Imaging apparatus

Abstract

To perform appropriate processing in regions with different imaging conditions.SOLUTION: An imaging apparatus includes: an imaging element including a first imaging region configured to image a subject under a first imaging condition, a second imaging region configured to image a subject under a second imaging condition different from the first imaging condition, and a third imaging region configured to image a subject under the first imaging condition; and a selection unit for selecting a pixel to be used for reducing signal noise from a pixel included in the first imaging region from among the pixels included in the second imaging region and the pixels included in the third imaging region. The second imaging region is a region adjacent to the first imaging region. The third imaging region is a region that is different from the first imaging region and is adjacent to at least one of the first imaging region and the second imaging region.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging device.

Background Art

[0002] There is known an imaging device equipped with an imaging element capable of setting different imaging conditions for each area of a screen (see Patent Document 1). However, when using image data generated in areas with different imaging conditions, there has been a problem that it cannot be done in the same way as when using image data generated in areas with the same imaging conditions.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] An imaging device according to a first aspect of the present invention includes an imaging element having a first imaging area set to image a subject under a first imaging condition, a second imaging area set to image a subject under a second imaging condition different from the first imaging condition, and a third imaging area set to image a subject under the first imaging condition, and a selection unit that selects pixels to be used for reducing noise of signals from the pixels included in the first imaging area from among the pixels included in the second imaging area and the pixels included in the third imaging area. The second imaging area is an area adjacent to the first imaging area, the third imaging area is an area different from the first imaging area and is an area adjacent to at least one of the first imaging area and the second imaging area.

Brief Description of the Drawings

[0005]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

MODE FOR CARRYING OUT THE INVENTION

[0006] As an example of an electronic device equipped with the image processing apparatus according to the present embodiment, a digital camera will be described as an example. The camera 1 (FIG. 1) is configured to be capable of imaging under different conditions for each region of the imaging surface in the imaging device 32a. The image processing unit 33 performs appropriate processing for each region where the imaging conditions are different. Details of such a camera 1 will be described with reference to the drawings.

[0007] <Explanation of the camera> (First Embodiment) FIG. 1 is a block diagram illustrating the configuration of the camera 1 according to the first embodiment. In FIG. 1, the camera 1 includes an imaging optical system 31, an imaging unit 32, an image processing unit 33, a control unit 34, a display unit 35, an operation member 36, and a recording unit 37.

[0008] The imaging optical system 31 guides a light beam from the object field to the imaging unit 32. The imaging unit 32 includes an imaging device 32a and a driving unit 32b, and photoelectrically converts the image of the subject formed by the imaging optical system 31. The imaging unit 32 can image under the same conditions over the entire imaging surface of the imaging device 32a, or can image under different conditions for each region of the imaging surface in the imaging device 32a. Details of the imaging unit 32 will be described later. The driving unit 32b generates a driving signal necessary to cause the imaging device 32a to perform accumulation control. Imaging instructions such as the charge accumulation time for the imaging unit 32 are transmitted from the control unit 34 to the driving unit 32b.

[0009] The image processing unit 33 includes an input unit 33a, a selection unit 33b, and a generation unit 33c. The input unit 33a receives the image data acquired by the imaging unit 32. The selection unit 33b performs preprocessing on the input image data. Details of the preprocessing will be described later. The generation unit 33c generates an image based on the input image data and the preprocessed image data. Also, the generation unit 33c performs image processing on the image data. The image processing includes, for example, color interpolation processing, pixel defect correction processing, edge enhancement processing, noise reduction processing, white balance adjustment processing, gamma correction processing, display brightness adjustment processing, saturation adjustment processing, etc. Further, the generation unit 33c generates an image to be displayed by the display unit 35.

[0010] The control unit 34 is configured by, for example, a CPU, and controls the overall operation of the camera 1. For example, the control unit 34 performs a predetermined exposure calculation based on the photoelectric conversion signal acquired by the imaging unit 32, and determines exposure conditions such as the charge accumulation time (exposure time) of the imaging element 32a, the aperture value of the imaging optical system 31, and the ISO sensitivity required for proper exposure, and instructs the drive unit 32b. Also, according to the imaging scene mode set in the camera 1 and the type of the detected subject element, it determines image processing conditions for adjusting saturation, contrast, sharpness, etc., and instructs the image processing unit 33. Detection of the subject element will be described later.

[0011] The control unit 34 includes an object detection unit 34a, a setting unit 34b, an imaging control unit 34c, and an AF calculation unit 34d. These are realized software-wise by the control unit 34 executing a program stored in a non-volatile memory (not shown), but they may be configured by an ASIC or the like.

[0012] The object detection unit 34a performs known object recognition processing to detect subject elements such as a person (human face), animals such as a dog or a cat (animal face), plants, a bicycle, a vehicle such as a car or a train, a building, a stationary object, a landscape such as a mountain or a cloud, and a predetermined specific object from the image acquired by the imaging unit 32. The setting unit 34b divides the imaging screen by the imaging unit 32 into a plurality of regions including the subject elements detected as described above.

[0013] The setting unit 34b further sets imaging conditions for the plurality of regions. The imaging conditions include the above exposure conditions (charge accumulation time, gain, ISO sensitivity, frame rate, etc.) and the above image processing conditions (for example, white balance adjustment parameters, gamma correction curves, display luminance adjustment parameters, saturation adjustment parameters, etc.). Note that it is possible to set the same imaging conditions for all of the plurality of regions or to set different imaging conditions among the plurality of regions.

[0014] The imaging control unit 34c controls the imaging unit 32 (imaging element 32a) and the image processing unit 33 by applying the imaging conditions set for each region by the setting unit 34b. As a result, it is possible to cause the imaging unit 32 to perform imaging with different exposure conditions for each of the plurality of regions, and it is possible to cause the image processing unit 33 to perform image processing with different image processing conditions for each of the plurality of regions. The number of pixels constituting the region may be any number, for example, 1000 pixels or 1 pixel. Also, the number of pixels may be different between regions.

[0015] The AF calculation unit 34d controls an autofocus (AF) operation for focusing on a corresponding subject at a predetermined position (referred to as a focus detection position) of the imaging screen. The AF calculation unit 34d sends a drive signal for moving the focus lens of the imaging optical system 31 to the in-focus position to the drive unit 32b based on the calculation result. The process performed by the AF calculation unit 34d for autofocus adjustment is also called focus detection processing. Details of the focus detection processing will be described later.

[0016] The display unit 35 reproduces and displays images generated by the image processing unit 33, images that have been image-processed, images read by the recording unit 37, and the like. The display unit 35 also displays an operation menu screen, a setting screen for setting imaging conditions, and the like.

[0017] The operation member 36 is composed of various operation members such as a release button and a menu button. The operation member 36 sends an operation signal corresponding to each operation to the control unit 34. The operation member 36 includes a touch operation member provided on the display surface of the display unit 35.

[0018] The recording unit 37 records image data and the like on a recording medium composed of a memory card (not shown) and the like according to an instruction from the control unit 34. Further, the recording unit 37 reads out the image data recorded on the recording medium according to an instruction from the control unit 34.

[0019] <Description of the stacked image sensor> As an example of the above-described image sensor 32a, a stacked image sensor 100 will be described. FIG. 2 is a cross-sectional view of the image sensor 100. The image sensor 100 includes an imaging chip 111, a signal processing chip 112, and a memory chip 113. The imaging chip 111 is stacked on the signal processing chip 112. The signal processing chip 112 is stacked on the memory chip 113. The imaging chip 111 and the signal processing chip 112, and the signal processing chip 112 and the memory chip 113 are electrically connected by connection portions 109, respectively. The connection portions 109 are, for example, bumps or electrodes. The imaging chip 111 captures an optical image from a subject and generates image data. The imaging chip 111 outputs the image data from the imaging chip 111 to the signal processing chip 112. The signal processing chip 112 performs signal processing on the image data output from the imaging chip 111. The memory chip 113 has a plurality of memories and stores the image data. Note that the image sensor 100 may be composed of an imaging chip and a signal processing chip. When the image sensor 100 is composed of an imaging chip and a signal processing chip, the storage unit for storing the image data may be provided in the signal processing chip or may be provided separately from the image sensor 100.

[0020] As shown in FIG. 2, incident light is incident mainly in the +Z-axis direction indicated by the white arrow. Also, as shown in the coordinate axes, the left direction of the paper surface orthogonal to the Z-axis is the +X-axis direction, and the front direction of the paper surface orthogonal to the Z-axis and the X-axis is the +Y-axis direction. In some subsequent figures, the coordinate axes are displayed so that the orientation of each figure can be understood based on the coordinate axes of FIG. 2.

[0021] The imaging chip 111 is, for example, a CMOS image sensor. Specifically, the imaging chip 111 is a back-illuminated CMOS image sensor. The imaging chip 111 includes a microlens layer 101, a color filter layer 102, a passivation layer 103, a semiconductor layer 106, and a wiring layer 108. The imaging chip 111 is arranged in the order of the microlens layer 101, the color filter layer 102, the passivation layer 103, the semiconductor layer 106, and the wiring layer 108 in the +Z-axis direction.

[0022] The microlens layer 101 has a plurality of microlenses L. The microlenses L condense the incident light onto a photoelectric conversion unit 104 described later. The color filter layer 102 has a plurality of color filters F. The color filter layer 102 has a plurality of types of color filters F with different spectral characteristics. Specifically, the color filter layer 102 has a first filter (R) with spectral characteristics that mainly transmit light of a red component, a second filter (Gb, Gr) with spectral characteristics that mainly transmit light of a green component, and a third filter (B) with spectral characteristics that mainly transmit light of a blue component. The color filter layer 102 has, for example, the first filter, the second filter, and the third filter arranged in a Bayer array. The passivation layer 103 is composed of a nitride film or an oxide film and protects the semiconductor layer 106.

[0023] The semiconductor layer 106 has a photoelectric conversion unit 104 and a readout circuit 105. The semiconductor layer 106 has a plurality of photoelectric conversion units 104 between a first surface 106a which is the light incident surface and a second surface 106b on the opposite side of the first surface 106a. In the semiconductor layer 106, a plurality of photoelectric conversion units 104 are arranged in the X-axis direction and the Y-axis direction. The photoelectric conversion unit 104 has a photoelectric conversion function of converting light into electric charges. Also, the photoelectric conversion unit 104 accumulates the electric charges by the photoelectric conversion signal. The photoelectric conversion unit 104 is, for example, a photodiode. The semiconductor layer 106 has a readout circuit 105 on the second surface 106b side rather than the photoelectric conversion unit 104. In the semiconductor layer 106, a plurality of readout circuits 105 are arranged in the X-axis direction and the Y-axis direction. The readout circuit 105 is composed of a plurality of transistors, reads out the image data generated by the electric charges photoelectrically converted by the photoelectric conversion unit 104, and outputs it to the wiring layer 108.

[0024] The wiring layer 108 has a plurality of metal layers. The metal layer is, for example, an Al wiring, a Cu wiring, or the like. The wiring layer 108 outputs the image data read out by the readout circuit 105. The image data is output from the wiring layer 108 to the signal processing chip 112 via the connection part 109.

[0025] Note that the connection part 109 may be provided for each photoelectric conversion unit 104. Also, the connection part 109 may be provided for a plurality of photoelectric conversion units 104. When the connection part 109 is provided for a plurality of photoelectric conversion units 104, the pitch of the connection part 109 may be larger than the pitch of the photoelectric conversion unit 104. Also, the connection part 109 may be provided in the peripheral area of the area where the photoelectric conversion unit 104 is arranged.

[0026] The signal processing chip 112 has a plurality of signal processing circuits. The signal processing circuit performs signal processing on the image data output from the imaging chip 111. The signal processing circuit is, for example, an amplifier circuit that amplifies the signal value of the image data, a correlated double sampling circuit that performs noise reduction processing on the image data, and an analog / digital (A / D) conversion circuit that converts an analog signal into a digital signal, etc. The signal processing circuit may be provided for each photoelectric conversion unit 104.

[0027] Further, the signal processing circuit may be provided for each of the plurality of photoelectric conversion units 104. The signal processing chip 112 has a plurality of through electrodes 110. The through electrodes 110 are, for example, silicon through electrodes. The through electrodes 110 connect the circuits provided in the signal processing chip 112 to each other. The through electrodes 110 may also be provided in the peripheral region of the imaging chip 111 and the memory chip 113. Note that some elements constituting the signal processing circuit may be provided in the imaging chip 111. For example, in the case of an analog / digital conversion circuit, a comparator that compares an input voltage with a reference voltage may be provided in the imaging chip 111, and circuits such as a counter circuit and a latch circuit may be provided in the signal processing chip 112.

[0028] The memory chip 113 has a plurality of storage units. The storage unit stores the image data on which signal processing has been performed by the signal processing chip 112. The storage unit is, for example, a volatile memory such as a DRAM. The storage unit may be provided for each of the photoelectric conversion units 104. Further, the storage unit may be provided for each of the plurality of photoelectric conversion units 104. The image data stored in the storage unit is output to the subsequent image processing unit.

[0029] FIG. 3 is a diagram for explaining the pixel array and the unit area 131 of the imaging chip 111. In particular, a state of observing the imaging chip 111 from the back surface (imaging surface) side is shown. For example, more than 20 million pixels are arranged in a matrix in the pixel region. In the example of FIG. 3, four pixels of 2 adjacent pixels × 2 pixels form one unit area 131. The grid lines in the figure indicate the concept that adjacent pixels are grouped to form the unit area 131. The number of pixels forming the unit area 131 is not limited to this, and may be about 1000, for example, 32 pixels × 32 pixels, more than that, less than that, or even 1 pixel.

[0030] As shown in the partial enlarged view of the pixel region, the unit region 131 in FIG. 3 includes a so-called Bayer array consisting of four pixels: green pixels Gb and Gr, a blue pixel B, and a red pixel R. The green pixels Gb and Gr are pixels having a green filter as the color filter F, and receive light in the green wavelength band of the incident light. Similarly, the blue pixel B is a pixel having a blue filter as the color filter F and receives light in the blue wavelength band, and the red pixel R is a pixel having a red filter as the color filter F and receives light in the red wavelength band.

[0031] In the present embodiment, a plurality of blocks are defined so as to include at least one unit region 131 per block. That is, the minimum unit of one block is one unit region 131. As described above, among the values that can be taken as the number of pixels forming one unit region 131, the smallest number of pixels is one pixel. Therefore, when a block is defined in pixel units, the smallest number of pixels that can define one block is one pixel. Each block can control the pixels included in each block with different control parameters. Each block has all the unit regions 131 within that block, that is, all the pixels within that block controlled under the same imaging conditions. That is, it is possible to obtain photoelectric conversion signals with different imaging conditions between a pixel group included in a certain block and a pixel group included in another block. Examples of the control parameters are frame rate, gain, decimation rate, number of addition rows or addition columns for adding photoelectric conversion signals, charge accumulation time or accumulation times, number of bits (word length) for digitization, and the like. The imaging device 100 can freely perform decimation not only in the row direction (X-axis direction of the imaging chip 111) but also in the column direction (Y-axis direction of the imaging chip 111). Furthermore, the control parameter may be a parameter in image processing.

[0032] FIG. 4 is a diagram for explaining the circuit in the unit region 131. In the example of FIG. 4, one unit region 131 is formed by four adjacent 2×2 pixels. Note that as described above, the number of pixels included in the unit region 131 is not limited to this, and may be 1000 pixels or more, or may be at least one pixel. The two-dimensional position of the unit region 131 is indicated by symbols A to D.

[0033] The reset transistor (RST) of the pixel included in the unit area 131 is configured to be individually turned on and off for each pixel. In FIG. 4, a reset wiring 300 for turning on and off the reset transistor of pixel A is provided, and a reset wiring 310 for turning on and off the reset transistor of pixel B is provided separately from the reset wiring 300. Similarly, a reset wiring 320 for turning on and off the reset transistor of pixel C is provided separately from the reset wirings 300 and 310. For other pixel D, a dedicated reset wiring 330 for turning on and off the reset transistor is provided.

[0034] Regarding the transfer transistor (TX) of the pixel included in the unit area 131, it is also configured to be individually turned on and off for each pixel. In FIG. 4, a transfer wiring 302 for turning on and off the transfer transistor of pixel A, a transfer wiring 312 for turning on and off the transfer transistor of pixel B, and a transfer wiring 322 for turning on and off the transfer transistor of pixel C are provided separately. For other pixel D, a dedicated transfer wiring 332 for turning on and off the transfer transistor is provided.

[0035] Furthermore, regarding the selection transistor (SEL) of the pixel included in the unit area 131, it is also configured to be individually turned on and off for each pixel. In FIG. 4, a selection wiring 306 for turning on and off the selection transistor of pixel A, a selection wiring 316 for turning on and off the selection transistor of pixel B, and a selection wiring 326 for turning on and off the selection transistor of pixel C are provided separately. For other pixel D, a dedicated selection wiring 336 for turning on and off the selection transistor is provided.

[0036] Note that the power supply wiring 304 is commonly connected from pixel A to pixel D included in the unit area 131. Similarly, the output wiring 308 is commonly connected from pixel A to pixel D included in the unit area 131. Also, the power supply wiring 304 is commonly connected between a plurality of unit areas, while the output wiring 308 is provided individually for each unit area 131. The load current source 309 supplies current to the output wiring 308. The load current source 309 may be provided on the imaging chip 111 side or on the signal processing chip 112 side.

[0037] By individually turning on and off the reset transistor and the transfer transistor in the unit area 131, it is possible to control the charge accumulation including the charge accumulation start time, the charge accumulation end time, and the transfer timing for pixels A to D included in the unit area 131. Also, by individually turning on and off the selection transistors in the unit area 131, the photoelectric conversion signals of each of pixels A to D can be output via the common output wiring 308.

[0038] Here, a so-called rolling shutter method is known in which charge accumulation is controlled in a regular order with respect to rows and columns for pixels A to D included in the unit area 131. When pixels are selected row by row and then columns are specified by the rolling shutter method, the photoelectric conversion signals are output in the order of "ABCD" in the example of FIG. 4.

[0039] By configuring the circuit based on the unit area 131 in this way, the charge accumulation time can be controlled for each unit area 131. In other words, it is possible to output photoelectric conversion signals with different frame rates between the unit areas 131. Also, while causing the unit areas 131 included in some blocks in the imaging chip 111 to perform charge accumulation (imaging), by resting the unit areas 131 included in other blocks, imaging can be performed only in a predetermined block of the imaging chip 111, and the photoelectric conversion signal can be output. Furthermore, by switching the block (the block to be subjected to accumulation control) that performs charge accumulation (imaging) between frames, sequential imaging can be performed in different blocks of the imaging chip 111, and the photoelectric conversion signal can be output.

[0040] As described above, output wirings 308 are provided corresponding to each of the unit areas 131. Since the imaging device 100 stacks the imaging chip 111, the signal processing chip 112, and the memory chip 113, by using the electrical connection between chips using the connection portion 109 for these output wirings 308, the wirings can be routed without increasing the chips in the plane direction.

[0041] <Block Control of Imaging Device> In the present embodiment, the imaging conditions can be set for each of a plurality of blocks in the imaging device 32a. The control unit 34 (imaging control unit 34c) causes imaging to be performed under the imaging conditions set for each region by associating the plurality of regions with the blocks.

[0042] FIG. 5 is a diagram schematically showing an image of a subject imaged on the imaging device 32a of the camera 1. Before an imaging instruction is given, the camera 1 photoelectrically converts the subject image to obtain a live view image. The live view image refers to an image for monitoring that is repeatedly imaged at a predetermined frame rate (for example, 60 fps).

[0043] Before the control unit 34 divides the area by the setting unit 34b, it sets the same imaging conditions for the entire area of the imaging chip 111 (i.e., the entire imaging screen). The same imaging conditions mean setting imaging conditions common to the entire imaging screen. For example, even if there is a variation of less than about 0.3 steps in the apex value, it is regarded as the same. The imaging conditions set identically across the entire area of the imaging chip 111 are determined based on the exposure conditions corresponding to the photometric value of the subject brightness or the exposure conditions manually set by the user.

[0044] In FIG. 5, an image including a person 61a, an automobile 62a, a bag 63a, a mountain 64a, and clouds 65a, 66a is formed on the imaging surface of the imaging chip 111. The person 61a is holding the bag 63a with both hands. An automobile 62a is stopped right behind the person 61a.

[0045] <Division of the area> Based on the live view image, the control unit 34 divides the screen of the live view image into a plurality of areas as follows. First, the object detection unit 34a detects subject elements from the live view image. The detection of subject elements uses known subject recognition techniques. In the example of FIG. 5, the object detection unit 34a detects the person 61a, the automobile 62a, the bag 63a, the mountain 64a, the cloud 65a, and the cloud 66a as subject elements.

[0046] Next, the setting unit 34b divides the screen of the live view image into areas including the above subject elements. In the present embodiment, the area including the person 61a is referred to as area 61, the area including the automobile 62a is referred to as area 62, the area including the bag 63a is referred to as area 63, the area including the mountain 64a is referred to as area 64, the area including the cloud 65a is referred to as area 65, and the area including the cloud 66a is referred to as area 66 for explanation.

[0047] <Setting of imaging conditions for each block> When the control unit 34 divides the screen into a plurality of areas by the setting unit 34b, it causes the display unit 35 to display a setting screen as illustrated in FIG. 6. In FIG. 6, a live view image 60a is displayed, and a setting screen 70 for imaging conditions is displayed on the right side of the live view image 60a.

[0048] In the setting screen 70, as an example of the setting items for the imaging conditions, the frame rate, shutter speed (TV), and gain (ISO) are listed in order from the top. The frame rate is the number of frames of the live view image acquired per second or the moving image recorded by the camera 1. The gain is the ISO sensitivity. The setting items for the imaging conditions may be appropriately added in addition to those illustrated in FIG. 6. When all the setting items do not fit within the setting screen 70, other setting items may be displayed by scrolling the setting items up and down.

[0049] In the present embodiment, the control unit 34 sets the area selected by the user among the areas divided by the setting unit 34b as the target for setting (changing) the imaging conditions. For example, in the camera 1 capable of touch operation, the user taps the display position of the main subject for which the imaging conditions are to be set (changed) on the display surface of the display unit 35 on which the live view image 60a is displayed. When the display position of the person 61a is tapped, for example, the control unit 34 sets the area 61 including the person 61a in the live view image 60a as the target area for setting (changing) the imaging conditions, and emphasizes and displays the outline of the area 61.

[0050] In FIG. 6, an area 61 where the outline is emphasized and displayed (displayed thickly, brightly, with a color change, with a broken line, blinking display, etc.) indicates an area that is the target of setting (changing) the imaging conditions. In the example of FIG. 6, it is assumed that a live view image 60a with the outline of area 61 emphasized is being displayed. In this case, area 61 is the target of setting (changing) the imaging conditions. For example, in camera 1 that enables touch operations, when the user taps on the display 71 of the shutter speed (TV), the control unit 34 displays the current set value of the shutter speed for the area being emphasized and displayed (area 61) on the screen (reference numeral 68). In the following description, the explanation of camera 1 is based on the premise of touch operations, but the imaging conditions may be set (changed) by operating buttons or the like that make up the operation member 36.

[0051] When the upper icon 71a or the lower icon 71b of the shutter speed (TV) is tapped by the user, the setting unit 34b increases or decreases the display 68 of the shutter speed from the current set value according to the above tap operation, and sends an instruction to the imaging unit 32 (FIG. 1) to change the imaging conditions of the unit area 131 (FIG. 3) of the imaging element 32a corresponding to the area being emphasized and displayed (area 61) according to the above tap operation. The determination icon 72 is an operation icon for finalizing the set imaging conditions. The setting unit 34b performs the setting (changing) of the frame rate and gain (ISO) in the same manner as in the case of setting (changing) the shutter speed (TV).

[0052] Note that although the setting unit 34b has been described as setting the imaging conditions based on the user's operation, it is not limited to this. The setting unit 34b may set the imaging conditions based on the determination of the control unit 34 without relying on the user's operation. For example, in an area including a subject element that is the maximum or minimum luminance in the image, when blooming or black clipping occurs, the setting unit 34b may set the imaging conditions to eliminate blooming or black clipping based on the determination of the control unit 34. For regions that are not highlighted (regions other than region 61), the set imaging conditions are maintained.

[0053] Instead of highlighting the contour of the region targeted for imaging condition setting (modification), the control unit 34 may display the entire target region brightly, increase the contrast of the entire target region for display, or cause the entire target region to blink. Also, the target region may be surrounded by a frame. The display of the frame surrounding the target region may be a double frame or a single frame, and the display modes such as the line type, color, and brightness of the surrounding frame may be changed as appropriate. Further, the control unit 34 may display an indication such as an arrow near the target region to indicate the region targeted for imaging condition setting. The control unit 34 may also display the area other than the target region targeted for imaging condition setting (modification) darkly or reduce the contrast of the area other than the target region for display.

[0054] As described above, after the imaging conditions for each region are set, when a release button (not shown) constituting the operation member 36 or a display (release icon) instructing the start of imaging is operated, the control unit 34 controls the imaging unit 32 to cause imaging to be performed on the divided regions with the imaging conditions set for each. Then, the image processing unit 33 performs image processing on the image data acquired by the imaging unit 32. The image processing can be performed under different image processing conditions for each region as described above.

[0055] After the image processing by the image processing unit 33, the recording unit 37 instructed by the control unit 34 records the image data after the image processing on a recording medium constituted by a memory card (not shown) or the like. Thereby, a series of imaging processes is completed.

[0056] <Data Selection Process> As described above, in the present embodiment, after the imaging screen area is divided by the setting unit 34b, imaging conditions can be set (changed) for the area selected by the user or the area determined by the control unit 34. When different imaging conditions are set for the divided areas, the control unit 34 causes the following data selection process to be performed as necessary.

[0057] 1. When performing image processing When the image processing for the image data obtained by applying different imaging conditions among the divided areas by the image processing unit 33 (selection unit 33b) is a predetermined image processing, the data selection process is performed on the image data located near the boundary of the area as a pre-process of the image processing. The predetermined image processing is a process of calculating the image data of the attention position to be processed in the image with reference to the image data of a plurality of reference positions around the attention position. For example, pixel defect correction processing, color interpolation processing, edge enhancement processing, noise reduction processing, etc. are applicable.

[0058] The data selection process is performed to alleviate the discomfort that appears in the image after image processing due to different imaging conditions among the divided areas. When the attention position is located near the boundary of the divided area, there may be a mixture of image data to which the same imaging conditions as the image data of the attention position are applied and image data to which different imaging conditions from the image data of the attention position are applied among the plurality of reference positions around the attention position. In the present embodiment, based on the idea that it is preferable to calculate the image data of the attention position with reference to the image data of the reference position to which the same imaging conditions as the attention position are applied rather than calculating the image data of the attention position by directly referring to the image data of the reference position to which different imaging conditions are applied, the data used for image processing is selected as follows.

[0059] FIG. 7(a) is a diagram illustrating a region 80 near the boundary between a region 61 and a region 64 in a live view image 60a. In this example, it is assumed that a first imaging condition is set for at least the region 61 including a person, and a second imaging condition is set for the region 64 including a mountain. FIG. 7(b) is an enlarged view of the region 80 near the boundary of FIG. 7(a). Image data from pixels on the imaging element 32a corresponding to the region 61 where the first imaging condition is set is shown in white, and image data from pixels on the imaging element 32a corresponding to the region 64 where the second imaging condition is set is shown hatched. In FIG. 7(b), the image data from the target pixel P is located at a boundary portion that is on the region 61 and near the boundary 81 between the region 61 and the region 64. Pixels (8 pixels in this example) around the target pixel P included in a predetermined range 90 (for example, 3×3 pixels) centered on the target pixel P are used as reference pixels. FIG. 7(c) is an enlarged view of the target pixel P and the reference pixels. The position of the target pixel P is the target position, and the positions of the reference pixels surrounding the target pixel P are the reference positions.

[0060] The image processing unit 33 (generation unit 33c) can also perform image processing by directly referring to the image data of the reference pixels without performing data selection processing. However, when the imaging condition applied to the target pixel P (assumed to be the first imaging condition) is different from the imaging condition applied to the reference pixels around the target pixel P (assumed to be the second imaging condition), the selection unit 33b selects the image data of the first imaging condition for use in image processing from the image data of the reference pixels as follows in (Example 1) to (Example 3). Then, the generation unit 33c performs image processing to calculate the image data of the target pixel P by referring to the image data of the reference pixels after the image data selection. In FIG. 7(c), the data output from the pixels shown in white is the image data of the first imaging condition, and the data output from the pixels shown hatched is the image data of the second imaging condition. In this embodiment, by not selecting the image data of the second imaging condition, the image data output from the pixels shown hatched is not used for image processing.

[0061] (Example 1) The image processing unit 33 (selection unit 33b) selects, for example, the image data of the first imaging condition for use in image processing from the image data of the reference pixels when only the ISO sensitivity differs between the first imaging condition and the second imaging condition, the ISO sensitivity of the first imaging condition is 100, and the ISO sensitivity of the second imaging condition is 800. That is, the image data of the second imaging condition, which is different from the first imaging condition, among the image data of the reference pixels is not used for image processing.

[0062] (Example 2) The image processing unit 33 (selection unit 33b) selects, for example, the image data of the first imaging condition for use in image processing from the image data of the reference pixels when only the shutter speed differs between the first imaging condition and the second imaging condition, the shutter speed of the first imaging condition is 1 / 1000 second, and the shutter speed of the second imaging condition is 1 / 100 second. That is, the image data of the second imaging condition, which is different from the first imaging condition, among the image data of the reference pixels is not used for image processing.

[0063] (Example 3) The image processing unit 33 (selection unit 33b) selects, for example, the image data of the frame image with an acquisition timing close to that of the frame image acquired under the first imaging condition (30 fps) for the image data of the second imaging condition (60 fps) among the image data of the reference pixels when only the frame rate differs between the first imaging condition and the second imaging condition (the charge accumulation time is the same), the frame rate of the first imaging condition is 30 fps, and the frame rate of the second imaging condition is 60 fps. That is, the image data of the frame image, which has a different acquisition timing from the frame image of the first imaging condition (30 fps), among the image data of the reference pixels is not used for image processing.

[0064] On the other hand, the image processing unit 33 (selection unit 33b) selects all the image data when the imaging condition applied to the target pixel P is the same as the imaging conditions applied to all the reference pixels around the target pixel P. For example, when it is determined that the same imaging condition as the imaging condition applied to the target pixel P is applied to all the reference pixels, image processing is performed to calculate the image data of the target pixel P by referring to the image data of all the reference pixels as they are. In addition, as described above, even if there are some differences in the imaging conditions (for example, within 0.3 steps in terms of the apex value), it may be regarded as the same imaging conditions.

[0065] <Exemplification of Image Processing> Exemplify image processing involving data selection processing. (1) Pixel Defect Correction Processing In the present embodiment, the pixel defect correction processing is one of the image processing performed during imaging. Generally, the imaging element 100, which is a solid-state imaging element, may have pixel defects during the manufacturing process or after manufacturing, and may output abnormal-level image data. Therefore, the image processing unit 33 (generation unit 33c) corrects the image data output from the pixels where pixel defects have occurred, so that the image data at the pixel positions where pixel defects have occurred is not conspicuous.

[0066] Describe an example of pixel defect correction processing. The image processing unit 33 (generation unit 33c) designates, for example, the pixels at the positions of pixel defects recorded in a non-volatile memory (not shown in advance) in an image of one frame as the target pixels P (pixels to be processed), and uses the pixels around the target pixel P (8 pixels in this example) included in a predetermined range 90 (for example, 3×3 pixels) (Fig. 7(c)) centered on the target pixel P as reference pixels.

[0067] The image processing unit 33 (generation unit 33c) calculates the maximum value and minimum value of the image data in the reference pixels, and when the image data output from the target pixel P exceeds these maximum values or minimum values, performs Max, Min filter processing to replace the image data output from the target pixel P with the above maximum value or minimum value. Perform such processing for all pixel defects whose position information is recorded in the non-volatile memory.

[0068] In the present embodiment, when the pixel to which the second imaging condition different from the first imaging condition applied to the target pixel P is included in the reference pixel, the image processing unit 33 (selection unit 33b) selects the image data to which the first imaging condition is applied from the image data in the reference pixel. Thereafter, the image processing unit 33 (generation unit 33c) performs the above-described Max, Min filter processing with reference to the selected image data. Note that it may be possible to perform pixel defect correction processing by taking the average of the selected image data.

[0069] (2) Color interpolation processing In the present embodiment, the color interpolation processing is one of the image processes performed during imaging. As illustrated in FIG. 3, in the imaging chip 111 of the image sensor 100, green pixels Gb, Gr, blue pixels B, and red pixels R are arranged in a Bayer array. Since the image data of color components different from the color components of the color filter F arranged at each pixel position is insufficient, the image processing unit 33 (generation unit 33c) performs color interpolation processing to generate the image data of the insufficient color components with reference to the image data at the surrounding pixel positions.

[0070] An example of the color interpolation processing will be described. FIG. 8(a) is a diagram illustrating the arrangement of the image data output from the image sensor 100. Corresponding to each pixel position, it has a color component of either R, G, or B according to the rule of the Bayer array. <G color interpolation> First, general G-color interpolation will be described. The image processing unit 33 (generation unit 33c) that performs G-color interpolation uses the positions of the R-color component and the B-color component as the target positions in order, and refers to the image data of the four G-color components at the reference positions around the target position to generate the image data of the G-color component at the target position. For example, when generating the image data of the G-color component at the target position indicated by the thick frame in FIG. 8(b) (the second row and the second column counted from the upper left position; hereinafter, the target position counted from the upper left position shall be the same), the image data G1 to G4 of the four G-color components located in the vicinity of the target position (the second row and the second column) are referred to. The image processing unit 33 (generation unit 33c) sets, for example, (aG1 + bG2 + cG3 + dG4) / 4 as the image data of the G-color component at the target position (the second row and the second column). Note that a to d are weight coefficients provided according to the distance between the reference position and the target position and the image structure.

[0071] Next, the G-color interpolation of the present embodiment will be described. In FIGS. 8(a) to 8(c), it is assumed that the first imaging condition is applied to the regions to the left and above the thick line, and the second imaging condition is applied to the regions to the right and below the thick line. Note that in FIGS. 8(a) to 8(c), the first imaging condition and the second imaging condition are different. Also, the image data G1 to G4 of the G-color component in FIG. 8(b) are reference positions for image-processing the pixel at the target position (the second row and the second column). In FIG. 8(b), the first imaging condition is applied to the target position (the second row and the second column). Among the reference positions, the first imaging condition is applied to the image data G1 to G3. Also, among the reference positions, the second imaging condition is applied to the image data G4. Therefore, the image processing unit 33 (selection unit 33b) selects the image data G1 to G3 to which the first imaging condition is applied from the image data G1 to G4 of the G-color component. In this way, the image processing unit 33 (generation unit 33c) calculates the image data of the G-color component at the target position (the second row and the second column) with reference to the selected image data. The image processing unit 33 (generation unit 33c) sets, for example, (a1G1 + b1G2 + c1G3) / 3 as the image data of the G-color component at the target position (the second row and the second column). Note that a1 to c1 are weight coefficients provided according to the distance between the reference position and the target position and the image structure.

[0072] The image processing unit 33 (generation unit 33c) generates the image data of the G color component at the positions of the B color component and the R color component in FIG. 8(a), respectively, so that the image data of the G color component can be obtained at each pixel position as shown in FIG. 8(c).

[0073] <R color interpolation> FIG. 9(a) is a diagram in which the image data of the R color component is extracted from FIG. 8(a). The image processing unit 33 (generation unit 33c) calculates the image data of the color difference component Cr shown in FIG. 9(b) based on the image data of the G color component shown in FIG. 8(c) and the image data of the R color component shown in FIG. 9(a).

[0074] First, the interpolation of the general color difference component Cr will be described. When the image processing unit 33 (generation unit 33c) generates the image data of the color difference component Cr at the target position indicated by the thick frame (second row and second column) in FIG. 9(b), for example, it refers to the image data Cr1 to Cr4 of the four color difference components located near the target position (second row and second column). The image processing unit 33 (generation unit 33c) uses, for example, (eCr1 + fCr2 + gCr3 + hCr4) / 4 as the image data of the color difference component Cr at the target position (second row and second column). Here, e to h are weight coefficients provided according to the distance between the reference position and the target position and the image structure.

[0075] Similarly, when the image processing unit 33 (generation unit 33c) generates the image data of the color difference component Cr at the target position indicated by the thick frame (second row and third column) in FIG. 9(c), for example, it refers to the image data Cr2, Cr4 to Cr6 of the four color difference components located near the target position (second row and third column). The image processing unit 33 (generation unit 33c) uses, for example, (qCr2 + rCr4 + sCr5 + tCr6) / 4 as the image data of the color difference component Cr at the target position (second row and third column). Here, q to t are weight coefficients provided according to the distance between the reference position and the target position and the image structure. In this way, the image data of the color difference component Cr is generated for each pixel position.

[0076] Next, the interpolation of the color difference component Cr of the present embodiment will be described. In FIGS. 9(a) to 9(c), for example, it is assumed that the first imaging condition is applied to the regions to the left and above the thick line, and the second imaging condition is applied to the regions to the right and below the thick line. Note that in FIGS. 9(a) to 9(c), the first imaging condition and the second imaging condition are different. In FIG. 9(b), the position indicated by the thick frame (second row, second column) is the position of interest of the color difference component Cr. Also, the image data Cr1 to Cr4 of the color difference components in FIG. 9(b) are reference positions for image processing of the pixel at the position of interest (second row, second column). In FIG. 9(b), the first imaging condition is applied to the position of interest (second row, second column). Among the reference positions, the first imaging condition is applied to the image data Cr1, Cr3, and Cr4. Also, the second imaging condition is applied to the image data Cr2 among the reference positions. Therefore, the image processing unit 33 (selection unit 33b) selects the image data Cr1, Cr3, and Cr4 to which the first imaging condition is applied from the image data Cr1 to Cr4 of the color difference component Cr. Thereafter, the image processing unit 33 (generation unit 33c) calculates the image data Cr of the color difference component at the position of interest (second row, second column) with reference to the selected image data. The image processing unit 33 (generation unit 33c) sets, for example, (e1Cr1 + g1Cr3 + h1Cr4) / 3 as the image data of the color difference component Cr at the position of interest (second row, second column). Note that e1, g1, and h1 are weighting factors provided according to the distance between the reference position and the position of interest and the image structure.

[0077] In addition, in FIG. 9(c), the position indicated by the thick frame (second row, third column) is the position of interest for the chrominance component Cr. Also, the image data Cr2, Cr4, Cr5, Cr6 of the chrominance components in FIG. 9(c) are reference positions for image processing of the pixels at the position of interest (second row, third column). In FIG. 9(c), the second imaging condition is applied to the position of interest (second row, third column). Among the reference positions, the first imaging condition is applied to the image data Cr4 and Cr5. Also, among the reference positions, the second imaging condition is applied to the image data Cr2 and Cr6. Therefore, the image processing unit 33 (selection unit 33b) selects the image data Cr2 and Cr6 to which the second imaging condition is applied from the image data Cr2, Cr4 to Cr6 of the chrominance component Cr in the same manner as in the example of FIG. 9(b) described above. After that, the image processing unit 33 (generation unit 33c) calculates the image data Cr of the chrominance component at the position of interest (second row, third column) with reference to the selected image data. The image processing unit 33 (generation unit 33c) sets, for example, (g2Cr2 + h2Cr6) / 2 as the image data of the chrominance component Cr at the position of interest (second row, third column). Note that g2 and h2 are weighting coefficients provided according to the distance between the reference position and the position of interest and the image structure.

[0078] After the image processing unit 33 (generation unit 33c) obtains the image data of the chrominance component Cr at each pixel position, the image data of the R color component can be obtained at each pixel position by adding the image data of the G color component shown in FIG. 8(c) corresponding to each pixel position.

[0079] FIG. 10(a) is a diagram in which the image data of the B color component is extracted from FIG. 8(a). The image processing unit 33 (generation unit 33c) calculates the image data of the chrominance component Cb shown in FIG. 10(b) based on the image data of the G color component shown in FIG. 8(c) and the image data of the B color component shown in FIG. 10(a).

[0080] First, the interpolation of the general color difference component Cb will be described. When the image processing unit 33 (generation unit 33c) generates the image data of the color difference component Cb at the attention position indicated by the thick frame (the third row and the third column) in FIG. 10(b), for example, the image data Cb1 to Cb4 of four color difference components located in the vicinity of the attention position (the third row and the third column) are referred to. The image processing unit 33 (generation unit 33c) sets, for example, (uCb1 + vCb2 + wCb3 + xCb4) / 4 as the image data of the color difference component Cb at the attention position (the third row and the third column). Here, u to x are weighting coefficients provided according to the distance between the reference position and the attention position and the image structure.

[0081] Similarly, when the image processing unit 33 (generation unit 33c) generates the image data of the color difference component Cb at the attention position indicated by the thick frame (the third row and the fourth column) in FIG. 10(c), for example, the image data Cb2, Cb4 to Cb6 of four color difference components located in the vicinity of the attention position (the third row and the fourth column) are referred to. The image processing unit 33 (generation unit 33c) sets, for example, (yCb2 + zCb4 + αCb5 + βCb6) / 4 as the image data of the color difference component Cb at the attention position (the third row and the fourth column). Here, y, z, α, and β are weighting coefficients provided according to the distance between the reference position and the attention position and the image structure. In this way, the image data of the color difference component Cb is generated for each pixel position.

[0082] Next, the interpolation of the color difference component Cb in the present embodiment will be described. In FIGS. 10(a) to 10(c), for example, it is assumed that the first imaging condition is applied to the regions to the left and above the thick line, and the second imaging condition is applied to the regions to the right and below the thick line. Note that in FIGS. 10(a) to 10(c), the first imaging condition and the second imaging condition are different. In FIG. 10(b), the position indicated by the thick frame (the third row and the third column) is the position of interest of the color difference component Cb. Also, the image data Cb1 to Cb4 of the color difference component in FIG. 10(b) are the reference positions for image processing of the pixel at the position of interest (the third row and the third column). In FIG. 10(b), the second imaging condition is applied to the position of interest (the third row and the third column). Among the reference positions, the first imaging condition is applied to the image data Cb1 and Cb3. Also, among the reference positions, the second imaging condition is applied to the image data Cb2 and Cb4. Therefore, the image processing unit 33 (selection unit 33b) selects the image data Cb2 and Cb4 to which the second imaging condition is applied from the image data Cb1 to Cb4 of the color difference component Cb. Thereafter, the image processing unit 33 (generation unit 33c) calculates the image data Cb of the color difference component at the position of interest (the third row and the third column) with reference to the selected image data. The image processing unit 33 (generation unit 33c) uses, for example, (v1Cb2 + x1Cb4) / 2 as the image data of the color difference component Cb at the position of interest (the third row and the third column). Note that v1 and x1 are weighting coefficients provided according to the distance between the reference position and the position of interest and the image structure.

[0083] The same applies to the case of generating the image data Cb of the color difference component for the position of interest (the third row and the fourth column) indicated by the thick frame in FIG. 10(c). In FIG. 10(c), the position indicated by the thick frame (the third row and fourth column) is the position of interest for the chrominance component Cb. Also, the image data Cb2, Cb4 to Cb6 of the chrominance components in FIG. 10(c) are the reference positions for image processing of the pixels at the position of interest (the third row and fourth column). In FIG. 10(c), the second imaging condition is applied to the position of interest (the third row and fourth column). Also, the second imaging condition is applied to the image data Cb2, Cb4 to Cb6 of all the reference positions. Therefore, the image processing unit 33 (generation unit 33c) calculates the image data of the chrominance component Cb at the position of interest (the third row and fourth column) with reference to the image data Cb2, Cb4 to Cb6 of the four chrominance components located near the position of interest (the third row and fourth column).

[0084] After obtaining the image data of the chrominance component Cb at each pixel position, the image processing unit 33 (generation unit 33c) can obtain the image data of the B color component at each pixel position by adding the image data of the G color component shown in FIG. 8(c) corresponding to each pixel position.

[0085] (3) Edge enhancement processing An example of edge enhancement processing will be described. The image processing unit 33 (generation unit 33c) performs a known linear filter operation using a kernel of a predetermined size centered on the pixel of interest P (the pixel to be processed) in, for example, one frame of the image. When the kernel size of the sharpening filter, which is an example of the linear filter, is N×N pixels, the position of the pixel of interest P is the position of interest, and the positions of the (N 2 -1) reference pixels surrounding the pixel of interest P are the reference positions. Note that the kernel size may be N×M pixels.

[0086] The image processing unit 33 (generation unit 33c) performs a filter process of replacing the image data at the pixel of interest P with the linear filter operation result, for example, from the upper horizontal line to the lower horizontal line of the frame image, while shifting the pixel of interest from left to right on each horizontal line.

[0087] In the present embodiment, when the pixel to which the second imaging condition different from the first imaging condition applied to the target pixel P is included in the reference pixels, the image processing unit 33 (selection unit 33b) selects the image data to which the first imaging condition is applied from the image data in the reference pixels. Thereafter, the image processing unit 33 (generation unit 33c) performs the above-described linear filter processing with reference to the selected image data.

[0088] (4) Noise reduction processing An example of noise reduction processing will be described. The image processing unit 33 (generation unit 33c) performs, for example, a known linear filter operation using a kernel of a predetermined size centered on the target pixel P (processing target pixel) in one frame of an image. When the kernel size of a smoothing filter, which is an example of a linear filter, is N × N pixels, the position of the target pixel P is the target position, and the positions of (N 2 -1) reference pixels surrounding the target pixel P are the reference positions. Note that the kernel size may be N × M pixels.

[0089] The image processing unit 33 (generation unit 33c) performs a filter process of replacing the image data at the target pixel P with the linear filter operation result, for example, from the upper horizontal line to the lower horizontal line of the frame image, while shifting the target pixel from left to right on each horizontal line.

[0090] In the present embodiment, when the pixel to which the second imaging condition different from the first imaging condition applied to the target pixel P is included in the reference pixels, the image processing unit 33 (selection unit 33b) selects the image data to which the first imaging condition is applied from the image data in the reference pixels. Thereafter, the image processing unit 33 (generation unit 33c) performs the above-described linear filter processing with reference to the selected image data. As described above, when the pixel to which the second imaging condition different from the first imaging condition applied to the target pixel P is included in the reference pixel, the image processing unit 33 (selection unit 33b) selects the image data to which the first imaging condition is applied from the image data in the reference pixel. Then, thereafter, the image processing unit 33 (generation unit 33c) performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing with reference to the selected image data. The image processing unit 33 (selection unit 33b) performs the same processing on the image data from other pixels output from the imaging device (including the image data to which the second imaging condition is applied) to generate an image. The generated image is displayed on a display unit such as a display device.

[0091] 2. When performing focus detection processing The control unit 34 (AF calculation unit 34d) performs focus detection processing using the image data corresponding to a predetermined position (focus detection position) on the imaging screen. When different imaging conditions are set between the divided regions and the focus detection position of the AF operation is located at the boundary portion of the divided regions, the control unit 34 (AF calculation unit 34d) performs data selection processing as a preprocessing of the focus detection processing on the image data for focus detection located near the boundary of the region.

[0092] The data selection processing is performed to suppress a decrease in the accuracy of the focus detection processing due to different imaging conditions between the regions of the imaging screen divided by the setting unit 34b. For example, when the image data for focus detection at the focus detection position for detecting the image shift amount (phase difference) in the image is located near the boundary of the divided regions, there may be a mixture of image data to which different imaging conditions are applied in the image data for focus detection. In the present embodiment, based on the idea that it is preferable to detect the image shift amount (phase difference) using the image data to which the same imaging condition is applied rather than detecting the image shift amount (phase difference) using the image data to which different imaging conditions are applied as it is, the data selection processing is performed as follows.

[0093] <Exemplification of focus detection processing> An example of focus detection processing involving data selection processing will be described. The AF operation of this embodiment focuses on a subject corresponding to a focus detection position selected by a user from among a plurality of focus detection positions. The control unit 34 (AF calculation unit 34d) calculates the defocus amount of the imaging optical system 31 by detecting the amount of image shift (phase difference) of a plurality of subject images formed by light beams passing through different pupil regions of the imaging optical system 31. The control unit 34 (AF calculation unit 34d) moves the focus lens of the imaging optical system 31 to a position where the defocus amount is zero (equal to or less than the allowable value), that is, the in-focus position.

[0094] FIG. 11 is a diagram illustrating the positions of the focus detection pixels on the imaging surface of the imaging device 32a. In this embodiment, the focus detection pixels are provided discretely along the X-axis direction (horizontal direction) of the imaging chip 111. In the example of FIG. 11, 15 focus detection pixel lines 160 are provided at predetermined intervals. The focus detection pixels constituting the focus detection pixel line 160 output photoelectric conversion signals for focus detection. Normal imaging pixels are provided at pixel positions other than the focus detection pixel line 160 in the imaging chip 111. The imaging pixels output photoelectric conversion signals for live view images and recording.

[0095] FIG. 12 is an enlarged view of a partial region of the focus detection pixel line 160 corresponding to the focus detection position 80A shown in FIG. 11. In FIG. 12, a red pixel R, green pixels G (Gb, Gr), and a blue pixel B, and focus detection pixels S1 and focus detection pixels S2 are illustrated. The red pixel R, green pixels G (Gb, Gr), and blue pixel B are arranged according to the rules of the above-described Bayer array.

[0096] The square regions exemplified for the red pixel R, green pixels G (Gb, Gr), and blue pixel B indicate the light receiving regions of the imaging pixels. Each imaging pixel receives a light beam passing through the exit pupil of the imaging optical system 31 (FIG. 1). That is, the red pixel R, green pixels G (Gb, Gr), and blue pixel B each have a square mask opening, and the light passing through these mask openings reaches the light receiving portion of the imaging pixel.

[0097] Note that the shapes of the light-receiving areas (mask openings) of the red pixels R, green pixels G (Gb, Gr), and blue pixels B are not limited to rectangles, and may be, for example, circular.

[0098] The semi-circular regions exemplified for the focus detection pixels S1 and S2 indicate the light-receiving areas of the focus detection pixels. That is, the focus detection pixel S1 has a semi-circular mask opening on the left side of the pixel position in FIG. 12, and the light passing through this mask opening reaches the light-receiving portion of the focus detection pixel S1. On the other hand, the focus detection pixel S2 has a semi-circular mask opening on the right side of the pixel position in FIG. 12, and the light passing through this mask opening reaches the light-receiving portion of the focus detection pixel S2. Thus, the focus detection pixel S1 and the focus detection pixel S2 respectively receive a pair of light beams passing through different regions of the exit pupil of the imaging optical system 31 (FIG. 1).

[0099] Note that the position of the focus detection pixel line 160 in the imaging chip 111 is not limited to the position exemplified in FIG. 11. Also, the number of focus detection pixel lines 160 is not limited to the example of FIG. 11. Further, the shapes of the mask openings in the focus detection pixels S1 and S2 are not limited to semi-circular, and may be, for example, rectangular shapes obtained by horizontally dividing the rectangular light-receiving areas (mask openings) in the imaging pixels R, imaging pixels G, and imaging pixels B.

[0100] Also, the focus detection pixel line 160 in the imaging chip 111 may be provided by arranging focus detection pixels along the Y-axis direction (vertical direction) of the imaging chip 111. Image sensors in which imaging pixels and focus detection pixels are arranged two-dimensionally as shown in FIG. 12 are well-known, and detailed illustrations and descriptions of these pixels are omitted.

[0101] In the example of FIG. 12, the configuration in which the focus detection pixels S1 and S2 receive one of a pair of light beams for focus detection has been described. Instead, the focus detection pixels may be configured to receive both of the pair of light beams for focus detection. By configuring the focus detection pixels to receive both of the pair of light beams for focus detection, it becomes possible to use the photoelectric conversion signal obtained by the focus detection pixels as a photoelectric conversion signal for recording.

[0102] The control unit 34 (AF calculation unit 34d) detects the amount of image shift (phase difference) between a pair of images formed by a pair of light beams passing through different regions of the imaging optical system 31 (FIG. 1) based on the focus detection photoelectric conversion signals (signal data) output from the focus detection pixel S1 and the focus detection pixel S2. Then, the defocus amount is calculated based on the amount of image shift (phase difference). Since the defocus amount calculation by such a pupil division phase difference method is known in the field of cameras, a detailed description thereof will be omitted.

[0103] The focus detection position 80A (FIG. 11) is assumed to be selected by the user at a position corresponding to the region 80 near the boundary of the region 61 in the live view image 60a illustrated in FIG. 7(a). FIG. 13 is an enlarged view of the focus detection position 80A. The white pixels indicate that the first imaging condition is set, and the shaded pixels indicate that the second imaging condition is set. The position surrounded by the frame 170 in FIG. 13 corresponds to the focus detection pixel line 160 (FIG. 11).

[0104] The control unit 34 (AF calculation unit 34d) usually performs a focus detection process using, as it is, the signal data for focus detection of the pixels for focus detection shown by the frame 170 without performing data selection processing. However, when the signal data for focus detection to which the first imaging condition is applied and the signal data for focus detection to which the second imaging condition is applied are mixed in the signal data for focus detection surrounded by the frame 170, the control unit 34 (AF calculation unit 34d) selects, as follows in (Example 1) to (Example 3), the signal data for focus detection of the first imaging condition to be used for the focus detection process from the signal data for focus detection surrounded by the frame 170. Then, the control unit 34 (AF calculation unit 34d) performs a focus detection process using the signal data for focus detection after the data selection process. In FIG. 13, the data output from the pixels shown in white is the signal data for focus detection of the first imaging condition, and the data output from the pixels shown by hatching is the signal data for focus detection of the second imaging condition. In the present embodiment, by not selecting the signal data for focus detection of the second imaging condition, the signal data for focus detection output from the pixels shown by hatching is not used for the focus detection process.

[0105] (Example 1) For example, when only the ISO sensitivity differs between the first imaging condition and the second imaging condition, and the ISO sensitivity of the first imaging condition is 100 and the ISO sensitivity of the second imaging condition is 800, the control unit 34 (AF calculation unit 34d) selects the signal data for focus detection of the first imaging condition to be used for the focus detection process from the image data surrounded by the frame 170. That is, the signal data for focus detection of the second imaging condition that is different from the first imaging condition among the signal data for focus detection surrounded by the frame 170 is not used for the focus detection process.

[0106] (Example 2) The control unit 34 (AF calculation unit 34d) selects, for example, the signal data for focus detection under the first imaging condition to be used in the focus detection process from the signal data for focus detection surrounded by the frame 170 when only the shutter speed differs between the first imaging condition and the second imaging condition, the shutter speed of the first imaging condition is 1 / 1000 second, and the shutter speed of the second imaging condition is 1 / 100 second. That is, among the signal data for focus detection surrounded by the frame 170, the signal data for focus detection under the second imaging condition that is different from the first imaging condition is not used in the focus detection process.

[0107] (Example 3) The control unit 34 (AF calculation unit 34d) selects, for example, the signal data for focus detection under the first imaging condition to be used in the focus detection process from the signal data for focus detection surrounded by the frame 170 when only the frame rate differs between the first imaging condition and the second imaging condition (the charge accumulation time is the same), the frame rate of the first imaging condition is 30 fps, and the frame rate of the second imaging condition is 60 fps. That is, among the signal data for focus detection surrounded by the frame 170, the signal data for focus detection under the second imaging condition obtained at a timing different from the image data under the first imaging condition is not used in the focus detection process.

[0108] On the other hand, the control unit 34 (AF calculation unit 34d) does not perform the above data selection process when the imaging conditions applied to the signal data for focus detection surrounded by the frame 170 are the same. That is, the control unit 34 (AF calculation unit 34d) performs the focus detection process using the signal data for focus detection of the focus detection pixels indicated by the frame 170 as they are.

[0109] Note that, as described above, even if there are slight differences in the imaging conditions, they are regarded as the same imaging conditions. Also, in the above example, an example of selecting the signal data for focus detection under the first imaging condition from the signal data for focus detection surrounded by the frame 170 has been described, but it may be possible to select the signal data for focus detection under the second imaging condition among the signal data for focus detection surrounded by the frame 170. In the above example, since the focus detection is performed by designating the area where the first imaging condition is set, an example of selecting the photoelectric conversion signal for focus detection under the first imaging condition has been described. An example will be described in which the subject to be focused is located across the area where the first imaging condition is set and the area where the second imaging condition is set. When the subject to be focused is located across the area where the first imaging condition is set and the area where the second imaging condition is set, the control unit 34 (AF calculation unit 34d) selects the photoelectric conversion signal for focus detection under the first imaging condition to be used for the focus detection process from the photoelectric conversion signals for focus detection surrounded by the frame 170. Then, the control unit 34 (AF calculation unit 34d) calculates the first defocus amount from the selected photoelectric conversion signal for focus detection. Further, the control unit 34 (AF calculation unit 34d) selects the photoelectric conversion signal for focus detection under the second imaging condition to be used for the focus detection process from the photoelectric conversion signals for focus detection surrounded by the frame 170. Then, the control unit 34 (AF calculation unit 34d) calculates the second defocus amount from the selected photoelectric conversion signal for focus detection. Then, the control unit 34 (AF calculation unit 34d) performs the focus detection process using the first defocus amount and the second defocus amount. Specifically, for example, the control unit 34 (AF calculation unit 34d) calculates the average value of the first defocus amount and the second defocus amount, and calculates the moving distance of the lens. Further, the control unit 34 (AF calculation unit 34d) may select the value with the smaller moving distance of the lens between the first defocus amount and the second defocus amount. Further, the control unit 34 (AF calculation unit 34d) may select a value indicating that the subject is closer from the first defocus amount and the second defocus amount. Further, when the subject to be focused on is located across the region where the first imaging condition is set and the region where the second imaging condition is set, the control unit 34 (AF calculation unit 34d) may select the region with the larger area of the subject region and select the photoelectric conversion signal for focus detection. For example, when 70% of the area of the face of the subject to be focused on is in the region where the first imaging condition is set and 30% is in the second region, the control unit 34 (AF calculation unit 34d) selects the photoelectric conversion signal for focus detection under the first imaging condition. Note that the ratios (percentages) regarding the areas described above are merely examples and are not limited thereto.

[0110] In the above description, the focus detection process using the pupil division phase difference method has been exemplified. However, it can be performed in the same manner for the contrast detection method in which the focus lens of the imaging optical system 31 is moved to the in-focus position based on the magnitude of the contrast of the subject image.

[0111] When using the contrast detection method, while moving the focus lens of the imaging optical system 31, the control unit 34 performs a known focus evaluation value calculation based on the image data output from the imaging pixels of the imaging element 32a corresponding to the focus detection position at each position of the focus lens. Then, the position of the focus lens that maximizes the focus evaluation value is obtained as the in-focus position.

[0112] The control unit 34 usually performs the focus evaluation value calculation directly using the image data output from the imaging pixels corresponding to the focus detection position without performing data selection processing. However, when the image data corresponding to the focus detection position contains a mixture of the image data to which the first imaging condition is applied and the image data to which the second imaging condition is applied, the control unit 34 selects the image data of the first imaging condition or the image data of the second imaging condition from the image data corresponding to the focus detection position. Then, the control unit 34 performs the focus evaluation value calculation using the image data after the data selection processing. As described above, when the subject to be focused on is located across the region where the first imaging condition is set and the region where the second imaging condition is set, the control unit 34 (AF calculation unit 34d) may select the region with the larger area of the subject region and select the signal data for focus detection.

[0113] 3. When performing subject detection processing FIG. 14(a) is a diagram illustrating a template image 180 representing an object to be detected, and FIG. 14(b) is a diagram illustrating a live view image 60(a) and a search range 190. The control unit 34 (object detection unit 34a) detects an object (for example, a bag 63a which is one of the subject elements in FIG. 5) from the live view image. The control unit 34 (object detection unit 34a) may set the entire range of the live view image 60a as the range for detecting the object, but in order to lighten the detection processing, a part of the live view image 60a may be set as the search range 190.

[0114] When different imaging conditions are set between the divided regions and the search range 190 includes the boundary of the divided regions, the control unit 34 (object detection unit 34a) performs data selection processing on the image data located near the boundary of the region as a preprocessing of the subject detection processing.

[0115] The data selection process is performed to suppress a decrease in the accuracy of the subject element detection process due to different imaging conditions among the regions of the imaging screen divided by the setting unit 34b. Generally, when the search range 190 used for detecting the subject element includes the boundary of the divided region, there may be a case where image data to which different imaging conditions are applied is mixed in the image data of the search range 190. In the present embodiment, based on the idea that it is more preferable to detect the subject element using the image data to which the same imaging condition is applied than to detect the subject element using the image data to which different imaging conditions are applied as they are, the data selection process is performed as follows.

[0116] In the live view image 60a illustrated in FIG. 5, the case of detecting the bag 63a which is an object held by the person 61a will be described. The control unit 34 (object detection unit 34a) sets the search range 190 in the vicinity of the region including the person 61a. Note that the region 61 including the person 61a may be set as the search range.

[0117] When the control unit 34 (object detection unit 34a) determines that the search range 190 is not divided by two regions with different imaging conditions, it performs subject detection processing using the image data that constitutes the search range 190 without performing data selection processing. However, if the image data of the search range 190 contains a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied, the control unit 34 (object detection unit 34a) selects the image data of the first imaging condition to be used for subject detection processing from the image data of the search range 190 in the same manner as in (Example 1) to (Example 3) when performing the above-described focus detection processing. Then, the control unit 34 (object detection unit 34a) performs subject detection processing using the image data after data selection processing for the region where the first imaging condition is set. Here, the subject detection processing is, for example, a process of detecting by obtaining the similarity between the template image 180 and the selected image data of the first imaging condition (so-called template matching). Further, the control unit 34 (object detection unit 34a) selects the image data of the second imaging condition to be used for subject detection processing from the image data of the search range 190. Then, the control unit 34 (object detection unit 34a) performs the same subject detection processing as described above using the image data after data selection processing for the region where the second imaging condition is set. In this way, the control unit 34 (object detection unit 34a) detects a subject within the search range 190. Then, by aligning the boundaries between the subject regions detected using the image data of the first imaging condition and the subject regions detected using the image data of the second imaging condition, it is possible to detect the subject within the search range 190. In the above example, the control unit 34 (object detection unit 34a) has described the case of using the image data of the first imaging condition and the image data of the second imaging condition, but it is also possible to perform subject detection using only the image data of either one. For example, when a region where the first imaging condition is set and a region where the second imaging condition is set are set within the search range 190, but the region where the first imaging condition is set occupies a large portion, it may be possible to perform subject detection using only the image data of the first imaging condition. Here, when the region where the first imaging condition is set occupies a large portion means, for example, that the area of the region where the first imaging condition is set is 70% or more.Further, it is not limited to 70% or more, and the area of the region where the first imaging condition is set may be 80% or more, or 90% or more. Of course, the ratio of the area of the region where the first imaging condition is set is not limited to these and can be changed as appropriate.

[0118] The data selection process for the image data of the search range 190 described above may be applied to the search range used for detecting a specific subject such as a person's face or the region used for determining the imaging scene. For example, when the control unit 34 (object detection unit 34a) detects a person's face from within the search range 190, it selects the image data of the first imaging condition used for the subject detection process from the image data of the search range 190. Then, the control unit 34 (object detection unit 34a) performs a known face detection process on the region where the first imaging condition is set. Further, the control unit 34 (object detection unit 34a) selects the image data of the second imaging condition used for the subject detection process from the image data of the search range 190. Then, the control unit 34 (object detection unit 34a) performs a known face detection process on the region where the second imaging condition is set. Then, the control unit 34 (object detection unit 34a) performs face detection from the image data of the search range 190 by aligning the boundary between the face region detected within the region where the first imaging condition is set and the face region detected within the region where the second imaging condition is set.

[0119] Further, the data selection process for the image data of the search range 190 described above is not limited to the search range used in the pattern matching method using the template image, and can be similarly applied to the search range when detecting feature amounts based on the color or edges of the image.

[0120] Further, the present invention may be applied to a tracking process of a moving body that searches for a region similar to a tracking target in a previously acquired frame image from a frame image acquired later by performing known template matching processing using image data of a plurality of frames having different acquisition times. In this case, when the image data to which the first imaging condition is applied and the image data to which the second imaging condition is applied are mixed in the search range set for the frame image acquired later, the control unit 34 selects the image data of the first imaging condition used for the tracking process from the image data of the search range. Then, for the region where the first imaging condition is set, the control unit 34 performs the tracking process using the image data after the data selection process. Thereafter, in the same manner as described above, the image data of the second imaging condition used for the tracking process is selected from the image data of the search range, and for the region where the second imaging condition is set, the control unit 34 may perform the tracking process using the image data after the data selection process.

[0121] Furthermore, the same applies when detecting a known motion vector using image data of a plurality of frames having different acquisition times. When the image data to which the first imaging condition is applied and the image data to which the second imaging condition is applied are mixed in the detection region used for detecting the motion vector, the control unit 34 selects the image data of the first imaging condition used for the detection process from the image data of the detection region used for detecting the motion vector. Then, the control unit 34 detects the motion vector using the image data after the data selection process for the region where the first imaging condition is set. Thereafter, in the same manner as described above, the image data of the second imaging condition used for the motion vector detection process is selected from the image data of the search range, and for the region where the second imaging condition is set, the control unit 34 may perform the motion vector detection process using the image data after the data selection process. The control unit 34 may obtain the motion vector of the entire image from the motion vector detected from the region where the first imaging condition is set and the motion vector detected from the region where the second imaging condition is set, or may obtain the motion vector for each region.

[0122] 4. When setting the imaging conditions When the control unit 34 (setting unit 34b) re - measures photometry anew and determines exposure conditions in a state where the area of the imaging screen is divided and different imaging conditions are set between the divided areas, a data selection process is performed on the image data located near the boundary of the area as pre - processing for setting the exposure conditions.

[0123] The data selection process is performed to suppress a decrease in the accuracy of the process for determining exposure conditions due to different imaging conditions between the areas of the imaging screen divided by the setting unit 34b. For example, when the photometry range set in the central part of the imaging screen includes the boundary of the divided areas, there may be a mixture of image data to which different imaging conditions are applied among the image data of the photometry range. In the present embodiment, based on the idea that it is more preferable to perform exposure calculation processing using image data to which the same imaging condition is applied than to perform exposure calculation processing using image data to which different imaging conditions are applied as they are, the data selection process is performed as follows.

[0124] When the photometric range is not divided into a plurality of regions with different imaging conditions, the control unit 34 (setting unit 34b) performs exposure calculation processing using the image data constituting the photometric range as it is without performing data selection processing. However, if the image data of the photometric range contains a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied, the control unit 34 (setting unit 34b) performs the above-described focus detection processing and subject detection processing. Similar to (Example 1) to (Example 3) in the case of, the image data of the first imaging condition used for the exposure calculation process is selected from the image data of the photometric range. Then, the control unit 34 (setting unit 34b) performs exposure calculation processing on the region where the first imaging condition is set using the image data after the data selection process. Then, the control unit 34 (setting unit 34b) selects the image data of the second imaging condition used for the exposure calculation process from the image data of the photometric range. Then, the control unit 34 (setting unit 34b) performs exposure calculation processing on the region where the second imaging condition is set using the image data after the data selection process. Thus, when there are a plurality of regions with different imaging conditions in the photometric range, the control unit 34 (setting unit 34b) performs data selection processing for measuring each region, and performs exposure calculation processing using the image data of the data selection process. Also, when the photometric range straddles a region where the first imaging condition is set and a region where the second imaging condition is set, the control unit 34 (setting unit 34b) may select the region with the larger area in the same manner as in the case of the above-described focus detection and subject detection.

[0125] This is the same not only for the photometric range when performing the above-described exposure calculation processing, but also for the photometric (colorimetric) range when determining the white balance adjustment value, the photometric range when determining the necessity of emitting the shooting auxiliary light by the light source that emits the shooting auxiliary light, and further, the photometric range when determining the emission amount of the shooting auxiliary light by the above light source.

[0126] Also, when the readout resolution of the photoelectric conversion signal is made different between the regions obtained by dividing the imaging screen, the same can be applied to the regions used for determining the imaging scene when determining the readout resolution for each region.

[0127] <Explanation of Flowchart> FIG. 15 is a flowchart for explaining the flow of the process of setting imaging conditions for each area and performing imaging. When the main switch of camera 1 is turned on, control unit 34 starts a program that executes the process shown in FIG. 15. In step S10, control unit 34 causes display unit 35 to start live view display and proceeds to step S20.

[0128] Specifically, control unit 34 sequentially displays on display unit 35, as a live view image, an image obtained by subjecting the image data sequentially output from imaging unit 32 to predetermined image processing. As described above, at this point, the same imaging conditions are set for the entire area of imaging chip 111, that is, the entire screen. Note that when a setting is made to perform an AF operation during live view display, control unit 34 (AF calculation unit 34d) controls the AF operation of focusing on a subject element corresponding to a predetermined focus detection position by performing a focus detection process. AF calculation unit 34d performs the above-described data selection process and then performs the focus detection process as necessary. Also, when a setting is not made to perform an AF operation during live view display, control unit 34 (AF calculation unit 34d) performs the AF operation when an AF operation is instructed later.

[0129] In step S20, control unit 34 (object detection unit 34a) detects a subject element from the live view image and proceeds to step S30. Object detection unit 34a performs the subject detection process after performing the above-described data selection process as necessary. In step S30, control unit 34 (setting unit 34b) divides the screen of the live view image into areas including the subject element and proceeds to step S40.

[0130] In step S40, control unit 34 causes display unit 35 to display the areas. Control unit 34 highlights, as exemplified in FIG. 6, the areas to be set (changed) for imaging conditions among the divided areas. Also, control unit 34 causes display unit 35 to display imaging condition setting screen 70 and proceeds to step S50. Note that when the display position of another main subject on the display screen is tapped with the user's finger, the control unit 34 changes the area including the main subject to the area to be set (changed) for the imaging conditions and highlights it.

[0131] In step S50, the control unit 34 determines whether an AF operation is necessary. For example, when the focus adjustment state changes due to the movement of the subject, when the position of the focus detection position is changed by a user operation, or when the execution of the AF operation is instructed by a user operation, the control unit 34 makes an affirmative determination in step S50 and proceeds to step S70. When the focus adjustment state does not change, the position of the focus detection position is not changed by a user operation, and the execution of the AF operation is not instructed by a user operation, the control unit 34 makes a negative determination in step S50 and proceeds to step 60.

[0132] In step S70, the control unit 34 causes the AF operation to be performed and returns to step S40. The AF calculation unit 34d performs the focus detection process, which is the AF operation, after performing the above data selection process as necessary. The control unit 34 that has returned to step S40 repeats the same process as the above-described process based on the live view image acquired after the AF operation.

[0133] In step S60, the control unit 34 (setting unit 34b) sets the imaging conditions for the highlighted area according to the user operation and proceeds to step S80. Note that the display transition of the display unit 35 and the setting of the imaging conditions according to the user operation in step S60 are as described above. The control unit 34 (setting unit 34b) performs the exposure calculation process after performing the above data selection process as necessary.

[0134] In step S80, the control unit 34 determines whether there is an imaging instruction. When a release button (not shown) that constitutes the operation member 36 or a display icon for instructing imaging is operated, the control unit 34 makes an affirmative determination in step S80 and proceeds to step S90. When no imaging instruction is given, the control unit 34 makes a negative determination in step S80 and returns to step S60.

[0135] In step S90, the control unit 34 performs predetermined imaging processing. That is, the imaging control unit 34c controls the imaging element 32a to perform imaging under the imaging conditions set for each of the above regions, and proceeds to step S100.

[0136] In step S100, the control unit 34 (imaging control unit 34c) sends an instruction to the image processing unit 33, causes the image processing unit 33 to perform predetermined image processing on the image data obtained by the above imaging, and proceeds to step S110. The image processing includes the above pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing. Note that the image processing unit 33 (selection unit 33b) performs data selection processing on the image data located near the boundary of the region, if necessary, and then performs image processing.

[0137] In step S110, the control unit 34 sends an instruction to the recording unit 37, causes the recording unit 37 to record the image data after the image processing on a recording medium (not shown), and proceeds to step S120.

[0138] In step S120, the control unit 34 determines whether or not an end operation has been performed. If the end operation has been performed, the control unit 34 makes an affirmative determination in step S120 and ends the processing according to FIG. 15. If the end operation has not been performed, the control unit 34 makes a negative determination in step S120 and returns to step S20. When returning to step S20, the control unit 34 repeats the above-described processing.

[0139] In the above description, the stacked imaging element 100 is exemplified as the imaging element 32a. However, if imaging conditions can be set for each of a plurality of blocks in the imaging element (imaging chip 111), it is not necessarily configured as a stacked imaging element.

[0140] According to the above-described embodiment, the following operational effects can be obtained. (1) The camera 1 equipped with the image processing device inputs the first image data generated by imaging the subject image incident on the first region of the imaging unit 32 under the first imaging condition, and the second image data generated by imaging the subject image incident on the second region of the imaging unit 32 under the second imaging condition different from the first imaging condition, into the image processing unit 33 (input unit 33a). The selection unit 33b selects one of the first image data and the second image data input to the input unit 33a. The image processing unit (generation unit 33) causes the display unit 35 to display the image generated from the selected image data. Thereby, appropriate processing can be performed in regions with different imaging conditions. That is, an image based on the image data generated in each region can be appropriately generated. For example, the sense of incongruity appearing in the generated image can be suppressed due to differences in imaging conditions for each region.

[0141] (2) The generation unit 33c of the camera 1 generates an image using the image data of predetermined reference pixels around each pixel at the position of each pixel of the image including the first image data and the second image data. The selection unit 33b selects, at the position of each pixel, one of the first image data and the second image data as the image data of the reference pixel. Thereby, appropriate processing can be performed in regions with different imaging conditions.

[0142] (3) The selection unit 33b of the camera 1 selects the image data generated by imaging under the same imaging condition as that of each pixel from among the first image data and the second image data. Thereby, appropriate processing can be performed in regions with different imaging conditions.

[0143] (4) The selection unit 33b of the camera 1 further selects, at other pixels other than the reference pixels, the image data generated by imaging under the same imaging condition as that of each pixel. Thereby, appropriate processing can be performed in regions with different imaging conditions.

[0144] (5) When all of the image data is the first image data, the selection unit 33b of the camera 1 selects all of the image data. When all of the image data is the second image data, the selection unit 33b of the camera 1 selects all of the image data. When the image data includes the first image data and the second image data, the selection unit 33b of the camera 1 selects the first image data or the second image data. Thereby, appropriate processing can be performed in regions with different imaging conditions respectively.

[0145] (6) Since the first imaging condition and the second imaging condition of the camera 1 include at least the accumulation time or the ISO sensitivity, appropriate processing can be performed in regions with different charge accumulation times and imaging sensitivities respectively.

[0146] (7) The generation unit 33c of the camera 1 generates the third image data as at least pixel defect correction processing, color interpolation processing, edge enhancement processing, or noise reduction processing. Thereby, image data subjected to pixel defect correction processing, color interpolation processing, edge enhancement processing, or noise reduction processing appropriately can be generated in regions with different imaging conditions respectively.

[0147] (Second Embodiment) In the case of performing the image data selection process in the image processing in the first embodiment, when the imaging condition (referred to as the first imaging condition) applied to the target pixel P is different from the imaging condition (referred to as the second imaging condition) applied to the reference pixels around the target pixel P, the image processing unit 33 (selection unit 33b) selects the image data to which the first imaging condition common to the target pixel is applied from the image data of the pixels located inside the predetermined range 90, and the image processing unit 33 (generation unit 33c) refers to the selected image data.

[0148] In the second embodiment, the image processing unit 33 (selection unit 33b) also selects the image data to which the first imaging condition common to the target pixel is applied from the image data of the pixels located outside the predetermined range 90, and increases the number of data referred to by the image processing unit 33 (generation unit 33c). That is, the image processing unit 33 (selection unit 33b) changes the selection position and selects the image data to which the first imaging condition is applied.

[0149] FIG. 7(d) is an enlarged view of the target pixel P and the reference pixel in the second embodiment. In FIG. 7(d), among the predetermined range 90 centered on the target pixel P, the data output from the pixels shown in white is the image data under the first imaging condition, and the data output from the pixels shown by hatching is the image data under the second imaging condition. Even for the image data of the pixels located inside the predetermined range 90, the image data of the pixels (hatched) to which the second imaging condition is applied is not selected by the image processing unit 33 (selection unit 33b), which is the same as in the first embodiment.

[0150] In the second embodiment, as illustrated in FIG. 7(d), the image processing unit 33 (selection unit 33b) further selects the image data of the white pixels located outside the predetermined range 90 to which the first imaging condition is applied, together with the image data of the white pixels located inside the predetermined range 90 to which the first imaging condition is applied. The image processing unit 33 (generation unit 33c) performs image processing with reference to the image data thus selected. In the above example, the distance from the position of the target pixel P to the position of the image data under the second imaging condition inside the predetermined range 90 is longer than the distance from the position of the target pixel P to the position of the image data of the white pixels located outside the predetermined range 90 to which the first imaging condition is applied. Thus, the image processing unit 33 (generation unit 33c) can also select the image data under the first imaging condition with a longer distance from the position of the target pixel P instead of selecting the image data under the second imaging condition with a shorter distance from the position of the target pixel P.

[0151] Note that the image processing unit 33 (selection unit 33b) preferentially selects the image data of the pixels at a position closer to the predetermined range 90 rather than the image data of the pixels at a position farther from the predetermined range 90. This is based on the idea that the pixels at a position closer to the predetermined range 90 are more likely to have common image information with the target pixel P than the pixels at a position farther from the predetermined range 90.

[0152] Further, for example, when the length of one side of the predetermined range 90 is L, the image processing unit 33 (selection unit 33b) selects image data of white pixels (i.e., pixels to which the first imaging condition is applied) whose distance from the pixels indicated by the above diagonal lines is L or less. The reason for this is based on the idea that if it is too far from the predetermined range 90, the possibility of having common image information with the target pixel P becomes low, so it is preferable not to include it in the reference data.

[0153] <Exemplification of Image Processing> Exemplify the image processing in the second embodiment. (1) Pixel Defect Correction Processing When the same imaging condition is applied to all the pixels in the predetermined range 90 centered on the target pixel P, the image processing unit 33 (selection unit 33b) selects all the image data of the pixels located inside the predetermined range 90. Then, the image processing unit 33 (generation unit 33c) performs Max, Min filter processing with reference to the selected image data. Note that it may be possible to perform pixel defect correction processing by taking the average of the selected image data.

[0154] As shown in FIG. 7(d), when a pixel to which a second imaging condition different from the first imaging condition applied to the target pixel P at the time of imaging is included in the predetermined range 90 centered on the target pixel P, the image processing unit 33 (selection unit 33b) selects the image data of the pixels located inside the predetermined range 90 to which the first imaging condition is applied. Further, the image data of the white pixels located outside the predetermined range 90 to which the first imaging condition is applied is selected. The image processing unit 33 (generation unit 33c) performs the above-described Max, Min filter processing with reference to the image data thus selected. Note that it may be possible to perform pixel defect correction processing by taking the average of the selected image data.

[0155] The image processing unit 33 performs such processing for all pixel defects whose position information is recorded in the non-volatile memory.

[0156] (2) Color Interpolation Processing <G Color Interpolation> The G-color interpolation of the second embodiment will be described. Similar to the case of the first embodiment, in FIGS. 8(a) to 8(c), for example, the first imaging condition is applied to the left and upper regions with respect to the thick line, and the second imaging condition is applied to the right and lower regions with respect to the thick line.

[0157] In the example of FIG. 8(b), a second imaging condition different from the first imaging condition applied to the attention position (second row and second column) is applied to the reference position corresponding to the image data G4 of the G-color component indicated by the hatching. The image processing unit 33 (selection unit 33b) selects the image data G1 to G3 to which the first imaging condition is applied from the image data G1 to G4 of the G-color component. Further, the image processing unit 33 (selection unit 33b) selects the image data G6 of the G-color component located in the vicinity of the reference position corresponding to the data G4 and to which the first imaging condition is applied. That is, the image processing unit 33 (selection unit 33b) changes the position for selecting the image data with respect to the first embodiment and selects the image data to which the first imaging condition is applied. Note that when the second imaging condition is also applied at the position of the data G6, data to which the first imaging condition is applied may be selected from the image data at the positions in the vicinity of the data G6.

[0158] The image processing unit 33 (generation unit 33c) calculates the image data of the G-color component at the attention position (second row and second column) with reference to the image data thus selected. The image processing unit 33 (generation unit 33c) sets, for example, (a2G1 + b2G2 + c2G3 + d2G6) / 4 as the image data of the G-color component at the attention position (second row and second column). Note that a2, b2, c2, and d2 are weighting coefficients provided according to the distance between the reference position and the attention position and the image structure.

[0159] The image processing unit 33 (generation unit 33c) generates the image data of the G-color component at the positions of the B-color component and the R-color component in FIG. 8(a), respectively, to obtain the image data of the G-color component at each pixel position as shown in FIG. 8(c).

[0160] <R-color interpolation> The R color interpolation of the second embodiment will be described. Similar to the case of the first embodiment, in FIGS. 9(a) to 9(c), for example, the first imaging condition is applied to the left and upper regions with respect to the thick line, and the second imaging condition is applied to the right and lower regions with respect to the thick line.

[0161] In the example of FIG. 9(b), the image processing unit 33 (selection unit 33b) applies a second imaging condition different from the first imaging condition applied to the attention position indicated by the thick frame (second row, second column) to the reference position corresponding to the image data Cr2 of the color difference component Cr indicated by the hatching. Therefore, the image processing unit 33 (selection unit 33b) selects the image data Cr1, Cr3 to Cr4 to which the first imaging condition is applied from the image data Cr1 to Cr4 of the color difference component. Further, the image processing unit 33 (selection unit 33b) selects the image data Cr15 (or Cr16) of the color difference component Cr located in the vicinity of the reference position corresponding to the data Cr2 and to which the first imaging condition is applied. That is, the image processing unit 33 (selection unit 33b) changes the position for selecting the image data with respect to the first embodiment and selects the image data to which the first imaging condition is applied. If the second imaging condition is also applied at the position of the data Cr15 or Cr16, the data to which the first imaging condition is applied may be selected from the image data at the position in the vicinity of the data Cr15 or Cr16.

[0162] The image processing unit 33 (generation unit 33c) calculates the image data of the color difference component at the attention position (second row, second column) with reference to the image data selected in this way. The image processing unit 33 (generation unit 33c) sets, for example, (e3Cr1 + f3Cr15 + g3Cr3 + h3Cr4) / 4 as the image data of the color difference component Cr at the attention position (second row, second column). Note that e3, f3, g3, and h3 are weighting coefficients provided according to the distance between the reference position and the attention position and the image structure.

[0163] The same applies when generating the image data of the chrominance component Cr for the attention position indicated by the thick frame (second row, third column) in Fig. 9(c). In the example of Fig. 9(c), the first imaging condition different from the second imaging condition applied to the attention position (second row, third column) is applied to the reference positions corresponding to the image data Cr4 and Cr5 of the chrominance component Cr indicated by the hatching. Therefore, the image processing unit 33 (selection unit 33b) selects the image data Cr2 and Cr6 to which the second imaging condition is applied from the image data Cr2, Cr4 to Cr6 of the chrominance component. Further, the image processing unit 33 (selection unit 33b) selects the image data Cr8 and Cr7 of the chrominance component Cr to which the second imaging condition is applied and which are located in the vicinity of the reference positions corresponding to the data Cr4 and Cr5. That is, the image processing unit 33 (selection unit 33b) changes the position for selecting the image data with respect to the first embodiment and selects the image data to which the second imaging condition is applied. Incidentally, if the first imaging condition is also applied at the positions of the data Cr8 and Cr7, data to which the second imaging condition is applied may be selected from the image data at the positions in the vicinity of the data Cr8 and Cr7.

[0164] The image processing unit 33 (generation unit 33c) calculates the image data of the chrominance component Cr at the attention position (second row, third column) with reference to the image data thus selected. The image processing unit 33 (generation unit 33c) sets, for example, (q3Cr2 + r3Cr8 + s3Cr7 + t3r6) / 4 as the image data of the chrominance component Cr at the attention position. Note that q3, r3, s3, and t3 are weighting factors provided according to the distance between the reference position and the attention position and the image structure.

[0165] After obtaining the image data of the chrominance component Cr at each pixel position, the image processing unit 33 (generation unit 33c) adds the image data of the G color component shown in Fig. 8(c) corresponding to each pixel position to obtain the image data of the R color component at each pixel position.

[0166] The B color interpolation of the second embodiment will be described. Similar to the case of the first embodiment, in FIGS. 10(a) to 10(c), for example, the first imaging condition is applied to the regions to the left and above the thick line, and the second imaging condition is applied to the regions to the right and below the thick line.

[0167] In the example of FIG. 10(b), a first imaging condition different from the second imaging condition applied to the attention position indicated by the thick frame (the third row and third column) is applied to the reference positions corresponding to the image data Cb1 and Cb3 of the color difference component Cb indicated by the hatching. Therefore, the image processing unit 33 (selection unit 33b) selects the image data Cb2 and Cb4 to which the second imaging condition is applied from the image data Cb1 to Cb4 of the color difference component. Further, the image processing unit 33 (selection unit 33b) selects the image data Cb16 and Cb17 of the color difference component Cb located in the vicinity of the reference positions corresponding to the data Cb1 and Cb3 and to which the second imaging condition is applied. That is, the image processing unit 33 (selection unit 33b) changes the position for selecting the image data with respect to the first embodiment and selects the image data to which the second imaging condition is applied.

[0168] The image processing unit 33 (generation unit 33c) calculates the image data Cb of the color difference component at the attention position (the third row and third column) with reference to the image data thus selected. The image processing unit 33 (generation unit 33c) sets, for example, (u3Cb16 + v3Cb2 + w3Cb4 + x3Cb17) / 4 as the image data of the color difference component Cb at the attention position (the third row and third column). Note that u3, v3, w3, and x3 are weighting coefficients provided according to the distance between the reference position and the attention position and the image structure.

[0169] The same applies to the case of generating the image data Cb of the color difference component for the attention position (the third row and fourth column) indicated by the thick frame in FIG. 10(c). In the example of FIG. 10(c), the same second imaging condition as that of the target position (the fourth column of the third row) is applied to the reference positions corresponding to the image data Cb2, Cb4 to Cb6 of the four color difference components located near the target position (the fourth column of the third row). The image processing unit 33 (generation unit 33c) calculates the image data of the color difference component Cb at the target position with reference to the image data Cb2, Cb4 to Cb6 of the four color difference components located near the target position.

[0170] After obtaining the image data of the color difference component Cb at each pixel position, the image processing unit 33 (generation unit 33c) adds the image data of the G color component shown in FIG. 8(c) corresponding to each pixel position to obtain the image data of the B color component at each pixel position.

[0171] According to the second embodiment described above, in addition to the same operational effects as those of the first embodiment, the following operational effects can be obtained. That is, the selection unit 33b is configured to select the image data of the pixels in the region where the imaging condition common to the pixel position (the position of the target pixel) is set, among the pixels other than the reference pixels. Thereby, for example, the data referred to for the generation of the third image data by the generation unit 33c can be increased, and appropriate image processing can be performed.

[0172] (Modifications of the First and Second Embodiments) The following modifications are also within the scope of the present invention, and one or more of the modifications can be combined with the above-described embodiments. (Modification 1) FIGS. 16(a) to 16(c) are diagrams illustrating the arrangements of the first region and the second region on the imaging surface of the imaging device 32a. According to the example of FIG. 16(a), the first region is composed of even-numbered columns, and the second region is composed of odd-numbered columns. That is, the imaging surface is divided into even-numbered columns and odd-numbered columns.

[0173] According to the example of FIG. 16(b), the first region is composed of odd-numbered rows, and the second region is composed of even-numbered rows. That is, the imaging surface is divided into odd-numbered rows and even-numbered rows.

[0174] According to the example of Fig. 16(c), the first region is composed of blocks in even rows in odd columns and blocks in odd rows in even columns. Also, the second region is composed of blocks in even rows in even columns and blocks in odd rows in odd columns. That is, the imaging surface is divided in a checkered pattern.

[0175] In any of the cases of Figs. 16(a) to 16(c), a first image based on the photoelectric conversion signal read from the first region and a second image based on the photoelectric conversion signal read from the second region are respectively generated by the photoelectric conversion signal read from the image sensor 32a that has performed imaging for one frame. According to Modification 1, the first image and the second image are imaged at the same angle of view and include a common subject image.

[0176] In Modification 1, the control unit 34 uses the first image for display and uses the second image for detection. Specifically, the control unit 34 causes the display unit 35 to display the first image as a live view image. Also, the control unit 34 causes the object detection unit 34a to perform subject detection processing using the second image, causes the AF calculation unit 34 to perform focus detection processing using the second image, and causes the setting unit 34b to perform exposure calculation processing using the second image.

[0177] In Modification 1, the imaging conditions set for the first region that captures the first image are referred to as first imaging conditions, and the imaging conditions set for the second region that captures the second image are referred to as second imaging conditions. The control unit 34 may make the first imaging conditions different from the second imaging conditions.

[0178] 1. As an example, the control unit 34 sets the first imaging conditions to conditions suitable for display by the display unit 35. The first imaging conditions are made the same throughout the entire first region of the imaging screen. On the other hand, the control unit 34 sets the second imaging conditions to conditions suitable for focus detection processing, subject detection processing, and exposure calculation processing. The second imaging conditions are also made the same throughout the entire second region of the imaging screen. In addition, when the conditions suitable for the focus detection process, the subject detection process, and the exposure calculation process are different, the control unit 34 may vary the second imaging conditions set for the second region for each frame. For example, the second imaging conditions for the first frame are set as the conditions suitable for the focus detection process, the second imaging conditions for the second frame are set as the conditions suitable for the subject detection process, and the second imaging conditions for the third frame are set as the conditions suitable for the exposure calculation process. In these cases, the second imaging conditions for each frame are the same for the entire second region of the imaging screen.

[0179] 2. As another example, the control unit 34 may vary the first imaging conditions set for the first region. The control unit 34 (the setting unit 34b) sets different first imaging conditions for each region including the subject elements divided by the setting unit 34b. On the other hand, the control unit 34 makes the second imaging conditions the same for the entire second region of the imaging screen. The control unit 34 sets the second imaging conditions as the conditions suitable for the focus detection process, the subject detection process, and the exposure calculation process. However, when the conditions suitable for the focus detection process, the subject detection process, and the exposure calculation process are different, the imaging conditions set for the second region may be varied for each frame.

[0180] 3. Also, as another example, the control unit 34 may make the first imaging conditions the same for the entire first region of the imaging screen while varying the second imaging conditions set for the second region on the imaging screen. For example, different second imaging conditions are set for each region including the subject elements divided by the setting unit 34b. Also in this case, when the conditions suitable for the focus detection process, the subject detection process, and the exposure calculation process are different, the imaging conditions set for the second region may be varied for each frame.

[0181] 4. Furthermore, as another example, the control unit 34 varies the first imaging conditions set for the first region on the imaging screen and also varies the second imaging conditions set for the second region on the imaging screen. For example, while setting different first imaging conditions for each region including the subject elements divided by the setting unit 34b, different second imaging conditions are set for each region including the subject elements divided by the setting unit 34b.

[0182] In FIGS. 16(a) to 16(c), the area ratio between the first region and the second region may be varied. For example, based on an operation by the user or a determination by the control unit 34, the control unit 34 may set the ratio of the first region to be higher than that of the second region, may set the ratio of the first region and the second region to be equal as illustrated in FIGS. 16(a) to 16(c), or may set the ratio of the first region to be lower than that of the second region. By varying the area ratio between the first region and the second region, the first image can be made higher definition than the second image, the resolutions of the first image and the second image can be made equal, or the second image can be made higher definition than the first image.

[0183] (Modification Example 2) In the above-described embodiment, an example in which the control unit 34 (setting unit 34b) detects a subject element based on the live view image and divides the screen of the live view image into regions including the subject element has been described. In Modification Example 2, when the control unit 34 includes a photometric sensor separately from the imaging device 32a, the control unit 34 may divide the region based on the output signal from the photometric sensor.

[0184] The control unit 34 divides the image into a foreground and a background based on the output signal from the photometric sensor. Specifically, the live view image acquired by the imaging device 32b is divided into a foreground region corresponding to the region determined as the foreground from the output signal of the photometric sensor and a background region corresponding to the region determined as the background from the output signal of the photometric sensor.

[0185] The control unit 34 further arranges the first region and the second region as illustrated in FIGS. 16(a) to 16(c) at positions corresponding to the foreground region of the imaging surface of the imaging device 32a. On the other hand, the control unit 34 arranges only the first region on the imaging surface of the imaging device 32a at a position corresponding to the background region of the imaging surface of the imaging device 32a. The control unit 34 uses the first image for display and uses the second image for detection.

[0186] According to Modification Example 2, by using the output signal from the photometric sensor, it is possible to perform region division of the live view image acquired by the imaging device 32b. Also, for the foreground region, a first image for display and a second image for detection can be obtained, and for the background region, only the first image for display can be obtained.

[0187] (Modification Example 3) In Modification Example 3, the image processing unit 33 (generation unit 33c) performs contrast adjustment processing so as to mitigate the discontinuity of the image based on the difference between the first imaging condition and the second imaging condition. That is, the generation unit 33c mitigates the discontinuity of the image based on the difference between the first imaging condition and the second imaging condition by varying the tone curve (gamma curve).

[0188] For example, assume that only the ISO sensitivity differs between the first imaging condition and the second imaging condition, the ISO sensitivity of the first imaging condition is 100, and the ISO sensitivity of the second imaging condition is 800. The generation unit 33c compresses the value of the image data of the second imaging condition among the image data at the reference position to 1 / 8 by flattening the tone curve.

[0189] Alternatively, the generation unit 33c may expand the value of the image data at the attention position and the image data of the first imaging condition among the image data at the reference position by 8 times by raising the tone curve.

[0190] According to Modification Example 3, similar to the above-described embodiment, appropriate image processing can be performed on the image data generated in the regions with different imaging conditions. For example, it is possible to suppress the discontinuity and discomfort that appear in the image after image processing due to the difference in imaging conditions at the boundary of the region.

[0191] (Modification Example 4) In Modification 4, the image processing unit 33 (generation unit 33c) prevents the outline of the subject element from being damaged in the above-described image processing (for example, noise reduction processing). Generally, when performing noise reduction, a smoothing filter process is adopted. When using a smoothing filter, in addition to the noise reduction effect, the boundary of the subject element may be blurred.

[0192] Therefore, the image processing unit 33 (generation unit 33c) compensates for the blurring of the boundary of the subject element by performing contrast adjustment processing, for example, in addition to the noise reduction processing or together with the noise reduction processing. In Modification 4, the image processing unit 33 (generation unit 33c) sets a curve that draws an S shape as the density conversion (tone conversion) curve (so-called S-curve conversion). By performing contrast adjustment using the S-curve conversion, the image processing unit 33 (generation unit 33c) stretches the tone portions of bright data and dark data, respectively, to increase the number of tones of the bright data (and dark data), and shrinks the image data of the intermediate tones to reduce the number of tones. As a result, the number of image data with medium brightness of the image decreases, and the data classified as either bright / dark increases, thereby compensating for the blurring of the boundary of the subject element.

[0193] According to Modification 4, by clarifying the light and dark of the image, the blurring of the boundary of the subject element can be compensated for.

[0194] (Modification 5) In Modification 5, the image processing unit 33 (generation unit 33c) changes the white balance adjustment gain so as to mitigate the discontinuity of the image based on the difference between the first imaging condition and the second imaging condition.

[0195] For example, when the imaging conditions applied during imaging at the target position (referred to as the first imaging conditions) are different from the imaging conditions applied during imaging at a reference position around the target position (referred to as the second imaging conditions), the image processing unit 33 (generation unit 33c) changes the white balance adjustment gain so that the white balance of the image data of the second imaging conditions among the image data of the reference position approaches the white balance of the image data acquired under the first imaging conditions.

[0196] Note that the image processing unit 33 (generation unit 33c) may also change the white balance adjustment gain so that the white balance of the image data of the first imaging conditions among the image data of the reference position and the white balance of the image data of the target position approach the white balance of the image data acquired under the second imaging conditions.

[0197] According to Modification 5, by aligning the white balance adjustment gain of the image data generated in regions with different imaging conditions with the adjustment gain of either region with different imaging conditions, the discontinuity of the image based on the difference between the first imaging conditions and the second imaging conditions can be alleviated.

[0198] (Modification 6) A plurality of image processing units 33 may be provided to perform image processing in parallel. For example, while performing image processing on the image data captured in region A of the imaging unit 32, image processing is performed on the image data captured in region B of the imaging unit 32. The plurality of image processing units 33 may perform the same image processing or different image processing. That is, the same parameters or the like can be applied to the image data of regions A and B to perform similar image processing, or different parameters or the like can be applied to the image data of regions A and B to perform different image processing.

[0199] When a plurality of image processing units 33 are provided, image processing may be performed on the image data to which the first imaging condition is applied by one image processing unit, and image processing may be performed on the image data to which the second imaging condition is applied by another image processing unit. The number of image processing units is not limited to the above two, and for example, the same number as the number of imaging conditions that can be set may be provided. That is, each image processing unit is in charge of image processing for each area to which different imaging conditions are applied. According to Modification 6, imaging using different imaging conditions for each area and image processing on the image data of the images obtained for each area can proceed in parallel.

[0200] (Modification 7) In the above description, the camera 1 has been described as an example, but it may be configured by a high - performance mobile phone 250 (FIG. 14) equipped with a camera function such as a smartphone, or a mobile device such as a tablet terminal.

[0201] (Modification 8) In the above - described embodiment, the camera 1 configured by integrating the imaging unit 32 and the control unit 34 as a single electronic device has been described as an example. Instead, for example, an imaging system 1B may be configured in which the imaging unit 32 and the control unit 34 are provided separately, and the imaging unit 32 is controlled via communication from the control unit 34. Hereinafter, an example of controlling an imaging device 1001 including an imaging unit 32 from a control device 1002 including a control unit 34 will be described with reference to FIG. 17.

[0202] FIG. 17 is a block diagram illustrating the configuration of an imaging system 1B according to Modification 8. In FIG. 17, the imaging system 1B is composed of an imaging device 1001 and a display device 1002. The imaging device 1001 includes a first communication unit 1003 in addition to the imaging optical system 31 and the imaging unit 32 described in the above - described embodiment. Further, the display device 1002 includes a second communication unit 1004 in addition to the image processing unit 33, the control unit 34, the display unit 35, the operation member 36, and the recording unit 37 described in the above - described embodiment.

[0203] The first communication unit 1003 and the second communication unit 1004 can perform bidirectional image data communication by, for example, well-known wireless communication technologies, optical communication technologies, etc. Note that the imaging device 1001 and the display device 1002 may be wired-connected by a wired cable, and the first communication unit 1003 and the second communication unit 1004 may be configured to perform bidirectional image data communication.

[0204] In the imaging system 1B, the control unit 34 controls the imaging unit 32 by performing data communication via the second communication unit 1004 and the first communication unit 1003. For example, by transmitting and receiving predetermined control data between the imaging device 1001 and the display device 1002, the display device 1002 divides the screen into a plurality of regions based on the image as described above, sets different imaging conditions for each divided region, or reads out the photoelectric conversion signals photoelectrically converted in each region.

[0205] According to Modification 8, since the live view image acquired on the imaging device 1001 side and transmitted to the display device 1002 is displayed on the display unit 35 of the display device 1002, the user can perform remote operations from the display device 1002 located at a position away from the imaging device 1001. The display device 1002 can be configured by, for example, a high-performance mobile phone 250 such as a smartphone. Also, the imaging device 1001 can be configured by an electronic device including the above-described stacked imaging element 100. Note that although an example in which the control unit 34 of the display device 1002 is provided with an object detection unit 34a, a setting unit 34b, an imaging control unit 34c, and an AF calculation unit 34d has been described, a part of the object detection unit 34a, the setting unit 34b, the imaging control unit 34c, and the AF calculation unit 34d may be provided in the imaging device 1001.

[0206] (Modification 9) The supply of the program to mobile devices such as the above-described camera 1, high-performance mobile phone 250, or tablet terminal can be performed by transmitting it from the personal computer 205 storing the program to the mobile device via infrared communication or short-range wireless communication, as exemplified in FIG. 18, for example.

[0207] The supply of the program to the personal computer 205 may be performed by setting a recording medium 204 such as a CD-ROM storing the program in the personal computer 205, or by loading it into the personal computer 205 by a method via a communication line 201 such as a network. When passing through the communication line 201, the program is stored in the storage device 203 of the server 202 connected to the communication line.

[0208] Also, the program can be directly transmitted to the mobile device via an access point (not shown) of a wireless LAN connected to the communication line 201. Further, a recording medium 204B such as a memory card storing the program may be set in the mobile device. In this way, the program can be supplied as computer program products in various forms such as being provided via a recording medium or a communication line.

[0209] (Third Embodiment) With reference to FIGS. 19 to 25, as an example of an electronic device equipped with an image processing apparatus according to the third embodiment, a digital camera will be described as an example. In the following description, the same reference numerals are given to the same components as those in the first or second embodiment, and the differences will be mainly described. Regarding points not particularly described, they are the same as those in the first or second embodiment. In this embodiment, mainly, instead of providing the image processing unit 33 of the first embodiment, the imaging unit 32A further includes an image processing unit 32c having the same function as the image processing unit 33 of the first embodiment, which is different from the first embodiment.

[0210] FIG. 19 is a block diagram illustrating the configuration of the camera 1C according to the third embodiment. In FIG. 19, the camera 1C includes an imaging optical system 31, an imaging unit 32A, a control unit 34, a display unit 35, an operation member 36, and a recording unit 37. The imaging unit 32A further includes an image processing unit 32c having the same functions as the image processing unit 33 of the first embodiment.

[0211] The image processing unit 32c includes an input unit 321, a selection unit 322, and a generation unit 323. Image data from the imaging device 32a is input to the input unit 321. The selection unit 322 performs preprocessing on the input image data. The preprocessing performed by the selection unit 322 is the same as the preprocessing performed by the selection unit 33b in the first embodiment. The generation unit 323 performs image processing on the input image data and the preprocessed image data to generate an image. The image processing performed by the generation unit 323 is the same as the image processing performed by the generation unit 33c in the first embodiment.

[0212] FIG. 20 is a diagram schematically showing the correspondence between each block in the present embodiment and a plurality of selection units 322. In FIG. 20, one cell of the imaging chip 111 represented by a rectangle represents one block 111a. Similarly, one cell of the image processing chip 114, which will be described later and is represented by a rectangle, represents one selection unit 322.

[0213] In the present embodiment, the selection unit 322 is provided corresponding to each block 111a. In other words, the selection unit 322 is provided for each block, which is the minimum unit of the area where the imaging conditions can be changed on the imaging surface. For example, in FIG. 20, the hatched block 111a and the hatched selection unit 322 are in a corresponding relationship. The hatched selection unit 322 in FIG. 20 performs preprocessing on the image data from the pixels included in the hatched block 111a. Each selection unit 322 performs preprocessing on the image data from the pixels included in the corresponding block 111a. As a result, preprocessing of the image data can be performed in parallel by a plurality of selection units 322, so that the processing load on the selection unit 322 can be reduced, and an appropriate image can be generated in a short time from the image data generated in regions with different imaging conditions. In the following description, when explaining the relationship between a certain block 111a and the pixels included in the block 111a, the block 111a may be referred to as the block 111a to which the pixel belongs. Further, the block 111a may be referred to as a unit division, and a collection of a plurality of block 111a, that is, a collection of a plurality of unit divisions may be referred to as a composite division.

[0214] FIG. 21 is a cross-sectional view of the stacked image sensor 100A. The stacked image sensor 100A further includes an image processing chip 114 that performs the above-described preprocessing and image processing, in addition to the back-illuminated imaging chip 111, the signal processing chip 112, and the memory chip 113. That is, the above-described image processing unit 32c is provided in the image processing chip 114. These imaging chip 111, signal processing chip 112, memory chip 113, and image processing chip 114 are stacked and electrically connected to each other by bumps 109 having conductivity such as Cu.

[0215] A plurality of bumps 109 are arranged on the opposing surfaces of the memory chip 113 and the image processing chip 114. These bumps 109 are aligned with each other, and when the memory chip 113 and the image processing chip 114 are pressed, etc., the aligned bumps 109 are joined to each other and electrically connected.

[0216] <Data selection process> Similar to the first embodiment, in the second embodiment, after the region of the imaging screen is divided by the setting unit 34b, the imaging conditions can be set (changed) for the region selected by the user or the region determined by the control unit 34. When different imaging conditions are set in the divided regions, the control unit 34 causes the selection unit 322 to perform the following data selection process as necessary.

[0217] 1. When performing image processing 1-1. When the imaging conditions of the target pixel P are the same as the imaging conditions of a plurality of reference pixels around the target pixel P In this case, the selection unit 322 selects all the image data of the plurality of reference pixels and outputs them to the generation unit 323. The generation unit 323 performs image processing using the image data of the plurality of reference pixels.

[0218] 1-2. When the imaging conditions of the target pixel P are different from the imaging conditions of at least one of the plurality of reference pixels around the target pixel P Let the imaging conditions applied to the target pixel P be the first imaging conditions, the imaging conditions applied to a part of the plurality of reference pixels be the first imaging conditions, and the imaging conditions applied to the remaining reference pixels be the second imaging conditions. In this case, the selection unit 322 corresponding to the block 111a to which the reference pixel to which the first imaging condition is applied belongs, and the selection unit 322 corresponding to the block 111a to which the reference pixel to which the second imaging condition is applied belongs, perform data selection processing on the image data of the reference pixels as follows in (Example 1) to (Example 3). Then, the generation unit 323 performs image processing to calculate the image data of the target pixel P with reference to the image data of the reference pixels after the data selection processing.

[0219] (Example 1) For example, assume that only the ISO sensitivity differs between the first imaging conditions and the second imaging conditions, the ISO sensitivity of the first imaging conditions is 100, and the ISO sensitivity of the second imaging conditions is 800. In this case, the selection unit 322 corresponding to the block 111a to which the reference pixel to which the first imaging condition is applied belongs selects the image data of the first imaging conditions. However, the selection unit 322 corresponding to the block 111a to which the reference pixel to which the second imaging condition is applied belongs does not select the image data of the second imaging conditions. That is, the image data of the second imaging conditions different from the first imaging conditions is not used for image processing.

[0220] (Example 2) For example, assume that only the shutter speed differs between the first imaging condition and the second imaging condition, the shutter speed of the first imaging condition is 1 / 1000 second, and the shutter speed of the second imaging condition is 1 / 100 second. In this case, the selection unit 322 corresponding to the block 111a to which the reference pixel to which the first imaging condition is applied belongs selects the image data of the first imaging condition. However, the selection unit 322 corresponding to the block 111a to which the reference pixel to which the second imaging condition is applied belongs does not select the image data of the second imaging condition. That is, the image data of the second imaging condition different from the first imaging condition is not used for image processing.

[0221] (Example 3) For example, assume that only the frame rate differs between the first imaging condition and the second imaging condition (the charge accumulation time is the same), the frame rate of the first imaging condition is 30 fps, and the frame rate of the second imaging condition is 60 fps. In this case, the selection unit 322 corresponding to the block 111a to which the reference pixel to which the first imaging condition is applied belongs selects the image data of the pixel of the first imaging condition. Also, the selection unit 322 corresponding to the block 111a to which the reference pixel to which the second imaging condition is applied belongs selects, for the image data of the second imaging condition (60 fps) among the image data of the reference pixel, the image data of the frame image whose acquisition timing is close to the frame image acquired under the first imaging condition (30 fps). That is, among the image data of the reference pixel, the image data of the frame image whose acquisition timing is close to the frame image of the first imaging condition (30 fps) is used for image processing, and the image data of the frame image whose acquisition timing is different from the frame image of the first imaging condition (30 fps) is not used for image processing.

[0222] Note that the same applies when the imaging condition applied to the target pixel P is the second imaging condition and the imaging condition applied to the reference pixels around the target pixel P is the first imaging condition. That is, in this case, the selection unit 322 corresponding to the block 111a to which the reference pixel to which the first imaging condition is applied belongs, and the selection unit 322 corresponding to the block 111a to which the reference pixel to which the second imaging condition is applied belongs perform data selection processing on the image data of the reference pixel as described in the above (Example 1) to (Example 3).

[0223] Note that, as described above, even if there are slight differences in the imaging conditions, they are regarded as the same imaging conditions.

[0224] Based on the image data of the reference pixel selected by the selection unit 322, the generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing, in the same manner as the image processing unit 33 (generation unit 33c) in the first embodiment.

[0225] FIG. 22 is a diagram schematically showing the processing of the image data (hereinafter referred to as first image data) from each pixel included in a partial region (hereinafter referred to as first region 141) of the imaging surface to which the first imaging condition is applied and the image data (hereinafter referred to as second image data) from each pixel included in a partial region (hereinafter referred to as second region 142) of the imaging surface to which the second imaging condition is applied. Note that FIG. 22 is a diagram for explaining the case where the imaging condition applied to the target pixel P is the first imaging condition in the above (Example 1) and (Example 2).

[0226] From each pixel included in the first region 141, first image data captured under the first imaging condition is output, and from each pixel included in the second region 142, second image data captured under the second imaging condition is output. The first image data is output to the selection unit 322 corresponding to the block 111a to which the pixel that generated the first image data belongs among the plurality of selection units 322 provided in the processing chip 114. In the following description, the plurality of selection units 322 respectively corresponding to the plurality of blocks 111a to which the pixels that generated the respective first image data belong are referred to as first selection units 151. Similarly, the second image data is output to the selection unit 322 corresponding to the block 111a to which each pixel that generated the second image data belongs among the plurality of selection units 322 provided in the processing chip 114. In the following description, the plurality of selection units 322 respectively corresponding to the plurality of blocks 111a to which the pixels that generated the respective second image data belong are referred to as second selection units 152.

[0227] For example, when the target pixel P is included in the first region 141, the first selection unit 151 selects the image data of the target pixel P and the image data of the reference pixels captured under the first imaging condition, and outputs them to the generation unit 323. Here, the selection unit 322 selects the image data from the same block, but the image data of another block captured under the first imaging condition may also be used. At this time, the selection unit 322 that has received the target pixel P and the selection unit 322 of another block captured under the first imaging condition may transmit and receive the information 181 about the first imaging condition required for the data selection process. On the other hand, the second selection unit 152 does not select the image data of the reference pixels captured under the second imaging condition and does not output the image data of the reference pixels captured under the second imaging condition to the generation unit 323. Note that the second selection unit 152 receives the information 181 about the first imaging condition required for the data selection process from the first selection unit 151, for example. Similarly, for example, when the target pixel P is included in the second region, the second selection unit 152 selects the image data of the target pixel P and the image data of the reference pixels captured under the second imaging condition, and outputs them to the generation unit 323. On the other hand, the first selection unit 151 does not select the image data of the reference pixels captured under the first imaging condition and does not output the image data of the reference pixels captured under the first imaging condition to the generation unit 323. Note that the first selection unit 151 receives the information about the second imaging condition required for the data selection process from the second selection unit 152, for example.

[0228] After the above-described preprocessing, the generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing based on the image data from the first selection unit 151 and the second selection unit 152, and outputs the image data after the image processing.

[0229] 2. When performing focus detection processing Similar to the first embodiment, the control unit 34 (AF calculation unit 34d) performs focus detection processing using the image data corresponding to a predetermined position (focus detection position) on the imaging screen. Note that different imaging conditions are set between the divided regions. When the focus detection position in the AF operation is located at the boundary portion of the divided regions, that is, when the focus detection position is divided into the first region and the second region, in this embodiment, as described in 2-2 below, the control unit 34 (AF calculation unit 34d) causes the selection unit 322 to perform data selection processing.

[0230] 2-1. When the signal data for focus detection to which the first imaging condition is applied and the signal data for focus detection to which the second imaging condition is applied are not mixed in the image data from the pixels within the frame 170 in FIG. 13 In this case, the selection unit 322 selects all the signal data for focus detection from the pixels within the frame 170 and outputs it to the generation unit 323. The control unit 34 (AF calculation unit 34d) performs focus detection processing using the signal data for focus detection by the focus detection pixels indicated by the frame 170.

[0231] 2-2. When the signal data for focus detection to which the first imaging condition is applied and the signal data for focus detection to which the second imaging condition is applied are mixed in the signal data for focus detection from the pixels within the frame 170 in FIG. 13 In this case, the control unit 34 (AF calculation unit 34d) causes the selection unit 322 corresponding to the block 111a to which the pixels within the frame 170 belong to perform data selection processing as in the following (Example 1) to (Example 3). Then, the control unit 34 (AF calculation unit 34d) performs focus detection processing using the signal data for focus detection after the data selection processing.

[0232] (Example 1) For example, assume that only the ISO sensitivity differs between the first imaging condition and the second imaging condition, the ISO sensitivity of the first imaging condition is 100, and the ISO sensitivity of the second imaging condition is 800. In this case, the selection unit 322 corresponding to the block 111a to which the pixels to which the first imaging condition is applied belong selects the signal data for focus detection under the first imaging condition. Then, the selection unit 322 corresponding to the block 111a to which the pixels to which the second imaging condition is applied belong does not select the signal data for focus detection under the second imaging condition. That is, among the signal data for focus detection from the pixels within the frame 170, the signal data for focus detection under the first imaging condition is used for the focus detection process, and the signal data for focus detection under the second imaging condition different from the first imaging condition is not used for the focus detection process.

[0233] (Example 2) For example, assume that only the shutter speed differs between the first imaging condition and the second imaging condition, the shutter speed of the first imaging condition is 1 / 1000 second, and the shutter speed of the second imaging condition is 1 / 100 second. In this case, the selection unit 322 corresponding to the block 111a to which the pixels to which the first imaging condition is applied belong selects the signal data for focus detection under the first imaging condition. Then, the selection unit 322 corresponding to the block 111a to which the pixels to which the second imaging condition is applied belong does not select the signal data for focus detection under the second imaging condition. That is, among the signal data for focus detection from the pixels within the frame 170, the signal data for focus detection under the first imaging condition is used for the focus detection process, and the signal data for focus detection under the second imaging condition different from the first imaging condition is not used for the focus detection process.

[0234] (Example 3) For example, assume that only the frame rate differs between the first imaging condition and the second imaging condition (the charge accumulation time is the same), the frame rate of the first imaging condition is 30 fps, and the frame rate of the second imaging condition is 60 fps. In this case, the selection unit 322 corresponding to the block 111a to which the pixels to which the first imaging condition is applied belong selects the signal data for focus detection of the pixels of the first imaging condition. Also, the selection unit 322 corresponding to the block 111a to which the pixels to which the second imaging condition is applied belong selects, for the image data of the second imaging condition (60 fps), the signal data for focus detection of the frame image whose acquisition timing is close to that of the frame image acquired under the first imaging condition (30 fps). That is, among the signal data for focus detection of the second imaging condition (60 fps), the signal data for focus detection of the frame image whose acquisition timing is close to that of the frame image of the first imaging condition (30 fps) is used for the focus detection process, and the signal data for focus detection of the frame image whose acquisition timing is different from that of the frame image of the first imaging condition (30 fps) is not used for the focus detection process.

[0235] Note that, as described above, even if there are some differences in the imaging conditions, they are regarded as the same imaging conditions. Also, in the above (Example 1) and (Example 2), an example of selecting the signal data for focus detection of the first imaging condition from the signal data for focus detection surrounded by the frame 170 has been described, but the signal data for focus detection of the second imaging condition among the signal data for focus detection surrounded by the frame 170 may be selected. Note that when the focus detection position is divided into the first and second regions and the area of the first region is larger than the area of the second region, it is desirable to select the image data of the first imaging condition, and conversely, when the area of the second region is larger than the area of the first region, it is desirable to select the image data of the second imaging condition.

[0236] FIG. 23 is a diagram schematically showing the processing of the first signal data and the second signal data related to the focus detection process. Note that FIG. 23 is a diagram for explaining the case where, in the above (Example 3), the signal data for focus detection of the first imaging condition is selected from the signal data generated from the region surrounded by the frame 170, and the signal data for focus detection of the second imaging condition is selected. From each pixel included in the first region 141, first signal data for focus detection captured under the first imaging condition is output, and from each pixel included in the second region 142, second signal data for focus detection captured under the second imaging condition is output. The first signal data from the first region 141 is output to the first selection unit 151. Similarly, the second signal data from the second region 142 is output to the second selection unit 152. The first processing unit 151 selects the first signal data of the first imaging condition and outputs it to the AF calculation unit 34d. The second processing unit 152 selects the second signal data of the second imaging condition and outputs it to the AF calculation unit 34d. After the above-described preprocessing, the AF calculation unit 34d calculates the first defocus amount from the first signal data from the first processing unit 151. Further, the AF calculation unit 34d calculates the second defocus amount from the second signal data from the first processing unit 151. Then, the AF calculation unit 34d outputs a drive signal for moving the focus lens of the imaging optical system 31 to the in-focus position using the first defocus amount and the second defocus amount.

[0237] Note that, as in the above (Example 1) and (Example 2), when the signal data for focus detection under the first imaging condition is selected from the signal data of the region surrounded by the frame 170 and the signal data for focus detection under the second imaging condition is not selected, the following processing is performed on the first signal data and the second signal data.

[0238] For example, when performing focus detection processing using the first signal data of the first imaging condition, the first processing unit 151 selects the first signal data of the first imaging condition and outputs it to the generation unit 323. The second processing unit 152 does not select the second signal data of the second imaging condition and does not output the second signal data of the reference pixel captured under the second imaging condition to the generation unit 323. Note that the second processing unit 152 receives information 181 about the first imaging condition necessary for the data selection process, for example, from the first processing unit 151.

[0239] After the above-described preprocessing, the AF calculation unit 34d performs a focus detection process based on the first signal data from the first processing unit 151, and outputs a drive signal for moving the focus lens of the imaging optical system 31 to the in-focus position based on the calculation result.

[0240] Here, an example will be described in which the subject to be focused is located across the region where the first imaging condition is set and the region where the second imaging condition is set. When the subject to be focused is located across the region where the first imaging condition is set and the region where the second imaging condition is set, the selection unit 322 corresponding to the block 111a to which the pixels to which the first imaging condition is applied belong selects the first signal data for focus detection of the first imaging condition. Further, the selection unit 322 corresponding to the block 111a to which the pixels to which the second imaging condition is applied belong selects the second signal data for focus detection of the second imaging condition. Then, the control unit 34 (AF calculation unit 34d) calculates the first defocus amount from the selected first signal data for focus detection. Further, the control unit 34 (AF calculation unit 34d) calculates the second defocus amount from the selected second signal data for focus detection. Then, the control unit 34 (AF calculation unit 34d) performs a focus detection process using the first defocus amount and the second defocus amount. Specifically, for example, the control unit 34 (AF calculation unit 34d) calculates the average of the first defocus amount and the second defocus amount, and calculates the moving distance of the lens. Further, the control unit 34 (AF calculation unit 34d) may select the value with the smaller moving distance of the lens among the first defocus amount and the second defocus amount. Further, the control unit 34 (AF calculation unit 34d) may select a value indicating that the subject is closer from the first defocus amount and the second defocus amount. Also, when the subject to be focused on is located across the area where the first imaging condition is set and the area where the second imaging condition is set, the control unit 34 (AF calculation unit 34d) may select the area with the larger area of the subject area and select the photoelectric conversion signal for focus detection. For example, when 70% of the area of the face of the subject to be focused on is in the area where the first imaging condition is set and 30% is in the second area, the control unit 34 (AF calculation unit 34d) selects the photoelectric conversion signal for focus detection under the first imaging condition. Note that the ratios (percentages) of the areas described above are merely examples and are not limited thereto.

[0241] 3. When performing subject detection processing A case will be described where different imaging conditions are set between the divided areas and the search range 190 includes the boundaries of the divided areas.

[0242] 3-1. When the image data to which the first imaging condition is applied and the image data to which the second imaging condition is applied are not mixed in the image data of the search range 190 in FIG. 14 In this case, the selection unit 322 selects all the image data from the pixels of the search range 190 and outputs it to the generation unit 323. The control unit 34 (AF calculation unit 34d) performs subject detection processing using the image data from the pixels of the search range 190.

[0243] 3-2. When the image data to which the first imaging condition is applied and the image data to which the second imaging condition is applied are mixed in the image data of the search range 190 in FIG. 14 (a) When performing the above-described focus detection processing, in the same manner as (Example 1) and (Example 2), when only the ISO sensitivity differs between the first imaging condition and the second imaging condition, or when only the shutter speed differs between the first imaging condition and the second imaging condition In this case, as shown in FIG. 24, the control unit 34 (object detection unit 34a) causes the selection unit 322 (first selection unit 151) corresponding to the block 111a to which the pixels to which the first imaging condition is applied belong to select the image data of the first imaging condition used for the subject detection process from the image data in the search range 190. Note that FIG. 24 is a diagram schematically showing the processing of the first image data and the second image data related to the subject detection process. Then, the control unit 34 (object detection unit 34a) performs a subject detection process using the image data after the data selection process. In addition, the control unit 34 (object detection unit 34a) causes the selection unit 322 (second selection unit 152) corresponding to the block 111a to which the pixels to which the second imaging condition is applied belong to select the image data of the second imaging condition used for the subject detection process from the image data in the search range 190. Then, the control unit 34 (object detection unit 34a) performs a subject detection process using the image data after the data selection process. Then, the control unit 34 (object detection unit 34a) can detect the subject detection within the search range 190 by aligning the boundaries between the subject region detected from the image data of the first imaging condition and the subject region detected from the image data of the second imaging condition.

[0244] (b) When only the frame rate is different between the first imaging condition and the second imaging condition in the same manner as in (Example 3) when performing the above-described focus detection process In this case, the control unit 34 (object detection unit 34a) causes the selection unit 322 (first selection unit 151) corresponding to the block 111a to which the pixels to which the first imaging condition is applied belong to select the image data of the first imaging condition used for the subject detection process from the image data in the search range 190. Then, the control unit 34 (object detection unit 34a) performs the subject detection process using the image data after the data selection process. Also, the control unit 34 (object detection unit 34a) causes the selection unit 322 (second selection unit 152) corresponding to the block 111a to which the pixels to which the second imaging condition is applied belong to select only the image data of the frame image acquired under the first imaging condition (30 fps) and the image data of the frame image with a timing close to the acquisition timing as the image data of the second imaging condition (60 fps) used for the subject detection process from the image data in the search range 190. Then, the control unit 34 (object detection unit 34a) performs the subject detection process using the image data after the data selection process. Then, the control unit 34 can detect the subject detection within the search range 190 by aligning the boundaries between the subject regions detected from the image data of the first imaging condition and the subject regions detected from the image data of the second imaging condition.

[0245] In addition, when the search range 190 is divided into the first and second regions and the area of the first region is larger than the area of the second region, the image data of the first imaging condition may be selected and the image data of the second imaging condition may not be selected. Also, when the area of the second region is larger than the area of the first region, the image data of the second imaging condition may be selected and the image data of the first imaging condition may not be selected.

[0246] 4. When setting the imaging conditions A case will be described in which the area of the imaging screen is divided and new photometry is performed to determine the exposure conditions in a state where different imaging conditions are set between the divided regions.

[0247] 4-1. When the image data to which the first imaging condition is applied and the image data to which the second imaging condition is applied are not mixed in the image data of the photometry range In this case, the selection unit 322 selects all the image data from the pixels in the photometric range and outputs it to the generation unit 323. The control unit 34 (setting unit 34b) performs exposure calculation processing using the image data from the pixels constituting the photometric range.

[0248] 4-2. When the image data to which the first imaging condition is applied and the image data to which the second imaging condition is applied are mixed in the image data of the photometric range (a) When performing the above-described focus detection processing, (Example 1), (Example 2) When only the ISO sensitivity differs between the first imaging condition and the second imaging condition, or when only the shutter speed differs between the first imaging condition and the second imaging condition In this case, as shown in FIG. 25, the control unit 34 (object detection unit 34a) causes the selection unit 322 corresponding to the block 111a to which the pixels to which the first imaging condition is applied belong, to select, from the image data of the photometric range, the image data of the first imaging condition used for exposure calculation processing, in the same manner as (a) in the case of performing the above-described subject detection processing. Further, the control unit 34 (object detection unit 34a) causes the selection unit 322 corresponding to the block 111a to which the pixels to which the second imaging condition is applied belong, to select, from the image data of the photometric range, the image data of the second imaging condition used for exposure calculation processing. Note that FIG. 25 is a diagram schematically showing the processing of the first image data and the second image data regarding the setting of imaging conditions such as exposure calculation processing. Then, the control unit 34 (setting unit 34b) performs exposure calculation processing for each of the region where the first imaging condition is set and the region where the second imaging condition is set, using the image data after the data selection processing. In this way, when there are a plurality of regions with different imaging conditions in the photometric range, the control unit 34 (setting unit 34b) performs data selection processing for measuring each region, and performs exposure calculation processing using the image data of the data selection processing.

[0249] (b) When only the frame rate differs between the first imaging condition and the second imaging condition, in the same manner as (Example 3) in the case of performing the above-described focus detection processing In this case, similar to (b) when the control unit 34 (object detection unit 34a) performs subject detection processing, the control unit 34 (object detection unit 34a) causes the selection unit 322 corresponding to the block 111a to which the pixels to which the first imaging condition is applied belong to select the image data of the first imaging condition to be used for exposure calculation processing from the image data of the photometry range. Also, similar to (Example 3) when performing the above-described focus detection processing, the control unit 34 (object detection unit 34a) causes the selection unit 322 corresponding to the block 111a to which the pixels to which the second imaging condition is applied belong to select, from the image data of the photometry range, only the image data of the frame image acquired under the first imaging condition (30 fps) and the image data of the frame image with a timing close to the acquisition timing as the image data of the second imaging condition (60 fps) to be used for exposure calculation processing. Then, the control unit 34 (setting unit 34b) performs exposure calculation processing using the image data after the data selection processing, similar to the case of (a) described above.

[0250] Note that when the photometry range is divided into the first and second regions and the area of the first region is larger than the area of the second region, the image data of the first imaging condition may be selected and the image data of the second imaging condition may not be selected. Also, when the area of the second region is larger than the area of the first region, the image data of the second imaging condition may be selected and the image data of the first imaging condition may not be selected.

[0251] According to the third embodiment described above, the following operational effects can be obtained. (1) The camera 1C can perform imaging by changing the imaging conditions for each unit division of the imaging surface, and includes an imaging element 32a that generates first image data from a first region composed of at least one unit division imaged under the first imaging condition and second image data from a second region composed of at least one unit division imaged under a second imaging condition different from the first imaging condition. The camera 1C includes a plurality of selection units 322 provided corresponding to each unit division or each composite division having a plurality of unit divisions, and that select or do not select the image data from the corresponding unit division or the unit divisions within the corresponding composite division. The imaging element 32a is provided on the back-illuminated imaging chip 111. The plurality of selection units 322 are provided on the image processing chip 114. As a result, the data selection process of the image data can be processed in parallel by a plurality of selection units 322, so that the processing load on the selection unit 322 can be reduced.

[0252] (2) The back-illuminated imaging chip 111 and the image processing chip 114 are stacked. As a result, the image sensor 32a and the image processing unit 32c can be easily connected.

[0253] (3) The camera 1C includes a generation unit 323 that generates an image from the selected image data selected by the selection unit 322. As a result, since the preprocessing by the plurality of selection units 322 is performed in parallel in a short time, the time until an image is generated can be shortened.

[0254] (4) The camera 1C can perform imaging by changing the imaging conditions for each unit section of the imaging surface, and includes an image sensor 32a that generates first image data from a first region composed of at least one unit section that has imaged the optical image incident through the imaging optical system under a first imaging condition, and second image data from a second region composed of at least one unit section that has imaged the incident optical image under a second imaging condition different from the first imaging condition. The camera 1C is provided corresponding to each unit section or each composite section having a plurality of unit sections, and includes a plurality of selection units 322 that select or do not select the image data from the corresponding unit section or the unit sections within the corresponding composite section. The camera 1C includes an AF calculation unit 34d that detects information for moving the imaging optical system based on the selected selected image data. The image sensor 32a is provided on the back-illuminated imaging chip 111. The plurality of correction units 322 are provided on the image processing chip 114. As a result, the data selection process of the image data can be processed in parallel by a plurality of selection units 322, so that the processing load on the selection unit 322 can be reduced, and since the preprocessing by the plurality of selection units 322 is performed in parallel in a short time, the time until the start of the focus detection process in the AF calculation unit 34d can be shortened, contributing to the speeding up of the focus detection process.

[0255] (5) The camera 1C can capture images by changing the imaging conditions for each unit division of the imaging surface, and generates first image data from a first region composed of at least one unit division that has captured the subject image incident through the imaging optical system under a first imaging condition, and second image data from a second region composed of at least one unit division that has captured the incident subject image under a second imaging condition different from the first imaging condition. The camera 1C includes an imaging element 32a. The camera 1C is provided corresponding to each unit division or each composite division having a plurality of unit divisions, and includes a plurality of selection units 322 that select or do not select image data from the corresponding unit division or the unit divisions within the corresponding composite division. The camera 1C includes an object detection unit 34a that detects an object from the subject image based on the selected selected image data. The imaging element 32a is provided on the back-illuminated imaging chip 111. The plurality of correction units 322 are provided on the image processing chip 114. Accordingly, since the data selection process of the image data can be parallel-processed by the plurality of selection units 322, the processing load on the selection unit 322 can be reduced, and since the pre-processing by the plurality of selection units 322 is performed in a short time by parallel processing, the time until the start of the subject detection process in the object detection unit 34a can be shortened, contributing to the speeding up of the subject detection process.

[0256] (6) The camera 1C can capture images by changing the imaging conditions for each unit division of the imaging surface, and generates first image data from a first region composed of at least one unit division that has captured the light image incident through the imaging optical system under a first imaging condition, and second image data from a second region composed of at least one unit division that has captured the incident light image under a second imaging condition different from the first imaging condition. The camera 1C includes an imaging element 32a. The camera 1C is provided corresponding to each unit division or each composite division having a plurality of unit divisions, and includes a plurality of selection units 322 that select or do not select image data from the corresponding unit division or the unit divisions within the corresponding composite division. The camera 1C includes a setting unit 34b that sets imaging conditions based on the selected selected image data. The imaging element 32a is provided on the back-illuminated imaging chip 111. The plurality of correction units 322 are provided on the image processing chip 114. As a result, the data selection process of the image data can be processed in parallel by a plurality of selection units 322. Therefore, the processing load on the selection unit 322 can be reduced, and since the preprocessing by the plurality of selection units 322 is performed in a short time by parallel processing, the time until the start of the imaging condition setting process in the setting unit 34b can be shortened, contributing to the speeding up of the imaging condition setting process.

[0257] (Modification Example of the Third Embodiment) The following modifications are also within the scope of the present invention, and it is also possible to combine one or more of the modification examples with the above-described embodiments. (Modification Example 10) As shown in FIGS. 16(a) to 16(c) in Modification Example 1 of the first and second embodiments, the processing of the first image data and the second image data when the first region and the second region are arranged on the imaging surface of the imaging device 32a will be described. Also in this modification example, similar to Modification Example 1, in any of the cases of FIGS. 16(a) to 16(c), the first image based on the image signal read from the first region and the second image based on the image signal read from the second region are respectively generated by the pixel signals read from the imaging device 32a that has performed imaging of one frame. Also in this modification example, similar to Modification Example 1, the control unit 34 uses the first image for display and uses the second image for detection. It is assumed that the first imaging condition is set for the first region that captures the first image, and the second imaging condition different from the first imaging condition is set for the second region that captures the second image.

[0258] 1. As an example, the case where the first imaging condition is the same for the entire first region of the imaging screen and the second imaging condition is also the same for the entire second region of the imaging screen will be described with reference to FIG. 26. FIG. 26 is a diagram schematically showing the processing of the first image data and the second image data.

[0259] From each pixel included in the first area 141, first image data captured under the first imaging condition is output, and from each pixel included in the second area 142, second image data and second signal data captured under the second imaging condition are output. The first image data from the first area 141 is output to the first selection unit 151. Similarly, the second image data and the second signal data from the second area 142 are output to the second selection unit 152.

[0260] In this example, since the first imaging condition is the same for the entire first area of the imaging screen, the first selection unit 151 selects all the first image data from each pixel in the first area. Also, since the second imaging condition is the same for the entire second area of the imaging screen, the second selection unit 152 selects all the second image data from each pixel in the second area. Note that since the first imaging condition and the second imaging condition are different, the second selection unit 152 does not select the second image data as data for image processing of the image data in the first area. Also, the second selection unit 152 receives information 181 about the first imaging condition, for example, from the first selection unit 151.

[0261] The generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing based on the first image data from the first selection unit 151, and outputs the image data after the image processing. The object detection unit 34a performs a process of detecting a subject element based on the second image data from the second selection unit 152, and outputs the detection result. The setting unit 34b performs a calculation process of imaging conditions such as exposure calculation processing based on the second image data from the second selection unit 152, and based on the calculation result, divides the imaging screen by the imaging unit 32 into a plurality of areas including the detected subject element, and re-sets the imaging conditions for the plurality of areas. The AF calculation unit 34d performs a focus detection process based on the second signal data from the second selection unit 152, and outputs a drive signal for moving the focus lens of the imaging optical system 31 to the in-focus position based on the calculation result.

[0262] 2. As another example, where the first imaging condition varies depending on the area of the imaging screen, that is, the first imaging condition varies depending on the partial area within the first area, and the second imaging condition is the same for the entire second area of the imaging screen, this will be described with reference to FIG. 27. FIG. 27 is a diagram schematically showing the processing of the first image data, the second image data, and the second signal data.

[0263] From each pixel included in the first area 141, first image data captured under the first imaging condition is output, and from each pixel included in the second area 142, second image data and second signal data captured under the second imaging condition are output. The first image data from the first area 141 is output to the first selection unit 151. Similarly, the second image data from the second area 142 is output to the second selection unit 152.

[0264] As described above, in this example, the first imaging condition varies depending on the area of the imaging screen. That is, the first imaging condition varies depending on the partial area within the first area. The first selection unit 151 selects only the first image data under a certain imaging condition from the first image data from each pixel in the first area, and does not select the first image data under other imaging conditions. Also, since the second imaging condition is the same for the entire second area of the imaging screen, the second selection unit 152 selects all the second image data from each pixel in the second area. Note that since the first imaging condition and the second imaging condition are different, the second selection unit 152 does not select the second image data as data for image processing of the image data in the first area. Also, the second selection unit 152 receives information 181 about the first imaging condition, for example, from the first selection unit 151.

[0265] The generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing based on some of the first image data selected by the first selection unit 151, and outputs the image data after the image processing. The object detection unit 34a performs a process of detecting subject elements based on the second image data from the second selection unit 152, and outputs the detection result. The setting unit 34b performs calculation processing of imaging conditions such as exposure calculation processing based on the second image data from the second selection unit 152, and based on the calculation result, divides the imaging screen by the imaging unit 32 into a plurality of regions including the detected subject elements, and re-sets the imaging conditions for the plurality of regions. The AF calculation unit 34d performs focus detection processing based on the second signal data from the second selection unit 152, and outputs a drive signal for moving the focus lens of the imaging optical system 31 to the in-focus position based on the calculation result.

[0266] 3. Further, as another example, a case where the first imaging condition is the same for the entire first region of the imaging screen and the second imaging condition varies depending on the region of the imaging screen will be described with reference to FIG. 28. FIG. 28 is a diagram schematically showing the processing of the first image data and the second image data.

[0267] From each pixel included in the first region 141, first image data captured under the same first imaging condition for the entire first region of the imaging screen is output, and from each pixel included in the second region 142, second image data captured under different second imaging conditions depending on the region of the imaging screen is output. The first image data from the first region 141 is output to the first selection unit 151. Similarly, the second image data and the second signal data from the second region 142 are output to the second selection unit 152.

[0268] In this example, since the first imaging condition is the same for the entire first region of the imaging screen, the first selection unit 151 selects all the first image data from each pixel of the first region. Also, the second imaging condition varies depending on the region of the imaging screen. That is, the second imaging condition varies depending on the partial regions within the second region. The second selection unit 152 selects only the second image data of a certain imaging condition from the second image data from each pixel of the second region, and does not select the second image data of other imaging conditions. Note that since the first imaging condition and the second imaging condition are different, the second selection unit 152 does not select the second image data as data for image processing of the image data of the first region. Further, the second selection unit 152 receives information 181 about the first imaging condition, for example, from the first selection unit 151.

[0269] Based on the first image data from the first selection unit 151, the generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, edge enhancement processing, and noise reduction processing, and outputs the image data after the image processing. The object detection unit 34a performs a process of detecting subject elements based on a part of the second image data selected by the second selection unit 152, and outputs the detection result. The setting unit 34b performs a calculation process of imaging conditions such as exposure calculation processing based on a part of the second image data selected by the second selection unit 152. Based on the calculation result, the imaging screen by the imaging unit 32 is divided into a plurality of regions including the detected subject elements, and the imaging conditions are reset for the plurality of regions. The AF calculation unit 34d performs a focus detection process based on a part of the second signal data selected by the second selection unit 152, and outputs a drive signal for moving the focus lens of the imaging optical system 31 to the in-focus position based on the calculation result.

[0270] 4. Furthermore, as another example, a case where the first imaging condition varies depending on the region of the imaging screen and the second imaging condition varies depending on the region of the imaging screen will be described with reference to FIG. 29. FIG. 29 is a diagram schematically showing the processing of the first image data, the second image data, and the second signal data.

[0271] From each pixel included in the first region 141, first image data captured under a first imaging condition that varies depending on the region of the imaging screen is output. From each pixel included in the second region 142, second image data and second signal data captured under a second imaging condition that varies depending on the region of the imaging screen are output. The first image data from the first region 141 is output to the first selection unit 151. Similarly, the second image data and the second signal data from the second region 142 are output to the second selection unit 152.

[0272] As described above, in this example, the first imaging condition varies depending on the area of the imaging screen. That is, the first imaging condition varies depending on the partial area within the first area. The first selection unit 151 selects only the first image data of a certain imaging condition from the first image data from each pixel in the first area, and does not select the first image data of other imaging conditions. Also, the second imaging condition varies depending on the area of the imaging screen. That is, the second imaging condition varies depending on the partial area within the second area. The second selection unit 152 selects only the second image data of a certain imaging condition from the second image data from each pixel in the second area, and does not select the second image data of other imaging conditions. Note that since the first imaging condition and the second imaging condition are different, the second selection unit 152 does not select the second image data as data for image processing of the image data in the first area. Also, the second selection unit 152 receives information 181 about the first imaging condition, for example, from the first selection unit 151.

[0273] The generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing based on some of the first image data selected by the first selection unit 151, and outputs the image data after the image processing. The object detection unit 34a performs a process of detecting a subject element based on some of the second image data selected by the second selection unit 152, and outputs the detection result. The setting unit 34b performs a calculation process of imaging conditions such as exposure calculation processing based on some of the second image data selected by the second selection unit 152, and based on the calculation result, divides the imaging screen by the imaging unit 32 into a plurality of regions including the detected subject element, and re-sets the imaging conditions for the plurality of regions. The AF calculation unit 34d performs a focus detection process based on some of the second signal data selected by the second selection unit 152, and outputs a drive signal for moving the focus lens of the imaging optical system 31 to the in-focus position based on the calculation result.

[0274] (Modification Example 11) In the above-described third embodiment, one of the selection units 322 corresponds to one of the blocks 111a (unit division). However, one of the selection units 322 may correspond to one of the composite blocks (composite division) having a plurality of blocks 111a (unit division). In this case, the selection unit 322 sequentially performs data selection processing on the image data from the pixels belonging to the plurality of blocks 111a included in the composite block. Even if a plurality of selection units 322 are provided corresponding to each composite block having a plurality of blocks 111a, since the data selection processing of the image data can be performed in parallel by the plurality of selection units 322, the processing load on the selection unit 322 can be reduced, and an appropriate image can be generated in a short time from the image data generated in regions with different imaging conditions.

[0275] (Modification Example 12) In the above-described third embodiment, the generation unit 323 is provided inside the imaging unit 32A. However, the generation unit 323 may be provided outside the imaging unit 32A. Even if the generation unit 323 is provided outside the imaging unit 32A, the same effects as those described above can be obtained.

[0276] (Modification Example 13) In the above-described third embodiment, the stacked imaging device 100A further includes an image processing chip 114 that performs the above-described pre-processing and image processing, in addition to the back-illuminated imaging chip 111, the signal processing chip 112, and the memory chip 113. However, the image processing chip 114 may not be provided in the stacked imaging device 100A, and the image processing unit 32c may be provided in the signal processing chip 112. Note that the above-described embodiments and modification examples may be combined with each other.

[0277] In the above, various embodiments and modification examples have been described, but the present invention is not limited to these contents. Other aspects conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.

[0278] The above-described embodiments and modification examples also include the following apparatuses. (1) An imaging device having an imaging region for imaging a subject, a setting unit for setting imaging conditions of the imaging region, a selection unit for selecting pixels for use in interpolation from the pixels included in the imaging region, and a generation unit for generating an image of the subject imaged in the imaging region using a signal interpolated by a signal output from the pixels selected by the selection unit, wherein the selection unit is an imaging device in which at least some of the pixels to be selected are different depending on the imaging conditions set by the setting unit. (2) In the imaging device as described in (1), the imaging device has a first imaging region for imaging a subject and a second imaging region for imaging a subject, the setting unit sets imaging conditions of the first imaging region and imaging conditions of the second imaging region, the selection unit varies at least some of the pixels to be selected for use in interpolation of the pixels included in the first imaging region depending on the imaging conditions of the second imaging region set by the setting unit from among the pixels included in the first imaging region and the pixels included in the second imaging region, and the generation unit generates an image of the subject imaged in the first imaging region using a signal interpolated by a signal output from the pixels selected by the selection unit. (3) In the imaging device as described in (2), the selection unit varies at least some of the pixels to be selected for use in interpolation of the pixels included in the first imaging region depending on the imaging conditions of the first imaging region and the imaging conditions of the second imaging region set by the setting unit. (4) In the imaging device as described in (2) or (3), the selection unit selects pixels for use in the interpolation from at least one of the first imaging region and the second imaging region. (5) In the imaging device as described in (4), the selection unit selects pixels of the second imaging region as the pixels for use in the interpolation depending on the imaging conditions of the second imaging region set by the setting unit. (6) In the imaging device as described in (2) to (5), when a first imaging condition is set for the first imaging region and the second imaging region by the setting unit, the selection unit selects pixels included in the second imaging region. In the imaging device as described in (7) and (6), the selection unit selects pixels in the second imaging area as the pixels to be used for the interpolation based on the value related to the exposure according to the imaging conditions of the second imaging area set by the setting unit. In the imaging device as described in (8) and (7), the selection unit selects pixels in the second imaging area as the pixels to be used for the interpolation based on the value related to the exposure according to the imaging conditions of the first imaging area set by the setting unit and the value related to the exposure according to the imaging conditions of the second imaging area. In the imaging device as described in (9) and (8), the selection unit selects pixels in the second imaging area as the pixels to be used for the interpolation when the difference between the number of exposure steps according to the imaging conditions of the first imaging area set by the setting unit and the number of exposure steps according to the imaging conditions of the second imaging area is 0.3 steps or less. In the imaging device as described in (10) to (2) and (9), when the first imaging conditions are set in the first imaging area and the second imaging conditions are set in the second imaging area by the setting unit, the selection unit selects pixels included in the first imaging area. In the imaging device as described in (11) and (10), when the first imaging conditions are set in the first imaging area and the second imaging conditions are set in the second imaging area by the setting unit, the selection unit selects pixels included in the first imaging area without selecting pixels in the second imaging area. In the imaging device as described in (12) to (2) and (11), the selection unit selects a third pixel in the first imaging area, the distance of which from the first pixel is longer than the distance between the first pixel and the second pixel in the second imaging area, as the pixel for interpolating the first pixel in the first imaging area. In the imaging device as described in (13) to (2) and (12), the number of pixels to be selected by the selection unit varies according to the imaging conditions set for the second imaging area by the setting unit. In the imaging device as described in (14) and (13), when the first imaging conditions are set in the first imaging area and the second imaging conditions are set in the second imaging area by the setting unit, the selection unit selects fewer pixels than when the first imaging conditions are set in both the first imaging area and the second imaging area. An imaging device includes: an imaging element having a first imaging region set to image a subject under a first imaging condition, a second imaging region set to image a subject under a second imaging condition different from the first imaging condition, and a third imaging region set to image a subject under a third imaging condition different from the second imaging condition; a selection unit that selects pixels for use in interpolating pixels included in the first imaging region from among pixels included in the second imaging region and pixels included in the third imaging region; and a generation unit that generates an image of the subject imaged in the first imaging region using a signal interpolated by a signal output from the pixels selected by the selection unit. An imaging device includes: an imaging element having a first imaging region set to image a subject under a first imaging condition and a second imaging region set to image a subject under a second imaging condition different from the first imaging condition; a selection unit that selects pixels for use in interpolating pixels included in the first imaging region from among pixels included in the first imaging region and pixels included in the second imaging region; and a generation unit that generates an image of the subject imaged in the first imaging region using a signal interpolated by a signal output from the pixels selected by the selection unit. An imaging device includes: an imaging element having a first imaging region for imaging a subject, a second imaging region for imaging a subject, and a third imaging region for imaging a subject; a setting unit that sets the imaging condition of the first imaging region to a first imaging condition, sets the imaging condition of the second imaging region to a second imaging condition different from the first imaging condition, and sets the imaging condition of the third imaging region to a third imaging condition in which the difference from the first imaging condition is smaller than the difference between the first imaging condition and the second imaging condition; a selection unit that selects pixels for use in interpolating pixels included in the first imaging region from among pixels included in the first imaging region, pixels included in the second imaging region, and pixels included in the third imaging region; and a generation unit that generates an image of the subject imaged in the first imaging region using a signal interpolated by a signal output from the pixels selected by the selection unit. An imaging device including: an imaging element having a first imaging region for imaging a subject, a second imaging region for imaging the subject, and a third imaging region for imaging the subject, the distance between the first imaging region and the third imaging region being longer than the distance between the first imaging region and the second imaging region; a setting unit configured to set imaging conditions for the second imaging region to be different from the imaging conditions for the first imaging region; a selection unit configured to select pixels to be used for interpolation of the pixels included in the first imaging region from among the pixels included in the first imaging region, the pixels included in the second imaging region, and the pixels included in the third imaging region; and a generation unit configured to generate an image of the subject imaged in the first imaging region using a signal interpolated by a signal output from the pixels selected by the selection unit. (19) An imaging device including: an imaging element having an imaging region for imaging a subject; a setting unit configured to set imaging conditions for the imaging region; and a generation unit configured to generate an image of the subject imaged in the imaging region using a signal interpolated by a signal output from a pixel selected as a pixel to be used for interpolation and included in the imaging region, wherein at least a part of the pixels to be selected are different depending on the imaging conditions set by the setting unit. (20) An imaging device including: an imaging element having a first imaging region set to image a subject under a first imaging condition and a second imaging region set to image the subject under a second imaging condition different from the first imaging condition; and a generation unit configured to generate an image of the subject imaged in the first imaging region using a signal interpolated by a signal output from a pixel selected as a pixel to be used for interpolation of the pixels included in the first imaging region from among the pixels included in the first imaging region and the pixels included in the second imaging region. (21) An imaging device including: an imaging element having an imaging region for imaging a subject; a setting unit configured to set imaging conditions for the imaging region; and a generation unit configured to generate an image of the subject imaged in the imaging region using a signal in which noise is reduced by a signal output from a pixel selected from among the pixels included in the imaging region and configured to output a signal for reducing noise, wherein at least a part of the pixels to be selected are different depending on the imaging conditions set by the setting unit. An imaging device includes: an imaging element having a first imaging region set to image a subject under a first imaging condition, a second imaging region set to image the subject under a second imaging condition different from the first imaging condition, and a third imaging region set to image the subject under a third imaging condition different from the second imaging condition; a selection unit that selects, from among pixels included in the second imaging region and pixels included in the third imaging region, pixels included in the first imaging region for use in reducing noise; and a generation unit that generates an image of the subject imaged in the first imaging region using a signal with reduced noise, the signal being output from a pixel selected from among pixels included in the second imaging region and pixels included in the third imaging region and used as a signal for reducing noise of a signal of a pixel included in the first imaging region. An imaging device includes: an imaging element having a first imaging region set to image a subject under a first imaging condition and a second imaging region set to image the subject under a second imaging condition different from the first imaging condition; and a generation unit that generates an image of the subject imaged in the first imaging region using a signal interpolated by a signal output from a pixel selected from among pixels included in the first imaging region and pixels included in the second imaging region and used as a pixel for outputting a signal for reducing noise of pixels included in the first imaging region. An imaging device includes: an imaging element having an imaging region for imaging a subject; a setting unit that sets an imaging condition of the imaging region; and a generation unit that generates an image of the subject imaged in the imaging region using a signal processed by image processing with a signal output from a pixel selected as a pixel for image processing, wherein at least some of the pixels to be selected are different depending on the imaging condition set by the setting unit. An image processing device includes: a selection unit that selects a signal for use in interpolation from signals output from pixels included in an imaging region of an imaging element; and a generation unit that generates an image of a subject imaged in the imaging region using a signal interpolated by the signal selected by the selection unit, wherein at least some of the pixels to be selected are different depending on the imaging condition set for the imaging region. A selection unit that selects a signal for reducing noise from a signal output from pixels included in an imaging region of an imaging device, and a generation unit that generates an image of a subject imaged in the imaging region using a signal with reduced noise by the signal selected by the selection unit, wherein the selection unit is an image processing apparatus in which at least some of the pixels to be selected are different depending on imaging conditions set in the imaging region. A selection unit that selects a signal for use in interpolation from a signal output from pixels included in an imaging region of an imaging device, and a display unit that displays an image of a subject imaged in the imaging region generated using a signal interpolated by the signal selected by the selection unit, wherein the selection unit is an image processing apparatus in which at least some of the pixels to be selected are different depending on imaging conditions set in the imaging region. A selection unit that selects a signal for reducing noise from a signal output from pixels included in an imaging region of an imaging device, and a display unit that displays an image of a subject imaged in the imaging region generated using a signal with reduced noise by the signal selected by the selection unit, wherein the selection unit is an image processing apparatus in which at least some of the pixels to be selected are different depending on imaging conditions set in the imaging region.

[0279] Also, the above-described embodiments and modified examples include the following apparatuses. An imaging device including: an imaging device having a first region that images incident light under first imaging conditions and outputs first image data, a second region that images incident light under second imaging conditions different from the first imaging conditions and outputs second image data, and a third region that images incident light under the first imaging conditions and outputs third image data; and an image processing unit that generates an image based on first image data obtained by image processing using at least one of the first image data and the third image data and the second image data. In the imaging device as described in (1), the image processing unit performs image processing on the first image data using the third image data without using the second image data. (3) In the imaging device such as (2), the first image data is output from the pixel at the first position in the first region, the second image data is output from the pixel at the second position in the second region, the third image data is output from the pixel at the third position in the first region, and the distance from the first position to the third position is longer than the distance from the first position to the second position. (4) In the imaging device such as (1) to (3), the imaging element has a fourth region that captures incident light under the second imaging condition and outputs fourth image data, and the image processing unit generates an image using at least one of the second image data and the fourth image data and the processed first image data. (5) An imaging device including: an imaging element having a first region that captures incident light under a first imaging condition and outputs first image data, and a second region that captures incident light under a second imaging condition different from the first imaging condition and outputs second image data; and an image processing unit that generates an image based on the first image data processed using the first image data and the second image data. (6) In the imaging device such as (5), the image processing unit processes the first image data using the first image data without using the second image data. (7) In the imaging device such as (6), the image processing unit generates an image using the second image data processed using the second image data and the processed first image data. (8) In the imaging device such as (1) to (7), the first region outputs a plurality of the first image data, and the image processing unit processes the first image data to be processed among the plurality of first image data using other first image data among the plurality of first image data. (9) In the imaging device such as (8), the first image data is output from the pixel at the first position in the first region, the second image data is output from the pixel at the second position in the second region, the other first image data is output from the pixel at the third position in the first region, and the distance from the first position to the third position is longer than the distance from the first position to the second position. (10) In an imaging device such as (1) to (9), the first imaging condition and the second imaging condition include at least an accumulation time or an ISO sensitivity. (11) In an imaging device such as (1) to (10), the image processing unit processes the first image data by performing at least pixel defect correction processing, color interpolation processing, edge enhancement processing, or noise reduction processing. (12) In an imaging device such as (1) to (11), the image processing unit includes a selection unit that selects image data for processing the first image data. (13) In the imaging device such as (12), the imaging element is capable of imaging by changing imaging conditions for each unit region of the imaging surface, and the selection unit is provided corresponding to each unit region or each composite region having a plurality of the unit regions, and selects image data from the corresponding unit region or the unit regions within the corresponding composite region. (14) In the imaging device such as (13), the imaging element is provided on a first semiconductor substrate, and the selection unit is provided on a second semiconductor substrate. (15) In the imaging device such as (14), the first semiconductor substrate and the second semiconductor substrate are laminated. (16) A display device having a display unit that displays an image generated based on first image data output by imaging light incident on a first region of an imaging unit under a first imaging condition, second image data output by imaging light incident on a second region of the imaging unit under a second imaging condition different from the first imaging condition, and third image data output by imaging light incident on a third region of the imaging unit under the first imaging condition, the display device including an image processing unit that generates an image based on at least one of the first image data and the third image data after image processing and the second image data, and the display unit displays the image generated by the image processing unit. An image processing unit that generates an image based on the first image data obtained by performing image processing using the first image data output by capturing the light incident on the first region of the imaging unit under the first imaging conditions, and the second image data output by capturing the light incident on the second region of the imaging unit under second imaging conditions different from the first imaging conditions, and a display unit that displays the image generated by the image processing unit. (18) An image processing apparatus including an image processing unit that generates an image based on the first image data output by capturing the light incident on the first region of the imaging unit under the first imaging conditions, the second image data output by capturing the light incident on the second region of the imaging unit under second imaging conditions different from the first imaging conditions, and the third image data output by capturing the light incident on the third region of the imaging unit under the first imaging conditions, wherein the image processing unit generates an image based on the first image data obtained by performing image processing using at least one of the first image data and the third image data and the second image data. (19) An image processing apparatus including an image processing unit that generates an image based on the first image data obtained by performing image processing using the first image data output by capturing the light incident on the first region of the imaging unit under the first imaging conditions, and the second image data output by capturing the light incident on the second region of the imaging unit under second imaging conditions different from the first imaging conditions.

[0280] The disclosure of the following priority-based application is incorporated herein by reference. Japanese Patent Application No. 2015-195288 (filed on September 30, 2015)

Explanation of Reference Numerals

[0281] 1,1C… Camera 1B… Imaging System 32… Imaging Unit 32a, 100… Image Sensor 33… Image Processing Unit 33a, 321… Input Unit 33b, 322… Selection Unit 33c, 323… Generation Unit 34… Control Unit 34a… Object Detection Unit 34b… Region division unit 34d… Imaging control unit 35… Display unit 90… Predetermined range 1001… Imaging device 1002… Display device P… Pixel of interest

Claims

【Claim 1】 An imaging device having a first imaging region set to image a subject under a first imaging condition, a second imaging region set to image a subject under a second imaging condition different from the first imaging condition, and a third imaging region set to image a subject under the first imaging condition; a selection unit that selects pixels to be used for reducing noise in signals from the pixels included in the first imaging region from among the pixels included in the second imaging region and the pixels included in the third imaging region; and having; wherein the second imaging region is a region adjacent to the first imaging region; wherein the third imaging region is a region different from the first imaging region and is a region adjacent to at least one of the first imaging region and the second imaging region; an imaging device.

Images