Imaging device

Abstract

Perform appropriate processing in areas with different imaging conditions. [Solution] The imaging device includes an image sensor having a first imaging area set to image a subject under first imaging conditions, a second imaging area set to image a subject under second imaging conditions different from the first imaging conditions, and a third imaging area set to image a subject under first imaging conditions, and a selection unit that selects pixels from among the pixels included in the second imaging area and the pixels included in the third imaging area to be used to reduce noise from the signals of the pixels included in the first imaging area, wherein the second imaging area is an area adjacent to the first imaging area, and the third imaging area is an area different from the first imaging area and is an area adjacent to at least one of the first imaging area and the second imaging area.

Description

Technical Field

[0001] The present invention relates to an imaging device.

Background Art

[0002] There is known an imaging device equipped with an imaging element capable of setting different imaging conditions for each area of a screen (see Patent Document 1). However, when using image data generated in areas with different imaging conditions, there is a problem that it cannot be treated in the same manner as when using image data generated in areas with the same imaging conditions.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] An imaging device according to a first aspect of the present invention includes an imaging element having a first imaging area set to image a subject under a first imaging condition, a second imaging area set to image a subject under a second imaging condition different from the first imaging condition, and a third imaging area set to image a subject under the first imaging condition, and a selection unit that selects pixels to be used for reducing noise in signals from the pixels included in the first imaging area from among the pixels included in the second imaging area and the pixels included in the third imaging area. The second imaging area is an area adjacent to the first imaging area, the third imaging area is an area different from the first imaging area and is an area adjacent to at least one of the first imaging area and the second imaging area.

Brief Description of the Drawings

[0005] [Figure 1] It is a block diagram illustrating the configuration of a camera according to a first embodiment. [Figure 2]This is a cross-sectional view of a stacked image sensor. [Figure 3] This diagram illustrates the pixel arrangement and unit region of the imaging chip. [Figure 4] This is a diagram illustrating a circuit in a unit region. [Figure 5] This diagram schematically shows the image of a subject formed on the image sensor of a camera. [Figure 6] This diagram illustrates the screen for setting imaging conditions. [Figure 7] Figure 7(a) illustrates the vicinity of the boundary of the first region in the live view image, Figure 7(b) is a magnified view of the vicinity of the boundary, and Figure 7(c) is a magnified view of the pixel of interest and the reference pixel. Figure 7(d) is a magnified view of the pixel of interest and the reference pixel in the second embodiment. [Figure 8] Figure 8(a) illustrates the sequence of photoelectric conversion signals output from a pixel, Figure 8(b) illustrates the interpolation of the G color component image data, and Figure 8(c) illustrates the image data of the G color component after interpolation. [Figure 9] Figure 9(a) shows the image data of the R color component extracted from Figure 8(a), Figure 9(b) is a diagram illustrating the interpolation of the chrominance component Cr, and Figure 9(c) is a diagram illustrating the interpolation of the chrominance component Cr in the image data. [Figure 10] Figure 10(a) shows the image data of the B color component extracted from Figure 8(a), Figure 10(b) is a diagram illustrating the interpolation of the color difference component Cb, and Figure 10(c) is a diagram illustrating the interpolation of the image data of the color difference component Cb. [Figure 11] This diagram illustrates the position of focus detection pixels on the imaging surface. [Figure 12] This is a magnified view of a portion of the focus detection pixel line. [Figure 13] This is a magnified view of the focus detection area. [Figure 14] Figure 14(a) is an example of a template image representing the object to be detected, and Figure 14(b) is an example of a live view image and search area. [Figure 15] This flowchart explains the process of setting imaging conditions for each region and then performing the imaging. [Figure 16] FIGS. 16(a) to 16(c) are diagrams illustrating the arrangement of the first region and the second region on the imaging surface of the imaging device. [Figure 17] It is a block diagram illustrating the configuration of the imaging system according to Modification 8. [Figure 18] It is a diagram for explaining the supply of a program to a mobile device. [Figure 19] It is a block diagram illustrating the configuration of the camera according to the third embodiment. [Figure 20] It is a diagram schematically showing the correspondence between each block in the third embodiment and a plurality of selection units. [Figure 21] It is a cross-sectional view of a stacked imaging device. [Figure 22] It is a diagram schematically showing the processing of the first image data and the second image data related to image processing. [Figure 23] It is a diagram schematically showing the processing of the first image data and the second image data related to focus detection processing. [Figure 24] It is a diagram schematically showing the processing of the first image data and the second image data related to subject detection processing. [Figure 25] It is a diagram schematically showing the processing of the first image data and the second image data related to the setting of imaging conditions such as exposure calculation processing. [Figure 26] It is a diagram schematically showing the processing of the first image data and the second image data according to Modification 10. [Figure 27] It is a diagram schematically showing the processing of the first image data and the second image data according to Modification 10. [Figure 28] It is a diagram schematically showing the processing of the first image data and the second image data according to Modification 10. [Figure 29] It is a diagram schematically showing the processing of the first image data and the second image data according to Modification 10.

MODE FOR CARRYING OUT THE INVENTION

[0006] As an example of an electronic device equipped with an image processing apparatus according to the present embodiment, a digital camera will be described. The camera 1 (FIG. 1) is configured to be able to perform imaging under different conditions for each region of the imaging surface in the imaging device 32a. The image processing unit 33 performs appropriate processing for each region where the imaging conditions are different. Details of such a camera 1 will be described with reference to the drawings.

[0007] <Description of the camera> (First Embodiment) FIG. 1 is a block diagram illustrating the configuration of the camera 1 according to the first embodiment. In FIG. 1, the camera 1 includes an imaging optical system 31, an imaging unit 32, an image processing unit 33, a control unit 34, a display unit 35, an operation member 36, and a recording unit 37.

[0008] The imaging optical system 31 guides a light beam from the subject field to the imaging unit 32. The imaging unit 32 includes an imaging device 32a and a driving unit 32b, and photoelectrically converts the image of the subject formed by the imaging optical system 31. The imaging unit 32 can perform imaging under the same conditions over the entire imaging surface in the imaging device 32a, or can perform imaging under different conditions for each region of the imaging surface in the imaging device 32a. Details of the imaging unit 32 will be described later. The driving unit 32b generates a driving signal necessary to cause the imaging device 32a to perform accumulation control. An imaging instruction such as the charge accumulation time for the imaging unit 32 is transmitted from the control unit 34 to the driving unit 32b.

[0009] The image processing unit 33 includes an input unit 33a, a selection unit 33b, and a generation unit 33c. Image data acquired by the imaging unit 32 is input to the input unit 33a. The selection unit 33b performs preprocessing on the input image data. Details of the preprocessing will be described later. The generation unit 33c generates an image based on the input image data and the preprocessed image data. The generation unit 33c also performs image processing on the image data. Image processing includes, for example, color interpolation, pixel defect correction, edge enhancement, noise reduction, white balance adjustment, gamma correction, display brightness adjustment, and saturation adjustment. Furthermore, the generation unit 33c generates an image to be displayed by the display unit 35.

[0010] The control unit 34, for example, is composed of a CPU and controls the overall operation of the camera 1. For example, the control unit 34 performs a predetermined exposure calculation based on the photoelectric conversion signal acquired by the imaging unit 32, determines the exposure conditions necessary for proper exposure, such as the charge accumulation time (exposure time) of the image sensor 32a, the aperture value of the imaging optical system 31, and the ISO sensitivity, and instructs the drive unit 32b. It also determines image processing conditions to adjust saturation, contrast, sharpness, etc., according to the imaging scene mode set in the camera 1 and the type of subject element detected, and instructs the image processing unit 33. The detection of subject elements will be described later.

[0011] The control unit 34 includes an object detection unit 34a, a setting unit 34b, an imaging control unit 34c, and an AF calculation unit 34d. These are implemented in software by the control unit 34 executing a program stored in a non-volatile memory (not shown), but they may also be configured using ASICs or the like.

[0012] The object detection unit 34a detects subject elements from the image acquired by the imaging unit 32 by performing known object recognition processing, such as people (people's faces), animals such as dogs and cats (animal faces), plants, vehicles such as bicycles, cars and trains, buildings, stationary objects, landscapes such as mountains and clouds, and predetermined specific objects. The setting unit 34b divides the image captured by the imaging unit 32 into multiple regions that include the subject elements detected as described above.

[0013] The setting unit 34b further sets imaging conditions for multiple regions. The imaging conditions include the above-mentioned exposure conditions (charge accumulation time, gain, ISO sensitivity, frame rate, etc.) and the above-mentioned image processing conditions (for example, white balance adjustment parameters, gamma correction curve, display brightness adjustment parameters, saturation adjustment parameters, etc.). It is possible to set the same imaging conditions for all of the multiple regions, or to set different imaging conditions for the multiple regions.

[0014] The imaging control unit 34c controls the imaging unit 32 (image sensor 32a) and the image processing unit 33 by applying imaging conditions set for each region by the setting unit 34b. This makes it possible to have the imaging unit 32 perform imaging with different exposure conditions for each of the multiple regions, and to have the image processing unit 33 perform image processing with different image processing conditions for each of the multiple regions. The number of pixels constituting a region can be any number; for example, it could be 1000 pixels or 1 pixel. Also, the number of pixels may differ between regions.

[0015] The AF calculation unit 34d controls the autofocus (AF) operation to focus on the corresponding subject at a predetermined position on the imaging screen (referred to as the focus detection position). Based on the calculation result, the AF calculation unit 34d sends a drive signal to the drive unit 32b to move the focus lens of the imaging optical system 31 to the focus position. The processing performed by the AF calculation unit 34d for autofocus adjustment is also called focus detection processing. Details of the focus detection processing will be described later.

[0016] The display unit 35 displays images generated by the image processing unit 33, images that have been processed, and images read out by the recording unit 37. The display unit 35 also displays operation menu screens and setting screens for setting imaging conditions.

[0017] The operating components 36 consist of various operating components such as a release button and a menu button. The operating components 36 send operation signals corresponding to each operation to the control unit 34. The operating components 36 also include touch operation components provided on the display surface of the display unit 35.

[0018] The recording unit 37 records image data and other data on a recording medium, such as a memory card (not shown), in response to instructions from the control unit 34. The recording unit 37 also reads out image data recorded on the recording medium in response to instructions from the control unit 34.

[0019] <Explanation of stacked image sensors> As an example of the image sensor 32a described above, a stacked image sensor 100 will be explained. Figure 2 is a cross-sectional view of the image sensor 100. The image sensor 100 comprises an imaging chip 111, a signal processing chip 112, and a memory chip 113. The imaging chip 111 is stacked on the signal processing chip 112. The signal processing chip 112 is stacked on the memory chip 113. The imaging chip 111 and the signal processing chip 112, and the signal processing chip 112 and the memory chip 113 are electrically connected by connection parts 109. The connection parts 109 are, for example, bumps or electrodes. The imaging chip 111 captures a light image from a subject and generates image data. The imaging chip 111 outputs the image data from the imaging chip 111 to the signal processing chip 112. The signal processing chip 112 performs signal processing on the image data output from the imaging chip 111. The memory chip 113 has multiple memories and stores the image data. The image sensor 100 may be composed of an imaging chip and a signal processing chip. If the image sensor 100 is composed of an imaging chip and a signal processing chip, the storage unit for storing image data may be provided on the signal processing chip or separately from the image sensor 100.

[0020] As shown in Figure 2, incident light primarily enters in the positive Z-axis direction, indicated by the white arrow. Furthermore, as shown in the coordinate axes, the direction to the left of the paper perpendicular to the Z-axis is defined as the positive X-axis direction, and the direction towards the viewer, perpendicular to both the Z-axis and X-axis, is defined as the positive Y-axis direction. In the following figures, the coordinate axes are displayed using the coordinate axes in Figure 2 as a reference, so that the orientation of each figure is clear.

[0021] The imaging chip 111 is, for example, a CMOS image sensor. Specifically, the imaging chip 111 is a back-illuminated CMOS image sensor. The imaging chip 111 has a microlens layer 101, a color filter layer 102, a passivation layer 103, a semiconductor layer 106, and a wiring layer 108. The imaging chip 111 is arranged in the order of microlens layer 101, color filter layer 102, passivation layer 103, semiconductor layer 106, and wiring layer 108 in the Z-axis positive direction.

[0022] The microlens layer 101 has a plurality of microlenses L. The microlenses L focus the incident light onto the photoelectric conversion unit 104, which will be described later. The color filter layer 102 has a plurality of color filters F. The color filter layer 102 has multiple types of color filters F with different spectral characteristics. Specifically, the color filter layer 102 has a first filter (R) with spectral characteristics that mainly transmit red light, a second filter (Gb, Gr) with spectral characteristics that mainly transmit green light, and a third filter (B) with spectral characteristics that mainly transmit blue light. In the color filter layer 102, for example, the first filter, second filter and third filter are arranged in a Bayer array. The passivation layer 103 is composed of a nitride film or an oxide film and protects the semiconductor layer 106.

[0023] The semiconductor layer 106 has photoelectric conversion units 104 and readout circuits 105. The semiconductor layer 106 has a plurality of photoelectric conversion units 104 between a first surface 106a, which is the light incident surface, and a second surface 106b opposite to the first surface 106a. The semiconductor layer 106 has a plurality of photoelectric conversion units 104 arranged in the X-axis direction and the Y-axis direction. The photoelectric conversion unit 104 has a photoelectric conversion function that converts light into electric charge. The photoelectric conversion unit 104 also stores charge due to the photoelectric conversion signal. The photoelectric conversion unit 104 is, for example, a photodiode. The semiconductor layer 106 has readout circuits 105 on the second surface 106b side of the photoelectric conversion units 104. The semiconductor layer 106 has a plurality of readout circuits 105 arranged in the X-axis direction and the Y-axis direction. The readout circuit 105 is composed of multiple transistors and reads out image data generated by the charge photoelectrically converted by the photoelectric conversion unit 104 and outputs it to the wiring layer 108.

[0024] The wiring layer 108 has multiple metal layers. The metal layers are, for example, Al wiring, Cu wiring, etc. The wiring layer 108 outputs image data read by the read circuit 105. The image data is output from the wiring layer 108 to the signal processing chip 112 via the connection part 109.

[0025] The connection portion 109 may be provided for each photoelectric conversion unit 104. Alternatively, the connection portion 109 may be provided for each of multiple photoelectric conversion units 104. If the connection portion 109 is provided for each of multiple photoelectric conversion units 104, the pitch of the connection portion 109 may be greater than the pitch of the photoelectric conversion units 104. Furthermore, the connection portion 109 may be provided in the peripheral region of the area where the photoelectric conversion units 104 are located.

[0026] The signal processing chip 112 has multiple signal processing circuits. The signal processing circuits perform signal processing on image data output from the imaging chip 111. Examples of signal processing circuits include an amplifier circuit that amplifies the signal value of the image data, a correlated double sampling circuit that performs noise reduction processing on the image data, and an analog-to-digital (A / D) conversion circuit that converts an analog signal into a digital signal. A signal processing circuit may be provided for each photoelectric conversion unit 104.

[0027] Furthermore, a signal processing circuit may be provided for each of the multiple photoelectric conversion units 104. The signal processing chip 112 has multiple through electrodes 110. The through electrodes 110 are, for example, silicon through electrodes. The through electrodes 110 connect the circuits provided on the signal processing chip 112 to each other. The through electrodes 110 may also be provided in the peripheral area of ​​the imaging chip 111 and in the memory chip 113. Note that some of the elements constituting the signal processing circuit may be provided on the imaging chip 111. For example, in the case of an analog / digital conversion circuit, a comparator that compares the input voltage and the reference voltage may be provided on the imaging chip 111, and circuits such as a counter circuit and a latch circuit may be provided on the signal processing chip 112.

[0028] The memory chip 113 has multiple storage units. Each storage unit stores image data that has been signal-processed by the signal processing chip 112. The storage units are, for example, volatile memory such as DRAM. A storage unit may be provided for each photoelectric conversion unit 104. Alternatively, multiple storage units may be provided for each photoelectric conversion unit 104. The image data stored in the storage units is output to a subsequent image processing unit.

[0029] Figure 3 illustrates the pixel arrangement and unit region 131 of the imaging chip 111. In particular, it shows the imaging chip 111 as observed from the back (imaging surface) side. The pixel region contains, for example, more than 20 million pixels arranged in a matrix. In the example in Figure 3, four adjacent 2x2 pixels form one unit region 131. The grid lines in the figure illustrate the concept that adjacent pixels are grouped together to form a unit region 131. The number of pixels forming a unit region 131 is not limited to this; it could be around 1000, for example, 32x32 pixels, or more or less, or even just one pixel.

[0030] As shown in the magnified view of the pixel region, the unit region 131 in Figure 3 contains a so-called Bayer array consisting of four pixels: green pixels Gb and Gr, blue pixel B, and red pixel R. The green pixels Gb and Gr are pixels that have a green filter as the color filter F and receive light in the green wavelength band of the incident light. Similarly, the blue pixel B is a pixel that has a blue filter as the color filter F and receives light in the blue wavelength band, and the red pixel R is a pixel that has a red filter as the color filter F and receives light in the red wavelength band.

[0031] In this embodiment, multiple blocks are defined such that each block contains at least one unit region 131. That is, the smallest unit of a block is one unit region 131. As described above, the smallest number of pixels that can form one unit region 131 is 1 pixel. Therefore, when a block is defined on a pixel-by-pixel basis, the smallest number of pixels that can define a block is 1 pixel. Each block can control the pixels contained within it using different control parameters. In each block, all unit regions 131 within that block, i.e., all pixels within that block, are controlled under the same imaging conditions. In other words, photoelectric conversion signals can be obtained with different imaging conditions for a group of pixels contained in one block and a group of pixels contained in another block. Examples of control parameters include frame rate, gain, decimation rate, number of addition rows or columns for adding the photoelectric conversion signals, charge accumulation time or number of accumulations, and the number of bits (word length) for digitization. The image sensor 100 can freely perform decimation not only in the row direction (the X-axis direction of the imaging chip 111) but also in the column direction (the Y-axis direction of the imaging chip 111). Furthermore, the control parameters may also be parameters used in image processing.

[0032] Figure 4 is a diagram illustrating the circuit in a unit region 131. In the example in Figure 4, four adjacent 2x2 pixels form one unit region 131. As mentioned above, the number of pixels included in a unit region 131 is not limited to this; it may be 1000 pixels or more, or as few as 1 pixel. The two-dimensional positions of the unit region 131 are indicated by symbols A to D.

[0033] The reset transistors (RSTs) of the pixels included in the unit region 131 are configured to be individually switched on and off for each pixel. In Figure 4, a reset wire 300 is provided to switch the reset transistor of pixel A on and off, and a reset wire 310 is provided separately from the reset wire 300 to switch the reset transistor of pixel B on and off. Similarly, a reset wire 320 is provided separately from the reset wires 300 and 310 to switch the reset transistor of pixel C on and off. A dedicated reset wire 330 is also provided for other pixels D to switch their reset transistors on and off.

[0034] The transfer transistors (TX) of the pixels included in the unit region 131 are also configured to be individually switched on and off for each pixel. In Figure 4, separate transfer wirings 302 for switching the transfer transistor of pixel A, 312 for switching the transfer transistor of pixel B, and 322 for switching the transfer transistor of pixel C are provided. A dedicated transfer wiring 332 is also provided for switching the transfer transistors of other pixels D.

[0035] Furthermore, the selection transistors (SELs) of the pixels included in the unit region 131 are also configured to be individually switched on and off for each pixel. In Figure 4, separate selection wires 306 for switching the selection transistor of pixel A on and off, selection wires 316 for switching the selection transistor of pixel B on and off, and selection wires 326 for switching the selection transistor of pixel C on and off are provided. A dedicated selection wire 336 for switching the selection transistors of other pixels D is also provided.

[0036] The power supply wiring 304 is commonly connected to pixels A through D included in the unit region 131. Similarly, the output wiring 308 is commonly connected to pixels A through D included in the unit region 131. In addition, the power supply wiring 304 is commonly connected between multiple unit regions, but the output wiring 308 is provided individually for each unit region 131. The load current source 309 supplies current to the output wiring 308. The load current source 309 may be provided on the imaging chip 111 side or on the signal processing chip 112 side.

[0037] By individually switching the reset transistor and transfer transistor of the unit region 131 on and off, charge accumulation, including the charge accumulation start time, accumulation end time, and transfer timing, can be controlled for pixels A to D contained within the unit region 131. Furthermore, by individually switching the selection transistor of the unit region 131 on and off, the photoelectric conversion signals from each pixel A to D can be output via a common output wiring 308.

[0038] Here, a known rolling shutter method is used to control charge accumulation in a regular order for rows and columns for pixels A to D contained in the unit region 131. When pixels are selected row by row and then the column is specified using the rolling shutter method, the photoelectric conversion signals are output in the order "ABCD" in the example shown in Figure 4.

[0039] By configuring the circuit based on the unit region 131 in this way, the charge accumulation time can be controlled for each unit region 131. In other words, different photoelectric conversion signals with different frame rates can be output between the unit regions 131. Furthermore, by allowing charge accumulation (imaging) to be performed in some of the unit regions 131 in the imaging chip 111 while the unit regions 131 in other blocks are left idle, imaging can be performed only in predetermined blocks of the imaging chip 111, and the resulting photoelectric conversion signal can be output. Moreover, by switching the block in which charge accumulation (imaging) is performed between frames (the block targeted for accumulation control), sequential imaging can be performed in different blocks of the imaging chip 111, and the resulting photoelectric conversion signal can be output.

[0040] As described above, output wiring 308 is provided corresponding to each of the unit regions 131. Since the image sensor 100 has an imaging chip 111, a signal processing chip 112, and a memory chip 113 stacked on top of each other, by using electrical connections between the chips with connection parts 109 for these output wiring 308, the wiring can be routed without increasing the size of each chip in the planar direction.

[0041] <Block control of the image sensor> In this embodiment, imaging conditions can be set for each of the multiple blocks in the image sensor 32a. The control unit 34 (image control unit 34c) corresponds the multiple regions to the blocks and performs imaging according to the imaging conditions set for each region.

[0042] Figure 5 schematically shows the image of the subject formed on the image sensor 32a of camera 1. Before an imaging command is given, camera 1 converts the subject image into a live view image using photoelectric conversion. A live view image is a monitor image that is repeatedly captured at a predetermined frame rate (e.g., 60fps).

[0043] Before the setting unit 34b divides the area, the control unit 34 sets the same imaging conditions for the entire area of ​​the imaging chip 111 (i.e., the entire imaging screen). The same imaging conditions mean setting common imaging conditions for the entire imaging screen; for example, even if there is a variation of less than approximately 0.3 stops in the apex value, it is considered the same. The imaging conditions set uniformly for the entire area of ​​the imaging chip 111 are determined based on exposure conditions corresponding to the metered value of the subject's brightness, or exposure conditions manually set by the user.

[0044] In Figure 5, an image including a person 61a, a car 62a, a bag 63a, a mountain 64a, and clouds 65a and 66a is formed on the imaging surface of the imaging chip 111. Person 61a is holding the bag 63a in both hands. The car 62a is parked to the right and behind person 61a.

[0045] <Division of Regions> The control unit 34 divides the live view image screen into multiple regions based on the live view image as follows. First, the object detection unit 34a detects subject elements from the live view image. The detection of subject elements uses known subject recognition techniques. In the example in Figure 5, the object detection unit 34a detects a person 61a, a car 62a, a bag 63a, a mountain 64a, a cloud 65a, and a cloud 66a as subject elements.

[0046] Next, the setting unit 34b divides the live view image screen into regions containing the subject elements. In this embodiment, the region containing the person 61a is referred to as region 61, the region containing the car 62a as region 62, the region containing the bag 63a as region 63, the region containing the mountain 64a as region 64, the region containing the cloud 65a as region 65, and the region containing the cloud 66a as region 66.

[0047] <Setting imaging conditions for each block> When the control unit 34 divides the screen into multiple areas using the setting unit 34b, it displays a setting screen on the display unit 35, as illustrated in Figure 6. In Figure 6, the live view image 60a is displayed, and the imaging condition setting screen 70 is displayed to the right of the live view image 60a.

[0048] The settings screen 70 shows, from top to bottom, frame rate, shutter speed (TV), and gain (ISO) as examples of settings for imaging conditions. Frame rate is the number of frames of live view images or video recorded by camera 1 per second. Gain is the ISO sensitivity. In addition to the examples shown in Figure 6, additional settings for imaging conditions may be added as appropriate. If all settings do not fit on the settings screen 70, the settings can be displayed by scrolling up and down.

[0049] In this embodiment, the control unit 34 makes the area selected by the user from the areas divided by the setting unit 34b the target for setting (changing) the imaging conditions. For example, in a touch-operable camera 1, the user taps the display position of the main subject for which they want to set (change) the imaging conditions on the display surface of the display unit 35 on which the live view image 60a is displayed. When, for example, the display position of a person 61a is tapped, the control unit 34 makes the area 61 including the person 61a in the live view image 60a the target area for setting (changing) the imaging conditions, and also highlights the outline of the area 61.

[0050] In Figure 6, the area 61, which is displayed with its outline emphasized (thickened, brightened, changed color, dashed, blinking, etc.), indicates the area that is subject to setting (changing) the imaging conditions. In the example in Figure 6, it is assumed that a live view image 60a with the outline of area 61 emphasized is displayed. In this case, area 61 is the area subject to setting (changing) the imaging conditions. For example, in a touch-operable camera 1, when the user taps the shutter speed (TV) display 71, the control unit 34 displays the current shutter speed setting value for the emphasized area (area 61) on the screen (reference numeral 68). In the following explanation, the camera 1 will be described assuming touch operation, but the imaging conditions may also be set (changed) by operating buttons or other components of the operating member 36.

[0051] When the user taps the upper icon 71a or lower icon 71b for shutter speed (TV), the setting unit 34b increases or decreases the shutter speed display 68 from the current setting value according to the tap operation, and also sends an instruction to the imaging unit 32 (Figure 1) to change the imaging conditions of the unit area 131 (Figure 3) of the image sensor 32a corresponding to the highlighted area (area 61) according to the tap operation. The confirmation icon 72 is an operation icon for confirming the set imaging conditions. The setting unit 34b also sets (changes) the frame rate and gain (ISO) in the same way as it sets (changes) the shutter speed (TV).

[0052] Although the setting unit 34b has been described as setting imaging conditions based on user operation, it is not limited to this. The setting unit 34b may also set imaging conditions based on the judgment of the control unit 34, without user operation. For example, if overexposure or underexposure occurs in an area containing a subject element with maximum or minimum brightness in the image, the setting unit 34b may, based on the judgment of the control unit 34, set imaging conditions to eliminate the overexposure or underexposure. For areas that are not highlighted (areas other than area 61), the set imaging conditions are maintained.

[0053] Instead of highlighting the outline of the area to be set (changed) in imaging conditions, the control unit 34 may display the entire target area brightly, increase the contrast of the entire target area, or make the entire target area blink. Alternatively, the target area may be enclosed in a frame. The frame surrounding the target area may be a double frame or a single frame, and the line type, color, brightness, and other display characteristics of the frame may be changed as appropriate. The control unit 34 may also display an arrow or other indicator near the target area to point to the area to be set in imaging conditions. The control unit 34 may also display areas other than the target area to be set (changed) in darkness, or display areas other than the target area with low contrast.

[0054] As explained above, after the imaging conditions for each region are set, when the release button (not shown) or the display (release icon) that instructs the start of imaging, which constitutes the operating member 36, is operated, the control unit 34 controls the imaging unit 32 to perform imaging according to the imaging conditions set for each of the divided regions. The image processing unit 33 then performs image processing on the image data acquired by the imaging unit 32. As described above, the image processing can be performed with different image processing conditions for each region.

[0055] After the image processing by the image processing unit 33, the recording unit 37, instructed by the control unit 34, records the processed image data onto a recording medium, such as a memory card (not shown). This completes the series of imaging processes.

[0056] <Data Selection Process> As described above, in this embodiment, after the setting unit 34b divides the area of ​​the imaging screen, it is possible to set (change) imaging conditions for the area selected by the user or the area determined by the control unit 34. If different imaging conditions are set for the divided areas, the control unit 34 will perform the following data selection process as necessary.

[0057] 1. When performing image processing The image processing unit 33 (selection unit 33b) performs a data selection process as a preprocessing step for image data located near the boundaries of the regions when the image processing performed on the image data obtained by applying different imaging conditions between the divided regions is a predetermined image processing step. The predetermined image processing step is a process that calculates the image data of a point of interest to be processed in the image by referring to image data of multiple reference points around the point of interest, and includes, for example, pixel defect correction processing, color interpolation processing, edge enhancement processing, and noise reduction processing.

[0058] The data selection process is performed to mitigate any inconsistencies that may appear in the processed image due to differing imaging conditions between the divided regions. When the point of interest is located near the boundary of a divided region, the surrounding reference locations may contain a mixture of image data with the same imaging conditions as the image data of the point of interest and image data with different imaging conditions. In this embodiment, based on the idea that it is preferable to calculate the image data of the point of interest by referring to image data of reference locations with the same imaging conditions as the point of interest, rather than directly referring to image data of reference locations with different imaging conditions, the data used for image processing is selected as follows.

[0059] Figure 7(a) illustrates a region 80 near the boundary between region 61 and region 64 in the live view image 60a. In this example, a first imaging condition is set for region 61, which includes at least a person, and a second imaging condition is set for region 64, which includes a mountain. Figure 7(b) is an enlarged view of the region 80 near the boundary in Figure 7(a). Image data from pixels on the image sensor 32a corresponding to region 61, where the first imaging condition is set, is shown in white, and image data from pixels on the image sensor 32a corresponding to region 64, where the second imaging condition is set, is shown in shaded areas. In Figure 7(b), the image data from the pixel of interest P is located in the boundary area, which is the vicinity of the boundary 81 between region 61 and region 64, on region 61. The pixels surrounding the pixel of interest P (8 pixels in this example) that are included in a predetermined range 90 (e.g., 3x3 pixels) centered on the pixel of interest P are used as reference pixels. Figure 7(c) is an enlarged view of the pixel of interest P and the reference pixels. The position of the pixel of interest P is the point of interest, and the positions of the reference pixels surrounding the pixel of interest P are the reference positions.

[0060] The image processing unit 33 (generation unit 33c) can also perform image processing by directly referencing the image data of the reference pixels without performing data selection processing. However, if the imaging conditions applied to the pixel of interest P (referred to as the first imaging conditions) and the imaging conditions applied to the reference pixels surrounding the pixel of interest P (referred to as the second imaging conditions) are different, the selection unit 33b selects the image data of the first imaging conditions to be used for image processing from the image data of the reference pixels, as shown in (Example 1) to (Example 3) below. Then, the generation unit 33c performs image processing to calculate the image data of the pixel of interest P by referring to the image data of the reference pixels after image data selection. In Figure 7(c), the data output from the pixels shown in white is the image data of the first imaging conditions, and the data output from the pixels shown in diagonal lines is the image data of the second imaging conditions. In this embodiment, by not selecting the image data of the second imaging conditions, the image data output from the pixels shown in diagonal lines is not used for image processing.

[0061] (Example 1) The image processing unit 33 (selection unit 33b) selects the image data of the first imaging condition to be used for image processing from the image data of the reference pixels, for example, if the only difference between the first imaging condition and the second imaging condition is the ISO sensitivity, and the ISO sensitivity of the first imaging condition is 100 and the ISO sensitivity of the second imaging condition is 800. In other words, it does not use the image data of the second imaging condition that differs from the first imaging condition among the image data of the reference pixels for image processing.

[0062] (Example 2) The image processing unit 33 (selection unit 33b) selects the image data of the first imaging condition to be used for image processing from the image data of the reference pixels, for example, if the only difference between the first imaging condition and the second imaging condition is the shutter speed, and the shutter speed of the first imaging condition is 1 / 1000 second and the shutter speed of the second imaging condition is 1 / 100 second. In other words, it does not use the image data of the second imaging condition that differs from the first imaging condition among the image data of the reference pixels for image processing.

[0063] (Example 3) The image processing unit 33 (selection unit 33b) selects, for example, if the only difference between the first imaging condition and the second imaging condition is the frame rate (the charge accumulation time is the same), and the frame rate for the first imaging condition is 30fps and the frame rate for the second imaging condition is 60fps, then it selects the image data of the second imaging condition (60fps) from the image data of the reference pixels that has an acquisition timing close to that of the frame images acquired under the first imaging condition (30fps). In other words, it does not use the image data of the reference pixels that has an acquisition timing different from that of the frame images acquired under the first imaging condition (30fps).

[0064] On the other hand, the image processing unit 33 (selection unit 33b) selects all image data if the imaging conditions applied to the pixel of interest P are the same as the imaging conditions applied to all reference pixels surrounding the pixel of interest P. For example, if it is determined that the same imaging conditions as those applied to the pixel of interest P are applied to all reference pixels, it performs image processing to calculate the image data of the pixel of interest P by directly referencing the image data of all reference pixels. As mentioned above, even if there are slight differences in imaging conditions (for example, 0.3 stops or less in apex value), they may be considered to be the same imaging conditions.

[0065] <Example of image processing> This section provides an example of image processing involving data selection. (1) Pixel defect correction process In this embodiment, pixel defect correction processing is one of the image processing steps performed during imaging. Generally, a solid-state image sensor 100 may have pixel defects during the manufacturing process or after manufacturing, resulting in the output of abnormally high-quality image data. Therefore, the image processing unit 33 (generation unit 33c) corrects the image data output from the pixel with the pixel defect to make the image data at the pixel location where the pixel defect occurred less noticeable.

[0066] An example of pixel defect correction processing will be described. The image processing unit 33 (generation unit 33c) selects, for example, the pixel at the location of a pixel defect in a single frame of image, which is pre-recorded in a non-volatile memory (not shown), as the pixel of interest P (the pixel to be processed), and the pixels surrounding the pixel of interest P (8 pixels in this example) that are included in a predetermined range 90 (for example, 3 x 3 pixels) (Figure 7(c)) centered on the pixel of interest P, are used as reference pixels.

[0067] The image processing unit 33 (generation unit 33c) calculates the maximum and minimum values ​​of the image data at the reference pixel, and performs a Max, Min filter process to replace the image data output from the pixel of interest P with the above maximum or minimum value if the image data exceeds these maximum or minimum values. This process is performed for all pixel defects whose position information is recorded in non-volatile memory.

[0068] In the present embodiment, when the pixel to which the second imaging condition different from the first imaging condition applied to the target pixel P is included in the reference pixel, the image processing unit 33 (selection unit 33b) selects the image data to which the first imaging condition is applied from the image data in the reference pixel. Then, the image processing unit 33 (generation unit 33c) performs the above-described Max, Min filter processing by referring to the selected image data. Note that pixel defect correction processing may be performed by taking the average of the selected image data.

[0069] (2) Color interpolation processing In the present embodiment, the color interpolation processing is one of the image processes performed during imaging. As illustrated in FIG. 3, the imaging chip 111 of the image sensor 100 has green pixels Gb, Gr, blue pixels B, and red pixels R arranged in a Bayer array. Since the image data of color components different from the color components of the color filter F arranged at each pixel position is insufficient, the image processing unit 33 (generation unit 33c) performs color interpolation processing to generate the image data of the insufficient color components by referring to the image data at the surrounding pixel positions.

[0070] An example of the color interpolation processing will be described. FIG. 8(a) is a diagram illustrating the arrangement of the image data output from the image sensor 100. Corresponding to each pixel position, it has a color component of either R, G, or B according to the rule of the Bayer array. <G color interpolation> First, let's explain general G color interpolation. The image processing unit 33 (generation unit 33c) that performs G color interpolation uses the positions of the R color component and B color component as the focus position in order, and generates image data of the G color component at the focus position by referring to the image data of the four G color components at the reference positions surrounding the focus position. For example, when generating image data of the G color component at the focus position indicated by the thick frame in Figure 8(b) (2nd row, 2nd column counting from the top left position; similarly, focus positions will be represented counting from the top left position hereafter), the image data of the four G color components G1 to G4 located in the vicinity of the focus position (2nd row, 2nd column) are referred to. The image processing unit 33 (generation unit 33c) uses, for example, (aG1+bG2+cG3+dG4) / 4 as the image data of the G color component at the focus position (2nd row, 2nd column). a to d are weighting coefficients set according to the distance between the reference position and the focus position and the image structure.

[0071] Next, the G color interpolation of this embodiment will be described. In Figures 8(a) to 8(c), the first imaging condition is applied to the areas to the left and above the thick line, and the second imaging condition is applied to the areas to the right and below the thick line. Note that the first and second imaging conditions are different in Figures 8(a) to 8(c). Also, the G color component image data G1 to G4 in Figure 8(b) are reference positions for image processing the pixels at the point of interest (2nd row, 2nd column). In Figure 8(b), the first imaging condition is applied to the point of interest (2nd row, 2nd column). Of the reference positions, the first imaging condition is applied to image data G1 to G3. Also, of the reference positions, the second imaging condition is applied to image data G4. Therefore, the image processing unit 33 (selection unit 33b) selects image data G1 to G3 to which the first imaging condition is applied from the G color component image data G1 to G4. Thus, the image processing unit 33 (generation unit 33c) calculates the image data of the G color component at the point of interest (2nd row, 2nd column) by referring to the selected image data. The image processing unit 33 (generation unit 33c) uses, for example, (a1G1+b1G2+c1G3) / 3 as the image data of the G color component at the point of interest (2nd row, 2nd column). a1 to c1 are weighting coefficients that are set according to the distance between the reference point and the point of interest and the image structure.

[0072] The image processing unit 33 (generation unit 33c) generates image data of the G color component at the positions of the B color component and the R color component in FIG. 8(a), respectively, so that, as shown in FIG. 8(c), image data of the G color component can be obtained at each pixel position.

[0073] <R color interpolation> FIG. 9(a) is a diagram in which image data of the R color component is extracted from FIG. 8(a). The image processing unit 33 (generation unit 33c) calculates image data of the color difference component Cr shown in FIG. 9(b) based on the image data of the G color component shown in FIG. 8(c) and the image data of the R color component shown in FIG. 9(a).

[0074] First, interpolation of the general color difference component Cr will be described. When the image processing unit 33 (generation unit 33c) generates image data of the color difference component Cr at the attention position indicated by the thick frame (second row, second column) in FIG. 9(b), for example, it refers to the image data Cr1 to Cr4 of the four color difference components located near the attention position (second row, second column). The image processing unit 33 (generation unit 33c) uses, for example, (eCr1 + fCr2 + gCr3 + hCr4) / 4 as the image data of the color difference component Cr at the attention position (second row, second column). Here, e to h are weight coefficients provided according to the distance between the reference position and the attention position and the image structure.

[0075] Similarly, when the image processing unit 33 (generation unit 33c) generates image data of the color difference component Cr at the attention position indicated by the thick frame (second row, third column) in FIG. 9(c), for example, it refers to the image data Cr2, Cr4 to Cr6 of the four color difference components located near the attention position (second row, third column). The image processing unit 33 (generation unit 33c) uses, for example, (qCr2 + rCr4 + sCr5 + tCr6) / 4 as the image data of the color difference component Cr at the attention position (second row, third column). Here, q to t are weight coefficients provided according to the distance between the reference position and the attention position and the image structure. In this way, image data of the color difference component Cr is generated for each pixel position.

[0076] Next, the interpolation of the color difference component Cr in this embodiment will be explained. In Figures 9(a) to 9(c), for example, the first imaging condition is applied to the area to the left and above the thick line, and the second imaging condition is applied to the area to the right and below the thick line. Note that the first and second imaging conditions are different in Figures 9(a) to 9(c). In Figure 9(b), the position indicated by the thick frame (2nd row, 2nd column) is the position of interest for the color difference component Cr. Also, the image data Cr1 to Cr4 of the color difference component in Figure 9(b) are reference positions for image processing the pixels at the position of interest (2nd row, 2nd column). In Figure 9(b), the first imaging condition is applied to the position of interest (2nd row, 2nd column). Of the reference positions, the image data Cr1, Cr3, and Cr4 are to which the first imaging condition is applied. Also, of the reference positions, the image data Cr2 is to which the second imaging condition is applied. Therefore, the image processing unit 33 (selection unit 33b) selects image data Cr1, Cr3, and Cr4 to which the first imaging condition has been applied from the image data Cr1 to Cr4 of the color difference component Cr. Then, the image processing unit 33 (generation unit 33c) calculates the image data Cr of the color difference component at the point of interest (2nd row, 2nd column) by referring to the selected image data. The image processing unit 33 (generation unit 33c) uses, for example, (e1Cr1 + g1Cr3 + h1Cr4) / 3 as the image data of the color difference component Cr at the point of interest (2nd row, 2nd column). e1, g1, and h1 are weighting coefficients that are set according to the distance between the reference position and the point of interest and the image structure.

[0077] Also, in FIG. 9(c), the position indicated by the thick frame (second row, third column) is the attention position of the color difference component Cr. Also, the image data Cr2, Cr4, Cr5, Cr6 of the color difference components in FIG. 9(c) are reference positions for image processing of the pixels at the attention position (second row, third column). In FIG. 9(c), the second imaging condition is applied to the attention position (second row, third column). Among the reference positions, the first imaging condition is applied to the image data Cr4 and Cr5. Also, among the reference positions, the second imaging condition is applied to the image data Cr2 and Cr6. Therefore, the image processing unit 33 (selection unit 33b) selects the image data Cr2 and Cr6 to which the second imaging condition is applied from the image data Cr2, Cr4 to Cr6 of the color difference component Cr, similar to the example of FIG. 9(b) described above. After that, the image processing unit 33 (generation unit 33c) calculates the image data Cr of the color difference component at the attention position (second row, third column) by referring to the selected image data. The image processing unit 33 (generation unit 33c) uses, for example, (g2Cr2 + h2Cr6) / 2 as the image data of the color difference component Cr at the attention position (second row, third column). Note that g2 and h2 are weighting coefficients provided according to the distance between the reference position and the attention position and the image structure.

[0078] After obtaining the image data of the color difference component Cr at each pixel position, the image processing unit 33 (generation unit 33c) can obtain the image data of the R color component at each pixel position by adding the image data of the G color component shown in FIG. 8(c) corresponding to each pixel position.

[0079] FIG. 10(a) is a diagram in which the image data of the B color component is extracted from FIG. 8(a). The image processing unit 33 (generation unit 33c) calculates the image data of the color difference component Cb shown in FIG. 10(b) based on the image data of the G color component shown in FIG. 8(c) and the image data of the B color component shown in FIG. 10(a).

[0080] First, let's explain the interpolation of the general color difference component Cb. When the image processing unit 33 (generation unit 33c) generates image data of the color difference component Cb at the point of interest indicated by the thick frame (3rd row, 3rd column) in Figure 10(b), for example, it refers to the image data Cb1 to Cb4 of the four color difference components located in the vicinity of the point of interest (3rd row, 3rd column). The image processing unit 33 (generation unit 33c) uses, for example, (uCb1 + vCb2 + wCb3 + xCb4) / 4 as the image data of the color difference component Cb at the point of interest (3rd row, 3rd column). Note that u to x are weighting coefficients that are set according to the distance between the reference position and the point of interest and the image structure.

[0081] Similarly, when the image processing unit 33 (generation unit 33c) generates image data of the color difference component Cb at the point of interest indicated by the thick frame (3rd row, 4th column) in Figure 10(c), it refers to the image data of four color difference components Cb2, Cb4~Cb6 located in the vicinity of the point of interest (3rd row, 4th column). The image processing unit 33 (generation unit 33c) uses, for example, (yCb2+zCb4+αCb5+βCb6) / 4 as the image data of the color difference component Cb at the point of interest (3rd row, 4th column). Hereinafter, y, z, α, and β are weighting coefficients set according to the distance between the reference position and the point of interest and the image structure. In this way, image data of the color difference component Cb is generated for each pixel position.

[0082] Next, the interpolation of the color difference component Cb in this embodiment will be explained. In Figures 10(a) to 10(c), for example, the first imaging condition is applied to the area to the left and above the thick line, and the second imaging condition is applied to the area to the right and below the thick line. Note that the first and second imaging conditions are different in Figures 10(a) to 10(c). In Figure 10(b), the position indicated by the thick frame (3rd row, 3rd column) is the position of interest for the color difference component Cb. Also, the image data Cb1 to Cb4 of the color difference component in Figure 10(b) are reference positions for image processing the pixels at the position of interest (3rd row, 3rd column). In Figure 10(b), the second imaging condition is applied to the position of interest (3rd row, 3rd column). Of the reference positions, the first imaging condition is applied to the image data Cb1 and Cb3. Also, of the reference positions, the second imaging condition is applied to the image data Cb2 and Cb4. Therefore, the image processing unit 33 (selection unit 33b) selects image data Cb2 and Cb4 to which the second imaging condition has been applied from the image data Cb1 to Cb4 of the color difference component Cb. Then, the image processing unit 33 (generation unit 33c) calculates the image data Cb of the color difference component at the point of interest (3rd row, 3rd column) by referring to the selected image data. The image processing unit 33 (generation unit 33c) uses, for example, (v1Cb2+x1Cb4) / 2 as the image data of the color difference component Cb at the point of interest (3rd row, 3rd column). Note that v1 and x1 are weighting coefficients set according to the distance between the reference position and the point of interest and the image structure.

[0083] The same procedure applies when generating the color difference component image data Cb for the point of interest (3rd row, 4th column) indicated by the thick border in Figure 10(c). In Figure 10(c), the position indicated by the thick border (3rd row, 4th column) is the position of interest for the chrominance component Cb. Furthermore, the chrominance component image data Cb2, Cb4-Cb6 in Figure 10(c) are reference positions for image processing the pixel at the position of interest (3rd row, 4th column). In Figure 10(c), the second imaging condition is applied to the position of interest (3rd row, 4th column). Additionally, the second imaging condition is applied to the image data Cb2, Cb4-Cb6 of all reference positions. Therefore, the image processing unit 33 (generation unit 33c) calculates the image data for the chrominance component Cb at the position of interest (3rd row, 4th column) by referring to the image data Cb2, Cb4-Cb6 of the four chrominance components located near the position of interest (3rd row, 4th column).

[0084] The image processing unit 33 (generation unit 33c) obtains image data of the color difference component Cb at each pixel position, and then adds the image data of the G color component shown in Figure 8(c) corresponding to each pixel position to obtain image data of the B color component at each pixel position.

[0085] (3) Edge enhancement processing An example of edge enhancement processing is described below. The image processing unit 33 (generation unit 33c) performs a known linear filter operation using a predetermined size kernel centered on the pixel of interest P (the pixel to be processed) in one frame of the image. In the case of a sharpening filter, which is an example of a linear filter, if the kernel size is N × N pixels, the position of the pixel of interest P is the position of interest, and the area surrounding the pixel of interest P is (N 2 The position of the -1) reference pixels is the reference position. The kernel size may also be N×M pixels.

[0086] The image processing unit 33 (generation unit 33c) performs a filtering process to replace the image data of the pixel of interest P with the result of a linear filter operation, for example, from the upper horizontal line of the frame image to the lower horizontal line, while shifting the pixel of interest from left to right on each horizontal line.

[0087] In this embodiment, the image processing unit 33 (selection unit 33b) selects image data to which the first imaging condition is applied from the image data of the reference pixel if the reference pixel includes a pixel to which a second imaging condition different from the first imaging condition applied to the pixel of interest P is applied. Subsequently, the image processing unit 33 (generation unit 33c) performs the linear filtering process described above by referring to the selected image data.

[0088] (4) Noise reduction processing An example of noise reduction processing is described below. The image processing unit 33 (generation unit 33c) performs a known linear filter operation using a predetermined size kernel centered on the pixel of interest P (the pixel to be processed) in one frame of the image. In the case of a smoothing filter, which is an example of a linear filter, if the kernel size is N × N pixels, the position of the pixel of interest P is the position of interest, and the area surrounding the pixel of interest P is (N 2 The position of the -1) reference pixels is the reference position. The kernel size may also be N×M pixels.

[0089] The image processing unit 33 (generation unit 33c) performs a filtering process to replace the image data of the pixel of interest P with the result of a linear filter operation, for example, from the upper horizontal line of the frame image to the lower horizontal line, while shifting the pixel of interest from left to right on each horizontal line.

[0090] In this embodiment, the image processing unit 33 (selection unit 33b) selects image data to which the first imaging condition is applied from the image data of the reference pixel if the reference pixel includes a pixel to which a second imaging condition different from the first imaging condition applied to the pixel of interest P is applied. Subsequently, the image processing unit 33 (generation unit 33c) performs the linear filtering process described above by referring to the selected image data. As described above, the image processing unit 33 (selection unit 33b) selects image data to which the first imaging condition is applied from the image data of the reference pixel if the reference pixel includes a pixel to which a second imaging condition different from the first imaging condition applied to the pixel of interest P is applied. Then, the image processing unit 33 (generation unit 33c) performs image processing such as pixel defect correction, color interpolation, edge enhancement, and noise reduction by referring to the selected image data. The image processing unit 33 (selection unit 33b) performs similar processing on image data from other pixels output from the image sensor (including image data to which the second imaging condition is applied) to generate an image. The generated image is displayed on a display unit such as a display device.

[0091] 2. When performing focus detection processing The control unit 34 (AF calculation unit 34d) performs focus detection processing using image data corresponding to a predetermined position (focus detection position) on the imaging screen. If different imaging conditions are set between divided regions, and the focus detection position for AF operation is located at the boundary of the divided region, the control unit 34 (AF calculation unit 34d) performs data selection processing as a preprocessing step for focus detection processing on the image data for focus detection located near the boundary of the region.

[0092] The data selection process is performed to suppress a decrease in the accuracy of the focus detection process due to differences in imaging conditions between regions of the imaging screen divided by the setting unit 34b. For example, if the image data for focus detection of the focus detection position where the amount of image shift (phase difference) is detected in the image is located near the boundary of the divided region, the image data for focus detection may contain image data to which different imaging conditions have been applied. In this embodiment, based on the idea that it is preferable to detect the amount of image shift (phase difference) using image data to which the same imaging conditions have been applied rather than using image data to which different imaging conditions have been applied as is, the data selection process is performed as follows.

[0093] <Example of focus detection processing> An example of focus detection processing involving data selection processing is provided. In this embodiment, the AF operation focuses on a subject corresponding to a focus detection position selected by the user from among multiple focus detection positions. The control unit 34 (AF calculation unit 34d) calculates the amount of defocus of the imaging optical system 31 by detecting the amount of image shift (phase difference) of multiple subject images caused by light beams passing through different pupil regions of the imaging optical system 31. The control unit 34 (AF calculation unit 34d) moves the focus lens of the imaging optical system 31 to a position where the amount of defocus is zero (below an acceptable value), i.e., the in-focus position.

[0094] Figure 11 illustrates the position of focus detection pixels on the imaging surface of the image sensor 32a. In this embodiment, focus detection pixels are discretely arranged along the X-axis direction (horizontal direction) of the imaging chip 111. In the example of Figure 11, 15 focus detection pixel lines 160 are provided at predetermined intervals. The focus detection pixels constituting the focus detection pixel lines 160 output a photoelectric conversion signal for focus detection. In the imaging chip 111, normal imaging pixels are provided at pixel positions other than those of the focus detection pixel lines 160. The imaging pixels output a photoelectric conversion signal for live view images and recording.

[0095] Figure 12 is an enlarged view of a portion of the focus detection pixel line 160 corresponding to the focus detection position 80A shown in Figure 11. In Figure 12, red pixels R, green pixels G (Gb, Gr), and blue pixels B, as well as focus detection pixels S1 and S2, are shown as examples. The red pixels R, green pixels G (Gb, Gr), and blue pixels B are arranged according to the Bayer array rules described above.

[0096] The square-shaped regions exemplified for the red pixel R, green pixel G(Gb, Gr), and blue pixel B represent the light-receiving areas of the imaging pixels. Each imaging pixel receives the light beam passing through the exit pupil of the imaging optical system 31 (Figure 1). That is, the red pixel R, green pixel G(Gb, Gr), and blue pixel B each have square-shaped mask openings, and the light passing through these mask openings reaches the light-receiving area of ​​the imaging pixel.

[0097] Furthermore, the shape of the light-receiving area (mask aperture) of the red pixel R, green pixel G (Gb, Gr), and blue pixel B is not limited to a rectangle; for example, it may be circular.

[0098] The semicircular regions illustrated for focus detection pixels S1 and S2 indicate the light-receiving areas of the focus detection pixels. Specifically, focus detection pixel S1 has a semicircular mask opening to the left of its pixel position in Figure 12, and light passing through this mask opening reaches the light-receiving area of ​​focus detection pixel S1. On the other hand, focus detection pixel S2 has a semicircular mask opening to the right of its pixel position in Figure 12, and light passing through this mask opening reaches the light-receiving area of ​​focus detection pixel S2. In this way, focus detection pixels S1 and S2 each receive a pair of light beams passing through different regions of the exit pupil of the imaging optical system 31 (Figure 1).

[0099] The position of the focus detection pixel line 160 in the imaging chip 111 is not limited to the position exemplified in Figure 11. Furthermore, the number of focus detection pixel lines 160 is not limited to the example in Figure 11. In addition, the shape of the mask aperture in the focus detection pixels S1 and S2 is not limited to a semicircle; for example, the rectangular light-receiving area (mask aperture) in the imaging pixels R, G, and B may be divided horizontally to form a rectangle.

[0100] Furthermore, the focus detection pixel line 160 in the imaging chip 111 may be formed by arranging focus detection pixels along the Y-axis direction (vertical direction) of the imaging chip 111. Image sensors in which imaging pixels and focus detection pixels are arranged in a two-dimensional manner, as shown in Figure 12, are well known, and detailed illustrations and explanations of these pixels are omitted.

[0101] In the example shown in Figure 12, a configuration was described in which focus detection pixels S1 and S2 each receive one of a pair of light beams used for focus detection. Alternatively, the focus detection pixels may each receive both of the pair of light beams used for focus detection. By configuring the focus detection pixels to receive both of the pair of light beams used for focus detection, it becomes possible to use the photoelectric conversion signal obtained by the focus detection pixels as a photoelectric conversion signal for recording.

[0102] The control unit 34 (AF calculation unit 34d) detects the amount of image shift (phase difference) between a pair of images caused by a pair of light beams passing through different regions of the imaging optical system 31 (Figure 1), based on the photoelectric conversion signals (signal data) for focus detection output from the focus detection pixels S1 and S2. Then, it calculates the amount of defocus based on the amount of image shift (phase difference). Since this type of pupil-splitting phase-difference defocus calculation is well known in the field of cameras, a detailed explanation is omitted.

[0103] The focus detection position 80A (Figure 11) is assumed to be selected by the user at a position corresponding to region 80 near the boundary of region 61 in the live view image 60a illustrated in Figure 7(a). Figure 13 is an enlarged view of the focus detection position 80A. White pixels indicate that the first imaging condition is set, and shaded pixels indicate that the second imaging condition is set. In Figure 13, the position enclosed by frame 170 corresponds to the focus detection pixel line 160 (Figure 11).

[0104] The control unit 34 (AF calculation unit 34d) normally performs focus detection processing using the focus detection signal data of the focus detection pixels indicated by the frame 170 without performing data selection processing. However, if the focus detection signal data enclosed by the frame 170 contains a mixture of focus detection signal data to which the first imaging condition is applied and focus detection signal data to which the second imaging condition is applied, the control unit 34 (AF calculation unit 34d) selects the focus detection signal data for the first imaging condition to be used for focus detection processing from the focus detection signal data enclosed by the frame 170, as shown in (Example 1) to (Example 3) below. Then, the control unit 34 (AF calculation unit 34d) performs focus detection processing using the focus detection signal data after data selection processing. In Figure 13, the data output from pixels indicated by a white background is the focus detection signal data for the first imaging condition, and the data output from pixels indicated by diagonal lines is the focus detection signal data for the second imaging condition. In this embodiment, by not selecting the signal data for focus detection under the second imaging condition, the signal data for focus detection output from the pixels indicated by the diagonal lines is not used in the focus detection process.

[0105] (Example 1) The control unit 34 (AF calculation unit 34d) selects the focus detection signal data for the first imaging condition to be used for focus detection processing from the image data enclosed by frame 170, for example, if the only difference between the first imaging condition and the second imaging condition is the ISO sensitivity, and the ISO sensitivity of the first imaging condition is 100 and the ISO sensitivity of the second imaging condition is 800. In other words, the focus detection signal data for the second imaging condition that differs from the first imaging condition among the focus detection signal data enclosed by frame 170 is not used for focus detection processing.

[0106] (Example 2) The control unit 34 (AF calculation unit 34d) selects the focus detection signal data for the first imaging condition to be used for focus detection processing from the focus detection signal data enclosed by frame 170, for example, if the only difference between the first imaging condition and the second imaging condition is the shutter speed, and the shutter speed for the first imaging condition is 1 / 1000 second and the shutter speed for the second imaging condition is 1 / 100 second. In other words, it does not use the focus detection signal data for the second imaging condition that differs from the first imaging condition among the focus detection signal data enclosed by frame 170 for focus detection processing.

[0107] (Example 3) The control unit 34 (AF calculation unit 34d) selects the focus detection signal data for the first imaging condition to be used for focus detection processing from the focus detection signal data enclosed in frame 170, for example, if the only difference between the first imaging condition and the second imaging condition is the frame rate (the charge accumulation time is the same), and the frame rate for the first imaging condition is 30fps and the frame rate for the second imaging condition is 60fps. In other words, among the focus detection signal data enclosed in frame 170, the focus detection signal data for the second imaging condition that was acquired at a different timing than the image data for the first imaging condition is not used for focus detection processing.

[0108] On the other hand, the control unit 34 (AF calculation unit 34d) does not perform the above data selection process if the imaging conditions applied to the signal data for focus detection enclosed by the frame 170 are the same. In other words, the control unit 34 (AF calculation unit 34d) performs the focus detection process using the signal data for focus detection of the focus detection pixels indicated by the frame 170 as is.

[0109] As mentioned above, even if there are slight differences in imaging conditions, they will be considered the same. Furthermore, while the above example described selecting the focus detection signal data for the first imaging condition from the focus detection signal data enclosed in frame 170, it is also possible to select the focus detection signal data for the second imaging condition from the focus detection signal data enclosed in frame 170. In the example above, we described an example of selecting a photoelectric conversion signal for focus detection of the first imaging condition, as this is an example of performing focus detection by specifying an area where the first imaging condition is set. We will now describe an example where the subject to be focused on is located across the area where the first imaging condition is set and the area where the second imaging condition is set. When the subject to be focused on is located across the area where the first imaging condition is set and the area where the second imaging condition is set, the control unit 34 (AF calculation unit 34d) selects the photoelectric conversion signal for focus detection of the first imaging condition to be used for focus detection processing from the photoelectric conversion signals for focus detection enclosed by the frame 170. Then, the control unit 34 (AF calculation unit 34d) calculates the first defocus amount from the selected photoelectric conversion signal for focus detection. Furthermore, the control unit 34 (AF calculation unit 34d) selects the photoelectric conversion signal for focus detection of the second imaging condition to be used for focus detection processing from the photoelectric conversion signals for focus detection enclosed by the frame 170. The control unit 34 (AF calculation unit 34d) then calculates a second defocus amount from the selected photoelectric conversion signal for focus detection. The control unit 34 (AF calculation unit 34d) then performs focus detection processing using the first defocus amount and the second defocus amount. Specifically, for example, the control unit 34 (AF calculation unit 34d) calculates the average value of the first defocus amount and the second defocus amount to calculate the lens movement distance. Alternatively, the control unit 34 (AF calculation unit 34d) may select the value of the first defocus amount and the second defocus amount that results in a smaller lens movement distance. Alternatively, the control unit 34 (AF calculation unit 34d) may select a value from the first defocus amount and the second defocus amount that indicates the subject is closer. Furthermore, if the subject to be focused on is located across the area where the first imaging condition is set and the area where the second imaging condition is set, the control unit 34 (AF calculation unit 34d) may select the area with the larger subject area and select the photoelectric conversion signal for focus detection. For example, if the area of ​​the face of the subject to be focused on is 70% in the area where the first imaging condition is set and 30% in the second area, the control unit 34 (AF calculation unit 34d) will select the photoelectric conversion signal for focus detection for the first imaging condition. Note that the area ratios (percentages) mentioned above are examples and are not limited to them.

[0110] The above explanation illustrates focus detection processing using a pupil-splitting phase-difference method, but the same method can also be used for contrast detection methods, which move the focus lens of the imaging optical system 31 to the focus position based on the contrast of the subject image.

[0111] When using the contrast detection method, the control unit 34 moves the focus lens of the imaging optical system 31 and, at each position of the focus lens, performs a known focus evaluation value calculation based on the image data output from the imaging pixel of the image sensor 32a corresponding to the focus detection position. The control unit then determines the position of the focus lens that maximizes the focus evaluation value as the in-focus position.

[0112] Normally, the control unit 34 calculates the focus evaluation value using the image data output from the imaging pixels corresponding to the focus detection position without performing data selection processing. However, if the image data corresponding to the focus detection position contains a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied, the control unit 34 selects either the image data to which the first imaging condition is applied or the image data to which the second imaging condition is applied from among the image data corresponding to the focus detection position. Then, the control unit 34 calculates the focus evaluation value using the image data after data selection processing. Furthermore, if the subject to be focused on is located across the region to which the first imaging condition is set and the region to which the second imaging condition is set, the control unit 34 (AF calculation unit 34d) may select the region with the larger area of ​​the subject region and select the signal data for focus detection.

[0113] 3. When performing subject detection processing Figure 14(a) is an example of a template image 180 representing an object to be detected, and Figure 14(b) is an example of a live view image 60(a) and a search range 190. The control unit 34 (object detection unit 34a) detects an object (for example, a bag 63a, which is one of the subject elements in Figure 5) from the live view image. The control unit 34 (object detection unit 34a) may set the range for detecting the object to the entire range of the live view image 60a, but to reduce the detection processing load, it may also set a part of the live view image 60a as the search range 190.

[0114] The control unit 34 (object detection unit 34a) performs data selection processing as a preprocessing step for subject detection processing on image data located near the boundary of a divided region when different imaging conditions are set between the divided regions and the search range 190 includes the boundary of the divided region.

[0115] The data selection process is performed to suppress a decrease in the accuracy of the subject element detection process due to differences in imaging conditions between the regions of the imaging screen divided by the setting unit 34b. Generally, when the search range 190 used for detecting subject elements includes the boundaries of the divided regions, the image data in the search range 190 may contain image data to which different imaging conditions have been applied. In this embodiment, based on the idea that it is preferable to perform subject element detection using image data to which the same imaging conditions have been applied rather than using image data to which different imaging conditions have been applied as is, the data selection process is performed as follows.

[0116] The following describes the case of detecting a bag 63a, which is an item belonging to person 61a, in the live view image 60a illustrated in Figure 5. The control unit 34 (object detection unit 34a) sets the search range 190 in the vicinity of the area containing person 61a. Alternatively, the area 61 containing person 61a may be set as the search range.

[0117] The control unit 34 (object detection unit 34a) performs subject detection processing using the image data constituting the search range 190 without performing data selection processing if the search range 190 is not divided by two regions with different imaging conditions. However, if the image data in the search range 190 contains a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied, the control unit 34 (object detection unit 34a) selects the image data to be used for subject detection processing from the image data in the search range 190, similar to (Examples 1) to (Examples 3) when performing the focus detection processing described above. Then, the control unit 34 (object detection unit 34a) performs subject detection processing using the image data after data selection processing for the region where the first imaging condition is set. Here, subject detection processing is, for example, a process that calculates the similarity between the template image 180 and the selected image data to be used for subject detection processing (so-called template matching). Furthermore, the control unit 34 (object detection unit 34a) selects the image data to be used for subject detection processing from the image data in the search range 190. Then, the control unit 34 (object detection unit 34a) performs the same subject detection process as described above using the image data after data selection processing for the area where the second imaging condition is set. In this way, the control unit 34 (object detection unit 34a) performs subject detection within the search range 190. By aligning the boundary between the subject area detected using the image data of the first imaging condition and the subject area detected using the image data of the second imaging condition, the detection of a subject within the search range 190 can be achieved. In the above example, the case in which the control unit 34 (object detection unit 34a) uses both the image data of the first and second imaging conditions has been described, but subject detection may be performed using only one of the image data. For example, if the search range 190 contains both an area where the first imaging condition is set and an area where the second imaging condition is set, but the area where the first imaging condition is set occupies a large portion, subject detection may be performed using only the image data of the first imaging condition. Here, "the area where the first imaging condition is set occupies a large portion" means, for example, that the area where the first imaging condition is set accounts for 70% or more of the total area.Furthermore, the area of ​​the region where the first imaging condition is set may be 80% or more, or even 90% or more, rather than being limited to 70% or more. Naturally, the percentage of the area of ​​the region where the first imaging condition is set is not limited to these figures and can be changed as appropriate.

[0118] The data selection process for image data within the search range 190 described above may also be applied to search ranges used to detect specific subjects such as human faces, or to regions used to determine the imaging scene. For example, when the control unit 34 (object detection unit 34a) detects a human face within the search range 190, it selects image data for a first imaging condition used for subject detection processing from the image data within the search range 190. Then, the control unit 34 (object detection unit 34a) performs known face detection processing on the region where the first imaging condition is set. Furthermore, the control unit 34 (object detection unit 34a) selects image data for a second imaging condition used for subject detection processing from the image data within the search range 190. Then, the control unit 34 (object detection unit 34a) performs known face detection processing on the region where the second imaging condition is set. Finally, the control unit 34 (object detection unit 34a) performs face detection from the image data within the search range 190 by aligning the boundary between the face region detected within the region where the first imaging condition is set and the face region detected within the region where the second imaging condition is set.

[0119] Furthermore, the data selection process for image data within the search range of 190 described above may be applied not only to the search range used in pattern matching methods using template images, but also to the search range when detecting features based on image color, edges, etc.

[0120] Furthermore, by applying a known template matching process using image data from multiple frames acquired at different acquisition times, the tracking process for a moving object may be applied to search for areas similar to the target object in the frame image acquired earlier, using the frame image acquired later. In this case, if the search range set in the frame image acquired later contains a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied, the control unit 34 selects the image data to which the first imaging condition is applied for use in the tracking process from the image data in the search range. Then, for the area to which the first imaging condition is set, the control unit 34 performs the tracking process using the image data after the data selection process. After that, similarly to the above, the control unit 34 may select the image data to which the second imaging condition is applied for use in the tracking process from the image data in the search range, and perform the tracking process using the image data after the data selection process for the area to which the second imaging condition is set.

[0121] Furthermore, the same applies when detecting known motion vectors using image data from multiple frames acquired at different times. If the detection region used for motion vector detection contains a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied, the control unit 34 selects the image data to which the first imaging condition is applied for use in the detection process from the image data of the detection region used for motion vector detection. Then, the control unit 34 detects motion vectors in the region to which the first imaging condition is set using the image data after the data selection process. Subsequently, similarly to the above, the control unit 34 may select the image data to which the second imaging condition is used for motion vector detection processing from the image data of the search range and perform motion vector detection processing in the region to which the second imaging condition is set using the image data after the data selection process. The control unit 34 may determine the motion vector of the entire image from the motion vector detected from the region to which the first imaging condition is set and the motion vector detected from the region to which the second imaging condition is set, or it may determine the motion vector for each region.

[0122] 4. When setting imaging conditions When the control unit 34 (setting unit 34b) divides the area of ​​the imaging screen and sets different imaging conditions between the divided areas, and then remeasures the image to determine the exposure conditions, it performs a data selection process on the image data located near the boundary of the area as a preprocessing step before setting the exposure conditions.

[0123] The data selection process is performed to suppress a decrease in the accuracy of the exposure condition determination process due to differences in imaging conditions between regions of the imaging screen divided by the setting unit 34b. For example, if the metering range set in the center of the imaging screen includes the boundaries of the divided regions, the image data within the metering range may contain image data to which different imaging conditions have been applied. In this embodiment, based on the idea that it is preferable to perform exposure calculations using image data to which the same imaging conditions have been applied rather than using image data to which different imaging conditions have been applied, the data selection process is performed as follows.

[0124] The control unit 34 (setting unit 34b) performs exposure calculation processing using the image data constituting the photometric range without performing data selection processing if the photometric range is not divided by multiple regions with different imaging conditions. However, if the image data of the photometric range contains a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied, the control unit 34 (setting unit 34b) selects the image data of the first imaging condition to be used for exposure calculation processing from the image data of the photometric range, similar to (Examples 1) to (Examples 3) when performing focus detection processing and subject detection processing described above. Then, the control unit 34 (setting unit 34b) performs exposure calculation processing for the region to which the first imaging condition is set using the image data after data selection processing. Then, the control unit 34 (setting unit 34b) selects the image data of the second imaging condition to be used for exposure calculation processing from the image data of the photometric range. Then, the control unit 34 (setting unit 34b) performs exposure calculation processing for the region to which the second imaging condition is set using the image data after data selection processing. Thus, when there are multiple regions with different imaging conditions within the photometric range, the control unit 34 (setting unit 34b) performs data selection processing for photometric measurement of each region and performs exposure calculation processing using the image data from the data selection processing. Furthermore, if the photometering range spans both the area where the first imaging condition is set and the area where the second imaging condition is set, the control unit 34 (setting unit 34b) may select the area with the larger area, similar to the case of focus detection and subject detection described above.

[0125] This applies not only to the metering range used when performing the exposure calculation process described above, but also to the metering (color measurement) range used when determining the white balance adjustment value, the metering range used when determining whether or not to emit auxiliary light from a light source that emits auxiliary light, and the metering range used when determining the amount of auxiliary light emitted from the above light source.

[0126] Furthermore, when the readout resolution of the photoelectric conversion signal differs between regions of the divided imaging screen, the same treatment can be applied to the region used for determining the imaging scene when determining the readout resolution for each region.

[0127] <Explanation of the flowchart> Figure 15 is a flowchart illustrating the process of setting imaging conditions for each region and taking an image. When the main switch of camera 1 is turned on, the control unit 34 starts a program that executes the process shown in Figure 15. In step S10, the control unit 34 starts a live view display on the display unit 35 and proceeds to step S20.

[0128] Specifically, the control unit 34 applies predetermined image processing to the image data output sequentially from the imaging unit 32 and displays the resulting image as a live view image on the sequential display unit 35. As described above, at this point, the same imaging conditions are set for the entire area of ​​the imaging chip 111, i.e., the entire screen. Furthermore, if the setting is configured to perform AF operation during live view display, the control unit 34 (AF calculation unit 34d) controls the AF operation to focus on the subject element corresponding to a predetermined focus detection position by performing focus detection processing. The AF calculation unit 34d performs the above data selection processing before performing focus detection processing as necessary. Furthermore, if the setting to perform AF operation during live view display is not configured, the control unit 34 (AF calculation unit 34d) will perform AF operation when AF operation is instructed later.

[0129] In step S20, the control unit 34 (object detection unit 34a) detects subject elements from the live view image and proceeds to step S30. The object detection unit 34a performs the data selection process described above before performing the subject detection process, if necessary. In step S30, the control unit 34 (setting unit 34b) divides the live view image screen into regions containing subject elements and proceeds to step S40.

[0130] In step S40, the control unit 34 displays the regions on the display unit 35. As illustrated in Figure 6, the control unit 34 highlights the region among the divided regions that is the target of setting (changing) the imaging conditions. The control unit 34 then displays the imaging condition setting screen 70 on the display unit 35 and proceeds to step S50. Furthermore, if the user taps the display position of another main subject on the display screen with their finger, the control unit 34 changes the area containing that main subject to be changed and highlights it as an area subject to setting (changing) the imaging conditions.

[0131] In step S50, the control unit 34 determines whether or not AF operation is necessary. For example, if the focus adjustment state changes due to the movement of the subject, if the position of the focus detection position is changed by user operation, or if the execution of AF operation is instructed by user operation, the control unit 34 determines step S50 to be positive and proceeds to step S70. If the focus adjustment state does not change, the position of the focus detection position is not changed by user operation, and the execution of AF operation is not instructed by user operation, the control unit 34 determines step S50 to be negative and proceeds to step 60.

[0132] In step S70, the control unit 34 performs the AF operation and returns to step S40. The AF calculation unit 34d performs the data selection process as necessary and then performs the focus detection process, which is the AF operation. After returning to step S40, the control unit 34 repeats the same process as described above based on the live view image acquired after the AF operation.

[0133] In step S60, the control unit 34 (setting unit 34b) sets the imaging conditions for the highlighted area in response to user operation and proceeds to step S80. The display transitions of the display unit 35 and the setting of imaging conditions in response to user operation in step S60 are as described above. The control unit 34 (setting unit 34b) performs the above data selection process as necessary and then performs the exposure calculation process.

[0134] In step S80, the control unit 34 determines whether or not an imaging instruction has been given. If the release button (not shown) or the display icon indicating imaging, which constitutes the operating member 36, is operated, the control unit 34 affirms step S80 and proceeds to step S90. If no imaging instruction has been given, the control unit 34 negates step S80 and returns to step S60.

[0135] In step S90, the control unit 34 performs a predetermined imaging process. That is, the imaging control unit 34c controls the image sensor 32a to take images according to the imaging conditions set for each region, and then proceeds to step S100.

[0136] In step S100, the control unit 34 (imaging control unit 34c) sends an instruction to the image processing unit 33 to perform predetermined image processing on the image data obtained by the imaging, and then proceeds to step S110. The image processing includes the pixel defect correction process, color interpolation process, edge enhancement process, and noise reduction process. Furthermore, the image processing unit 33 (selection unit 33b) performs data selection processing on image data located near the boundary of the region, if necessary, before performing image processing.

[0137] In step S110, the control unit 34 sends an instruction to the recording unit 37 to record the image data after image processing on a recording medium (not shown) and proceed to step S120.

[0138] In step S120, the control unit 34 determines whether or not a termination operation has been performed. If a termination operation has been performed, the control unit 34 affirms step S120 and terminates the process shown in Figure 15. If a termination operation has not been performed, the control unit 34 negates step S120 and returns to step S20. If the process returns to step S20, the control unit 34 repeats the process described above.

[0139] In the above explanation, a stacked image sensor 100 was used as an example for the image sensor 32a, but it is not necessary to configure it as a stacked image sensor as long as imaging conditions can be set for each of the multiple blocks in the image sensor (imaging chip 111).

[0140] According to the above-described embodiment, the following effects and advantages can be obtained. (1) The camera 1 equipped with an image processing device includes an image processing unit 33 (input unit 33a) that receives first image data generated by capturing an image of a subject incident on a first region of the imaging unit 32 under first imaging conditions, and second image data generated by capturing an image of a subject incident on a second region of the imaging unit 32 under second imaging conditions different from the first imaging conditions, a selection unit 33b that selects one of the first image data and second image data input to the input unit 33a, and an image processing unit (generation unit 33) that displays the image generated from the selected image data on the display unit 35. This allows for appropriate processing in regions with different imaging conditions. In other words, it is possible to appropriately generate images from image data generated in each region. For example, it is possible to suppress the unnaturalness that may appear in the generated image due to differences in imaging conditions for each region.

[0141] (2) The generation unit 33c of the camera 1 generates an image at the position of each pixel in an image including the first image data and the second image data, using image data of predetermined reference pixels surrounding each pixel, and the selection unit 33b selects one of the first image data and the second image data as the image data of the reference pixel at the position of each pixel. This allows for appropriate processing in regions with different imaging conditions.

[0142] (3) The selection unit 33b of the camera 1 selects from the first image data and the second image data the image data generated by capturing images under the same imaging conditions as each pixel. This allows for appropriate processing in areas with different imaging conditions.

[0143] (4) The selection unit 33b of the camera 1 further selects image data generated by imaging other pixels other than the reference pixel using the same imaging conditions as each of the aforementioned pixels. This allows for appropriate processing in areas with different imaging conditions.

[0144] (5) The selection unit 33b of camera 1 selects all image data if all of the image data is first image data, selects all image data if all of the image data is second image data, and selects either the first image data or the second image data if the image data includes both first and second image data. This allows for appropriate processing in areas with different imaging conditions.

[0145] (6) Since the first and second imaging conditions of camera 1 include at least the storage time or ISO sensitivity, appropriate processing can be performed in different regions for charge storage time and imaging sensitivity.

[0146] (7) The generation unit 33c of the camera 1 generates third image data as at least pixel defect correction processing, color interpolation processing, edge enhancement processing, or noise reduction processing. This makes it possible to generate image data in which pixel defect correction processing, color interpolation processing, edge enhancement processing, or noise reduction processing is appropriately applied in regions with different imaging conditions.

[0147] (Second Embodiment) In the first embodiment, when performing image processing, the image data selection process is such that if the imaging conditions applied to the pixel of interest P (referred to as the first imaging conditions) and the imaging conditions applied to the reference pixels surrounding the pixel of interest P (referred to as the second imaging conditions) are different, the image processing unit 33 (selection unit 33b) selects image data from the image data of pixels located inside a predetermined range 90 to which the first imaging conditions common to the pixel of interest are applied, and the image processing unit 33 (generation unit 33c) then refers to the selected image data.

[0148] In the second embodiment, the image processing unit 33 (selection unit 33b) selects image data to which the first imaging conditions common to the pixel of interest are applied, even from image data of pixels located outside the predetermined range 90, thereby increasing the number of data referenced by the image processing unit 33 (generation unit 33c). In other words, the image processing unit 33 (selection unit 33b) selects image data to which the first imaging conditions are applied by changing the selection position.

[0149] Figure 7(d) is an enlarged view of the pixel of interest P and the reference pixel in the second embodiment. In Figure 7(d), within a predetermined range 90 centered on the pixel of interest P, the data output from pixels shown in white is image data for the first imaging condition, and the data output from pixels shown in diagonal lines is image data for the second imaging condition. Similar to the first embodiment, even if the image data of a pixel is located inside the predetermined range 90, the image processing unit 33 (selection unit 33b) does not select the image data of pixels to which the second imaging condition is applied (diagonal lines).

[0150] In the second embodiment, the image processing unit 33 (selection unit 33b) further selects image data of white pixels located outside the predetermined range 90 to which the first imaging condition is applied, along with image data of white pixels located inside the predetermined range 90 to which the first imaging condition is applied, as illustrated in Figure 7(d). The image processing unit 33 (generation unit 33c) then performs image processing by referring to the image data thus selected. In the above example, the distance from the position of the pixel of interest P to the position of the image data of the second imaging condition located inside the predetermined range 90 is longer than the distance from the position of the pixel of interest P to the position of the image data of white pixels located outside the predetermined range 90 to which the first imaging condition is applied. Thus, the image processing unit 33 (generation unit 33c) can also choose not to select image data of the second imaging condition which is shorter from the position of the pixel of interest P, and instead select image data of the first imaging condition which is longer from the position of the pixel of interest P.

[0151] Furthermore, the image processing unit 33 (selection unit 33b) prioritizes selecting image data of pixels closer to the predetermined range 90 over image data of pixels further away from the predetermined range 90. This is based on the idea that pixels closer to the predetermined range 90 are more likely to have common image information with the pixel of interest P than pixels further away from the predetermined range 90.

[0152] Further, for example, when the length of one side of the predetermined range 90 is L, the image processing unit 33 (selection unit 33b) selects image data of white pixels (i.e., the first imaging condition is applied) whose distance from the pixels indicated by the above diagonal lines is L or less. The reason is based on the idea that if it is too far from the predetermined range 90, the possibility of having the same image information as the target pixel P becomes low, so it is preferable not to include it in the reference data.

[0153] <Example Illustration of Image Processing> An example of the image processing in the second embodiment will be illustrated. (1) Pixel Defect Correction Processing When the same imaging condition is applied to all the pixels in the predetermined range 90 centered on the target pixel P, the image processing unit 33 (selection unit 33b) selects all the image data of the pixels located inside the predetermined range 90. Then, referring to the selected image data, the image processing unit 33 (generation unit 33c) performs Max, Min filter processing. Note that pixel defect correction processing may be performed by taking the average of the selected image data.

[0154] As shown in FIG. 7(d), when a pixel to which a second imaging condition different from the first imaging condition applied to the target pixel P at the time of imaging is included in the predetermined range 90 centered on the target pixel P, the image processing unit 33 (selection unit 33b) selects the image data of the pixels located inside the predetermined range 90 to which the first imaging condition is applied. Further, the image data of the white pixels located outside the predetermined range 90 to which the first imaging condition is applied is selected. The image processing unit 33 (generation unit 33c) performs the above-described Max, Min filter processing referring to the image data thus selected. Note that pixel defect correction processing may be performed by taking the average of the selected image data.

[0155] The image processing unit 33 performs such processing for all pixel defects whose position information is recorded in the non-volatile memory.

[0156] (2) Color Interpolation Processing <G Color Interpolation> The G color interpolation of the second embodiment will be described. Similar to the case of the first embodiment, in FIGS. 8(a) to 8(c), for example, the first imaging conditions are applied to the regions to the left and above the thick line, and the second imaging conditions are applied to the regions to the right and below the thick line.

[0157] In the example of FIG. 8(b), different second imaging conditions from the first imaging conditions applied to the attention position (the second row and the second column) are applied to the reference position corresponding to the image data G4 of the G color component indicated by the slanted lines. The image processing unit 33 (selection unit 33b) selects the image data G1 to G3 to which the first imaging conditions are applied from the image data G1 to G4 of the G color component. Further, the image processing unit 33 (selection unit 33b) selects the image data G6 of the G color component located in the vicinity of the reference position corresponding to the data G4 and to which the first imaging conditions are applied. That is, the image processing unit 33 (selection unit 33b) changes the position for selecting the image data with respect to the first embodiment and selects the image data to which the first imaging conditions are applied. If the second imaging conditions are also applied at the position of the data G6, data to which the first imaging conditions are applied may be selected from the image data at the positions in the vicinity of the data G6.

[0158] The image processing unit 33 (generation unit 33c) calculates the image data of the G color component at the attention position (the second row and the second column) by referring to the image data selected in this way. The image processing unit 33 (generation unit 33c) uses, for example, (a2G1 + b2G2 + c2G3 + d2G6) / 4 as the image data of the G color component at the attention position (the second row and the second column). Here, a2, b2, c2, and d2 are weighting coefficients provided according to the distance between the reference position and the attention position and the image structure.

[0159] The image processing unit 33 (generation unit 33c) generates the image data of the G color component at the positions of the B color component and the R color component in FIG. 8(a), respectively, to obtain the image data of the G color component at each pixel position as shown in FIG. 8(c).

[0160] <R color interpolation> The R color interpolation of the second embodiment will now be described. Similar to the first embodiment, in Figures 9(a) to 9(c), for example, the first imaging condition is applied to the areas to the left and above the thick line, and the second imaging condition is applied to the areas to the right and below the thick line.

[0161] In the example shown in Figure 9(b), the image processing unit 33 (selection unit 33b) applies a second imaging condition, different from the first imaging condition applied to the point of interest indicated by the thick frame (2nd row, 2nd column), to the reference position corresponding to the image data Cr2 of the color difference component Cr, indicated by the diagonal lines. Therefore, the image processing unit 33 (selection unit 33b) selects image data Cr1, Cr3, and Cr4 to which the first imaging condition is applied from the image data Cr1 to Cr4 of the color difference components. Furthermore, the image processing unit 33 (selection unit 33b) selects image data Cr15 (or Cr16) of the color difference component Cr, which is located near the reference position corresponding to data Cr2 and to which the first imaging condition is applied. In other words, compared to the first embodiment, the image processing unit 33 (selection unit 33b) changes the position at which it selects image data to which the first imaging condition is applied. If the second imaging condition was also applied at the location of data Cr15 or Cr16, data to which the first imaging condition was applied may be selected from image data at a location near data Cr15 or Cr16.

[0162] The image processing unit 33 (generation unit 33c) calculates the image data of the color difference component at the point of interest (2nd row, 2nd column) by referring to the image data selected in this manner. The image processing unit 33 (generation unit 33c) uses, for example, (e3Cr1 + f3Cr15 + g3Cr3 + h3Cr4) / 4 as the image data of the color difference component Cr at the point of interest (2nd row, 2nd column). e3, f3, g3, and h3 are weighting coefficients that are set according to the distance between the reference point and the point of interest and the image structure.

[0163] The same applies when generating the image data of the chromatic aberration component Cr for the attention position indicated by the thick frame (second row, third column) in FIG. 9(c). In the example of FIG. 9(c), the first imaging condition different from the second imaging condition applied to the attention position (second row, third column) is applied to the reference positions corresponding to the image data Cr4 and Cr5 of the chromatic aberration component Cr indicated by the oblique lines. Therefore, the image processing unit 33 (selection unit 33b) selects the image data Cr2 and Cr6 to which the second imaging condition is applied from the image data Cr2, Cr4 to Cr6 of the chromatic aberration component. Further, the image processing unit 33 (selection unit 33b) selects the image data Cr8 and Cr7 of the chromatic aberration component Cr to which the second imaging condition is applied and which are located in the vicinity of the reference positions corresponding to the data Cr4 and Cr5. That is, the image processing unit 33 (selection unit 33b) changes the position for selecting the image data with respect to the first embodiment and selects the image data to which the second imaging condition is applied. Incidentally, if the first imaging condition is also applied at the positions of the data Cr8 and Cr7, data to which the second imaging condition is applied may be selected from the image data at the positions in the vicinity of the data Cr8 and Cr7.

[0164] The image processing unit 33 (generation unit 33c) calculates the image data of the chromatic aberration component Cr at the attention position (second row, third column) by referring to the image data selected in this way. The image processing unit 33 (generation unit 33c) uses, for example, (q3Cr2 + r3Cr8 + s3Cr7 + t3r6) / 4 as the image data of the chromatic aberration component Cr at the attention position. Here, q3, r3, s3, and t3 are weighting coefficients provided according to the distance between the reference position and the attention position and the image structure.

[0165] After obtaining the image data of the chromatic aberration component Cr at each pixel position, the image processing unit 33 (generation unit 33c) adds the image data of the G color component shown in FIG. 8(c) corresponding to each pixel position to obtain the image data of the R color component at each pixel position.

[0166] Next, we will describe the B color interpolation of the second embodiment. Similar to the first embodiment, in Figures 10(a) to 10(c), for example, the first imaging condition is applied to the area to the left and above the thick line, and the second imaging condition is applied to the area to the right and below the thick line.

[0167] In the example in Figure 10(b), the first imaging condition, which is different from the second imaging condition applied to the point of interest indicated by the thick frame (3rd row, 3rd column), is applied to the reference positions corresponding to the image data Cb1 and Cb3 of the color difference component Cb, which are indicated by the diagonal lines. Therefore, the image processing unit 33 (selection unit 33b) selects image data Cb2 and Cb4 to which the second imaging condition is applied from the image data Cb1 to Cb4 of the color difference component. Furthermore, the image processing unit 33 (selection unit 33b) selects image data Cb16 and Cb17 of the color difference component Cb, which are located near the reference positions corresponding to the data Cb1 and Cb3 and to which the second imaging condition is applied. In other words, compared to the first embodiment, the image processing unit 33 (selection unit 33b) changes the position to which it selects image data and selects image data to which the second imaging condition is applied.

[0168] The image processing unit 33 (generation unit 33c) calculates the image data Cb of the color difference component at the point of interest (3rd row, 3rd column) by referring to the image data selected in this manner. The image processing unit 33 (generation unit 33c) uses, for example, (u3Cb16+v3Cb2+w3Cb4+x3Cb17) / 4 as the image data of the color difference component Cb at the point of interest (3rd row, 3rd column). Note that u3, v3, w3, and x3 are weighting coefficients that are set according to the distance between the reference point and the point of interest and the image structure.

[0169] The same procedure applies when generating the color difference component image data Cb for the point of interest (3rd row, 4th column) indicated by the thick border in Figure 10(c). In the example shown in Figure 10(c), the same second imaging conditions as those applied to the position of interest (3rd row, 4th column) are applied to the reference position corresponding to the image data Cb2, Cb4-Cb6 of the four color difference components located near the position of interest (3rd row, 4th column). The image processing unit 33 (generation unit 33c) calculates the image data of the color difference component Cb at the position of interest by referring to the image data Cb2, Cb4-Cb6 of the four color difference components located near the position of interest.

[0170] The image processing unit 33 (generation unit 33c) obtains image data of the color difference component Cb at each pixel position, and then adds the image data of the G color component shown in Figure 8(c) corresponding to each pixel position to obtain image data of the B color component at each pixel position.

[0171] According to the second embodiment described above, in addition to the same effects as the first embodiment, the following effects can be obtained. Specifically, the selection unit 33b selects image data of pixels other than the reference pixel, in a region where imaging conditions common to the pixel position (position of the pixel of interest) are set. This allows, for example, an increase in the data referenced for the generation of the third image data by the generation unit 33c, enabling appropriate image processing.

[0172] (Variations of the first and second embodiments) The following modifications are also within the scope of the present invention, and it is possible to combine one or more of these modifications with the embodiments described above. (Variation 1) Figures 16(a) to 16(c) illustrate the arrangement of the first and second regions on the imaging surface of the image sensor 32a. In the example shown in Figure 16(a), the first region is composed of even-numbered rows, and the second region is composed of odd-numbered rows. That is, the imaging surface is divided into even-numbered and odd-numbered rows.

[0173] In the example shown in Figure 16(b), the first region is composed of odd-numbered rows, and the second region is composed of even-numbered rows. In other words, the imaging surface is divided into odd-numbered and even-numbered rows.

[0174] In the example shown in Figure 16(c), the first region is composed of blocks of even-numbered rows in odd-numbered columns and blocks of odd-numbered rows in even-numbered columns. Similarly, the second region is composed of blocks of even-numbered rows in even-numbered columns and blocks of odd-numbered rows in odd-numbered columns. In other words, the imaging surface is divided into a checkerboard pattern.

[0175] In all cases shown in Figures 16(a) to 16(c), the photoelectric conversion signal read from the image sensor 32a that captured one frame generates a first image based on the photoelectric conversion signal read from the first region and a second image based on the photoelectric conversion signal read from the second region, respectively. According to Modification 1, the first and second images are captured with the same field of view and contain a common subject image.

[0176] In Modification 1, the control unit 34 uses the first image for display and the second image for detection. Specifically, the control unit 34 displays the first image as a live view image on the display unit 35. The control unit 34 also has the object detection unit 34a perform subject detection processing using the second image, the AF calculation unit 34 perform focus detection processing using the second image, and the setting unit 34b perform exposure calculation processing using the second image.

[0177] In Modification 1, the imaging conditions set for the first region where the first image is captured are called the first imaging conditions, and the imaging conditions set for the second region where the second image is captured are called the second imaging conditions. The control unit 34 may set the first imaging conditions and the second imaging conditions to be different.

[0178] 1. As an example, the control unit 34 sets the first imaging conditions to conditions suitable for display by the display unit 35. The first imaging conditions are made the same throughout the entire first region of the imaging screen. On the other hand, the control unit 34 sets the second imaging conditions to conditions suitable for focus detection processing, subject detection processing, and exposure calculation processing. The second imaging conditions are also made the same throughout the entire second region of the imaging screen. Furthermore, if the conditions suitable for focus detection processing, subject detection processing, and exposure calculation processing differ, the control unit 34 may set different second imaging conditions for each frame in the second region. For example, the second imaging conditions for the first frame may be set to conditions suitable for focus detection processing, the second imaging conditions for the second frame to be conditions suitable for subject detection processing, and the second imaging conditions for the third frame to be conditions suitable for exposure calculation processing. In these cases, the second imaging conditions in each frame are made the same for the entire second region of the imaging screen.

[0179] 2. As another example, the control unit 34 may set different first imaging conditions for the first region. The control unit 34 (setting unit 34b) sets different first imaging conditions for each region containing subject elements divided by the setting unit 34b. On the other hand, the control unit 34 sets the second imaging conditions to be the same for the entire second region of the imaging screen. The control unit 34 sets the second imaging conditions to conditions suitable for focus detection processing, subject detection processing, and exposure calculation processing, but if the conditions suitable for focus detection processing, subject detection processing, and exposure calculation processing are different, the imaging conditions set for the second region may be different for each frame.

[0180] 3. As another example, the control unit 34 may set the first imaging conditions to be the same for the entire first region of the imaging screen, while setting different second imaging conditions for the second region on the imaging screen. For example, the setting unit 34b may set different second imaging conditions for each region containing the divided subject elements. In this case as well, if the conditions suitable for focus detection processing, subject detection processing, and exposure calculation processing are different, the imaging conditions set for the second region may be different for each frame.

[0181] 4. As another example, the control unit 34 may vary the first imaging conditions set for the first region on the imaging screen, and also vary the second imaging conditions set for the second region on the imaging screen. For example, the setting unit 34b may set different first imaging conditions for each region containing the subject elements it has divided, while simultaneously setting different second imaging conditions for each region containing the subject elements it has divided.

[0182] In Figures 16(a) to 16(c), the area ratio of the first region and the second region may be different. The control unit 34 may, for example, set the ratio of the first region higher than that of the second region, set the ratio of the first and second regions to be the same as illustrated in Figures 16(a) to 16(c), or set the ratio of the first region to be lower than that of the second region, based on user operation or the control unit 34's judgment. By making the area ratio of the first and second regions different, it is possible to make the first image more detailed than the second image, to make the resolution of the first and second images the same, or to make the second image more detailed than the first image.

[0183] (Modification 2) In the above embodiment, an example was described in which the control unit 34 (setting unit 34b) detects subject elements based on the live view image and divides the live view image screen into regions containing the subject elements. In Modification 2, if the control unit 34 is equipped with a photometering sensor in addition to the image sensor 32a, the regions may be divided based on the output signal from the photometering sensor.

[0184] The control unit 34 divides the image into a foreground and a background based on the output signal from the photometering sensor. Specifically, it divides the live view image acquired by the image sensor 32b into a foreground region corresponding to the area determined to be the foreground based on the output signal from the photometering sensor, and a background region corresponding to the area determined to be the background based on the output signal from the photometering sensor.

[0185] The control unit 34 further positions the first and second regions at locations corresponding to the foreground region of the imaging surface of the image sensor 32a, as illustrated in Figures 16(a) to 16(c). On the other hand, the control unit 34 positions only the first region on the imaging surface of the image sensor 32a at locations corresponding to the background region of the imaging surface of the image sensor 32a. The control unit 34 uses the first image for display and the second image for detection.

[0186] According to Modification 2, the live view image acquired by the image sensor 32b can be segmented by using the output signal from the photometric sensor. Furthermore, a first image for display and a second image for detection can be obtained for the foreground region, while only the first image for display can be obtained for the background region.

[0187] (Variation 3) In the modified example 3, the image processing unit 33 (generation unit 33c) performs contrast adjustment processing to mitigate the discontinuity in the image caused by the difference between the first imaging conditions and the second imaging conditions. Specifically, the generation unit 33c mitigates the discontinuity in the image caused by the difference between the first imaging conditions and the second imaging conditions by making the gradation curve (gamma curve) different.

[0188] For example, consider a case where the only difference between the first and second imaging conditions is the ISO sensitivity, with the ISO sensitivity of the first imaging condition being 100 and the ISO sensitivity of the second imaging condition being 800. The generation unit 33c compresses the image data values ​​of the second imaging condition among the image data at the reference position to 1 / 8 by flattening the gradation curve.

[0189] Alternatively, the generation unit 33c may increase the value of the image data of the position of interest and the image data of the first imaging condition among the image data of the reference position by eight times by raising the grayscale curve.

[0190] According to Modification 3, similar to the embodiment described above, image processing can be appropriately performed on image data generated in regions with different imaging conditions. For example, discontinuities and inconsistencies that appear in the processed image due to differences in imaging conditions at the boundaries of the regions can be suppressed.

[0191] (Modification 4) In the modified example 4, the image processing unit 33 (generation unit 33c) ensures that the contours of the subject elements are not damaged during the image processing described above (for example, noise reduction processing). Generally, when noise reduction is performed, a smoothing filter is used. When a smoothing filter is used, while noise reduction is achieved, the boundaries of the subject elements may become blurred.

[0192] Therefore, the image processing unit 33 (generation unit 33c) compensates for the blurring of the boundaries of the subject elements by performing contrast adjustment processing in addition to, or together with, noise reduction processing. In modified example 4, the image processing unit 33 (generation unit 33c) sets a curve that draws an S shape as the density conversion (grayscale conversion) curve (so-called S-curve conversion). By performing contrast adjustment using the S-curve conversion, the image processing unit 33 (generation unit 33c) stretches the grayscale portions of the bright data and the dark data respectively to increase the number of grayscales of the bright data (and dark data), and compresses the image data of the intermediate grayscale to reduce the number of grayscales. As a result, the number of image data with moderate brightness decreases and the number of data classified as either bright or dark increases, thereby compensating for the blurring of the boundaries of the subject elements.

[0193] According to variation 4, blurring of the boundaries of subject elements can be compensated for by making the light and dark areas of the image sharper.

[0194] (Variation 5) In modified example 5, the image processing unit 33 (generation unit 33c) changes the white balance adjustment gain to mitigate the discontinuity in the image caused by the difference between the first imaging conditions and the second imaging conditions.

[0195] For example, if the imaging conditions applied during imaging at the point of interest (referred to as the first imaging conditions) are different from the imaging conditions applied during imaging at the reference positions surrounding the point of interest (referred to as the second imaging conditions), the image processing unit 33 (generation unit 33c) changes the white balance adjustment gain so that the white balance of the image data under the second imaging conditions among the image data at the reference positions is brought closer to the white balance of the image data acquired under the first imaging conditions.

[0196] The image processing unit 33 (generation unit 33c) may also change the white balance adjustment gain so that the white balance of the image data under the first imaging condition and the image data at the point of interest among the image data at the reference position is brought closer to the white balance of the image data acquired under the second imaging condition.

[0197] According to Modification 5, by adjusting the white balance adjustment gain of the image data generated in regions with different imaging conditions to the adjustment gain of one of the regions with different imaging conditions, the discontinuity in the image caused by the difference between the first and second imaging conditions can be mitigated.

[0198] (Experimental variation 6) Multiple image processing units 33 may be provided to perform image processing in parallel. For example, image processing may be performed on image data captured in area A of the imaging unit 32 while simultaneously performing image processing on image data captured in area B of the imaging unit 32. Multiple image processing units 33 may perform the same image processing or different image processing. That is, the same parameters can be applied to the image data of areas A and B to perform similar image processing, or different parameters can be applied to the image data of areas A and B to perform different image processing.

[0199] In cases where there are multiple image processing units 33, one image processing unit may perform image processing on image data to which the first imaging condition is applied, and another image processing unit may perform image processing on image data to which the second imaging condition is applied. The number of image processing units is not limited to the above two; for example, the number of units may be equal to the number of imaging conditions that can be set. That is, each image processing unit is responsible for image processing for each region to which different imaging conditions are applied. According to Modification 6, imaging with different imaging conditions for each region and image processing of the image data obtained for each region can be carried out in parallel.

[0200] (Example 7) In the explanation above, camera 1 was used as an example, but the system may also be composed of a high-performance mobile phone 250 (Figure 14) equipped with a camera function, such as a smartphone, or a mobile device such as a tablet terminal.

[0201] (Variation 8) In the embodiments described above, a camera 1 was described as an example in which the imaging unit 32 and the control unit 34 are configured as a single electronic device. Alternatively, for example, an imaging system 1B may be configured in which the imaging unit 32 and the control unit 34 are provided separately, and the imaging unit 32 is controlled from the control unit 34 via communication. The following describes an example in which an imaging device 1001 equipped with an imaging unit 32 is controlled by a control device 1002 equipped with a control unit 34, with reference to Figure 17.

[0202] Figure 17 is a block diagram illustrating the configuration of the imaging system 1B according to the modified example 8. In Figure 17, the imaging system 1B is composed of an imaging device 1001 and a display device 1002. The imaging device 1001 includes the imaging optical system 31 and imaging unit 32 described in the above embodiment, as well as a first communication unit 1003. The display device 1002 includes the image processing unit 33, control unit 34, display unit 35, operating member 36, and recording unit 37 described in the above embodiment, as well as a second communication unit 1004.

[0203] The first communication unit 1003 and the second communication unit 1004 can perform bidirectional image data communication using, for example, well-known wireless communication technology or optical communication technology. Alternatively, the imaging device 1001 and the display device 1002 may be connected via a wired cable, and the first communication unit 1003 and the second communication unit 1004 may be configured to perform bidirectional image data communication.

[0204] The imaging system 1B controls the imaging unit 32 through data communication by the control unit 34 via the second communication unit 1004 and the first communication unit 1003. For example, by sending and receiving predetermined control data between the imaging device 1001 and the display device 1002, the display device 1002 divides the screen into multiple regions based on the image, sets different imaging conditions for each divided region, and reads out the photoelectric conversion signals converted by photoelectric conversion in each region, as described above.

[0205] According to Modification 8, the live view image acquired by the imaging device 1001 and transmitted to the display device 1002 is displayed on the display unit 35 of the display device 1002, so the user can remotely operate the display device 1002, which is located away from the imaging device 1001. The display device 1002 can be configured, for example, by a high-function mobile phone 250 such as a smartphone. The imaging device 1001 can be configured by an electronic device equipped with the stacked image sensor 100 described above. Although an example has been described in which the control unit 34 of the display device 1002 is provided with an object detection unit 34a, a setting unit 34b, an imaging control unit 34c, and an AF calculation unit 34d, some of the object detection unit 34a, setting unit 34b, imaging control unit 34c, and parts of the AF calculation unit 34d may be provided in the imaging device 1001.

[0206] (Extreme variation 9) The program can be supplied to the aforementioned camera 1, high-performance mobile phone 250, or mobile device such as a tablet terminal by transmitting it from a personal computer 205 that stores the program to the mobile device via infrared communication or short-range wireless communication, as illustrated in Figure 18, for example.

[0207] The program for the personal computer 205 may be supplied by inserting a recording medium 204, such as a CD-ROM, containing the program into the personal computer 205, or by loading it into the personal computer 205 via a communication line 201, such as a network. If using the communication line 201, the program is stored in a storage device 203 of a server 202 connected to the communication line.

[0208] Furthermore, the program can be directly transmitted to a mobile device via a wireless LAN access point (not shown) connected to the communication line 201. Alternatively, a recording medium 204B, such as a memory card containing the program, may be inserted into the mobile device. In this way, the program can be supplied as a computer program product in various forms, including via recording media and communication lines.

[0209] (Third embodiment) Referring to Figures 19-25, a digital camera will be used as an example of an electronic device equipped with an image processing device according to the third embodiment. In the following description, the same reference numerals are used for components that are the same as those in the first or second embodiment, and the differences will be mainly explained. Points that are not specifically explained are the same as those in the first or second embodiment. This embodiment differs from the first embodiment mainly in that, instead of providing the image processing unit 33 of the first embodiment, the imaging unit 32A further includes an image processing unit 32c that has the same function as the image processing unit 33 of the first embodiment.

[0210] Figure 19 is a block diagram illustrating the configuration of camera 1C according to the third embodiment. In Figure 19, camera 1C includes an imaging optical system 31, an imaging unit 32A, a control unit 34, a display unit 35, an operating member 36, and a recording unit 37. The imaging unit 32A further includes an image processing unit 32c having the same functions as the image processing unit 33 of the first embodiment.

[0211] The image processing unit 32c includes an input unit 321, a selection unit 322, and a generation unit 323. Image data from the image sensor 32a is input to the input unit 321. The selection unit 322 performs preprocessing on the input image data. The preprocessing performed by the selection unit 322 is the same as the preprocessing performed by the selection unit 33b in the first embodiment. The generation unit 323 performs image processing on the input image data and the preprocessed image data to generate an image. The image processing performed by the generation unit 323 is the same as the image processing performed by the generation unit 33c in the first embodiment.

[0212] Figure 20 is a schematic diagram showing the correspondence between each block and the multiple selection units 322 in this embodiment. In Figure 20, one square of the imaging chip 111, represented by a rectangle, represents one block 111a. Similarly, one square of the image processing chip 114, which will be described later, represented by a rectangle, represents one selection unit 322.

[0213] In this embodiment, the selection unit 322 is provided in correspondence to each block 111a. In other words, the selection unit 322 is provided for each block, which is the smallest unit of the region on the imaging plane in which the imaging conditions can be changed. For example, in Figure 20, the hatched block 111a corresponds to the hatched selection unit 322. In Figure 20, the hatched selection unit 322 performs preprocessing on the image data from the pixels included in the hatched block 111a. Each selection unit 322 performs preprocessing on the image data from the pixels included in the corresponding block 111a. This allows image data preprocessing to be performed in parallel by multiple selection units 322, reducing the processing load on the selection units 322 and enabling the generation of appropriate images in a short time from image data generated in areas with different imaging conditions. In the following explanation, when describing the relationship between a block 111a and the pixels contained within it, the block 111a may be referred to as the block 111a to which the pixel belongs. Furthermore, a block 111a may be referred to as a unit division, and a collection of multiple blocks 111a, i.e., a collection of multiple unit divisions, may be referred to as a composite division.

[0214] Figure 21 is a cross-sectional view of the stacked image sensor 100A. The stacked image sensor 100A comprises a back-illuminated imaging chip 111, a signal processing chip 112, a memory chip 113, and an image processing chip 114 that performs the pre-processing and image processing described above. In other words, the image processing unit 32c described above is located on the image processing chip 114. These imaging chips 111, signal processing chip 112, memory chip 113, and image processing chip 114 are stacked and electrically connected to each other by conductive bumps 109 made of Cu or the like.

[0215] Multiple bumps 109 are arranged on the opposing surfaces of the memory chip 113 and the image processing chip 114. When these bumps 109 are aligned with each other and pressure is applied to the memory chip 113 and the image processing chip 114, the aligned bumps 109 are joined together and electrically connected.

[0216] <Data Selection Process> Similar to the first embodiment, in the second embodiment, after the setting unit 34b divides the area of ​​the imaging screen, it is possible to set (change) imaging conditions for the area selected by the user or the area determined by the control unit 34. If different imaging conditions are set for the divided areas, the control unit 34 causes the selection unit 322 to perform the following data selection process as necessary.

[0217] 1. When performing image processing 1-1. When the imaging conditions for the pixel of interest P are the same as the imaging conditions for multiple reference pixels surrounding the pixel of interest P. In this case, the selection unit 322 selects all the image data of multiple reference pixels and outputs it to the generation unit 323. The generation unit 323 performs image processing using the image data of multiple reference pixels.

[0218] 1-2. When the imaging conditions for the pixel of interest P are different from the imaging conditions for at least one of the multiple reference pixels surrounding the pixel of interest P. The imaging conditions applied to the pixel of interest P are defined as the first imaging conditions. The imaging conditions applied to some of the multiple reference pixels are also defined as the first imaging conditions, and the imaging conditions applied to the remaining reference pixels are defined as the second imaging conditions. In this case, the selection unit 322 corresponding to block 111a to which the reference pixel to which the first imaging condition is applied belongs, and the selection unit 322 corresponding to block 111a to which the reference pixel to which the second imaging condition is applied belongs, perform data selection processing on the image data of the reference pixel as shown in (Example 1) to (Example 3) below. Then, the generation unit 323 performs image processing to calculate the image data of the pixel of interest P by referring to the image data of the reference pixel after the data selection processing.

[0219] (Example 1) For example, suppose the only difference between the first and second imaging conditions is the ISO sensitivity, with the ISO sensitivity of the first condition being 100 and the ISO sensitivity of the second condition being 800. In this case, the selection unit 322 corresponding to block 111a to which the reference pixel to which the first imaging condition is applied belongs selects the image data for the first imaging condition. However, the selection unit 322 corresponding to block 111a to which the reference pixel to which the second imaging condition is applied does not select the image data for the second imaging condition. In other words, the image data for the second imaging condition, which is different from the first imaging condition, is not used for image processing.

[0220] (Example 2) For example, suppose the only difference between the first and second imaging conditions is the shutter speed, with the shutter speed for the first condition being 1 / 1000 second and the shutter speed for the second condition being 1 / 100 second. In this case, the selection unit 322 corresponding to block 111a to which the reference pixel to which the first imaging condition is applied belongs selects the image data for the first imaging condition. However, the selection unit 322 corresponding to block 111a to which the reference pixel to which the second imaging condition is applied does not select the image data for the second imaging condition. In other words, the image data for the second imaging condition, which is different from the first imaging condition, is not used for image processing.

[0221] (Example 3) For example, suppose the only difference between the first and second imaging conditions is the frame rate (the charge accumulation time is the same), and the frame rate for the first imaging condition is 30fps, while the frame rate for the second imaging condition is 60fps. In this case, the selection unit 322 corresponding to block 111a to which the reference pixel to which the first imaging condition is applied belongs selects the image data of the pixel to which the first imaging condition is applied. The selection unit 322 corresponding to block 111a to which the reference pixel to which the second imaging condition is applied belongs selects the image data of the frame image from the reference pixel to which the acquisition timing is close to that of the frame image acquired under the first imaging condition (30fps), based on the image data of the second imaging condition (60fps). In other words, the image data of the reference pixel to which the acquisition timing is close to that of the frame image to which the first imaging condition (30fps) belong is used for image processing, and the image data of the frame image to which the acquisition timing is different from that of the frame image to which the first imaging condition (30fps) belong is not used for image processing.

[0222] The same applies when the imaging conditions applied to the pixel of interest P are designated as the second imaging conditions, and the imaging conditions applied to the reference pixels surrounding the pixel of interest P are designated as the first imaging conditions. In this case, the selection unit 322 corresponding to block 111a to which the reference pixel to which the first imaging conditions are applied belongs, and the selection unit 322 corresponding to block 111a to which the reference pixel to which the second imaging conditions are applied, perform data selection processing on the image data of the reference pixels as described in (Example 1) to (Example 3).

[0223] As mentioned above, even if there are slight differences in imaging conditions, they will be considered the same.

[0224] Based on the image data of the reference pixels selected by the selection unit 322, the generation unit 323 performs image processing such as pixel defect correction, color interpolation, edge enhancement, and noise reduction, similar to the image processing unit 33 (generation unit 33c) in the first embodiment.

[0225] Figure 22 schematically illustrates the processing of image data from each pixel in a portion of the imaging surface to which the first imaging condition is applied (hereinafter referred to as the first region 141) (hereinafter referred to as the first image data) and image data from each pixel in a portion of the imaging surface to which the second imaging condition is applied (hereinafter referred to as the second region 142) (hereinafter referred to as the second image data). Figure 22 is a diagram illustrating the case in (Example 1) and (Example 2) above where the imaging condition applied to the pixel of interest P is the first imaging condition.

[0226] Each pixel in the first region 141 outputs first image data captured under first imaging conditions, and each pixel in the second region 142 outputs second image data captured under second imaging conditions. The first image data is output to one of the multiple selection units 322 provided on the processing chip 114, which corresponds to the selection unit 322 corresponding to the block 111a to which the pixel that generated the first image data belongs. In the following description, the multiple selection units 322 corresponding to the multiple blocks 111a to which each pixel that generated the first image data belongs will be referred to as the first selection unit 151. Similarly, the second image data is output to one of the multiple selection units 322 provided on the processing chip 114, which corresponds to the selection unit 322 corresponding to the block 111a to which each pixel that generated the second image data belongs. In the following description, the multiple selection units 322 corresponding to the multiple blocks 111a to which each pixel that generated the second image data belongs will be referred to as the second selection unit 152.

[0227] For example, if the pixel of interest P is included in the first region 141, the first selection unit 151 selects the image data of the pixel of interest P and the image data of the reference pixel captured under the first imaging condition and outputs them to the generation unit 323. Here, the selection unit 322 selected image data from the same block, but it may also use image data from another block captured under the first imaging condition. In this case, the selection unit 322 that received the pixel of interest P as input and the selection unit 322 of the other block captured under the first imaging condition only need to send and receive information 181 about the first imaging condition necessary for the data selection process. On the other hand, the second selection unit 152 does not select the image data of the reference pixel captured under the second imaging condition and does not output the image data of the reference pixel captured under the second imaging condition to the generation unit 323. The second selection unit 152 receives the information 181 about the first imaging condition necessary for the data selection process, for example, from the first selection unit 151. Similarly, for example, if the pixel of interest P is included in the second region, the second selection unit 152 selects the image data of the pixel of interest P and the image data of the reference pixel captured under the second imaging condition and outputs them to the generation unit 323. On the other hand, the first selection unit 151 does not select the image data of the reference pixel captured under the first imaging condition and does not output the image data of the reference pixel captured under the first imaging condition to the generation unit 323. The first selection unit 151 receives information about the second imaging condition necessary for data selection processing, for example, from the second selection unit 152.

[0228] After the preprocessing described above, the generation unit 323 performs image processing such as pixel defect correction, color interpolation, edge enhancement, and noise reduction based on the image data from the first selection unit 151 and the second selection unit 152, and outputs the image data after image processing.

[0229] 2. When performing focus detection processing Similar to the first embodiment, the control unit 34 (AF calculation unit 34d) performs focus detection processing using image data corresponding to a predetermined position (focus detection position) on the imaging screen. If different imaging conditions are set between the divided regions, and the focus detection position for AF operation is located at the boundary of the divided regions, i.e., the focus detection position is divided between the first region and the second region, in this embodiment, as described in 2-2 below, the control unit 34 (AF calculation unit 34d) causes the selection unit 322 to perform data selection processing.

[0230] 2-1. In the image data from pixels within frame 170 in Figure 13, when the signal data for focus detection applied to the first imaging condition and the signal data for focus detection applied to the second imaging condition are not mixed. In this case, the selection unit 322 selects all the signal data for focus detection from the pixels within the frame 170 and outputs it to the generation unit 323. The control unit 34 (AF calculation unit 34d) performs focus detection processing using the signal data for focus detection from the pixels for focus detection indicated by the frame 170.

[0231] 2-2. When the focus detection signal data from pixels within frame 170 in Figure 13 contains a mixture of focus detection signal data to which the first imaging condition is applied and focus detection signal data to which the second imaging condition is applied. In this case, the control unit 34 (AF calculation unit 34d) causes the selection unit 322 corresponding to the block 111a to which the pixels within the frame 170 belong to perform data selection processing as shown in (Example 1) to (Example 3) below. Then, the control unit 34 (AF calculation unit 34d) performs focus detection processing using the signal data for focus detection after the data selection processing.

[0232] (Example 1) For example, suppose the only difference between the first and second imaging conditions is the ISO sensitivity, with the ISO sensitivity of the first imaging condition being 100 and the ISO sensitivity of the second imaging condition being 800. In this case, the selection unit 322 corresponding to block 111a to which the pixels to which the first imaging condition is applied belong selects the signal data for focus detection of the first imaging condition. The selection unit 322 corresponding to block 111a to which the pixels to which the second imaging condition is applied does not select the signal data for focus detection of the second imaging condition. In other words, of the signal data for focus detection from pixels within the frame 170, the signal data for focus detection of the first imaging condition is used for the focus detection process, and the image data of the second imaging condition, which is different from the first imaging condition, is not used for the focus detection process.

[0233] (Example 2) For example, suppose the only difference between the first and second imaging conditions is the shutter speed, with the shutter speed for the first condition being 1 / 1000 second and the shutter speed for the second condition being 1 / 100 second. In this case, the selection unit 322 corresponding to block 111a to which the pixels to which the first imaging condition is applied belong selects the signal data for focus detection of the first imaging condition. The selection unit 322 corresponding to block 111a to which the pixels to which the second imaging condition is applied does not select the signal data for focus detection of the second imaging condition. In other words, of the signal data for focus detection from pixels within the frame 170, the signal data for focus detection of the first imaging condition is used for focus detection processing, and the signal data for focus detection of the second imaging condition, which is different from the first imaging condition, is not used for focus detection processing.

[0234] (Example 3) For example, suppose that the only difference between the first and second imaging conditions is the frame rate (the charge accumulation time is the same), and the frame rate for the first imaging condition is 30fps and the frame rate for the second imaging condition is 60fps. In this case, the selection unit 322 corresponding to block 111a to which the pixels to which the first imaging condition is applied belong selects the signal data for focus detection of the pixels to which the first imaging condition is applied. The selection unit 322 corresponding to block 111a to which the pixels to which the second imaging condition is applied belong selects the signal data for focus detection of frame images with acquisition timings close to those of the frame images acquired under the first imaging condition (30fps) for the image data of the second imaging condition (60fps). In other words, the signal data for focus detection of frame images with acquisition timings close to those of the frame images to which the first imaging condition (30fps) belong is used for focus detection processing, while the signal data for focus detection of frame images with acquisition timings different from those of the frame images to which the first imaging condition (30fps) belong is not used for focus detection processing.

[0235] As mentioned above, even if there are slight differences in imaging conditions, they will be considered the same. Furthermore, while the above examples (1) and (2) describe selecting the focus detection signal data for the first imaging condition from the focus detection signal data enclosed in frame 170, it is also possible to select the focus detection signal data for the second imaging condition from the focus detection signal data enclosed in frame 170. Furthermore, if the focus detection position is divided into two regions, the first region, and the area of ​​the first region is larger than the area of ​​the second region, it is desirable to select the image data for the first imaging condition. Conversely, if the area of ​​the second region is larger than the area of ​​the first region, it is desirable to select the image data for the second imaging condition.

[0236] Figure 23 schematically illustrates the processing of the first signal data and the second signal data related to focus detection. Figure 23 illustrates the case in (Example 3) above where the signal data for focus detection for the first imaging condition is selected from the signal data generated from the area enclosed by frame 170, and the signal data for focus detection for the second imaging condition is selected. Each pixel in the first region 141 outputs first signal data for focus detection, captured under first imaging conditions, and each pixel in the second region 142 outputs second signal data for focus detection, captured under second imaging conditions. The first signal data from the first region 141 is output to the first selection unit 151. Similarly, the second signal data from the second region 142 is output to the second selection unit 152. The first processing unit 151 selects the first signal data for the first imaging condition and outputs it to the AF calculation unit 34d. The second processing unit 152 selects the second signal data for the second imaging condition and outputs it to the AF calculation unit 34d. After the preprocessing described above, the AF calculation unit 34d calculates a first defocus amount from the first signal data from the first processing unit 151. Furthermore, the AF calculation unit 34d calculates a second defocus amount from the second signal data from the first processing unit 151. Then, using the first and second defocus amounts, the AF calculation unit 34d outputs a drive signal to move the focus lens of the imaging optical system 31 to the focus position.

[0237] Furthermore, as in (Example 1) and (Example 2) above, if the signal data for focus detection for the first imaging condition is selected from the signal data of the area enclosed by frame 170, and the signal data for focus detection for the second imaging condition is not selected, the first signal data and the second signal data will be processed as follows.

[0238] For example, when performing focus detection processing with the first signal data of the first imaging condition, the first processing unit 151 selects the first signal data of the first imaging condition and outputs it to the generation unit 323. The second processing unit 152 does not select the second signal data of the second imaging condition and does not output the second signal data of the reference pixel captured under the second imaging condition to the generation unit 323. The second processing unit 152 receives information 181 about the first imaging condition necessary for data selection processing from, for example, the first processing unit 151.

[0239] After the preprocessing described above, the AF calculation unit 34d performs focus detection processing based on the first signal data from the first processing unit 151, and outputs a drive signal to move the focus lens of the imaging optical system 31 to the focus position based on the calculation result.

[0240] Here, we will explain an example where the subject to be focused is located across the area to which the first imaging condition is set and the area to which the second imaging condition is set. When the subject to be focused is located across the area to which the first imaging condition is set and the area to which the second imaging condition is set, the selection unit 322 corresponding to block 111a to which the pixels to which the first imaging condition is applied belong selects the first signal data for focus detection of the first imaging condition. Furthermore, the selection unit 322 corresponding to block 111a to which the pixels to which the second imaging condition is applied belong selects the second signal data for focus detection of the second imaging condition. Then, the control unit 34 (AF calculation unit 34d) calculates the first defocus amount from the selected first signal data for focus detection. Furthermore, the control unit 34 (AF calculation unit 34d) calculates the second defocus amount from the selected second signal data for focus detection. Then, the control unit 34 (AF calculation unit 34d) performs focus detection processing using the first defocus amount and the second defocus amount. Specifically, for example, the control unit 34 (AF calculation unit 34d) calculates the average of the first defocus amount and the second defocus amount to calculate the lens movement distance. Alternatively, the control unit 34 (AF calculation unit 34d) may select the value of the first defocus amount and the second defocus amount that results in a smaller lens movement distance. Alternatively, the control unit 34 (AF calculation unit 34d) may select a value from the first defocus amount and the second defocus amount that indicates the subject is closer. Also, when the subject to be focused on is located across the region where the first imaging condition is set and the region where the second imaging condition is set, the control unit 34 (AF calculation unit 34d) may select the region with the larger area of the subject region and select the photoelectric conversion signal for focus detection. For example, when 70% of the area of the face of the subject to be focused on is in the region where the first imaging condition is set and 30% is in the second region, the control unit 34 (AF calculation unit 34d) selects the photoelectric conversion signal for focus detection under the first imaging condition. Note that the ratios (percentages) regarding the areas described above are merely examples and are not limited thereto.

[0241] 3. When performing subject detection processing A case where different imaging conditions are set between the divided regions and the search range 190 includes the boundaries of the divided regions will be described.

[0242] 3-1. When the image data to which the first imaging condition is applied and the image data to which the second imaging condition is applied are not mixed in the image data of the search range 190 in FIG. 14 In this case, the selection unit 322 selects all the image data from the pixels of the search range 190 and outputs it to the generation unit 323. The control unit 34 (AF calculation unit 34d) performs subject detection processing using the image data from the pixels of the search range 190.

[0243] 3-2. When the image data to which the first imaging condition is applied and the image data to which the second imaging condition is applied are mixed in the image data of the search range 190 in FIG. 14 (a) When performing the focus detection processing described above, (Example 1), (Example 2) When only the ISO sensitivity is different between the first imaging condition and the second imaging condition, or when only the shutter speed is different between the first imaging condition and the second imaging condition In this case, as shown in FIG. 24, the control unit 34 (object detection unit 34a) causes the selection unit 322 (first selection unit 151) corresponding to the block 111a to which the pixels to which the first imaging condition is applied belong to select the image data of the first imaging condition used for the subject detection process from the image data in the search range 190. Note that FIG. 24 is a diagram schematically showing the processing of the first image data and the second image data related to the subject detection process. Then, the control unit 34 (object detection unit 34a) performs the subject detection process using the image data after the data selection process. Further, the control unit 34 (object detection unit 34a) causes the selection unit 322 (second selection unit 152) corresponding to the block 111a to which the pixels to which the second imaging condition is applied belong to select the image data of the second imaging condition used for the subject detection process from the image data in the search range 190. Then, the control unit 34 (object detection unit 34a) performs the subject detection process using the image data after the data selection process. Then, the control unit 34 (object detection unit 34a) can detect the subject detection within the search range 190 by aligning the boundary between the subject region detected from the image data of the first imaging condition and the subject region detected from the image data of the second imaging condition.

[0244] (b) When only the frame rate is different between the first imaging condition and the second imaging condition in the same manner as in (Example 3) when performing the above-described focus detection process In this case, the control unit 34 (object detection unit 34a) instructs the selection unit 322 (first selection unit 151) corresponding to block 111a to which the pixels to which the first imaging condition is applied belong to select image data of the first imaging condition to be used for subject detection processing from the image data of the search range 190. Then, the control unit 34 (object detection unit 34a) performs subject detection processing using the image data after the data selection processing. Furthermore, the control unit 34 (object detection unit 34a) instructs the selection unit 322 (second selection unit 152) corresponding to block 111a to which the pixels to which the second imaging condition is applied belong to select only image data of frames acquired at a similar timing to the frame images acquired under the first imaging condition (30fps) from the image data of the search range 190 to be used for subject detection processing under the second imaging condition (60fps). Then, the control unit 34 (object detection unit 34a) performs subject detection processing using the image data after the data selection processing. The control unit 34 can then detect the detection of a subject within the search range 190 by aligning the boundary between the subject area detected from the image data under the first imaging condition and the subject area detected from the image data under the second imaging condition.

[0245] Furthermore, if the search range 190 is divided into a first and second region, and the area of ​​the first region is larger than the area of ​​the second region, the image data for the first imaging condition may be selected and the image data for the second imaging condition may not be selected. Also, if the area of ​​the second region is larger than the area of ​​the first region, the image data for the second imaging condition may be selected and the image data for the first imaging condition may not be selected.

[0246] 4. When setting imaging conditions This section describes a method for determining exposure conditions by dividing the image area, setting different imaging conditions for each divided area, and then re-measuring the image.

[0247] 4-1. When the image data within the photometric range does not contain a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied. In this case, the selection unit 322 selects all image data from pixels within the photometric range and outputs it to the generation unit 323. The control unit 34 (setting unit 34b) uses the image data from pixels constituting the photometric range to perform exposure calculation processing.

[0248] 4-2. When the image data within the metering range contains a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied: (a) When the focus detection processing described above is performed, the only difference between the first and second imaging conditions is the ISO sensitivity, or the only difference between the first and second imaging conditions is the shutter speed. In this case, as shown in Figure 25, the control unit 34 (object detection unit 34a) causes the selection unit 322 corresponding to the block 111a to which the pixels to which the first imaging condition is applied belong to be selected from the image data of the photometric range to be used for exposure calculation processing, similar to (a) when performing the subject detection processing described above. The control unit 34 (object detection unit 34a) also causes the selection unit 322 corresponding to the block 111a to which the pixels to which the second imaging condition is applied belong to be selected from the image data of the photometric range to be used for exposure calculation processing to be selected for the second imaging condition. Figure 25 is a schematic diagram illustrating the processing of the first image data and the second image data related to setting imaging conditions such as exposure calculation processing. Then, the control unit 34 (setting unit 34b) performs exposure calculation processing for each of the regions where the first imaging condition is set and the regions where the second imaging condition is set, using the image data after the data selection process. In this way, when there are multiple regions with different imaging conditions in the photometering range, the control unit 34 (setting unit 34b) performs data selection processing to photometer each region and performs exposure calculation processing using the image data from the data selection process.

[0249] (b) In the case where only the frame rate differs between the first imaging condition and the second imaging condition, similar to (Example 3) when performing the focus detection processing described above. In this case, the control unit 34 (object detection unit 34a) causes the selection unit 322 corresponding to the block 111a to which the pixels to which the first imaging condition is applied belong to belong to select the image data of the first imaging condition to be used for exposure calculation processing from the image data of the photometric range, similar to (b) when performing subject detection processing. Furthermore, the control unit 34 (object detection unit 34a) causes the selection unit 322 corresponding to the block 111a to which the pixels to which the second imaging condition is applied belong to belong to select only the image data of frames acquired with the first imaging condition (30fps) that are close in acquisition timing to the frame images acquired with the first imaging condition (30fps) as the image data of the second imaging condition (60fps) to be used for exposure calculation processing, similar to (Example 3) when performing focus detection processing as described above. Then, the control unit 34 (setting unit 34b) performs exposure calculation processing using the image data after the data selection process, similar to the case in (a) described above.

[0250] Furthermore, if the photometric range is divided into two regions, a first and a second region, and the area of ​​the first region is larger than the area of ​​the second region, the image data for the first imaging condition may be selected and the image data for the second imaging condition may not be selected. Also, if the area of ​​the second region is larger than the area of ​​the first region, the image data for the second imaging condition may be selected and the image data for the first imaging condition may not be selected.

[0251] According to the third embodiment described above, the following effects and advantages can be obtained. (1) Camera 1C is capable of taking images by changing the imaging conditions for each unit division of the imaging surface, and includes an image sensor 32a that generates first image data from a first region consisting of at least one unit division captured under first imaging conditions, and second image data from a second region consisting of at least one unit division captured under second imaging conditions different from the first imaging conditions. Camera 1C includes a plurality of selection units 322 that are provided corresponding to each unit division or each composite division having multiple unit divisions, and that select or not select image data from the corresponding unit division or unit divisions within the corresponding composite division. The image sensor 32a is provided on a back-illuminated imaging chip 111. The plurality of selection units 322 are provided on an image processing chip 114. This allows the data selection process for image data to be processed in parallel by multiple selection units 322, thereby reducing the processing load on the selection units 322.

[0252] (2) The back-illuminated imaging chip 111 and the image processing chip 114 are stacked. This allows for easy connection between the image sensor 32a and the image processing unit 32c.

[0253] (3) Camera 1C includes a generation unit 323 that generates an image using selected image data selected by the selection unit 322. As a result, preprocessing by multiple selection units 322 is performed in parallel in a short time, thus shortening the time until the image is generated.

[0254] (4) Camera 1C is capable of taking images by changing the imaging conditions for each unit division of the imaging surface, and includes an image sensor 32a that generates first image data from a first region consisting of at least one unit division, in which the incident light image is captured under first imaging conditions via the imaging optical system, and second image data from a second region consisting of at least one unit division, in which the incident light image is captured under second imaging conditions different from the first imaging conditions. Camera 1C includes a plurality of selection units 322 that are provided corresponding to each unit division or each composite division having multiple unit divisions, and that select or not select image data from the corresponding unit division or unit division within the corresponding composite division. Camera 1C includes an AF calculation unit 34d that detects information for moving the imaging optical system based on the selected selected image data. The image sensor 32a is provided on a back-illuminated imaging chip 111. The plurality of correction units 322 are provided on an image processing chip 114. This allows the data selection process for image data to be processed in parallel by multiple selection units 322, thereby reducing the processing load on the selection units 322. Furthermore, since the pre-processing by multiple selection units 322 is performed in parallel, the time until the start of focus detection processing in the AF calculation unit 34d can be shortened, contributing to faster focus detection processing.

[0255] (5) Camera 1C is capable of taking images by changing the imaging conditions for each unit division of the imaging surface, and includes an image sensor 32a that generates first image data from a first region consisting of at least one unit division, in which an incident subject image is captured with first imaging conditions via the imaging optical system, and second image data from a second region consisting of at least one unit division, in which an incident subject image is captured with second imaging conditions different from the first imaging conditions. Camera 1C includes a plurality of selection units 322, which are provided corresponding to each unit division or each composite division having multiple unit divisions, and which select or not select image data from the corresponding unit division or unit divisions within the corresponding composite division. Camera 1C includes an object detection unit 34a that detects an object from the subject image based on the selected image data. The image sensor 32a is provided on a back-illuminated imaging chip 111. The plurality of correction units 322 are provided on an image processing chip 114. As a result, the data selection process for image data can be processed in parallel by multiple selection units 322, reducing the processing load on the selection units 322. Furthermore, since the pre-processing by multiple selection units 322 is performed in parallel in a short time, the time until the start of subject detection processing in the object detection unit 34a can be shortened, contributing to faster subject detection processing.

[0256] (6) Camera 1C is capable of taking images by changing the imaging conditions for each unit division of the imaging surface, and includes an image sensor 32a that generates first image data from a first region consisting of at least one unit division, in which the incident light image is captured under first imaging conditions via the imaging optical system, and second image data from a second region consisting of at least one unit division, in which the incident light image is captured under second imaging conditions different from the first imaging conditions. Camera 1C includes a plurality of selection units 322, which are provided corresponding to each unit division or each composite division having multiple unit divisions, and which select or not select image data from the corresponding unit division or unit division within the corresponding composite division. Camera 1C includes a setting unit 34b that sets the shooting conditions based on the selected image data. The image sensor 32a is provided on the back-illuminated imaging chip 111. The plurality of correction units 322 are provided on the image processing chip 114. As a result, since the data selection process of the image data can be performed in parallel by a plurality of selection units 322, the processing load on the selection unit 322 can be reduced, and since the preprocessing by the plurality of selection units 322 is performed in a short time by parallel processing, the time until the start of the imaging condition setting process in the setting unit 34b can be shortened, contributing to the speeding up of the imaging condition setting process.

[0257] (Modification example of the third embodiment) The following modifications are also within the scope of the present invention, and it is also possible to combine one or more of the modification examples with the above-described embodiments. (Modification example 10) As shown in FIGS. 16(a) to 16(c) in the modification example 1 of the first and second embodiments, the processing of the first image data and the second image data when the first region and the second region are arranged on the imaging surface of the imaging device 32a will be described. Also in this modification example, as in the modification example 1, in any of FIGS. 16(a) to 16(c), the first image based on the image signal read from the first region and the second image based on the image signal read from the second region are respectively generated by the pixel signals read from the imaging device 32a that has performed imaging of one frame. Also in this modification example, as in the modification example 1, the control unit 34 uses the first image for display and the second image for detection. It is assumed that a first imaging condition is set for the first region that captures the first image, and a second imaging condition different from the first imaging condition is set for the second region that captures the second image.

[0258] 1. As an example, the case where the first imaging condition is the same throughout the first region of the imaging screen and the second imaging condition is also the same throughout the second region of the imaging screen will be described with reference to FIG. 26. FIG. 26 is a diagram schematically showing the processing of the first image data and the second image data.

[0259] Each pixel in the first region 141 outputs first image data captured under first imaging conditions, and each pixel in the second region 142 outputs second image data and second signal data captured under second imaging conditions. The first image data from the first region 141 is output to the first selection unit 151. Similarly, the second image data and second signal data from the second region 142 are output to the second selection unit 152.

[0260] In this example, since the first imaging condition is the same throughout the entire first region of the imaging screen, the first selection unit 151 selects all first image data from each pixel in the first region. Similarly, since the second imaging condition is the same throughout the entire second region of the imaging screen, the second selection unit 152 selects all second image data from each pixel in the second region. However, because the first and second imaging conditions are different, the second selection unit 152 does not select the second image data as data for image processing of the image data from the first region. Furthermore, the second selection unit 152 receives information 181 about the first imaging conditions from, for example, the first selection unit 151.

[0261] The generation unit 323 performs image processing such as pixel defect correction, color interpolation, edge enhancement, and noise reduction based on the first image data from the first selection unit 151, and outputs the image data after image processing. The object detection unit 34a performs a process to detect subject elements based on the second image data from the second selection unit 152 and outputs the detection result. The setting unit 34b calculates imaging conditions such as exposure calculation based on the second image data from the second selection unit 152, and based on the calculation results, divides the image captured by the imaging unit 32 into multiple regions including the detected subject elements, and resets the imaging conditions for the multiple regions. The AF calculation unit 34d performs focus detection processing based on the second signal data from the second selection unit 152, and outputs a drive signal to move the focus lens of the imaging optical system 31 to the focus position based on the calculation result.

[0262] 2. As another example, a case in which the first imaging conditions differ depending on the region of the imaging screen, that is, the first imaging conditions differ depending on a subregion within the first region, and the second imaging conditions are the same for the entire second region of the imaging screen, will be explained with reference to Figure 27. Figure 27 is a schematic diagram showing the processing of the first image data, the second image data, and the second signal data.

[0263] Each pixel in the first region 141 outputs first image data captured under first imaging conditions, and each pixel in the second region 142 outputs second image data and second signal data captured under second imaging conditions. The first image data from the first region 141 is output to the first selection unit 151. Similarly, the second image data from the second region 142 is output to the second selection unit 152.

[0264] As described above, in this example, the first imaging condition differs depending on the region of the imaging screen. That is, the first imaging condition differs depending on the subregion within the first region. The first selection unit 151 selects only the first image data for a certain imaging condition from the first image data from each pixel of the first region, and does not select the first image data for other imaging conditions. Also, since the second imaging condition is the same for the entire second region of the imaging screen, the second selection unit 152 selects all the second image data from each pixel of the second region. Note that because the first and second imaging conditions are different, the second selection unit 152 does not select the second image data as data for image processing of the image data of the first region. Furthermore, the second selection unit 152 receives information 181 about the first imaging conditions from, for example, the first selection unit 151.

[0265] The generation unit 323 performs image processing such as pixel defect correction, color interpolation, edge enhancement, and noise reduction based on a portion of the first image data selected by the first selection unit 151, and outputs the image data after image processing. The object detection unit 34a performs a process to detect subject elements based on the second image data from the second selection unit 152 and outputs the detection result. The setting unit 34b calculates imaging conditions such as exposure calculation based on the second image data from the second selection unit 152, and based on the calculation results, divides the image captured by the imaging unit 32 into multiple regions including the detected subject elements, and resets the imaging conditions for the multiple regions. The AF calculation unit 34d performs focus detection processing based on the second signal data from the second selection unit 152, and outputs a drive signal to move the focus lens of the imaging optical system 31 to the focus position based on the calculation result.

[0266] 3. As another example, we will explain the case where the first imaging conditions are the same throughout the entire first region of the imaging screen, and the second imaging conditions differ depending on the region of the imaging screen, with reference to Figure 28. Figure 28 is a schematic diagram illustrating the processing of the first image data and the second image data.

[0267] From each pixel in the first region 141, first image data captured under the same first imaging conditions across the entire first region of the imaging screen is output, and from each pixel in the second region 142, second image data captured under different second imaging conditions depending on the region of the imaging screen is output. The first image data from the first region 141 is output to the first selection unit 151. Similarly, the second image data and second signal data from the second region 142 are output to the second selection unit 152.

[0268] In this example, since the first imaging condition is the same throughout the entire first region of the imaging screen, the first selection unit 151 selects all first image data from each pixel in the first region. Also, the second imaging condition differs depending on the region of the imaging screen. That is, the second imaging condition differs depending on the subregion within the second region. The second selection unit 152 selects only the second image data for a certain imaging condition from the second image data from each pixel in the second region, and does not select the second image data for other imaging conditions. Note that because the first and second imaging conditions are different, the second selection unit 152 does not select the second image data as data for image processing of the image data of the first region. Furthermore, the second selection unit 152 receives information 181 about the first imaging conditions from, for example, the first selection unit 151.

[0269] The generation unit 323 performs image processing such as pixel defect correction, color interpolation, edge enhancement, and noise reduction based on the first image data from the first selection unit 151, and outputs the image data after image processing. The object detection unit 34a performs a process to detect subject elements based on a portion of the second image data selected by the second selection unit 152, and outputs the detection result. The setting unit 34b calculates imaging conditions such as exposure calculation based on a portion of the second image data selected by the second selection unit 152, and based on the calculation results, divides the image captured by the imaging unit 32 into multiple regions including the detected subject elements, and resets the imaging conditions for the multiple regions. The AF calculation unit 34d performs focus detection processing based on some of the second signal data selected by the second selection unit 152, and outputs a drive signal to move the focus lens of the imaging optical system 31 to the focus position based on the calculation result.

[0270] 4. Furthermore, as another example, a case in which the first imaging conditions differ depending on the region of the imaging screen and the second imaging conditions also differ depending on the region of the imaging screen will be explained with reference to Figure 29. Figure 29 is a schematic diagram showing the processing of the first image data, the second image data, and the second signal data.

[0271] Each pixel in the first region 141 outputs first image data captured under first imaging conditions that differ depending on the region of the imaging screen, and each pixel in the second region 142 outputs second image data and second signal data captured under second imaging conditions that also differ depending on the region of the imaging screen. The first image data from the first region 141 is output to the first selection unit 151. Similarly, the second image data and second signal data from the second region 142 are output to the second selection unit 152.

[0272] As described above, in this example, the first imaging condition differs depending on the region of the imaging screen. That is, the first imaging condition differs depending on the subregion within the first region. The first selection unit 151 selects only the first image data for a certain imaging condition from the first image data from each pixel of the first region, and does not select the first image data for other imaging conditions. Furthermore, the second imaging condition differs depending on the region of the imaging screen. That is, the second imaging condition differs depending on the subregion within the second region. The second selection unit 152 selects only the second image data for a certain imaging condition from the second image data from each pixel of the second region, and does not select the second image data for other imaging conditions. Note that because the first and second imaging conditions are different, the second selection unit 152 does not select the second image data as data for image processing of the image data of the first region. Furthermore, the second selection unit 152 receives information 181 about the first imaging conditions from, for example, the first selection unit 151.

[0273] The generation unit 323 performs image processing such as pixel defect correction, color interpolation, edge enhancement, and noise reduction based on a portion of the first image data selected by the first selection unit 151, and outputs the image data after image processing. The object detection unit 34a performs a process to detect subject elements based on a portion of the second image data selected by the second selection unit 152, and outputs the detection result. The setting unit 34b calculates imaging conditions such as exposure calculation based on a portion of the second image data selected by the second selection unit 152, and based on the calculation results, divides the image captured by the imaging unit 32 into multiple regions including the detected subject elements, and resets the imaging conditions for the multiple regions. The AF calculation unit 34d performs focus detection processing based on some of the second signal data selected by the second selection unit 152, and outputs a drive signal to move the focus lens of the imaging optical system 31 to the focus position based on the calculation result.

[0274] (Variation 11) In the third embodiment described above, one of the selection units 322 corresponds to one of the blocks 111a (unit divisions). However, one of the selection units 322 may correspond to one of the composite blocks (composite divisions) having multiple blocks 111a (unit divisions). In this case, the selection unit 322 sequentially performs data selection processing on image data from pixels belonging to the multiple blocks 111a included in the composite block. Even if multiple selection units 322 are provided corresponding to each composite block having multiple blocks 111a, the data selection processing of image data can be processed in parallel by the multiple selection units 322, thereby reducing the processing load on the selection units 322 and enabling the generation of appropriate images in a short time from image data generated in regions with different imaging conditions.

[0275] (Example 12) In the third embodiment described above, the generation unit 323 is located inside the imaging unit 32A. However, the generation unit 323 may be located outside the imaging unit 32A. Even if the generation unit 323 is located outside the imaging unit 32A, it will produce the same effects and advantages as described above.

[0276] (Example 13) In the third embodiment described above, the stacked image sensor 100A further includes an image processing chip 114 that performs the pre-processing and image processing described above, in addition to the back-illuminated imaging chip 111, the signal processing chip 112, and the memory chip 113. However, the stacked image sensor 100A may not have an image processing chip 114, and the signal processing chip 112 may have an image processing unit 32c. Furthermore, the embodiments and variations described above may be combined in any way.

[0277] Although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention.

[0278] The embodiments and variations described above also include the following devices. (1) An imaging device comprising: an image sensor having an imaging area for imaging a subject; a setting unit for setting imaging conditions for the imaging area; a selection unit for selecting pixels to be used for interpolation from pixels included in the imaging area; and a generation unit for generating an image of the subject imaged in the imaging area using a signal interpolated by signals output from the pixels selected by the selection unit, wherein at least some of the pixels selected by the selection unit differ depending on the imaging conditions set by the setting unit. (2) In an imaging device as described in (1), the image sensor has a first imaging area for imaging a subject and a second imaging area for imaging a subject, the setting unit sets imaging conditions for the first imaging area and imaging conditions for the second imaging area, the selection unit differentiates at least some of the pixels selected from the pixels included in the first imaging area and the pixels included in the second imaging area to be used for interpolation of the pixels included in the first imaging area according to the imaging conditions for the second imaging area set by the setting unit, and the generation unit generates an image of the subject imaged in the first imaging area using a signal interpolated by the signals output from the pixels selected by the selection unit. (3) In an imaging device like the one in (2), the selection unit makes at least some of the pixels selected for interpolation of pixels included in the first imaging region different depending on the imaging conditions of the first imaging region and the imaging conditions of the second imaging region set by the setting unit. (4) In an imaging device such as (2) or (3), the selection unit selects pixels to be used for interpolation from at least one of the imaging regions, the first imaging region and the second imaging region. (5) In an imaging device like the one in (4), the selection unit selects pixels of the second imaging region to be used for interpolation based on the imaging conditions of the second imaging region set by the setting unit. (6) In an imaging device as described in (2) to (5), the selection unit selects pixels included in the second imaging region when the setting unit sets the first imaging conditions for the first imaging region and the second imaging region. (7) In an imaging device like the one in (6), the selection unit selects pixels of the second imaging region to be used for interpolation based on exposure values ​​determined by the imaging conditions of the second imaging region set by the setting unit. In an imaging device like the one described in (8)(7), the selection unit selects pixels in the second imaging region to be used for interpolation based on the exposure values ​​of the imaging conditions of the first imaging region set by the setting unit and the exposure values ​​of the imaging conditions of the second imaging region. (9)(8) In an imaging device as described above, the selection unit selects a pixel in the second imaging region to be used for interpolation when the difference between the number of exposure stops based on the imaging conditions of the first imaging region set by the setting unit and the number of exposure stops based on the imaging conditions of the second imaging region is 0.3 stops or less. In imaging devices such as those described in (10)(2) to (9), the selection unit selects pixels included in the first imaging region when the setting unit sets the first imaging condition for the first imaging region and sets the second imaging condition for the second imaging region. In an imaging device like the one described in (11)(10), when the setting unit sets a first imaging condition for the first imaging area and a second imaging condition for the second imaging area, the selection unit selects pixels included in the first imaging area without selecting pixels in the second imaging area. In an imaging device as described in (12)(2)~(11), the selection unit selects a third pixel in the first imaging region as a pixel for interpolating the first pixel in the first imaging region, wherein the distance to the first pixel is longer than the distance between the first pixel and the second pixel in the second imaging region. In imaging devices such as (13)(2)~(12), the number of pixels selected by the selection unit differs depending on the imaging conditions set in the second imaging area by the setting unit. In an imaging device like the one described in (14)(13), when the setting unit sets the first imaging condition for the first imaging area and the second imaging condition for the second imaging area, the selection unit selects fewer pixels than when the first imaging condition is set for both the first and second imaging areas. (15) An imaging device comprising: an image sensor having a first imaging area set to image a subject under first imaging conditions; a second imaging area set to image a subject under second imaging conditions different from the first imaging conditions; and a third imaging area set to image a subject under third imaging conditions different from the second imaging conditions; a selection unit that selects pixels from among the pixels in the second imaging area and the pixels in the third imaging area to be used for interpolation of the pixels in the first imaging area; and a generation unit that generates an image of a subject imaged in the first imaging area using a signal interpolated by the signals output from the pixels selected by the selection unit. (16) An imaging device comprising: an image sensor having a first imaging area set to image a subject under first imaging conditions and a second imaging area set to image a subject under second imaging conditions different from the first imaging conditions; a selection unit that selects pixels from among the pixels included in the first imaging area and the pixels included in the second imaging area to be used for interpolation of the pixels included in the first imaging area; and a generation unit that generates an image of a subject imaged in the first imaging area using a signal interpolated by the signals output from the pixels selected by the selection unit. (17) An imaging device comprising: an image sensor having a first imaging area for imaging a subject, a second imaging area for imaging a subject, and a third imaging area for imaging a subject; a setting unit that sets the imaging conditions of the first imaging area to a first imaging condition, sets the imaging conditions of the second imaging area to a second imaging condition different from the first imaging condition, and sets the imaging conditions of the third imaging area to a third imaging condition in which the difference from the first imaging condition is smaller than the difference between the first imaging condition and the second imaging condition; a selection unit that selects pixels from among the pixels included in the first imaging area, the pixels included in the second imaging area, and the pixels included in the third imaging area to be used for interpolation of the pixels included in the first imaging area; and a generation unit that generates an image of a subject imaged in the first imaging area using a signal interpolated by the signals output from the pixels selected by the selection unit. (18) An imaging device comprising: an image sensor having a first imaging area for imaging a subject, a second imaging area for imaging a subject, and a third imaging area for imaging a subject, the distance to the first imaging area being longer than the distance between the first imaging area and the second imaging area; a setting unit for setting the imaging conditions of the second imaging area to different imaging conditions from those of the first imaging area; a selection unit for selecting pixels from among the pixels included in the first imaging area, the pixels included in the second imaging area, and the pixels included in the third imaging area to be used for interpolation of the pixels included in the first imaging area; and a generation unit for generating an image of a subject imaged in the first imaging area using a signal interpolated by the signals output from the pixels selected by the selection unit. (19) An imaging device comprising: an image sensor having an imaging area for imaging a subject; a setting unit for setting imaging conditions for the imaging area; and a generation unit for generating an image of a subject imaged in the imaging area using a signal interpolated by signals output from pixels included in the imaging area, which are selected as pixels to be used for interpolation, wherein at least some of the pixels selected differ depending on the imaging conditions set by the setting unit. (20) An imaging device comprising: an image sensor having a first imaging area set to image a subject under first imaging conditions and a second imaging area set to image a subject under second imaging conditions different from the first imaging conditions; and a generation unit that generates an image of a subject imaged in the first imaging area using a signal interpolated by signals output from pixels selected from among the pixels in the first imaging area and the pixels in the second imaging area to be used for interpolating the pixels in the first imaging area. (21) An imaging device comprising: an image sensor having an imaging area for imaging a subject; a setting unit for setting imaging conditions for the imaging area; and a generation unit for generating an image of a subject imaged in the imaging area using a signal from which noise has been reduced by a signal output from a pixel that outputs a noise reduction signal, selected from the pixels included in the imaging area, wherein at least some of the pixels selected differ depending on the imaging conditions set by the setting unit. (22) An imaging device comprising: an image sensor having a first imaging area set to image a subject under first imaging conditions; a second imaging area set to image a subject under second imaging conditions different from the first imaging conditions; and a third imaging area set to image a subject under third imaging conditions different from the second imaging conditions; a selection unit that selects pixels included in the first imaging area from among the pixels included in the second imaging area and the pixels included in the third imaging area for use in noise reduction; and a generation unit that generates an image of a subject imaged in the first imaging area using a signal from which noise has been reduced by the signal output from the selected pixel from among the pixels included in the second imaging area and the pixels included in the third imaging area, as a pixel to output a signal for use in reducing the noise of the signal of the pixels included in the first imaging area. (23) An imaging device comprising: an image sensor having a first imaging area set to image a subject under first imaging conditions and a second imaging area set to image a subject under second imaging conditions different from the first imaging conditions; and a generation unit that generates an image of a subject imaged in the first imaging area using a signal interpolated by signals output from pixels selected from among the pixels in the first imaging area and the pixels in the second imaging area, which are used to output signals to reduce noise in the pixels included in the first imaging area. (24) An imaging device comprising: an image sensor having an imaging area for imaging a subject; a setting unit for setting imaging conditions for the imaging area; and a generation unit for generating an image of the subject imaged in the imaging area using signals processed by signals output from pixels selected as pixels for image processing, wherein at least some of the pixels selected are different depending on the imaging conditions set by the setting unit. (25) An image processing apparatus comprising: a selection unit that selects a signal to be used for interpolation from signals output from pixels included in the imaging area of ​​an image sensor; and a generation unit that generates an image of a subject captured in the imaging area using the signal interpolated by the signal selected by the selection unit, wherein at least some of the pixels selected by the selection unit differ depending on the imaging conditions set in the imaging area. (26) An image processing apparatus comprising: a selection unit that selects a signal for reducing noise from signals output from pixels included in the imaging area of ​​an image sensor; and a generation unit that generates an image of a subject captured in the imaging area using the signal from which noise has been reduced by the signal selected by the selection unit, wherein at least some of the pixels selected by the selection unit differ depending on the imaging conditions set in the imaging area. (27) An image processing apparatus comprising: a selection unit that selects a signal to be used for interpolation from signals output from pixels included in the imaging area of ​​an image sensor; and a display unit that displays an image of a subject captured in the imaging area, which is generated using the signal interpolated by the signal selected by the selection unit, wherein at least some of the pixels to be selected differ depending on the imaging conditions set in the imaging area. (28) An image processing apparatus comprising: a selection unit that selects a signal for reducing noise from signals output from pixels included in the imaging area of ​​an image sensor; and a display unit that displays an image of a subject captured in the imaging area, which is generated using the signal from which noise has been reduced by the signal selected by the selection unit, wherein at least some of the pixels selected by the selection unit differ depending on the imaging conditions set in the imaging area.

[0279] Furthermore, the embodiments and variations described above also include the following devices. (1) An imaging device comprising: an image sensor having a first region for capturing incident light under first imaging conditions and outputting first image data; a second region for capturing incident light under second imaging conditions different from the first imaging conditions and outputting second image data; and a third region for capturing incident light under first imaging conditions and outputting third image data; and an image processing unit for generating an image based on first image data obtained by image processing using at least one of the first image data and the third image data, and the second image data. (2) In an imaging device as described in (1), the image processing unit processes the first image data using the third image data without using the second image data. (3) In an imaging device like the one in (2), the first image data is output from a pixel at a first position in the first region, the second image data is output from a pixel at a second position in the second region, and the third image data is output from a pixel at a third position in the first region, and the distance from the first position to the third position is longer than the distance from the first position to the second position. (4) In an imaging device as described in (1) to (3), the image sensor has a fourth region that captures incident light under the second imaging conditions and outputs fourth image data, and the image processing unit generates an image using a second image data obtained by image processing using at least one of the second image data and the fourth image data, and the first image data obtained by image processing. (5) An imaging device comprising: an image sensor having a first region for capturing incident light under first imaging conditions and outputting first image data, and a second region for capturing incident light under second imaging conditions different from the first imaging conditions and outputting second image data; and an image processing unit for generating an image based on the first image data obtained by image processing using the first image data and the second image data. In an imaging device like the one described in (6)(5), the image processing unit processes the first image data using the first image data without using the second image data. In an imaging device like the one described in (7)(6), the image processing unit generates an image using the second image data processed using the second image data and the first image data processed using the second image data. (8) In an imaging device as described in (1) to (7), the first region outputs a plurality of first image data, and the image processing unit processes the first image data to be processed from the plurality of first image data using the other first image data from the plurality of first image data. (9)(8) In an imaging device as described above, the first image data is output from a pixel at a first position in the first region, the second image data is output from a pixel at a second position in the second region, and the other first image data is output from a pixel at a third position in the first region, and the distance from the first position to the third position is longer than the distance from the first position to the second position. (10) In imaging devices such as those described in (1) to (9), the first imaging condition and the second imaging condition include at least an accumulation time or ISO sensitivity. (11) In an imaging device as described in (1) to (10), the image processing unit processes the first image data by performing at least pixel defect correction processing, color interpolation processing, edge enhancement processing, or noise reduction processing. (12) In an imaging device as described in (1) to (11), the image processing unit includes a selection unit that selects image data to be processed from the first image data. In an imaging device like (13)(12), the image sensor is capable of taking images by changing the imaging conditions for each unit region of the imaging surface, and the selection unit is provided corresponding to each unit region or each composite region having multiple unit regions, and selects image data from the corresponding unit region or the unit region within the corresponding composite region. In an imaging device like the one described in (14)(13), the image sensor is provided on a first semiconductor substrate, and the selection unit is provided on a second semiconductor substrate. In an imaging device like the one described in (15)(14), the first semiconductor substrate and the second semiconductor substrate are stacked. (16) A display device having a display unit that displays an image generated based on: first image data output by imaging light incident on a first region of an imaging unit under first imaging conditions; second image data output by imaging light incident on a second region of the imaging unit under second imaging conditions different from the first imaging conditions; and third image data output by imaging light incident on a third region of the imaging unit under first imaging conditions, wherein the display unit displays the image generated by the image processing unit, and the display unit displays the image generated by the image processing unit. (17) A display device comprising: (17) an image processing unit that generates an image based on first image data obtained by image processing first image data output by image capturing light incident on a first region of an imaging unit under first imaging conditions; second image data output by image capturing light incident on a second region of the imaging unit under second imaging conditions different from the first imaging conditions; and a display unit that displays the image generated by the image processing unit. (18) An image processing apparatus comprising an image processing unit that generates an image based on: first image data output by imaging light incident on a first region of an imaging unit under first imaging conditions; second image data output by imaging light incident on a second region of the imaging unit under second imaging conditions different from the first imaging conditions; and third image data output by imaging light incident on a third region of the imaging unit under first imaging conditions, wherein the image processing unit generates an image based on first image data processed using at least one of the first image data and the third image data, and the second image data. (19) An image processing apparatus comprising an image processing unit that generates an image based on first image data obtained by image processing using first image data output by image capturing light incident on a first region of an imaging unit under first imaging conditions, and second image data output by image capturing light incident on a second region of the imaging unit under second imaging conditions different from the first imaging conditions.

[0280] The disclosures of the following priority application are incorporated herein by reference. Japanese Patent Application No. 195288 of 2015 (filed September 30, 2015) [Explanation of Symbols]

[0281] 1.1C...Camera 1B…Imaging System 32…Imaging Unit 32a, 100... Image sensor 33…Image Processing Unit 33a, 321... Input section 33b, 322… Selection section 33c,323...Generation part 34…Control Unit 34a...Object detection unit 34b…Area division part 34d...Imaging Control Unit 35...Display section 90...Prescribed range 1001...Imaging device 1002...Display device P...Featured Pixel

Claims

[Claim 1] An image sensor having a first imaging region set to image a subject under first imaging conditions, a second imaging region set to image a subject under second imaging conditions different from the first imaging conditions, and a third imaging region set to image a subject under first imaging conditions, A selection unit that selects pixels from among the pixels included in the second imaging region and the pixels included in the third imaging region to be used for reducing noise in the signal from the pixels included in the first imaging region, It has, The second imaging region is an area adjacent to the first imaging region. The third imaging region is a region different from the first imaging region and is adjacent to at least one of the first imaging region and the second imaging region. Imaging device.

Images