Imaging device

Abstract

In imaging devices equipped with image processing technology that generates images from signals from an image sensor, there was a demand for improved image quality. [Solution] An imaging unit having a first imaging region that is imaged under first imaging conditions and a second imaging region that is imaged under second imaging conditions different from the first imaging conditions, A generation unit that generates an image of a subject captured in the first imaging area using image data of the subject captured in the second imaging area, It is equipped with.

Description

Technical Field

[0001] The present invention relates to an imaging device.

Background Art

[0002] An imaging device equipped with an image processing technology that generates an image from a signal from an imaging element is known (see Patent Document 1). Conventionally, improvement in image quality of an image has been required.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] According to a first aspect, an imaging device includes an imaging unit having a first imaging region that images under a first imaging condition and a second imaging region that images under a second imaging condition different from the first imaging condition, and a generation unit that generates an image of a subject imaged in the first imaging region based on image data of the subject imaged in the second imaging region.

Brief Description of the Drawings

[0005] [Figure 1] It is a block diagram illustrating the configuration of a camera according to a first embodiment. [Figure 2] It is a cross-sectional view of a stacked imaging element. [Figure 3] It is a diagram for explaining a pixel array and a unit region of an imaging chip. [Figure 4] It is a diagram for explaining a circuit in a unit region. [Figure 5] It is a diagram schematically showing an image of a subject imaged on an imaging element of a camera. [Figure 6] It is a diagram illustrating a setting screen for imaging conditions. [Figure 7]Figure 7(a) is an example of a predetermined range in the live view image, and Figure 7(b) is a magnified view of the predetermined range. [Figure 8] Figure 8 is an example of image data corresponding to Figure 7(b). [Figure 9] Figure 9(a) is an example of a region of interest in a live view image, and Figure 9(b) is a magnified view of the pixel of interest and the reference pixel Pr. [Figure 10] Figure 10(a) illustrates the sequence of photoelectric conversion signals output from a pixel, Figure 10(b) illustrates the interpolation of the G color component image data, and Figure 10(c) illustrates the image data of the G color component after interpolation. [Figure 11] Figure 11(a) shows the R color component image data extracted from Figure 10(a), Figure 11(b) is a diagram illustrating the interpolation of the chrominance component Cr, and Figure 11(c) is a diagram illustrating the interpolation of the chrominance component Cr in image data. [Figure 12] Figure 12(a) shows the image data of the B color component extracted from Figure 10(a), Figure 12(b) is a diagram illustrating the interpolation of the color difference component Cb, and Figure 12(c) is a diagram illustrating the interpolation of the image data of the color difference component Cb. [Figure 13] This diagram illustrates the position of focus detection pixels on the imaging surface. [Figure 14] This is a magnified view of a portion of the focus detection pixel line. [Figure 15] This is a magnified view of the focus point. [Figure 16] Figure 16(a) is an example of a template image representing the object to be detected, and Figure 16(b) is an example of a live view image and search area. [Figure 17] This flowchart explains the process of setting imaging conditions for each region and then performing the imaging. [Figure 18] Figures 18(a) to 18(c) illustrate the arrangement of the first imaging region and the second imaging region on the imaging surface of the image sensor. [Figure 19] This is a block diagram illustrating the configuration of the imaging system according to modified example 11. [Figure 20]This is a diagram for explaining the supply of a program to a mobile device. [Figure 21] This is a block diagram illustrating the configuration of a camera according to a second embodiment. [Figure 22] This is a diagram schematically showing the correspondence between each block in the second embodiment and a plurality of correction units. [Figure 23] This is a cross-sectional view of a stacked image sensor. [Figure 24] This is a diagram schematically showing the processing of first image data and second image data related to image processing. [Figure 25] This is a diagram schematically showing the processing of first image data and second image data related to focus detection processing. [Figure 26] This is a diagram schematically showing the processing of first image data and second image data related to subject detection processing. [Figure 27] This is a diagram schematically showing the processing of first image data and second image data related to setting imaging conditions such as exposure calculation processing. [Figure 28] This is a diagram schematically showing the processing of first image data and second image data according to Modification 13. [Figure 29] FIG. 29(a) is a diagram illustrating a predetermined range in a live view image, and FIG. 29(b) is an enlarged view of the predetermined range. [Figure 30] This is a diagram illustrating image data corresponding to FIG. 29(b). [Figure 31] This is an enlarged view of a boundary block.

Embodiments for Carrying Out the Invention

[0006] ---First Embodiment--- As an example of an electronic device equipped with an image processing device according to the first embodiment, a digital camera will be used as an example. Camera 1 (Figure 1) is configured to perform imaging under different conditions for each region of the imaging surface of the image sensor 32a. The image processing unit 33 performs appropriate processing for each region with different imaging conditions. Details of such camera 1 will be explained with reference to the drawings.

[0007] <Camera Description> Figure 1 is a block diagram illustrating the configuration of camera 1 according to the first embodiment. In Figure 1, camera 1 includes an imaging optical system 31, an imaging unit 32, an image processing unit 33, a control unit 34, a display unit 35, an operating member 36, and a recording unit 37.

[0008] The imaging optical system 31 guides the light beam from the field of view to the imaging unit 32. The imaging unit 32 includes an image sensor 32a and a drive unit 32b, and converts the image of the subject formed by the imaging optical system 31 into photoelectric power. The imaging unit 32 can image under the same conditions across the entire imaging surface of the image sensor 32a, or under different conditions for each region of the imaging surface of the image sensor 32a. Details of the imaging unit 32 will be described later. The drive unit 32b generates the drive signals necessary to control the charge accumulation of the image sensor 32a. Imaging instructions, such as the charge accumulation time for the imaging unit 32, are transmitted from the control unit 34 to the drive unit 32b.

[0009] The image processing unit 33 includes an input unit 33a, a correction unit 33b, and a generation unit 33c. Image data acquired by the imaging unit 32 is input to the input unit 33a. The correction unit 33b performs preprocessing to correct the input image data. Details of the preprocessing will be described later. The generation unit 33c performs image processing on the input image data and the preprocessed image data to generate an image. Image processing includes, for example, color interpolation, pixel defect correction, edge enhancement, noise reduction, white balance adjustment, gamma correction, display brightness adjustment, and saturation adjustment. Furthermore, the generation unit 33c generates an image to be displayed by the display unit 35.

[0010] The control unit 34, for example, is composed of a CPU and controls the overall operation of the camera 1. For example, the control unit 34 performs a predetermined exposure calculation based on the photoelectric conversion signal acquired by the imaging unit 32, determines the exposure conditions necessary for proper exposure, such as the charge accumulation time (exposure time) of the image sensor 32a, the aperture value of the imaging optical system 31, and the ISO sensitivity, and instructs the drive unit 32b. It also determines image processing conditions to adjust saturation, contrast, sharpness, etc., according to the imaging scene mode set in the camera 1 and the type of subject element detected, and instructs the image processing unit 33. The detection of subject elements will be described later.

[0011] The control unit 34 includes an object detection unit 34a, a setting unit 34b, an imaging control unit 34c, and a lens movement control unit 34d. These are implemented in software by the control unit 34 executing a program stored in a non-volatile memory (not shown), but they may also be configured using ASICs or the like.

[0012] The object detection unit 34a performs known object recognition processing to detect subject elements from the image data acquired by the imaging unit 32, such as people (people's faces), animals such as dogs and cats (animal faces), plants, vehicles such as bicycles, cars and trains, buildings, stationary objects, landscapes such as mountains and clouds, and predetermined specific objects. The setting unit 34b divides the image data acquired by the imaging unit 32 into multiple regions, each containing the subject elements detected as described above.

[0013] The setting unit 34b further sets imaging conditions for multiple regions. The imaging conditions include the above-mentioned exposure conditions (charge accumulation time, gain, ISO sensitivity, frame rate, etc.) and the above-mentioned image processing conditions (for example, white balance adjustment parameters, gamma correction curve, display brightness adjustment parameters, saturation adjustment parameters, etc.). It is possible to set the same imaging conditions for all of the multiple regions, or to set different imaging conditions for the multiple regions.

[0014] The imaging control unit 34c controls the imaging unit 32 (image sensor 32a) and the image processing unit 33 by applying imaging conditions set for each region by the setting unit 34b. This makes it possible to have the imaging unit 32 perform imaging with different exposure conditions for each of the multiple regions, and to have the image processing unit 33 perform image processing with different image processing conditions for each of the multiple regions. The number of pixels constituting a region can be any number; for example, it could be 1000 pixels or 1 pixel. Also, the number of pixels may differ between regions.

[0015] The lens movement control unit 34d controls the autofocus (AF) operation, which focuses on a corresponding subject at a predetermined position (called the focus point) on the imaging screen. When focus is achieved, the sharpness of the subject's image increases. In other words, the image produced by the imaging optical system 31 is adjusted by moving the focus lens of the imaging optical system 31 in the optical axis direction. Based on the calculation results, the lens movement control unit 34d sends a drive signal to the lens movement mechanism 31m of the imaging optical system 31 to move the focus lens of the imaging optical system 31 to the focus position, for example, a signal to adjust the image of the subject with the focus lens of the imaging optical system 31. In this way, the lens movement control unit 34d functions as a movement unit that moves the focus lens of the imaging optical system 31 in the optical axis direction based on the calculation results. The processing performed by the lens movement control unit 34d for AF operation is also called focus detection processing. Details of focus detection processing will be described later.

[0016] The display unit 35 displays images generated by the image processing unit 33, images that have been processed, and images read out by the recording unit 37. The display unit 35 also displays operation menu screens and setting screens for setting imaging conditions.

[0017] The operating components 36 consist of various operating components such as a release button and a menu button. The operating components 36 send operation signals corresponding to each operation to the control unit 34. The operating components 36 also include touch operation components provided on the display surface of the display unit 35.

[0018] The recording unit 37 records image data and other data on a recording medium, such as a memory card (not shown), in response to instructions from the control unit 34. The recording unit 37 also reads out image data recorded on the recording medium in response to instructions from the control unit 34.

[0019] <Explanation of stacked image sensors> As an example of the image sensor 32a described above, a stacked image sensor 100 will be explained. Figure 2 is a cross-sectional view of the image sensor 100. The image sensor 100 comprises an imaging chip 111, a signal processing chip 112, and a memory chip 113. The imaging chip 111 is stacked on the signal processing chip 112. The signal processing chip 112 is stacked on the memory chip 113. The imaging chip 111 and the signal processing chip 112, and the signal processing chip 112 and the memory chip 113 are electrically connected by connection parts 109. The connection parts 109 are, for example, bumps or electrodes. The imaging chip 111 captures a light image from a subject and generates image data. The imaging chip 111 outputs the image data from the imaging chip 111 to the signal processing chip 112. The signal processing chip 112 performs signal processing on the image data output from the imaging chip 111. The memory chip 113 has multiple memories and stores the image data. The image sensor 100 may consist of an imaging chip and a signal processing chip. If the image sensor 100 consists of an imaging chip and a signal processing chip, the storage unit for storing image data may be provided on the signal processing chip or separately from the image sensor 100.

[0020] As shown in Figure 2, incident light primarily enters in the positive Z-axis direction, indicated by the white arrow. Furthermore, as shown in the coordinate axes, the direction to the left of the paper perpendicular to the Z-axis is defined as the positive X-axis direction, and the direction towards the viewer, perpendicular to both the Z-axis and X-axis, is defined as the positive Y-axis direction. In the following figures, the coordinate axes are displayed using the coordinate axes in Figure 2 as a reference, so that the orientation of each figure is clear.

[0021] The imaging chip 111 is, for example, a CMOS image sensor. Specifically, the imaging chip 111 is a back-illuminated CMOS image sensor. The imaging chip 111 has a microlens layer 101, a color filter layer 102, a passivation layer 103, a semiconductor layer 106, and a wiring layer 108. The imaging chip 111 is arranged in the order of microlens layer 101, color filter layer 102, passivation layer 103, semiconductor layer 106, and wiring layer 108 in the Z-axis positive direction.

[0022] The microlens layer 101 has a plurality of microlenses L. The microlenses L focus the incident light onto the photoelectric conversion unit 104, which will be described later. The color filter layer 102 has a plurality of color filters F. The color filter layer 102 has multiple types of color filters F with different spectral characteristics. Specifically, the color filter layer 102 has a first filter (R) with spectral characteristics that mainly transmit red light, a second filter (Gb, Gr) with spectral characteristics that mainly transmit green light, and a third filter (B) with spectral characteristics that mainly transmit blue light. In the color filter layer 102, for example, the first filter, second filter and third filter are arranged in a Bayer array. The passivation layer 103 is composed of a nitride film or an oxide film and protects the semiconductor layer 106.

[0023] The semiconductor layer 106 has photoelectric conversion units 104 and readout circuits 105. The semiconductor layer 106 has a plurality of photoelectric conversion units 104 between a first surface 106a, which is the light incident surface, and a second surface 106b opposite to the first surface 106a. The semiconductor layer 106 has a plurality of photoelectric conversion units 104 arranged in the X-axis direction and the Y-axis direction. The photoelectric conversion unit 104 has a photoelectric conversion function that converts light into electric charge. The photoelectric conversion unit 104 also stores charge due to the photoelectric conversion signal. The photoelectric conversion unit 104 is, for example, a photodiode. The semiconductor layer 106 has readout circuits 105 on the second surface 106b side of the photoelectric conversion units 104. The semiconductor layer 106 has a plurality of readout circuits 105 arranged in the X-axis direction and the Y-axis direction. The readout circuit 105 is composed of multiple transistors and reads out image data generated by the charge photoelectrically converted by the photoelectric conversion unit 104 and outputs it to the wiring layer 108.

[0024] The wiring layer 108 has multiple metal layers. The metal layers are, for example, Al wiring, Cu wiring, etc. The wiring layer 108 outputs image data read by the read circuit 105. The image data is output from the wiring layer 108 to the signal processing chip 112 via the connection part 109.

[0025] The connection portion 109 may be provided for each photoelectric conversion unit 104. Alternatively, the connection portion 109 may be provided for each of multiple photoelectric conversion units 104. If the connection portion 109 is provided for each of multiple photoelectric conversion units 104, the pitch of the connection portion 109 may be greater than the pitch of the photoelectric conversion units 104. Furthermore, the connection portion 109 may be provided in the peripheral region of the area where the photoelectric conversion units 104 are located.

[0026] The signal processing chip 112 has multiple signal processing circuits. The signal processing circuits perform signal processing on image data output from the imaging chip 111. Examples of signal processing circuits include an amplifier circuit that amplifies the signal value of the image data, a correlated double sampling circuit that performs noise reduction processing on the image data, and an analog-to-digital (A / D) conversion circuit that converts an analog signal into a digital signal. A signal processing circuit may be provided for each photoelectric conversion unit 104.

[0027] Furthermore, a signal processing circuit may be provided for each of the multiple photoelectric conversion units 104. The signal processing chip 112 has multiple through electrodes 110. The through electrodes 110 are, for example, silicon through electrodes. The through electrodes 110 connect circuits (not shown) provided on the signal processing chip 112 to each other. The through electrodes 110 may also be provided in the peripheral area of ​​the imaging chip 111 and the memory chip 113. Note that some of the elements constituting the signal processing circuit may be provided on the imaging chip 111. For example, in the case of an analog / digital conversion circuit, a comparator that compares the input voltage and the reference voltage may be provided on the imaging chip 111, and circuits such as a counter circuit and a latch circuit may be provided on the signal processing chip 112.

[0028] The memory chip 113 has multiple storage units. Each storage unit stores image data that has been signal-processed by the signal processing chip 112. The storage units are, for example, volatile memory such as DRAM. A storage unit may be provided for each photoelectric conversion unit 104. Alternatively, multiple storage units may be provided for each photoelectric conversion unit 104. The image data stored in the storage units is output to a subsequent image processing unit.

[0029] Figure 3 illustrates the pixel arrangement and unit region 131 of the imaging chip 111. In particular, it shows the imaging chip 111 as observed from the back (imaging surface) side. The pixel region contains, for example, more than 20 million pixels arranged in a matrix. In the example in Figure 3, four adjacent pixels (2x2) form one unit region 131. The grid lines in the figure illustrate the concept that adjacent pixels are grouped together to form a unit region 131. The number of pixels forming a unit region 131 is not limited to this; it could be around 1000, for example, 32x32 pixels, or more or less, or even just one pixel.

[0030] As shown in the magnified view of the pixel region, the unit region 131 in Figure 3 contains a so-called Bayer array consisting of four pixels: green pixels Gb and Gr, blue pixel B, and red pixel R. The green pixels Gb and Gr are pixels that have a green filter as the color filter F and receive light in the green wavelength band of the incident light. Similarly, the blue pixel B is a pixel that has a blue filter as the color filter F and receives light in the blue wavelength band, and the red pixel R is a pixel that has a red filter as the color filter F and receives light in the red wavelength band.

[0031] In this embodiment, multiple blocks are defined such that each block contains at least one unit region 131. That is, the smallest unit of a block is one unit region 131. As described above, the smallest number of pixels that can form one unit region 131 is 1 pixel. Therefore, when a block is defined on a pixel-by-pixel basis, the smallest number of pixels that can define a block is 1 pixel. Each block can control the pixels contained within it using different control parameters. In each block, all unit regions 131 within that block, i.e., all pixels within that block, are controlled under the same imaging conditions. In other words, photoelectric conversion signals can be obtained with different imaging conditions for a group of pixels contained in one block and a group of pixels contained in another block. Examples of control parameters include frame rate, gain, decimation rate, number of addition rows or columns for adding the photoelectric conversion signals, charge accumulation time or number of accumulations, and the number of bits (word length) for digitization. The image sensor 100 can freely perform decimation not only in the row direction (the X-axis direction of the imaging chip 111) but also in the column direction (the Y-axis direction of the imaging chip 111). Furthermore, the control parameters may also be parameters used in image processing.

[0032] Figure 4 illustrates the circuit in a unit region 131. In the example in Figure 4, four adjacent 2x2 pixels form one unit region 131. As mentioned above, the number of pixels included in a unit region 131 is not limited to this; it may be 1000 pixels or more, or as few as 1 pixel. The two-dimensional positions of the unit region 131 are indicated by symbols A to D.

[0033] The reset transistors (RSTs) of the pixels included in the unit region 131 are configured to be individually switched on and off for each pixel. In Figure 4, a reset wire 300 is provided to switch the reset transistor of pixel A on and off, and a reset wire 310 is provided separately from the reset wire 300 to switch the reset transistor of pixel B on and off. Similarly, a reset wire 320 is provided separately from the reset wires 300 and 310 to switch the reset transistor of pixel C on and off. A dedicated reset wire 330 is also provided for other pixels D to switch their reset transistors on and off.

[0034] The transfer transistors (TX) of the pixels included in the unit region 131 are also configured to be individually switched on and off for each pixel. In Figure 4, separate transfer wirings 302 for switching the transfer transistor of pixel A, 312 for switching the transfer transistor of pixel B, and 322 for switching the transfer transistor of pixel C are provided. A dedicated transfer wiring 332 is also provided for switching the transfer transistors of other pixels D.

[0035] Furthermore, the selection transistors (SELs) of the pixels included in the unit region 131 are also configured to be individually switched on and off for each pixel. In Figure 4, separate selection wires 306 for switching the selection transistor of pixel A on and off, selection wires 316 for switching the selection transistor of pixel B on and off, and selection wires 326 for switching the selection transistor of pixel C on and off are provided. A dedicated selection wire 336 for switching the selection transistors of other pixels D is also provided.

[0036] The power supply wiring 304 is commonly connected to pixels A through D included in the unit region 131. Similarly, the output wiring 308 is commonly connected to pixels A through D included in the unit region 131. In addition, the power supply wiring 304 is commonly connected between multiple unit regions, but the output wiring 308 is provided individually for each unit region 131. The load current source 309 supplies current to the output wiring 308. The load current source 309 may be provided on the imaging chip 111 side or on the signal processing chip 112 side.

[0037] By individually switching the reset transistor and transfer transistor of the unit region 131 on and off, charge accumulation, including the charge accumulation start time, accumulation end time, and transfer timing, can be controlled for pixels A to D contained within the unit region 131. Furthermore, by individually switching the selection transistor of the unit region 131 on and off, the photoelectric conversion signals from each pixel A to D can be output via a common output wiring 308.

[0038] Here, a known rolling shutter method is used to control charge accumulation in a regular order for rows and columns for pixels A to D contained in the unit region 131. When pixels are selected row by row and then the column is specified using the rolling shutter method, the photoelectric conversion signals are output in the order "ABCD" in the example shown in Figure 4.

[0039] By configuring the circuit based on the unit region 131 in this way, the charge accumulation time can be controlled for each unit region 131. In other words, different photoelectric conversion signals with different frame rates can be output between the unit regions 131. Furthermore, by allowing charge accumulation (imaging) to be performed in some of the unit regions 131 in the imaging chip 111 while the unit regions 131 in other blocks are left idle, imaging can be performed only in predetermined blocks of the imaging chip 111, and the resulting photoelectric conversion signal can be output. Moreover, by switching the block in which charge accumulation (imaging) is performed between frames (the block targeted for accumulation control), sequential imaging can be performed in different blocks of the imaging chip 111, and the resulting photoelectric conversion signal can be output.

[0040] As described above, output wiring 308 is provided corresponding to each of the unit regions 131. Since the image sensor 100 has an imaging chip 111, a signal processing chip 112, and a memory chip 113 stacked on top of each other, by using electrical connections between the chips with connection parts 109 for these output wiring 308, the wiring can be routed without increasing the size of each chip in the planar direction.

[0041] <Block control of the image sensor> In this embodiment, imaging conditions can be set for each of the multiple blocks in the image sensor 32a. The imaging control unit 34c of the control unit 34 corresponds the multiple regions to the blocks and performs imaging according to the imaging conditions set for each region.

[0042] Figure 5 schematically shows the image of the subject formed on the image sensor 32a of camera 1. Before an imaging command is given, camera 1 converts the subject image into a live view image using photoelectric conversion. A live view image is a monitor image that is repeatedly captured at a predetermined frame rate (e.g., 60fps).

[0043] Before the setting unit 34b divides the area, the control unit 34 sets the same imaging conditions for the entire area of ​​the imaging chip 111 (i.e., the entire imaging screen). The same imaging conditions mean setting common imaging conditions for the entire imaging screen; for example, even if there is a variation of less than approximately 0.3 stops in the apex value, it is considered the same. The imaging conditions set uniformly for the entire area of ​​the imaging chip 111 are determined based on exposure conditions corresponding to the metered value of the subject's brightness, or exposure conditions manually set by the user.

[0044] In Figure 5, an image including a person 61a, a car 62a, a bag 63a, a mountain 64a, and clouds 65a and 66a is formed on the imaging surface of the imaging chip 111. Person 61a is holding the bag 63a in both hands. The car 62a is parked to the right and behind person 61a.

[0045] <Division of Regions> The control unit 34 divides the live view image screen into multiple regions based on the live view image as follows. First, the object detection unit 34a detects subject elements from the live view image. The detection of subject elements uses known subject recognition techniques. In the example in Figure 5, the object detection unit 34a detects a person 61a, a car 62a, a bag 63a, a mountain 64a, a cloud 65a, and a cloud 66a as subject elements.

[0046] Next, the setting unit 34b divides the live view image screen into regions containing the subject elements. In this embodiment, the region containing the person 61a is referred to as the first region 61, the region containing the car 62a as the second region 62, the region containing the bag 63a as the third region 63, the region containing the mountain 64a as the fourth region 64, the region containing the cloud 65a as the fifth region 65, and the region containing the cloud 66a as the sixth region 66.

[0047] <Setting imaging conditions for each block> When the control unit 34 divides the screen into multiple areas using the setting unit 34b, it displays a setting screen on the display unit 35, as illustrated in Figure 6. In Figure 6, the live view image 60a is displayed, and the imaging condition setting screen 70 is displayed to the right of the live view image 60a.

[0048] The settings screen 70 shows, from top to bottom, frame rate, shutter speed (TV), and gain (ISO) as examples of settings for imaging conditions. Frame rate is the number of frames of live view images or video recorded by camera 1 per second. Gain is the ISO sensitivity. In addition to the examples shown in Figure 6, additional settings for imaging conditions may be added as appropriate. If all settings do not fit on the settings screen 70, the settings can be displayed by scrolling up and down.

[0049] In this embodiment, the control unit 34 makes the area selected by the user from the areas divided by the setting unit 34b the target for setting (changing) the imaging conditions. For example, in a touch-operable camera 1, the user taps the display position of the main subject for which they want to set (change) the imaging conditions on the display surface of the display unit 35 on which the live view image 60a is displayed. When, for example, the display position of a person 61a is tapped, the control unit 34 makes the first region 61 including the person 61a the target area for setting (changing) the imaging conditions in the live view image 60a, and also highlights the outline of the first region 61.

[0050] In Figure 6, the first region 61, which is displayed with its outline emphasized (thickened, brightened, changed color, dashed, blinking, etc.), indicates the region that is subject to setting (changing) the imaging conditions. In the example in Figure 6, it is assumed that a live view image 60a with the outline of the first region 61 emphasized is displayed. In this case, the first region 61 is the target of setting (changing) the imaging conditions. For example, in a touch-operable camera 1, when the user taps the shutter speed (TV) display 71, the control unit 34 displays the current shutter speed setting value for the emphasized region (first region 61) on the screen (reference numeral 68). In the following explanation, the camera 1 will be described assuming touch operation, but the imaging conditions may also be set (changed) by operating buttons or other components of the operating member 36.

[0051] When the user taps the upper icon 71a or lower icon 71b for shutter speed (TV), the setting unit 34b increases or decreases the shutter speed display 68 from the current setting value according to the tap operation, and also sends an instruction to the imaging unit 32 (Figure 1) to change the imaging conditions of the unit area 131 (Figure 3) of the image sensor 32a corresponding to the highlighted area (first area 61) according to the tap operation. The confirmation icon 72 is an operation icon for confirming the set imaging conditions. The setting unit 34b also sets (changes) the frame rate and gain (ISO) in the same way as it sets (changes) the shutter speed (TV).

[0052] Although the setting unit 34b has been described as setting imaging conditions based on user operation, it is not limited to this. The setting unit 34b may also set imaging conditions based on the judgment of the control unit 34, without relying on user operation. For areas that are not highlighted (areas other than the first region 61), the set imaging conditions are maintained.

[0053] Instead of highlighting the outline of the area to be set (changed) in imaging conditions, the control unit 34 may display the entire target area brightly, increase the contrast of the entire target area, or make the entire target area blink. Alternatively, the target area may be enclosed in a frame. The frame surrounding the target area may be a double frame or a single frame, and the line type, color, brightness, and other display characteristics of the frame may be changed as appropriate. The control unit 34 may also display an arrow or other indicator near the target area to point to the area to be set in imaging conditions. The control unit 34 may also display areas other than the target area to be set (changed) in darkness, or display areas other than the target area with low contrast.

[0054] As explained above, after the imaging conditions for each region are set, when the release button (not shown) or the display (release icon) that instructs the start of imaging, which constitutes the operating member 36, is operated, the control unit 34 controls the imaging unit 32 to perform imaging according to the imaging conditions set for each of the divided regions. The image processing unit 33 then performs image processing on the image data acquired by the imaging unit 32. As described above, the image processing can be performed with different image processing conditions for each region.

[0055] After the image processing by the image processing unit 33, the recording unit 37, instructed by the control unit 34, records the processed image data onto a recording medium, such as a memory card (not shown). This completes the series of imaging processes.

[0056] <First Correction Process> The correction unit 33b of the image processing unit 33 performs a first correction process as needed, as one of the preprocessing steps performed before image processing, focus detection processing, subject detection (detection of subject elements) processing, and imaging condition setting processing.

[0057] As described above, in this embodiment, after the area of ​​the imaging screen is divided by the setting unit 34b, it is possible to set (change) imaging conditions for the area selected by the user or the area determined by the control unit 34. For example, let's assume that the divided regions are designated as the first region 61 to the sixth region 66 (see Figure 7(a)), and that different first to sixth imaging conditions are set for each of the first to sixth regions 61 to 66. In such a case, there exists a block that includes the boundary between the first to sixth regions 66. As described above, a block is the smallest unit on the image sensor 32a where imaging conditions can be set individually.

[0058] Figure 7(a) is an example of a predetermined range 80 in the live view image 60a that includes the boundary between the first region 61 and the fourth region 64. Figure 7(b) is an enlarged view of the predetermined range 80 in Figure 7(a). In Figure 7(b), the predetermined range 80 includes multiple blocks 81 to 89. In this example, blocks 81 and 84, which capture people, are included in the first region 61, and blocks 82, 85, and 87, which capture people and mountains, are also included in the first region 61. Therefore, the first imaging condition is set for blocks 81, 82, 84, 85, and 87. On the other hand, blocks 83, 86, 88, and 89, which capture mountains, are included in the fourth region 64. Therefore, the fourth imaging condition is set for blocks 83, 86, 88, and 89.

[0059] The white areas in Figure 7(b) represent the parts corresponding to the people. The shaded areas in Figure 7(b) represent the parts corresponding to the mountains. Blocks 82, 85, and 87 contain the boundary B1 between the first region 61 and the fourth region 64. The shaded areas in Figure 7(b) represent the parts corresponding to the mountains.

[0060] In this embodiment, since a block is the smallest unit for setting imaging conditions, the same imaging conditions are set for each block. As described above, the first imaging conditions are set for blocks 82, 85, and 87, which include the boundary B1 between the first region 61 and the fourth region 64. Therefore, the first imaging conditions are also set for the shaded areas of blocks 82, 85, and 87, i.e., the parts corresponding to mountains. In other words, the shaded areas within blocks 82, 85, and 87 are set to imaging conditions different from the fourth imaging conditions set for blocks 83, 86, 88, and 89, which image the mountains.

[0061] In this case, there may be differences in brightness, contrast, and hue between the shaded areas of blocks 82, 85, and 87 and the shaded areas of blocks 83, 86, 88, and 89. In extreme cases, it is conceivable that the image data corresponding to the shaded areas may suffer from overexposure or underexposure. For example, in block 85, the first imaging conditions suitable for a person may not be suitable for the shaded area (i.e., the mountain portion) of block 85, and the image data corresponding to the shaded area may suffer from overexposure or underexposure. Overexposure refers to the loss of tonal gradation in the high-luminance parts of the image due to overexposure. Underexposure refers to the loss of tonal gradation in the low-luminance parts of the image due to underexposure.

[0062] Figure 8 is an example of image data corresponding to Figure 7(b). In Figure 8, blocks 81 to 89 are each composed of 4 pixels (2 pixels x 2 pixels). Of these, pixels 85b and 85d in block 85, located in the center of Figure 8, are assumed to be underexposed. The correction unit 33b in this embodiment corrects the image by performing a replacement process in which the image data with blown-out highlights or underexposed highlights in a block is replaced with image data from other blocks within the same imaging screen. This correction will be referred to as the first correction process.

[0063] The correction unit 33b performs the first correction process on all blocks where the image data from block 85 contains blown-out highlights or crushed blacks, provided that the block 85 includes the boundary of a region based on multiple subject elements, as described above. Note that the first correction process is unnecessary if there is no overexposure or underexposure.

[0064] (Example of the first correction process) The correction unit 33b selects a block containing image data that has experienced overexposure or underexposure as the block of interest and performs a first correction process on the block of interest. Here, the block of interest is defined as an area containing image data that has experienced overexposure or underexposure, but it does not have to be completely overexposed or underexposed. For example, an area where the signal value is above the first threshold or below the second threshold may be defined as the block of interest. In the examples of Figures 7(b) and 8, the eight blocks surrounding the block of interest 85, which are included in a predetermined range 80 (e.g., a 3x3 block) centered on the predetermined block of interest 85, are defined as reference blocks. That is, blocks 81-84 and blocks 86-89 surrounding the predetermined block of interest 85 are the reference blocks. The number of blocks constituting the predetermined range 80 is not limited to the 3x3 blocks described above, and may be changed as appropriate.

[0065] 1. The same correction is applied to the entire area where overexposure or underexposure has occurred. (1-1) As a first correction process, the correction unit 33b corrects a portion of the area within the block of interest using image data acquired by one reference block. Specifically, the correction unit 33b corrects all of the image data where overexposure or underexposure has occurred using image data acquired by one reference block. At this time, the area of ​​the reference block is the same as that of the block of interest. The mode of processing (1-1) may be any of the following modes (i) to (iv).

[0066] (i) The correction unit 33b replaces the image data in the block of interest that is overexposed or underexposed with image data acquired from one of the reference blocks located around the block of interest that is closest to the area that is overexposed or underexposed. Even if there are multiple pixels that are overexposed or underexposed within the block of interest, the image data of those multiple pixels that are overexposed or underexposed is replaced with the same image data acquired from the reference block located closest to them as described above. For example, among the reference blocks 81 to 84 and 86 to 89 around the block of interest 85, the image data corresponding to pixels 86a to 86d in the reference block 86 located closest to the underexposed pixels (pixels 85b and 85d) is used to replace the image data corresponding to underexposed pixel 85b and the image data corresponding to underexposed pixel 85d with the same data (for example, the image data corresponding to pixel 86c).

[0067] (ii) The correction unit 33b uses image data acquired from one of the reference blocks located around the block of interest that has overexposed or underexposed subject elements (e.g., mountains) and is set most frequently for the same subject element (mountain) as the subject element (mountain) (in this example, the fourth imaging condition) to replace the overexposed or underexposed image data within the block of interest with the same data. For example, from the reference blocks 81-84 and 86-89 around the block of interest 85, the correction unit 33b uses image data corresponding to pixels 88a-88d in one of the reference blocks 83, 86, 88, and 89 for which the fourth imaging condition is set for the mountain to replace the image data corresponding to underexposed pixel 85b and the image data corresponding to underexposed pixel 85d with the same data (e.g., image data corresponding to pixel 88b) based on the image data corresponding to pixels 88a-88d in one of the reference blocks 83, 86, 88, and 89, for example, reference block 88.

[0068] (iii) The correction unit 33b may select a pixel from the image data corresponding to the four pixels acquired in one reference block selected in (i) or (ii) above that has a short interval with the overexposed or underexposed pixel in the block of interest. Specifically, the correction unit 33b replaces the underexposed pixel 85b with the pixel 86a that has a short interval with the underexposed pixel 85b, among the interval between the underexposed pixel 85b and pixel 86a and the interval between the underexposed pixel 85b and pixel 86b. Here, the interval is the interval between the centers of the underexposed pixel 85b and pixel 86a, for example. The interval may also be the interval between the centroids of the underexposed pixel 85b and pixel 86a. Furthermore, if the underexposed pixels are consecutive (underexposed pixel 85b and underexposed pixel 86a), the interval may be the center or centroid of the cluster of two underexposed pixels. The same applies to 86a etc. in the reference block. Furthermore, the correction unit 33b may replace the image data that has been overexposed or underexposed with image data corresponding to adjacent pixels. For example, if the reference block 86 is selected, the correction unit 33b replaces the image data corresponding to the underexposed pixel 85b and the image data corresponding to the underexposed pixel 85d with the same data (image data corresponding to pixel 86a or pixel 86c of the reference block 86).

[0069] (iv) The correction unit 33b may replace the image data in the block of interest that has been blown out or crushed in white using image data generated based on the image data corresponding to the four pixels acquired in one reference block selected in (i) or (ii) above. If, for example, reference block 88 is selected, the correction unit 33b replaces the image data corresponding to the crushed pixel 85b and the image data corresponding to the crushed pixel 85d with image data based on multiple pixels in reference block 88 (for example, the average value of the image data corresponding to pixels 88a to 88d included in reference block 88).

[0070] Furthermore, when calculating the average value of the image data, instead of a simple average, a weighted average value may be used, which is weighted according to the distance from pixels where overexposure or underexposure occurred. For example, since pixel 88b is closer to the underexposed pixel 85d than pixel 88d, the contribution rate of the image data corresponding to pixel 88b is weighted higher than the contribution rate of the image data corresponding to pixel 88d.

[0071] Alternatively, instead of calculating the average value of the image data corresponding to pixels 88a to 88d included in the reference block 88, the median value of the image data corresponding to pixels 88a to 88d may be calculated, and the image data corresponding to the blacked-out pixels 85b and 85d may be replaced with this median value.

[0072] (1-2) As a first correction process, the correction unit 33b uses image data acquired from multiple reference blocks to replace all image data in the block of interest that has been overexposed or underexposed. Here, multiple candidate reference blocks are extracted for replacing the underexposed pixels (85b, 85d). Ultimately, only pixels within a single block are used for replacement. The mode of processing (1-2) may be one of the following modes (i) to (iv).

[0073] (i) The correction unit 33b uses image data acquired from multiple reference blocks located around the block of interest that are overexposed or underexposed, to replace the overexposed or underexposed image data within the block of interest with the same data. For example, among the reference blocks 81-84 and 86-89 around the block of interest 85, the unit performs the following replacement based on the image data corresponding to pixels 86a-86d and 88a-88d contained in the two reference blocks 86 and 88 adjacent to the underexposed pixels (pixels 85b and 85d). For example, the image data corresponding to underexposed pixel 85b and the image data corresponding to underexposed pixel 85d are replaced with the same data (for example, the image data corresponding to pixel 88b). At this time, the area of ​​underexposed pixels 85b and 85d that are replaced by pixel 88b is smaller than the area of ​​reference block 88.

[0074] (ii) The correction unit 33b uses image data acquired from multiple reference blocks located around the block of interest that have been selected from reference blocks with the imaging condition (in this example, the fourth imaging condition) that is most frequently set for the same subject element (e.g., a mountain) as the subject element (e.g., a mountain) that has been blown out or crushed in color, to replace the image data in the block of interest that has been blown out or crushed in color with the same data. For example, from the reference blocks 81-84 and 86-89 around the block of interest 85, the correction unit 33b uses image data acquired from multiple reference blocks selected from reference blocks 83, 86, 88, and 89 for which the fourth imaging condition is set for a mountain, for example, image data corresponding to pixels 86a-86d and 88a-88d contained in reference blocks 86 and 88 to replace the image data corresponding to the crushed pixel 85b and the image data corresponding to the crushed pixel 85d with the same data (e.g., image data corresponding to pixel 86c).

[0075] (iii) The correction unit 33b may replace the image data in which the highlighting or blacking occurred with the image data in which the highlighting or blacking occurred, using the image data in which the highlighting or blacking occurred is adjacent to the image data in which the highlighting or blacking occurred, from among the image data in which the highlighting or blacking occurred obtained in the highlighting or blacking

[0076] (iv) The correction unit 33b may replace the image data in the block of interest that has been blown out or crushed in white using image data generated based on the image data corresponding to multiple pixels acquired in the multiple reference blocks selected in (i) or (ii) above. If the correction unit 33b selects reference blocks 86 and 88, for example, it replaces the image data corresponding to the crushed pixel 85b and the image data corresponding to the crushed pixel 85d with the same data (the average value of the image data corresponding to pixels 86a to 86d in reference block 86 and the image data corresponding to pixels 88a to 88d in reference block 88). In this case, the area of ​​the pixels used for replacement is larger than the area of ​​the crushed pixels 85b and 85d.

[0077] Furthermore, when calculating the average value of the image data, instead of a simple average, a weighted average value may be used, which is weighted according to the distance from pixels where overexposure or underexposure occurred. For example, since pixel 86a is closer to the underexposed pixel 85b than pixel 86b, a weight is assigned to the image data corresponding to pixel 86a so that its contribution rate is higher than that of the image data corresponding to pixel 86b.

[0078] Alternatively, instead of calculating the average value of the image data corresponding to pixels 86a-86d and 88a-88d included in reference blocks 86 and 88, the median value of the image data corresponding to pixels 86a-86d and 88a-88d may be calculated, and the image data corresponding to the blacked-out pixels 85b and 85d may be replaced with this median value.

[0079] 2. Perform multiple corrections on the entire area where overexposure or underexposure has occurred. (2-1) As a first correction process, the correction unit 33b replaces all of the image data in the block of interest that has overexposed or underexposed using image data acquired in one reference block. The mode of processing (2-1) may be any of the following modes (i) to (iii).

[0080] (i) The correction unit 33b uses image data corresponding to pixels adjacent to the clipped or clipped pixels among the reference blocks located around the block of interest to replace multiple clipped or clipped images within the block of interest with different data. For example, among the reference blocks 81-84 and 86-89 surrounding the block of interest 85, the replacement is performed as follows based on the image data corresponding to pixels 86a-86d included in the reference block 86 adjacent to the clipped pixels (pixels 85b and 85d): For example, the image data corresponding to the clipped pixel 85b is replaced with the image data of pixel 86a in the adjacent reference block 86, and the image data corresponding to the clipped pixel 85d is replaced with the image data of pixel 86c in the adjacent reference block 86.

[0081] (ii) The correction unit 33b uses image data acquired from one of the reference blocks located around the block of interest that has blown-out highlights or crushed blacks, selected from the reference blocks with the imaging condition (in this example, the fourth imaging condition) that is most frequently set for the same subject element (e.g., a mountain) as the subject element (e.g., a mountain), to replace multiple image data in the block of interest that have blown-out highlights or crushed blacks with different data. For example, from the reference blocks 81-84 and 86-89 around the block of interest 85, the correction unit 33b uses image data corresponding to pixels 86a-86d in one of the reference blocks 83, 86, 88, and 89 for which the fourth imaging condition is set for a mountain, for example, image data in reference block 86, to which pixels 86a-86d are included, to perform the replacement as follows: For example, the image data corresponding to the crushed black pixel 85b is replaced with the image data of pixel 86b in reference block 86, and the image data corresponding to the crushed black pixel 85d is replaced with the image data of pixel 86d in reference block 86.

[0082] (iii) The correction unit 33b may replace the image data with image data generated based on the image data corresponding to the four pixels acquired in one reference block selected in (i) or (ii) above. If, for example, reference block 86 is selected, the correction unit 33b replaces the image data corresponding to the blacked-out pixel 85b in the block of interest with the average value of the image data corresponding to pixels 86a and 86b included in reference block 86. It also replaces the image data corresponding to the blacked-out pixel 85d in the block of interest with the average value of the image data corresponding to pixels 86c and 86d included in reference block 86.

[0083] Furthermore, when calculating the average value of the image data, instead of a simple average, a weighted average value may be used, which is weighted according to the distance from the pixel where overexposure or underexposure occurred. For example, since the image data corresponding to pixel 86a is closer to the image data corresponding to the underexposed pixel 85b than the image data corresponding to pixel 86b, a weight is assigned to the image data corresponding to pixel 86a so that its contribution rate is higher than that of the image data corresponding to pixel 86b.

[0084] (2-2) As a first correction process, the correction unit 33b replaces all image data in the block of interest that has been overexposed or underexposed using image data acquired from multiple reference blocks. The mode of processing (2-2) may be any of the following modes (i) to (iii).

[0085] (i) The correction unit 33b uses image data acquired from multiple reference blocks located around the block of interest that are overexposed or underexposed, to replace multiple image data within the block of interest that are overexposed or underexposed with different data. For example, among the reference blocks 81-84 and 86-89 around the block of interest 85, the correction unit 33b uses image data corresponding to pixels 86a-86d and 88a-88d contained in the two adjacent reference blocks 86 and 88 that are underexposed (pixels 85b and 85d) to perform the following replacements: For example, the image data corresponding to the underexposed pixel 85b is replaced with the image data of pixel 86a in the adjacent reference block 86, and the image data corresponding to the underexposed pixel 85d is replaced with the image data of pixel 88b in the adjacent reference block 88.

[0086] (ii) The correction unit 33b uses image data acquired from multiple reference blocks located around the block of interest that have been set to the same subject element (e.g., a mountain) as the subject element that has been blown out or crushed in black (e.g., a mountain), and which have the same imaging condition (in this example, the fourth imaging condition), to replace multiple image data in the block of interest that have been blown out or crushed in black with different data. For example, from the reference blocks 81-84 and 86-89 around the block of interest 85, the correction unit 33b selects two reference blocks 83, 86, 88, and 89 for which the fourth imaging condition is set for a mountain, for example, image data corresponding to pixels 86a-86d and 88a-88d in reference blocks 86 and 88, and replaces them as follows: For example, the image data corresponding to the crushed pixel 85b is replaced with the image data of pixel 86a of reference block 86, and the image data corresponding to the crushed pixel 85d is replaced with the image data of pixel 88b of reference block 88.

[0087] (iii) The correction unit 33b may replace the image data with image data generated based on image data corresponding to multiple pixels acquired in multiple reference blocks selected in (i) or (ii) above. If, for example, reference blocks 86 and 88 are selected, the correction unit 33b replaces the image data corresponding to blacked-out pixels 85b and 85d in the block of interest as follows: That is, the image data corresponding to blacked-out pixel 85b is replaced with the average value of the image data corresponding to pixels 86a to 86d included in reference block 86. Also, the image data corresponding to blacked-out pixel 85d is replaced with the average value of the image data corresponding to pixels 88a to 88d included in reference block 88.

[0088] Furthermore, when calculating the average value of the image data, instead of a simple average, a weighted average value may be used, which is weighted according to the distance from the pixel where overexposure or underexposure occurred. For example, since the image data corresponding to pixel 86a is closer to the image data corresponding to the underexposed pixel 85b than the image data corresponding to pixel 86b, a weight is assigned to the image data corresponding to pixel 86a so that its contribution rate is higher than that of the image data corresponding to pixel 86b.

[0089] Alternatively, instead of calculating the average value of the image data corresponding to pixels 86a-86d and 88a-88d included in reference blocks 86 and 88, the median value of the image data corresponding to pixels 86a-86d and 88a-88d may be calculated, and the image data corresponding to the blacked-out pixels 85b and 85d may be replaced with this median value.

[0090] The above description has explained the first correction process in various forms. The control unit 34 may decide which of these forms of the first correction process to perform based, for example, on the setting status (including the setting of the operation menu) by the operation member 36. Furthermore, the control unit 34 may determine which form of the first correction processing to perform based on the imaging scene mode set in camera 1 and the type of subject element detected.

[0091] <Second Correction Process> The correction unit 33b of the image processing unit 33 further performs the following second correction process as needed before image processing, focus detection processing, subject detection (detection of subject elements) processing, and imaging condition setting processing. Note that the correction unit 33b performs the second correction process after replacing pixels that are overexposed or underexposed as described above. Furthermore, for image data at the location of the blacked-out pixel 85b (or 85d) that has been replaced by another pixel, the second correction process described below should be performed as if it were captured under the same imaging conditions as the replaced pixel (for example, 86a). Also, if the blacked-out pixel 85b is replaced using pixels from multiple blocks with different imaging conditions, it may be treated as being captured under an imaging condition of a value (average, median) between the imaging conditions of each block. For example, when correcting the blacked-out pixel 85d with pixel 86c captured at ISO 100 and pixel 88b captured at ISO 1600, it may be treated as data captured at ISO 800, which is between ISO 100 and ISO 1600. 1. When performing image processing The correction unit 33b of the image processing unit 33 performs a second correction process as a pre-processing step for image data located at the boundaries of the regions when the image processing performed on the image data obtained by applying different imaging conditions between the divided regions is a predetermined image processing step. The predetermined image processing step is a process that calculates the image data of a point of interest to be processed in the image by referring to image data of multiple reference points around the point of interest, and includes, for example, pixel defect correction processing, color interpolation processing, edge enhancement processing, and noise reduction processing.

[0092] The second correction process is performed to mitigate discontinuities that occur in the image after image processing due to differences in imaging conditions between the divided regions. Generally, when the point of interest is located at the boundary of a divided region, the reference locations surrounding the point of interest may contain a mixture of image data applied under the same imaging conditions as the image data of the point of interest and image data applied under different imaging conditions. In this embodiment, based on the idea that it is preferable to calculate the image data of the point of interest by referring to the image data of the reference locations that have undergone the second correction process to suppress differences between image data due to differences in imaging conditions, rather than directly referring to the image data of the reference locations to which different imaging conditions have been applied, the second correction process is performed as follows.

[0093] Figure 9(a) is an enlarged view of the area of ​​interest 90 at the boundary between the first area 61 and the fourth area 64 in the live view image 60a of Figure 7(a). Image data from pixels on the image sensor 32a corresponding to the first area 61, where the first imaging condition is set, is shown in white, and image data from pixels on the image sensor 32a corresponding to the fourth area 64, where the fourth imaging condition is set, is shown in shaded area. In Figure 9(a), the image data from the pixel of interest P is located on the first area 61, in the vicinity of the boundary 91 between the first area 61 and the fourth area 64, i.e., at the boundary. The pixels surrounding the pixel of interest P (e.g., 8 pixels in this example) that are included in the area of ​​interest 90 (e.g., 3x3 pixels) centered on the pixel of interest P are called reference pixels Pr. Figure 9(b) is an enlarged view of the pixel of interest P and reference pixels Pr1 to Pr8. The position of the pixel of interest P is the position of interest, and the positions of the reference pixels Pr1 to Pr8 surrounding the pixel of interest P are the reference positions. First imaging conditions are set for reference pixels Pr1 to Pr6 and the pixel of interest P corresponding to the first region 61, and fourth imaging conditions are set for reference pixels Pr7 and Pr8 corresponding to the fourth region 64. In the following explanation, the code Pr will be used to refer collectively to the reference pixels Pr1 to Pr8.

[0094] The generation unit 33c of the image processing unit 33 normally performs image processing by directly referencing the image data of the reference pixel Pr without performing the second correction process. However, if the imaging conditions applied to the pixel of interest P (referred to as the first imaging conditions) and the imaging conditions applied to the reference pixels Pr surrounding the pixel of interest P (referred to as the fourth imaging conditions) are different, the correction unit 33b performs the second correction process on the image data of the reference pixel Pr corresponding to the fourth imaging conditions, as shown in (Example 1) to (Example 3) below. Then, the generation unit 33c performs image processing to calculate the image data of the pixel of interest P by referring to the image data of the reference pixel Pr after the second correction process.

[0095] (Example 1) The correction unit 33b of the image processing unit 33, for example, if only the ISO sensitivity differs between the first imaging condition and the fourth imaging condition, and the ISO sensitivity of the first imaging condition is 100 and the ISO sensitivity of the fourth imaging condition is 800, applies a second correction process of 100 / 800 to the image data of reference pixels Pr7 and Pr8 of the reference pixel Pr image data under the fourth imaging condition. This reduces the difference between image data due to the difference in imaging conditions. Note that when the amount of light incident on the pixel of interest P is the same as the amount of light incident on the reference pixel Pr, the difference in image data will be small. However, if the amount of light incident on the pixel of interest P is originally different from the amount of light incident on the reference pixel Pr, the difference in image data may not be small. The same applies to the examples described later.

[0096] (Example 2) The correction unit 33b of the image processing unit 33, for example, if only the shutter speed differs between the first imaging condition and the fourth imaging condition, and the shutter speed for the first imaging condition is 1 / 1000 second and the shutter speed for the fourth imaging condition is 1 / 100 second, then multiplies the image data of reference pixels Pr7 and Pr8 of the fourth imaging condition by (1 / 1000) / (1 / 100) = 1 / 10 as a second correction process. This reduces the difference between image data due to the difference in imaging conditions.

[0097] (Example 3) The correction unit 33b of the image processing unit 33 performs a second correction process in which, for example, if only the frame rate differs between the first imaging condition and the fourth imaging condition (the charge accumulation time is the same), and the frame rate of the first imaging condition is 30fps and the frame rate of the fourth imaging condition is 60fps, it selects the image data of the frame image acquired under the first imaging condition (30fps) from the image data of the reference pixel Pr under the fourth imaging condition (60fps) and uses the image data of the frame image acquired under the first imaging condition (30fps) that has a similar acquisition start time. This reduces the difference between image data due to the difference in imaging conditions. Furthermore, the second correction process may involve interpolating and calculating image data of frames acquired under the first imaging condition (30fps) and frames with similar acquisition start times, based on multiple consecutive frame images acquired under the fourth imaging condition (60fps).

[0098] On the other hand, if the imaging conditions applied to the pixel of interest P (referred to as the first imaging conditions) and the imaging conditions applied to all the reference pixels Pr surrounding the pixel of interest P (referred to as the fourth imaging conditions) are the same, the correction unit 33b of the image processing unit 33 does not perform the second correction processing on the image data of the reference pixels Pr. In other words, the generation unit 33c performs image processing to calculate the image data of the pixel of interest P by directly referring to the image data of the reference pixels Pr. As mentioned above, even if there are slight differences in imaging conditions, they will be considered the same.

[0099] <Example of image processing> An example of image processing involving a second correction process is provided. (1) Pixel defect correction process In this embodiment, pixel defect correction processing is one of the image processing steps performed during imaging. Generally, a solid-state image sensor 32a may have pixel defects during the manufacturing process or after manufacturing, resulting in the output of abnormally high-level image data. Therefore, the generation unit 33c of the image processing unit 33 corrects the image data output from the pixel with the pixel defect, thereby making the image data at the pixel location where the pixel defect occurred less noticeable.

[0100] An example of pixel defect correction processing will be described. The generation unit 33c of the image processing unit 33 designates, for example, a pixel at the position of a pixel defect recorded in a non-volatile memory (not shown) in advance in an image of one frame as a target pixel P (pixel to be processed), and sets pixels (eight pixels in this example) around the target pixel P included in a target area 90 (for example, 3×3 pixels) centered on the target pixel P as reference pixels Pr.

[0101] The generation unit 33c of the image processing unit 33 calculates the maximum value and the minimum value of the image data at the reference pixels Pr, and performs Max, Min filter processing to replace the image data output from the target pixel P with the above maximum value or minimum value when the image data output from the target pixel P exceeds these maximum value or minimum value. Such processing is performed for all pixel defects whose position information is recorded in a non-volatile memory (not shown).

[0102] In the present embodiment, when pixels to which a fourth imaging condition different from the first imaging condition applied to the target pixel P are included in the reference pixels Pr, the correction unit 33b of the image processing unit 33 performs a second correction process on the image data to which the fourth imaging condition is applied. Then, the generation unit 33c of the image processing unit 33 performs the above-described Max, Min filter processing.

[0103] (2) Color interpolation processing In the present embodiment, color interpolation processing is one of the image processes performed at the time of imaging. As illustrated in FIG. 3, in the imaging chip 111 of the image sensor 100, green pixels Gb, Gr, blue pixels B, and red pixels R are arranged in a Bayer array. Since the generation unit 33c of the image processing unit 33 lacks image data of color components different from the color components of the color filter F arranged at each pixel position, color interpolation processing is performed to generate image data of the lacking color components by referring to the image data at the surrounding pixel positions.

[0104] An example of color interpolation processing will be described. FIG. 10(a) is a diagram illustrating the arrangement of image data output from the image sensor 32a. Corresponding to each pixel position, it has a color component of either R, G, or B according to the rule of the Bayer array. <G color interpolation> First, let's explain general G color interpolation. The generation unit 33c of the image processing unit 33 that performs G color interpolation uses the positions of the R color component and B color component in order as the position of interest, and generates image data of the G color component at the position of interest by referring to the image data of the four G color components at the reference positions surrounding the position of interest. For example, when generating image data of the G color component at the position of interest indicated by the thick frame in Figure 10(b) (2nd row, 2nd column counting from the top left position; similarly, the position of interest will be indicated counting from the top left position hereafter), it refers to the image data of the four G color components G1 to G4 located in the vicinity of the position of interest (2nd row, 2nd column). The generation unit 33c of the image processing unit 33 uses, for example, (aG1+bG2+cG3+dG4) / 4 as the image data of the G color component at the position of interest (2nd row, 2nd column). a to d are weighting coefficients set according to the distance between the reference position and the position of interest and the image structure.

[0105] Next, the G color interpolation of this embodiment will be described. In Figures 10(a) to 10(c), the first imaging condition is applied to the areas to the left and above the thick line, and the fourth imaging condition is applied to the areas to the right and below the thick line. Note that the first imaging condition and the fourth imaging condition are different in Figures 10(a) to 10(c). Also, the G color component image data G1 to G4 in Figure 10(b) are reference positions for image processing the pixels at the point of interest (2nd row, 2nd column). In Figure 10(b), the first imaging condition is applied to the point of interest (2nd row, 2nd column). Of the reference positions, the first imaging condition is applied to image data G1 to G3. Also, of the reference positions, the fourth imaging condition is applied to image data G4. Therefore, the correction unit 33b of the image processing unit 33 performs the second correction process on image data G4. After that, the generation unit 33c of the image processing unit 33 calculates the G color component image data at the point of interest (2nd row, 2nd column).

[0106] The generation unit 33c of the image processing unit 33 generates image data of the G color component at the positions of the B color component and the R color component in Figure 10(a), respectively, thereby obtaining image data of the G color component at each pixel position, as shown in Figure 10(c).

[0107] <R color interpolation> FIG. 11(a) is a diagram obtained by extracting the image data of the R color component from FIG. 10(a). The generation unit 33c of the image processing unit 33 calculates the image data of the color difference component Cr shown in FIG. 11(b) based on the image data of the G color component shown in FIG. 10(c) and the image data of the R color component shown in FIG. 11(a).

[0108] First, the interpolation of the general color difference component Cr will be described. When the generation unit 33c of the image processing unit 33 generates the image data of the color difference component Cr at the attention position indicated by the thick frame (the second row and the second column) in FIG. 11(b), for example, it refers to the image data Cr1 to Cr4 of the four color difference components located near the attention position (the second row and the second column). The generation unit 33c of the image processing unit 33 uses, for example, (eCr1 + fCr2 + gCr3 + hCr4) / 4 as the image data of the color difference component Cr at the attention position (the second row and the second column). Here, e to h are weighting coefficients provided according to the distance between the reference position and the attention position and the image structure.

[0109] Similarly, when the generation unit 33c of the image processing unit 33 generates the image data of the color difference component Cr at the attention position indicated by the thick frame (the second row and the third column) in FIG. 11(c), for example, it refers to the image data Cr2, Cr4 to Cr6 of the four color difference components located near the attention position (the second row and the third column). The generation unit 33c of the image processing unit 33 uses, for example, (qCr2 + rCr4 + sCr5 + tCr6) / 4 as the image data of the color difference component Cr at the attention position (the second row and the third column). Here, q to t are weighting coefficients provided according to the distance between the reference position and the attention position and the image structure. In this way, the image data of the color difference component Cr is generated for each pixel position.

[0110] Next, the interpolation of the color difference component Cr in this embodiment will be explained. In Figures 11(a) to 11(c), for example, the first imaging condition is applied to the areas to the left and above the thick line, and the fourth imaging condition is applied to the areas to the right and below the thick line. Note that the first imaging condition and the fourth imaging condition are different in Figures 11(a) to 11(c). In Figure 11(b), the position indicated by the thick frame (2nd row, 2nd column) is the position of interest for the color difference component Cr. Also, the image data Cr1 to Cr4 of the color difference component in Figure 11(b) are reference positions for image processing the pixels at the position of interest (2nd row, 2nd column). In Figure 11(b), the first imaging condition is applied to the position of interest (2nd row, 2nd column). Of the reference positions, the first imaging condition is applied to the image data Cr1, Cr3, and Cr4. Also, of the reference positions, the fourth imaging condition is applied to the image data Cr2. Therefore, the correction unit 33b of the image processing unit 33 performs a second correction process on the image data Cr2. After that, the generation unit 33c of the image processing unit 33 calculates the image data of the color difference component Cr at the position of interest (2nd row, 2nd column). Furthermore, in Figure 11(c), the position indicated by the thick border (2nd row, 3rd column) is the position of interest for the chrominance component Cr. Also, the image data Cr2, Cr4, Cr5, and Cr6 of the chrominance components in Figure 11(c) are reference positions for image processing the pixels at the position of interest (2nd row, 3rd column). In Figure 11(c), the 4th imaging condition is applied to the position of interest (2nd row, 3rd column). Of the reference positions, the image data Cr4 and Cr5 are applied to the 1st imaging condition. Also, of the reference positions, the image data Cr2 and Cr6 are applied to the 4th imaging condition. Therefore, the correction unit 33b of the image processing unit 33 performs the 2nd correction process on the image data Cr4 and Cr5, respectively. After that, the generation unit 33c of the image processing unit 33 calculates the image data of the chrominance component Cr at the position of interest (2nd row, 3rd column).

[0111] The generation unit 33c of the image processing unit 33 obtains image data of the color difference component Cr at each pixel position, and then adds the image data of the G color component shown in Figure 10(c) corresponding to each pixel position to obtain image data of the R color component at each pixel position.

[0112] Figure 12(a) is a diagram obtained by extracting the image data of the B color component from Figure 10(a). The generation unit 33c of the image processing unit 33 calculates the image data of the color difference component Cb shown in Figure 12(b) based on the image data of the G color component shown in Figure 10(c) and the image data of the B color component shown in Figure 12(a).

[0113] First, the interpolation of the general color difference component Cb will be described. When the generation unit 33c of the image processing unit 33 generates the image data of the color difference component Cb at the target position indicated by the thick frame (the 3rd row and 3rd column) in Figure 12(b), for example, it refers to the image data Cb1 to Cb4 of the four color difference components located near the target position (the 3rd row and 3rd column). The generation unit 33c of the image processing unit 33 uses, for example, (uCb1 + vCb2 + wCb3 + xCb4) / 4 as the image data of the color difference component Cb at the target position (the 3rd row and 3rd column). Here, u to x are weighting coefficients provided according to the distance between the reference position and the target position and the image structure.

[0114] Similarly, when the generation unit 33c of the image processing unit 33 generates the image data of the color difference component Cb at the target position indicated by the thick frame (the 3rd row and 4th column) in Figure 12(c), for example, it refers to the image data Cb2, Cb4 to Cb6 of the four color difference components located near the target position (the 3rd row and 4th column). The generation unit 33c of the image processing unit 33 uses, for example, (yCb2 + zCb4 + αCb5 + βCb6) / 4 as the image data of the color difference component Cb at the target position (the 3rd row and 4th column). Here, y, z, α, and β are weighting coefficients provided according to the distance between the reference position and the target position and the image structure. In this way, the image data of the color difference component Cb is generated for each pixel position.

[0115] Next, the interpolation of the color difference component Cb in this embodiment will be explained. In Figures 12(a) to 12(c), for example, the first imaging condition is applied to the areas to the left and above the thick line, and the fourth imaging condition is applied to the areas to the right and below the thick line. Note that the first and fourth imaging conditions are different in Figures 12(a) to 12(c). In Figure 12(b), the position indicated by the thick frame (3rd row, 3rd column) is the position of interest for the color difference component Cb. Also, the image data Cb1 to Cb4 of the color difference component in Figure 12(b) are reference positions for image processing the pixels at the position of interest (3rd row, 3rd column). In Figure 12(b), the fourth imaging condition is applied to the position of interest (3rd row, 3rd column). Of the reference positions, the first imaging condition is applied to the image data Cb1 and Cb3. Also, of the reference positions, the fourth imaging condition is applied to the image data Cb2 and Cb4. Therefore, the correction unit 33b of the image processing unit 33 performs a second correction process on data Cb1 and Cb3, respectively. After that, the generation unit 33c of the image processing unit 33 calculates image data of the color difference component Cb at the position of interest (3rd row, 3rd column). Furthermore, in Figure 12(c), the position indicated by the thick border (3rd row, 4th column) is the position of interest for the chrominance component Cb. Also, the chrominance component image data Cb2, Cb4-Cb6 in Figure 12(c) are reference positions for image processing the pixels at the position of interest (3rd row, 4th column). In Figure 12(c), the fourth imaging condition is applied to the position of interest (3rd row, 4th column). Also, the fourth imaging condition is applied to the image data Cb2, Cb4-Cb6 of all reference positions. Therefore, the generation unit 33c of the image processing unit 33 calculates the image data of the chrominance component Cb at the position of interest (3rd row, 4th column) by referring to the image data Cb2, Cb4-Cb6 of the reference positions that have not undergone the second correction processing by the correction unit 33b of the image processing unit 33.

[0116] The generation unit 33c of the image processing unit 33 obtains image data of the color difference component Cb at each pixel position, and then adds the image data of the G color component shown in Figure 10(c) corresponding to each pixel position to obtain image data of the B color component at each pixel position. In the "G color interpolation" described above, for example, when generating image data of the G color component at the point of interest indicated by the thick frame (2nd row, 2nd column) in Figure 10(b), the image data G1 to G4 of the four G color components located near the point of interest are referenced. However, the number of G color component image data referenced may be changed depending on the image structure. For example, if the images near the point of interest have similarity in the vertical direction (e.g., a vertical stripe pattern), interpolation is performed using only the image data above and below the point of interest (G1 and G2 in Figure 10(b)). Also, for example, if the images near the point of interest have similarity in the horizontal direction (e.g., a horizontal stripe pattern), interpolation is performed using only the image data to the left and right of the point of interest (G3 and G4 in Figure 10(b)). In these cases, the image data G4 corrected by the correction unit 33b may or may not be used. In this way, by performing the first correction process, the second correction process, and interpolation process, the correction unit 33b can correct the black crushing pixels 85b and 85d and generate an image.

[0117] (3) Edge enhancement processing An example of edge enhancement processing is described below. The generation unit 33c of the image processing unit 33 performs a known linear filter operation using a predetermined size kernel centered on the pixel of interest P (the pixel to be processed) in one frame of the image. In the case of a sharpening filter, which is an example of a linear filter, if the kernel size is N × N pixels, the position of the pixel of interest P is the position of interest, and the area surrounding the pixel of interest P is (N 2 The position of the -1) reference pixels Pr is the reference position. The kernel size may also be N×M pixels.

[0118] The generation unit 33c of the image processing unit 33 performs a filtering process that replaces the image data of the pixel of interest P with the result of a linear filter operation, for example, from the upper horizontal line of the frame image to the lower horizontal line, while shifting the pixel of interest from left to right on each horizontal line.

[0119] In this embodiment, the correction unit 33b of the image processing unit 33 performs a second correction process on the image data to which a fourth imaging condition different from the first imaging condition applied to the target pixel P is applied, if the reference pixel Pr includes a pixel to which a fourth imaging condition different from the first imaging condition applied to the target pixel P is applied. Subsequently, the generation unit 33c of the image processing unit 33 performs the linear filter processing described above.

[0120] (4) Noise reduction processing An example of noise reduction processing is described below. The generation unit 33c of the image processing unit 33 performs a known linear filter operation using a predetermined size kernel centered on the pixel of interest P (the pixel to be processed) in one frame of the image. In the case of a smoothing filter, which is an example of a linear filter, if the kernel size is N × N pixels, the position of the pixel of interest P is the position of interest, and the area surrounding the pixel of interest P is (N 2 The position of the -1) reference pixels Pr is the reference position. The kernel size may also be N×M pixels.

[0121] The generation unit 33c of the image processing unit 33 performs a filtering process that replaces the image data of the pixel of interest P with the result of a linear filter operation, for example, from the upper horizontal line of the frame image to the lower horizontal line, while shifting the pixel of interest from left to right on each horizontal line.

[0122] In this embodiment, the correction unit 33b of the image processing unit 33 performs a second correction process on the image data to which a fourth imaging condition different from the first imaging condition applied to the target pixel P is applied, if the reference pixel Pr includes a pixel to which a fourth imaging condition different from the first imaging condition applied to the target pixel P is applied. Subsequently, the generation unit 33c of the image processing unit 33 performs the linear filter processing described above.

[0123] 2. When performing focus detection processing In the example above, as the first correction process, pixels that were overexposed or underexposed were replaced with image data from pixels in other blocks. However, if the sole purpose is focus adjustment, the signal from the overexposed or underexposed focus detection pixel can be replaced with the signal from other focus detection pixels. The method for replacing with other focus detection signals is the same as the method for replacing the image data of the overexposed or underexposed pixel, so the details are omitted. For focus adjustment based on image contrast, the image data replaced by the first correction process described above can be used. The lens movement control unit 34d of the control unit 34 performs focus detection processing using signal data (image data) corresponding to a predetermined position (focus point) on the imaging screen. If different imaging conditions are set between the divided regions, and the focus point for AF operation is located at the boundary of the divided region, the lens movement control unit 34d of the control unit 34 performs a second correction process as a preprocessing step for focus detection processing on the signal data for focus detection of at least one region.

[0124] The second correction process is performed to suppress a decrease in the accuracy of the focus detection process due to differences in imaging conditions between the regions of the imaging screen divided by the setting unit 34b. For example, if the signal data for focus detection of a focus point that detects the amount of image shift (phase difference) in an image is located at the boundary of a divided region, the signal data for focus detection may contain signal data to which different imaging conditions are applied. In this embodiment, based on the idea that it is preferable to perform image shift amount (phase difference) detection using signal data to which the second correction process has been applied to suppress differences between signal data due to differences in imaging conditions, rather than using signal data to which different imaging conditions are applied as is, the second correction process is performed as follows.

[0125] <Example of focus detection processing> An example of focus detection processing with a second correction process is given. In this embodiment, the AF operation focuses on a subject corresponding to a focus point selected by the user from among multiple focus points on the imaging screen. The lens movement control unit 34d (generation unit) of the control unit 34 calculates the amount of defocus of the imaging optical system 31 by detecting the amount of image shift (phase difference) of multiple subject images caused by light beams passing through different pupil regions of the imaging optical system 31. The lens movement control unit 34d of the control unit 34 adjusts the focus of the imaging optical system 31 by moving the focus lens of the imaging optical system 31 to a position where the amount of defocus is zero (below the allowable value), i.e., the in-focus position.

[0126] Figure 13 illustrates the position of focus detection pixels on the imaging surface of the image sensor 32a. In this embodiment, focus detection pixels are discretely arranged along the X-axis direction (horizontal direction) of the imaging chip 111. In the example of Figure 13, 15 focus detection pixel lines 160 are provided at predetermined intervals. The focus detection pixels constituting the focus detection pixel lines 160 output a photoelectric conversion signal for focus detection. In the imaging chip 111, normal imaging pixels are provided at pixel positions other than those of the focus detection pixel lines 160. The imaging pixels output a photoelectric conversion signal for live view images and recording.

[0127] Figure 14 is an enlarged view of a portion of the focus detection pixel line 160 corresponding to the focus point 80A shown in Figure 13. In Figure 14, red pixels R, green pixels G (Gb, Gr), and blue pixels B, as well as focus detection pixels S1 and S2, are shown as examples. The red pixels R, green pixels G (Gb, Gr), and blue pixels B are arranged according to the Bayer array rules described above.

[0128] The square-shaped regions exemplified for the red pixel R, green pixel G(Gb, Gr), and blue pixel B represent the light-receiving areas of the imaging pixels. Each imaging pixel receives the light beam passing through the exit pupil of the imaging optical system 31 (Figure 1). That is, the red pixel R, green pixel G(Gb, Gr), and blue pixel B each have square-shaped mask openings, and the light passing through these mask openings reaches the light-receiving area of ​​the imaging pixel.

[0129] Furthermore, the shape of the light-receiving area (mask aperture) of the red pixel R, green pixel G (Gb, Gr), and blue pixel B is not limited to a rectangle; for example, it may be circular.

[0130] The semicircular regions illustrated for focus detection pixels S1 and S2 indicate the light-receiving areas of the focus detection pixels. Specifically, focus detection pixel S1 has a semicircular mask opening to the left of its pixel position in Figure 14, and light passing through this mask opening reaches the light-receiving area of ​​focus detection pixel S1. On the other hand, focus detection pixel S2 has a semicircular mask opening to the right of its pixel position in Figure 14, and light passing through this mask opening reaches the light-receiving area of ​​focus detection pixel S2. In this way, focus detection pixels S1 and S2 each receive a pair of light beams passing through different regions of the exit pupil of the imaging optical system 31 (Figure 1).

[0131] The position of the focus detection pixel line 160 in the imaging chip 111 is not limited to the position exemplified in Figure 13. Furthermore, the number of focus detection pixel lines 160 is not limited to the example in Figure 13. In addition, the shape of the mask aperture in the focus detection pixels S1 and S2 is not limited to a semicircle; for example, the rectangular light-receiving area (mask aperture) in the imaging pixels R, G, and B may be divided horizontally to form a rectangle.

[0132] Furthermore, the focus detection pixel line 160 in the imaging chip 111 may be formed by arranging focus detection pixels along the Y-axis direction (vertical direction) of the imaging chip 111. Image sensors in which imaging pixels and focus detection pixels are arranged in a two-dimensional manner, as shown in Figure 14, are well known, and detailed illustrations and explanations of these pixels are omitted.

[0133] In the example shown in Figure 14, a configuration in which focus detection pixels S1 and S2 each receive one of a pair of light beams for focus detection, known as a 1PD structure, was described. Alternatively, a configuration in which each focus detection pixel receives both of the pair of light beams for focus detection, known as a 2PD structure, may be used. By using a 2PD structure, the photoelectric conversion signal obtained by the focus detection pixel can be used as the photoelectric conversion signal for recording.

[0134] The lens movement control unit 34d of the control unit 34 detects the amount of image shift (phase difference) between a pair of images caused by a pair of light beams passing through different regions of the imaging optical system 31 (Figure 1), based on the photoelectric conversion signals for focus detection output from the focus detection pixels S1 and S2. Then, it calculates the amount of defocus based on the amount of image shift (phase difference). Since this type of pupil-splitting phase-difference defocus calculation is well known in the field of cameras, a detailed explanation is omitted.

[0135] The focus point 80A (Figure 13) is assumed to be selected by the user in the live view image 60a illustrated in Figure 7(a), for example, at a position corresponding to the area of ​​interest 90 at the boundary between the first area 61 and the fourth area 64. Figure 15 is an enlarged view of the focus point 80A. White pixels indicate that the first imaging condition is set, and shaded pixels indicate that the fourth imaging condition is set. In Figure 15, the area enclosed by frame 170 corresponds to the focus detection pixel line 160 (Figure 13).

[0136] The lens movement control unit 34d of the control unit 34 normally performs focus detection processing using the signal data from the focus detection pixels indicated by the frame 170 without performing the second correction processing. However, if the signal data enclosed by the frame 170 contains a mixture of signal data to which the first imaging condition is applied and signal data to which the fourth imaging condition is applied, the lens movement control unit 34d of the control unit 34 performs the second correction processing on the signal data of the fourth imaging condition among the signal data enclosed by the frame 170, as shown in (Example 1) to (Example 3) below. Then, the lens movement control unit 34d of the control unit 34 performs focus detection processing using the signal data after the second correction processing.

[0137] (Example 1) The lens movement control unit 34d of the control unit 34, for example, if only the ISO sensitivity differs between the first imaging condition and the fourth imaging condition, and the ISO sensitivity of the first imaging condition is 100 and the ISO sensitivity of the fourth imaging condition is 800, applies a second correction process of 100 / 800 to the signal data of the fourth imaging condition. This reduces the difference between signal data due to the difference in imaging conditions. Furthermore, if the amount of incident light to a pixel to which the first imaging condition is applied is the same as the amount of incident light to a pixel to which the fourth imaging condition is applied, the difference in signal data will be small. However, if the amount of incident light to a pixel to which the first imaging condition is applied is originally different from the amount of incident light to a pixel to which the fourth imaging condition is applied, the difference in signal data may not be small. The same applies to the examples described later.

[0138] (Example 2) The lens movement control unit 34d of the control unit 34, for example, if only the shutter speed differs between the first imaging condition and the fourth imaging condition, and the shutter speed for the first imaging condition is 1 / 1000 second and the shutter speed for the fourth imaging condition is 1 / 100 second, then multiplies the signal data for the fourth imaging condition by 1 / 1000 / 1 / 100 = 1 / 10 as a second correction process. This reduces the difference between signal data due to the difference in imaging conditions.

[0139] (Example 3) The lens movement control unit 34d of the control unit 34 performs a second correction process in which, for example, if only the frame rate differs between the first imaging condition and the fourth imaging condition (the charge accumulation time is the same), and the frame rate of the first imaging condition is 30fps and the frame rate of the fourth imaging condition is 60fps, the second correction process adopts the signal data of the frame image acquired under the first imaging condition (30fps) and the frame image whose acquisition start timing is close to that of the frame image acquired under the first imaging condition (30fps). This reduces the difference between signal data due to the difference in imaging conditions. Furthermore, the second correction process may involve interpolating signal data from frame images acquired under the first imaging condition (30fps) and frame images with similar acquisition start times, based on multiple frame images acquired under the fourth imaging condition (60fps) in sequential order.

[0140] On the other hand, the lens movement control unit 34d of the control unit 34 does not perform the second correction process if the imaging conditions applied to the signal data enclosed by the frame 170 are the same. In other words, the lens movement control unit 34d of the control unit 34 performs focus detection processing using the signal data from the focus detection pixels indicated by the frame 170 as is.

[0141] As mentioned above, even if there are slight differences in imaging conditions, they will be considered the same. Furthermore, although the above example describes an example where the second correction process is performed on the signal data of the fourth imaging condition using the first imaging condition, the second correction process may also be performed on the signal data of the first imaging condition using the fourth imaging condition. The lens movement control unit 34d of the control unit 34 may determine, for example, whether to perform the second correction processing on the signal data of the first imaging condition or on the signal data of the fourth imaging condition based on the ISO sensitivity. If the ISO sensitivity differs between the first and fourth imaging conditions, it is desirable to perform the second correction processing on the signal data obtained under the imaging condition with the lower ISO sensitivity, provided that the signal data obtained under the imaging condition with the higher ISO sensitivity is not saturated. In other words, if the ISO sensitivity differs between the first and fourth imaging conditions, it is desirable to perform the second correction processing on the darker signal data to reduce the difference with the brighter signal data.

[0142] Furthermore, a second correction process may be applied to the signal data for the first imaging condition and the signal data for the fourth imaging condition, respectively, in order to reduce the difference between the two sets of signal data after the second correction process.

[0143] The above explanation illustrates focus detection processing using a pupil-splitting phase-difference method, but the same method can also be used for contrast detection methods, which move the focus lens of the imaging optical system 31 to the focus position based on the contrast of the subject image.

[0144] When using the contrast detection method, the control unit 34 moves the focus lens of the imaging optical system 31 and, at each position of the focus lens, performs a known focus evaluation value calculation based on the signal data output from the imaging pixel of the image sensor 32a corresponding to the focus point. Then, it determines the position of the focus lens that maximizes the focus evaluation value as the in-focus position.

[0145] Normally, the control unit 34 calculates the focus evaluation value using the signal data output from the imaging pixel corresponding to the focus point without performing the second correction process. However, if the signal data corresponding to the focus point contains a mixture of signal data to which the first imaging condition is applied and signal data to which the fourth imaging condition is applied, the control unit 34 performs the second correction process described above on the signal data of the fourth imaging condition among the signal data corresponding to the focus point. Then, the control unit 34 calculates the focus evaluation value using the signal data after the second correction process. In this way, by performing the first correction process, the second correction process, and interpolation process, the correction unit 33b can correct the black clipping and adjust the focus even if black clipping pixels 85b and 85d occur. Therefore, even if there are black clipping pixels 85b or 85d, the focus can be adjusted by moving the lens. In the example above, the focus adjustment process was performed after the second correction process, but it is also possible to perform the focus adjustment using the image data obtained by the first correction process without performing the second correction process.

[0146] 3. When performing subject detection processing Figure 16(a) is an example of a template image representing the object to be detected, and Figure 16(b) is an example of a live view image 60a and a search range 190. The object detection unit 34a of the control unit 34 detects an object (for example, a bag 63a, which is one of the subject elements in Figure 5) from the live view image. The object detection unit 34a of the control unit 34 may set the range for detecting the object to the entire range of the live view image 60a, but to reduce the detection processing load, it may also set a part of the live view image 60a as the search range 190.

[0147] The object detection unit 34a of the control unit 34 performs a second correction process as a preprocessing step for subject detection processing on image data of at least one region within the search range 190 if different imaging conditions are set between the divided regions and the search range 190 includes the boundary of the divided region.

[0148] The second correction process is performed to suppress a decrease in the accuracy of the subject element detection process due to differences in imaging conditions between the regions of the imaging screen divided by the setting unit 34b. Generally, when the search range 190 used for detecting subject elements includes the boundaries of the divided regions, the image data in the search range 190 may contain image data to which different imaging conditions have been applied. In this embodiment, based on the idea that it is preferable to perform subject element detection using image data to which the second correction process has been applied to suppress differences between image data due to differences in imaging conditions, rather than using image data to which different imaging conditions have been applied as is, the second correction process is performed as follows.

[0149] The following describes the case of detecting a bag 63a, which is an item belonging to person 61a, in the image of the subject illustrated in Figure 5. The object detection unit 34a of the control unit 34 sets the search range 190 in the vicinity of the region containing person 61a. Alternatively, the region 61 containing person 61a may be set as the search range.

[0150] The object detection unit 34a of the control unit 34 performs subject detection processing using the image data constituting the search range 190 without performing the second correction process if the search range 190 is not divided by two regions with different imaging conditions. However, if the image data in the search range 190 contains a mixture of image data to which the first imaging condition is applied and image data to which the fourth imaging condition is applied, the object detection unit 34a of the control unit 34 performs the second correction process on the image data to which the fourth imaging condition is applied among the image data in the search range 190, as described in (Examples 1) to (Examples 3) above, in the case of performing focus detection processing. Then, the object detection unit 34a of the control unit 34 performs subject detection processing using the image data after the second correction process. As mentioned above, even if there are slight differences in imaging conditions, they will be considered the same. Furthermore, although the above example describes an example where the second correction process is performed on the image data of the fourth imaging condition using the first imaging condition, the second correction process may also be performed on the image data of the first imaging condition using the fourth imaging condition.

[0151] The second correction process for image data within the search range 190 described above may also be applied to the search range used to detect specific subjects such as human faces, or to the region used to determine the imaging scene.

[0152] Furthermore, the second correction process for image data within the search range of 190 described above may be applied not only to the search range used in pattern matching methods using template images, but also to the search range when detecting features based on image color, edges, etc.

[0153] Furthermore, by applying a known template matching process using image data from multiple frames acquired at different times, the tracking process for a moving object may be applied to search for regions similar to the target object in the frame image acquired earlier, using the frame image acquired later. In this case, if the search range set for the frame image acquired later contains a mixture of image data to which the first imaging condition is applied and image data to which the fourth imaging condition is applied, the control unit 34 performs a second correction process on the image data to which the fourth imaging condition is applied within the image data of the search range, as described in (Example 1) to (Example 3) above. Then, the control unit 34 performs the tracking process using the image data after the second correction process.

[0154] Furthermore, the same applies when detecting a known motion vector using image data from multiple frames acquired at different times. If the detection area used for detecting the motion vector contains a mixture of image data to which the first imaging condition is applied and image data to which the fourth imaging condition is applied, the control unit 34 performs a second correction process on the image data to which the fourth imaging condition is applied among the image data of the detection area used for detecting the motion vector, as described in (Example 1) to (Example 3) above. Then, the control unit 34 detects the motion vector using the image data after the second correction process. In this way, by performing the first correction process, the second correction process, and interpolation process, the correction unit 33b can correct for crushed black pixels 85b and 85d and perform the above-mentioned subject detection, etc. Therefore, subject detection can be performed even if there are crushed black pixels 85b or 85d. In the example above, subject detection was performed after the second correction process, but it is also possible to perform subject detection using the image data obtained by the first correction process without performing the second correction process.

[0155] 4. When setting imaging conditions When the setting unit 34b of the control unit 34 divides the area of ​​the imaging screen and sets different imaging conditions between the divided areas, and then remeasures and determines the exposure conditions, it performs a second correction process as a preprocessing step for setting the exposure conditions on the image data of at least one area.

[0156] The second correction process is performed to suppress a decrease in the accuracy of the process for determining exposure conditions due to differences in imaging conditions between regions of the imaging screen divided by the setting unit 34b. For example, if the metering range set in the center of the imaging screen includes the boundaries of the divided regions, the image data of the metering range may contain image data to which different imaging conditions have been applied. In this embodiment, based on the idea that it is preferable to perform exposure calculations using image data to which the second correction process has been applied to suppress differences between image data due to differences in imaging conditions, rather than using image data to which different imaging conditions have been applied as is, the second correction process is performed as follows.

[0157] The setting unit 34b of the control unit 34 performs exposure calculation processing using the image data constituting the photometering range without performing the second correction process if the photometering range is not divided by multiple regions with different imaging conditions. However, if the image data of the photometering range contains a mixture of image data to which the first imaging condition is applied and image data to which the fourth imaging condition is applied, the setting unit 34b of the control unit 34 performs the second correction process on the image data of the fourth imaging condition within the photometering range, as described in (Examples 1) to (Examples 3) above, in the case of performing focus detection processing or subject detection processing. Then, the setting unit 34b of the control unit 34 performs exposure calculation processing using the image data after the second correction process. As mentioned above, even if there are slight differences in imaging conditions, they will be considered the same. Furthermore, although the above example describes an example where the second correction process is performed on the image data of the fourth imaging condition using the first imaging condition, the second correction process may also be performed on the image data of the first imaging condition using the fourth imaging condition.

[0158] This applies not only to the metering range used when performing the exposure calculation process described above, but also to the metering (color measurement) range used when determining the white balance adjustment value, the metering range used when determining whether or not to emit auxiliary light from a light source that emits auxiliary light, and the metering range used when determining the amount of auxiliary light emitted from the above light source.

[0159] Furthermore, when the readout resolution of the photoelectric conversion signal differs between regions of the divided imaging screen, the same treatment can be applied to the region used for determining the imaging scene when determining the readout resolution for each region. In this way, the correction unit 33b performs the first correction process, the second correction process, and the interpolation process, so that even if crushed black pixels 85b and 85d occur, the crushing can be corrected and the shooting conditions can be set. Therefore, even if there are crushed black pixels 85b or 85d, the shooting conditions can be set. In the example above, the shooting conditions were set after the second correction process, but it is also possible to set the shooting conditions using the image data obtained by the first correction process without performing the second correction process.

[0160] <Explanation of the flowchart> Figure 17 is a flowchart illustrating the process of setting imaging conditions for each region and taking an image. When the main switch of camera 1 is turned on, the control unit 34 starts a program that executes the process shown in Figure 17. In step S10, the control unit 34 starts a live view display on the display unit 35 and proceeds to step S20.

[0161] Specifically, the control unit 34 instructs the imaging unit 32 to start acquiring live view images, and the acquired live view images are displayed sequentially on the display unit 35. As described above, at this point, the same imaging conditions are set for the entire area of ​​the imaging chip 111, that is, the entire screen. Furthermore, if the setting is configured to perform AF operation during live view display, the lens movement control unit 34d of the control unit 34 controls the AF operation to focus on the subject element corresponding to a predetermined focus point by performing focus detection processing. The lens movement control unit 34d performs the first correction processing and the second correction processing, or the first correction processing or the second correction processing, as necessary, before performing focus detection processing. Furthermore, if the setting to perform AF operation during live view display is not configured, the lens movement control unit 34d of the control unit 34 will perform AF operation when AF operation is instructed later.

[0162] In step S20, the object detection unit 34a of the control unit 34 detects subject elements from the live view image and proceeds to step S30. The object detection unit 34a performs the first correction process and the second correction process, or the first correction process or the second correction process, before performing the subject detection process, if necessary. In step S30, the setting unit 34b of the control unit 34 divides the live view image screen into areas containing subject elements and proceeds to step S40.

[0163] In step S40, the control unit 34 displays the regions on the display unit 35. As illustrated in Figure 6, the control unit 34 highlights the region among the divided regions that is the target of setting (changing) the imaging conditions. The control unit 34 then displays the imaging condition setting screen 70 on the display unit 35 and proceeds to step S50. Furthermore, if the user taps the display position of another main subject on the display screen with their finger, the control unit 34 changes the area containing that main subject to be changed and highlights it as an area subject to setting (changing) the imaging conditions.

[0164] In step S50, the control unit 34 determines whether or not AF operation is necessary. For example, if the focus adjustment state changes due to the movement of the subject, if the position of the focus point is changed by user operation, or if the user instructs the execution of AF operation, the control unit 34 determines step S50 to be positive and proceeds to step S70. If the focus adjustment state does not change, the position of the focus point is not changed by user operation, and the user does not instruct the execution of AF operation, the control unit 34 determines step S50 to be negative and proceeds to step 60.

[0165] In step S70, the control unit 34 performs the AF operation and returns to step S40. The lens movement control unit 34d performs the first correction process and the second correction process, or the first correction process or the second correction process, followed by the focus detection process, which is the AF operation, as necessary. The control unit 34, having returned to step S40, repeats the same process as described above based on the live view image acquired after the AF operation.

[0166] In step S60, the setting unit 34b of the control unit 34 sets the imaging conditions for the highlighted area according to the user operation and proceeds to step S80. The display transitions of the display unit 35 and the setting of imaging conditions in response to the user operation in step S60 are as described above. The setting unit 34b of the control unit 34 performs the exposure calculation process after the first correction process and the second correction process, or after the first correction process or the second correction process has been performed, as necessary.

[0167] In step S80, the control unit 34 determines whether or not an imaging instruction has been given. If the release button (not shown) or the display icon indicating imaging, which constitutes the operating member 36, is operated, the control unit 34 affirms step S80 and proceeds to step S90. If no imaging instruction has been given, the control unit 34 negates step S80 and returns to step S60.

[0168] In step S90, the control unit 34 performs a predetermined imaging process. That is, the imaging control unit 34c controls the image sensor 32a to take images according to the imaging conditions set for each region, and then proceeds to step S100.

[0169] In step S100, the imaging control unit 34c of the control unit 34 sends an instruction to the image processing unit 33 to perform predetermined image processing on the image data obtained by the imaging, and then proceeds to step S110. The image processing includes the pixel defect correction process, color interpolation process, edge enhancement process, and noise reduction process. Furthermore, the correction unit 33b of the image processing unit 33 performs the first correction process and the second correction process, or the first correction process or the second correction process, on image data located at the boundary of a region, as necessary, before performing image processing.

[0170] In step S110, the control unit 34 sends an instruction to the recording unit 37 to record the image data after image processing on a recording medium (not shown) and proceed to step S120.

[0171] In step S120, the control unit 34 determines whether or not a termination operation has been performed. If a termination operation has been performed, the control unit 34 affirms step S120 and terminates the process shown in Figure 17. If a termination operation has not been performed, the control unit 34 negates step S120 and returns to step S20. If the process returns to step S20, the control unit 34 repeats the process described above.

[0172] In the above explanation, a stacked image sensor 100 was used as an example for the image sensor 32a, but it is not necessary to configure it as a stacked image sensor as long as imaging conditions can be set for each of the multiple blocks in the image sensor (imaging chip 111).

[0173] According to the first embodiment described above, the following effects and advantages can be obtained. (1) The camera 1 equipped with an image processing device includes an imaging unit 32 having a first region 61 that is imaged under first imaging conditions and a fourth region 64 that is imaged under fourth imaging conditions different from the first imaging conditions, and an image processing unit 33 that generates an image of the subject imaged in the first region 61 using image data of the subject imaged in the fourth region 64. As a result, for example, if inappropriate image data is generated in the image of the subject imaged in the first region 61 due to the imaging conditions set for the first region 61, appropriate image data can be generated using the image of the subject imaged in the fourth region 64 under imaging conditions different from those of the first region 61. Specifically, it is possible to generate image data that suppresses image discontinuities and inconsistencies such as differences in brightness, contrast, and color between the shaded areas of blocks 82, 85, and 87 in Figure 7(b) and the shaded areas of blocks 83, 86, 88, and 89.

[0174] (2) The camera 1 equipped with an image processing device includes an imaging unit 32 having a first region 61 that is imaged under first imaging conditions and a fourth region 64 that is imaged under fourth imaging conditions different from the first imaging conditions, and an image processing unit 33 that generates image data of a subject imaged in the first region 61 using image data of the subject imaged in the fourth region 64. This allows, for example, if inappropriate image data is generated in the image of a subject imaged in the first region 61 due to the imaging conditions set for the first region 61, appropriate image data can be generated using the image of the subject imaged in the fourth region 64 under imaging conditions different from those of the first region 61. Specifically, it is possible to generate image data that suppresses discontinuities and inconsistencies in images, such as differences in brightness, contrast, and color between the shaded areas of blocks 82, 85, and 87 in Figure 7(b) and the shaded areas of blocks 83, 86, 88, and 89.

[0175] (3) The image processing unit 33 generates image data of the subject image captured in the first region 61 using the image data of the subject captured in the fourth region 64. This allows, for example, if inappropriate image data is generated in the image of the subject captured in the first region 61 due to the imaging conditions set in the first region 61, the image data can be appropriately generated using the image of the subject captured in the fourth region 64 under different imaging conditions than those set in the first region 61.

[0176] (4) The image processing unit 33 generates image data of a portion of the subject captured in the first region 61 using image data of the subject captured in the fourth region 64. As a result, if inappropriate image data is generated in a portion of the image captured in the first region 61 (the block of interest), appropriate image data can be generated using image data of the image captured in the fourth region 64 (the reference block), which has different imaging conditions than the first region 61.

[0177] (5) The image processing unit 33 generates image data of a portion of the subject captured in the first region 61 using image data of the subject captured in the fourth region 64. A portion of the subject captured in the first region 61 and at least a portion of the subject captured in the fourth region 64 are detected as the same subject. As a result, when the same subject, for example, the mountain in Figure 7, is captured in both the first region 61 and the fourth region 64, the image data of the mountain obtained from capturing in the first region 61 can be replaced with the image data of the mountain captured in the fourth region 64. Therefore, image data can be generated appropriately.

[0178] (6) The area of ​​the fourth region 64 is larger than the area of ​​the first region 61. As a result, image data of the subject captured in the first region 61 is generated based on the image data of the subject captured in the fourth region 64, which is larger than the area of ​​the first region 61, so that image data can be generated appropriately.

[0179] (7) The area of ​​the fourth region 64 is less than or equal to the area of ​​the first region 61. That is, when replacing image data acquired in the block of interest with image data acquired in the reference block, the computational load in the replacement process can be suppressed by replacing multiple image data corresponding to multiple pixels in the block of interest with the same number or fewer image data.

[0180] (8) The image processing unit 33 generates image data of the subject captured in the first region 61 using image data of the subject captured in a part of the fourth region 64. As a result, if inappropriate image data is generated in a part of the image captured in the first region 61 (the block of interest), appropriate image data can be generated using a part of the image captured in the fourth region 64 (the reference block), which has different imaging conditions than the first region 61.

[0181] (9) The fourth region 64 has a pixel 86a that includes a photoelectric conversion unit that converts light into electric charge, and a pixel 86b that is different from the pixel 86a that includes a photoelectric conversion unit that converts light into electric charge. The image processing unit 33 generates image data of the subject captured in the first region 61 using image data of the subject captured by either of the pixels 86a or 86b. As a result, for example, if inappropriate image data is generated in the image of the subject captured in the first region 61 due to the imaging conditions set in the first region 61, appropriate image data can be generated using the image of the subject captured in the fourth region 64 under imaging conditions different from those of the first region 61.

[0182] (10) If the distance between the first region 61 and pixel 86a is shorter than the distance between the first region 61 and pixel 86b, the image processing unit 33 generates image data of the subject captured in the first region 61 using image data of the subject captured by pixel 86a. That is, when replacing image data acquired in the block of interest with image data acquired in the reference block, the replacement is performed using image data from pixels in the reference block that are closer to the pixel to be replaced in the block of interest. This makes it possible to generate image data appropriately.

[0183] (11) The image processing unit 33 generates image data of the subject captured in the first region 61 using data calculated from the image data of the subject captured in the fourth region 64. This allows, for example, if inappropriate image data is generated in the image of the subject captured in the first region 61 due to the imaging conditions set in the first region 61, the image data can be appropriately generated using the image of the subject captured in the fourth region 64 under different imaging conditions than those of the first region 61.

[0184] (12) The image processing unit 33 generates image data of the subject captured in the first region 61 using data calculated by averaging the image data of the subject captured in the fourth region 64. This allows, for example, if inappropriate image data is generated in the image of the subject captured in the first region 61 due to the imaging conditions set in the first region 61, appropriate image data can be generated using the image of the subject captured in the fourth region 64 under different imaging conditions than those set in the first region 61.

[0185] (13) The image sensor 100 includes an imaging chip 111 that includes a first region 61 and a fourth region 64, and a signal processing chip 112 connected to the imaging chip 111 and having an analog-to-digital (A / D) conversion circuit that converts image data output from the imaging chip 111 into digital data. This allows for appropriate processing in regions with different imaging conditions. In other words, it is possible to appropriately generate an image using image data generated in each region.

[0186] (14) The image sensor 100 is connected to a signal processing chip 112 and has a memory chip 113 which has a storage unit for storing image data converted into digital data by the signal processing chip 112. The signal processing chip 112 is located between the image chip 111 and the memory chip 113. As a result, each chip is stacked in accordance with the data flow in the image sensor 100, so that the chips can be electrically connected to each other efficiently.

[0187] (15) The camera 1 equipped with a lens adjustment device includes an imaging unit 32 having a first region 61 that images light from the imaging optical system 31 under first imaging conditions and a fourth region 64 that images light from the imaging optical system 31 under fourth imaging conditions different from the first imaging conditions, and an image processing unit 33 and a control unit 34 that move the imaging optical system 31 to adjust the image of the light incident on the first region 61 based on a signal based on the subject imaged in the fourth region 64. As a result, for example, if inappropriate signal data occurs in the signal data of the image imaged in the first region 61 due to the imaging conditions, appropriate signal data can be generated using the signal data of the image imaged in the fourth region 64 under different imaging conditions than the first region 61. Specifically, it is possible to generate signal data that suppresses image discontinuities such as differences in brightness, contrast, etc. between the shaded areas of blocks 82, 85, and 87 and the shaded areas of blocks 83, 86, 88, and 89. As a result, the decrease in focus detection accuracy due to differences in imaging conditions for each block can be suppressed, enabling proper focusing.

[0188] (16) The camera 1 equipped with a lens adjustment device includes an imaging unit 32 having a first region 61 that images light from the imaging optical system 31 under first imaging conditions and a fourth region 64 that images light from the imaging optical system 31 under fourth imaging conditions different from the first imaging conditions, and an image processing unit 33 and a control unit 34 that generate signals for adjusting the image of light incident on the first region 61 in the imaging optical system 31 based on signals from a subject imaged in the fourth region 64. As a result, for example, if inappropriate signal data occurs in the signal data of the image imaged in the first region 61 due to the imaging conditions, appropriate signal data can be generated using the signal data of the image imaged in the fourth region 64 under imaging conditions different from those of the first region 61. Specifically, it is possible to generate signal data that suppresses image discontinuities such as differences in brightness, contrast, etc. between the shaded areas of blocks 82, 85, and 87 and the shaded areas of blocks 83, 86, 88, and 89. As a result, the decrease in focus detection accuracy due to differences in imaging conditions for each block can be suppressed, enabling proper focusing.

[0189] (17) The image processing unit 33 generates a signal based on the subject captured in the first region 61 using the signal based on the subject captured in the fourth region 64. This allows, for example, if inappropriate signal data occurs in the signal data of the image captured in the first region 61 due to the imaging conditions, the system to generate appropriate signal data using the signal data of the image captured in the fourth region 64 under different imaging conditions than those of the first region 61.

[0190] (18) The image processing unit 33 generates a signal based on a portion of the subject captured in the first region 61 using a signal based on the subject captured in the fourth region 64. As a result, if inappropriate signal data is generated in a portion of the image (the block of interest) captured in the first region 61, the signal data can be appropriately generated using the signal data of the image captured in the fourth region 64 under different imaging conditions than those of the first region 61. As a result, even for a portion of the subject, the influence of differences in imaging conditions for each block can be avoided, and appropriate focusing becomes possible.

[0191] (19) The image processing unit 33 generates a signal based on a portion of the subject captured in the first region 61 using a signal based on the subject captured in the fourth region 64. A portion of the subject captured in the first region 61 and at least a portion of the subject captured in the fourth region 64 are detected as the same subject. As a result, when the same subject, for example, the mountain in Figure 7, is captured in the first region 61 and the fourth region 64, the signal data of the image of the mountain obtained from capturing in the first region 61 can be used with the signal data of the image of the mountain captured in the fourth region 64 to appropriately generate signal data. Therefore, proper focusing becomes possible.

[0192] (20) The area of ​​the fourth region 64 is larger than the area of ​​the first region 61. This allows for the generation of appropriate signal data based on the signal data of the image captured in the fourth region 64, which is larger than the area of ​​the first region 61. Therefore, proper focusing becomes possible.

[0193] (21) The area of ​​the fourth region 64 is less than or equal to the area of ​​the first region 61. In other words, when generating appropriate signal data, the computational load in the replacement process can be suppressed by replacing the signal data of multiple images corresponding to multiple pixels in the block of interest with the same number or fewer signal data of images.

[0194] (22) The image processing unit 33 generates a signal based on the subject captured in the first region 61 using a signal based on the subject captured in a part of the fourth region 64. As a result, if inappropriate signal data is generated in a part of the image captured in the first region 61 (the block of interest), the signal data can be appropriately generated using the signal data of the image captured in a part of the image captured in the fourth region 64 (the reference block) under different imaging conditions than those of the first region 61. As a result, proper focusing becomes possible without being affected by differences in imaging conditions for each block.

[0195] (23) The fourth region 64 has a pixel 86a that includes a photoelectric conversion unit that converts light into electric charge, and a pixel 86b that is different from the pixel 86a that includes a photoelectric conversion unit that converts light into electric charge. The image processing unit 33 generates a signal based on the subject captured in the first region 61 using a signal based on the subject captured in either region of pixels 86a or pixels 86b. As a result, for example, if inappropriate signal data occurs in the signal data of the image captured in the first region 61 due to the imaging conditions, appropriate signal data can be generated using the signal data of the image captured in the fourth region 64 under different imaging conditions than those of the first region 61.

[0196] (24) If the distance between the first region 61 and pixel 86a is shorter than the distance between the first region 61 and pixel 86b, the image processing unit 33 generates a signal based on the subject captured in the first region 61 using a signal based on the subject captured by pixel 86a. That is, when replacing the signal data of the image acquired in the block of interest with the signal data of the image acquired in the reference block, the replacement is performed using the signal data of the image from the pixel in the reference block that is closer to the pixel to be replaced in the block of interest. This makes it possible to generate appropriate signal data.

[0197] (25) The image processing unit 33 generates a signal based on the subject captured in the first region 61 using data calculated from the signal data based on the subject captured in the fourth region 64. This allows, for example, if inappropriate signal data occurs in the signal data of the image captured in the first region 61 due to the imaging conditions, the system to generate appropriate signal data using the signal data of the image captured in the fourth region 64 under different imaging conditions than those of the first region 61.

[0198] (26) The image processing unit 33 generates a signal based on the subject captured in the first region 61 using data calculated by averaging the signal data based on the subject captured in the fourth region 64. This allows, for example, if inappropriate signal data occurs in the signal data of the image captured in the first region 61 due to imaging conditions, to appropriately generate signal data using the signal data of the image captured in the fourth region 64 under different imaging conditions than those of the first region 61.

[0199] (27) The lens movement control unit 34d of the camera 1 generates a signal to drive the imaging optical system 31 based on the amount of image shift of multiple light images that have passed through different pupils of the imaging optical system 31. This makes it possible to perform proper focusing without being affected by differences in imaging conditions for each block when performing focus detection processing using pupil division phase difference detection.

[0200] (28) The lens movement control unit 34d of the camera 1 generates a signal to drive the imaging optical system 31 based on the contrast of the image caused by the light incident on the imaging unit 32. This makes it possible to perform proper focusing when performing contrast detection type focus detection processing without being affected by differences in imaging conditions for each block.

[0201] (29) Camera 1 equipped with a subject detection device includes an imaging unit 32 having a first region 61 that is imaged under first imaging conditions and a fourth region 64 that is imaged under fourth imaging conditions different from the first imaging conditions, The system includes an image processing unit 33 and a control unit 34 that detect a subject captured in the first region 61 using image data of the subject captured in the fourth region 64. This allows, for example, if inappropriate image data is generated in the image data of the image captured in the first region 61 due to imaging conditions, appropriate image data can be generated using image data of the image captured in the fourth region 64 under different imaging conditions than those of the first region 61. Specifically, it is possible to generate image data that suppresses image discontinuities such as differences in brightness, contrast, and hue between the shaded areas of blocks 82, 85, and 87 and the shaded areas of blocks 83, 86, 88, and 89. As a result, it is possible to suppress a decrease in the detection accuracy of subject elements due to differences in imaging conditions for each block.

[0202] (30) Camera 1 equipped with a subject detection device has an imaging unit 32 having a first region 61 that is imaged under first imaging conditions and a fourth region 64 that is imaged under fourth imaging conditions different from the first imaging conditions, The system includes an image processing unit 33 and a control unit 34 that generate a signal for detecting a subject captured in the first region 61 using image data of the subject captured in the fourth region 64. This allows, for example, if inappropriate image data is generated in the image data of the image captured in the first region 61 due to imaging conditions, appropriate image data can be generated using image data of the image captured in the fourth region 64 under different imaging conditions than those of the first region 61. Specifically, it is possible to generate image data that suppresses image discontinuities such as differences in brightness, contrast, and hue between the shaded areas of blocks 82, 85, and 87 and the shaded areas of blocks 83, 86, 88, and 89. As a result, it is possible to suppress a decrease in the detection accuracy of subject elements due to differences in imaging conditions for each block.

[0203] (31) The image processing unit 33 generates image data of the subject captured in the first region 61 using image data of the subject captured in the fourth region 64. This allows, for example, if inappropriate image data is generated in the image data of the image captured in the first region 61 due to the imaging conditions, to appropriately generate image data using image data of the image captured in the fourth region 64 under different imaging conditions than those of the first region 61. As a result, it is possible to suppress a decrease in the detection accuracy of subject elements due to differences in imaging conditions for each block.

[0204] (32) The image processing unit 33 generates image data of a portion of the subject captured in the first region 61 using image data of the subject captured in the fourth region 64. As a result, if inappropriate image data is generated in a portion of the image captured in the first region 61 (the block of interest), appropriate image data can be generated using image data of the image captured in the fourth region 64, which has different imaging conditions than the first region 61. As a result, it becomes possible to appropriately detect subject elements even for a portion of the subject, avoiding the influence of differences in imaging conditions for each block.

[0205] (33) The image processing unit 33 generates image data of a portion of the subject captured in the first region 61 using image data of the subject captured in the fourth region 64. A portion of the subject captured in the first region 61 and at least a portion of the subject captured in the fourth region 64 are detected as the same subject. As a result, when the same subject, for example, the mountain in Figure 7, is captured in both the first region 61 and the fourth region 64, the image data of the mountain obtained from capturing in the first region 61 can be replaced with the image data of the mountain captured in the fourth region 64. Therefore, image data can be generated appropriately. As a result, it becomes possible to appropriately detect subject elements without being affected by differences in imaging conditions for each block.

[0206] (34) The area of ​​the fourth region 64 is larger than the area of ​​the first region 61. As a result, image data of the subject captured in the first region 61 is generated based on the image data of the subject captured in the fourth region 64, which is larger than the area of ​​the first region 61, so that image data can be generated appropriately. As a result, it becomes possible to appropriately detect subject elements without being affected by differences in imaging conditions for each block.

[0207] (35) The area of ​​the fourth region 64 is less than or equal to the area of ​​the first region 61. That is, when replacing image data acquired in the block of interest with image data acquired in the reference block, the computational load in the replacement process can be suppressed by replacing multiple image data corresponding to multiple pixels in the block of interest with the same number or fewer image data. Therefore, the computational load related to the detection of subject elements can be suppressed.

[0208] (36) The image processing unit 33 generates image data of the subject captured in the first region 61 using image data of the subject captured in a part of the fourth region 64. As a result, if inappropriate image data is generated in a part of the image captured in the first region 61 (the block of interest), appropriate image data can be generated using a part of the image captured in the fourth region 64 (the reference block), which has different imaging conditions from the first region 61. As a result, it becomes possible to appropriately detect subject elements without being affected by differences in imaging conditions for each block.

[0209] (37) The fourth region 64 has a pixel 86a that includes a photoelectric conversion unit that converts light into electric charge, and a pixel 86b that is different from the pixel 86a that includes a photoelectric conversion unit that converts light into electric charge. The image processing unit 33 generates image data of the subject captured in the first region 61 using image data of the subject captured in either region of pixels 86a or pixels 86b. As a result, for example, if inappropriate image data is generated in the image data of the image captured in the first region 61 due to the imaging conditions, appropriate image data can be generated using image data of the image captured in the fourth region 64 under different imaging conditions than those of the first region 61. As a result, it is possible to suppress a decrease in the detection accuracy of subject elements due to differences in imaging conditions for each block.

[0210] (38) If the distance between the first region 61 and pixel 86a is shorter than the distance between the first region 61 and pixel 86b, the image processing unit 33 generates image data of the subject captured in the first region 61 using image data of the subject captured by pixel 86a. That is, when replacing image data acquired in the block of interest with image data acquired in the reference block, the replacement is performed using image data from pixels in the reference block that are closer to the pixel to be replaced in the block of interest. This makes it possible to generate image data appropriately. As a result, it is possible to suppress a decrease in the detection accuracy of subject elements due to differences in imaging conditions for each block.

[0211] (39) The image processing unit 33 generates image data of the subject captured in the first region 61 using data calculated from the image data of the subject captured in the fourth region 64. This allows, for example, if inappropriate image data is generated in the image data of the image captured in the first region 61 due to the imaging conditions, to appropriately generate image data using the image data of the image captured in the fourth region 64 under different imaging conditions than those of the first region 61. As a result, it is possible to suppress a decrease in the detection accuracy of subject elements due to differences in imaging conditions for each block.

[0212] (40) The image processing unit 33 generates image data of the subject captured in the first region 61 using data calculated by averaging the image data of the subject captured in the fourth region 64. As a result, for example, if inappropriate image data is generated in the image data of the image captured in the first region 61 due to the imaging conditions, appropriate image data can be generated using the image data of the image captured in the fourth region 64 under different imaging conditions than those of the first region 61. As a result, it is possible to suppress a decrease in the detection accuracy of subject elements due to differences in imaging conditions for each block.

[0213] (41) The object detection unit 34a of the camera 1 detects subject elements as targets for focusing the imaging optical system 31, so it can appropriately detect targets for focusing the imaging optical system 31 without being affected by differences in imaging conditions for each block.

[0214] (42) Camera 1 is equipped with a control unit 34 that detects the brightness of a subject, and the object detection unit 34a detects subject elements as targets for photometering by the control unit 34, so that it can appropriately detect targets for photometering without being affected by differences in imaging conditions for each block.

[0215] (43) Camera 1 includes an imaging unit 32 having a first region 61 that is imaged under first imaging conditions and a fourth region 64 that is imaged under fourth imaging conditions different from the first imaging conditions, and an image processing unit 33 and a control unit 34 that set the imaging conditions of the first region 61 based on a signal based on a subject imaged in the fourth region 64. As a result, for example, if inappropriate image data is generated in the image data of an image captured in the first region 61 due to the imaging conditions, appropriate image data can be generated using the image data of an image captured in the fourth region 64 under different imaging conditions from the first region 61. Specifically, it is possible to generate image data that suppresses image discontinuities such as differences in brightness, contrast, and hue between the shaded areas of blocks 82, 85, and 87 and the shaded areas of blocks 83, 86, 88, and 89. As a result, it is possible to suppress a decrease in the accuracy of setting exposure conditions due to differences in imaging conditions for each block.

[0216] (44) Camera 1 includes an imaging unit 32 having a first region 61 that is imaged under first imaging conditions and a fourth region 64 that is imaged under fourth imaging conditions different from the first imaging conditions, and an image processing unit 33 and a control unit 34 that generate signals for setting the imaging conditions of the first region 61 based on signals based on the subject imaged in the fourth region 64. As a result, for example, if inappropriate image data is generated in the image data of the image captured in the first region 61 due to the imaging conditions, appropriate image data can be generated using image data of the image captured in the fourth region 64 under imaging conditions different from those of the first region 61. Specifically, it is possible to generate image data that suppresses image discontinuities such as differences in brightness, contrast, and hue between the shaded areas of blocks 82, 85, and 87 and the shaded areas of blocks 83, 86, 88, and 89. As a result, it is possible to suppress a decrease in the accuracy of setting exposure conditions due to differences in imaging conditions for each block.

[0217] (45) The image processing unit 33 generates a signal based on the subject captured in the first region 61 using a signal based on the subject captured in the fourth region 64. This allows, for example, if inappropriate image data is generated in the image data of the image captured in the first region 61 due to the imaging conditions, appropriate image data can be generated using the image data of the image captured in the fourth region 64 under different imaging conditions than those of the first region 61. As a result, it is possible to suppress a decrease in the accuracy of setting exposure conditions due to differences in imaging conditions for each block.

[0218] (46) The image processing unit 33 generates a signal based on a part of the subject captured in the first region 61 using a signal based on the subject captured in the fourth region 64. As a result, if inappropriate image data is generated in a part of the image captured in the first region 61 (the block of interest), appropriate image data can be generated using the image data of the image captured in the fourth region 64, which has different imaging conditions than the first region 61. As a result, it becomes possible to set the exposure conditions appropriately without being affected by the differences in imaging conditions for each block.

[0219] (47) The image processing unit 33 generates a signal based on a portion of the subject captured in the first region 61 using a signal based on the subject captured in the fourth region 64. A portion of the subject captured in the first region 61 and at least a portion of the subject captured in the fourth region 64 are detected as the same subject. As a result, when the same subject, for example, the mountain in Figure 7, is captured in both the first region 61 and the fourth region 64, the image data of the mountain captured in the first region 61 can be replaced with the image data of the mountain captured in the fourth region 64. Therefore, image data can be generated appropriately. As a result, it becomes possible to set exposure conditions appropriately without being affected by differences in imaging conditions for each block.

[0220] (48) The area of ​​the fourth region 64 is larger than the area of ​​the first region 61. As a result, image data of the subject captured in the first region 61 is generated based on the image data of the subject captured in the fourth region 64, which is larger than the area of ​​the first region 61, so that image data can be generated appropriately. As a result, it becomes possible to set exposure conditions appropriately without being affected by differences in imaging conditions for each block.

[0221] (49) The area of ​​the fourth region 64 is less than or equal to the area of ​​the first region 61. That is, when replacing image data acquired in the block of interest with image data acquired in the reference block, the computational load in the replacement process can be suppressed by replacing multiple image data corresponding to multiple pixels in the block of interest with the same number or fewer image data. Therefore, the computational load related to setting exposure conditions can be suppressed.

[0222] (50) The image processing unit 33 generates a signal based on the subject captured in the first region 61 using a signal based on the subject captured in a part of the fourth region 64. As a result, if inappropriate image data is generated in a part of the image captured in the first region 61 (the block of interest), the image data can be appropriately generated using a part of the image captured in the fourth region 64 (the reference block), which has different imaging conditions from the first region 61. As a result, it becomes possible to set the exposure conditions appropriately without being affected by the differences in imaging conditions for each block.

[0223] (51) The fourth region 64 has a pixel 86a that includes a photoelectric conversion unit that converts light into electric charge, and a pixel 86b that is different from the pixel 86a that includes a photoelectric conversion unit that converts light into electric charge. The image processing unit 33 generates a signal based on the subject captured in the first region 61 using a signal based on the subject captured in either region of pixels 86a or pixels 86b. As a result, for example, if inappropriate image data is generated in the image data of the image captured in the first region 61 due to the imaging conditions, appropriate image data can be generated using the image data of the image captured in the fourth region 64 under different imaging conditions than those of the first region 61. As a result, it is possible to suppress a decrease in the accuracy of setting exposure conditions due to differences in imaging conditions for each block.

[0224] (52) If the distance between the first region 61 and pixel 86a is shorter than the distance between the first region 61 and pixel 86b, the image processing unit 33 generates a signal based on the subject captured in the first region 61 using a signal based on the subject captured by pixel 86a. That is, when replacing image data acquired in the block of interest with image data acquired in the reference block, the replacement is performed using image data from pixels in the reference block that are closer to the pixel to be replaced in the block of interest. This makes it possible to generate image data appropriately. As a result, it is possible to suppress a decrease in the accuracy of setting exposure conditions due to differences in imaging conditions for each block.

[0225] (53) The image processing unit 33 generates a signal based on the subject captured in the first region 61 using data calculated from the signal data based on the subject captured in the fourth region 64. As a result, for example, if inappropriate image data is generated in the image data of the image captured in the first region 61 due to the imaging conditions, appropriate image data can be generated using the image data of the image captured in the fourth region 64 under different imaging conditions than those of the first region 61. As a result, it is possible to suppress a decrease in the accuracy of setting exposure conditions due to differences in imaging conditions for each block.

[0226] (54) The image processing unit 33 generates a signal based on the subject captured in the first region 61 using data calculated by averaging the signal data based on the subject captured in the fourth region 64. As a result, for example, if inappropriate image data is generated in the image data of the image captured in the first region 61 due to the imaging conditions, appropriate image data can be generated using the image data of the image captured in the fourth region 64 under different imaging conditions than those of the first region 61. As a result, it is possible to suppress a decrease in the accuracy of setting exposure conditions due to differences in imaging conditions for each block.

[0227] (55) The setting unit 34b of camera 1 sets the exposure conditions as shooting conditions, so even if there are differences in imaging conditions for each block, the exposure conditions can be set appropriately for each.

[0228] (56) Camera 1 is equipped with a control unit 34 that controls a light source that emits assistive light for shooting, and the setting unit 34b sets the presence or absence of light emission or the amount of light emission from the light source controlled by the control unit 34 as shooting conditions. This allows for appropriate setting processing even when there are differences in imaging conditions for each block.

[0229] The system may be configured to allow switching between Mode 1, in which the above-described second correction process is performed as a preprocessing step, and Mode 2, in which the second correction process is not performed as a preprocessing step. When Mode 1 is selected, the control unit 34 performs the above-described preprocessing step followed by image processing and other processing. On the other hand, when Mode 2 is selected, the control unit 34 performs image processing and other processing without performing the above-described preprocessing step. For example, if there is a shadow on a part of a face detected as a subject element, and the image is captured with different settings for the imaging conditions of the area including the shadowed part of the face and the area including the area not in shadow, so that the brightness of the shadowed part of the face is about the same as the brightness of the area not in shadow, then if the second correction process is performed on the generated image and then color interpolation processing is performed, unintended color interpolation may occur in the shadowed area due to the difference in the set imaging conditions. By configuring the system to allow switching between Mode 1 and Mode 2 so that color interpolation processing can be performed using the image data as is without performing the second correction process, it becomes possible to avoid unintended color interpolation.

[0230] ---Modification of the first embodiment--- The following modifications are also within the scope of the present invention, and it is possible to combine one or more of these modifications with the embodiments described above. (Variation 1) Figures 18(a) to 18(c) illustrate the arrangement of the first imaging region and the second imaging region on the imaging surface of the image sensor 32a. In the example shown in Figure 18(a), the first imaging region is composed of even-numbered rows, and the second imaging region is composed of odd-numbered rows. In other words, the imaging surface is divided into even-numbered and odd-numbered rows.

[0231] In the example shown in Figure 18(b), the first imaging region is composed of odd-numbered rows, and the second imaging region is composed of even-numbered rows. In other words, the imaging plane is divided into odd-numbered and even-numbered rows.

[0232] In the example shown in Figure 18(c), the first imaging region is composed of blocks of even-numbered rows in odd-numbered columns and blocks of odd-numbered rows in even-numbered columns. Similarly, the second imaging region is composed of blocks of even-numbered rows in even-numbered columns and blocks of odd-numbered rows in odd-numbered columns. In other words, the imaging surface is divided into a checkerboard pattern.

[0233] In all cases shown in Figures 18(a) to 18(c), the photoelectric conversion signal read from the image sensor 32a that captured one frame generates a first image based on the photoelectric conversion signal read from the first imaging area and a second image based on the photoelectric conversion signal read from the second imaging area, respectively. According to Modification 1, the first and second images are captured with the same field of view and contain a common subject image.

[0234] In Modification 1, the control unit 34 uses the first image for display and the second image for detection. Specifically, the control unit 34 displays the first image as a live view image on the display unit 35. The control unit 34 also has the object detection unit 34a perform subject detection processing using the second image, the lens movement control unit 34d perform focus detection processing using the second image, and the setting unit 34b perform exposure calculation processing using the second image.

[0235] In Modification 1, the imaging conditions set for the first imaging area that captures the first image are called the first imaging conditions, and the imaging conditions set for the second imaging area that captures the second image are called the second imaging conditions. The control unit 34 may set the first imaging conditions and the second imaging conditions to be different.

[0236] 1. As an example, the control unit 34 sets the first imaging conditions to conditions suitable for display by the display unit 35. For example, the first imaging conditions set for the first imaging area are made the same for the entire first imaging area of ​​the imaging screen. On the other hand, the control unit 34 sets the second imaging conditions set for the second imaging area to conditions suitable for focus detection processing, subject detection processing, and exposure calculation processing. The second imaging conditions are made the same for the entire second imaging area of ​​the imaging screen. Furthermore, if the conditions suitable for focus detection processing, subject detection processing, and exposure calculation processing differ, the control unit 34 may set different second imaging conditions for the second imaging area for each frame. For example, the second imaging conditions for the first frame may be set to conditions suitable for focus detection processing, the second imaging conditions for the second frame to be conditions suitable for subject detection processing, and the second imaging conditions for the third frame to be conditions suitable for exposure calculation processing. In these cases, the second imaging conditions in each frame are made the same for the entire second imaging area of ​​the imaging screen.

[0237] 2. As another example, the control unit 34 may make the first imaging conditions set for the first imaging area different depending on the area. The setting unit 34b of the control unit 34 sets different first imaging conditions for each area containing subject elements divided by the setting unit 34b. On the other hand, the control unit 34 makes the second imaging conditions set for the second imaging area the same for the entire second imaging area of ​​the imaging screen. The control unit 34 sets the second imaging conditions to conditions suitable for focus detection processing, subject detection processing, and exposure calculation processing, but if the conditions suitable for focus detection processing, subject detection processing, and exposure calculation processing are different, the imaging conditions set for the second imaging area may be different for each frame.

[0238] 3. As another example, the control unit 34 may set the first imaging conditions set for the first imaging area to be the same across the entire first imaging area of ​​the imaging screen, while setting the second imaging conditions set for the second imaging area to be different across the imaging screen. For example, the setting unit 34b may set different second imaging conditions for each area containing the divided subject elements. In this case as well, if the conditions suitable for focus detection processing, subject detection processing, and exposure calculation processing are different, the imaging conditions set for the second imaging area may be different for each frame.

[0239] 4. As another example, the control unit 34 may vary the first imaging conditions set for the first imaging area on the imaging screen, and also vary the second imaging conditions set for the second imaging area on the imaging screen. For example, the setting unit 34b may set different first imaging conditions for each area containing the subject elements it has divided, while simultaneously setting different second imaging conditions for each area containing the subject elements it has divided.

[0240] In Figures 18(a) to 18(c), the area ratio of the first imaging area and the second imaging area may be different. The control unit 34 may, for example, set the ratio of the first imaging area higher than that of the second imaging area, set the ratio of the first and second imaging areas to be the same as illustrated in Figures 18(a) to 18(c), or set the ratio of the first imaging area to be lower than that of the second imaging area, based on user operation or the control unit 34's judgment. By making the area ratio of the first and second imaging areas different, it is possible to make the first image more detailed than the second image, to make the resolution of the first and second images the same, or to make the second image more detailed than the first image.

[0241] (Modification 2) In the embodiment described above, the second correction process when performing image processing involves the correction unit 33b of the image processing unit 33 correcting the image data of the fourth imaging condition (the image data of the fourth imaging condition among the image data of the reference position) based on the first imaging condition when the imaging conditions applied at the point of interest (referred to as the first imaging condition) and the imaging conditions applied at the reference position surrounding the point of interest (referred to as the fourth imaging condition) are different. In other words, by performing the second correction process on the image data of the fourth imaging condition at the reference position, the discontinuity in the image caused by the difference between the first imaging condition and the fourth imaging condition is mitigated.

[0242] Alternatively, in the modified example 2, the correction unit 33b of the image processing unit 33 may correct the image data of the first imaging condition (the image data of the first imaging condition from the image data of the point of interest and the image data of the reference position) based on the fourth imaging condition. In this case as well, the discontinuity in the image due to the difference between the first imaging condition and the fourth imaging condition can be mitigated.

[0243] Alternatively, the correction unit 33b of the image processing unit 33 may correct both the image data for the first imaging condition and the image data for the fourth imaging condition. That is, the second correction process may be applied to the image data of the point of interest for the first imaging condition, the image data of the first imaging condition from the reference position image data, and the image data of the fourth imaging condition from the reference position image data, respectively, in order to mitigate the discontinuity in the image based on the difference between the first imaging condition and the fourth imaging condition. For example, in (Example 1) above, the image data of the reference pixel Pr under the first imaging condition (ISO sensitivity of 100) is multiplied by 400 / 100 as a second correction process, and the image data of the reference pixel Pr under the fourth imaging condition (ISO sensitivity of 800) is multiplied by 400 / 800 as a second correction process. This reduces the difference between image data due to differences in imaging conditions. The pixel data of the pixel of interest is subjected to a second correction process of multiplying by 100 / 400 after color interpolation. This second correction process can change the pixel data of the pixel of interest after color interpolation to the same value as when it was captured under the first imaging condition. Furthermore, in (Example 1) above, the degree of the second correction process may be changed depending on the distance from the boundary between the first and fourth regions. Compared to the case of (Example 1) above, the rate of increase or decrease in image data due to the second correction process can be reduced, and the noise generated by the second correction process can be reduced. The above explanation describes (Example 1), but the same can be applied to (Example 2).

[0244] According to Modification 2, similar to the embodiment described above, appropriate image processing can be performed on the image data generated for each region with different imaging conditions.

[0245] (Variation 3) In the above-described embodiment, when performing the second correction process on the image data, the corrected image data is obtained by performing an operation based on the difference between the first imaging condition and the fourth imaging condition. Instead of the operation, the corrected image data may be obtained by referring to a correction table. For example, the corrected image data is read out by inputting the first imaging condition and the fourth imaging condition as arguments. Alternatively, a configuration may be adopted in which a correction coefficient is read out by inputting the first imaging condition and the fourth imaging condition as arguments.

[0246] (Modification Example 4) In the second correction process of the above-described embodiment, the upper limit and the lower limit of the corrected image data may be determined. By providing the upper limit value and the lower limit value, it is possible to limit so as not to perform unnecessary correction. The upper limit value and the lower limit value may be determined in advance, or when a photometric sensor is provided separately from the imaging device 32a, they may be determined based on the output signal from the photometric sensor.

[0247] (Modification Example 5) In the above embodiment, an example has been described in which the setting unit 34b of the control unit 34 detects a subject element based on the live view image and divides the screen of the live view image into regions including the subject element. In Modification Example 5, when the control unit 34 includes a photometric sensor separately from the imaging device 32a, the control unit 34 may divide the region based on the output signal from the photometric sensor.

[0248] The control unit 34 divides the live view image into a foreground and a background based on the output signal from the photometric sensor. Specifically, the live view image acquired by the imaging device 32a is divided into a foreground region corresponding to the region determined as the foreground from the output signal of the photometric sensor and a background region corresponding to the region determined as the background from the output signal of the photometric sensor.

[0249] The control unit 34 further arranges the first imaging region and the second imaging region at positions corresponding to the foreground region of the imaging surface of the imaging device 32a, as illustrated in FIGS. 18(a) to 18(c). On the other hand, the control unit 34 arranges only the first imaging region on the imaging surface of the imaging device 32a at a position corresponding to the background region of the imaging surface of the imaging device 32a. The control unit 34 uses the first image for display and the second image for detection.

[0250] According to Modification 5, by using the output signal from the photometric sensor, it is possible to perform region division of the live view image acquired by the imaging device 32a. Further, for the foreground region, it is possible to obtain the first image for display and the second image for detection, and for the background region, it is possible to obtain only the first image for display.

[0251] (Modification 6) In Modification 6, the generation unit 33c of the image processing unit 33 performs contrast adjustment processing as an example of the second correction processing. That is, the generation unit 33c relaxes the image discontinuity based on the difference between the first imaging condition and the fourth imaging condition by varying the tone curve (gamma curve).

[0252] For example, assume that only the ISO sensitivity is different between the first imaging condition and the fourth imaging condition, the ISO sensitivity of the first imaging condition is 100, and the ISO sensitivity of the fourth imaging condition is 800. The generation unit 33c compresses the value of the image data of the fourth imaging condition among the image data at the reference position to 1 / 8 by flattening the tone curve.

[0253] Alternatively, the generation unit 33c may expand the values of the image data at the attention position and the image data of the first imaging condition among the image data at the reference position by 8 times by raising the tone curve.

[0254] According to Modification 6, similar to the embodiment described above, image processing can be appropriately performed on image data generated in regions with different imaging conditions. For example, discontinuities and inconsistencies that appear in the processed image due to differences in imaging conditions at the boundaries of the regions can be suppressed.

[0255] (Example 7) In Modification 7, the image processing unit 33 ensures that the contours of the subject elements are not damaged during the image processing described above (for example, noise reduction processing). Generally, when noise reduction is performed, a smoothing filter is used. When a smoothing filter is used, while noise reduction is achieved, the boundaries of the subject elements may become blurred.

[0256] Therefore, the generation unit 33c of the image processing unit 33 compensates for the blurring of the boundaries of the subject elements by performing contrast adjustment processing in addition to, or together with, noise reduction processing. In the modified example 7, the generation unit 33c of the image processing unit 33 sets a curve that draws an S shape as the density conversion (grayscale conversion) curve (so-called S-curve conversion). By performing contrast adjustment using S-curve conversion, the generation unit 33c of the image processing unit 33 stretches the grayscale portions of the bright data and the dark data respectively to increase the number of grayscales of the bright data (and dark data), and compresses the image data of the intermediate grayscale to reduce the number of grayscales. As a result, the number of image data with moderate brightness decreases and the number of data classified as either bright or dark increases, thereby compensating for the blurring of the boundaries of the subject elements.

[0257] According to variation 7, blurring of the boundaries of subject elements can be compensated for by making the light and dark areas of the image sharper.

[0258] (Variation 8) In the modified example 8, the generation unit 33c of the image processing unit 33 changes the white balance adjustment gain so as to mitigate the discontinuity in the image caused by the difference between the first imaging condition and the fourth imaging condition.

[0259] For example, if the imaging conditions applied at the point of interest (referred to as the first imaging conditions) are different from the imaging conditions applied at the reference points surrounding the point of interest (referred to as the fourth imaging conditions), the generation unit 33c of the image processing unit 33 changes the white balance adjustment gain so that the white balance of the image data under the fourth imaging conditions among the image data at the reference points approaches the white balance of the image data acquired under the first imaging conditions.

[0260] Furthermore, the generation unit 33c of the image processing unit 33 may change the white balance adjustment gain so that the white balance of the image data under the first imaging condition and the image data at the point of interest among the image data at the reference position is brought closer to the white balance of the image data acquired under the fourth imaging condition.

[0261] According to Modification 8, by adjusting the white balance adjustment gain of the image data generated in regions with different imaging conditions to the adjustment gain of one of the regions with different imaging conditions, the discontinuity in the image caused by the difference between the first imaging condition and the fourth imaging condition can be mitigated.

[0262] (Extreme variation 9) Multiple image processing units 33 may be provided to perform image processing in parallel. For example, image processing may be performed on image data captured in area A of the imaging unit 32 while simultaneously performing image processing on image data captured in area B of the imaging unit 32. Multiple image processing units 33 may perform the same image processing or different image processing. That is, the same parameters can be applied to the image data of areas A and B to perform similar image processing, or different parameters can be applied to the image data of areas A and B to perform different image processing.

[0263] In cases where there are multiple image processing units 33, one image processing unit may perform image processing on image data to which the first imaging condition is applied, and another image processing unit may perform image processing on image data to which the fourth imaging condition is applied. The number of image processing units is not limited to the two described above; for example, the number of units may be equal to the number of imaging conditions that can be set. That is, each image processing unit is responsible for image processing for each region to which different imaging conditions are applied. According to Modification 9, imaging with different imaging conditions for each region and image processing of the image data obtained for each region can be carried out in parallel.

[0264] (Variation 10) In the explanation above, camera 1 was used as an example, but the system may also be composed of a high-performance mobile phone 250 (Figure 20) equipped with a camera function, such as a smartphone, or a mobile device such as a tablet terminal.

[0265] (Variation 11) In the embodiments described above, a camera 1 was described as an example in which the imaging unit 32 and the control unit 34 are configured as a single electronic device. Alternatively, for example, an imaging system 1B may be configured in which the imaging unit 32 and the control unit 34 are provided separately, and the imaging unit 32 is controlled from the control unit 34 via communication. The following describes an example in which an imaging device 1001 equipped with an imaging unit 32 is controlled from a display device 1002 equipped with a control unit 34, with reference to Figure 19.

[0266] Figure 19 is a block diagram illustrating the configuration of the imaging system 1B according to the modified example 11. In Figure 19, the imaging system 1B is composed of an imaging device 1001 and a display device 1002. The imaging device 1001 includes the imaging optical system 31 and imaging unit 32 described in the above embodiment, as well as a first communication unit 1003. The display device 1002 includes the image processing unit 33, control unit 34, display unit 35, operating member 36, and recording unit 37 described in the above embodiment, as well as a second communication unit 1004.

[0267] The first communication unit 1003 and the second communication unit 1004 can perform two-way image data communication by, for example, well-known wireless communication technologies, optical communication technologies, and the like. Alternatively, the imaging device 1001 and the display device 1002 may be connected by a wired cable for wired connection, and the first communication unit 1003 and the second communication unit 1004 may be configured to perform two-way image data communication.

[0268] In the imaging system 1B, the control unit 34 controls the imaging unit 32 by performing data communication via the second communication unit 1004 and the first communication unit 1003. For example, by transmitting and receiving predetermined control data between the imaging device 1001 and the display device 1002, the display device 1002 divides the screen into a plurality of regions based on the image as described above, sets different imaging conditions for each divided region, or reads out the photoelectric conversion signals photoelectrically converted in each region.

[0269] According to the modified example 11, since the live view image acquired on the imaging device 1001 side and transmitted to the display device 1002 is displayed on the display unit 35 of the display device 1002, the user can perform remote operations from the display device 1002 located at a position away from the imaging device 1001. The display device 1002 can be configured by a high-function mobile phone 250 such as a smartphone. Further, the imaging device 1001 can be configured by an electronic device including the stacked imaging element 100 described above. Although an example in which the control unit 34 of the display device 1002 is provided with an object detection unit 34a, a setting unit 34b, an imaging control unit 34c, and a lens movement control unit 34d has been described, a part of the object detection unit 34a, the setting unit 34b, the imaging control unit 34c, and the lens movement control unit 34d may be provided in the imaging device 1001.

[0270] (Modified Example 12) The program can be supplied to the aforementioned camera 1, high-performance mobile phone 250, or mobile device such as a tablet terminal by transmitting it from a personal computer 205 that stores the program to the mobile device via infrared communication or short-range wireless communication, as illustrated in Figure 20.

[0271] The program for the personal computer 205 may be supplied by inserting a recording medium 204, such as a CD-ROM, containing the program into the personal computer 205, or by loading it into the personal computer 205 via a communication line 201, such as a network. If using the communication line 201, the program is stored in a storage device 203 of a server 202 connected to the communication line.

[0272] Furthermore, the program can be directly transmitted to a mobile device via a wireless LAN access point (not shown) connected to the communication line 201. Alternatively, a recording medium 204B, such as a memory card containing the program, may be inserted into the mobile device. In this way, the program can be supplied as a computer program product in various forms, including via recording media and communication lines.

[0273] ---Second Embodiment--- Referring to Figures 21-27, a digital camera will be used as an example of an electronic device equipped with the image processing apparatus according to the second embodiment. In the following description, the same reference numerals are used for components that are the same as in the first embodiment, and the differences will be mainly explained. Points that are not specifically explained are the same as in the first embodiment. This embodiment differs from the first embodiment mainly in that, instead of providing the image processing unit 33 of the first embodiment, the imaging unit 32A further includes an image processing unit 32c that has the same function as the image processing unit 33 of the first embodiment.

[0274] Figure 21 is a block diagram illustrating the configuration of camera 1C according to the second embodiment. In Figure 21, camera 1C includes an imaging optical system 31, an imaging unit 32A, a control unit 34, a display unit 35, an operating member 36, and a recording unit 37. The imaging unit 32A further includes an image processing unit 32c having the same functions as the image processing unit 33 of the first embodiment.

[0275] The image processing unit 32c includes an input unit 321, a correction unit 322, and a generation unit 323. Image data from the image sensor 32a is input to the input unit 321. The correction unit 322 performs preprocessing to correct the input image data. The preprocessing performed by the correction unit 322 is the same as the preprocessing performed by the correction unit 33b in the first embodiment. The generation unit 323 performs image processing on the input image data and the preprocessed image data to generate an image. The image processing performed by the generation unit 323 is the same as the image processing performed by the generation unit 33c in the first embodiment.

[0276] Figure 22 is a schematic diagram showing the correspondence between each block and the multiple correction units 322 in this embodiment. In Figure 22, one square of the imaging chip 111, represented by a rectangle, represents one block 111a. Similarly, one square of the image processing chip 114, which will be described later, represented by a rectangle, represents one correction unit 322.

[0277] In this embodiment, the correction unit 322 is provided in correspondence with each block 111a. In other words, the correction unit 322 is provided for each block, which is the smallest unit of the area on the imaging plane where the imaging conditions can be changed. For example, in Figure 22, the hatched block 111a corresponds to the hatched correction unit 322. In Figure 22, the hatched correction unit 322 performs preprocessing on the image data from the pixels included in the hatched block 111a. Each correction unit 322 performs preprocessing on the image data from the pixels included in the corresponding block 111a. This allows image data preprocessing to be performed in parallel by multiple correction units 322, reducing the processing load on the correction units 322 and enabling the generation of appropriate images in a short time from image data generated in areas with different imaging conditions. In the following explanation, when describing the relationship between a block 111a and the pixels contained within it, the block 111a may be referred to as the block 111a to which the pixel belongs. Furthermore, a block 111a may be referred to as a unit division, and a collection of multiple blocks 111a, i.e., a collection of multiple unit divisions, may be referred to as a composite division.

[0278] Figure 23 is a cross-sectional view of the stacked image sensor 100A. The stacked image sensor 100A comprises a back-illuminated imaging chip 111, a signal processing chip 112, a memory chip 113, and an image processing chip 114 that performs the pre-processing and image processing described above. In other words, the image processing unit 32c described above is located on the image processing chip 114. These imaging chips 111, signal processing chip 112, memory chip 113, and image processing chip 114 are stacked and electrically connected to each other by conductive bumps 109 made of Cu or the like.

[0279] Multiple bumps 109 are arranged on the opposing surfaces of the memory chip 113 and the image processing chip 114. When these bumps 109 are aligned with each other and pressure is applied to the memory chip 113 and the image processing chip 114, the aligned bumps 109 are joined together and electrically connected.

[0280] <First Correction Process> Similar to the first embodiment, in the second embodiment, after the setting unit 34b divides the area of ​​the imaging screen, it is possible to set (change) imaging conditions for the area selected by the user or the area determined by the control unit 34. If different imaging conditions are set in the divided area, the control unit 34 causes the correction unit 322 of the image processing unit 32c to perform preprocessing as necessary.

[0281] In other words, if the block, which is the smallest unit for setting imaging conditions, includes the boundary of a region based on multiple subject elements, and if there is overexposure or underexposure in the image data of this block, the control unit 34 causes the correction unit 322 to perform the following first correction process as one of the preprocessing steps performed before image processing, focus detection processing, subject detection processing, and the processing for setting imaging conditions.

[0282] 1. The same correction is applied to the entire area where overexposure or underexposure has occurred. (1-1) The correction unit 322, as in the first embodiment, replaces the image data where overexposure or underexposure has occurred with image data acquired in one block within the same imaging screen as a first correction process, in any of the following manners (i) to (iv).

[0283] (i) The correction unit 322 uses image data acquired from one of the reference blocks located around the block of interest that is closest to the area that is overexposed or underexposed, to replace the image data within the block of interest that is overexposed or underexposed with the same data.

[0284] (ii) The correction unit 322 uses image data acquired from one of the reference blocks located around the block of interest that is set most frequently for the same subject element (e.g., mountain) as the subject element (e.g., mountain) that is blown out or crushed in color, to replace the image data in the block of interest that is blown out or crushed in color with the same data.

[0285] (iii) The correction unit 322 replaces the image data in which the highlights or blacks are blown out with the image data that is adjacent to the highlights or blacks in the block of interest, using the image data corresponding to the pixels adjacent to the highlights or blacks in the reference block of interest, from among the image data corresponding to multiple pixels (four pixels in the example of Figure 8) acquired in one reference block selected in (i) or (ii) above.

[0286] (iv) The correction unit 322 uses image data generated based on image data corresponding to multiple pixels (four pixels in the example of Figure 8) acquired in one reference block selected in (i) or (ii) above to replace image data in the block of interest that has been blown out or crushed.

[0287] As with the first embodiment, when calculating the average value of the image data, a weighted average value, which is weighted according to the distance from pixels where overexposure or underexposure occurred, may be used instead of a simple average.

[0288] Furthermore, similar to the first embodiment, instead of calculating the average value of the image data corresponding to multiple pixels included in the reference block, an intermediate value of the image data corresponding to multiple pixels may be calculated, and the image data corresponding to pixels where overexposure or underexposure has occurred may be replaced by this intermediate value.

[0289] (1-2) The correction unit 322, as in the first embodiment, performs a first correction process in any of the following manner (i) to (iv) to replace the image data in which overexposure or underexposure has occurred with image data acquired in multiple blocks within the same imaging screen.

[0290] (i) The correction unit 322 uses image data acquired from multiple reference blocks located around the block of interest that are overexposed or underexposed, to replace the overexposed or underexposed image data within the block of interest with the same data.

[0291] (ii) The correction unit 322 uses image data acquired from multiple reference blocks located around the block of interest that are set most frequently for the same subject element (e.g., mountain) as the subject element (e.g., mountain) that is overexposed or underexposed, to replace the overexposed or underexposed image data within the block of interest with the same data.

[0292] (iii) The correction unit 322 replaces the image data in which the highlights or blacks are blown out with the image data that is adjacent to the highlights or blacks in the block of interest, from among the image data corresponding to multiple pixels acquired in the multiple reference blocks selected in (i) or (ii) above.

[0293] (iv) The correction unit 322 replaces the image data in the block of interest that has been blown out or crushed in white using the image data generated based on the image data corresponding to multiple pixels acquired in the multiple reference blocks selected in (i) or (ii) above.

[0294] As with the first embodiment, when calculating the average value of the image data, a weighted average value, which is weighted according to the distance from pixels where overexposure or underexposure occurred, may be used instead of a simple average.

[0295] Furthermore, similar to the first embodiment, instead of calculating the average value of image data corresponding to multiple pixels contained in multiple reference blocks, an intermediate value of image data corresponding to multiple pixels may be calculated, and the image data corresponding to pixels where overexposure or underexposure has occurred may be replaced by this intermediate value.

[0296] 2. Apply multiple corrections to the entire area where overexposure or underexposure has occurred. (2-1) The correction unit 322, as in the first embodiment, performs a first correction process in any of the following manner (i) to (iii) to replace the image data that has been overexposed or underexposed with image data acquired in one block within the same imaging screen.

[0297] (i) The correction unit 322 uses image data corresponding to pixels adjacent to the overexposed or underexposed pixels among the reference blocks located around the block of interest to replace multiple overexposed or underexposed image data within the block of interest with different data.

[0298] (ii) The correction unit 322 uses image data acquired from one of the reference blocks located around the block of interest that is set most frequently for the same subject element (e.g., mountain) as the subject element (e.g., mountain) that is overexposed or underexposed, to replace multiple image data within the block of interest that are overexposed or underexposed with different data.

[0299] (iii) The correction unit 33b uses image data generated based on image data corresponding to multiple pixels (four pixels in the example of Figure 8) acquired in one reference block selected in (i) or (ii) above to replace multiple image data in the block of interest that have been blown out or crushed with different data.

[0300] As with the first embodiment, when calculating the average value of the image data, a weighted average value, which is weighted according to the distance from pixels where overexposure or underexposure occurred, may be used instead of a simple average.

[0301] (2-2) The correction unit 322, as in the first embodiment, performs a first correction process in which it replaces image data that has been overexposed or underexposed with image data acquired in multiple blocks within the same imaging screen, in any of the following manner (i) to (iv).

[0302] (i) The correction unit 322 uses image data acquired from multiple reference blocks located around the block of interest that are overexposed or underexposed, to replace multiple image data within the block of interest that are overexposed or underexposed with different data.

[0303] (ii) The correction unit 322 uses image data acquired from multiple reference blocks located around the block of interest that are set most frequently for the same subject element (e.g., mountain) as the subject element (e.g., mountain) that is overexposed or underexposed, to replace multiple image data within the block of interest that are overexposed or underexposed with different data.

[0304] (iii) The correction unit 322 replaces the image data in the block of interest that has been blown out or crushed in white using the image data generated based on the image data corresponding to multiple pixels acquired in the multiple reference blocks selected in (i) or (ii) above.

[0305] As with the first embodiment, when calculating the average value of the image data, a weighted average value, which is weighted according to the distance from pixels where overexposure or underexposure occurred, may be used instead of a simple average.

[0306] Furthermore, similar to the first embodiment, instead of calculating the average value of image data corresponding to multiple pixels contained in multiple reference blocks, an intermediate value of image data corresponding to multiple pixels may be calculated, and the image data corresponding to pixels where black crushing occurred may be replaced with this intermediate value.

[0307] Of the various forms of correction described above for the first correction process, the control unit 34 determines which form of correction to perform based, for example, on the setting status (including the setting of the operation menu) by the operation member 36. Furthermore, the control unit 34 may be configured to determine which type of correction to perform based on the imaging scene mode set in camera 1 and the type of subject element detected.

[0308] <Second Correction Process> The control unit 34 further causes the correction unit 322 to perform the following second correction process as needed, before image processing, focus detection processing, subject detection (detection of subject elements) processing, and imaging condition setting processing.

[0309] 1. When performing image processing 1-1. When the imaging conditions for the pixel of interest P are the same as the imaging conditions for multiple reference pixels Pr surrounding the pixel of interest P. In this case, the image processing unit 32c performs image processing using the image data of multiple reference pixels Pr that have not undergone the second correction process, without the correction unit 322 performing the second correction process.

[0310] 1-2. When the imaging conditions for the pixel of interest P are different from the imaging conditions for at least one of the multiple reference pixels Pr surrounding the pixel of interest P. The imaging conditions applied to the pixel of interest P are defined as the first imaging conditions. The imaging conditions applied to some of the multiple reference pixels Pr are also defined as the first imaging conditions, and the imaging conditions applied to the remaining reference pixels Pr are defined as the second imaging conditions. In this case, the correction unit 322 corresponding to block 111a to which the reference pixel Pr to which the second imaging condition is applied belongs performs a second correction process on the image data of the reference pixel Pr to which the second imaging condition is applied, as shown in (Example 1) to (Example 3) below. Then, the generation unit 323 performs image processing to calculate the image data of the pixel of interest P by referring to the image data of the reference pixel Pr to which the first imaging condition is applied and the image data of the reference pixel Pr after the second correction process.

[0311] (Example 1) The correction unit 322, which corresponds to block 111a to which the reference pixel Pr to which the second imaging condition is applied belongs, applies a second correction process of 100 / 800 to the image data of the reference pixel Pr if, for example, the only difference between the first and second imaging conditions is the ISO sensitivity, with the ISO sensitivity of the first imaging condition being 100 and the ISO sensitivity of the second imaging condition being 800. This reduces the difference between image data due to the difference in imaging conditions.

[0312] (Example 2) The correction unit 322, which corresponds to block 111a to which the reference pixel Pr to which the second imaging condition is applied belongs, for example, if only the shutter speed differs between the first and second imaging conditions, and the shutter speed of the first imaging condition is 1 / 1000 second and the shutter speed of the second imaging condition is 1 / 100 second, then the correction unit 322 applies a second correction process to the image data of the reference pixel Pr by multiplying it by 1 / 1000 / 1 / 100 = 1 / 10. This reduces the difference between image data due to the difference in imaging conditions.

[0313] (Example 3) The correction unit 322, which corresponds to block 111a to which the reference pixel Pr to which the second imaging condition is applied belongs, performs a second correction process in which, for example, if only the frame rate differs between the first and second imaging conditions (the charge accumulation time is the same), and the frame rate of the first imaging condition is 30fps and the frame rate of the second imaging condition is 60fps, the image data of the reference pixel Pr, i.e., the image data of the second imaging condition (60fps), is selected from the image data of the frame image acquired under the first imaging condition (30fps) and the image data of the frame image acquired under the second imaging condition (60fps) with an acquisition start timing close to that of the frame image. This reduces the difference between image data due to the difference in imaging conditions. Furthermore, the second correction process may involve interpolating and calculating image data of frames acquired under the first imaging condition (30fps) and frames with similar acquisition start times, based on multiple consecutive frame images acquired under the second imaging condition (60fps).

[0314] The same applies when the imaging conditions applied to the pixel of interest P are designated as the second imaging conditions, and the imaging conditions applied to the reference pixels Pr surrounding the pixel of interest P are designated as the first imaging conditions. In other words, in this case, the correction unit 322 corresponding to the block 111a to which the reference pixel Pr to which the first imaging conditions are applied belongs performs the second correction process on the image data of the reference pixel Pr as described in (Example 1) to (Example 3) above.

[0315] As mentioned above, even if there are slight differences in imaging conditions, they will be considered the same.

[0316] The generation unit 323 performs image processing such as pixel defect correction, color interpolation, edge enhancement, and noise reduction based on the image data of the reference pixel Pr to which the same imaging conditions as those of the pixel of interest P are applied, and the image data of the reference pixel Pr that has undergone second correction processing by the correction unit 322, similar to the generation unit 33c of the image processing unit 33 in the first embodiment.

[0317] Figure 24 schematically illustrates the processing of image data from each pixel in a portion of the imaging surface to which the first imaging condition is applied (hereinafter referred to as the first imaging region 141) (hereinafter referred to as the first image data) and image data from each pixel in a portion of the imaging surface to which the second imaging condition is applied (hereinafter referred to as the second imaging region 142) (hereinafter referred to as the second image data).

[0318] Each pixel in the first imaging area 141 outputs first image data captured under the first imaging conditions, and each pixel in the second imaging area 142 outputs second image data captured under the second imaging conditions. The first image data is output to a correction unit 322 in the image processing chip 114 that corresponds to the block 111a to which the pixel that generated the first image data belongs. In the following description, the multiple correction units 322 corresponding to the multiple blocks 111a to which each pixel that generated the first image data belongs will be referred to as the first processing unit 151. The first processing unit 151 performs the first correction process and the second correction process, or the first correction process or the second correction process, on the first image data as necessary.

[0319] Similarly, the second image data is output to the correction unit 322 of the image processing chip 114 that corresponds to the block 111a to which the pixel that generated the second image data belongs. In the following description, the multiple correction units 322 that correspond to the multiple blocks 111a to which each pixel that generated the second image data belongs will be referred to as the second processing unit 152. The second processing unit 152 performs the first correction process and the second correction process, or the first correction process or the second correction process, on the second image data as necessary.

[0320] In the first correction process described above, for example, if the block of interest is included in the first imaging region 141, the first processing unit 151 performs the first correction process, i.e., replacement process, as shown in Figure 24. As a result, image data in the block of interest that has been overexposed or underexposed is replaced with second image data from a reference block included in the second imaging region 142. For this purpose, the first processing unit 151 receives the second image data from the reference block as information 182 from the second processing unit 152, for example.

[0321] In the second correction process described above, for example, if the pixel of interest P is included in the first imaging region 141, the second image data from the reference pixel Pr included in the second imaging region 142 is subjected to the second correction process described above by the second processing unit 152, as shown in Figure 24. The second processing unit 152 receives, for example, information 181 about the first imaging conditions necessary to reduce the differences between image data due to differences in imaging conditions from the first processing unit 151. Similarly, for example, if the pixel of interest P is included in the second imaging region 142, the first image data from the reference pixel Pr included in the first imaging region 141 undergoes the second correction process described above in the first processing unit 151. The first processing unit 151 also receives information from the second processing unit 152 regarding the second imaging conditions necessary to reduce the differences between image data due to differences in imaging conditions.

[0322] Furthermore, if the pixel of interest P and the reference pixel Pr are included in the first imaging region 141, the first processing unit 151 does not perform the second correction process on the first image data from the reference pixel Pr. Similarly, if the pixel of interest P and the reference pixel Pr are included in the second imaging region 142, the second processing unit 152 does not perform the second correction process on the second image data from the reference pixel Pr. Alternatively, the first processing unit 151 and the second processing unit 152 may correct both the image data for the first imaging condition and the image data for the second imaging condition, respectively. That is, by applying the second correction process to the image data of the point of interest for the first imaging condition, the image data of the first imaging condition among the reference position image data, and the image data of the second imaging condition among the reference position image data, respectively, the discontinuity in the image based on the difference between the first and second imaging conditions may be mitigated. For example, in (Example 1) above, the image data of the reference pixel Pr under the first imaging condition (ISO sensitivity of 100) is multiplied by 400 / 100 as a second correction process, and the image data of the reference pixel Pr under the second imaging condition (ISO sensitivity of 800) is multiplied by 400 / 800 as a second correction process. This reduces the difference between image data due to differences in imaging conditions. The pixel data of the pixel of interest is subjected to a second correction process of multiplying by 100 / 400 after color interpolation. This second correction process makes it possible to change the pixel data of the pixel of interest after color interpolation to the same value as when it was captured under the first imaging condition. Furthermore, in (Example 1) above, the degree of the second correction process may be changed depending on the distance from the boundary between the first and second regions. Compared to the case of (Example 1) above, the rate of increase or decrease in image data due to the second correction process can be reduced, and the noise generated by the second correction process can be reduced. The above explanation describes (Example 1), but the same can be applied to (Example 2).

[0323] The generation unit 323 performs image processing such as pixel defect correction, color interpolation, edge enhancement, and noise reduction based on the image data from the first processing unit 151 and the second processing unit 152, and outputs the image data after image processing.

[0324] Furthermore, when the pixel of interest P is located in the second imaging area 142, the first processing unit 151 may perform the second correction process on the first image data from all pixels included in the first imaging area 141, or it may perform the second correction process only on the first image data from pixels included in the first imaging area 141 that may be used for interpolation of the pixel of interest P in the second imaging area 142. Similarly, when the pixel of interest P is located in the first imaging area 141, the second processing unit 152 may perform the second correction process on the second image data from all pixels included in the second imaging area 142, or it may perform the second correction process only on the second image data from pixels included in the second imaging area 142 that may be used for interpolation of the pixel of interest P in the first imaging area 141.

[0325] 2. When performing focus detection processing Similar to the first embodiment, the lens movement control unit 34d of the control unit 34 performs focus detection processing using signal data (image data) corresponding to a predetermined position (focus point) on the imaging screen. Note that if different imaging conditions are set between the divided regions, and the focus point for AF operation is located at the boundary of the divided regions, i.e., the focus point is divided between the first region and the second region, in this embodiment, as described in 2-2 below, the lens movement control unit 34d of the control unit 34 causes the correction unit 322 to perform a second correction process on the signal data for focus detection of at least one region.

[0326] 2-1. In Figure 15, when the signal data from pixels within frame 170 does not contain a mixture of signal data to which the first imaging condition is applied and signal data to which the second imaging condition is applied. In this case, the correction unit 322 does not perform the second correction process, and the lens movement control unit 34d of the control unit 34 uses the signal data from the focus detection pixels indicated by the frame 170 as is to perform the focus detection process.

[0327] 2-2. When the signal data from pixels within frame 170 in Figure 15 contains a mixture of signal data to which the first imaging condition is applied and signal data to which the second imaging condition is applied. In this case, the lens movement control unit 34d of the control unit 34 causes the correction unit 322 corresponding to the block 111a to which the pixels to which the second imaging condition is applied belong, among the pixels within the frame 170, to perform a second correction process as shown in (Example 1) to (Example 3) below. Then, the lens movement control unit 34d of the control unit 34 performs focus detection processing using the signal data of the pixels to which the first imaging condition is applied and the signal data after the second correction process.

[0328] (Example 1) The correction unit 322, which corresponds to block 111a to which the pixels to which the second imaging condition is applied belong, for example, if only the ISO sensitivity differs between the first and second imaging conditions, and the ISO sensitivity of the first imaging condition is 100 and the ISO sensitivity of the second imaging condition is 800, then it applies a second correction process of 100 / 800 to the signal data of the second imaging condition. This reduces the difference between signal data due to the difference in imaging conditions.

[0329] (Example 2) The correction unit 322, which corresponds to block 111a to which the pixels to which the second imaging condition is applied belong, for example, if only the shutter speed differs between the first and second imaging conditions, and the shutter speed of the first imaging condition is 1 / 1000 second and the shutter speed of the second imaging condition is 1 / 100 second, then the correction unit 322 multiplies the signal data of the second imaging condition by 1 / 1000 / 1 / 100 = 1 / 10 as the second correction process. This reduces the difference between signal data due to the difference in imaging conditions.

[0330] (Example 3) The correction unit 322, which corresponds to block 111a to which the pixels to which the second imaging condition is applied belong, performs a second correction process if, for example, only the frame rate differs between the first and second imaging conditions (the charge accumulation time is the same), and the frame rate of the first imaging condition is 30fps and the frame rate of the second imaging condition is 60fps, by adopting the signal data of the frame image acquired under the first imaging condition (30fps) and the frame image whose acquisition start timing is close to that of the second imaging condition (60fps). This reduces the difference between signal data due to the difference in imaging conditions. Furthermore, the second correction process may involve interpolating signal data from frame images acquired under the first imaging condition (30fps) and frame images with similar acquisition start times, based on multiple frame images acquired under the second imaging condition (60fps) that occur sequentially.

[0331] As mentioned above, even if there are slight differences in imaging conditions, they will be considered the same. Furthermore, although the above example describes an example where the second correction process is performed on the signal data of the second imaging condition, the second correction process may also be performed on the signal data of the first imaging condition.

[0332] Furthermore, a second correction process may be applied to the signal data for the first imaging condition and the data for the second imaging condition, respectively, in order to reduce the difference between the two sets of signal data after the second correction process.

[0333] Figure 25 is a schematic diagram illustrating the processing of the first signal data and the second signal data related to focus detection.

[0334] Each pixel in the first imaging area 141 outputs first signal data captured under first imaging conditions, and each pixel in the second imaging area 142 outputs second signal data captured under second imaging conditions. The first signal data from the first imaging area 141 is output to the first processing unit 151. Similarly, the second signal data from the second imaging area 142 is output to the second processing unit 152.

[0335] The first processing unit 151 performs the first correction process and the second correction process, or the first correction process or the second correction process, on the first image data as necessary. The second processing unit 152 performs the first correction process and the second correction process, or the first correction process or the second correction process, on the second image data as necessary.

[0336] In the first correction process described above, for example, if the block of interest is included in the first imaging region 141, the first processing unit 151 performs the first correction process, i.e., replacement process, as shown in Figure 25. As a result, the first signal data in the block of interest that has been overexposed or underexposed is replaced with the second signal data from the reference block included in the second imaging region 142. For this purpose, the first processing unit 151 receives the second signal data from the reference block as information 182 from the second processing unit 152, for example.

[0337] In the second correction process described above, if the difference between the signal data after the second correction process and the signal data of the first imaging condition is reduced by performing the second correction process on the signal data of the second imaging condition among the signal data, the second processing unit 152 performs the process. The second processing unit 152 performs the second correction process described above on the second signal data from pixels included in the second imaging area 142. The second processing unit 152 receives information 181 about the first imaging condition, which is necessary to reduce the difference between signal data due to differences in imaging conditions, from, for example, the first processing unit 151. Furthermore, if the difference between the signal data after the second correction process and the signal data for the first imaging condition is reduced by performing the second correction process on the signal data for the second imaging condition, the first processing unit 151 does not perform the second correction process on the first signal data.

[0338] Furthermore, if the difference between the signal data after the second correction process and the signal data under the first imaging condition is to be reduced by performing a second correction process on the signal data under the first imaging condition, the first processing unit 151 performs the process. The first processing unit 151 performs the above-described second correction process on the first signal data from pixels included in the first imaging area 141. The first processing unit 151 also receives information about the second imaging condition necessary to reduce the difference between signal data due to differences in imaging conditions from the second processing unit 152. Furthermore, if the difference between the signal data after the second correction process and the signal data under the first imaging condition is reduced by performing the second correction process on the signal data under the first imaging condition, the second processing unit 152 does not perform the second correction process on the second signal data.

[0339] Furthermore, if the difference between the two sets of signal data after the second correction process is reduced by applying a second correction process to the signal data of the first imaging condition and the data of the second imaging condition, the first processing unit 151 and the second processing unit 152 perform the processing. The first processing unit 151 performs the above-described second correction process on the first signal data from pixels included in the first imaging area 141, and the second processing unit 152 performs the above-described second correction process on the second signal data from pixels included in the second imaging area 142.

[0340] The lens movement control unit 34d performs focus detection processing based on signal data from the first processing unit 151 and the second processing unit 152, and outputs a drive signal to move the focus lens of the imaging optical system 31 to the focus position based on the calculation result.

[0341] 3. When performing subject detection processing If different imaging conditions are set between the divided regions, and the search range 190 includes the boundaries of the divided regions, in this embodiment, as described in 3-2 below, the object detection unit 34a of the control unit 34 causes the correction unit 322 to perform a second correction process on the image data of at least one region within the search range 190.

[0342] 3-1. In Figure 16, if the image data within the search range of 190 does not contain a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied. In this case, the correction unit 322 does not perform the second correction process, and the object detection unit 34a of the control unit 34 performs subject detection processing using the image data constituting the search range 190 as is.

[0343] 3-2. In Figure 16, when the image data within the search range of 190 contains a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied. In this case, the object detection unit 34a of the control unit 34 performs a second correction process on the correction unit 322 corresponding to the block 111a to which the pixels to which the second imaging condition is applied belong, as described in (Example 1) to (Example 3) above, when performing focus detection processing on the image within the search range 190. Then, the object detection unit 34a of the control unit 34 performs subject detection processing using the image data of the pixels to which the first condition is applied and the image data after the second correction process.

[0344] Figure 26 is a schematic diagram illustrating the processing of the first image data and the second image data related to the subject detection process. In the first correction process described above, for example, if the block of interest is included in the first imaging region 141, the first processing unit 151 performs the first correction process, i.e., replacement process, as shown in Figure 26. As a result, the first image data in the block of interest that has been overexposed or underexposed is replaced with the second image data from a reference block included in the second imaging region 142. For this purpose, the first processing unit 151 receives the second image data from the reference block as information 182 from the second processing unit 152, for example.

[0345] In the second correction process, the second correction process performed in the first processing unit 151 and / or the second processing unit 152 is the same as the second correction process for Figure 25 described above, in the case of performing focus detection processing. The object detection unit 34a performs a process to detect subject elements based on image data from the first processing unit 151 and the second processing unit 152, and outputs the detection result.

[0346] 4. When setting imaging conditions This section describes a method for determining exposure conditions by dividing the image area, setting different imaging conditions for each divided area, and then re-measuring the image.

[0347] 4-1. When the image data within the photometric range does not contain a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied. In this case, the correction unit 322 does not perform the second correction process, and the setting unit 34b of the control unit 34 performs the exposure calculation process using the image data that constitutes the photometric range as is.

[0348] 4-2. When the image data of the photometric range contains a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied. In this case, the setting unit 34b of the control unit 34 causes the correction unit 322 corresponding to the block 111a to which the pixels to which the second imaging condition is applied belong, to perform a second correction process as described above in (Example 1) to (Example 3), assuming that focus detection processing is performed on the image data within the photometric range. Then, the setting unit 34b of the control unit 34 performs exposure calculation processing using the image data after the second correction process.

[0349] Figure 27 is a schematic diagram illustrating the processing of the first image data and the second image data in relation to setting imaging conditions such as exposure calculation. In the first correction process described above, for example, if the block of interest is included in the first imaging region 141, the first processing unit 151 performs the first correction process, i.e., replacement process, as shown in Figure 27. As a result, the first image data in the block of interest that has been overexposed or underexposed is replaced with the second image data from a reference block included in the second imaging region 142. For this purpose, the first processing unit 151 receives the second image data from the reference block as information 182 from the second processing unit 152, for example.

[0350] In the second correction process, the second correction process performed in the first processing unit 151 and / or the second processing unit 152 is the same as the second correction process for Figure 25 described above, in the case of performing focus detection processing. The setting unit 34b calculates imaging conditions such as exposure calculation based on image data from the first processing unit 151 and the second processing unit 152. Based on the calculation results, it divides the image captured by the imaging unit 32 into multiple regions including the detected subject elements, and resets the imaging conditions for each of the multiple regions.

[0351] According to the second embodiment described above, the following effects and advantages can be obtained. (1) The image processing unit 32c includes a first processing unit 151 that generates image data of a subject captured in the first imaging area 141, and a second processing unit 152 that generates image data of a subject captured in the second imaging area 142. The first processing unit 151 generates image data of a subject captured in the first imaging area 141 using image data of a subject captured in the second imaging area 142. As a result, preprocessing of the image data (first correction processing, second correction processing) can be processed in parallel by multiple correction units 322, thereby reducing the processing load on the correction units 322.

[0352] (2) The image processing unit 32c includes a first processing unit 151 that generates a signal based on a subject incident in the first imaging area 141, and a second processing unit 152 that generates a signal based on a subject incident in the second imaging area 142. The first processing unit 151 generates a signal based on the subject captured in the first imaging area 141 using the signal based on the subject incident in the second imaging area 142. As a result, the preprocessing of image data can be processed in parallel by multiple correction units 322, reducing the processing load on the correction units 322. Furthermore, since the preprocessing by multiple correction units 322 is performed in parallel in a short time, the time until the start of focus detection processing in the lens movement control unit 34d can be shortened, contributing to faster focus detection processing.

[0353] (3) The image processing unit 32c includes a first processing unit 151 that generates image data of a subject incident on the first imaging area 141, and a second processing unit 152 that generates image data of a subject incident on the second imaging area 142. The first processing unit 151 generates image data of a subject captured in the first imaging area 141 using image data of a subject incident on the second imaging area 142. As a result, the preprocessing of the image data can be processed in parallel by multiple correction units 322, reducing the processing load on the correction units 322. Furthermore, since the preprocessing by multiple correction units 322 is performed in parallel in a short time, the time until the start of subject detection processing in the object detection unit 34a can be shortened, contributing to faster subject detection processing.

[0354] (4) The image processing unit 32c includes a first processing unit 151 that generates a signal based on a subject incident in the first imaging area 141, and a second processing unit 152 that generates a signal based on a subject incident in the second imaging area 142. The first processing unit 151 generates a signal based on the subject captured in the first imaging area 141 using the signal based on the subject incident in the second imaging area 142. As a result, the preprocessing of image data can be processed in parallel by multiple correction units 322, reducing the processing load on the correction units 322. Furthermore, since the preprocessing by multiple correction units 322 is performed in parallel in a short time, the time until the start of the imaging condition setting process in the setting unit 34b can be shortened, contributing to faster imaging condition setting.

[0355] ---Modified version of the second embodiment--- The following modifications are also within the scope of the present invention, and it is possible to combine one or more of these modifications with the embodiments described above. (Example 13) The processing of the first image data and the second image data when the first imaging region and the second imaging region are arranged on the imaging surface of the image sensor 32a, as shown in Figures 18(a) to 18(c) in Modification 1 of the First Embodiment, will be described. In this modified example, as in Modification Example 1, in all cases shown in Figures 18(a) to 18(c), a first image based on the image signal read from the first imaging area and a second image based on the image signal read from the second imaging area are generated, respectively, by the pixel signals read from the image sensor 32a that captured one frame. In this modified example, as in Modification Example 1, the control unit 34 uses the first image for display and the second image for detection. The imaging conditions set for the first imaging area where the first image is captured will be called the first imaging conditions, and the imaging conditions set for the second imaging area where the second image is captured will be called the second imaging conditions. The control unit 34 may set the first imaging conditions and the second imaging conditions to be different.

[0356] 1. As an example, we will explain the case where the first imaging conditions set for the first imaging area are the same throughout the entire first imaging area of ​​the imaging screen, and the second imaging conditions set for the second imaging area are the same throughout the entire second imaging area of ​​the imaging screen, with reference to Figure 28. Figure 28 is a schematic diagram showing the processing of the first image data and the second image data.

[0357] From each pixel in the first imaging region 141, first image data captured under the first imaging conditions is output, and from each pixel in the second imaging region 142, second image data captured under the second imaging conditions is output. The first image data from the first imaging region 141 is output to the first processing unit 151. Similarly, the second image data from the second imaging region 142 is output to the second processing unit 152.

[0358] The first processing unit 151 performs the first correction process and the second correction process, or the first correction process or the second correction process, on the first image data as necessary. The second processing unit 152 performs the first correction process and the second correction process, or the first correction process or the second correction process, on the second image data as necessary.

[0359] In this example, since the first imaging conditions are the same throughout the entire first imaging region of the imaging screen, the first processing unit 151 does not perform the second correction process on the first image data from the reference pixels Pr included in the first imaging region. Also, since the second imaging conditions are the same throughout the entire second imaging region of the imaging screen, the second processing unit 152 does not perform the second correction process on the second image data used for focus detection processing, subject detection processing, and exposure calculation processing. However, the second processing unit 152 performs a second correction process on the second image data used for interpolation of the first image data to reduce the difference between the image data due to the difference between the first and second imaging conditions. The second processing unit 152 outputs the second image data after the second correction process to the first processing unit 151 as indicated by arrow 182. Alternatively, the second processing unit 152 may output the second image data after the second correction process to the generation unit 323 as indicated by the dashed arrow 183. The second processing unit 152 receives, for example, information 181 about the first imaging conditions necessary to reduce the differences between image data due to differences in imaging conditions from the first processing unit 151.

[0360] The generation unit 323 performs image processing such as pixel defect correction, color interpolation, edge enhancement, and noise reduction based on the first image data from the first processing unit 151 and the second image data that has undergone second correction processing by the second processing unit 152, and outputs the image data after image processing. The object detection unit 34a performs a process to detect subject elements based on the second image data from the second processing unit 152 and outputs the detection result. The setting unit 34b calculates imaging conditions such as exposure calculation based on the second image data from the second processing unit 152, and based on the calculation results, divides the image captured by the imaging unit 32 into multiple regions including the detected subject elements, and resets the imaging conditions for the multiple regions. The lens movement control unit 34d performs focus detection processing based on the second signal data from the second processing unit 152, and outputs a drive signal to move the focus lens of the imaging optical system 31 to the focus position based on the calculation result.

[0361] 2. As another example, a case in which the first imaging conditions set for the first imaging area differ depending on the area of ​​the imaging screen, and the second imaging conditions set for the second imaging area are the same throughout the entire second imaging area of ​​the imaging screen, will be explained with reference to Figure 28.

[0362] From each pixel in the first imaging area 141, first image data captured under different first imaging conditions depending on the area of ​​the imaging screen is output, and from each pixel in the second imaging area 142, second image data captured under the same second imaging conditions across the entire second imaging area of ​​the imaging screen is output. The first image data from the first imaging area 141 is output to the first processing unit 151. Similarly, the second image data from the second imaging area 142 is output to the second processing unit 152.

[0363] The first processing unit 151 performs the first correction process and the second correction process, or the first correction process or the second correction process, on the first image data as necessary. The second processing unit 152 performs the first correction process and the second correction process, or the first correction process or the second correction process, on the second image data as necessary.

[0364] As described above, in this example, the first imaging conditions set in the first imaging area 141 differ depending on the area of ​​the imaging screen. That is, the first imaging conditions differ depending on the sub-region within the first imaging area 141. If different first imaging conditions are set for the target pixel P and the reference pixel Pr, both located within the first imaging area 141, the first processing unit 151 performs a second correction process on the first image data from the reference pixel Pr, similar to the second correction process described in 1-2 above. However, if the same first imaging conditions are set for the target pixel P and the reference pixel Pr, the first processing unit 151 does not perform the second correction process on the first image data from the reference pixel Pr.

[0365] In this example, since the second imaging conditions set for the second imaging area 142 are the same for the entire second imaging area of ​​the imaging screen, the second processing unit 152 does not perform the second correction processing on the second image data used for focus detection processing, subject detection processing, and exposure calculation processing. For the second image data used for interpolation of the first image data, the second processing unit 152 performs a second correction processing to reduce the difference between the image data due to the difference between the imaging conditions and the second imaging conditions for the pixel of interest P included in the first imaging area 141. The second processing unit 152 outputs the second image data after the second correction processing to the first processing unit 151 (arrow 182). Alternatively, the second processing unit 152 may output the second image data after the second correction processing to the generation unit 323 (arrow 183). The second processing unit 152 receives, for example, information 181 about the imaging conditions for a pixel of interest P included in the first imaging region 141 from the first processing unit 151, which is necessary to reduce the differences between image data due to differences in imaging conditions.

[0366] The generation unit 323 performs image processing such as pixel defect correction, color interpolation, edge enhancement, and noise reduction based on the first image data from the first processing unit 151 and the second image data that has undergone second correction processing by the second processing unit 152, and outputs the image data after image processing. The object detection unit 34a performs a process to detect subject elements based on the second image data from the second processing unit 152 and outputs the detection result. The setting unit 34b calculates imaging conditions such as exposure calculation based on the second image data from the second processing unit 152, and based on the calculation results, divides the image captured by the imaging unit 32 into multiple regions including the detected subject elements, and resets the imaging conditions for the multiple regions. The lens movement control unit 34d performs focus detection processing based on the second signal data from the second processing unit 152, and outputs a drive signal to move the focus lens of the imaging optical system 31 to the focus position based on the calculation result.

[0367] 3. As another example, a case in which the first imaging conditions set for the first imaging area 141 are the same throughout the entire first imaging area 141 of the imaging screen, and the second imaging conditions set for the second imaging area 142 differ depending on the area of ​​the imaging screen, will be explained with reference to Figure 28.

[0368] From each pixel in the first imaging area 141, first image data captured under the same first imaging conditions across the entire first imaging area 141 of the imaging screen is output. From each pixel in the second imaging area 142, second image data captured under different fourth imaging conditions depending on the area of ​​the imaging screen is output. The first image data from the first imaging area 141 is output to the first processing unit 151. Similarly, the second image data from the second imaging area 142 is output to the second processing unit 152.

[0369] The first processing unit 151 performs the first correction process and the second correction process, or the first correction process or the second correction process, on the first image data as necessary. The second processing unit 152 performs the first correction process and the second correction process, or the first correction process or the second correction process, on the second image data as necessary.

[0370] In this example, since the first imaging conditions set for the first imaging area 141 are the same throughout the entire first imaging area 141 of the imaging screen, the first processing unit 151 does not perform the second correction process on the first image data from the reference pixels Pr included in the first imaging area 141.

[0371] Furthermore, in this example, since the second imaging conditions set in the second imaging area 142 differ depending on the area of ​​the imaging screen, the second processing unit 152 performs the second correction processing on the second image data as follows. The second processing unit 152 reduces the difference between the second image data after the second correction processing and the second image data captured under other imaging conditions different from the aforementioned imaging conditions by, for example, performing the second correction processing on the second image data captured under certain imaging conditions.

[0372] In this example, with respect to the second image data used for interpolating the first image data, the second processing unit 152 performs a second correction process to reduce the difference between the image data due to the difference between the imaging conditions and the second imaging conditions for the pixel of interest P included in the first imaging region 141. The second processing unit 152 outputs the second image data after the second correction process to the first processing unit 151 (arrow 182). Alternatively, the second processing unit 152 may output the second image data after the second correction process to the generation unit 323 (arrow 183). The second processing unit 152 receives, for example, information 181 about the imaging conditions for a pixel of interest P included in the first region from the first processing unit 151, which is necessary to reduce the differences between image data due to differences in imaging conditions.

[0373] The generation unit 323 performs image processing such as pixel defect correction, color interpolation, edge enhancement, and noise reduction based on the first image data from the first processing unit 151 and the second image data that has undergone second correction processing by the second processing unit 152, and outputs the image data after image processing. The object detection unit 34a performs a process to detect subject elements based on second image data captured under certain imaging conditions and second image data captured under other imaging conditions, which have been subjected to a second correction process by the second processing unit 152, and outputs the detection result. The setting unit 34b performs imaging condition calculation processing, such as exposure calculation processing, based on the second image data captured under certain imaging conditions and the second image data captured under other imaging conditions, which have been subjected to second correction processing by the second processing unit 152. Based on the calculation results, the setting unit 34b divides the image screen captured by the imaging unit 32 into multiple regions including the detected subject elements, and resets the imaging conditions for the multiple regions. The lens movement control unit 34d performs focus detection processing based on the second signal data captured under certain imaging conditions and the second signal data captured under other imaging conditions, which have been second-corrected by the second processing unit 152. Based on the calculation results, the lens movement control unit 34d outputs a drive signal to move the focus lens of the imaging optical system 31 to the focus position.

[0374] 4. Furthermore, as another example, a case in which the first imaging conditions set in the first imaging area 141 differ depending on the area of ​​the imaging screen, and the second imaging conditions set in the second imaging area 142 also differ depending on the area of ​​the imaging screen, will be explained with reference to Figure 28.

[0375] From each pixel in the first imaging area 141, first image data captured under first imaging conditions that differ depending on the area of ​​the imaging screen is output, and from each pixel in the second imaging area 142, second image data captured under second imaging conditions that differ depending on the area of ​​the imaging screen is output. The first image data from the first imaging area 141 is output to the first processing unit 151. Similarly, the second image data from the second imaging area 142 is output to the second processing unit 152.

[0376] The first processing unit 151 performs the first correction process and the second correction process, or the first correction process or the second correction process, on the first image data as necessary. The second processing unit 152 performs the first correction process and the second correction process, or the first correction process or the second correction process, on the second image data as necessary.

[0377] As described above, in this example, the first imaging conditions set in the first imaging area 141 differ depending on the area of ​​the imaging screen. That is, the first imaging conditions differ depending on the sub-region within the first imaging area 141. If different first imaging conditions are set for the target pixel P and the reference pixel Pr, both located within the first imaging area 141, the first processing unit 151 performs a second correction process on the first image data from the reference pixel Pr, similar to the second correction process described in 1-2 above. However, if the same first imaging conditions are set for the target pixel P and the reference pixel Pr, the first processing unit 151 does not perform the second correction process on the first image data from the reference pixel Pr.

[0378] Furthermore, in this example, since the second imaging conditions set in the second imaging area 142 differ depending on the area of ​​the imaging screen, the second processing unit 152 performs a second correction process on the second image data as in the example described in 3. above.

[0379] The generation unit 323 performs image processing such as pixel defect correction, color interpolation, edge enhancement, and noise reduction based on the first image data from the first processing unit 151 and the second image data that has undergone second correction processing by the second processing unit 152, and outputs the image data after image processing. The object detection unit 34a performs a process to detect subject elements based on second image data captured under certain imaging conditions and second image data captured under other imaging conditions, which have been subjected to a second correction process by the second processing unit 152, and outputs the detection result. The setting unit 34b performs imaging condition calculation processing, such as exposure calculation processing, based on the second image data captured under certain imaging conditions and the second image data captured under other imaging conditions, which have been subjected to second correction processing by the second processing unit 152. Based on the calculation results, the setting unit 34b divides the image screen captured by the imaging unit 32 into multiple regions including the detected subject elements, and resets the imaging conditions for the multiple regions. The lens movement control unit 34d performs focus detection processing based on the second signal data captured under certain imaging conditions and the second signal data captured under other imaging conditions, which have been second-corrected by the second processing unit 152. Based on the calculation results, the lens movement control unit 34d outputs a drive signal to move the focus lens of the imaging optical system 31 to the focus position.

[0380] (Variation 14) In the second embodiment described above, one correction unit 322 corresponds to one block 111a (unit division). However, one correction unit 322 may correspond to one composite block (composite division) having multiple blocks 111a (unit divisions). In this case, the correction unit 322 sequentially corrects the image data from pixels belonging to the multiple blocks 111a included in the composite block. Even if multiple correction units 322 are provided corresponding to each composite block having multiple blocks 111a, the second correction processing of image data can be processed in parallel by the multiple correction units 322, thereby reducing the processing load on the correction units 322 and enabling the generation of appropriate images in a short time from image data generated in areas with different imaging conditions.

[0381] (Variation 15) In the second embodiment described above, the generation unit 323 is located inside the imaging unit 32A. However, the generation unit 323 may be located outside the imaging unit 32A. Even if the generation unit 323 is located outside the imaging unit 32A, it will produce the same effects and advantages as described above.

[0382] (Variation 16) In the second embodiment described above, the stacked image sensor 100A further includes an image processing chip 114 that performs the pre-processing and image processing described above, in addition to the back-illuminated imaging chip 111, the signal processing chip 112, and the memory chip 113. However, the stacked image sensor 100A may not have an image processing chip 114, and the signal processing chip 112 may have an image processing unit 32c.

[0383] (Example 17) In the second embodiment described above, the second processing unit 152 received information from the first processing unit 151 regarding the first imaging conditions necessary to reduce the differences between image data due to differences in imaging conditions. The first processing unit 151 also received information from the second processing unit 152 regarding the second imaging conditions necessary to reduce the differences between image data due to differences in imaging conditions. However, the second processing unit 152 may also receive information from the drive unit 32b or the control unit 34 regarding the first imaging conditions necessary to reduce the differences between image data due to differences in imaging conditions. Similarly, the first processing unit 151 may also receive information from the drive unit 32b or the control unit 34 regarding the second imaging conditions necessary to reduce the differences between image data due to differences in imaging conditions.

[0384] The imaging optical system 31 described above may include a zoom lens and a tilt-shift lens. The lens movement control unit 34d adjusts the field of view of the imaging optical system 31 by moving the zoom lens in the optical axis direction. In other words, by moving the zoom lens, the image produced by the imaging optical system 31 can be adjusted, such as obtaining an image of a wide range of subjects or obtaining a large image of a distant subject. Furthermore, the lens movement control unit 34d can adjust the image distortion caused by the imaging optical system 31 by moving the tilt lens in a direction perpendicular to the optical axis. Furthermore, based on the idea that it is preferable to use the pre-processed image data described above in order to adjust the state of the image produced by the imaging optical system 31 (for example, the state of the field of view or the state of image distortion), the pre-processing described above should be performed.

[0385] ---Third Embodiment--- Referring to Figures 29 and 30, a digital camera will be used as an example of an electronic device equipped with an image processing device according to the third embodiment. In the following description, the same reference numerals will be used for components that are the same as in the first embodiment, and the differences will be mainly explained. Points that are not specifically explained are the same as in the first embodiment. In this embodiment, the case in which the boundary of the divided region is within a block will be described.

[0386] <First Correction Process> In the third embodiment, the correction unit 33b of the image processing unit 33 performs a first correction process as necessary, as one of the preprocessing steps performed before image processing, focus detection processing, subject detection (detection of subject elements) processing, and imaging condition setting processing.

[0387] Similar to the first embodiment, in the third embodiment, after the imaging screen area is divided by the setting unit 34b, it is possible to set (change) imaging conditions for the area selected by the user or the area determined by the control unit 34. For example, the divided regions 61 to 66 are designated as the first region 61 to the sixth region 66 (see Figure 29(a)), and different first to sixth imaging conditions are set for each of the first to sixth regions 66. In such a case, there exists a block that includes the boundary between the first to sixth regions 66. As described above, a block is the smallest unit on the image sensor 32a from which imaging conditions can be individually set.

[0388] Figure 29(a) illustrates a predetermined range 280 that includes the boundary between a first region 61 corresponding to a person and a fourth region 64 corresponding to a mountain in the live view image 60a. Figure 20(b) is an enlarged view of the predetermined range 280 in Figure 29(a). In Figure 29(b), the predetermined range 280 includes multiple blocks 281 to 289. The white areas in Figure 29(b) represent the parts corresponding to the people. The shaded and shaded areas in Figure 29(b) represent the parts corresponding to the mountains. Blocks 282, 285, and 287 contain the boundary B1 between the first region 61 and the fourth region 64. That is, in Figure 29(b), the parts corresponding to the mountains within blocks 82, 85, and 87, where boundary B1 exists, are shaded. In the following explanation, among blocks within the same subject element (same region), blocks that include the boundary with an adjacent region will be called boundary blocks, and blocks that do not include the boundary will be called main blocks. That is, in the example in Figure 29(b), blocks 282, 285, and 287 are boundary blocks, and the remaining blocks 281, 283, 284, 286, 288, and 289 are main blocks.

[0389] In this embodiment, the first region 61 corresponding to a person is set as the first imaging condition, and the fourth region 64 corresponding to a mountain is set as the fourth imaging condition. In this embodiment, since a block is the smallest unit for setting imaging conditions, the same imaging conditions are set for one block. Therefore, the setting unit 34b sets the imaging conditions for the main blocks 281 and 284 that image the person as the first imaging condition, and also sets the imaging conditions for the boundary blocks 282, 285, and 287 that image the person and the mountain as the first imaging condition. On the other hand, the setting unit 34b sets the imaging conditions for the main blocks 283, 286, 288, and 289 that image the mountain as the fourth imaging condition. In this way, since the first imaging conditions are set for boundary blocks 282, 285, and 287, which include the boundary B1 between the first region 61 and the fourth region 64, the first imaging conditions are also set for the shaded areas of these boundary blocks 282, block 285, and block 287, i.e., the parts corresponding to the mountains. In other words, different imaging conditions are set for the shaded areas within boundary blocks 282, boundary block 285, and boundary block 287 than the fourth imaging conditions set for the main blocks 283, 286, 288, and 289 that image the mountains.

[0390] As described above, if the shaded areas of boundary blocks 282, 285, and 287 within the fourth region 64 are set as the first imaging condition, and the shaded main blocks 283, 286, 288, and 289 are set as the fourth imaging condition, there may be differences in brightness, contrast, and hue between the shaded and shaded areas of the image. In other words, even within the same fourth region 64, the imaging conditions differ between the boundary blocks and the main blocks, which may result in discontinuities in the image after image processing. Therefore, in this embodiment, by performing the correction process described below on the signals from pixels belonging to the boundary block, a signal similar to that obtained when imaging under the same imaging conditions as the main block is obtained. This correction process is called the first correction process. The first correction process is performed to mitigate discontinuities that occur in the image after image processing due to the existence of parts with different imaging conditions within the same region.

[0391] Figure 30 is an example of image data corresponding to Figure 29(b). In Figure 30, blocks 281 to 289 are each composed of 4 pixels, 2 pixels × 2 pixels. In Figure 30, boundary B1 is shown as a thick dashed line. Of the pixels shown in Figure 30, the white pixels 281a to 281d, 282a to 282c, 284a to 284d, 285a, and 287a are incident on subject light from person 61a (see Figure 5), and the shaded pixels 282d, 283a to 283d, 285b to 285d, 286a to 286d, 287b to 287d, 288a to 288d, and 289a to 289d are incident on subject light from mountain 64a (see Figure 5).

[0392] The correction unit 33b determines which subject element's light is incident on each pixel belonging to the boundary block. Specifically, the correction unit 33b calculates the boundary position on the imaging surface of the image sensor 32a from the subject element detection result by the object detection unit 34a. Then, the correction unit 33b extracts boundary blocks based on the calculated boundary position and calculates which subject element's light is incident on each pixel belonging to the extracted boundary block.

[0393] For example, using Figures 29 and 30 as an example, the correction unit 33b calculates the position of the boundary B1 between the first region 61 and the fourth region 64 from the object detection result of the object detection unit 34a. Then, the correction unit 33b extracts blocks 282, 285, and 287 as boundary blocks based on the calculated position of boundary B1. Then, based on the calculated position of boundary B1, the correction unit 33b calculates that subject light from the person 61a is incident on pixels 282a, 282b, and 282c of boundary block 282, pixel 285a of boundary block 285, and pixel 287a of boundary block 287. Furthermore, the correction unit 33b calculates, based on the calculated position of boundary B1, that subject light from the mountain 64a is incident on pixels 282d of boundary block 282, pixels 285b, 285c, and 285d of boundary block 285, and pixels 287b, 287c, and 287d of boundary block 287.

[0394] As described above, the main blocks 283, 286, 288, and 289 of the fourth region 64 are capturing subject light from the mountain 64a according to the fourth imaging condition. However, since the boundary blocks 282, 285, and 287 are set to the first imaging condition, the pixels 282d of boundary block 282, pixels 285b to 285d of boundary block 285, and pixels 287b to 287d of boundary block 287 are capturing subject light from the mountain 64a according to the first imaging condition. Therefore, the correction unit 33b performs a first correction process on the signals from pixels 282d of boundary block 282, pixels 285b to 285d of boundary block 285, and pixels 287b to 287d of boundary block 287, so that a signal similar to that obtained when subject light from mountain 64a is captured under the fourth imaging condition is obtained. In this embodiment, the correction process is performed under the same conditions on multiple pixels that are subject to correction processing within a boundary block. In the following description, boundary block 285 will be used as an example, but the same correction process will be performed on other boundary blocks as needed.

[0395] (Example of the first correction process) 1. Perform the first correction process based on the imaging conditions. As one aspect of the first correction process, the correction process is performed based on the difference in imaging conditions set for two regions (first region 61 and fourth region 64) related to the boundary block, as described below. (Example 1) For example, if the only difference between the fourth imaging condition and the first imaging condition is the ISO sensitivity, and the ISO sensitivity of the fourth imaging condition is 800 and the ISO sensitivity of the first imaging condition is 100, the correction unit 33b multiplies the signals from each pixel 285b to 285d of the boundary block 285 by 800 / 100=8 as the first correction process. In this case, the correction coefficient α multiplied by the pixel value of the signal from each pixel 285b to 285d is 8, which is calculated as described above based on the difference in imaging conditions set for two adjacent regions. The signals of each pixel 285b to 285d in the boundary block 285, corrected by this first correction process, become identical to the signals obtained when the light incident on each pixel 285b to 285d is captured under the fourth imaging condition (ISO sensitivity 800).

[0396] (Example 2) For example, if the only difference between the fourth imaging condition and the first imaging condition is the shutter speed, and the shutter speed for the fourth imaging condition is 1 / 100 second and the shutter speed for the first imaging condition is 1 / 1000 second, the correction unit 33b multiplies the signals from each pixel 285b to 285d of the boundary block 285 by (1 / 100) / (1 / 1000)=10 as the first correction process. In this case, the correction coefficient α multiplied by the pixel value of the signal from each pixel 285b to 285d is 10, which is calculated as described above based on the difference in imaging conditions set for two adjacent regions. The signals of each pixel 285b to 285d of the boundary block 285, corrected by this first correction process, become identical to the signals obtained when the light incident on each pixel 285b to 285d is captured under the fourth imaging condition (shutter speed of 1 / 100 second).

[0397] (Example 3) For example, if the only difference between the fourth imaging condition and the first imaging condition is the frame rate, and the frame rate for the fourth imaging condition is 60fps and the frame rate for the first imaging condition is 30fps, the correction unit 33b selects the signals from multiple frame images with a frame rate of 30fps from each pixel 285b to 285d of the boundary block 285, and the signals from frame images whose acquisition start timing is close to that of the frame image acquired under the fourth imaging condition (60fps), and converts the frame rate from 30fps to 60fps. This frame rate conversion is achieved, for example, by storing the signals from each pixel 285b to 285d of the boundary block 285 in memory and reading the same signal from that memory twice. Furthermore, the conversion from a frame rate of 30fps to a frame rate of 60fps through the first correction process can also be achieved by adding together the signals of multiple consecutive frame images acquired at a frame rate of 30fps to generate a composite signal, and then supplementing this composite signal with the signal of the frame image at a frame rate of 30fps.

[0398] In this way, the correction unit 33b performs the first correction process as needed on the signals from each pixel in all boundary blocks. That is, the correction unit 33b performs the first correction process on the signal from a pixel belonging to a boundary block if the imaging conditions applied to the boundary block are different from the imaging conditions applied to the main block for the same subject element as that pixel. However, if the imaging conditions applied to the main block for the same subject element as that pixel are the same as the imaging conditions applied to the boundary block, there is no need to perform the first correction process, and therefore the correction unit 33b does not perform the first correction process. As mentioned above, even if there are slight differences in imaging conditions, they will be considered to be the same imaging conditions. Furthermore, if the discontinuities in the image can be mitigated by performing image processing that reduces sharpness or contrast on the image data obtained by capturing images of boundary blocks or neighboring main blocks of boundary blocks, then the first correction process does not need to be performed.

[0399] 2. Perform the first correction process based on the pixel values. As another aspect of the first correction process, as described below, a correction process is performed based on the pixel values ​​of the signals output from the pixels of the boundary block and the pixel values ​​of the signals output from the pixels of the main block. (Example 1) The correction unit 33b calculates a correction coefficient α based on the average value of the pixel values ​​of the signals from each pixel of the boundary block, which was imaged with subject light from the mountain 64a under the first imaging condition, and the average value of the pixel values ​​of the signals from each pixel of the main block, which is adjacent to the boundary block, and which was imaged with subject light from the mountain 64a under the fourth imaging condition. The correction unit 33b then multiplies each of the pixel values ​​of the signals from each pixel of the boundary block, which was imaged with subject light from the mountain 64a, by the correction coefficient α.

[0400] For example, let's assume that Vave1 is the average value of the pixel signals from each pixel 285b to 285d of the boundary block 285, which is imaged with subject light from mountain 64a, and Vave2 is the average value of the pixel signals from each pixel 286a to 286d of the main block 286 adjacent to the boundary block 285. In this case, the correction unit 33b calculates α = Vave2 / Vave1 as a correction coefficient α for the pixel values ​​of the signals from pixels 285b to 285d of the boundary block 285. Then, the correction unit 33b multiplies the pixel value of the signal from pixel 285b of the boundary block 285 by the correction coefficient α and uses this value as the pixel value of the signal from pixel 285b after the first correction process. Similarly, the correction unit 33b multiplies the pixel value of the signal from pixel 285c of the boundary block 285 by the correction coefficient α and uses this value as the pixel value of the signal from pixel 285d after the first correction process. As described above, the difference between the pixel values ​​of the signals from pixels 285b to 285d of boundary block 285 and the pixel values ​​of the signals from each pixel of the main block adjacent to the boundary block is mainly due to the difference between the first and fourth imaging conditions. Therefore, by determining a correction coefficient α based on the difference in pixel values ​​and correcting the signals of pixels 285b to 285d with the correction coefficient α, the signals of pixels 285b to 285d become substantially identical to the signals obtained when the light incident on each pixel 285b to 285d is imaged under the fourth imaging condition.

[0401] In this way, the correction unit 33b performs the first correction process as needed on the signals from each pixel of all boundary blocks.

[0402] The above description has explained the first correction process in two different forms. The control unit 34 may decide which of these forms of the first correction process to perform based, for example, on the setting state (including the setting of the operation menu) by the operation member 36. Furthermore, the control unit 34 may determine which form of the first correction processing to perform based on the imaging scene mode set in camera 1 and the type of subject element detected.

[0403] <Second Correction Process> The correction unit 33b of the image processing unit 33 further performs the same correction process as in the first embodiment as necessary, before image processing, focus detection processing, subject detection (detection of subject elements) processing, and imaging condition setting processing. In the following description, the same correction process as in the first embodiment will also be referred to as the second correction process. The correction unit 33b performs the second correction process after the first correction process, which has been performed as necessary as described above. In the second correction process, the signals from the pixels of the boundary block corrected by the first correction process are processed as signals obtained by imaging with the imaging conditions set for the main block applied, rather than the imaging conditions set for the boundary block. For example, when performing the second correction process, the signals from the pixels 282d of the boundary block 282 corrected by the first correction process are processed by the correction unit 33b as signals obtained by imaging with the fourth imaging condition applied, rather than the first imaging condition.

[0404] According to the third embodiment described above, in addition to the effects and advantages of the first and second embodiments, the following effects and advantages can be obtained. (1) It is possible to process different regions with varying imaging conditions appropriately. In other words, it is possible to generate images appropriately using image data generated in each region. For example, it is possible to suppress discontinuities and inconsistencies that may appear in the generated images due to differences in imaging conditions for each region. Furthermore, because signals from pixels in boundary blocks can be appropriately corrected, image data can be generated appropriately. Furthermore, the amount of defocus can be appropriately detected based on the focus detection signal data generated in each region. For example, it is possible to suppress the decrease in focus detection accuracy due to differences in imaging conditions for each region. Furthermore, subject elements can be appropriately detected based on the image data generated in each region. For example, it is possible to suppress the decrease in detection accuracy due to differences in imaging conditions for each region. Furthermore, the imaging conditions can be appropriately set based on the image data generated in each region. For example, differences in imaging conditions for each region can suppress a decrease in the accuracy of exposure condition settings.

[0405] (2) Since the signals from the pixels of the boundary block can be appropriately corrected, image data can be generated appropriately. Furthermore, the amount of defocus can be appropriately detected based on the focus detection signal data generated in each region. Furthermore, subject elements can be appropriately detected based on the image data generated in each region. Additionally, imaging conditions can be appropriately set based on the image data generated in each region.

[0406] (3) A portion of the image data for the first region 61 is obtained by imaging light incident on a portion of the first region 61 adjacent to the fourth region 64. The correction unit 33b of the image processing unit 33 corrects a portion of the image data for the first region 61 according to the fourth imaging condition, so that the signals from the pixels of the boundary block can be appropriately corrected and image data can be appropriately generated. Furthermore, the amount of defocus can be appropriately detected based on the focus detection signal data generated in each region. Furthermore, subject elements can be appropriately detected based on the image data generated in each region. Additionally, imaging conditions can be appropriately set based on the image data generated in each region.

[0407] (4) The correction unit 33b of the image processing unit 33 corrects a portion of the image data for the first region 61 according to the fourth imaging condition, so that the signals from the pixels of the boundary block can be appropriately corrected and image data can be appropriately generated. Furthermore, the amount of defocus can be appropriately detected based on the focus detection signal data generated in each region. Furthermore, subject elements can be appropriately detected based on the image data generated in each region. Additionally, imaging conditions can be appropriately set based on the image data generated in each region.

[0408] (5) Among the pixels in the boundary block, the signals from pixels to which light from mountain 64a is incident can be appropriately corrected, and image data can be appropriately generated. Furthermore, the amount of defocus can be appropriately detected based on the focus detection signal data generated in each region. Furthermore, subject elements can be appropriately detected based on the image data generated in each region. Additionally, imaging conditions can be appropriately set based on the image data generated in each region.

[0409] ---Modified version of the third embodiment--- In the third embodiment described above, when the first correction processing is performed on the boundary blocks relating to the first region 61 and the fourth region 64 based on the imaging conditions, the correction coefficient α was set to the ratio of the first imaging conditions to the fourth imaging conditions. However, in this modified example, even when the first correction processing is performed on the boundary blocks relating to the first region 61 and the fourth region 64 based on the imaging conditions, the correction coefficient α is set to a value closer to 1 than the ratio of the first imaging conditions to the fourth imaging conditions. (Example 1) For example, if the only difference between the fourth imaging condition and the first imaging condition is the ISO sensitivity, and the ISO sensitivity of the fourth imaging condition is 800 and the ISO sensitivity of the first imaging condition is 100, the correction unit 33b will adopt a value closer to 1 than the ratio of the first imaging condition to the fourth imaging condition, which is 800 / 100 = 8, such as 7 or 6.5, as the value of the correction coefficient α. Note that the value of the correction coefficient α given here is merely an example, and the correction coefficient α is not limited to this value.

[0410] (Example 2) For example, if the only difference between the fourth imaging condition and the first imaging condition is the shutter speed, and the shutter speed for the fourth imaging condition is 1 / 100 second and the shutter speed for the first imaging condition is 1 / 1000 second, the correction unit 33b will adopt a value closer to 1 than the ratio of the first imaging condition to the fourth imaging condition, (1 / 100) / (1 / 1000)=10, such as 8.5 or 7, as the value of the correction coefficient α. Note that the value of the correction coefficient α given here is merely an example, and the correction coefficient α is not limited to this value.

[0411] In other words, by correcting the signal values ​​of each pixel 285b to 285d in the boundary block 285 where the first imaging condition is set, it is sufficient if the difference between the corrected signal value and the signal value of the pixels in the block where the fourth imaging condition is set is smaller (smoothed) than the difference between the signal value before correction and the signal value of the pixels in the block where the fourth imaging condition is set. In this way, if it is possible to mitigate the discontinuity that occurs in the image after image processing due to the existence of parts with different imaging conditions within the same region, the correction coefficient α may be calculated as in this modified example. For example, if the area containing the pixels to be corrected is small, or if the difference between the first imaging condition and the fourth imaging condition is small, the first correction process may be performed as shown in this modified example.

[0412] ---Fourth Embodiment--- Referring to Figures 29 and 31, a digital camera will be used as an example of an electronic device equipped with the image processing apparatus according to the fourth embodiment. In the following description, unless otherwise specified, the same principles apply as in the third embodiment. In the third embodiment, the pixel values ​​of signals from multiple pixels subject to correction processing within a boundary block were each multiplied by the same correction coefficient α. In contrast, in this embodiment, the pixel values ​​of signals from multiple pixels subject to correction processing within a single boundary block are multiplied by correction coefficients of different values ​​depending on their position.

[0413] <First Correction Process> (Example of the first correction process) 1. Perform the first correction process based on the imaging conditions. As one aspect of the first correction process in this embodiment, as described below, the correction process is performed based on the difference in imaging conditions set for two regions (first region 61 and fourth region 64) related to the boundary block. Specifically, even for pixels within the same region, if the imaging conditions applied to pixels within a boundary block differ from those applied to the main block, the correction coefficient α in the third embodiment is weighted according to the position of the pixel in the boundary block. The correction unit 33b calculates a correction coefficient α based on the imaging conditions, similar to (Example 1) and (Example 2) of the third embodiment. The correction unit 33b then changes the calculated correction coefficient α to different values ​​depending on the distance from the boundary of the region. For example, the correction unit 33b changes the value of the correction coefficient α so that the value approaches 1 as the distance from the boundary of the region decreases. That is, the correction unit 33b changes the value of the correction coefficient α so that the effect of the correction decreases as the position of the pixel to be corrected within the boundary block approaches the boundary of the region. The correction coefficient α after being changed in this way is called the correction coefficient β. The correction unit 33b multiplies the pixel value of the signal from the pixel to be corrected by the correction coefficient β.

[0414] (Example 1) For example, if the only difference between the fourth imaging condition and the first imaging condition is the ISO sensitivity, and the ISO sensitivity of the fourth imaging condition is 800 and the ISO sensitivity of the first imaging condition is 100, the correction unit 33b calculates a correction coefficient α = 800 / 100 = 8, similar to the third embodiment. Then, the correction unit 33b changes the value of the correction coefficient α so that the value approaches 1 as the distance from the boundary of the region decreases. For example, the correction unit 33b sets the correction coefficient β for the signals from each pixel 285b to 285d of the boundary block 285 as follows. Figure 31 is an enlarged view of the boundary block 285, with the centroids Gb of pixel 285b, Gc of pixel 285c, and Gd of pixel 285d indicated by black circles. As shown in Figure 31, the distances Lb and Lc between the centroid positions Gb and Gc of pixels 285b and 285c and the boundary B1 are smaller than the distance Ld between the centroid position Gd of pixel 285d and the boundary B1. Therefore, the correction unit 33b sets the value of the correction coefficient β for pixel 285d to, for example, 8, which is the same as the correction coefficient α, or 6, which is closer to 1 than the correction coefficient α, and sets the value of the correction coefficient β for pixels 285b and 285c to, for example, 4, which is even closer to 1. Note that the values ​​of the correction coefficient β given here are merely illustrative values ​​and are not limited to these values. The correction unit 33b uses the value obtained by multiplying the pixel value of the signal from pixel 285b of the boundary block 285 by a correction coefficient β=4 as the pixel value of the signal from pixel 285b after the first correction processing. Similarly, the correction unit 33b uses the value obtained by multiplying the pixel value of the signal from pixel 285c of the boundary block 285 by a correction coefficient β=4 as the pixel value of the signal from pixel 285c after the first correction processing, and uses the value obtained by multiplying the pixel value of the signal from pixel 285d by a correction coefficient β=6 or 8 as the pixel value of the signal from pixel 285d after the first correction processing. The signals of each pixel 285b to 285d in the boundary block 285, corrected by this first correction process, approach the signals obtained under the fourth imaging condition (ISO sensitivity 800) as they move away from boundary B1.

[0415] (Example 2) For example, if the only difference between the fourth imaging condition and the first imaging condition is the shutter speed, and the shutter speed for the fourth imaging condition is 1 / 100 second and the shutter speed for the first imaging condition is 1 / 1000 second, then, similar to the third embodiment, the correction coefficient α = (1 / 100) / (1 / 1000) = 10 is calculated. The correction unit 33b then changes the value of the correction coefficient α so that the value approaches 1 as the distance from the boundary of the region decreases. For example, the correction unit 33b sets the correction coefficient β for the signals from each pixel 285b to 285d of the boundary block 285 as follows. As described above, the distances Lb and Lc between the centroid positions Gb and Gc of pixels 285b and 285c and the boundary B1 are smaller than the distance Ld between the centroid position Gd of pixel 285d and the boundary B1. Therefore, the correction unit 33b sets the value of the correction coefficient β for pixel 285d to, for example, 10, which is the same as the correction coefficient α, or 7, which is closer to 1 than the correction coefficient α, and sets the value of the correction coefficient β for pixels 285b and 285c to, for example, 5, which is even closer to 1. Note that the values ​​of the correction coefficient β given here are merely illustrative values ​​and are not limited to these values. The correction unit 33b uses the value obtained by multiplying the pixel value of the signal from pixel 285b of the boundary block 285 by a correction coefficient β=5 as the pixel value of the signal from pixel 285b after the first correction processing. Similarly, the correction unit 33b uses the value obtained by multiplying the pixel value of the signal from pixel 285c of the boundary block 285 by a correction coefficient β=5 as the pixel value of the signal from pixel 285c after the first correction processing, and uses the value obtained by multiplying the pixel value of the signal from pixel 285d by a correction coefficient β=7 or 10 as the pixel value of the signal from pixel 285d after the first correction processing. The signals of each pixel 285b to 285d in the boundary block 285, corrected by this first correction process, approach the signals obtained when imaged under the fourth imaging condition (shutter speed of 1 / 100 second) as they move away from boundary B1.

[0416] In this way, the correction unit 33b performs the first correction process as needed on the signals from each pixel of all boundary blocks.

[0417] 2. Perform the first correction process based on the pixel values. As another aspect of the first correction process in this embodiment, as described below, the correction process is performed based on the signal values ​​(pixel values) output from the pixels of the boundary block and the signal values ​​(pixel values) output from the pixels of the main block. Specifically, even for pixels within the same region, if the imaging conditions applied to pixels within a boundary block differ from those applied to the main block, the correction coefficient α in the third embodiment is weighted according to the position of the pixel in the boundary block. The correction unit 33b calculates a correction coefficient α based on the imaging conditions, similar to (Example 1) of the third embodiment. The correction unit 33b then changes the calculated correction coefficient α to a different value depending on the distance from the boundary of the region. For example, the correction unit 33b changes the value of the correction coefficient α so that the value approaches 1 as the distance from the boundary of the region decreases. That is, the correction unit 33b changes the value of the correction coefficient α so that the effect of the correction decreases as the position of the pixel to be corrected within the boundary block approaches the boundary of the region. The correction coefficient α after being changed in this way is called the correction coefficient β, as in the case above. The correction unit 33b multiplies the pixel value of the signal from the pixel to be corrected by the correction coefficient β.

[0418] (Example 1) For example, suppose that Vave1 is the average value of the pixel signals from each pixel 285b to 285d of the boundary block 285, which is imaged with subject light from mountain 64a, and Vave2 is the average value of the pixel signals from each pixel 286a to 286d of the main block 286 adjacent to the boundary block 285. In this case, the correction unit 33b calculates α = Vave2 / Vave1 as the correction coefficient α for the pixel values ​​of the signals from pixels 285b to 285d of the boundary block 285. Then, the correction unit 33b changes the value of the correction coefficient α so that the value approaches 1 as the distance from the boundary of the region decreases. For example, the correction unit 33b sets the correction coefficient β for the signals from each pixel 285b to 285d of the boundary block 285 as follows. As described above, the distances Lb and Lc between the centroid positions Gb and Gc of pixels 285b and 285c and the boundary B1 are smaller than the distance Ld between the centroid position Gd of pixel 285d and the boundary B1. Therefore, the correction unit 33b sets the value of the correction coefficient β for pixel 285d to be the same as the correction coefficient α (i.e., Vave2 / Vave1) or closer to 1 than the correction coefficient α, for example, β=β1, and sets the value of the correction coefficient β for pixels 285b and 285c to be even closer to 1 than β1, for example, β=β2. Furthermore, if α (=Vave2 / Vave1) is greater than 1, then 1 ≤ β2 < β1 < α, and if α is less than 1, then α < β1 < β2 ≤ 1. The correction unit 33b uses the value obtained by multiplying the pixel value of the signal from pixel 285b of the boundary block 285 by the correction coefficient β=β2 as the pixel value of the signal from pixel 285b after the first correction processing. Similarly, the correction unit 33b uses the value obtained by multiplying the pixel value of the signal from pixel 285c of the boundary block 285 by the correction coefficient β=β2 as the pixel value of the signal from pixel 285c after the first correction processing, and uses the value obtained by multiplying the pixel value of the signal from pixel 285d by the correction coefficient β=β1 or Vave2 / Vave1 as the pixel value of the signal from pixel 285d after the first correction processing.

[0419] In this way, the correction unit 33b performs the first correction process as needed on the signals from each pixel of all boundary blocks.

[0420] The above description has explained the first correction process in two different forms. The control unit 34 may decide which of these forms of the first correction process to perform based, for example, on the setting state (including the setting of the operation menu) by the operation member 36. Furthermore, the control unit 34 may determine which form of the first correction processing to perform based on the imaging scene mode set in camera 1 and the type of subject element detected.

[0421] <Second Correction Process> The correction unit 33b of the image processing unit 33 further performs the same correction process as in the first embodiment as necessary, before image processing, focus detection processing, subject detection (detection of subject elements) processing, and imaging condition setting processing. In the following description, the same correction process as in the first embodiment will also be referred to as the second correction process. The correction unit 33b performs the second correction process after the first correction process, which has been performed as necessary as described above.

[0422] As described in the first to fourth embodiments above, the control unit 34 determines, based on the image data (pixel signal value) of block 85, whether or not to use the pixel values ​​of a block captured under different imaging conditions than those set for block 85. That is, if the image data of block 85 is overexposed or underexposed, the control unit 34 selects a pixel of a block captured under different imaging conditions than those set for block 85 and replaces the overexposed or underexposed image data with the image data (pixel signal value) of the selected pixel. The condition for using the pixel values ​​of a block captured under different imaging conditions than those set for block 85 may be that the image data (pixel signal value) of block 85 is above a first threshold or below a second threshold. If the image data of block 85 is not overexposed or underexposed, the control unit 34 uses the image data (pixel value) of block 85. In this case, the first correction process described in the third or fourth embodiment above is performed. Furthermore, even if there is no overexposure or underexposure in the image data of block 85, the control unit 34 may select pixels from a block captured under different imaging conditions than those set for block 85, and replace the image data with overexposure or underexposure using the image data (signal value of the pixel) of the selected pixels. The control unit 34 may also perform subject recognition and perform the first correction process based on the recognition result. For example, before the actual shooting, the setting unit 34b sets different imaging conditions from the first imaging conditions, and the object detection unit 34a performs subject recognition. Then, the signal values ​​of pixels that captured the same subject as the area to be corrected (for example, pixels 85b, 85d) captured under the first imaging conditions may be used. In this way, as described in the first to fourth embodiments, a subject captured under the first imaging conditions is corrected by the first correction process to appear as if it had been captured under the fourth imaging conditions.

[0423] In the above explanation, subject elements were detected using known subject recognition techniques. Examples of known subject recognition techniques include labeling and template matching, among others. Furthermore, in the above description, the control unit 34 performed subject element detection and region division using the live view image captured by the image sensor 32a. However, in the case of a camera equipped with a photometering sensor capable of capturing a subject image, such as a single-lens reflex camera, the control unit 34 may perform subject element detection and region division using the live view image captured by the photometering sensor. Furthermore, the embodiments and variations described above may be combined in any way.

[0424] Although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention.

[0425] The disclosures of the following priority application are incorporated herein by reference. Japanese Patent Application No. 71970 of 2016 (filed March 31, 2016) [Explanation of Symbols]

[0426] 1.1C...Camera 1B…Imaging System 31…Imaging Optical System 32…Imaging Unit 32a, 100.....

Claims

[Claim 1] An imaging unit having a first imaging region that is imaged under first imaging conditions and a second imaging region that is imaged under second imaging conditions different from the first imaging conditions, A generation unit that generates an image of a subject captured in the first imaging area using image data of the subject captured in the second imaging area, An imaging device equipped with the following features.

Images