Imaging device

Abstract

SOLUTION: An imaging device includes: an imaging unit having a first imaging region 61 for performing imaging under a first imaging condition and a second imaging region 64 for performing imaging under a second imaging condition different from the first imaging condition; and a generation unit for generating an image of an object imaged in the first imaging region by image data of the object imaged in the second imaging region.EFFECT: When inappropriate image data is generated because of the first imaging condition set in the first imaging region, in an image of an object image imaged in the first imaging region, image data can be properly generated by using an image of the object image imaged in the second imaging region under the second imaging condition different from the first imaging region.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an imaging device. [Background technology]

[0002] 2. Description of the Related Art Imaging devices incorporating image processing technology that generates an image from a signal from an imaging element are known (see Patent Document 1). There has been a demand for improving image quality. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2006-197192 Summary of the Invention

[0004] According to a first aspect, an imaging device includes an imaging unit having a first imaging area that is imaged under a first imaging condition and a second imaging area that is imaged under a second imaging condition different from the first imaging condition, and a generation unit that generates an image of a subject imaged in the first imaging area using image data of the subject imaged in the second imaging area. [Brief description of the drawings]

[0005] [Figure 1] 1 is a block diagram illustrating a configuration of a camera according to a first embodiment. [Diagram 2] FIG. 2 is a cross-sectional view of a stacked type imaging element. [Diagram 3] 2 is a diagram illustrating a pixel array and a unit area of ​​an imaging chip. FIG. [Figure 4] FIG. 2 is a diagram illustrating a circuit in a unit area. [Diagram 5] FIG. 2 is a diagram illustrating an example of an image of a subject formed on an image sensor of a camera. [Figure 6] FIG. 13 is a diagram illustrating an example of a setting screen for imaging conditions. [Figure 7]FIG. 7(a) is a diagram illustrating an example of a predetermined range in a live view image, and FIG. 7(b) is an enlarged view of the predetermined range. [Figure 8] FIG. 8 is a diagram illustrating image data corresponding to FIG. [Figure 9] FIG. 9(a) is a diagram illustrating an example of a region of interest in a live view image, and FIG. 9(b) is an enlarged view of a pixel of interest and a reference pixel Pr. [Figure 10] Figure 10(a) is a diagram illustrating an example of an arrangement of photoelectric conversion signals output from pixels, Figure 10(b) is a diagram explaining the interpolation of image data of the G color component, and Figure 10(c) is a diagram illustrating image data of the G color component after interpolation. [Figure 11] FIG. 11(a) is a diagram showing image data of the R color component extracted from FIG. 10(a), FIG. 11(b) is a diagram explaining the interpolation of the color difference component Cr, and FIG. 11(c) is a diagram explaining the interpolation of image data of the color difference component Cr. [Figure 12] FIG. 12(a) is a diagram showing image data of the B color component extracted from FIG. 10(a), FIG. 12(b) is a diagram explaining the interpolation of the color difference component Cb, and FIG. 12(c) is a diagram explaining the interpolation of image data of the color difference component Cb. [Figure 13] 4 is a diagram illustrating an example of the positions of focus detection pixels on an imaging surface; [Figure 14] FIG. 4 is an enlarged view of a partial area of ​​a focus detection pixel line. [Figure 15] FIG. 13 is an enlarged view of a focus point. [Figure 16] FIG. 16(a) is a diagram illustrating a template image representing an object to be detected, and FIG. 16(b) is a diagram illustrating a live view image and a search range. [Figure 17] 11 is a flowchart illustrating a process of setting imaging conditions for each region and capturing an image. [Figure 18] 18(a) to 18(c) are diagrams illustrating examples of the arrangement of the first imaging region and the second imaging region on the imaging surface of an imaging element. [Figure 19] FIG. 23 is a block diagram illustrating the configuration of an imaging system according to an eleventh modification. [Figure 20]FIG. 2 is a diagram illustrating the supply of a program to a mobile device. [Figure 21] FIG. 11 is a block diagram illustrating the configuration of a camera according to a second embodiment. [Figure 22] 13 is a diagram illustrating a schematic diagram of a correspondence relationship between each block and a plurality of correction units according to the second embodiment. FIG. [Diagram 23] FIG. 2 is a cross-sectional view of a stacked imaging element. [Figure 24] 3A to 3C are diagrams illustrating a schematic diagram of processing of first image data and second image data relating to image processing. [Diagram 25] 5A to 5C are diagrams illustrating the processing of first image data and second image data in relation to focus detection processing. [Figure 26] 5A to 5C are diagrams illustrating the processing of first image data and second image data in relation to the subject detection processing. [Figure 27] 5 is a diagram showing a schematic representation of processing of first image data and second image data relating to setting of imaging conditions such as exposure calculation processing. FIG. [Figure 28] 23 is a diagram showing a schematic diagram of processing of the first image data and the second image data according to a thirteenth modification example. FIG. [Figure 29] FIG. 29(a) is a diagram illustrating a predetermined range in a live view image, and FIG. 29(b) is an enlarged view of the predetermined range. [Diagram 30] FIG. 29(b) is a diagram illustrating image data corresponding to FIG. [Diagram 31] FIG. 2 is an enlarged view of a boundary block. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] ---First embodiment --- A digital camera will be described as an example of an electronic device equipped with an image processing device according to the first embodiment. Camera 1 (FIG. 1) is configured to be able to capture images under different conditions for each region of the imaging surface of imaging element 32a. Image processing unit 33 performs appropriate processing for each region with different imaging conditions. Details of camera 1 will be described with reference to the drawings.

[0007] <Camera description> Fig. 1 is a block diagram illustrating the configuration of a camera 1 according to the first embodiment. In Fig. 1, camera 1 has an imaging optical system 31, an imaging unit 32, an image processing unit 33, a control unit 34, a display unit 35, an operation member 36, and a recording unit 37.

[0008] The imaging optical system 31 guides a light beam from the subject field to the imaging section 32. The imaging section 32 includes an imaging element 32a and a driving section 32b, and photoelectrically converts the image of the subject formed by the imaging optical system 31. The imaging section 32 can capture images under the same conditions over the entire imaging surface of the imaging element 32a, or capture images under different conditions for each region of the imaging surface of the imaging element 32a. The imaging section 32 will be described in detail later. The driving section 32b generates a driving signal required for the imaging element 32a to perform accumulation control. Imaging instructions such as charge accumulation time for the imaging section 32 are transmitted from the control section 34 to the driving section 32b.

[0009] The image processing unit 33 includes an input unit 33a, a correction unit 33b, and a generation unit 33c. The input unit 33a receives image data acquired by the imaging unit 32. The correction unit 33b performs pre-processing to correct the input image data. The details of the pre-processing will be described later. The generation unit 33c performs image processing on the input image data and the pre-processed image data to generate an image. The image processing includes, for example, color interpolation processing, pixel defect correction processing, contour emphasis processing, noise reduction processing, white balance adjustment processing, gamma correction processing, display brightness adjustment processing, saturation adjustment processing, and the like. Furthermore, the generation unit 33c generates an image to be displayed by the display unit 35.

[0010] The control unit 34 is constituted by, for example, a CPU, and controls the overall operation of the camera 1. For example, the control unit 34 performs a predetermined exposure calculation based on the photoelectric conversion signal acquired by the imaging unit 32, and determines exposure conditions such as the charge accumulation time (exposure time) of the image sensor 32a required for proper exposure, the aperture value of the imaging optical system 31, and ISO sensitivity, and instructs the driving unit 32b. In addition, the control unit 34 determines image processing conditions for adjusting saturation, contrast, sharpness, etc. according to the imaging scene mode set in the camera 1 and the type of the detected subject element, and instructs the image processing unit 33. Detection of the subject element will be described later.

[0011] The control unit 34 includes an object detection unit 34a, a setting unit 34b, an imaging control unit 34c, and a lens movement control unit 34d. These are realized in software by the control unit 34 executing a program stored in a non-volatile memory (not shown), but they may also be configured by ASIC or the like.

[0012] The object detection unit 34a performs a known object recognition process to detect subject elements such as people (human faces), animals (animal faces) such as dogs and cats, plants, vehicles such as bicycles, automobiles, and trains, buildings, stationary objects, landscapes such as mountains and clouds, and predetermined specific objects from the image data acquired by the imaging unit 32. The setting unit 34b divides the image data acquired by the imaging unit 32 into a plurality of regions including the subject elements detected as described above.

[0013] The setting unit 34b further sets imaging conditions for the multiple regions. The imaging conditions include the above-mentioned exposure conditions (charge accumulation time, gain, ISO sensitivity, frame rate, etc.) and the above-mentioned image processing conditions (for example, white balance adjustment parameters, gamma correction curve, display brightness adjustment parameters, saturation adjustment parameters, etc.). It is possible to set the same imaging conditions for all of the multiple regions, or to set different imaging conditions between the multiple regions.

[0014] The imaging control unit 34c applies the imaging conditions set for each region by the setting unit 34b to control the imaging unit 32 (imaging element 32a) and image processing unit 33. This makes it possible for the imaging unit 32 to perform imaging under different exposure conditions for each of the multiple regions, and for the image processing unit 33 to perform image processing under different image processing conditions for each of the multiple regions. The number of pixels constituting each region may be any number, and may be, for example, 1000 pixels or 1 pixel. Furthermore, the number of pixels may differ between regions.

[0015] The lens movement control unit 34d controls an automatic focus adjustment (AF) operation that focuses on a corresponding subject at a predetermined position (called a focus point) on the imaging screen. When the focus is adjusted, the sharpness of the image of the subject is increased. That is, the image by the imaging optical system 31 is adjusted by moving the focus lens of the imaging optical system 31 in the optical axis direction. The lens movement control unit 34d sends a drive signal for moving the focus lens of the imaging optical system 31 to a focus position based on the calculation result, for example, a signal for adjusting the image of the subject with the focus lens of the imaging optical system 31, to the lens movement mechanism 31m of the imaging optical system 31. In this way, the lens movement control unit 34d functions as a movement unit that moves the focus lens of the imaging optical system 31 in the optical axis direction based on the calculation result. The process performed by the lens movement control unit 34d for the AF operation is also called a focus detection process. The details of the focus detection process will be described later.

[0016] The display unit 35 reproduces and displays images generated by the image processing unit 33, images that have been subjected to image processing, images that have been read out by the recording unit 37, etc. The display unit 35 also displays an operation menu screen, a setting screen for setting imaging conditions, etc.

[0017] The operation members 36 are composed of various operation members such as a release button, a menu button, etc. The operation members 36 send operation signals corresponding to each operation to the control unit 34. The operation members 36 also include a touch operation member provided on the display surface of the display unit 35.

[0018] The recording unit 37 records image data and the like in a recording medium such as a memory card (not shown) in response to an instruction from the control unit 34. The recording unit 37 also reads out image data recorded in the recording medium in response to an instruction from the control unit 34.

[0019] <Explanation of stacked image sensor> As an example of the above-mentioned imaging element 32a, a stacked imaging element 100 will be described. FIG. 2 is a cross-sectional view of the imaging element 100. The imaging element 100 includes an imaging chip 111, a signal processing chip 112, and a memory chip 113. The imaging chip 111 is stacked on the signal processing chip 112. The signal processing chip 112 is stacked on the memory chip 113. The imaging chip 111 and the signal processing chip 112, and the signal processing chip 112 and the memory chip 113 are electrically connected by a connection portion 109. The connection portion 109 is, for example, a bump or an electrode. The imaging chip 111 captures a light image from a subject and generates image data. The imaging chip 111 outputs the image data from the imaging chip 111 to the signal processing chip 112. The signal processing chip 112 performs signal processing on the image data output from the imaging chip 111. The memory chip 113 has a plurality of memories and stores the image data. The imaging element 100 may be composed of an imaging chip and a signal processing chip. When the imaging element 100 is composed of an imaging chip and a signal processing chip, a storage unit for storing image data may be provided in the signal processing chip or may be provided separately from the imaging element 100.

[0020] As shown in Figure 2, incident light mainly travels in the positive Z-axis direction, as indicated by the white arrow. Also, as shown by the coordinate axes, the left direction on the paper, perpendicular to the Z-axis, is the positive X-axis direction, and the front direction on the paper, perpendicular to the Z-axis and X-axis, is the positive Y-axis direction. In the following figures, the coordinate axes are displayed so that the orientation of each figure can be understood based on the coordinate axes in Figure 2.

[0021] The imaging chip 111 is, for example, a CMOS image sensor. Specifically, the imaging chip 111 is a back-illuminated CMOS image sensor. The imaging chip 111 has a microlens layer 101, a color filter layer 102, a passivation layer 103, a semiconductor layer 106, and a wiring layer 108. In the imaging chip 111, the microlens layer 101, the color filter layer 102, the passivation layer 103, the semiconductor layer 106, and the wiring layer 108 are arranged in this order in the positive direction of the Z axis.

[0022] The microlens layer 101 has a plurality of microlenses L. The microlenses L focus incident light on a photoelectric conversion unit 104, which will be described later. The color filter layer 102 has a plurality of color filters F. The color filter layer 102 has a plurality of types of color filters F with different spectral characteristics. Specifically, the color filter layer 102 has a first filter (R) with a spectral characteristic that mainly transmits red component light, a second filter (Gb, Gr) with a spectral characteristic that mainly transmits green component light, and a third filter (B) with a spectral characteristic that mainly transmits blue component light. In the color filter layer 102, the first filter, the second filter, and the third filter are arranged, for example, in a Bayer array. The passivation layer 103 is made of a nitride film or an oxide film, and protects the semiconductor layer 106.

[0023] The semiconductor layer 106 has a photoelectric conversion unit 104 and a readout circuit 105. The semiconductor layer 106 has a plurality of photoelectric conversion units 104 between a first surface 106a, which is a light incidence surface, and a second surface 106b opposite to the first surface 106a. The semiconductor layer 106 has a plurality of photoelectric conversion units 104 arranged in the X-axis direction and the Y-axis direction. The photoelectric conversion unit 104 has a photoelectric conversion function of converting light into electric charge. The photoelectric conversion unit 104 accumulates electric charge due to a photoelectric conversion signal. The photoelectric conversion unit 104 is, for example, a photodiode. The semiconductor layer 106 has a readout circuit 105 on the second surface 106b side of the photoelectric conversion unit 104. The semiconductor layer 106 has a plurality of readout circuits 105 arranged in the X-axis direction and the Y-axis direction. The readout circuit 105 is configured with a plurality of transistors, and reads out image data generated from charges photoelectrically converted by the photoelectric conversion unit 104, and outputs the image data to the wiring layer .

[0024] The wiring layer 108 has a plurality of metal layers. The metal layers are, for example, Al wiring, Cu wiring, etc. The wiring layer 108 outputs image data read by the readout circuit 105. The image data is output from the wiring layer 108 to the signal processing chip 112 via the connection unit 109.

[0025] The connection portion 109 may be provided for each photoelectric conversion portion 104. Furthermore, the connection portion 109 may be provided for each of a plurality of photoelectric conversion portions 104. When the connection portion 109 is provided for each of a plurality of photoelectric conversion portions 104, the pitch of the connection portions 109 may be larger than the pitch of the photoelectric conversion portions 104. Furthermore, the connection portions 109 may be provided in a peripheral region of the region in which the photoelectric conversion portions 104 are arranged.

[0026] The signal processing chip 112 has a plurality of signal processing circuits. The signal processing circuits perform signal processing on the image data output from the imaging chip 111. The signal processing circuits are, for example, an amplifier circuit that amplifies the signal value of the image data, a correlated double sampling circuit that performs noise reduction processing on the image data, and an analog / digital (A / D) conversion circuit that converts an analog signal into a digital signal. A signal processing circuit may be provided for each photoelectric conversion unit 104.

[0027] Furthermore, a signal processing circuit may be provided for each of the photoelectric conversion units 104. The signal processing chip 112 has a plurality of through electrodes 110. The through electrodes 110 are, for example, silicon through electrodes. The through electrodes 110 connect circuits (not shown) provided in the signal processing chip 112 to each other. The through electrodes 110 may also be provided in a peripheral region of the imaging chip 111 and in the memory chip 113. Note that some of the elements constituting the signal processing circuit may be provided in the imaging chip 111. For example, in the case of an analog / digital conversion circuit, a comparator that compares an input voltage with a reference voltage may be provided in the imaging chip 111, and circuits such as a counter circuit and a latch circuit may be provided in the signal processing chip 112.

[0028] The memory chip 113 has a plurality of storage units. The storage units store image data that has been subjected to signal processing by the signal processing chip 112. The storage units are, for example, volatile memories such as DRAMs. A storage unit may be provided for each photoelectric conversion unit 104. Also, a storage unit may be provided for each of the plurality of photoelectric conversion units 104. The image data stored in the storage units is output to a downstream image processing unit.

[0029] FIG. 3 is a diagram for explaining the pixel array of the imaging chip 111 and the unit area 131. In particular, the imaging chip 111 is shown as observed from the back surface (imaging surface) side. For example, 20 million or more pixels are arranged in a matrix in the pixel area. In the example of FIG. 3, four pixels, 2 pixels by 2 pixels adjacent to each other, form one unit area 131. The grid lines in the figure show the concept of forming the unit area 131 by grouping adjacent pixels. The number of pixels forming the unit area 131 is not limited to this, and may be about 1000, for example, 32 pixels by 32 pixels, or may be more or less than that, or may be 1 pixel.

[0030] As shown in the partially enlarged view of the pixel region, the unit region 131 in Fig. 3 includes a so-called Bayer array consisting of four pixels: green pixels Gb, Gr, blue pixel B, and red pixel R. The green pixels Gb and Gr are pixels having a green filter as a color filter F, and receive light in the green wavelength band of incident light. Similarly, the blue pixel B is a pixel having a blue filter as a color filter F, and receives light in the blue wavelength band, and the red pixel R is a pixel having a red filter as a color filter F, and receives light in the red wavelength band.

[0031] In this embodiment, a plurality of blocks are defined so that each block includes at least one unit area 131. That is, the minimum unit of one block is one unit area 131. As described above, the smallest possible value of the number of pixels forming one unit area 131 is one pixel. Therefore, when one block is defined in pixel units, the smallest possible number of pixels that can define one block is one pixel. Each block can control the pixels included in each block with different control parameters. In each block, all unit areas 131 in that block, that is, all pixels in that block are controlled under the same imaging conditions. That is, photoelectric conversion signals with different imaging conditions can be obtained between a pixel group included in one block and a pixel group included in another block. Examples of the control parameters are a frame rate, a gain, a thinning rate, the number of summation rows or columns for summing photoelectric conversion signals, the charge accumulation time or number of accumulations, the number of digitization bits (word length), and the like. The imaging element 100 can freely perform thinning not only in the row direction (X-axis direction of the imaging chip 111) but also in the column direction (Y-axis direction of the imaging chip 111). Furthermore, the control parameters may be parameters in image processing.

[0032] Fig. 4 is a diagram for explaining a circuit in a unit area 131. In the example of Fig. 4, one unit area 131 is formed by four pixels, ie, 2 pixels x 2 pixels adjacent to each other. As described above, the number of pixels included in the unit area 131 is not limited to this, and may be 1000 pixels or more, or may be as small as 1 pixel. The two-dimensional position of the unit area 131 is indicated by symbols A to D.

[0033] The reset transistors (RST) of the pixels included in the unit region 131 are configured to be able to be turned on and off individually for each pixel. In Fig. 4, a reset wiring 300 is provided to turn on and off the reset transistor of pixel A, and a reset wiring 310 to turn on and off the reset transistor of pixel B is provided separately from the reset wiring 300. Similarly, a reset wiring 320 to turn on and off the reset transistor of pixel C is provided separately from the reset wirings 300 and 310. A dedicated reset wiring 330 is also provided for another pixel D to turn on and off the reset transistor.

[0034] The transfer transistors (TX) of the pixels included in the unit region 131 are also configured to be able to be turned on and off individually for each pixel. In Fig. 4, a transfer wiring 302 that turns on and off the transfer transistor of pixel A, a transfer wiring 312 that turns on and off the transfer transistor of pixel B, and a transfer wiring 322 that turns on and off the transfer transistor of pixel C are provided separately. For another pixel D, a dedicated transfer wiring 332 is also provided to turn on and off the transfer transistor.

[0035] Furthermore, the selection transistors (SEL) of the pixels included in the unit region 131 are also configured to be able to be turned on and off individually for each pixel. In Fig. 4, a selection wiring 306 that turns on and off the selection transistor of pixel A, a selection wiring 316 that turns on and off the selection transistor of pixel B, and a selection wiring 326 that turns on and off the selection transistor of pixel C are provided separately. For another pixel D, a dedicated selection wiring 336 is also provided to turn on and off the selection transistor.

[0036] The power supply wiring 304 is commonly connected to pixels A to D included in the unit area 131. Similarly, the output wiring 308 is commonly connected to pixels A to D included in the unit area 131. The power supply wiring 304 is commonly connected between a plurality of unit areas, but the output wiring 308 is provided individually for each unit area 131. The load current source 309 supplies a current to the output wiring 308. The load current source 309 may be provided on the imaging chip 111 side or on the signal processing chip 112 side.

[0037] By individually turning on and off the reset transistors and transfer transistors of the unit area 131, it is possible to control charge accumulation including the accumulation start time, accumulation end time, and transfer timing for pixels A to D included in the unit area 131. In addition, by individually turning on and off the selection transistors of the unit area 131, it is possible to output the photoelectric conversion signals of each of pixels A to D via a common output wiring 308.

[0038] Here, a so-called rolling shutter method is known in which charge accumulation is controlled in a regular order for rows and columns for pixels A to D included in the unit area 131. When pixels are selected for each row using the rolling shutter method and then columns are specified, photoelectric conversion signals are output in the order of "ABCD" in the example of FIG.

[0039] In this way, by configuring the circuit based on the unit area 131, it is possible to control the charge accumulation time for each unit area 131. In other words, it is possible to output photoelectric conversion signals at different frame rates between the unit areas 131. Also, by resting the unit areas 131 included in some blocks of the imaging chip 111 while causing the unit areas 131 included in other blocks to accumulate charge (imaging), it is possible to cause imaging to be performed only in a predetermined block of the imaging chip 111 and output the photoelectric conversion signal. Furthermore, it is also possible to switch the block (block subject to accumulation control) that accumulates charge (imaging) between frames, and cause different blocks of the imaging chip 111 to sequentially capture images and output photoelectric conversion signals.

[0040] As described above, output wiring 308 is provided corresponding to each unit area 131. Since image pickup element 100 has imaging chip 111, signal processing chip 112, and memory chip 113 stacked on top of each other, by using electrical connection between the chips using connection parts 109 for output wiring 308, it is possible to route wiring without increasing the size of each chip in the planar direction.

[0041] <Image sensor block control> In this embodiment, imaging conditions can be set for each of a plurality of blocks in the image sensor 32a. The imaging control unit 34c of the control unit 34 associates the plurality of regions with the blocks and performs imaging under the imaging conditions set for each region.

[0042] 5 is a diagram showing a schematic diagram of an image of a subject formed on the imaging element 32a of the camera 1. Before an image capturing instruction is issued, the camera 1 photoelectrically converts the subject image to obtain a live view image. A live view image is an image for monitoring that is repeatedly captured at a predetermined frame rate (e.g., 60 fps).

[0043] Before the setting unit 34b divides the region, the control unit 34 sets the same imaging conditions for the entire imaging chip 111 (i.e., the entire imaging screen). The same imaging conditions refer to setting common imaging conditions for the entire imaging screen, and are considered to be the same even if there is a variation of less than about 0.3 stops in the Apex value, for example. The imaging conditions set to be the same for the entire imaging chip 111 are determined based on the exposure conditions according to the photometric value of the subject brightness, or the exposure conditions manually set by the user.

[0044] 5, an image including a person 61a, a car 62a, a bag 63a, a mountain 64a, and clouds 65a and 66a is formed on the imaging surface of the imaging chip 111. The person 61a is holding the bag 63a with both hands. A car 62a is parked to the right rear of the person 61a.

[0045] <Dividing the area> Based on the live view image, the control unit 34 divides the screen of the live view image into a plurality of regions as follows. First, the object detection unit 34a detects subject elements from the live view image. The detection of the subject elements uses a known subject recognition technique. In the example of Fig. 5, the object detection unit 34a detects a person 61a, a car 62a, a bag 63a, a mountain 64a, a cloud 65a, and a cloud 66a as subject elements.

[0046] Next, the setting unit 34b divides the screen of the live view image into regions including the above-mentioned subject elements. In this embodiment, the region including the person 61a is a first region 61, the region including the car 62a is a second region 62, the region including the bag 63a is a third region 63, the region including the mountain 64a is a fourth region 64, the region including the cloud 65a is a fifth region 65, and the region including the cloud 66a is a sixth region 66.

[0047] <Setting imaging conditions for each block> When the control unit 34 divides the screen into a plurality of regions by the setting unit 34b, the control unit 34 causes the display unit 35 to display a setting screen as exemplified in Fig. 6. In Fig. 6, a live view image 60a is displayed, and a setting screen 70 for setting imaging conditions is displayed to the right of the live view image 60a.

[0048] The setting screen 70 lists, from the top, frame rate, shutter speed (TV), and gain (ISO) as examples of setting items for imaging conditions. The frame rate is the number of frames of live view images captured per second or moving images recorded by the camera 1. The gain is the ISO sensitivity. Setting items for imaging conditions may be added as appropriate in addition to those exemplified in FIG. 6. If all setting items do not fit within the setting screen 70, other setting items may be displayed by scrolling the setting items up and down.

[0049] In this embodiment, the control unit 34 sets the area selected by the user among the areas divided by the setting unit 34b as the target for setting (changing) the imaging conditions. For example, in the camera 1 capable of touch operation, the user taps the display position of the main subject for which the imaging conditions are to be set (changed) on the display surface of the display unit 35 on which the live view image 60a is displayed. For example, when the display position of a person 61a is tapped, the control unit 34 sets the first area 61 including the person 61a in the live view image 60a as the target area for setting (changing) the imaging conditions, and displays the outline of the first area 61 with emphasis.

[0050] In FIG. 6, a first region 61 displayed with an emphasized outline (bold, bright, changed color, dashed, blinking, etc.) indicates a region for which imaging conditions are to be set (changed). In the example of FIG. 6, a live view image 60a is displayed with an emphasized outline of the first region 61. In this case, the first region 61 is a region for which imaging conditions are to be set (changed). For example, in a camera 1 capable of touch operation, when a user taps on the shutter speed (TV) display 71, the control unit 34 causes the current setting value of the shutter speed for the emphasized region (first region 61) to be displayed on the screen (reference numeral 68). In the following explanation, the camera 1 will be described on the assumption that touch operations are performed. However, the imaging conditions may be set (changed) by operating the buttons and the like that constitute the operation member .

[0051] When the user taps the upper icon 71a or the lower icon 71b of the shutter speed (TV), the setting unit 34b increases or decreases the shutter speed display 68 from the current setting value in response to the tap operation, and sends an instruction to the imaging unit 32 (FIG. 1) to change the imaging conditions of the unit area 131 (FIG. 3) of the imaging element 32a corresponding to the area (first area 61) displayed in an emphasized manner in response to the tap operation. The decision icon 72 is an operation icon for finalizing the set imaging conditions. The setting unit 34b also sets (changes) the frame rate and gain (ISO) in the same manner as in the case of setting (changing) the shutter speed (TV).

[0052] In addition, although the setting unit 34b has been described as setting the imaging conditions based on the user's operation, the present invention is not limited to this. The setting unit 34b may set the imaging conditions based on the judgment of the control unit 34, not based on the user's operation. For areas that are not highlighted (areas other than the first area 61), the imaging conditions that have been set are maintained.

[0053] Instead of highlighting the outline of the region for which the imaging conditions are to be set (changed), the control unit 34 may display the entire target region brightly, display the entire target region with increased contrast, or display the entire target region with blinking. The target region may also be surrounded by a frame. The frame surrounding the target region may be displayed as a double frame or a single frame, and the display mode of the surrounding frame, such as the line type, color, brightness, etc., may be changed as appropriate. The control unit 34 may also display an arrow or the like near the target region to indicate the region for which the imaging conditions are to be set. The control unit 34 may display the region other than the target region for which the imaging conditions are to be set (changed) darkly, or display the region other than the target region with reduced contrast.

[0054] As described above, after the imaging conditions for each region are set, when a release button (not shown) constituting the operation member 36 or a display (release icon) instructing the start of imaging is operated, the control unit 34 controls the imaging unit 32 to capture images under the imaging conditions set for each of the divided regions. Then, the image processing unit 33 performs image processing on the image data acquired by the imaging unit 32. As described above, the image processing can be performed under different image processing conditions for each region.

[0055] After the image processing by the image processing unit 33, the recording unit 37 receives an instruction from the control unit 34 and records the image data after the image processing in a recording medium such as a memory card (not shown). This completes the series of imaging processes.

[0056] <First correction process> The correction unit 33b of the image processing unit 33 performs a first correction process as necessary as one of pre-processing processes performed before image processing, focus detection processing, subject detection (detection of subject elements) processing, and processing for setting imaging conditions.

[0057] As described above, in this embodiment, after the area of ​​the imaging screen is divided by the setting unit 34b, it is possible to set (change) the imaging conditions for the area selected by the user or the area determined by the control unit 34. For example, the divided regions are respectively designated as a first region 61 to a sixth region 66 (see FIG. 7(a)), and different first to sixth imaging conditions are set for the first to sixth regions 61 to 66. In such a case, there exist blocks including the boundaries between the first to sixth regions 61 to 66. As described above, a block is the smallest unit for which imaging conditions can be set individually in the imaging element 32a.

[0058] FIG. 7(a) is a diagram illustrating a predetermined range 80 including the boundary between the first region 61 and the fourth region 64 in the live view image 60a. FIG. 7(b) is a diagram in which the predetermined range 80 in FIG. 7(a) is enlarged. In FIG. 7(b), the predetermined range 80 includes a plurality of blocks 81 to 89. In this example, the blocks 81 and 84 capturing an image of a person are included in the first region 61, and the blocks 82, 85, and 87 capturing an image of a person and a mountain are also included in the first region 61. Therefore, a first imaging condition is set for the blocks 81, 82, 84, 85, and 87. Meanwhile, the blocks 83, 86, 88, and 89 capturing an image of a mountain are included in the fourth region 64. Therefore, a fourth imaging condition is set for the blocks 83, 86, 88, and 89.

[0059] The white areas in Fig. 7(b) indicate areas corresponding to people. The shaded areas in Fig. 7(b) indicate areas corresponding to mountains. Blocks 82, 85, and 87 include the boundary B1 between the first region 61 and the fourth region 64. The shaded areas in Fig. 7(b) indicate areas corresponding to mountains.

[0060] In this embodiment, since a block is the smallest unit for setting an imaging condition, the same imaging condition is set for one block. As described above, the first imaging condition is set for blocks 82, 85, and 87 including the boundary B1 between the first region 61 and the fourth region 64, and therefore the first imaging condition is also set for the shaded areas of blocks 82, 85, and 87, i.e., the portions corresponding to the mountain. In other words, an imaging condition different from the fourth imaging condition set for blocks 83, 86, 88, and 89 capturing an image of a mountain is set for the shaded areas in blocks 82, 85, and 87.

[0061] In this case, the brightness, contrast, color, etc. of the image may differ between the shaded areas of blocks 82, 85, and 87 and the shaded areas of blocks 83, 86, 88, and 89. In an extreme example, it is assumed that the image data corresponding to the shaded areas may be blown out or crushed. For example, in block 85, the first imaging condition suitable for a person may not be suitable for the shaded area (i.e., the mountain portion) of block 85, and blown out or crushed in the image data corresponding to the shaded area may be caused. Blown out highlights refer to the loss of gradation in data of high brightness areas of an image due to overexposure. Crushed blacks refer to the loss of gradation in data of low brightness areas of an image due to underexposure.

[0062] FIG. 8 is a diagram illustrating image data corresponding to FIG. 7(b). In FIG. 8, each of blocks 81 to 89 is assumed to be composed of four pixels, ie, 2 pixels by 2 pixels. Of these, it is assumed that pixels 85b and 85d in block 85 located at the center of FIG. 8 have black crush. Correction unit 33b according to this embodiment corrects the image by performing a replacement process in which image data in a block where whiteout or black crush has occurred is replaced with image data of another block in the same captured image. This correction is referred to as a first correction process.

[0063] When a block includes boundaries of an area based on a plurality of subject elements, such as block 85, and when the image data of this block 85 includes whiteout or blackout, the correction unit 33b performs a first correction process on all blocks that include whiteout or blackout. It should be noted that the first correction process is not necessary if there is no whiteout or blackout.

[0064] (Example of the first correction process) The correction unit 33b performs a first correction process on a block including image data with whiteout or blackout as a block of interest. Here, the block of interest is an area including image data with whiteout or blackout, but the block of interest does not have to be completely whiteout or blackout. For example, an area with a signal value equal to or greater than a first threshold or equal to or less than a second threshold may be the block of interest. In the example of FIG. 7(b) and FIG. 8, eight blocks around the block of interest 85 included in a predetermined range 80 (e.g., 3×3 blocks) centered on the block of interest 85 are set as reference blocks. That is, blocks 81 to 84 and blocks 86 to 89 around the block of interest 85 are the reference blocks. The number of blocks constituting the predetermined range 80 is not limited to the above 3×3 blocks, and may be changed as appropriate.

[0065] 1. The same correction is performed on the entire area where whiteout or blackout has occurred. (1-1) As a first correction process, the correction unit 33b corrects a portion of an area in the target block using image data acquired in one reference block. Specifically, the correction unit 33b corrects all of the image data where whiteout or blackout has occurred using image data acquired in one reference block. At this time, the area of ​​the reference block is the same as that of the target block. The process (1-1) can be performed in any of the following modes (i) to (iv), for example.

[0066] (i) The correction unit 33b replaces the image data in the target block where whiteout or blackout occurs with image data acquired from one reference block located closest to the whiteout or blackout area among the reference blocks located around the target block. Even if there are multiple whiteout or blackout pixels in the target block, the image data of the multiple whiteout or blackout pixels is replaced with the same image data acquired from the reference block located closest to the reference block. For example, based on image data corresponding to pixels 86a to 86d included in the reference block 86 located closest to the blackout pixels (pixels 85b and 85d) among the reference blocks 81 to 84 and 86 to 89 around the target block 85, the image data corresponding to the blackout pixel 85b and the image data corresponding to the blackout pixel 85d are replaced with the same data (for example, image data corresponding to pixel 86c).

[0067] (ii) The correction unit 33b replaces the image data in the target block in which whiteout or blackout has occurred with the same data, using image data acquired from one reference block selected from the reference blocks with the most frequently set imaging condition (fourth imaging condition in this example) for the same target element (mountain) as the whiteout or blackout target element (e.g. mountain) among the reference blocks located around the target block. For example, based on image data corresponding to pixels 88a to 88d included in one reference block selected from the reference blocks 83, 86, 88, and 89 in which the fourth imaging condition is set for the mountain among the reference blocks 81 to 84, 86 to 89 around the target block 85, for example, the correction unit 33b replaces the image data corresponding to the blackout pixel 85b and the image data corresponding to the blackout pixel 85d with the same data (e.g. image data corresponding to pixel 88b). In this way, the correction unit 33b may replace the blackout pixel 85d with a part of the pixels of the reference block.

[0068] (iii) The correction unit 33b may select a pixel that is closer to the blown-out or crushed-black pixel in the target block from the image data corresponding to the four pixels acquired in one reference block selected by (i) or (ii) above. Specifically, the correction unit 33b replaces the crushed-black pixel 85b with the pixel 86a that is closer to the crushed-black pixel 85b than the pixel 86a. Here, the interval is the interval between the centers of the crushed-black pixel 85b and the pixel 86a when the crushed-black pixel 85b and the pixel 86a are taken as an example. The interval may also be the interval between the centers of the crushed-black pixel 85b and the pixel 86a. Furthermore, when the crushed-black pixels are consecutive (the crushed-black pixel 85b and the crushed-black pixel 86a), the interval may be the center or center of gravity of the two crushed-black pixel clusters. The same applies to the pixel 86a in the reference block. Furthermore, the correction unit 33b may replace the image data in which whiteout or blackout occurs by using image data corresponding to adjacent pixels. For example, when the correction unit 33b selects the reference block 86, the correction unit 33b replaces the image data corresponding to the blackout pixel 85b and the image data corresponding to the blackout pixel 85d by the same data (image data corresponding to the pixel 86a in the reference block 86 or the pixel 86c in the reference block 86).

[0069] (iv) The correction unit 33b may replace the image data in the target block in which whiteout or blackout has occurred, using image data generated based on image data corresponding to four pixels acquired in one reference block selected by (i) or (ii) above. For example, when the correction unit 33b selects the reference block 88, the correction unit 33b replaces the image data corresponding to the blackout pixel 85b and the image data corresponding to the blackout pixel 85d with image data based on a plurality of pixels in the reference block 88 (for example, the average value of the image data corresponding to the pixels 88a to 88d included in the reference block 88).

[0070] In addition, when calculating the average value of the image data, instead of the simple average, a weighted average value weighted according to the distance from the pixel where whiteout or blackout has occurred may be used. For example, pixel 88b is closer to blackout pixel 85d than pixel 88d, so the contribution rate of the image data corresponding to pixel 88b is weighted higher than the contribution rate of the image data corresponding to pixel 88d.

[0071] In addition, instead of calculating the average value of the image data corresponding to pixels 88a to 88d included in reference block 88, the intermediate value of the image data corresponding to pixels 88a to 88d may be calculated, and the image data corresponding to blacked-out pixel 85b and pixel 85d may be replaced with this intermediate value.

[0072] (1-2) As a first correction process, the correction unit 33b uses image data acquired from multiple reference blocks to replace all image data in the target block that has whiteout or blackout. Here, multiple reference block candidates for replacing the blackout pixels (85b, 85d) are extracted. Ultimately, pixels in one block are used for replacement. The process (1-2) can be performed in any of the following modes (i) to (iv), for example.

[0073] (i) The correction unit 33b replaces the image data in the target block where whiteout or blackout has occurred with the same data, using image data acquired from a plurality of reference blocks around the whiteout or blackout region among the reference blocks located around the target block. For example, the correction unit 33b replaces the image data in the target block where whiteout or blackout has occurred with the same data, based on the image data corresponding to the pixels 86a to 86d and 88a to 88d included in the two reference blocks 86 and 88 adjacent to the blackout (pixels 85b and 85d) among the reference blocks 81 to 84 and 86 to 89 around the target block 85, as follows. For example, the image data corresponding to the blackout pixel 85b and the image data corresponding to the blackout pixel 85d are replaced with the same data (for example, the image data corresponding to the pixel 88b). At this time, the areas of the blackout pixel 85b and the blackout pixel 85d replaced by the pixel 88b are smaller than the area of ​​the reference block 88.

[0074] (ii) The correction unit 33b replaces the image data in the target block in which whiteout or blackout has occurred with the same data, using image data acquired from a plurality of reference blocks selected from the reference blocks with the most frequently set imaging condition (fourth imaging condition in this example) for the same target element (mountain) as the whiteout or blackout target element (e.g. mountain) among the reference blocks located around the target block 85. For example, based on image data corresponding to pixels 86a-86d and 88a-88d included in two reference blocks selected from the reference blocks 83, 86, 88, and 89 in which the fourth imaging condition is set for the mountain among the reference blocks 81-84, 86-89 around the target block 85, the correction unit 33b replaces the image data corresponding to the blackout pixel 85b and the image data corresponding to the blackout pixel 85d with the same data (e.g. image data corresponding to pixel 86c).

[0075] (iii) The correction unit 33b may replace the image data in which whiteout or blackout has occurred with image data corresponding to pixels adjacent to the whiteout or blackout pixel in the target block among the image data corresponding to the multiple pixels acquired in the multiple reference blocks selected by (i) or (ii) above. For example, when the correction unit 33b selects the reference blocks 86 and 88, the correction unit 33b replaces the image data corresponding to the blackout pixel 85b and the image data corresponding to the blackout pixel 85d with the same data (image data corresponding to the pixel 86a or pixel 86c of the reference block 86, or the pixel 86c or pixel 88a of the reference block 88).

[0076] (iv) The correction unit 33b may replace the image data in the target block in which whiteout or blackout has occurred, using image data generated based on image data corresponding to a plurality of pixels acquired in a plurality of reference blocks selected by the above (i) or (ii). For example, when the correction unit 33b selects the reference blocks 86 and 88, the correction unit 33b replaces the image data corresponding to the blackout pixel 85b and the image data corresponding to the blackout pixel 85d with the same data (the average value of the image data corresponding to the pixels 86a to 86d included in the reference block 86 and the image data corresponding to the pixels 88a to 88d included in the reference block 88). At this time, the area of ​​the pixel used for replacement is larger than the area of ​​the blackout pixels 85b and 85d.

[0077] When calculating the average value of the image data, a weighted average value weighted according to the distance from the pixel where whiteout or blackout has occurred may be used instead of a simple average. For example, pixel 86a is closer to blackout pixel 85b than pixel 86b, so a weight is applied so that the contribution rate of the image data corresponding to pixel 86a is higher than that of the image data corresponding to pixel 86b.

[0078] In addition, instead of calculating the average value of the image data corresponding to pixels 86a-86d and pixels 88a-88d contained in reference blocks 86 and 88, the intermediate value of the image data corresponding to pixels 86a-86d and pixels 88a-88d may be calculated, and the image data corresponding to blacked-out pixels 85b and pixel 85d may be replaced with this intermediate value.

[0079] 2. A plurality of corrections are performed on the entire area where whiteout or blackout has occurred. (2-1) As a first correction process, the correction unit 33b replaces all of the image data in the target block where whiteout or blackout has occurred with image data acquired from one reference block. The process (2-1) may be performed in any of the following modes (i) to (iii), for example.

[0080] (i) The correction unit 33b replaces multiple pieces of image data in the target block where whiteout or blackout has occurred with different data, using image data corresponding to pixels adjacent to the whiteout or blackout pixel in the reference blocks located around the target block. For example, the correction unit 33b replaces the multiple pieces of image data in the target block where whiteout or blackout has occurred with different data, based on image data corresponding to pixels 86a to 86d included in the reference block 86 located adjacent to the blackout (pixels 85b and 85d) among the reference blocks 81 to 84 and 86 to 89 around the target block 85, as follows. For example, the image data corresponding to the blackout pixel 85b is replaced with the image data of pixel 86a in the adjacent reference block 86, and the image data corresponding to the blackout pixel 85d is replaced with the image data of pixel 86c in the adjacent reference block 86.

[0081] (ii) The correction unit 33b replaces multiple pieces of image data in the target block where whiteout or blackout has occurred with different data, using image data acquired from one reference block selected from the reference blocks with the most frequently set imaging condition (fourth imaging condition in this example) for the same target element (mountain) as the whiteout or blackout target element (e.g. mountain) among the reference blocks located around the target block. For example, the correction unit 33b replaces multiple pieces of image data in the target block where whiteout or blackout has occurred with different data, using image data acquired from one reference block selected from the reference blocks 83, 86, 88, and 89 with the fourth imaging condition set for the mountain among the reference blocks 81-84, 86-89 around the target block 85, for example, image data corresponding to pixels 86a-86d included in the reference block 86, as follows. For example, the image data corresponding to the blackout pixel 85b is replaced with the image data of pixel 86b in the reference block 86, and the image data corresponding to the blackout pixel 85d is replaced with the image data of pixel 86d in the reference block 86.

[0082] (iii) The correction unit 33b may use image data generated based on image data corresponding to four pixels acquired in one reference block selected by (i) or (ii) above to perform the replacement. For example, when the correction unit 33b selects the reference block 86, the correction unit 33b replaces the image data corresponding to the black crushed pixel 85b in the target block with the average value of the image data corresponding to the pixels 86a and 86b included in the reference block 86. The correction unit 33b also replaces the image data corresponding to the black crushed pixel 85d in the target block with the average value of the image data corresponding to the pixels 86c and 86d included in the reference block 86.

[0083] When calculating the average value of image data, a weighted average value weighted according to the distance from a pixel where whiteout or blackout has occurred may be used instead of a simple average. For example, since the image data corresponding to pixel 86a is closer to the image data corresponding to blackout pixel 85b than the image data corresponding to pixel 86b, the image data corresponding to pixel 86a is weighted so that the contribution rate of the image data corresponding to pixel 86a is higher than that of the image data corresponding to pixel 86b.

[0084] (2-2) As a first correction process, the correction unit 33b replaces all image data in the target block in which whiteout or blackout has occurred, using image data acquired from a plurality of reference blocks. The process (2-2) may be performed in any of the following modes (i) to (iii).

[0085] (i) The correction unit 33b replaces multiple pieces of image data in the target block where whiteout or blackout has occurred with different data, using image data acquired from multiple reference blocks around the whiteout or blackout region among the reference blocks located around the target block. For example, the correction unit 33b replaces multiple pieces of image data in the target block where whiteout or blackout has occurred with different data, based on image data corresponding to pixels 86a to 86d and 88a to 88d included in two reference blocks 86 and 88 adjacent to the blackout (pixels 85b and 85d) among the reference blocks 81 to 84 and 86 to 89 around the target block 85, as follows. For example, the image data corresponding to the blackout pixel 85b is replaced with the image data of pixel 86a of the adjacent reference block 86, and the image data corresponding to the blackout pixel 85d is replaced with the image data of pixel 88b of the adjacent reference block 88.

[0086] (ii) The correction unit 33b replaces the multiple image data in the target block where whiteout or blackout has occurred with different data, using image data acquired from multiple reference blocks selected from the reference blocks with the most frequently set imaging condition (fourth imaging condition in this example) for the same target element (mountain) as the whiteout or blackout target element (e.g. mountain) among the reference blocks located around the target block. For example, the correction unit 33b replaces the multiple image data in the target block where whiteout or blackout has occurred with different data, using image data acquired from multiple reference blocks selected from the reference blocks with the most frequently set imaging condition (fourth imaging condition in this example) for the same target element (mountain) among the reference blocks located around the target block 85, as follows, based on image data corresponding to the pixels 86a to 86d and 88a to 88d included in the reference blocks 86 and 88. For example, the correction unit 33b replaces the image data corresponding to the blackout pixel 85b with image data of the pixel 86a in the reference block 86, and replaces the image data corresponding to the blackout pixel 85d with image data of the pixel 88b in the reference block 88.

[0087] (iii) The correction unit 33b may use image data generated based on image data corresponding to a plurality of pixels acquired from a plurality of reference blocks selected by (i) or (ii) above for replacement. When the correction unit 33b selects the reference blocks 86 and 88, for example, the correction unit 33b replaces the image data corresponding to the crushed black pixel 85b and the crushed black pixel 85d in the target block as follows. That is, the image data corresponding to the crushed black pixel 85b is replaced by the average value of the image data corresponding to the pixels 86a to 86d included in the reference block 86. Also, the image data corresponding to the crushed black pixel 85d is replaced by the average value of the image data corresponding to the pixels 88a to 88d included in the reference block 88.

[0088] When calculating the average value of image data, a weighted average value weighted according to the distance from a pixel where whiteout or blackout has occurred may be used instead of a simple average. For example, since the image data corresponding to pixel 86a is closer to the image data corresponding to blackout pixel 85b than the image data corresponding to pixel 86b, the image data corresponding to pixel 86a is weighted so that the contribution rate of the image data corresponding to pixel 86a is higher than that of the image data corresponding to pixel 86b.

[0089] In addition, instead of calculating the average value of the image data corresponding to pixels 86a-86d and pixels 88a-88d contained in reference blocks 86 and 88, the intermediate value of the image data corresponding to pixels 86a-86d and pixels 88a-88d may be calculated, and the image data corresponding to blacked-out pixels 85b and pixel 85d may be replaced with this intermediate value.

[0090] In the above description, various aspects of the first correction process have been described. The control unit 34 may determine which aspect of the first correction process to perform based on, for example, the setting state (including the operation menu setting) of the operation member 36. Note that the control unit 34 may determine which mode of the first correction process to perform depending on the imaging scene mode set in the camera 1 and the type of the detected subject element.

[0091] <Second correction process> The correction unit 33b of the image processing unit 33 further performs the following second correction process as necessary before the image processing, the focus detection process, the subject detection (detection of subject elements) process, and the process of setting the imaging conditions. Note that the correction unit 33b performs the second correction process after replacing the blown out or crushed black pixels as described above. Note that the image data at the position of the crushed black pixel 85b (or 85d) replaced by another pixel can be subjected to the second correction process described below as if it was captured under the imaging conditions under which the replaced pixel (e.g. 86a) was captured. Furthermore, when crushed black pixel 85b is replaced using pixels from multiple blocks with different imaging conditions, it may be treated as a shooting condition with a value (average value, median value) between the imaging conditions of the blocks. For example, when crushed black pixel 85d is corrected using pixel 86c captured at ISO sensitivity 100 and pixel 88b captured at ISO sensitivity 1600, it may be treated as data captured at ISO sensitivity 800, which is between ISO sensitivity 100 and ISO sensitivity 1600. 1. When image processing is performed When the image processing for the image data acquired by applying different imaging conditions between the divided regions is a predetermined image processing, the correction unit 33b of the image processing unit 33 performs a second correction processing on the image data located at the boundary between the regions as a pre-processing of the image processing. The predetermined image processing is a processing for calculating the image data of a target position to be processed in the image by referring to the image data of a plurality of reference positions around the target position, and corresponds to, for example, a pixel defect correction processing, a color interpolation processing, a contour enhancement processing, a noise reduction processing, etc.

[0092] The second correction process is performed to reduce discontinuity occurring in the image after image processing due to differences in imaging conditions between the divided regions. In general, when the attention position is located at the boundary between the divided regions, image data to which the same imaging conditions as the image data of the attention position are applied and image data to which imaging conditions different from those of the image data of the attention position are applied may be mixed at a plurality of reference positions around the attention position. In this embodiment, the second correction process is performed as follows based on the idea that it is preferable to calculate the image data of the attention position by referring to the image data of the reference position to which the second correction process has been applied so as to suppress differences between the image data due to differences in imaging conditions, rather than calculating the image data of the attention position by directly referring to the image data of the reference position to which different imaging conditions have been applied.

[0093] FIG. 9(a) is an enlarged view of a region of interest 90 at the boundary between the first region 61 and the fourth region 64 in the live view image 60a in FIG. 7(a). Image data from pixels on the image sensor 32a corresponding to the first region 61 for which the first imaging condition is set is shown in white, and image data from pixels on the image sensor 32a corresponding to the fourth region 64 for which the fourth imaging condition is set is shown in shaded. In FIG. 9(a), image data from a pixel of interest P is located in the first region 61, in the vicinity of the boundary 91 between the first region 61 and the fourth region 64, that is, in the boundary portion. Pixels (eight pixels in this example) surrounding the pixel of interest P included in the region of interest 90 (e.g., 3×3 pixels) centered on the pixel of interest P are set as reference pixels Pr. FIG. 9(b) is an enlarged view of the pixel of interest P and reference pixels Pr1 to Pr8. The position of the pixel of interest P is the position of interest, and the positions of the reference pixels Pr1 to Pr8 surrounding the pixel of interest P are the reference positions. A first imaging condition is set for the reference pixels Pr1 to Pr6 and the target pixel P corresponding to the first region 61, and a fourth imaging condition is set for the reference pixels Pr7 and Pr8 corresponding to the fourth region 64. In the following description, the reference pixels Pr1 to Pr8 will be collectively referred to as Pr.

[0094] The generation unit 33c of the image processing unit 33 normally performs image processing by directly referring to the image data of the reference pixel Pr without performing the second correction processing. However, when the imaging condition (first imaging condition) applied to the pixel of interest P differs from the imaging condition (fourth imaging condition) applied to the reference pixel Pr surrounding the pixel of interest P, the correction unit 33b performs the second correction processing on the image data of the reference pixel Pr under the fourth imaging condition as shown in the following (Example 1) to (Example 3). Then, the generation unit 33c performs image processing to calculate image data of the pixel of interest P by referring to the image data of the reference pixel Pr after the second correction processing.

[0095] (Example 1) For example, when only the ISO sensitivity differs between the first and fourth imaging conditions, the ISO sensitivity of the first imaging condition is 100, and the ISO sensitivity of the fourth imaging condition is 800, the correction unit 33b of the image processing unit 33 applies 100 / 800 as a second correction process to the image data of the reference pixels Pr7 and Pr8 under the fourth imaging condition among the image data of the reference pixel Pr. This reduces the difference between the image data due to the difference in imaging conditions. Note that, when the amount of light incident on the pixel of interest P is the same as the amount of light incident on the reference pixel Pr, the difference in image data becomes small, but when the amount of light incident on the pixel of interest P is originally different from the amount of light incident on the reference pixel Pr, the difference in image data may not become small. The same applies to the examples described later.

[0096] (Example 2) For example, when only the shutter speed differs between the first and fourth imaging conditions, and the shutter speed of the first imaging condition is 1 / 1000 seconds and the shutter speed of the fourth imaging condition is 1 / 100 seconds, the correction unit 33b of the image processing unit 33 multiplies the image data of the reference pixels Pr7 and Pr8 under the fourth imaging condition among the image data of the reference pixel Pr by (1 / 1000) / (1 / 100)=1 / 10 as a second correction process, thereby reducing the difference between the image data due to the difference in imaging conditions.

[0097] (Example 3) For example, when only the frame rate differs between the first and fourth imaging conditions (the charge accumulation time is the same), the frame rate of the first imaging condition is 30 fps, and the frame rate of the fourth imaging condition is 60 fps, the correction unit 33b of the image processing unit 33 performs the second correction process to adopt image data of a frame image acquired under the first imaging condition (30 fps) and having an acquisition start timing close to that of the frame image acquired under the first imaging condition (30 fps) for the image data of the reference pixel Pr under the fourth imaging condition (60 fps). This reduces the difference between the image data due to the difference in the imaging conditions. The second correction process may involve interpolating and calculating image data of a frame image whose acquisition start timing is close to that of a frame image acquired under the first imaging condition (30 fps) based on multiple adjacent frame images acquired under the fourth imaging condition (60 fps).

[0098] On the other hand, the correction unit 33b of the image processing unit 33 does not perform the second correction process on the image data of the reference pixels Pr when the imaging condition (first imaging condition) applied to the pixel of interest P is the same as the imaging condition (fourth imaging condition) applied to all reference pixels Pr surrounding the pixel of interest P. In other words, the generation unit 33c performs image processing to calculate the image data of the pixel of interest P by directly referring to the image data of the reference pixels Pr. As described above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions.

[0099] <Image processing example> Image processing involving the second correction processing will be illustrated. (1) Pixel defect correction process In this embodiment, pixel defect correction processing is one of the image processing performed during imaging. In general, the imaging element 32a, which is a solid-state imaging element, may have pixel defects during or after manufacturing, and output image data of an abnormal level. Therefore, the generation unit 33c of the image processing unit 33 corrects the image data output from the pixel where the pixel defect occurred, so that the image data at the pixel position where the pixel defect occurred is not noticeable.

[0100] An example of pixel defect correction processing will be described. The generation unit 33c of the image processing unit 33 designates, for example, a pixel at the position of a pixel defect recorded in a non-volatile memory (not shown in advance) in an image of one frame as a target pixel P (pixel to be processed), and sets pixels (eight pixels in this example) around the target pixel P included in a target area 90 (for example, 3×3 pixels) centered on the target pixel P as reference pixels Pr.

[0101] The generation unit 33c of the image processing unit 33 calculates the maximum value and the minimum value of the image data in the reference pixels Pr, and when the image data output from the target pixel P exceeds these maximum values or minimum values, performs Max, Min filter processing to replace the image data output from the target pixel P with the above maximum value or minimum value. Such processing is performed for all pixel defects whose position information is recorded in a non-volatile memory (not shown).

[0102] In the present embodiment, when a pixel to which a fourth imaging condition different from the first imaging condition applied to the target pixel P is included in the reference pixels Pr, the correction unit 33b of the image processing unit 33 performs second correction processing on the image data to which the fourth imaging condition is applied. Thereafter, the generation unit 33c of the image processing unit 33 performs the above-described Max, Min filter processing.

[0103] (2) Color interpolation processing In the present embodiment, color interpolation processing is one of the image processes performed during imaging. As illustrated in FIG. 3, in the imaging chip 111 of the image sensor 100, green pixels Gb, Gr, blue pixels B, and red pixels R are arranged in a Bayer array. Since the generation unit 33c of the image processing unit 33 lacks image data of color components different from the color components of the color filter F arranged at each pixel position, color interpolation processing is performed to generate image data of the lacking color components by referring to the image data at the surrounding pixel positions.

[0104] An example of color interpolation processing will be described. FIG. 10(a) is a diagram illustrating the arrangement of image data output from the image sensor 32a. Corresponding to each pixel position, it has a color component of either R, G, or B according to the rule of the Bayer array. <G color interpolation> First, a general G color interpolation will be described. The generating unit 33c of the image processing unit 33 that performs G color interpolation generates G color component image data at the attention position by referring to the image data of the four G color components at the reference position around the attention position, with the positions of the R color component and the B color component in order as the attention position. For example, when generating G color component image data at the attention position shown in the thick frame (second row, second column counting from the upper left position. Similarly, the following positions are counted from the upper left position) in FIG. 10(b), the generating unit 33c of the image processing unit 33 refers to the image data G1 to G4 of the four G color components located near the attention position (second row, second column). For example, the generating unit 33c of the image processing unit 33 sets (aG1+bG2+cG3+dG4) / 4 as the G color component image data at the attention position (second row, second column). Note that a to d are weighting coefficients set according to the distance between the reference position and the attention position and the image structure.

[0105] Next, the G color interpolation of this embodiment will be described. In Fig. 10(a) to Fig. 10(c), the first imaging condition is applied to the left and upper regions of the thick line, and the fourth imaging condition is applied to the right and lower regions of the thick line. Note that the first imaging condition and the fourth imaging condition are different in Fig. 10(a) to Fig. 10(c). Also, the image data G1 to G4 of the G color component in Fig. 10(b) are reference positions for image processing of the pixel at the target position (second row, second column). In Fig. 10(b), the first imaging condition is applied to the target position (second row, second column). Among the reference positions, the first imaging condition is applied to the image data G1 to G3. Also, among the reference positions, the fourth imaging condition is applied to the image data G4. Therefore, the correction unit 33b of the image processing unit 33 performs a second correction process on the image data G4. Thereafter, the generation unit 33c of the image processing unit 33 calculates the image data of the G color component at the target position (second row, second column).

[0106] The generation unit 33c of the image processing unit 33 generates image data of the G color component at the B color component position and the R color component position in Figure 10(a), respectively, to obtain image data of the G color component at each pixel position, as shown in Figure 10(c).

[0107] <R color interpolation> FIG. 11(a) is a diagram obtained by extracting the image data of the R color component from FIG. 10(a). The generation unit 33c of the image processing unit 33 calculates the image data of the color difference component Cr shown in FIG. 11(b) based on the image data of the G color component shown in FIG. 10(c) and the image data of the R color component shown in FIG. 11(a).

[0108] First, the interpolation of the general color difference component Cr will be described. When the generation unit 33c of the image processing unit 33 generates the image data of the color difference component Cr at the target position indicated by the thick frame (the second row and the second column) in FIG. 11(b), for example, it refers to the image data Cr1 to Cr4 of the four color difference components located near the target position (the second row and the second column). The generation unit 33c of the image processing unit 33 sets, for example, (eCr1 + fCr2 + gCr3 + hCr4) / 4 as the image data of the color difference component Cr at the target position (the second row and the second column). Here, e to h are weight coefficients provided according to the distance between the reference position and the target position and the image structure.

[0109] Similarly, when the generation unit 33c of the image processing unit 33 generates the image data of the color difference component Cr at the target position indicated by the thick frame (the second row and the third column) in FIG. 11(c), for example, it refers to the image data Cr2, Cr4 to Cr6 of the four color difference components located near the target position (the second row and the third column). The generation unit 33c of the image processing unit 33 sets, for example, (qCr2 + rCr4 + sCr5 + tCr6) / 4 as the image data of the color difference component Cr at the target position (the second row and the third column). Here, q to t are weight coefficients provided according to the distance between the reference position and the target position and the image structure. In this way, the image data of the color difference component Cr is generated for each pixel position.

[0110] Next, the interpolation of the color difference component Cr in this embodiment will be described. In Fig. 11(a) to Fig. 11(c), for example, the first imaging condition is applied to the region to the left and above the thick line, and the fourth imaging condition is applied to the region to the right and below the thick line. Note that the first imaging condition and the fourth imaging condition are different in Fig. 11(a) to Fig. 11(c). In Fig. 11(b), the position indicated by the thick frame (second row, second column) is the target position of the color difference component Cr. Also, the image data Cr1 to Cr4 of the color difference component in Fig. 11(b) are reference positions for image processing the pixel at the target position (second row, second column). In Fig. 11(b), the first imaging condition is applied to the target position (second row, second column). Of the reference positions, the first imaging condition is applied to the image data Cr1, Cr3, and Cr4. Also, of the reference positions, the fourth imaging condition is applied to the image data Cr2. Therefore, the correction unit 33b of the image processing unit 33 performs a second correction process on the image data Cr2. Thereafter, the generation unit 33c of the image processing unit 33 calculates image data of the color difference component Cr at the target position (second row, second column). In addition, in FIG. 11(c), the position indicated by the thick frame (second row, third column) is the target position of the color difference component Cr. In addition, the image data Cr2, Cr4, Cr5, and Cr6 of the color difference components in FIG. 11(c) are reference positions for image processing the pixel at the target position (second row, third column). In FIG. 11(c), the fourth imaging condition is applied to the target position (second row, third column). Among the reference positions, the first imaging condition is applied to the image data Cr4 and Cr5. Among the reference positions, the fourth imaging condition is applied to the image data Cr2 and Cr6. Therefore, the correction unit 33b of the image processing unit 33 performs the second correction process on the image data Cr4 and Cr5, respectively. Thereafter, the generation unit 33c of the image processing unit 33 calculates the image data of the color difference component Cr at the target position (second row, third column).

[0111] The generation unit 33c of the image processing unit 33 obtains image data of the color difference component Cr at each pixel position, and then adds the image data of the G color component shown in Figure 10 (c) corresponding to each pixel position, thereby obtaining image data of the R color component at each pixel position.

[0112] FIG. 12(a) is a diagram obtained by extracting the image data of the B color component from FIG. 10(a). The generation unit 33c of the image processing unit 33 calculates the image data of the color difference component Cb shown in FIG. 12(b) based on the image data of the G color component shown in FIG. 10(c) and the image data of the B color component shown in FIG. 12(a).

[0113] First, the interpolation of the general color difference component Cb will be described. When the generation unit 33c of the image processing unit 33 generates the image data of the color difference component Cb at the attention position indicated by the thick frame (the third row and the third column) in FIG. 12(b), for example, it refers to the image data Cb1 to Cb4 of the four color difference components located in the vicinity of the attention position (the third row and the third column). The generation unit 33c of the image processing unit 33 uses, for example, (uCb1 + vCb2 + wCb3 + xCb4) / 4 as the image data of the color difference component Cb at the attention position (the third row and the third column). Here, u to x are weight coefficients provided according to the distance between the reference position and the attention position and the image structure.

[0114] Similarly, when the generation unit 33c of the image processing unit 33 generates the image data of the color difference component Cb at the attention position indicated by the thick frame (the third row and the fourth column) in FIG. 12(c), for example, it refers to the image data Cb2, Cb4 to Cb6 of the four color difference components located in the vicinity of the attention position (the third row and the fourth column). The generation unit 33c of the image processing unit 33 uses, for example, (yCb2 + zCb4 + αCb5 + βCb6) / 4 as the image data of the color difference component Cb at the attention position (the third row and the fourth column). Here, y, z, α, and β are weight coefficients provided according to the distance between the reference position and the attention position and the image structure. In this way, the image data of the color difference component Cb is generated for each pixel position.

[0115] Next, the interpolation of the color difference component Cb in this embodiment will be described. In Fig. 12(a) to Fig. 12(c), for example, the first imaging condition is applied to the left and upper regions of the thick line, and the fourth imaging condition is applied to the right and lower regions of the thick line. Note that the first imaging condition and the fourth imaging condition are different in Fig. 12(a) to Fig. 12(c). In Fig. 12(b), the position indicated by the thick frame (third row, third column) is the target position of the color difference component Cb. Also, the image data Cb1 to Cb4 of the color difference components in Fig. 12(b) are reference positions for image processing the pixel at the target position (third row, third column). In Fig. 12(b), the fourth imaging condition is applied to the target position (third row, third column). Of the reference positions, the first imaging condition is applied to the image data Cb1 and Cb3. Also, of the reference positions, the fourth imaging condition is applied to the image data Cb2 and Cb4. Therefore, the correction unit 33b of the image processing unit 33 performs a second correction process on each of the data Cb1 and Cb3. After that, the generation unit 33c of the image processing unit 33 calculates image data of the color difference component Cb at the target position (third row, third column). In addition, in FIG. 12(c), the position indicated by the thick frame (third row, fourth column) is the target position of the chrominance component Cb. In addition, the image data Cb2, Cb4 to Cb6 of the chrominance components in FIG. 12(c) are reference positions for performing image processing on the pixel at the target position (third row, fourth column). In FIG. 12(c), the fourth imaging condition is applied to the target position (third row, fourth column). In addition, the fourth imaging condition is applied to the image data Cb2, Cb4 to Cb6 at all reference positions. Therefore, the generation unit 33c of the image processing unit 33 calculates the image data of the chrominance component Cb at the target position (third row, fourth column) by referring to the image data Cb2, Cb4 to Cb6 at the reference positions on which the second correction process has not been performed by the correction unit 33b of the image processing unit 33.

[0116] The generation unit 33c of the image processing unit 33 obtains image data of the color difference component Cb at each pixel position, and then adds the image data of the G color component shown in Figure 10 (c) corresponding to each pixel position, thereby obtaining image data of the B color component at each pixel position. In the above-mentioned "G color interpolation", for example, when generating image data of G color components at the attention position shown in the thick frame (second row, second column) in FIG. 10(b), the image data G1 to G4 of the four G color components located near the attention position are referred to, but the number of image data of G color components to be referred to may be changed depending on the image structure. For example, when the image near the attention position has similarity in the vertical direction (for example, a vertical stripe pattern), the interpolation process is performed using only the image data above and below the attention position (G1 and G2 in FIG. 10(b)). Also, for example, when the image near the attention position has similarity in the horizontal direction (for example, a horizontal stripe pattern), the interpolation process is performed using only the image data to the left and right of the attention position (G3 and G4 in FIG. 10(b)). In these cases, the image data G4 to be corrected by the correction unit 33b may or may not be used. In this way, even if the black crushed pixels 85b and 85d occur, the correction unit 33b performs the first correction process, the second correction process, and the interpolation process, so that the black crushed pixels can be corrected to generate an image.

[0117] (3) Edge enhancement processing An example of the edge enhancement process will be described. The generation unit 33c of the image processing unit 33 performs a known linear filter operation using a kernel of a predetermined size centered on a pixel of interest P (a pixel to be processed) in an image of one frame, for example. When the kernel size of a sharpening filter, which is an example of a linear filter, is N×N pixels, the position of the pixel of interest P is the focus position, and the (N 2 The positions of the −1) reference pixels Pr are reference positions. The kernel size may be N×M pixels.

[0118] The generation unit 33c of the image processing unit 33 performs a filter process to replace the image data at the target pixel P with the result of a linear filter calculation, for example, by shifting the target pixel from left to right on each horizontal line from the top horizontal line to the bottom horizontal line of the frame image.

[0119] In this embodiment, the correction unit 33b of the image processing unit 33 performs a second correction process on the image data to which the fourth imaging condition has been applied, when the reference pixels Pr include a pixel to which a fourth imaging condition that is different from the first imaging condition applied to the pixel of interest P has been applied. Thereafter, the generation unit 33c of the image processing unit 33 performs the above-mentioned linear filter process.

[0120] (4) Noise reduction processing An example of noise reduction processing will be described. For example, in one frame of image, the generation unit 33c of the image processing unit 33 performs a known linear filter calculation using a kernel of a predetermined size centered on a pixel of interest P (a pixel to be processed). When the kernel size of a smoothing filter, which is an example of a linear filter, is N×N pixels, the position of the pixel of interest P is the position of interest, and the pixels (N 2 The positions of the −1) reference pixels Pr are reference positions. The kernel size may be N×M pixels.

[0121] The generation unit 33c of the image processing unit 33 performs a filter process to replace the image data at the target pixel P with the result of a linear filter calculation, for example, by shifting the target pixel from left to right on each horizontal line from the top horizontal line to the bottom horizontal line of the frame image.

[0122] In this embodiment, the correction unit 33b of the image processing unit 33 performs a second correction process on the image data to which the fourth imaging condition has been applied, when the reference pixels Pr include a pixel to which a fourth imaging condition that is different from the first imaging condition applied to the pixel of interest P has been applied. Thereafter, the generation unit 33c of the image processing unit 33 performs the above-mentioned linear filter process.

[0123] 2. When performing focus detection processing In the above example, as the first correction process, pixels where whiteout or blackout has occurred are replaced with image data of pixels in other blocks, but if the purpose is only focus adjustment, the signal from the focus detection pixel where whiteout or blackout has occurred can be replaced with a signal from another focus detection pixel. The method of replacing with another focus detection signal is similar to the method of replacing image data of a pixel where whiteout or blackout has occurred, so details will be omitted. In the case of focus adjustment based on image contrast, the image data replaced by the first correction process can be used. The lens movement control unit 34d of the control unit 34 performs focus detection processing using signal data (image data) corresponding to a predetermined position (focus point) on the imaging screen. When different imaging conditions are set between the divided areas and the focus point for AF operation is located at the boundary portion of the divided areas, the lens movement control unit 34d of the control unit 34 performs a second correction processing as pre-processing of the focus detection processing on the signal data for focus detection of at least one area.

[0124] The second correction process is performed to suppress a decrease in the accuracy of the focus detection process due to differences in imaging conditions between the regions of the imaging screen divided by the setting unit 34b. For example, when signal data for focus detection of a focus point that detects an image shift amount (phase difference) in an image is located at the boundary of the divided regions, signal data to which different imaging conditions are applied may be mixed in the signal data for focus detection. In this embodiment, the second correction process is performed as follows based on the idea that it is preferable to detect the image shift amount (phase difference) using signal data that has been subjected to the second correction process so as to suppress differences between signal data due to differences in imaging conditions, rather than detecting the image shift amount (phase difference) using signal data to which different imaging conditions are applied as is.

[0125] <Example of focus detection processing> A focus detection process accompanied by a second correction process will be illustrated. In the AF operation of this embodiment, for example, the focus is adjusted to a subject corresponding to a focus point selected by the user from among a plurality of focus points on the imaging screen. The lens movement control unit 34d (generator) of the control unit 34 calculates a defocus amount of the imaging optical system 31 by detecting an image shift amount (phase difference) of a plurality of subject images formed by light beams passing through different pupil regions of the imaging optical system 31. The lens movement control unit 34d of the control unit 34 moves a focus lens of the imaging optical system 31 to a position where the defocus amount is zero (below a permissible value), that is, to a focusing position, and adjusts the focus of the imaging optical system 31.

[0126] Fig. 13 is a diagram illustrating the positions of focus detection pixels on the imaging surface of the image sensor 32a. In this embodiment, focus detection pixels are arranged discretely along the X-axis direction (horizontal direction) of the imaging chip 111. In the example of Fig. 13, 15 focus detection pixel lines 160 are provided at predetermined intervals. The focus detection pixels constituting the focus detection pixel lines 160 output photoelectric conversion signals for focus detection. In the imaging chip 111, normal imaging pixels are provided at pixel positions other than the focus detection pixel lines 160. The imaging pixels output photoelectric conversion signals for live view images and recording.

[0127] Fig. 14 is an enlarged view of a partial area of ​​the focus detection pixel line 160 corresponding to the focus point 80A shown in Fig. 13. In Fig. 14, red pixels R, green pixels G (Gb, Gr), and blue pixels B, focus detection pixels S1, and focus detection pixels S2 are illustrated. The red pixels R, green pixels G (Gb, Gr), and blue pixels B are arranged according to the Bayer array rules described above.

[0128] The square regions illustrated for the red pixel R, green pixel G (Gb, Gr), and blue pixel B indicate the light receiving regions of the imaging pixels. Each imaging pixel receives a light beam that passes through the exit pupil of the imaging optical system 31 (FIG. 1). That is, the red pixel R, green pixel G (Gb, Gr), and blue pixel B each have a square mask opening, and light that passes through these mask openings reaches the light receiving section of the imaging pixel.

[0129] The shape of the light receiving regions (mask openings) of the red pixel R, the green pixel G (Gb, Gr), and the blue pixel B is not limited to a rectangle, and may be, for example, a circle.

[0130] The semicircular regions illustrated for the focus detection pixels S1 and S2 indicate the light receiving regions of the focus detection pixels. That is, the focus detection pixel S1 has a semicircular mask opening on the left side of the pixel position in FIG. 14, and light passing through this mask opening reaches the light receiving portion of the focus detection pixel S1. On the other hand, the focus detection pixel S2 has a semicircular mask opening on the right side of the pixel position in FIG. 14, and light passing through this mask opening reaches the light receiving portion of the focus detection pixel S2. In this way, the focus detection pixels S1 and S2 each receive a pair of light beams that pass through different regions of the exit pupil of the imaging optical system 31 (FIG. 1).

[0131] The positions of the focus detection pixel lines 160 in the imaging chip 111 are not limited to the positions exemplified in Fig. 13. Furthermore, the number of focus detection pixel lines 160 is not limited to the example in Fig. 13. Furthermore, the shape of the mask openings in the focus detection pixels S1 and S2 is not limited to a semicircular shape, and may be, for example, a rectangular shape obtained by horizontally dividing the square-shaped light receiving regions (mask openings) in the imaging pixels R, G, and B.

[0132] Furthermore, the focus detection pixel line 160 in the imaging chip 111 may be formed by arranging focus detection pixels along the Y-axis direction (vertical direction) of the imaging chip 111. An imaging element in which imaging pixels and focus detection pixels are two-dimensionally arranged as in Fig. 14 is well known, and detailed illustration and description of these pixels will be omitted.

[0133] 14, a so-called 1PD structure has been described in which each of the focus detection pixels S1 and S2 receives one of a pair of light beams for focus detection. Alternatively, a so-called 2PD structure may be used in which each of the focus detection pixels receives both of a pair of light beams for focus detection. The 2PD structure makes it possible to use the photoelectric conversion signal obtained by the focus detection pixel as a photoelectric conversion signal for recording.

[0134] The lens movement control unit 34d of the control unit 34 detects the image shift amount (phase difference) of a pair of images formed by a pair of light beams passing through different regions of the imaging optical system 31 (FIG. 1) based on the photoelectric conversion signals for focus detection output from the focus detection pixel S1 and the focus detection pixel S2. Then, the lens movement control unit 34d calculates the defocus amount based on the image shift amount (phase difference). Such defocus amount calculation using the pupil division phase difference method is well known in the field of cameras, so a detailed description will be omitted.

[0135] It is assumed that focus point 80A (FIG. 13) is selected by the user at a position corresponding to attention area 90 at the boundary between first area 61 and fourth area 64 in live view image 60a illustrated in FIG. 7(a). FIG. 15 is an enlarged view of focus point 80A. White pixels indicate that the first imaging condition is set, and shaded pixels indicate that the fourth imaging condition is set. The position surrounded by a frame 170 in FIG. 15 corresponds to focus detection pixel line 160 (FIG. 13).

[0136] The lens movement control unit 34d of the control unit 34 normally performs focus detection processing using signal data from the focus detection pixels indicated by the frame 170 as is, without performing the second correction processing. However, when the signal data surrounded by the frame 170 contains a mixture of signal data to which the first imaging condition has been applied and signal data to which the fourth imaging condition has been applied, the lens movement control unit 34d of the control unit 34 performs the second correction processing as in the following (Example 1) to (Example 3) on the signal data under the fourth imaging condition among the signal data surrounded by the frame 170. Then, the lens movement control unit 34d of the control unit 34 performs focus detection processing using the signal data after the second correction processing.

[0137] (Example 1) For example, when only the ISO sensitivity differs between the first and fourth imaging conditions, and the ISO sensitivity of the first imaging condition is 100 and that of the fourth imaging condition is 800, the lens movement control unit 34d of the control unit 34 applies 100 / 800 to the signal data of the fourth imaging condition as the second correction process, thereby reducing the difference between the signal data due to the difference in imaging conditions. In addition, when the amount of light incident on a pixel to which the first imaging condition is applied is the same as the amount of light incident on a pixel to which the fourth imaging condition is applied, the difference in signal data becomes small, but in cases where the amount of light incident on a pixel to which the first imaging condition is applied is different from the amount of light incident on a pixel to which the fourth imaging condition is applied, the difference in signal data may not become small.

[0138] (Example 2) For example, when only the shutter speed differs between the first and fourth imaging conditions and the shutter speed of the first imaging condition is 1 / 1000 seconds and the shutter speed of the fourth imaging condition is 1 / 100 seconds, the lens movement control unit 34d of the control unit 34 applies 1 / 1000 / 1 / 100=1 / 10 to the signal data of the fourth imaging condition as the second correction process, thereby reducing the difference between the signal data due to the difference in imaging conditions.

[0139] (Example 3) For example, when only the frame rate differs between the first and fourth imaging conditions (the charge accumulation time is the same), the frame rate of the first imaging condition is 30 fps, and the frame rate of the fourth imaging condition is 60 fps, the lens movement control unit 34d of the control unit 34 sets the second correction process to adopt, for the signal data of the fourth imaging condition (60 fps), the signal data of the frame image whose acquisition start timing is close to that of the frame image acquired under the first imaging condition (30 fps). This reduces the difference between the signal data due to the difference in imaging conditions. The second correction process may involve interpolating signal data of a frame image whose acquisition start timing is close to that of a frame image acquired under the first imaging condition (30 fps) based on multiple adjacent frame images acquired under the fourth imaging condition (60 fps).

[0140] On the other hand, the lens movement control unit 34d of the control unit 34 does not perform the second correction process when the imaging conditions applied to the signal data surrounded by the frame 170 are the same. In other words, the lens movement control unit 34d of the control unit 34 performs focus detection processing using the signal data from the focus detection pixels shown in the frame 170 as is.

[0141] As described above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions. In the above example, an example was described in which the second correction processing is performed on signal data of the fourth imaging condition among the signal data using the first imaging condition, but the second correction processing may also be performed on signal data of the first imaging condition among the signal data using the fourth imaging condition. The lens movement control unit 34d of the control unit 34 may determine whether to perform the second correction process on the signal data under the first imaging condition or the signal data under the fourth imaging condition based on, for example, the ISO sensitivity. When the ISO sensitivity is different between the first imaging condition and the fourth imaging condition, it is desirable to perform the second correction process on the signal data obtained under the imaging condition with the lower ISO sensitivity if the signal data obtained under the imaging condition with the higher ISO sensitivity is not saturated. In other words, when the ISO sensitivity is different between the first imaging condition and the fourth imaging condition, it is desirable to perform the second correction process on the darker signal data so as to reduce the difference with the brighter signal data.

[0142] Furthermore, a second correction process may be performed on each of the signal data under the first imaging condition and the signal data under the fourth imaging condition, thereby reducing the difference between the two pieces of signal data after the second correction process.

[0143] In the above explanation, an example of focus detection processing using the pupil division phase difference method has been given, but the same can also be performed using a contrast detection method in which the focus lens of the imaging optical system 31 is moved to the in-focus position based on the level of contrast of the subject image.

[0144] When the contrast detection method is used, the control unit 34 performs a known focus evaluation value calculation based on signal data output from the imaging pixels of the image sensor 32a corresponding to the focus point at each position of the focus lens while moving the focus lens of the imaging optical system 31. Then, the position of the focus lens that maximizes the focus evaluation value is obtained as the in-focus position.

[0145] The control unit 34 usually performs focus evaluation value calculation using the signal data output from the imaging pixel corresponding to the focus point without performing the second correction process. However, when the signal data corresponding to the focus point includes a mixture of signal data to which the first imaging condition is applied and signal data to which the fourth imaging condition is applied, the control unit 34 performs the second correction process as described above on the signal data of the fourth imaging condition among the signal data corresponding to the focus point. Then, the control unit 34 performs focus evaluation value calculation using the signal data after the second correction process. In this way, the correction unit 33b performs the first correction process, the second correction process, and the interpolation process, so that even if the black crushed pixels 85b and 85d occur, the black crushed pixels can be corrected and focus adjustment can be performed. Therefore, even if the black crushed pixels 85b and 85d exist, the focus can be adjusted by moving the lens. In the above example, the focus adjustment process is performed after the second correction process, but the focus adjustment may be performed using the image data obtained by the first correction process without performing the second correction process.

[0146] 3. When performing subject detection processing Fig. 16(a) is a diagram illustrating a template image representing an object to be detected, and Fig. 16(b) is a diagram illustrating a live view image 60a and a search range 190. The object detection unit 34a of the control unit 34 detects an object (for example, a bag 63a, which is one of the subject elements in Fig. 5) from the live view image. The object detection unit 34a of the control unit 34 may set the range for detecting the object to the entire range of the live view image 60a, but may also set a part of the live view image 60a as the search range 190 to reduce the load of the detection process.

[0147] When different imaging conditions are set between the divided areas and the search range 190 includes the boundary between the divided areas, the object detection unit 34a of the control unit 34 performs a second correction process on the image data of at least one area within the search range 190 as preprocessing of the subject detection process.

[0148] The second correction process is performed to suppress a decrease in accuracy of the detection process of the subject element due to differences in imaging conditions between the regions of the imaging screen divided by the setting unit 34b. In general, when the search range 190 used to detect the subject element includes the boundaries of the divided regions, image data to which different imaging conditions are applied may be mixed in the image data of the search range 190. In this embodiment, the second correction process is performed as follows based on the idea that it is preferable to detect the subject element using image data that has been subjected to the second correction process so as to suppress differences between image data due to differences in imaging conditions, rather than detecting the subject element using image data to which different imaging conditions are applied as is.

[0149] A case will be described where a bag 63a belonging to a person 61a is detected in the image of the subject shown in Fig. 5. The object detection unit 34a of the control unit 34 sets a search range 190 in the vicinity of the area including the person 61a. Note that the area 61 including the person 61a may be set as the search range.

[0150] When the search range 190 is not divided by two regions with different imaging conditions, the object detection unit 34a of the control unit 34 performs the subject detection process using the image data constituting the search range 190 as is without performing the second correction process. However, if the image data of the search range 190 includes a mixture of image data to which the first imaging condition is applied and image data to which the fourth imaging condition is applied, the object detection unit 34a of the control unit 34 performs the second correction process as in (Example 1) to (Example 3) described above as the case of performing the focus detection process on the image data of the fourth imaging condition among the image data of the search range 190. Then, the object detection unit 34a of the control unit 34 performs the subject detection process using the image data after the second correction process. As described above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions. In addition, in the above example, an example was described in which the second correction processing is performed on image data of the fourth imaging condition among the image data using the first imaging condition, but the second correction processing may also be performed on image data of the first imaging condition among the image data using the fourth imaging condition.

[0151] The second correction process for the image data in the search range 190 described above may be applied to a search range used for detecting a specific subject such as a human face, or to an area used for determining an imaging scene.

[0152] Furthermore, the second correction process for the image data in the search range 190 described above may be applied not only to a search range used in a pattern matching method using a template image, but also to a search range when detecting features based on the color, edges, etc. of an image.

[0153] Also, a known template matching process may be performed using image data of a plurality of frames acquired at different times, to apply to a moving object tracking process in which an area similar to a tracking target in a previously acquired frame image is searched for in a later acquired frame image. In this case, when image data to which the first imaging condition is applied and image data to which the fourth imaging condition is applied are mixed in a search range set in a later acquired frame image, the control unit 34 performs a second correction process as described above in (Example 1) to (Example 3) on image data of the fourth imaging condition among the image data in the search range. Then, the control unit 34 performs a tracking process using the image data after the second correction process.

[0154] The same is true when detecting a known motion vector using image data of a plurality of frames acquired at different times. When image data to which the first imaging condition is applied and image data to which the fourth imaging condition is applied are mixed in the detection area used for detecting the motion vector, the control unit 34 performs the second correction process as described above in (Example 1) to (Example 3) on the image data of the fourth imaging condition among the image data of the detection area used for detecting the motion vector. Then, the control unit 34 detects the motion vector using the image data after the second correction process. In this way, even if the black crushed pixels 85b and 85d occur, the correction unit 33b performs the first correction process, the second correction process, and the interpolation process, so that the above-mentioned subject detection and the like can be performed by correcting the black crush. Therefore, even if the black crushed pixels 85b and 85d exist, the subject detection can be performed. In the above example, the subject detection process is performed after the second correction process, but the second correction process may not be performed and subject detection may be performed using image data obtained by the first correction process.

[0155] 4. Setting imaging conditions When the setting unit 34b of the control unit 34 divides an area of ​​the imaging screen, sets different imaging conditions between the divided areas, and then performs new photometry to determine exposure conditions, the setting unit 34b performs a second correction process on the image data of at least one area as a preprocessing step for setting the exposure conditions.

[0156] The second correction process is performed to suppress a decrease in accuracy of the process for determining the exposure conditions due to differences in the imaging conditions between the regions of the imaging screen divided by the setting unit 34b. For example, when the photometry range set in the center of the imaging screen includes the boundaries of the divided regions, image data to which different imaging conditions are applied may be mixed in the image data of the photometry range. In this embodiment, the second correction process is performed as follows based on the idea that it is preferable to perform exposure calculation processing using image data that has been subjected to the second correction process so as to suppress differences between image data due to differences in imaging conditions, rather than performing exposure calculation processing using image data to which different imaging conditions are applied as is.

[0157] If the photometric range is not divided by a plurality of regions with different imaging conditions, the setting unit 34b of the control unit 34 performs exposure calculation processing using the image data constituting the photometric range as is without performing the second correction processing. However, if the image data of the photometric range includes a mixture of image data to which the first imaging condition is applied and image data to which the fourth imaging condition is applied, the setting unit 34b of the control unit 34 performs the second correction processing as in (Example 1) to (Example 3) described above as the case of performing focus detection processing or subject detection processing on the image data of the fourth imaging condition among the image data of the photometric range. Then, the setting unit 34b of the control unit 34 performs exposure calculation processing using the image data after the second correction processing. As described above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions. In addition, in the above example, an example was described in which the second correction processing is performed on image data of the fourth imaging condition among the image data using the first imaging condition, but the second correction processing may also be performed on image data of the first imaging condition among the image data using the fourth imaging condition.

[0158] This is not limited to the metering range used when performing the above-mentioned exposure calculation processing, but also applies to the metering (colorimetry) range used when determining a white balance adjustment value, the metering range used when determining whether or not to emit fill light from a light source that emits fill light, and further to the metering range used when determining the amount of fill light emitted by the light source.

[0159] In addition, when the readout resolution of the photoelectric conversion signal is made different between the divided regions of the imaging screen, the same can be applied to the regions used for determining the imaging scene when determining the readout resolution for each region. In this way, even if the black crushed pixels 85b and 85d occur, the correction unit 33b performs the first correction process, the second correction process, and the interpolation process, so that the black crush can be corrected and the imaging conditions can be set. Therefore, even if the black crushed pixels 85b and 85d exist, the imaging conditions can be set. In the above example, the shooting conditions are set after the second correction process is performed, but the shooting conditions may be set based on the image data obtained by the first correction process without performing the second correction process.

[0160] <Explanation of the flow chart> Fig. 17 is a flowchart for explaining the flow of processing for setting imaging conditions for each area and capturing images. When the main switch of the camera 1 is turned on, the control unit 34 starts a program for executing the processing shown in Fig. 17. In step S10, the control unit 34 causes the display unit 35 to start live view display, and the process proceeds to step S20.

[0161] Specifically, the control unit 34 instructs the imaging unit 32 to start acquiring live view images, and causes the acquired live view images to be sequentially displayed on the display unit 35. As described above, at this point in time, the same imaging conditions are set for the entire area of ​​the imaging chip 111, i.e., the entire screen. When a setting is made to perform an AF operation during live view display, the lens movement control unit 34d of the control unit 34 controls the AF operation to focus on a subject element corresponding to a predetermined focus point by performing a focus detection process. The lens movement control unit 34d performs the focus detection process after the first correction process and the second correction process, or after the first correction process or the second correction process, as necessary. Furthermore, if a setting to perform an AF operation during live view display is not made, the lens movement control section 34d of the control section 34 performs the AF operation when an AF operation is later instructed.

[0162] In step S20, the object detection unit 34a of the control unit 34 detects subject elements from the live view image and proceeds to step S30. The object detection unit 34a performs the first correction process and the second correction process, or the first correction process or the second correction process, and then performs the subject detection process, as necessary. In step S30, the setting unit 34b of the control unit 34 divides the screen of the live view image into areas including subject elements, and proceeds to step S40.

[0163] In step S40, the control unit 34 displays the regions on the display unit 35. As illustrated in Fig. 6, the control unit 34 highlights the region among the divided regions that is the target of setting (changing) the imaging conditions. The control unit 34 also displays a setting screen 70 for the imaging conditions on the display unit 35, and the process proceeds to step S50. When the user taps the display position of another main subject on the display screen with his / her finger, the control unit 34 changes the area including the main subject to an area for setting (changing) the imaging conditions and highlights it.

[0164] In step S50, the control unit 34 determines whether or not an AF operation is necessary. For example, when the focus adjustment state has changed due to the movement of the subject, when the position of the focus point has been changed by a user operation, or when an AF operation is instructed by a user operation, the control unit 34 makes a positive determination in step S50 and proceeds to step S70. When the focus adjustment state has not changed, the position of the focus point has not been changed by a user operation, and the AF operation is not instructed by a user operation, the control unit 34 makes a negative determination in step S50 and proceeds to step 60.

[0165] In step S70, the control unit 34 performs an AF operation and returns to step S40. The lens movement control unit 34d performs the first correction process and the second correction process, or the first correction process or the second correction process, followed by a focus detection process, which is an AF operation, as necessary. After returning to step S40, the control unit 34 repeats the same process as described above based on a live view image acquired after the AF operation.

[0166] In step S60, the setting unit 34b of the control unit 34 sets the imaging conditions for the highlighted area in response to a user operation, and the process proceeds to step S80. The display transition of the display unit 35 and the setting of the imaging conditions in response to a user operation in step S60 are as described above. The setting unit 34b of the control unit 34 performs the exposure calculation process after the first correction process and the second correction process, or after the first correction process or the second correction process, as necessary.

[0167] In step S80, the control unit 34 determines whether or not an image capture instruction has been issued. If a release button (not shown) constituting the operation member 36 or a display icon instructing image capture has been operated, the control unit 34 makes an affirmative decision in step S80 and proceeds to step S90. If an image capture instruction has not been issued, the control unit 34 makes a negative decision in step S80 and returns to step S60.

[0168] In step S90, the control unit 34 performs a predetermined imaging process. That is, the imaging control unit 34c controls the imaging element 32a to perform imaging under the imaging conditions set for each of the above-mentioned regions, and the process proceeds to step S100.

[0169] In step S100, the imaging control unit 34c of the control unit 34 sends an instruction to the image processing unit 33 to perform a predetermined image processing on the image data obtained by the imaging, and the process proceeds to step S110. The image processing includes the pixel defect correction processing, color interpolation processing, edge enhancement processing, and noise reduction processing. In addition, the correction unit 33b of the image processing unit 33 performs the first correction process and the second correction process, or the first correction process or the second correction process, on image data located at the boundary part of the region as necessary, and then performs image processing.

[0170] In step S110, the control unit 34 sends an instruction to the recording unit 37 to record the image data after image processing on a recording medium (not shown), and the process proceeds to step S120.

[0171] In step S120, the control unit 34 judges whether or not a termination operation has been performed. If a termination operation has been performed, the control unit 34 makes an affirmative decision in step S120 and ends the processing in Fig. 17. If a termination operation has not been performed, the control unit 34 makes a negative decision in step S120 and returns to step S20. If the processing returns to step S20, the control unit 34 repeats the above-mentioned processing.

[0172] In the above description, the stacked imaging element 100 has been exemplified as the imaging element 32a. However, as long as imaging conditions can be set for each of a plurality of blocks in the imaging element (imaging chip 111), it is not necessarily required to configure the imaging element as a stacked imaging element.

[0173] According to the first embodiment described above, the following advantageous effects can be obtained. (1) The camera 1 equipped with an image processing device includes an imaging section 32 having a first region 61 imaged under a first imaging condition and a fourth region 64 imaged under a fourth imaging condition different from the first imaging condition, and an image processing section 33 that generates an image of a subject imaged in the first region 61 using image data of the subject imaged in the fourth region 64. As a result, when inappropriate image data occurs in an image of a subject imaged in the first region 61 due to the imaging conditions set in the first region 61, appropriate image data can be generated using an image of a subject imaged in the fourth region 64 under imaging conditions different from the first region 61. Specifically, image data can be generated that suppresses discontinuity and discomfort in images such as differences in brightness, contrast, color, etc. between the shaded parts of blocks 82, 85, and 87 and the shaded parts of blocks 83, 86, 88, and 89 in FIG. 7(b).

[0174] (2) The camera 1 equipped with the image processing device includes an imaging section 32 having a first region 61 imaged under a first imaging condition and a fourth region 64 imaged under a fourth imaging condition different from the first imaging condition, and an image processing section 33 that generates image data of a subject imaged in the first region 61 using image data of the subject imaged in the fourth region 64. In this way, for example, when inappropriate image data occurs in an image of a subject imaged in the first region 61 due to the imaging conditions set in the first region 61, appropriate image data can be generated using an image of a subject imaged in the fourth region 64 under imaging conditions different from the first region 61. Specifically, image data can be generated that suppresses discontinuity and discomfort in images such as differences in brightness, contrast, color, etc. between the shaded parts of blocks 82, 85, and 87 and the shaded parts of blocks 83, 86, 88, and 89 in FIG. 7(b).

[0175] (3) The image processing unit 33 generates image data of the image of the subject captured in the first region 61 from image data of the subject captured in the fourth region 64. As a result, for example, when inappropriate image data occurs in the image of the subject captured in the first region 61 due to the imaging conditions set in the first region 61, appropriate image data can be generated using the image of the subject captured in the fourth region 64 under imaging conditions different from those of the first region 61.

[0176] (4) The image processing unit 33 generates image data of a part of the subject captured in the first region 61 from image data of the subject captured in the fourth region 64. As a result, when inappropriate image data occurs in a part (target block) of the image captured in the first region 61, appropriate image data can be generated using image data of an image (reference block) captured in the fourth region 64, which has different imaging conditions from the first region 61.

[0177] (5) Image processing unit 33 generates image data of a portion of the subject imaged in first region 61 from image data of the subject imaged in fourth region 64. The portion of the subject imaged in first region 61 and at least the portion of the subject imaged in fourth region 64 are detected as the same subject. As a result, when the same subject, for example a mountain in FIG. 7, is imaged in first region 61 and fourth region 64, image data of the mountain imaged in first region 61 can be replaced with image data of the mountain imaged in fourth region 64. Therefore, image data can be generated appropriately.

[0178] (6) The area of ​​the fourth region 64 is larger than the area of ​​the first region 61. As a result, image data of the subject captured in the first region 61 is generated based on image data of the subject captured in the fourth region 64, which is larger than the area of ​​the first region 61, so that the image data can be generated appropriately.

[0179] (7) The area of ​​the fourth region 64 is equal to or smaller than the area of ​​the first region 61. That is, when replacing image data acquired in the block of interest with image data acquired in the reference block, for example, by replacing a plurality of image data corresponding to a plurality of pixels included in the block of interest with the same number or a smaller number of image data, the computational load in the replacement process can be reduced.

[0180] (8) The image processing unit 33 generates image data of the subject captured in the first region 61 from image data of the subject captured in a portion of the fourth region 64. As a result, when inappropriate image data occurs in a portion (target block) of the image captured in the first region 61, appropriate image data can be generated using a portion (reference block) of the image captured in the fourth region 64, which has different imaging conditions from the first region 61.

[0181] (9) The fourth region 64 has a pixel 86a including a photoelectric conversion unit that converts light into an electric charge, and a pixel 86b different from the pixel 86a including a photoelectric conversion unit that converts light into an electric charge. The image processing unit 33 generates image data of the subject captured in the first region 61 using image data of the subject captured in either pixel 86a or pixel 86b. Thereby, for example, when inappropriate image data occurs in the image of the subject captured in the first region 61 due to the imaging conditions set in the first region 61, appropriate image data can be generated using the image of the subject captured in the fourth region 64 under imaging conditions different from those in the first region 61.

[0182] (10) When the distance between the first region 61 and pixel 86a is shorter than the distance between the first region 61 and pixel 86b, the image processing unit 33 generates image data of the object captured in the first region 61 from image data of the object captured in pixel 86a. That is, when replacing the image data acquired in the target block with image data acquired in the reference block, the image data is replaced with image data from a pixel that is closest to the pixel to be replaced in the target block among the pixels included in the reference block. This allows appropriate image data to be generated.

[0183] (11) The image processing unit 33 generates image data of the subject captured in the first area 61 using data calculated from image data of the subject captured in the fourth area 64. As a result, for example, when inappropriate image data occurs in the image of the subject captured in the first area 61 due to the imaging conditions set in the first area 61, appropriate image data can be generated using the image of the subject captured in the fourth area 64 under imaging conditions different from those of the first area 61.

[0184] (12) The image processing unit 33 generates image data of the subject captured in the first area 61 from data calculated by averaging the image data of the subject captured in the fourth area 64. As a result, for example, when inappropriate image data occurs in the image of the subject captured in the first area 61 due to the imaging conditions set in the first area 61, appropriate image data can be generated using the image of the subject captured in the fourth area 64 under imaging conditions different from those of the first area 61.

[0185] (13) The imaging element 100 includes an imaging chip 111 including a first region 61 and a fourth region 64, and a signal processing chip 112 connected to the imaging chip 111 and having an analog / digital (A / D) conversion circuit that converts image data output from the imaging chip 111 into digital data. This allows appropriate processing to be performed in each of the regions with different imaging conditions. In other words, an image can be appropriately generated based on image data generated in each region.

[0186] (14) The imaging element 100 includes a memory chip 113 connected to the signal processing chip 112 and having a storage unit that stores image data converted into digital data by the signal processing chip 112. The signal processing chip 112 is disposed between the imaging chip 111 and the memory chip 113. This allows the chips to be stacked in accordance with the flow of data in the imaging element 100, thereby enabling efficient electrical connection between the chips.

[0187] (15) The camera 1 equipped with the lens adjustment device includes an imaging section 32 having a first region 61 that images light from the imaging optical system 31 under a first imaging condition and a fourth region 64 that images light from the imaging optical system 31 under a fourth imaging condition different from the first imaging condition, and an image processing section 33 and a control section 34 that move the imaging optical system 31 so as to adjust the image of the light incident on the first region 61 by a signal based on the subject imaged in the fourth region 64. In this way, for example, when inappropriate signal data due to the imaging conditions occurs in the signal data of the image imaged in the first region 61, appropriate signal data can be generated using the signal data of the image imaged in the fourth region 64 under imaging conditions different from the first region 61. Specifically, signal data can be generated that suppresses discontinuities in images such as differences in brightness and contrast between the shaded portions of blocks 82, 85, and 87 and the shaded portions of blocks 83, 86, 88, and 89. As a result, it is possible to prevent a decrease in focus detection accuracy due to differences in imaging conditions for each block, making it possible to perform appropriate focusing.

[0188] (16) The camera 1 equipped with the lens adjustment device includes an imaging section 32 having a first region 61 that images the light of the imaging optical system 31 under a first imaging condition and a fourth region 64 that images the light from the imaging optical system 31 under a fourth imaging condition different from the first imaging condition, and an image processing section 33 and a control section 34 that generate a signal for adjusting the image of the light incident on the first region 61 in the imaging optical system 31 by a signal based on the subject imaged in the fourth region 64. In this way, for example, when inappropriate signal data due to the imaging conditions occurs in the signal data of the image imaged in the first region 61, appropriate signal data can be generated using the signal data of the image imaged in the fourth region 64 under imaging conditions different from the first region 61. Specifically, signal data can be generated that suppresses discontinuities in images such as differences in brightness and contrast between the shaded portions of blocks 82, 85, and 87 and the shaded portions of blocks 83, 86, 88, and 89. As a result, it is possible to prevent a decrease in focus detection accuracy due to differences in imaging conditions for each block, making it possible to perform appropriate focusing.

[0189] (17) The image processing unit 33 generates a signal based on the subject imaged in the first region 61 from a signal based on the subject imaged in the fourth region 64. As a result, when inappropriate signal data due to imaging conditions occurs in the signal data of the image imaged in the first region 61, appropriate signal data can be generated using the signal data of the image imaged in the fourth region 64 under imaging conditions different from those of the first region 61.

[0190] (18) The image processing unit 33 generates a signal based on a part of the object imaged in the first region 61 from a signal based on the object imaged in the fourth region 64. As a result, when inappropriate signal data occurs in a part (block of interest) of the image imaged in the first region 61, appropriate signal data can be generated using signal data of an image imaged in the fourth region 64 under imaging conditions different from those of the first region 61. As a result, it becomes possible to perform appropriate focusing on a part of the object by avoiding the influence of differences in imaging conditions for each block.

[0191] (19) The image processing unit 33 generates a signal based on a part of the subject imaged in the first region 61 from a signal based on the subject imaged in the fourth region 64. The part of the subject imaged in the first region 61 and at least a part of the subject imaged in the fourth region 64 are detected as the same subject. As a result, when the same subject, for example a mountain in FIG. 7, is imaged in the first region 61 and the fourth region 64, appropriate signal data can be generated using signal data of the image of the mountain imaged in the first region 61 and signal data of the image of the mountain imaged in the fourth region 64. This enables appropriate focusing.

[0192] (20) The area of ​​the fourth region 64 is larger than the area of ​​the first region 61. This makes it possible to generate appropriate signal data based on the signal data of an image captured in the fourth region 64, which is larger than the area of ​​the first region 61. This makes it possible to perform appropriate focusing.

[0193] (21) The area of ​​the fourth region 64 is equal to or smaller than the area of ​​the first region 61. That is, in generating appropriate signal data, for example, signal data of a plurality of images corresponding to a plurality of pixels included in a block of interest is replaced with signal data of the same number of images or a smaller number of images, thereby reducing the computational load in the replacement process.

[0194] (22) The image processing unit 33 generates a signal based on the object captured in the first region 61 from a signal based on the object captured in a part of the fourth region 64. As a result, when inappropriate signal data occurs in a part (target block) of the image captured in the first region 61, appropriate signal data can be generated using signal data of an image captured in a part (reference block) of the image captured in the fourth region 64 under imaging conditions different from those of the first region 61. As a result, appropriate focusing is possible without being affected by differences in imaging conditions for each block.

[0195] (23) The fourth region 64 has pixels 86a including a photoelectric conversion unit that converts light into electric charges, and pixels 86b different from the pixels 86a including a photoelectric conversion unit that converts light into electric charges. The image processing unit 33 generates a signal based on an object captured in the first region 61 using a signal based on an object captured in either the pixel 86a or the pixel 86b. Thereby, for example, when inappropriate signal data due to imaging conditions occurs in the signal data of an image captured in the first region 61, it is possible to generate appropriate signal data using signal data of an image captured in the fourth region 64 under imaging conditions different from those of the first region 61.

[0196] (24) When the distance between the first region 61 and pixel 86a is shorter than the distance between the first region 61 and pixel 86b, the image processing unit 33 generates a signal based on the object captured in the first region 61 from a signal based on the object captured at pixel 86a. That is, when replacing the signal data of the image acquired in the target block with the signal data of the image acquired in the reference block, the signal data of the image from a pixel that is closer to the pixel to be replaced in the target block among the pixels included in the reference block is used for replacement. This makes it possible to generate appropriate signal data.

[0197] (25) The image processing unit 33 generates a signal based on the subject imaged in the first region 61 using data calculated from signal data based on the subject imaged in the fourth region 64. As a result, for example, when inappropriate signal data due to imaging conditions occurs in the signal data of the image imaged in the first region 61, appropriate signal data can be generated using signal data of an image imaged in the fourth region 64 under imaging conditions different from those of the first region 61.

[0198] (26) The image processing unit 33 generates a signal based on the subject imaged in the first region 61 using data calculated by averaging signal data based on the subject imaged in the fourth region 64. As a result, for example, when inappropriate signal data due to imaging conditions occurs in the signal data of the image imaged in the first region 61, appropriate signal data can be generated using signal data of an image imaged in the fourth region 64 under imaging conditions different from those of the first region 61.

[0199] (27) The lens movement control unit 34d of the camera 1 generates a signal for driving the imaging optical system 31 based on the image shift amounts of multiple optical images that have passed through different pupils of the imaging optical system 31. This enables appropriate focusing without being affected by differences in imaging conditions for each block when performing focus detection processing using the split-pupil phase difference detection method.

[0200] (28) The lens movement control unit 34d of the camera 1 generates a signal for driving the imaging optical system 31 based on the contrast of an image formed by light incident on the imaging unit 32. This enables appropriate focusing without being affected by differences in imaging conditions for each block when performing focus detection processing using a contrast detection method.

[0201] (29) A camera 1 equipped with a subject detection device includes an imaging unit 32 having a first region 61 imaged under a first imaging condition and a fourth region 64 imaged under a fourth imaging condition different from the first imaging condition; The image processing unit 33 and the control unit 34 detect the subject captured in the first region 61 using image data of the subject captured in the fourth region 64. As a result, when inappropriate image data occurs in the image data of the image captured in the first region 61 due to the imaging conditions, for example, appropriate image data can be generated using image data of the image captured in the fourth region 64 under imaging conditions different from those of the first region 61. Specifically, image data can be generated that suppresses discontinuity of images such as differences in brightness, contrast, color, etc. between the shaded portions of the blocks 82, 85, and 87 and the shaded portions of the blocks 83, 86, 88, and 89. As a result, it is possible to suppress a decrease in the detection accuracy of the subject elements due to differences in the imaging conditions for each block.

[0202] (30) A camera 1 equipped with a subject detection device includes an imaging section 32 having a first region 61 imaged under a first imaging condition and a fourth region 64 imaged under a fourth imaging condition different from the first imaging condition; The image processing unit 33 and the control unit 34 generate a signal for detecting the object captured in the first area 61 from the image data of the object captured in the fourth area 64. As a result, when inappropriate image data occurs in the image data of the image captured in the first area 61 due to the imaging conditions, appropriate image data can be generated using the image data of the image captured in the fourth area 64 under imaging conditions different from those of the first area 61. Specifically, image data can be generated that suppresses discontinuity of images such as differences in brightness, contrast, color tone, etc. between the shaded areas of the blocks 82, 85, and 87 and the shaded areas of the blocks 83, 86, 88, and 89. As a result, it is possible to suppress a decrease in the detection accuracy of the object elements due to differences in the imaging conditions for each block.

[0203] (31) The image processing unit 33 generates image data of the subject captured in the first region 61 from image data of the subject captured in the fourth region 64. As a result, when inappropriate image data occurs in the image data of the image captured in the first region 61 due to the imaging conditions, for example, it is possible to generate appropriate image data using image data of the image captured in the fourth region 64 under imaging conditions different from those of the first region 61. As a result, it is possible to prevent a decrease in the detection accuracy of the subject elements due to differences in imaging conditions for each block.

[0204] (32) The image processing unit 33 generates image data of a part of the subject captured in the first region 61 from image data of the subject captured in the fourth region 64. As a result, when inappropriate image data occurs in a part (a block of interest) of the image captured in the first region 61, appropriate image data can be generated using image data of the image captured in the fourth region 64, which has imaging conditions different from those of the first region 61. As a result, it becomes possible to appropriately detect subject elements even for a part of the subject, avoiding the influence of differences in imaging conditions for each block.

[0205] (33) The image processing unit 33 generates image data of a part of the subject imaged in the first region 61 from image data of the subject imaged in the fourth region 64. The part of the subject imaged in the first region 61 and at least the part of the subject imaged in the fourth region 64 are detected as the same subject. As a result, when the same subject, for example a mountain in FIG. 7, is imaged in the first region 61 and the fourth region 64, the image data of the mountain imaged in the first region 61 can be replaced with the image data of the mountain imaged in the fourth region 64. Therefore, it is possible to generate image data appropriately. As a result, it becomes possible to appropriately detect subject elements without being affected by differences in imaging conditions for each block.

[0206] (34) The area of ​​the fourth region 64 is larger than the area of ​​the first region 61. As a result, image data of the subject captured in the first region 61 is generated based on image data of the subject captured in the fourth region 64, which is larger than the area of ​​the first region 61, so that the image data can be generated appropriately. As a result, it becomes possible to appropriately detect subject elements without being affected by differences in imaging conditions for each block.

[0207] (35) The area of ​​the fourth region 64 is equal to or smaller than the area of ​​the first region 61. That is, when replacing the image data acquired in the target block with the image data acquired in the reference block, for example, by replacing a plurality of image data corresponding to a plurality of pixels included in the target block with the same number or a smaller number of image data, the computational load in the replacement process can be reduced. Therefore, the computational load related to the detection of subject elements can be reduced.

[0208] (36) The image processing unit 33 generates image data of the subject captured in the first region 61 from image data of the subject captured in a portion of the fourth region 64. As a result, when inappropriate image data occurs in a portion (a target block) of the image captured in the first region 61, appropriate image data can be generated using a portion (a reference block) of the image captured in the fourth region 64, which has imaging conditions different from those of the first region 61. As a result, it becomes possible to appropriately detect subject elements without being affected by differences in imaging conditions for each block.

[0209] (37) The fourth region 64 has a pixel 86a including a photoelectric conversion unit that converts light into an electric charge, and a pixel 86b different from the pixel 86a including a photoelectric conversion unit that converts light into an electric charge. The image processing unit 33 generates image data of a subject captured in the first region 61 using image data of the subject captured in either the pixel 86a or the pixel 86b. As a result, when inappropriate image data occurs in the image data of an image captured in the first region 61 due to the imaging conditions, for example, it is possible to generate appropriate image data using image data of an image captured in the fourth region 64 under imaging conditions different from those of the first region 61. As a result, it is possible to suppress a decrease in the detection accuracy of subject elements due to differences in imaging conditions for each block.

[0210] (38) When the distance between the first region 61 and pixel 86a is shorter than the distance between the first region 61 and pixel 86b, the image processing unit 33 generates image data of the object captured in the first region 61 from image data of the object captured in pixel 86a. That is, when replacing the image data acquired in the target block with image data acquired in the reference block, the image data is replaced with image data from a pixel that is closer to the pixel to be replaced in the target block among the pixels included in the reference block. This makes it possible to generate image data appropriately. As a result, it is possible to suppress a decrease in the detection accuracy of the object element due to differences in imaging conditions for each block.

[0211] (39) The image processing unit 33 generates image data of the subject captured in the first region 61 using data calculated from image data of the subject captured in the fourth region 64. As a result, when inappropriate image data occurs in the image data of the image captured in the first region 61 due to the imaging conditions, for example, it is possible to generate appropriate image data using image data of the image captured in the fourth region 64 under imaging conditions different from those of the first region 61. As a result, it is possible to prevent a decrease in the detection accuracy of the subject elements due to differences in imaging conditions for each block.

[0212] (40) The image processing unit 33 generates image data of the subject captured in the first region 61 using data calculated by averaging the image data of the subject captured in the fourth region 64. As a result, if inappropriate image data occurs in the image data of the image captured in the first region 61 due to the imaging conditions, for example, it is possible to generate appropriate image data using image data of the image captured in the fourth region 64 under imaging conditions different from those of the first region 61. As a result, it is possible to prevent a decrease in the detection accuracy of the subject elements due to differences in imaging conditions for each block.

[0213] (41) The object detection unit 34a of the camera 1 detects subject elements as targets for focus adjustment of the imaging optical system 31, and can therefore appropriately detect targets for focus adjustment of the imaging optical system 31 without being affected by differences in imaging conditions for each block.

[0214] (42) Camera 1 is equipped with control unit 34 that detects the brightness of the subject, and object detection unit 34a detects subject elements as targets for photometry by control unit 34, so that the target for photometry can be appropriately detected without being affected by differences in imaging conditions for each block.

[0215] (43) The camera 1 includes an imaging section 32 having a first region 61 imaged under a first imaging condition and a fourth region 64 imaged under a fourth imaging condition different from the first imaging condition, and an image processing section 33 and a control section 34 that set the imaging condition of the first region 61 by a signal based on a subject imaged in the fourth region 64. As a result, when inappropriate image data due to the imaging conditions occurs in the image data of an image imaged in the first region 61, for example, it is possible to generate appropriate image data using image data of an image imaged in the fourth region 64 under imaging conditions different from the first region 61. Specifically, it is possible to generate image data that suppresses discontinuity of images such as differences in brightness, contrast, color, etc. between the shaded parts of blocks 82, 85, and 87 and the shaded parts of blocks 83, 86, 88, and 89. As a result, it is possible to suppress a decrease in the setting accuracy of the exposure condition due to differences in the imaging conditions for each block.

[0216] (44) The camera 1 includes an imaging section 32 having a first region 61 imaged under a first imaging condition and a fourth region 64 imaged under a fourth imaging condition different from the first imaging condition, and an image processing section 33 and a control section 34 that generate a signal for setting the imaging condition of the first region 61 by a signal based on a subject imaged in the fourth region 64. As a result, when inappropriate image data due to the imaging conditions occurs in the image data of an image imaged in the first region 61, for example, it is possible to generate appropriate image data using image data of an image imaged in the fourth region 64 under imaging conditions different from the first region 61. Specifically, it is possible to generate image data that suppresses discontinuity of images such as differences in brightness, contrast, color, etc. between the shaded parts of blocks 82, 85, and 87 and the shaded parts of blocks 83, 86, 88, and 89. As a result, it is possible to suppress a decrease in the setting accuracy of the exposure condition due to differences in the imaging conditions for each block.

[0217] (45) The image processing unit 33 generates a signal based on the subject captured in the first region 61 from a signal based on the subject captured in the fourth region 64. As a result, when inappropriate image data occurs in the image data of the image captured in the first region 61 due to the imaging conditions, for example, it is possible to generate appropriate image data using image data of the image captured in the fourth region 64 under imaging conditions different from those of the first region 61. As a result, it is possible to prevent a decrease in the setting accuracy of the exposure conditions due to differences in the imaging conditions for each block.

[0218] (46) The image processing unit 33 generates a signal based on a part of the subject captured in the first region 61 from a signal based on the subject captured in the fourth region 64. As a result, when inappropriate image data occurs in a part (a block of interest) of the image captured in the first region 61, appropriate image data can be generated using image data of an image captured in the fourth region 64, which has imaging conditions different from those of the first region 61. As a result, it becomes possible to set appropriate exposure conditions without being affected by differences in imaging conditions for each block.

[0219] (47) The image processing unit 33 generates a signal based on a part of the subject imaged in the first region 61 from a signal based on the subject imaged in the fourth region 64. The part of the subject imaged in the first region 61 and at least a part of the subject imaged in the fourth region 64 are detected as the same subject. As a result, when the same subject, for example a mountain in FIG. 7, is imaged in the first region 61 and the fourth region 64, the image data of the mountain imaged in the first region 61 can be replaced with the image data of the mountain imaged in the fourth region 64. Therefore, the image data can be generated appropriately. As a result, it becomes possible to set the exposure conditions appropriately without being affected by differences in the imaging conditions for each block.

[0220] (48) The area of ​​the fourth region 64 is larger than the area of ​​the first region 61. As a result, the image data of the subject captured in the first region 61 is generated based on the image data of the subject captured in the fourth region 64, which is larger than the area of ​​the first region 61, so that the image data can be generated appropriately. As a result, it becomes possible to set the exposure conditions appropriately without being affected by differences in the imaging conditions for each block.

[0221] (49) The area of ​​the fourth region 64 is equal to or smaller than the area of ​​the first region 61. That is, when replacing image data acquired in the target block with image data acquired in the reference block, for example, by replacing a plurality of image data corresponding to a plurality of pixels included in the target block with the same number or a smaller number of image data, the computation load in the replacement process can be reduced. Therefore, the computation load related to setting the exposure conditions can be reduced.

[0222] (50) The image processing unit 33 generates a signal based on an object captured in the first region 61 from a signal based on an object captured in a portion of the fourth region 64. As a result, when inappropriate image data occurs in a portion (a block of interest) of the image captured in the first region 61, appropriate image data can be generated using a portion (a reference block) of the image captured in the fourth region 64, which has imaging conditions different from those of the first region 61. As a result, it becomes possible to set appropriate exposure conditions without being affected by differences in imaging conditions for each block.

[0223] (51) The fourth region 64 has a pixel 86a including a photoelectric conversion unit that converts light into an electric charge, and a pixel 86b different from the pixel 86a including a photoelectric conversion unit that converts light into an electric charge. The image processing unit 33 generates a signal based on an object captured in the first region 61 using a signal based on an object captured in either the pixel 86a or the pixel 86b. As a result, when inappropriate image data due to the imaging conditions occurs in the image data of an image captured in the first region 61, for example, it is possible to generate appropriate image data using image data of an image captured in the fourth region 64 under imaging conditions different from the first region 61. As a result, it is possible to suppress a decrease in the setting accuracy of the exposure conditions due to differences in the imaging conditions for each block.

[0224] (52) When the interval between the first region 61 and pixel 86a is shorter than the interval between the first region 61 and pixel 86b, the image processing unit 33 generates a signal based on the object captured in the first region 61 from a signal based on the object captured in pixel 86a. That is, when replacing the image data acquired in the target block with the image data acquired in the reference block, the image data is replaced with image data from a pixel that is closer to the pixel to be replaced in the target block among the pixels included in the reference block. This makes it possible to generate image data appropriately. As a result, it is possible to suppress a decrease in the setting accuracy of the exposure condition due to differences in the imaging conditions for each block.

[0225] (53) The image processing unit 33 generates a signal based on the subject captured in the first region 61 using data calculated from signal data based on the subject captured in the fourth region 64. As a result, when inappropriate image data due to imaging conditions occurs in the image data of the image captured in the first region 61, appropriate image data can be generated using image data of the image captured in the fourth region 64 under imaging conditions different from those of the first region 61. As a result, it is possible to prevent a decrease in the setting accuracy of the exposure conditions due to differences in imaging conditions for each block.

[0226] (54) The image processing unit 33 generates a signal based on the subject captured in the first region 61 using data calculated by averaging signal data based on the subject captured in the fourth region 64. As a result, when inappropriate image data due to imaging conditions occurs in the image data of the image captured in the first region 61, appropriate image data can be generated using image data of the image captured in the fourth region 64 under imaging conditions different from those of the first region 61. As a result, it is possible to prevent a decrease in the setting accuracy of the exposure conditions due to differences in imaging conditions for each block.

[0227] (55) The setting unit 34b of the camera 1 sets the exposure conditions as the shooting conditions. Therefore, even if the shooting conditions differ for each block, appropriate exposure conditions can be set for each block.

[0228] (56) The camera 1 includes a control unit 34 that controls a light source that emits fill light, and a setting unit 34b sets, as a shooting condition, whether or not the light source controlled by the control unit 34 emits light or the amount of light emitted. This allows appropriate setting processing to be performed even if the imaging conditions differ for each block.

[0229] The system may be configured to be switchable between mode 1, in which the above-mentioned second correction process is performed as a pre-processing, and mode 2, in which the second correction process is not performed as a pre-processing. When mode 1 is selected, the control unit 34 performs the above-mentioned pre-processing and then performs image processing. On the other hand, when mode 2 is selected, the control unit 34 performs image processing and other processes without performing the above-mentioned pre-processing. For example, when a part of a face detected as a subject element has a shadow, if an image is captured under different imaging conditions for an area including a shadowed part of the face and an area including a non-shadowed part of the face so that the brightness of the shadowed part of the face is approximately the same as the brightness of the non-shadowed part of the face, and then the second correction process is performed on the image generated by the imaging conditions, unintended color interpolation may be performed on the shadowed part due to the difference in the imaging conditions set. By configuring the system to be switchable between mode 1 and mode 2 so that the color interpolation process can be performed using the image data as it is without performing the second correction process, it is possible to avoid unintended color interpolation.

[0230] ---Modification of the first embodiment--- The following modifications are also within the scope of the present invention, and one or more of the modifications may be combined with the above-described embodiment. (Variation 1) 18(a) to 18(c) are diagrams illustrating the arrangement of the first and second imaging regions on the imaging surface of the image sensor 32a. In the example of FIG. 18(a), the first imaging region is composed of even-numbered columns, and the second imaging region is composed of odd-numbered columns. That is, the imaging surface is divided into even-numbered columns and odd-numbered columns.

[0231] 18(b), the first imaging region is made up of odd-numbered rows, and the second imaging region is made up of even-numbered rows, i.e., the imaging surface is divided into odd-numbered rows and even-numbered rows.

[0232] 18(c), the first imaging region is made up of blocks in even rows and odd columns, and blocks in odd rows and even columns. The second imaging region is made up of blocks in even rows and even columns, and blocks in odd rows and odd columns. That is, the imaging surface is divided into a checkerboard pattern.

[0233] 18(a) to 18(c), a first image based on the photoelectric conversion signal read out from the first imaging region and a second image based on the photoelectric conversion signal read out from the second imaging region are generated by the photoelectric conversion signal read out from the imaging element 32a that has captured one frame. According to the first modification, the first image and the second image are captured at the same angle of view and include a common subject image.

[0234] In the first modification, the control unit 34 uses the first image for display and the second image for detection. Specifically, the control unit 34 causes the display unit 35 to display the first image as a live view image. The control unit 34 also causes the object detection unit 34a to perform subject detection processing using the second image, the lens movement control unit 34d to perform focus detection processing using the second image, and the setting unit 34b to perform exposure calculation processing using the second image.

[0235] In the first modification, the imaging conditions set in the first imaging area for capturing the first image are referred to as the first imaging conditions, and the imaging conditions set in the second imaging area for capturing the second image are referred to as the second imaging conditions. The control unit 34 may make the first imaging conditions and the second imaging conditions different from each other.

[0236] 1. As an example, the control unit 34 sets the first imaging condition to a condition suitable for display by the display unit 35. For example, the first imaging condition set in the first imaging area is made uniform throughout the entire first imaging area of ​​the imaging screen. Meanwhile, the control unit 34 sets the second imaging condition set in the second imaging area to a condition suitable for focus detection processing, subject detection processing, and exposure calculation processing. The second imaging condition is made uniform throughout the entire second imaging area of ​​the imaging screen. In addition, when the conditions suitable for the focus detection process, the subject detection process, and the exposure calculation process are different, the control unit 34 may set the second imaging condition in the second imaging area to be different for each frame. For example, the second imaging condition in the first frame is set to be the condition suitable for the focus detection process, the second imaging condition in the second frame is set to be the condition suitable for the subject detection process, and the second imaging condition in the third frame is set to be the condition suitable for the exposure calculation process. In these cases, the second imaging condition in each frame is set to be the same throughout the second imaging area of ​​the imaging screen.

[0237] 2. As another example, the control unit 34 may set the first imaging condition in the first imaging region differently depending on the region. The setting unit 34b of the control unit 34 sets different first imaging conditions for each region including the subject elements divided by the setting unit 34b. On the other hand, the control unit 34 sets the second imaging condition in the second imaging region to be the same for the entire second imaging region of the imaging screen. The control unit 34 sets the second imaging condition to a condition suitable for focus detection processing, subject detection processing, and exposure calculation processing, but when the conditions suitable for focus detection processing, subject detection processing, and exposure calculation processing are different, the imaging condition set in the second imaging region may be different for each frame.

[0238] 3. As another example, the control unit 34 may set the first imaging condition for the first imaging area to be the same for the entire first imaging area of ​​the imaging screen, while setting the second imaging condition for the second imaging area to be different in the imaging screen. For example, the setting unit 34b sets different second imaging conditions for each area including the divided subject elements. Even in this case, if the conditions suitable for the focus detection process, the subject detection process, and the exposure calculation process are different, the imaging condition set for the second imaging area may be different for each frame.

[0239] 4. Furthermore, as another example, the control unit 34 makes the first imaging condition set for the first imaging region different on the imaging screen, and makes the second imaging condition set for the second imaging region different on the imaging screen. For example, the setting unit 34b sets different first imaging conditions for each region including the divided subject elements, and sets different second imaging conditions for each region including the divided subject elements.

[0240] In Fig. 18(a) to Fig. 18(c), the area ratio of the first imaging region and the second imaging region may be made different. For example, the control unit 34 sets the ratio of the first imaging region higher than that of the second imaging region, sets the ratio of the first imaging region and the second imaging region equal as illustrated in Fig. 18(a) to Fig. 18(c), or sets the ratio of the first imaging region lower than that of the second imaging region based on the user's operation or the determination of the control unit 34. By making the area ratio of the first imaging region and the second imaging region different, it is possible to make the first image higher in resolution than the second image, make the resolution of the first image and the second image equal, or make the second image higher in resolution than the first image.

[0241] (Variation 2) In the above-described embodiment, in the second correction process for image processing, when the imaging condition (first imaging condition) applied to the position of interest is different from the imaging condition (fourth imaging condition) applied to the reference position around the position of interest, the correction unit 33b of the image processing unit 33 corrects the image data of the fourth imaging condition (image data of the fourth imaging condition among the image data of the reference position) based on the first imaging condition. That is, by performing the second correction process on the image data of the fourth imaging condition of the reference position, discontinuity of the image based on the difference between the first imaging condition and the fourth imaging condition is mitigated.

[0242] Instead, in Modification 2, the correction unit 33b of the image processing unit 33 may correct the image data of the first imaging condition (the image data of the interest position and the image data of the reference position under the first imaging condition) based on the fourth imaging condition. In this case as well, it is possible to reduce discontinuity of the image based on the difference between the first imaging condition and the fourth imaging condition.

[0243] Alternatively, the correction unit 33b of the image processing unit 33 may correct both the image data under the first imaging condition and the image data under the fourth imaging condition. That is, the second correction process may be performed on the image data of the position of interest under the first imaging condition, the image data under the first imaging condition among the image data of the reference position, and the image data under the fourth imaging condition among the image data of the reference position, to alleviate discontinuity of images based on the difference between the first imaging condition and the fourth imaging condition. For example, in the above (Example 1), the image data of the reference pixel Pr under the first imaging condition (ISO sensitivity is 100) is multiplied by 400 / 100 as the second correction process, and the image data of the reference pixel Pr under the fourth imaging condition (ISO sensitivity is 800) is multiplied by 400 / 800 as the second correction process. This reduces the difference between the image data due to the difference in the imaging conditions. Note that the pixel data of the target pixel is subjected to a second correction process in which the pixel data is multiplied by 100 / 400 after the color interpolation process. This second correction process can change the pixel data of the target pixel after the color interpolation process to a value similar to that when the pixel data is captured under the first imaging condition. Furthermore, in the above (Example 1), the degree of the second correction process may be changed depending on the distance from the boundary between the first region and the fourth region. Then, the rate at which the image data increases or decreases due to the second correction process can be reduced compared to the above (Example 1), and the noise generated by the second correction process can be reduced. Although the above (Example 1) has been described, the above (Example 2) can also be similarly applied.

[0244] According to the second modification, similarly to the above-described embodiment, it is possible to perform appropriate image processing on image data generated for areas with different imaging conditions.

[0245] (Variation 3) In the above-described embodiment, when the second correction process is performed on the image data, the corrected image data is obtained by performing a calculation based on the difference between the first and fourth imaging conditions. Instead of the calculation, the corrected image data may be obtained by referring to a correction table. For example, the corrected image data is read by inputting the first and fourth imaging conditions as arguments. Alternatively, the correction coefficient may be read by inputting the first and fourth imaging conditions as arguments.

[0246] (Variation 4) In the second correction process of the above-described embodiment, an upper limit and a lower limit of the corrected image data may be set. By setting the upper limit and the lower limit, it is possible to restrict the correction so as not to make more correction than necessary. The upper limit and the lower limit may be determined in advance, or, in the case where a photometry sensor is provided in addition to the image sensor 32a, they may be determined based on the output signal from the photometry sensor.

[0247] (Variation 5) In the above embodiment, an example has been described in which the setting unit 34b of the control unit 34 detects subject elements based on a live view image and divides the screen of the live view image into areas including the subject elements. In Modification 5, if the control unit 34 includes a photometry sensor separate from the image sensor 32a, the control unit 34 may divide the areas based on an output signal from the photometry sensor.

[0248] The control unit 34 divides the image into a foreground and a background based on the output signal from the photometry sensor. Specifically, the control unit 34 divides the live view image acquired by the image sensor 32a into a foreground region corresponding to the region determined to be the foreground based on the output signal from the photometry sensor, and a background region corresponding to the region determined to be the background based on the output signal from the photometry sensor.

[0249] The control unit 34 further arranges the first imaging region and the second imaging region at a position corresponding to the foreground region on the imaging surface of the image sensor 32a, as illustrated in Fig. 18(a) to Fig. 18(c). On the other hand, the control unit 34 arranges only the first imaging region on the imaging surface of the image sensor 32a at a position corresponding to the background region on the imaging surface of the image sensor 32a. The control unit 34 uses the first image for display and the second image for detection.

[0250] According to the fifth modification, the output signal from the photometry sensor can be used to perform area division of the live view image captured by the image sensor 32a. Also, for the foreground area, a first image for display and a second image for detection can be obtained, and for the background area, only the first image for display can be obtained.

[0251] (Variation 6) In the sixth modification, the generating unit 33c of the image processing unit 33 performs a contrast adjustment process as an example of a second correction process. That is, the generating unit 33c varies the gradation curve (gamma curve) to reduce discontinuity in the image based on the difference between the first and fourth imaging conditions.

[0252] For example, assume that only the ISO sensitivity differs between the first and fourth imaging conditions, that is, the ISO sensitivity of the first imaging condition is 100 and the ISO sensitivity of the fourth imaging condition is 800. The generation unit 33c compresses the value of the image data of the fourth imaging condition among the image data at the reference position to 1 / 8 by flattening the gradation curve.

[0253] Alternatively, the generating unit 33c may increase the value of the image data of the interest position and the image data of the reference position under the first imaging condition by eight times by increasing the gradation curve.

[0254] According to the sixth modification, similarly to the above-described embodiment, it is possible to appropriately perform image processing on image data generated in regions with different imaging conditions. For example, it is possible to suppress discontinuity or discomfort that appears in the image after image processing due to differences in imaging conditions at the boundary between regions.

[0255] (Variation 7) In the seventh modification, the image processing unit 33 performs the above-mentioned image processing (for example, noise reduction processing) so as not to impair the contours of the subject elements. Generally, smoothing filter processing is adopted when performing noise reduction. When a smoothing filter is used, while the noise reduction effect is achieved, the boundaries of the subject elements may become blurred.

[0256] Therefore, the generation unit 33c of the image processing unit 33 performs contrast adjustment processing in addition to or together with the noise reduction processing to compensate for the blurring of the boundaries of the above-mentioned subject elements. In the seventh modification, the generation unit 33c of the image processing unit 33 sets an S-shaped curve as a density conversion (gradation conversion) curve (so-called S-shaped conversion). By performing contrast adjustment using the S-shaped conversion, the generation unit 33c of the image processing unit 33 stretches the gradation parts of the bright data and the dark data, respectively, to increase the number of gradations of the bright data (and dark data), and compresses the image data of intermediate gradations to reduce the number of gradations. This reduces the number of image data with medium brightness in the image, and increases the data classified as either bright or dark, thereby making it possible to compensate for the blurring of the boundaries of the subject elements.

[0257] According to the seventh modification, the blurring of the boundaries between object elements can be compensated for by making the contrast between light and dark in the image clearer.

[0258] (Variation 8) In the eighth modification, the generation unit 33c of the image processing unit 33 changes the white balance adjustment gain so as to reduce discontinuity in the image based on the difference between the first and fourth capturing conditions.

[0259] For example, when the imaging conditions (called the first imaging conditions) applied when capturing an image at the focus position are different from the imaging conditions (called the fourth imaging conditions) applied when capturing an image at a reference position surrounding the focus position, the generation unit 33c of the image processing unit 33 changes the white balance adjustment gain so as to bring the white balance of the image data of the fourth imaging condition among the image data of the reference position closer to the white balance of the image data acquired under the first imaging conditions.

[0260] In addition, the generation unit 33c of the image processing unit 33 may change the white balance adjustment gain so as to bring the white balance of the image data of the first imaging condition among the image data of the reference position and the image data of the interest position closer to the white balance of the image data acquired under the fourth imaging condition.

[0261] According to variant example 8, by aligning the white balance adjustment gain for image data generated in areas with different imaging conditions to the adjustment gain for one of the areas with different imaging conditions, discontinuity in the image due to the difference between the first and fourth imaging conditions can be mitigated.

[0262] (Variation 9) A plurality of image processing units 33 may be provided to perform image processing in parallel. For example, image processing may be performed on image data captured in region A of the imaging unit 32 while image processing is performed on image data captured in region B of the imaging unit 32. The plurality of image processing units 33 may perform the same image processing, or may perform different image processing. In other words, the same parameters or the like may be applied to the image data of region A and region B to perform similar image processing, or different parameters or the like may be applied to the image data of region A and region B to perform different image processing.

[0263] In the case where a plurality of image processing units 33 are provided, image processing may be performed by one image processing unit on image data to which the first imaging condition is applied, and image processing may be performed by another image processing unit on image data to which the fourth imaging condition is applied. The number of image processing units is not limited to the above two, and may be the same number as the number of imaging conditions that can be set, for example. That is, each image processing unit is responsible for image processing for each region to which a different imaging condition is applied. According to the ninth modification, imaging under different imaging conditions for each region and image processing for image data of images obtained for each of the above regions can be carried out in parallel.

[0264] (Variation 10) In the above description, the camera 1 has been taken as an example, but the camera 1 may be configured as a highly functional mobile phone 250 (FIG. 20) equipped with a camera function such as a smartphone, or as a mobile device such as a tablet terminal.

[0265] (Variation 11) In the above-described embodiment, the camera 1 is described as an example in which the imaging unit 32 and the control unit 34 are configured as a single electronic device. Instead of this, for example, the imaging unit 32 and the control unit 34 may be provided separately, and the imaging system 1B may be configured in which the imaging unit 32 is controlled by the control unit 34 via communication. Hereinafter, an example in which an imaging device 1001 including an imaging section 32 is controlled from a display device 1002 including a control section 34 will be described with reference to FIG.

[0266] Fig. 19 is a block diagram illustrating a configuration of an imaging system 1B according to Modification 11. In Fig. 19, the imaging system 1B is configured with an imaging device 1001 and a display device 1002. The imaging device 1001 includes a first communication unit 1003 in addition to the imaging optical system 31 and imaging unit 32 described in the above embodiment. The display device 1002 includes a second communication unit 1004 in addition to the image processing unit 33, control unit 34, display unit 35, operation member 36, and recording unit 37 described in the above embodiment.

[0267] The first communication unit 1003 and the second communication unit 1004 can perform two-way image data communication using, for example, well-known wireless communication technology or optical communication technology. The imaging device 1001 and the display device 1002 may be connected by a wired cable, and the first communication unit 1003 and the second communication unit 1004 may perform bidirectional image data communication.

[0268] In the imaging system 1B, the control unit 34 controls the imaging unit 32 by performing data communication via the second communication unit 1004 and the first communication unit 1003. For example, by transmitting and receiving predetermined control data between the imaging device 1001 and the display device 1002, the display device 1002 divides the screen into a plurality of regions based on the image as described above, sets different imaging conditions for each divided region, and reads out photoelectric conversion signals photoelectrically converted in each region.

[0269] According to variant example 11, a live view image acquired by the imaging device 1001 and transmitted to the display device 1002 is displayed on the display unit 35 of the display device 1002, allowing the user to remotely control the imaging device 1001 from the display device 1002 located away from the imaging device 1001. The display device 1002 can be configured, for example, by a highly functional mobile phone 250 such as a smartphone. The imaging device 1001 can be configured by an electronic device including the stacked imaging element 100 described above. Although an example has been described in which the control unit 34 of the display device 1002 is provided with the object detection unit 34a, the setting unit 34b, the imaging control unit 34c, and the lens movement control unit 34d, part of the object detection unit 34a, the setting unit 34b, the imaging control unit 34c, and the lens movement control unit 34d may also be provided in the imaging device 1001.

[0270] (Variation 12) The program can be supplied to a mobile device such as the above-mentioned camera 1, high-function mobile phone 250, or tablet terminal by transmitting the program from a personal computer 205 storing the program to the mobile device via infrared communication or short-range wireless communication, as illustrated in FIG. 20, for example.

[0271] The program may be supplied to the personal computer 205 by inserting a recording medium 204 such as a CD-ROM storing the program into the personal computer 205, or by loading the program into the personal computer 205 via a communication line 201 such as a network. When the program is supplied via the communication line 201, the program is stored in a storage device 203 of a server 202 connected to the communication line.

[0272] The program can also be directly transmitted to the mobile device via a wireless LAN access point (not shown) connected to the communication line 201. Furthermore, a recording medium 204B such as a memory card storing the program may be set in the mobile device. In this way, the program can be supplied as a computer program product in various forms, such as a recording medium or via a communication line.

[0273] ---Second embodiment--- With reference to Figs. 21 to 27, a digital camera will be described as an example of an electronic device equipped with an image processing device according to the second embodiment. In the following description, the same components as those in the first embodiment are given the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as those in the first embodiment. This embodiment differs from the first embodiment mainly in that, instead of providing the image processing unit 33 of the first embodiment, the imaging unit 32A further includes an image processing unit 32c having the same function as the image processing unit 33 of the first embodiment.

[0274] Fig. 21 is a block diagram illustrating the configuration of a camera 1C according to the second embodiment. In Fig. 21, the camera 1C has an imaging optical system 31, an imaging unit 32A, a control unit 34, a display unit 35, an operation member 36, and a recording unit 37. The imaging unit 32A further includes an image processing unit 32c having the same function as the image processing unit 33 of the first embodiment.

[0275] The image processing unit 32c includes an input unit 321, a correction unit 322, and a generation unit 323. The input unit 321 receives image data from the imaging element 32a. The correction unit 322 performs pre-processing to correct the input image data. The pre-processing performed by the correction unit 322 is the same as the pre-processing performed by the correction unit 33b in the first embodiment. The generation unit 323 performs image processing on the input image data and the image data after the pre-processing to generate an image. The image processing performed by the generation unit 323 is the same as the image processing performed by the generation unit 33c in the first embodiment.

[0276] Fig. 22 is a diagram showing a schematic diagram of the correspondence between each block in this embodiment and a plurality of correction units 322. In Fig. 22, one square of the imaging chip 111 represented by a rectangle represents one block 111a. Similarly, one square of the image processing chip 114, which will be described later, represented by a rectangle represents one correction unit 322.

[0277] In this embodiment, the correction unit 322 is provided for each block 111a. In other words, the correction unit 322 is provided for each block, which is the smallest unit of an area on the imaging surface where the imaging conditions can be changed. For example, the hatched block 111a in FIG. 22 corresponds to the hatched correction unit 322. The hatched correction unit 322 in FIG. 22 performs preprocessing on image data from pixels included in the hatched block 111a. Each correction unit 322 performs preprocessing on image data from pixels included in the corresponding block 111a. This allows pre-processing of image data to be performed in parallel by multiple correction units 322, thereby reducing the processing burden on the correction units 322 and enabling an appropriate image to be generated in a short period of time from image data generated in areas with different imaging conditions. In the following description, when explaining the relationship between a certain block 111a and the pixels contained in the block 111a, the block 111a may be referred to as the block 111a to which the pixel belongs. The block 111a may be referred to as a unit segment, and a collection of multiple blocks 111a, i.e., a collection of multiple unit segments, may be referred to as a composite segment.

[0278] 23 is a cross-sectional view of the stacked imaging element 100A. The stacked imaging element 100A further includes an image processing chip 114 that performs the above-mentioned pre-processing and image processing, in addition to a back-illuminated imaging chip 111, a signal processing chip 112, and a memory chip 113. That is, the above-mentioned image processing unit 32c is provided in the image processing chip 114. The imaging chip 111, the signal processing chip 112, the memory chip 113 and the image processing chip 114 are stacked and electrically connected to one another by conductive bumps 109 such as Cu.

[0279] A plurality of bumps 109 are arranged on the mutually opposing surfaces of the memory chip 113 and the image processing chip 114. These bumps 109 are aligned with each other, and the memory chip 113 and the image processing chip 114 are pressed together, whereby the aligned bumps 109 are bonded together and electrically connected.

[0280] <First correction process> As in the first embodiment, in the second embodiment, after the area of ​​the imaging screen is divided by the setting unit 34b, it is possible to set (change) imaging conditions for an area selected by a user or an area determined by the control unit 34. When the control unit 34 sets different imaging conditions for the divided areas, it causes the correction unit 322 of the image processing unit 32c to perform preprocessing as necessary.

[0281] That is, when a block, which is the smallest unit for setting imaging conditions, contains boundaries of areas based on multiple subject elements and the image data from this block contains whiteout or blackout, the control unit 34 causes the correction unit 322 to perform the following first correction process as one of the pre-processing steps performed before image processing, focus detection processing, subject detection processing, and processing for setting imaging conditions.

[0282] 1. Applying the same correction to the entire area where whiteout or blackout has occurred. (1-1) As in the first embodiment, the correction unit 322 replaces image data where whiteout or blackout has occurred with image data acquired in one block within the same captured image by one of the following modes (i) to (iv) as the first correction process.

[0283] (i) The correction unit 322 uses image data obtained from one of the reference blocks located around the target block, which is located closest to the area where whiteout or blackout has occurred, to replace the image data in the target block where whiteout or blackout has occurred with the same data.

[0284] (ii) The correction unit 322 replaces the image data in the target block that has been blown out or crushed in the shadows with the same data, using image data obtained from one of the reference blocks located around the target block and selected from the reference blocks with the imaging conditions (e.g., the fourth imaging condition) that are most frequently set for the same subject element (e.g., a mountain) as the subject element that has been blown out or crushed in the shadows.

[0285] (iii) The correction unit 322 replaces the image data in which whiteout or blackout has occurred with image data corresponding to pixels adjacent to the whiteout or blackout pixel in the target block, among the image data corresponding to multiple pixels (four pixels in the example of FIG. 8) obtained from one reference block selected by (i) or (ii) above.

[0286] (iv) The correction unit 322 replaces the image data in the target block that has blown out highlights or crushed shadows with image data generated based on image data corresponding to multiple pixels (four pixels in the example of Figure 8) obtained from one reference block selected by (i) or (ii) above.

[0287] As in the first embodiment, when calculating the average value of image data, instead of a simple average, a weighted average value weighted according to the distance from a pixel where whiteout or blackout has occurred may be used.

[0288] Also, similar to the first embodiment, instead of calculating the average value of image data corresponding to multiple pixels included in the reference block, an intermediate value of the image data corresponding to multiple pixels may be calculated and image data corresponding to a pixel where whiteout or blackout has occurred may be replaced with this intermediate value.

[0289] (1-2) As in the first embodiment, the correction unit 322 replaces image data in which whiteout or blackout has occurred with image data acquired in multiple blocks within the same captured image by any one of the following aspects (i) to (iv) as a first correction process.

[0290] (i) The correction unit 322 uses image data obtained from multiple reference blocks located around the target block, which are located around the area where whiteout or blackout has occurred, to replace the image data in the target block where whiteout or blackout has occurred with the same data.

[0291] (ii) The correction unit 322 replaces the image data in the target block that has been blown out or crushed in the shadows with the same data, using image data obtained from a plurality of reference blocks selected from among the reference blocks located around the target block and that have the imaging condition (e.g., the fourth imaging condition) that is most frequently set for the same subject element (e.g., a mountain) as the subject element that has been blown out or crushed in the shadows.

[0292] (iii) The correction unit 322 replaces the image data in which whiteout or blackout has occurred with image data corresponding to pixels adjacent to the whiteout or blackout pixel in the target block, among the image data corresponding to the multiple pixels obtained from the multiple reference blocks selected by (i) or (ii) above.

[0293] (iv) The correction unit 322 replaces image data in the target block that has blown out highlights or crushed shadows with image data generated based on image data corresponding to multiple pixels obtained from multiple reference blocks selected by (i) or (ii) above.

[0294] As in the first embodiment, when calculating the average value of image data, instead of a simple average, a weighted average value weighted according to the distance from a pixel where whiteout or blackout has occurred may be used.

[0295] Also, similar to the first embodiment, instead of calculating the average value of image data corresponding to multiple pixels contained in multiple reference blocks, an intermediate value of the image data corresponding to multiple pixels may be calculated and image data corresponding to pixels where whiteout or blackout has occurred may be replaced with this intermediate value.

[0296] 2. Apply multiple corrections to the entire area where whiteout or blackout occurs. (2-1) As in the first embodiment, the correction unit 322 replaces image data in which whiteout or blackout has occurred with image data acquired in one block within the same captured image by any one of the following aspects (i) to (iii) as a first correction process.

[0297] (i) The correction unit 322 replaces multiple pieces of image data in the target block that have experienced whiteout or blackout with different data, using image data corresponding to pixels adjacent to the whiteout or blackout pixels in reference blocks located around the target block.

[0298] (ii) The correction unit 322 replaces multiple pieces of image data in the target block that have suffered from blown out highlights or crushed shadows with different data, using image data obtained from one reference block selected from among the reference blocks located around the target block and that have the imaging conditions (e.g., the fourth imaging condition) that are most frequently set for the same subject element (e.g., a mountain) as the subject element that has suffered from blown out highlights or crushed shadows.

[0299] (iii) The correction unit 33b replaces multiple pieces of image data in the target block that have blown out highlights or crushed shadows with different data, using image data generated based on image data corresponding to multiple pixels (four pixels in the example of Figure 8) obtained in one reference block selected by (i) or (ii) above.

[0300] As in the first embodiment, when calculating the average value of image data, instead of a simple average, a weighted average value weighted according to the distance from a pixel where whiteout or blackout has occurred may be used.

[0301] (2-2) As in the first embodiment, the correction unit 322 replaces image data in which whiteout or blackout has occurred with image data acquired in multiple blocks within the same captured image by any one of the following aspects (i) to (iv) as a first correction process.

[0302] (i) The correction unit 322 replaces multiple pieces of image data in the target block that have experienced blown out highlights or crushed shadows with different data, using image data obtained from multiple reference blocks located around the target block and surrounding the area that has blown out highlights or crushed shadows.

[0303] (ii) The correction unit 322 replaces multiple pieces of image data in the target block that have suffered from blown out highlights or crushed shadows with different data, using image data obtained from multiple reference blocks selected from among the reference blocks located around the target block and that have the imaging condition (e.g., the fourth imaging condition) that is most frequently set for the same subject element (e.g., a mountain) as the subject element that has suffered from blown out highlights or crushed shadows.

[0304] (iii) The correction unit 322 replaces image data in the target block that has blown out highlights or crushed shadows with image data generated based on image data corresponding to multiple pixels obtained from multiple reference blocks selected by (i) or (ii) above.

[0305] As in the first embodiment, when calculating the average value of image data, instead of a simple average, a weighted average value weighted according to the distance from a pixel where whiteout or blackout has occurred may be used.

[0306] Also, similar to the first embodiment, instead of calculating the average value of image data corresponding to multiple pixels contained in multiple reference blocks, an intermediate value of the image data corresponding to multiple pixels may be calculated and the image data corresponding to the pixel where black crush has occurred may be replaced with this intermediate value.

[0307] The control unit 34 determines which of the various types of correction in the first correction process described above to perform, based on the state of settings (including operation menu settings) made by the operation members 36, for example. Note that the control unit 34 may determine which type of correction to perform depending on the imaging scene mode set in the camera 1 and the type of the detected subject element.

[0308] <Second correction process> The control unit 34 further causes the correction unit 322 to perform the following second correction process as necessary before the image processing, the focus detection process, the subject detection process (detection of subject elements), and the process of setting the imaging conditions.

[0309] 1. When performing image processing 1-1. When the imaging conditions of the pixel of interest P and the imaging conditions of the multiple reference pixels Pr surrounding the pixel of interest P are the same In this case, in the image processing unit 32c, the correction unit 322 does not perform the second correction process, and the generation unit 323 performs image processing using image data of a plurality of reference pixels Pr that have not been subjected to the second correction process.

[0310] 1-2. When the imaging conditions of the pixel of interest P differ from the imaging conditions of at least one of the reference pixels Pr surrounding the pixel of interest P The imaging condition applied to the pixel of interest P is defined as the first imaging condition, the imaging condition applied to a portion of the multiple reference pixels Pr is defined as the first imaging condition, and the imaging condition applied to the remaining reference pixels Pr is defined as the second imaging condition. In this case, the correction unit 322 corresponding to the block 111a to which the reference pixel Pr to which the second imaging condition is applied belongs performs the second correction process on the image data of the reference pixel Pr to which the second imaging condition is applied as in the following (Example 1) to (Example 3). Then, the generation unit 323 performs image processing to calculate image data of the target pixel P by referring to the image data of the reference pixel Pr to which the first imaging condition is applied and the image data of the reference pixel Pr after the second correction process.

[0311] (Example 1) The correction unit 322 corresponding to the block 111a to which the reference pixel Pr to which the second imaging condition is applied belongs applies 100 / 800 to the image data of the reference pixel Pr as the second correction process when, for example, only the ISO sensitivity differs between the first and second imaging conditions and the ISO sensitivity of the first imaging condition is 100 and the ISO sensitivity of the second imaging condition is 800. This reduces the difference between the image data due to the difference in imaging conditions.

[0312] (Example 2) For example, when only the shutter speed differs between the first and second imaging conditions, and the shutter speed of the first imaging condition is 1 / 1000 second and the shutter speed of the second imaging condition is 1 / 100 second, the correction unit 322 corresponding to the block 111a to which the reference pixel Pr to which the second imaging condition is applied multiplies the image data of the reference pixel Pr by 1 / 1000 / 1 / 100=1 / 10 as the second correction process, thereby reducing the difference between the image data due to the difference in imaging conditions.

[0313] (Example 3) The correction unit 322 corresponding to the block 111a to which the reference pixel Pr to which the second imaging condition is applied belongs performs the second correction process in which, for example, when only the frame rate differs between the first imaging condition and the second imaging condition (the charge accumulation time is the same) and the frame rate of the first imaging condition is 30 fps and the frame rate of the second imaging condition is 60 fps, the image data of the reference pixel Pr, i.e., the image data of the second imaging condition (60 fps), is adopted as the image data of the frame image acquired under the first imaging condition (30 fps) and the acquisition start timing of the frame image close to that of the frame image acquired under the first imaging condition (30 fps). This reduces the difference between the image data due to the difference in the imaging conditions. The second correction process may involve interpolating and calculating image data of a frame image whose acquisition start timing is close to that of a frame image acquired under the first imaging condition (30 fps) based on multiple adjacent frame images acquired under the second imaging condition (60 fps).

[0314] The same applies to the case where the imaging condition applied to the pixel of interest P is the second imaging condition, and the imaging condition applied to the reference pixels Pr surrounding the pixel of interest P is the first imaging condition. That is, in this case, the correction unit 322 corresponding to the block 111a to which the reference pixel Pr to which the first imaging condition is applied belongs performs the second correction process on the image data of the reference pixel Pr as in the above-mentioned (Example 1) to (Example 3).

[0315] As described above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions.

[0316] The generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing, similar to the generation unit 33c of the image processing unit 33 in the first embodiment, based on the image data of the reference pixel Pr to which the same imaging conditions as those of the target pixel P have been applied and the image data of the reference pixel Pr to which the second correction processing has been performed by the correction unit 322.

[0317] Figure 24 is a schematic diagram showing the processing of image data (hereinafter referred to as first image data) from each pixel included in a partial area of ​​the imaging surface to which a first imaging condition is applied (hereinafter referred to as first imaging area 141), and image data (hereinafter referred to as second image data) from each pixel included in a partial area of ​​the imaging surface to which a second imaging condition is applied (hereinafter referred to as second imaging area 142).

[0318] Each pixel included in the first imaging region 141 outputs first image data captured under the first imaging condition, and each pixel included in the second imaging region 142 outputs second image data captured under the second imaging condition. The first image data is output to a correction unit 322 corresponding to the block 111a to which the pixel that generated the first image data belongs, among the correction units 322 provided in the image processing chip 114. In the following description, the multiple correction units 322 corresponding to the multiple blocks 111a to which the pixels that generated the respective first image data belong will be referred to as a first processing unit 151. The first processing unit 151 performs the first correction processing and / or the second correction processing on the first image data as necessary.

[0319] Similarly, the second image data is output to a correction unit 322 corresponding to the block 111a to which the pixel that generated the second image data belongs, among the correction units 322 provided in the image processing chip 114. In the following description, the multiple correction units 322 corresponding to the multiple blocks 111a to which the pixels that generated the second image data belong will be referred to as second processing units 152. The second processing unit 152 performs the first correction processing and / or the second correction processing on the second image data as necessary.

[0320] In the above-mentioned first correction process, for example, when the block of interest is included in the first imaging region 141, the first correction process described above, i.e., replacement process, is performed by the first processing unit 151 as shown in Fig. 24. As a result, image data in the block of interest in which whiteout or blackout has occurred is replaced with second image data from a reference block included in the second imaging region 142. For this purpose, the first processing unit 151 receives the second image data from the reference block as information 182 from the second processing unit 152, for example.

[0321] In the above-mentioned second correction process, for example, when the pixel of interest P is included in the first imaging region 141, the second image data from the reference pixel Pr included in the second imaging region 142 is subjected to the above-mentioned second correction process by the second processing unit 152 as shown in Fig. 24. Note that the second processing unit 152 receives information 181 on the first imaging condition required to reduce a difference between image data due to a difference in imaging condition from, for example, the first processing unit 151. Similarly, for example, when the pixel of interest P is included in the second imaging region 142, the first image data from the reference pixel Pr included in the first imaging region 141 is subjected to the above-mentioned second correction processing by the first processing unit 151. Note that the first processing unit 151 receives information on the second imaging condition required to reduce the difference between the image data due to the difference in the imaging condition from the second processing unit 152.

[0322] When the pixel of interest P and the reference pixel Pr are included in the first imaging region 141, the first processing unit 151 does not perform the second correction process on the first image data from the reference pixel Pr. Similarly, when the pixel of interest P and the reference pixel Pr are included in the second imaging region 142, the second processing unit 152 does not perform the second correction process on the second image data from the reference pixel Pr. Alternatively, both the image data under the first imaging condition and the image data under the second imaging condition may be corrected by the first processing unit 151 and the second processing unit 152. That is, the discontinuity of the images based on the difference between the first imaging condition and the second imaging condition may be mitigated by performing the second correction process on the image data of the position of interest under the first imaging condition, the image data under the first imaging condition among the image data of the reference position, and the image data under the second imaging condition among the image data of the reference position. For example, in the above (Example 1), the image data of the reference pixel Pr under the first imaging condition (ISO sensitivity is 100) is multiplied by 400 / 100 as the second correction process, and the image data of the reference pixel Pr under the second imaging condition (ISO sensitivity is 800) is multiplied by 400 / 800 as the second correction process. This reduces the difference between the image data due to the difference in the imaging conditions. Note that the pixel data of the target pixel is subjected to a second correction process in which the pixel data is multiplied by 100 / 400 after the color interpolation process. This second correction process can change the pixel data of the target pixel after the color interpolation process to a value similar to that when the pixel data is captured under the first imaging condition. Furthermore, in the above (Example 1), the degree of the second correction process may be changed depending on the distance from the boundary between the first region and the second region. Then, the rate at which the image data increases or decreases due to the second correction process can be reduced compared to the above (Example 1), and the noise generated by the second correction process can be reduced. Although the above (Example 1) has been described, the above (Example 2) can also be similarly applied.

[0323] The generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing based on the image data from the first processing unit 151 and the second processing unit 152, and outputs the image data after image processing.

[0324] When the pixel of interest P is located in the second imaging region 142, the first processing unit 151 may perform the second correction process on the first image data from all pixels included in the first imaging region 141, or may perform the second correction process only on the first image data from pixels that may be used to interpolate the pixel of interest P in the second imaging region 142 among the pixels included in the first imaging region 141. Similarly, when the pixel of interest P is located in the first imaging region 141, the second processing unit 152 may perform the second correction process on the second image data from all pixels included in the second imaging region 142, or may perform the second correction process only on the second image data from pixels that may be used to interpolate the pixel of interest P in the first imaging region 141 among the pixels included in the second imaging region 142.

[0325] 2. When performing focus detection processing As in the first embodiment, the lens movement control unit 34d of the control unit 34 performs focus detection processing using signal data (image data) corresponding to a predetermined position (focus point) on the imaging screen. When different imaging conditions are set between the divided areas and the focus point for AF operation is located at the boundary between the divided areas, that is, when the focus point is divided into two areas, the first area and the second area, in this embodiment, as described in 2-2 below, the lens movement control unit 34d of the control unit 34 causes the correction unit 322 to perform a second correction processing on the signal data for focus detection of at least one area.

[0326] 2-1. When signal data from pixels within frame 170 in FIG. 15 does not include a mixture of signal data to which the first imaging condition is applied and signal data to which the second imaging condition is applied In this case, the correction unit 322 does not perform the second correction process, and the lens movement control unit 34d of the control unit 34 performs focus detection processing using signal data from the focus detection pixels indicated by the frame 170 as is.

[0327] 2-2. When signal data from pixels within frame 170 in FIG. 15 includes a mixture of signal data to which the first imaging condition is applied and signal data to which the second imaging condition is applied In this case, the lens movement control unit 34d of the control unit 34 causes the correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied belongs among the pixels in the frame 170 to perform the second correction process as in the following (Example 1) to (Example 3). Then, the lens movement control unit 34d of the control unit 34 performs the focus detection process using the signal data of the pixel to which the first imaging condition is applied and the signal data after the second correction process.

[0328] (Example 1) The correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied belongs applies a second correction process to the signal data of the second imaging condition by 100 / 800 when, for example, only the ISO sensitivity differs between the first imaging condition and the second imaging condition, and the ISO sensitivity of the first imaging condition is 100 and the ISO sensitivity of the second imaging condition is 800. This reduces the difference between the signal data due to the difference in imaging conditions.

[0329] (Example 2) For example, when only the shutter speed differs between the first and second imaging conditions and the shutter speed of the first imaging condition is 1 / 1000 second and the shutter speed of the second imaging condition is 1 / 100 second, the correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied belongs applies 1 / 1000 / 1 / 100=1 / 10 to the signal data of the second imaging condition as a second correction process, thereby reducing the difference between the signal data due to the difference in imaging conditions.

[0330] (Example 3) The correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied belongs performs the second correction process in which, for example, when only the frame rate differs between the first imaging condition and the second imaging condition (the charge accumulation time is the same), the frame rate of the first imaging condition is 30 fps, and the frame rate of the second imaging condition is 60 fps, the signal data of the frame image acquired under the first imaging condition (30 fps) and the acquisition start timing of which are close to each other, are adopted for the signal data under the second imaging condition (60 fps). This reduces the difference between the signal data due to the difference in the imaging conditions. The second correction process may involve interpolating signal data of a frame image whose acquisition start timing is close to that of a frame image acquired under the first imaging condition (30 fps) based on multiple adjacent frame images acquired under the second imaging condition (60 fps).

[0331] As described above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions. In the above example, the second correction process is performed on the signal data under the second imaging condition among the signal data. However, the second correction process may be performed on the signal data under the first imaging condition among the signal data.

[0332] Furthermore, a second correction process may be performed on the signal data under the first imaging condition and the signal data under the second imaging condition, respectively, to reduce the difference between the two pieces of signal data after the second correction process.

[0333] FIG. 25 is a diagram showing a schematic diagram of processing of the first signal data and the second signal data relating to the focus detection processing.

[0334] First signal data captured under a first imaging condition is output from each pixel included in the first imaging region 141, and second signal data captured under a second imaging condition is output from each pixel included in the second imaging region 142. The first signal data from the first imaging region 141 is output to a first processing unit 151. Similarly, the second signal data from the second imaging region 142 is output to a second processing unit 152.

[0335] The first processing unit 151 performs the first correction process and the second correction process, or the first correction process or the second correction process, on the first image data as necessary. The second processing unit 152 performs the first correction process and the second correction process, or the first correction process or the second correction process, on the second image data as necessary.

[0336] In the above-mentioned first correction process, for example, when the block of interest is included in the first imaging region 141, the first correction process described above, i.e., replacement process, is performed by the first processing unit 151 as shown in Fig. 25. As a result, the first signal data in the block of interest in which whiteout or blackout has occurred is replaced by the second signal data from the reference block included in the second imaging region 142. For this purpose, the first processing unit 151 receives the second signal data from the reference block as information 182 from the second processing unit 152, for example.

[0337] In the above-mentioned second correction process, when the second correction process is performed on the signal data of the second imaging condition among the signal data to reduce the difference between the signal data after the second correction process and the signal data of the first imaging condition, the second processing unit 152 performs the process. The second processing unit 152 performs the above-mentioned second correction process on the second signal data from the pixels included in the second imaging region 142. The second processing unit 152 receives information 181 about the first imaging condition required to reduce the difference between the signal data due to the difference in the imaging condition from, for example, the first processing unit 151. In addition, when performing the second correction processing on the signal data of the second imaging condition among the signal data to reduce the difference between the signal data after the second correction processing and the signal data of the first imaging condition, the first processing unit 151 does not perform the second correction processing on the first signal data.

[0338] Furthermore, when performing a second correction process on the signal data under the first imaging condition among the signal data to reduce the difference between the signal data after the second correction process and the signal data under the first imaging condition, the first processing unit 151 performs the process. The first processing unit 151 performs the above-mentioned second correction process on the first signal data from the pixels included in the first imaging region 141. The first processing unit 151 receives information on the second imaging condition required to reduce the difference between the signal data due to the difference in the imaging condition from the second processing unit 152. In addition, when performing the second correction processing on the signal data of the first imaging condition among the signal data, in order to reduce the difference between the signal data after the second correction processing and the signal data of the first imaging condition, the second processing unit 152 does not perform the second correction processing on the second signal data.

[0339] Furthermore, when performing a second correction process on the signal data under the first imaging condition and the signal data under the second imaging condition among the signal data to reduce the difference between both the signal data after the second correction process, the first processing unit 151 and the second processing unit 152 perform the process. The first processing unit 151 performs the above-mentioned second correction process on the first signal data from the pixels included in the first imaging region 141, and the second processing unit 152 performs the above-mentioned second correction process on the second signal data from the pixels included in the second imaging region 142.

[0340] The lens movement control unit 34d performs focus detection processing based on signal data from the first processing unit 151 and the second processing unit 152, and outputs a drive signal for moving the focus lens of the imaging optical system 31 to a focus position based on the calculation results.

[0341] 3. When performing subject detection processing When different imaging conditions are set between the divided areas and the search range 190 includes the boundaries of the divided areas, in this embodiment, the object detection unit 34a of the control unit 34 causes the correction unit 322 to perform a second correction process on the image data of at least one area within the search range 190, as described in 3-2 below.

[0342] 3-1. When image data in the search range 190 in FIG. 16 does not include a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied In this case, the correction unit 322 does not perform the second correction process, and the object detection unit 34a of the control unit 34 performs the subject detection process using the image data constituting the search range 190 as is.

[0343] 3-2. When image data in the search range 190 in FIG. 16 includes image data to which the first imaging condition is applied and image data to which the second imaging condition is applied In this case, the object detection unit 34a of the control unit 34 causes the correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied belongs, in the image of the search range 190, to perform the second correction process as in (Example 1) to (Example 3) described above for the case of performing focus detection processing. Then, the object detection unit 34a of the control unit 34 performs the subject detection process using the image data of the pixel to which the first condition is applied and the image data after the second correction process.

[0344] FIG. 26 is a diagram showing a schematic diagram of processing of the first image data and the second image data in relation to the subject detection processing. In the above-mentioned first correction process, for example, when the block of interest is included in the first imaging region 141, the first correction process described above, i.e., replacement process, is performed by the first processing unit 151 as shown in Fig. 26. As a result, the first image data in the block of interest in which whiteout or blackout has occurred is replaced with the second image data from the reference block included in the second imaging region 142. For this purpose, the first processing unit 151 receives the second image data from the reference block as information 182 from the second processing unit 152, for example.

[0345] In the second correction processing, the second correction processing performed by the first processing unit 151 and / or the second processing unit 152 is the same as the second correction processing in FIG. 25 described above in relation to the case where focus detection processing is performed. The object detection unit 34a performs processing to detect subject elements based on the image data from the first processing unit 151 and the second processing unit 152, and outputs the detection result.

[0346] 4. Setting imaging conditions A case will be described in which the area of ​​the image capture screen is divided, different image capture conditions are set between the divided areas, and exposure conditions are determined by newly measuring the light.

[0347] 4-1. When image data in the photometric range does not include a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied In this case, the correction unit 322 does not perform the second correction process, and the setting unit 34b of the control unit 34 performs exposure calculation processing using the image data that constitutes the photometric range as is.

[0348] 4-2. When image data in the photometric range includes a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied In this case, the setting unit 34b of the control unit 34 causes the correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied belongs, among the image data of the photometric range, to perform the second correction process as in (Example 1) to (Example 3) described above for the case of performing focus detection process. Then, the setting unit 34b of the control unit 34 performs exposure calculation process using the image data after the second correction process.

[0349] FIG. 27 is a diagram showing a schematic representation of processing of the first image data and the second image data in relation to setting of imaging conditions such as exposure calculation processing. In the above-mentioned first correction process, for example, when the block of interest is included in the first imaging region 141, the first correction process described above, i.e., replacement process, is performed by the first processing unit 151 as shown in Fig. 27. As a result, the first image data in the block of interest in which whiteout or blackout has occurred is replaced with the second image data from the reference block included in the second imaging region 142. For this purpose, the first processing unit 151 receives the second image data from the reference block as information 182 from the second processing unit 152, for example.

[0350] In the second correction processing, the second correction processing performed by the first processing unit 151 and / or the second processing unit 152 is the same as the second correction processing in FIG. 25 described above in relation to the case where focus detection processing is performed. The setting unit 34b performs calculation processing of the imaging conditions such as exposure calculation processing based on the image data from the first processing unit 151 and the second processing unit 152, and based on the calculation results, divides the image captured by the imaging unit 32 into multiple areas including the detected subject elements, and resets the imaging conditions for the multiple areas.

[0351] According to the second embodiment described above, the following advantageous effects can be obtained. (1) The image processing unit 32c has a first processing unit 151 that generates image data of a subject imaged in the first imaging region 141, and a second processing unit 152 that generates image data of a subject imaged in the second imaging region 142. The first processing unit 151 generates image data of a subject imaged in the first imaging region 141 from the image data of the subject imaged in the second imaging region 142. This allows pre-processing (first correction processing, second correction processing) of the image data to be performed in parallel by multiple correction units 322, thereby reducing the processing load on the correction units 322.

[0352] (2) The image processing unit 32c has a first processing unit 151 that generates a signal based on the subject incident on the first imaging region 141, and a second processing unit 152 that generates a signal based on the subject incident on the second imaging region 142. The first processing unit 151 generates a signal based on the subject imaged in the first imaging region 141 from a signal based on the subject incident on the second imaging region 142. This allows the pre-processing of image data to be performed in parallel by the multiple correction units 322, thereby reducing the processing load on the correction unit 322 and, since the pre-processing by the multiple correction units 322 is performed in a short time by parallel processing, the time until the start of focus detection processing in the lens movement control unit 34d can be shortened, which contributes to speeding up the focus detection processing.

[0353] (3) The image processing unit 32c has a first processing unit 151 that generates image data of the subject incident on the first imaging region 141, and a second processing unit 152 that generates image data of the subject incident on the second imaging region 142. The first processing unit 151 generates image data of the subject imaged in the first imaging region 141 from the image data of the subject incident on the second imaging region 142. This allows the pre-processing of the image data to be performed in parallel by the multiple correction units 322, thereby reducing the processing load on the correction units 322 and also allowing the pre-processing by the multiple correction units 322 to be performed in a short time by parallel processing, thereby shortening the time until the subject detection process starts in the object detection unit 34a, which contributes to speeding up the subject detection process.

[0354] (4) The image processing unit 32c has a first processing unit 151 that generates a signal based on the subject incident on the first imaging region 141, and a second processing unit 152 that generates a signal based on the subject incident on the second imaging region 142. The first processing unit 151 generates a signal based on the subject imaged in the first imaging region 141 from a signal based on the subject incident on the second imaging region 142. This allows the pre-processing of image data to be performed in parallel by the multiple correction units 322, thereby reducing the processing load on the correction unit 322 and, since the pre-processing by the multiple correction units 322 is performed in a short time by parallel processing, the time until the setting unit 34b starts setting the imaging conditions can be shortened, which contributes to speeding up the setting process of the imaging conditions.

[0355] ---Modification of the second embodiment--- The following modifications are also within the scope of the present invention, and one or more of the modifications may be combined with the above-described embodiment. (Variation 13) We will explain the processing of the first image data and the second image data when the first imaging area and the second imaging area are arranged on the imaging surface of the imaging element 32a as shown in Figures 18(a) to 18(c) in variant example 1 of the first embodiment. 18(a) to 18(c), a first image based on an image signal read out from the first imaging region and a second image based on an image signal read out from the second imaging region are generated by pixel signals read out from the imaging element 32a that has captured one frame. In this modification, similar to modification 1, the control unit 34 uses the first image for display and the second image for detection. The imaging conditions set in the first imaging area for capturing the first image are referred to as the first imaging conditions, and the imaging conditions set in the second imaging area for capturing the second image are referred to as the second imaging conditions. The control unit 34 may make the first imaging conditions and the second imaging conditions different from each other.

[0356] 1. As an example, a case where the first imaging condition set in the first imaging area is the same in the entire first imaging area of ​​the imaging screen, and the second imaging condition set in the second imaging area is the same in the entire second imaging area of ​​the imaging screen will be described with reference to Fig. 28. Fig. 28 is a diagram that illustrates the processing of the first image data and the second image data.

[0357] First image data captured under a first imaging condition is output from each pixel included in the first imaging region 141, and second image data captured under a second imaging condition is output from each pixel included in the second imaging region 142. The first image data from the first imaging region 141 is output to a first processing unit 151. Similarly, the second image data from the second imaging region 142 is output to a second processing unit 152.

[0358] The first processing unit 151 performs the first correction processing and / or the second correction processing on the first image data as necessary. The second processing unit 152 performs the first correction processing and / or the second correction processing on the second image data as necessary.

[0359] In this example, since the first imaging condition is the same throughout the first imaging region of the imaging screen, the first processing unit 151 does not perform the second correction process on the first image data from the reference pixels Pr included in the first imaging region. Also, since the second imaging condition is the same throughout the second imaging region of the imaging screen, the second processing unit 152 does not perform the second correction process on the second image data used in the focus detection process, the subject detection process, and the exposure calculation process. However, the second processing unit 152 performs the second correction process on the second image data used for the interpolation of the first image data, which reduces the difference between the image data due to the difference between the first imaging condition and the second imaging condition. The second processing unit 152 outputs the second image data after the second correction process to the first processing unit 151 as indicated by the arrow 182. Note that the second processing unit 152 may output the second image data after the second correction process to the generation unit 323 as indicated by the dashed arrow 183. The second processing unit 152 receives, for example, from the first processing unit 151, information 181 about the first imaging condition required to reduce the difference between image data due to the difference in imaging condition.

[0360] The generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing based on the first image data from the first processing unit 151 and the second image data subjected to the second correction processing by the second processing unit 152, and outputs the image data after the image processing. The object detection unit 34a performs processing to detect subject elements based on the second image data from the second processing unit 152, and outputs the detection result. The setting unit 34b performs calculation processing of the imaging conditions such as exposure calculation processing based on the second image data from the second processing unit 152, and based on the calculation results, divides the image captured by the imaging unit 32 into multiple areas including the detected subject elements, and resets the imaging conditions for the multiple areas. The lens movement control unit 34d performs focus detection processing based on the second signal data from the second processing unit 152, and outputs a drive signal for moving the focus lens of the imaging optical system 31 to a focus position based on the calculation result.

[0361] 2. As another example, a case in which the first imaging condition set in the first imaging area varies depending on the area of ​​the imaging screen, and the second imaging condition set in the second imaging area is the same throughout the second imaging area of ​​the imaging screen will be described with reference to FIG. 28.

[0362] Each pixel included in the first imaging region 141 outputs first image data captured under first imaging conditions that differ depending on the region of the imaging screen, and each pixel included in the second imaging region 142 outputs second image data captured under the same second imaging conditions throughout the second imaging region of the imaging screen. The first image data from the first imaging region 141 is output to a first processing unit 151. Similarly, the second image data from the second imaging region 142 is output to a second processing unit 152.

[0363] The first processing unit 151 performs the first correction processing and / or the second correction processing on the first image data as necessary. The second processing unit 152 performs the first correction processing and / or the second correction processing on the second image data as necessary.

[0364] As described above, in this example, the first imaging condition set in the first imaging region 141 differs depending on the region of the imaging screen. That is, the first imaging condition differs depending on the partial region in the first imaging region 141. When different first imaging conditions are set for the pixel of interest P and the reference pixel Pr, both located in the first imaging region 141, the first processing unit 151 performs a second correction process similar to the second correction process described in 1-2 above on the first image data from the reference pixel Pr. Note that when the same first imaging condition is set for the pixel of interest P and the reference pixel Pr, the first processing unit 151 does not perform the second correction process on the first image data from the reference pixel Pr.

[0365] In this example, since the second imaging conditions set in the second imaging region 142 are the same throughout the entire second imaging region of the imaging screen, the second processing unit 152 does not perform the second correction process on the second image data used in the focus detection process, the subject detection process, and the exposure calculation process. For the second image data used to interpolate the first image data, the second processing unit 152 performs the second correction process to reduce the difference between the image data due to the difference between the imaging conditions for the target pixel P included in the first imaging region 141 and the second imaging conditions. The second processing unit 152 outputs the second image data after the second correction process to the first processing unit 151 (arrow 182). Note that the second processing unit 152 may output the second image data after the second correction process to the generation unit 323 (arrow 183). The second processing unit 152 receives, for example, from the first processing unit 151, information 181 about the imaging conditions for the pixel of interest P included in the first imaging area 141, which is necessary to reduce the difference between image data due to differences in imaging conditions.

[0366] The generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing based on the first image data from the first processing unit 151 and the second image data subjected to the second correction processing by the second processing unit 152, and outputs the image data after the image processing. The object detection unit 34a performs processing to detect subject elements based on the second image data from the second processing unit 152, and outputs the detection result. The setting unit 34b performs calculation processing of the imaging conditions such as exposure calculation processing based on the second image data from the second processing unit 152, and based on the calculation results, divides the image captured by the imaging unit 32 into multiple areas including the detected subject elements, and resets the imaging conditions for the multiple areas. The lens movement control unit 34d performs focus detection processing based on the second signal data from the second processing unit 152, and outputs a drive signal for moving the focus lens of the imaging optical system 31 to a focus position based on the calculation result.

[0367] 3.As another example, a case in which the first imaging condition set in the first imaging area 141 is the same throughout the first imaging area 141 of the imaging screen, and the second imaging condition set in the second imaging area 142 differs depending on the area of ​​the imaging screen, will be described with reference to FIG. 28.

[0368] Each pixel included in the first imaging region 141 outputs first image data captured under the same first imaging condition throughout the first imaging region 141 of the imaging screen, and each pixel included in the second imaging region 142 outputs second image data captured under a fourth imaging condition that differs depending on the region of the imaging screen. The first image data from the first imaging region 141 is output to a first processing unit 151. Similarly, the second image data from the second imaging region 142 is output to a second processing unit 152.

[0369] The first processing unit 151 performs the first correction processing and / or the second correction processing on the first image data as necessary. The second processing unit 152 performs the first correction processing and / or the second correction processing on the second image data as necessary.

[0370] In this example, since the first imaging condition set in the first imaging area 141 is the same throughout the entire first imaging area 141 of the imaging screen, the first processing unit 151 does not perform the second correction processing on the first image data from the reference pixel Pr included in the first imaging area 141.

[0371] In this example, since the second imaging conditions set in the second imaging region 142 differ depending on the region of the imaging screen, the second processing unit 152 performs the second correction process on the second image data as follows. For example, the second processing unit 152 performs the second correction process on the second image data captured under a certain imaging condition among the second image data, thereby reducing the difference between the second image data after the second correction process and the second image data captured under another imaging condition different from the above-mentioned certain imaging condition.

[0372] In this example, the second processing unit 152 performs a second correction process on the second image data used for interpolating the first image data, which reduces a difference between the image data due to a difference between the imaging conditions for the pixel of interest P included in the first imaging region 141 and the second imaging conditions. The second processing unit 152 outputs the second image data after the second correction process to the first processing unit 151 (arrow 182). Note that the second processing unit 152 may output the second image data after the second correction process to the generation unit 323 (arrow 183). The second processing unit 152 receives, for example, from the first processing unit 151, information 181 on the imaging conditions for the pixel of interest P included in the first region, which is necessary to reduce the difference between image data due to differences in imaging conditions.

[0373] The generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing based on the first image data from the first processing unit 151 and the second image data subjected to the second correction processing by the second processing unit 152, and outputs the image data after the image processing. The object detection unit 34a performs a process of detecting subject elements based on second image data captured under certain imaging conditions and second image data captured under other imaging conditions, which have been subjected to a second correction process by the second processing unit 152, and outputs the detection result. The setting unit 34b performs calculation processing of the imaging conditions such as exposure calculation processing based on the second image data captured under certain imaging conditions and the second image data captured under other imaging conditions, which have been subjected to the second correction processing by the second processing unit 152. Based on the calculation results, the setting unit 34b divides the image captured by the imaging unit 32 into a plurality of areas including the detected subject elements, and resets the imaging conditions for the plurality of areas. The lens movement control unit 34d performs focus detection processing based on the second signal data captured under certain imaging conditions and the second signal data captured under other imaging conditions, which have been subjected to the second correction processing by the second processing unit 152.The lens movement control unit 34d outputs a drive signal for moving the focus lens of the imaging optical system 31 to a focus position based on the calculation result.

[0374] 4.Furthermore, as another example, a case in which the first imaging condition set in the first imaging area 141 differs depending on the area of ​​the imaging screen, and the second imaging condition set in the second imaging area 142 differs depending on the area of ​​the imaging screen will be described with reference to FIG. 28.

[0375] Each pixel included in the first imaging region 141 outputs first image data captured under a first imaging condition that differs depending on the region of the imaging screen, and each pixel included in the second imaging region 142 outputs second image data captured under a second imaging condition that differs depending on the region of the imaging screen. The first image data from the first imaging region 141 is output to a first processing unit 151. Similarly, the second image data from the second imaging region 142 is output to a second processing unit 152.

[0376] The first processing unit 151 performs the first correction processing and / or the second correction processing on the first image data as necessary. The second processing unit 152 performs the first correction processing and / or the second correction processing on the second image data as necessary.

[0377] As described above, in this example, the first imaging condition set in the first imaging region 141 differs depending on the region of the imaging screen. That is, the first imaging condition differs depending on the partial region in the first imaging region 141. When different first imaging conditions are set for the pixel of interest P and the reference pixel Pr, both located in the first imaging region 141, the first processing unit 151 performs a second correction process similar to the second correction process described in 1-2 above on the first image data from the reference pixel Pr. Note that when the same first imaging condition is set for the pixel of interest P and the reference pixel Pr, the first processing unit 151 does not perform the second correction process on the first image data from the reference pixel Pr.

[0378] In addition, in this example, since the second imaging conditions set in the second imaging region 142 differ depending on the region of the imaging screen, the second processing unit 152 performs the second correction processing on the second image data as an example of 3. described above.

[0379] The generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing based on the first image data from the first processing unit 151 and the second image data subjected to the second correction processing by the second processing unit 152, and outputs the image data after the image processing. The object detection unit 34a performs a process of detecting subject elements based on second image data captured under certain imaging conditions and second image data captured under other imaging conditions, which have been subjected to a second correction process by the second processing unit 152, and outputs the detection result. The setting unit 34b performs calculation processing of the imaging conditions such as exposure calculation processing based on the second image data captured under certain imaging conditions and the second image data captured under other imaging conditions, which have been subjected to the second correction processing by the second processing unit 152. Based on the calculation results, the setting unit 34b divides the image captured by the imaging unit 32 into a plurality of areas including the detected subject elements, and resets the imaging conditions for the plurality of areas. The lens movement control unit 34d performs focus detection processing based on the second signal data captured under certain imaging conditions and the second signal data captured under other imaging conditions, which have been subjected to the second correction processing by the second processing unit 152.The lens movement control unit 34d outputs a drive signal for moving the focus lens of the imaging optical system 31 to a focus position based on the calculation result.

[0380] (Variation 14) In the second embodiment described above, one of the correction units 322 corresponds to one of the blocks 111a (unit divisions). However, one of the correction units 322 may correspond to one of the composite blocks (composite divisions) having a plurality of blocks 111a (unit divisions). In this case, the correction unit 322 sequentially corrects image data from pixels belonging to a plurality of blocks 111a included in the composite block. Even if a plurality of correction units 322 are provided corresponding to each composite block having a plurality of blocks 111a, the second correction process of the image data can be processed in parallel by the plurality of correction units 322, so that the processing load on the correction unit 322 can be reduced, and an appropriate image can be generated in a short time from image data generated in areas with different imaging conditions.

[0381] (Variation 15) In the second embodiment described above, the generation unit 323 is provided inside the imaging unit 32A. However, the generation unit 323 may be provided outside the imaging unit 32A. Even if the generation unit 323 is provided outside the imaging unit 32A, the same effects as those described above can be achieved.

[0382] (Variation 16) In the above-described second embodiment, the stacked imaging element 100A further includes an image processing chip 114 that performs the above-described pre-processing and image processing, in addition to the back-illuminated imaging chip 111, the signal processing chip 112, and the memory chip 113. However, the stacked imaging element 100A may not include the image processing chip 114, and the image processing unit 32c may be provided in the signal processing chip 112.

[0383] (Variation 17) In the above-described second embodiment, the second processing unit 152 receives information on the first imaging condition required to reduce the difference between the image data due to the difference between the imaging conditions from the first processing unit 151. Also, the first processing unit 151 receives information on the second imaging condition required to reduce the difference between the image data due to the difference between the imaging conditions from the second processing unit 152. However, the second processing unit 152 may receive information on the first imaging condition required to reduce the difference between the image data due to the difference between the imaging conditions from the driving unit 32b or the control unit 34. Similarly, the first processing unit 151 may receive information on the second imaging condition required to reduce the difference between the image data due to the difference between the imaging conditions from the driving unit 32b or the control unit 34.

[0384] The imaging optical system 31 described above may include a zoom lens or a tilt lens. The lens movement control unit 34d adjusts the angle of view of the imaging optical system 31 by moving the zoom lens in the optical axis direction. In other words, the image of the imaging optical system 31 can be adjusted by moving the zoom lens, for example, to obtain an image of a wide range of subjects or to obtain a large image of a distant subject. Furthermore, the lens movement control unit 34d can adjust the distortion of the image caused by the imaging optical system 31 by moving the tilt lens in a direction perpendicular to the optical axis. Then, in order to adjust the state of the image captured by the imaging optical system 31 (for example, the state of the angle of view or the state of image distortion), it is preferable to perform the above-mentioned preprocessing based on the idea that it is preferable to use image data after the above-mentioned preprocessing.

[0385] ---Third embodiment--- 29 and 30, a digital camera will be described as an example of an electronic device equipped with an image processing device according to the third embodiment. In the following description, the same components as in the first embodiment are given the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as in the first embodiment. In this embodiment, a case will be described where the boundary between divided areas exists within a block.

[0386] <First correction process> In the third embodiment, the correction section 33b of the image processing section 33 performs a first correction process as necessary as one of the pre-processing processes performed before the image processing, the focus detection process, the subject detection process (detection of subject elements), and the process of setting the imaging conditions.

[0387] As in the first embodiment, in the third embodiment, after the area of ​​the imaging screen is divided by the setting unit 34b, it is possible to set (change) imaging conditions for the area selected by the user or the area determined by the control unit 34. For example, the divided regions 61 to 66 are respectively designated as first region 61 to sixth region 66 (see FIG. 29(a)), and different first to sixth imaging conditions are set for the first to sixth regions 61 to 66. In such a case, there exists a block including the boundaries between the first to sixth regions 61 to 66. As described above, a block is the smallest unit for which imaging conditions can be set individually in the imaging element 32a.

[0388] Fig. 29(a) is a diagram illustrating a predetermined range 280 including the boundary between a first region 61 corresponding to a person and a fourth region 64 corresponding to a mountain in a live view image 60a. Fig. 20(b) is an enlarged view of the predetermined range 280 in Fig. 29(a). In Fig. 29(b), the predetermined range 280 includes a plurality of blocks 281-289. The white parts in Fig. 29(b) indicate parts corresponding to people. The diagonal line and hatched parts in Fig. 29(b) indicate parts corresponding to mountains. Blocks 282, 285, and 287 include the boundary B1 between the first region 61 and the fourth region 64. That is, in Fig. 29(b), of the parts corresponding to mountains, the parts corresponding to mountains in blocks 82, 85, and 87 where the boundary B1 exists are diagonally lined. In the following description, among blocks in the same subject element (same region), blocks that include a boundary with an adjacent region are called boundary blocks, and blocks that do not include a boundary are called main blocks. That is, in the example of Figure 29(b), blocks 282, 285, and 287 are boundary blocks, and the remaining blocks 281, 283, 284, 286, 288, and 289 are main blocks.

[0389] In this embodiment, the first region 61 corresponding to a person is set to the first imaging condition, and the fourth region 64 corresponding to a mountain is set to the fourth imaging condition. In this embodiment, since a block is the smallest unit for setting an imaging condition, the same imaging condition is set for one block. Therefore, the setting unit 34b sets the imaging condition of the main blocks 281 and 284 that capture a person to the first imaging condition, and also sets the imaging condition of the boundary blocks 282, 285, and 287 that capture a person and a mountain to the first imaging condition. On the other hand, the setting unit 34b sets the imaging condition of the main blocks 283, 286, 288, and 289 that capture a mountain to the fourth imaging condition. In this way, the first imaging condition is set for the boundary blocks 282, 285, and 287 including the boundary B1 between the first region 61 and the fourth region 64, and therefore the first imaging condition is also set for the shaded portions of these boundary blocks 282, 285, and 287, i.e., the portions corresponding to the mountains. In other words, an imaging condition different from the fourth imaging condition set for the main blocks 283, 286, 288, and 289 capturing images of the mountains is set for the shaded portions in the boundary blocks 282, 285, and 287.

[0390] As described above, when the hatched parts of the boundary blocks 282, 285, and 287 in the fourth region 64 are set to the first imaging condition and the shaded main blocks 283, 286, 288, and 289 are set to the fourth imaging condition, there may be differences in brightness, contrast, color, etc. between the hatched parts and the shaded parts of the image. In other words, even in the same fourth region 64, the imaging conditions are different between the boundary blocks and the main blocks, so there is a risk of discontinuity occurring in the image after image processing. Therefore, in this embodiment, the following correction process is performed on the signals from the pixels belonging to the boundary block to obtain signals similar to those captured under the same imaging conditions as those in the main block. This correction process is called the first correction process. The first correction process is performed to reduce discontinuity that occurs in the image after image processing due to the presence of parts with different imaging conditions within the same region.

[0391] Fig. 30 is a diagram illustrating image data corresponding to Fig. 29(b). In Fig. 30, each of blocks 281-289 is assumed to be composed of four pixels, ie, 2 pixels x 2 pixels. In Fig. 30, boundary B1 is indicated by a thick dashed line. Of the pixels shown in Fig. 30, subject light from person 61a (see Fig. 5) is assumed to be incident on blanked pixels 281a-281d, 282a-282c, 284a-284d, 285a, and 287a, and subject light from mountain 64a (see Fig. 5) is assumed to be incident on shaded pixels 282d, 283a-283d, 285b-285d, 286a-286d, 287b-287d, 288a-288d, and 289a-289d.

[0392] The correction unit 33b determines from which subject element light is incident on each pixel belonging to the boundary block. Specifically, the correction unit 33b calculates the position of the boundary on the imaging surface of the image sensor 32a from the detection result of the subject element by the object detection unit 34a. The correction unit 33b then extracts a boundary block based on the calculated boundary position, and calculates from which subject element subject light is incident on each pixel belonging to the extracted boundary block.

[0393] 29 and 30 as an example, correction unit 33b calculates the position of boundary B1 between first region 61 and fourth region 64 from the detection result of subject elements by object detection unit 34a. Then, correction unit 33b extracts blocks 282, 285, and 287 as boundary blocks based on the calculated position of boundary B1. Then, correction unit 33b calculates that subject light from person 61a is incident on pixels 282a, 282b, and 282c of boundary block 282, pixel 285a of boundary block 285, and pixel 287a of boundary block 287, based on the calculated position of boundary B1. In addition, based on the calculated position of boundary B1, correction unit 33b calculates that subject light from mountain 64a is incident on pixel 282d in boundary block 282, pixels 285b, 285c, and 285d in boundary block 285, and pixels 287b, 287c, and 287d in boundary block 287.

[0394] As described above, the main blocks 283, 286, 288, and 289 in the fourth region 64 capture the subject light from the mountain 64a under the fourth imaging condition. However, since the boundary blocks 282, 285, and 287 are set to the first imaging condition, the pixel 282d in the boundary block 282, the pixels 285b to 285d in the boundary block 285, and the pixels 287b to 287d in the boundary block 287 capture the subject light from the mountain 64a under the first imaging condition. Therefore, the correction unit 33b performs a first correction process on each signal from pixel 282d in boundary block 282, pixels 285b-285d in boundary block 285, and pixels 287b-287d in boundary block 287 so as to obtain a signal similar to that obtained when the subject light from mountain 64a is captured under the fourth imaging condition. In this embodiment, the correction process is performed under the same condition on multiple pixels that are the subject of the correction process within the boundary block. In the following description, boundary block 285 is taken as an example, but the same correction process as described below is also performed on other boundary blocks as necessary.

[0395] (Example of the first correction process) 1. The first correction process is performed based on the imaging conditions. As one aspect of the first correction process, as described below, the correction process is performed based on the difference in the imaging conditions set for the two regions (the first region 61 and the fourth region 64) related to the boundary block. (Example 1) For example, if only the ISO sensitivity differs between the fourth imaging condition and the first imaging condition, and the ISO sensitivity of the fourth imaging condition is 800 and the ISO sensitivity of the first imaging condition is 100, the correction unit 33b multiplies the signals from each pixel 285b to 285d of the boundary block 285 by 800 / 100=8 as a first correction process. In this case, the correction coefficient α by which the pixel value of the signal from each of the pixels 285b to 285d is multiplied is 8, which is calculated as described above based on the difference between the imaging conditions set for each of the two adjacent regions. The signals of the pixels 285b to 285d of the boundary block 285 corrected by this first correction process become the same as the signals when the light incident on each of the pixels 285b to 285d is captured under the fourth imaging condition (ISO sensitivity 800).

[0396] (Example 2) For example, if only the shutter speed differs between the fourth imaging condition and the first imaging condition, and the shutter speed of the fourth imaging condition is 1 / 100 seconds and the shutter speed of the first imaging condition is 1 / 1000 seconds, the correction unit 33b multiplies the signals from each pixel 285b to 285d of the boundary block 285 by (1 / 100) / (1 / 1000)=10 as a first correction process. In this case, the correction coefficient α by which the pixel value of the signal from each of the pixels 285b to 285d is multiplied is 10, which is calculated as described above based on the difference between the imaging conditions set for each of the two adjacent regions. The signals of the pixels 285b to 285d of the boundary block 285 corrected by this first correction process become the same as the signals when the light incident on each of the pixels 285b to 285d is captured under the fourth imaging condition (shutter speed is 1 / 100 seconds).

[0397] (Example 3) For example, when only the frame rate differs between the fourth imaging condition and the first imaging condition, the frame rate of the fourth imaging condition is 60 fps and the frame rate of the first imaging condition is 30 fps, the correction unit 33b converts the frame rate from 30 fps to 60 fps by adopting the signal of the frame image whose acquisition start timing is close to the frame image acquired under the fourth imaging condition (60 fps) for the signals of the multiple frame images having a frame rate of 30 fps from each pixel 285b-285d of the boundary block 285. This frame rate conversion is achieved, for example, by storing the signals from each pixel 285b-285d of the boundary block 285 in a memory and reading out the same signal twice from the memory. Note that the conversion from a frame rate of 30 fps to a frame rate of 60 fps using the first correction process can also be achieved by adding together the signals of multiple successive frame images acquired at a frame rate of 30 fps to generate a composite signal, and then supplementing the signal of the frame image with a frame rate of 30 fps with this composite signal.

[0398] In this way, the correction unit 33b performs the first correction process on the signals from each pixel of all the boundary blocks as necessary. That is, the correction unit 33b performs the first correction process on the signals from a pixel belonging to a boundary block if the imaging conditions applied to the main block for the same object element as the pixel in question and the imaging conditions applied to the boundary block are different. However, if the imaging conditions applied to the main block for the same object element as the pixel in question and the imaging conditions applied to the boundary block are the same, there is no need to perform the first correction process, so the correction unit 33b does not perform the first correction process. As described above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions. In addition, if the discontinuity that occurs in the image can be alleviated by performing image processing such as reducing sharpness or contrast on image data obtained by capturing an image in a boundary block or a main block adjacent to the boundary block, it is not necessary to perform the first correction process.

[0399] 2. The first correction process is performed based on the pixel value. As another aspect of the first correction process, as described below, the correction process is performed based on the pixel values ​​of the signals output from the pixels of the boundary block and the pixel values ​​of the signals output from the pixels of the main block. (Example 1) The correction unit 33b calculates a correction coefficient α based on the average value of pixel values ​​of signals from each pixel of a boundary block where the subject light from the mountain 64a is captured under the first imaging condition, and the average value of pixel values ​​of signals from each pixel of a main block adjacent to the boundary block where the subject light from the mountain 64a is captured under the fourth imaging condition. Then, the correction unit 33b multiplies each of the pixel values ​​of signals from each pixel of the boundary block where the subject light from the mountain 64a is captured by the correction coefficient α.

[0400] For example, assume that the average value of pixel values ​​of signals from pixels 285b-285d of boundary block 285 capturing an image of subject light from mountain 64a is Vave1, and the average value of pixel values ​​of signals from pixels 286a-286d of main block 286 adjacent to boundary block 285 is Vave2. In this case, correction unit 33b calculates a correction coefficient α for the pixel values ​​of signals from pixels 285b-285d of boundary block 285 as follows: α=Vave2 / Vave1. Then, correction unit 33b sets the value obtained by multiplying the pixel value of the signal from pixel 285b in boundary block 285 by correction coefficient α as the pixel value of the signal from pixel 285b after the first correction processing. Similarly, correction unit 33b sets the value obtained by multiplying the pixel value of the signal from pixel 285c in boundary block 285 by correction coefficient α as the pixel value of the signal from pixel 285c after the first correction processing, and sets the value obtained by multiplying the pixel value of the signal from pixel 285d by correction coefficient α as the pixel value of the signal from pixel 285d after the first correction processing. As described above, the difference between the pixel values ​​of the signals from the pixels 285b-285d of the boundary block 285 and the pixel values ​​of the signals from each pixel of the main block adjacent to the boundary block is mainly due to the difference between the first and fourth imaging conditions. Therefore, by determining the correction coefficient α based on the difference in pixel values ​​and correcting the signals of the pixels 285b-285d with the correction coefficient α, the signals of the pixels 285b-285d become substantially the same as the signals when light incident on each of the pixels 285b-285d is captured under the fourth imaging condition.

[0401] In this manner, the correction unit 33b performs the first correction process on the signals from each pixel of all the boundary blocks as necessary.

[0402] In the above description, the first correction process is performed in two different modes. The control unit 34 may determine which of these modes to perform the first correction process based on the setting state (including the operation menu setting) of the operation member 36, for example. Note that the control unit 34 may determine which mode of the first correction process to perform depending on the imaging scene mode set in the camera 1 and the type of the detected subject element.

[0403] <Second correction process> The correction unit 33b of the image processing unit 33 further performs the same correction processing as the correction processing in the first embodiment as necessary before the image processing, the focus detection processing, the subject detection (detection of subject elements) processing, and the processing for setting the imaging conditions. In the following description, the correction processing that is the same as the correction processing in the first embodiment is also referred to as the second correction processing. The correction unit 33b performs the second correction processing after the first correction processing that is performed as necessary as described above. In the second correction process, the signal from the pixel of the boundary block corrected by the first correction process is processed as a signal obtained by imaging under the imaging conditions set for the main block, not under the imaging conditions set for the boundary block. For example, when the second correction process is performed, the signal from the pixel 282d of the boundary block 282 corrected by the first correction process is processed by the correction unit 33b as a signal obtained by imaging under the fourth imaging condition, not under the first imaging condition.

[0404] According to the third embodiment described above, in addition to the effects of the first and second embodiments, the following effects can be obtained. (1) It is possible to appropriately process each area with different imaging conditions. That is, it is possible to appropriately generate an image based on image data generated in each area. For example, it is possible to suppress discontinuity and discomfort that appear in the generated image due to differences in imaging conditions for each area. Furthermore, since signals from pixels in the boundary blocks can be appropriately corrected, image data can be generated appropriately. In addition, the defocus amount can be appropriately detected based on the focus detection signal data generated in each region. For example, it is possible to prevent the focus detection accuracy from decreasing due to differences in imaging conditions between regions. Furthermore, subject elements can be appropriately detected based on image data generated for each region, and a decrease in detection accuracy due to differences in image capturing conditions for each region can be suppressed. In addition, the imaging conditions can be appropriately set based on the image data generated for each region. For example, it is possible to prevent a decrease in the accuracy of setting the exposure conditions due to differences in the imaging conditions for each region.

[0405] (2) Since signals from pixels in boundary blocks can be appropriately corrected, image data can be generated appropriately. Moreover, the defocus amount can be appropriately detected based on the signal data for focus detection generated in each area. In addition, subject elements can be appropriately detected based on the image data generated in each region, and imaging conditions can be appropriately set based on the image data generated in each region.

[0406] (3) A portion of the image data for the first region 61 is obtained by capturing an image of light incident on a portion of the first region 61 adjacent to the fourth region 64. The correction unit 33b of the image processing unit 33 corrects the portion of the image data for the first region 61 under the fourth imaging condition, so that signals from pixels in the boundary blocks can be appropriately corrected and image data can be appropriately generated. Moreover, the defocus amount can be appropriately detected based on the signal data for focus detection generated in each area. In addition, subject elements can be appropriately detected based on the image data generated in each region, and imaging conditions can be appropriately set based on the image data generated in each region.

[0407] (4) The correction unit 33b of the image processing unit 33 corrects a part of the image data for the first region 61 under the fourth imaging condition, so that the signals from the pixels of the boundary blocks can be appropriately corrected and image data can be appropriately generated. Moreover, the defocus amount can be appropriately detected based on the signal data for focus detection generated in each area. In addition, subject elements can be appropriately detected based on the image data generated in each region, and imaging conditions can be appropriately set based on the image data generated in each region.

[0408] (5) Of the pixels in the boundary block, signals from pixels on which light from the mountain 64a is incident can be appropriately corrected, and image data can be appropriately generated. Moreover, the defocus amount can be appropriately detected based on the signal data for focus detection generated in each area. In addition, subject elements can be appropriately detected based on the image data generated in each region, and imaging conditions can be appropriately set based on the image data generated in each region.

[0409] ---Modification of the third embodiment--- In the above-described third embodiment, when the first correction process is performed on the boundary blocks relating to the first region 61 and the fourth region 64 based on the imaging conditions, the ratio between the first and fourth imaging conditions is set as the correction coefficient α. However, in this modification, even when the first correction process is performed on the boundary blocks relating to the first region 61 and the fourth region 64 based on the imaging conditions, the correction coefficient α is set to a value closer to 1 than the ratio between the first and fourth imaging conditions. (Example 1) For example, if only the ISO sensitivity differs between the fourth imaging condition and the first imaging condition, with the ISO sensitivity of the fourth imaging condition being 800 and the ISO sensitivity of the first imaging condition being 100, the correction unit 33b adopts, as the value of the correction coefficient α, a value closer to 1 than the ratio between the first imaging condition and the fourth imaging condition (800 / 100=8), such as a value of 7 or 6.5. Note that the values ​​of the correction coefficient α given here are merely examples, and the correction coefficient α is not limited to these values.

[0410] (Example 2) For example, when only the shutter speed differs between the fourth imaging condition and the first imaging condition, the shutter speed of the fourth imaging condition is 1 / 100 seconds and the shutter speed of the first imaging condition is 1 / 1000 seconds, the correction unit 33b adopts, as the value of the correction coefficient α, a value closer to 1 than the ratio between the first imaging condition and the fourth imaging condition, (1 / 100) / (1 / 1000)=10, such as 8.5 or 7. Note that the values ​​of the correction coefficient α given here are merely examples, and the correction coefficient α is not limited to these values.

[0411] That is, by correcting the signal value of each pixel 285b-285d of the boundary block 285 to which the first imaging condition is set, the difference between the signal value after correction and the signal value of the pixel of the block to which the fourth imaging condition is set should be smaller (smoothed) than the difference between the signal value before correction and the signal value of the pixel of the block to which the fourth imaging condition is set. In this way, if it is possible to mitigate discontinuity that occurs in an image after image processing due to the presence of parts with different imaging conditions within the same region, the correction coefficient α may be calculated as in this modified example. For example, if the area in which the pixels to be corrected exist is small, or if the difference between the first and fourth imaging conditions is small, the first correction process may be performed as in this modified example.

[0412] ---Fourth embodiment--- 29 and 31, a digital camera will be described as an example of an electronic device equipped with an image processing device according to the fourth embodiment. In the following description, points that are not particularly described are the same as those in the third embodiment. In the third embodiment, the pixel values ​​of signals from multiple pixels that are the subject of correction processing within a boundary block are multiplied by the same correction coefficient α. In contrast, in this embodiment, the pixel values ​​of signals from multiple pixels that are the subject of correction processing within one boundary block are multiplied by correction coefficients that differ depending on their positions.

[0413] <First correction process> (Example of the first correction process) 1. The first correction process is performed based on the imaging conditions. As one aspect of the first correction process of this embodiment, the correction process is performed based on the difference in the imaging conditions set for each of the two regions (first region 61 and fourth region 64) related to the boundary block, as described below. Specifically, even if the pixels are in the same region, if the imaging conditions applied to the pixels in the boundary block differ from the imaging conditions applied to the main block, the correction coefficient α in the third embodiment is weighted according to the position of the pixel in the boundary block. The correction unit 33b calculates the correction coefficient α based on the imaging conditions, similarly to (Example 1) and (Example 2) of the third embodiment. Then, the correction unit 33b changes the calculated correction coefficient α to a different value according to the distance from the boundary of the region. For example, the correction unit 33b changes the value of the correction coefficient α so that the value approaches 1 as the distance from the boundary of the region decreases. In other words, the correction unit 33b changes the value of the correction coefficient α so that the effect of correction decreases as the position of the pixel to be corrected in the boundary block is closer to the boundary of the region. The correction coefficient α after being changed in this way is called a correction coefficient β. The correction unit 33b multiplies the pixel value of the signal from the pixel to be corrected by the correction coefficient β.

[0414] (Example 1) For example, if only the ISO sensitivity differs between the fourth imaging condition and the first imaging condition, and the ISO sensitivity of the fourth imaging condition is 800 and the ISO sensitivity of the first imaging condition is 100, the correction unit 33b calculates the correction coefficient α=800 / 100=8, as in the third embodiment. Then, the correction unit 33b changes the value of the correction coefficient α so that the value approaches 1 as the distance from the boundary of the region decreases. For example, the correction unit 33b sets the correction coefficient β for the signal from each pixel 285b to 285d of the boundary block 285 as follows. FIG. 31 is an enlarged view of the boundary block 285, and the center of gravity Gb of the pixel 285b, the center of gravity Gc of the pixel 285c, and the center of gravity Gd of the pixel 285d are indicated by black circles. As shown in FIG. 31, the distances Lb and Lc between the center of gravity positions Gb and Gc of the pixels 285b and 285c and the boundary B1 are smaller than the distance Ld between the center of gravity position Gd of the pixel 285d and the boundary B1. Therefore, the correction unit 33b sets the value of the correction coefficient β for the pixel 285d to, for example, 8, which is the same value as the correction coefficient α, or 6, which is closer to 1 than the correction coefficient α, and sets the value of the correction coefficient β for the pixels 285b and 285c to, for example, 4, which is even closer to 1. Note that the values ​​of the correction coefficient β given here are merely exemplary values ​​and are not limited to these values. The correction unit 33b sets the pixel value of the signal from pixel 285b in the boundary block 285 after the first correction processing to a value obtained by multiplying the pixel value of the signal from pixel 285b in the boundary block 285 by a correction coefficient β=4. Similarly, the correction unit 33b sets the pixel value of the signal from pixel 285c in the boundary block 285 after the first correction processing to a value obtained by multiplying the pixel value of the signal from pixel 285c in the boundary block 285 by a correction coefficient β=4 to a pixel value of the signal from pixel 285c after the first correction processing, and sets the pixel value of the signal from pixel 285d after the first correction processing to a value obtained by multiplying the pixel value of the signal from pixel 285d by a correction coefficient β=6 or 8 to a pixel value of the signal from pixel 285d after the first correction processing. The signals of the pixels 285b to 285d of the boundary block 285 corrected by this first correction process approach the signals obtained when the pixels are imaged under the fourth imaging condition (ISO sensitivity 800) as they move away from the boundary B1.

[0415] (Example 2) For example, if only the shutter speed differs between the fourth imaging condition and the first imaging condition, and the shutter speed in the fourth imaging condition is 1 / 100 second and the shutter speed in the first imaging condition is 1 / 1000 second, the correction coefficient α is calculated to be (1 / 100) / (1 / 1000)=10, as in the third embodiment. Then, the correction unit 33b changes the value of the correction coefficient α so that the value approaches 1 as the distance from the boundary of the region decreases. For example, the correction unit 33b sets the correction coefficient β for the signal from each pixel 285b to 285d of the boundary block 285 as follows. As described above, the distances Lb, Lc between the boundary B1 and the center of gravity positions Gb, Gc of the pixels 285b and 285c are smaller than the distance Ld between the center of gravity position Gd of the pixel 285d and the boundary B1. Therefore, the correction unit 33b sets the value of the correction coefficient β for the pixel 285d to, for example, 10, which is the same value as the correction coefficient α, or 7, which is closer to 1 than the correction coefficient α, and sets the value of the correction coefficient β for the pixels 285b and 285c to, for example, 5, which is even closer to 1. Note that the values ​​of the correction coefficient β given here are merely examples and are not limited to these values. The correction unit 33b multiplies the pixel value of the signal from pixel 285b in the boundary block 285 by a correction coefficient β=5, and sets the result as the pixel value of the signal from pixel 285b after the first correction processing. Similarly, the correction unit 33b multiplies the pixel value of the signal from pixel 285c in the boundary block 285 by a correction coefficient β=5, and sets the pixel value of the signal from pixel 285c after the first correction processing, and sets the pixel value of the signal from pixel 285d by a correction coefficient β=7 or 10, and sets the pixel value of the signal from pixel 285d after the first correction processing. The signals of the pixels 285b to 285d of the boundary block 285 corrected by this first correction process approach the signals obtained when the pixels are imaged under the fourth imaging condition (shutter speed is 1 / 100 seconds) as they move away from the boundary B1.

[0416] In this manner, the correction unit 33b performs the first correction process on the signals from each pixel of all the boundary blocks as necessary.

[0417] 2. The first correction process is performed based on the pixel value. As another aspect of the first correction process of this embodiment, the correction process is performed based on the values ​​(pixel values) of signals output from pixels in the boundary block and the values ​​(pixel values) of signals output from pixels in the main block, as described below. Specifically, even if the pixels are in the same region, if the imaging conditions applied to the pixels in the boundary block differ from the imaging conditions applied to the main block, the correction coefficient α in the third embodiment is weighted according to the position of the pixel in the boundary block. The correction unit 33b calculates the correction coefficient α based on the imaging conditions, similarly to (Example 1) of the third embodiment. Then, the correction unit 33b changes the calculated correction coefficient α to a different value according to the distance from the boundary of the region. For example, the correction unit 33b changes the value of the correction coefficient α so that the value approaches 1 as the distance from the boundary of the region decreases. In other words, the correction unit 33b changes the value of the correction coefficient α so that the effect of correction decreases as the position of the pixel to be corrected in the boundary block is closer to the boundary of the region. The correction coefficient α after being changed in this way is called a correction coefficient β, similarly to the above case. The correction unit 33b multiplies the pixel value of the signal from the pixel to be corrected by the correction coefficient β.

[0418] (Example 1) For example, assume that the average value of pixel values ​​of signals from pixels 285b-285d of boundary block 285 capturing an image of subject light from mountain 64a is Vave1, and the average value of pixel values ​​of signals from pixels 286a-286d of main block 286 adjacent to boundary block 285 is Vave2. In this case, correction unit 33b calculates a correction coefficient α for the pixel values ​​of signals from pixels 285b-285d of boundary block 285 as follows: α=Vave2 / Vave1. Then, correction unit 33b changes the value of correction coefficient α so that the value approaches 1 as the distance from the boundary of the region decreases. For example, the correction unit 33b sets the correction coefficient β for the signal from each pixel 285b to 285d of the boundary block 285 as follows: As described above, the distances Lb, Lc between the boundary B1 and the center of gravity positions Gb, Gc of the pixels 285b and 285c are smaller than the distance Ld between the boundary B1 and the center of gravity position Gd of the pixel 285d. Therefore, the correction unit 33b sets the value of the correction coefficient β for the pixel 285d to, for example, the same value as the correction coefficient α (i.e., Vave2 / Vave1) or closer to 1 than the correction coefficient α, for example, β=β1, and sets the value of the correction coefficient β for the pixels 285b and 285c to even closer to 1 than β1, for example, β=β2. When α (= Vave2 / Vave1) is greater than 1, 1≦β2<β1<α holds; when α is less than 1, α<β1<β2≦1 holds. The correction unit 33b multiplies the pixel value of the signal from pixel 285b in the boundary block 285 by the correction coefficient β=β2 and sets the result as the pixel value of the signal from pixel 285b after the first correction processing. Similarly, the correction unit 33b multiplies the pixel value of the signal from pixel 285c in the boundary block 285 by the correction coefficient β=β2 and sets the pixel value of the signal from pixel 285c after the first correction processing, and multiplies the pixel value of the signal from pixel 285d by the correction coefficient β=β1 or Vave2 / Vave1 and sets the pixel value of the signal from pixel 285d after the first correction processing.

[0419] In this manner, the correction unit 33b performs the first correction process on the signals from each pixel of all the boundary blocks as necessary.

[0420] In the above description, the first correction process is performed in two different modes. The control unit 34 may determine which of these modes to perform the first correction process based on the setting state (including the operation menu setting) of the operation member 36, for example. Note that the control unit 34 may determine which mode of the first correction process to perform depending on the imaging scene mode set in the camera 1 and the type of the detected subject element.

[0421] <Second correction process> The correction unit 33b of the image processing unit 33 further performs the same correction processing as the correction processing in the first embodiment as necessary before the image processing, the focus detection processing, the sub...

Claims

[Claim 1] an imaging unit having a first imaging region that is imaged under a first imaging condition and a second imaging region that is imaged under a second imaging condition different from the first imaging condition; a generation unit that generates an image of the subject captured in the first imaging area based on image data of the subject captured in the second imaging area; An imaging device comprising:

Images