Electronic equipment

Abstract

An electronic device is provided that allows easy setting of imaging conditions for each area. [Solution] An imaging device (10) that is an electronic device includes a control unit (23) that displays a first image of a subject captured in a first region of an imaging element and a second image of the subject captured in a second region of the imaging element on a liquid crystal monitor (24) that serves as a display unit, and an operation unit (26) that accepts an operation to select an area from the first region or the second region for setting imaging conditions. The operation unit accepts an operation on the display unit.

Description

[Technical field]

[0001] The present invention relates to electronic devices. [Background technology]

[0002] An imaging element that is configured with an imaging chip and a signal processing chip has been proposed (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2006-49361 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional techniques have had the problem that it is difficult to set imaging conditions for each region. [Means for solving the problem]

[0005] According to a first aspect of the present invention, an electronic device includes a control unit that causes a display unit to display a first image of a subject captured in a first region of an image sensor and a second image of the subject captured in a second region of the image sensor; Controlling the image sensor and an operation unit that receives a first operation for selecting an area for setting an imaging condition, the operation unit receiving the first operation on the display unit, and the control unit displaying on the display unit a third image used for setting an area selected by the first operation out of the first area and the second area to an imaging condition different from an area not selected by the first operation out of the first area and the second area. The third image has at least one of a function of selecting a setting item for setting the imaging condition of the area selected by the first operation out of the first area and the second area, a function of changing the setting of the imaging condition of the area selected by the first operation out of the first area and the second area, and a function of displaying the setting of the imaging condition of the area selected by the first operation out of the first area and the second area. . [Brief description of the drawings]

[0006] [Figure 1]1 is a block diagram showing a configuration of an electronic device according to a first embodiment. [Diagram 2] 1A is a plan view showing a schematic view of an imaging surface of an imaging element, and FIG. 1B is an enlarged plan view of a partial region of the imaging surface. [Diagram 3] (a) is a schematic diagram showing a state in which a subject image is displayed on a liquid crystal monitor of an information terminal device whose orientation has been determined to be vertical. (b) is a diagram showing an example of a display form in which a main subject region is explicitly displayed by being superimposed on the subject image of the subject, and a partial image is also displayed. (c) is a diagram showing an example of a display form in which an enlarged image of the main subject related to the partial image in (a) is displayed. (d) is a diagram showing an example of a display form in which an enlarged image of the main subject related to the partial image in (a) is displayed. [Figure 4] 1A is a diagram showing an example of a display form in which a person is displayed large on a liquid crystal monitor in an information terminal device whose orientation has been determined to be vertical by an orientation detection unit, and FIG. 1B is a diagram showing an example of a display form in which the outline of a main subject area is displayed on a subject image. [Diagram 5] 1A is a diagram showing an example of a display form when a person displayed on a liquid crystal monitor of an information terminal device whose orientation has been determined to be vertical by an orientation detection unit, and FIG. 1B is a diagram showing an example of a display form in which an enlarged image of a main subject in a subject image is displayed. [Figure 6] (a) is a schematic diagram showing a state in which a subject image is displayed on a liquid crystal monitor of an information terminal device whose orientation has been determined to be horizontal by an orientation detection unit. (b) is a diagram showing an example of a display form in which a main subject region is superimposed and explicitly displayed on the subject image, and a partial image is also displayed. (c) is a diagram showing an example of a display form in which an enlarged image of the main subject related to the partial image in (b) is displayed. (d) is a diagram showing an example of a display form in which an enlarged image of the main subject related to the partial image in (b) is displayed. [Figure 7] 11 is a flowchart of a process for setting imaging conditions for each main subject and capturing a subject image. [Figure 8] FIG. 11 is a block diagram showing a configuration of an information terminal device according to a second embodiment. [Figure 9] 1A is a diagram showing a state where a user touches a person in a subject image displayed on a liquid crystal monitor with a finger, and FIG. 1B is a diagram showing an example of a display form in which a main subject region is explicitly displayed by being superimposed on the subject image, and an enlarged image is also displayed. [Figure 10] (a) is a diagram that shows a schematic diagram of a state in which a user touches a person in a subject image displayed on a liquid crystal monitor with two fingers. (b) is a diagram that explains the operation of a frame displayed on a liquid crystal monitor. (c) is a diagram that explains the operation of a frame displayed on a liquid crystal monitor. (d) is a diagram that explains the operation of a frame displayed on a liquid crystal monitor. (e) is a diagram that explains the operation of a frame displayed on a liquid crystal monitor. [Figure 11] 1A is a diagram illustrating a case where a line following a path traced with a finger is displayed on a liquid crystal monitor, and FIG. 1B is a diagram illustrating a case where a line following a path traced with a finger is displayed on a liquid crystal monitor. [Figure 12] FIG. [Figure 13] FIG. [Figure 14] FIG. [Figure 15] FIG. [Figure 16] FIG. [Figure 17] FIG. [Figure 18] FIG. [Figure 19] FIG. [Figure 20] FIG. 13 is a block diagram showing a configuration of an imaging device according to a fourth embodiment of the present invention. [Figure 21] FIG. 2 is a diagram illustrating a schematic diagram of a subject and a luminance distribution of the subject. [Figure 22](a) is a histogram of luminance for each unit group when there is neither crushed shadows nor blown out highlights. (b) is a histogram of luminance for each unit group when there is crushed shadows but no blown out highlights. (c) is a histogram of luminance for each unit group when there is blown out highlights but not crushed shadows. (d) is a histogram of luminance for each unit group when there are both crushed shadows and blown out highlights. [Diagram 23] FIG. 13 is an explanatory diagram of a seventh modified example. [Figure 24] FIG. 13 is an explanatory diagram of Modification Example 8. [Diagram 25] FIG. 2 is a cross-sectional view of a stacked imaging element. [Figure 26] FIG. 2 is a diagram illustrating a pixel array and blocks of an imaging chip. [Figure 27] FIG. 4 is a circuit diagram corresponding to a unit of the imaging chip. [Figure 28] FIG. 2 is a block diagram showing a functional configuration of an imaging element. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] ---First embodiment --- First, an imaging element 22 mounted on an electronic device (e.g., imaging device 10) according to an embodiment of the present invention will be described. The imaging element 22 is described in Japanese Patent Application No. 2012-139026, previously filed by the applicant of the present application. FIG. 25 is a cross-sectional view of the imaging element 22. The imaging element 22 includes an imaging chip 2113 that outputs pixel signals corresponding to incident light, a signal processing chip 2111 that processes the pixel signals, and a memory chip 2112 that stores the pixel signals. The imaging chip 2113, the signal processing chip 2111, and the memory chip 2112 are stacked and electrically connected to each other by conductive bumps 2109 made of Cu or the like.

[0008] As shown in the figure, the incident light is mainly incident in the Z-axis positive direction indicated by the white arrow. In this embodiment, the surface of the imaging chip 2113 on which the incident light is incident is referred to as the back surface. Also, as shown in the coordinate axes, the left direction on the paper surface perpendicular to the Z-axis is the X-axis positive direction, and the front direction on the paper surface perpendicular to the Z-axis and the X-axis is the Y-axis positive direction. In the following figures, the coordinate axes are displayed so that the orientation of each figure can be understood based on the coordinate axes in FIG. 25.

[0009] An example of the imaging chip 2113 is a CMOS image sensor. Specifically, it is a back-illuminated CMOS image sensor. The PD layer 2106 is arranged on the back side of the wiring layer 2108. The PD layer 2106 has a plurality of photodiodes 2104 arranged two-dimensionally and accumulating charges according to incident light, and transistors 2105 provided corresponding to the photodiodes 2104. That is, the PD layer 2106 has a plurality of photodiodes 2104 having a photoelectric conversion function of converting light into charges, and a plurality of transistors 2105 for reading out signals generated by the charges accumulated in the photodiodes 2104 to the wiring layer 2108.

[0010] A color filter 2102 is provided on the incident side of the PD layer 2106 on which the incident light is incident via a passivation film 2103. The color filters 2102 have a plurality of types that transmit different wavelength regions, and have a specific arrangement corresponding to each of the photodiodes 2104. That is, the color filter 2102 has a plurality of types of filters with different spectral characteristics, and the photodiode 2104 receives the light that has transmitted through the filter. The arrangement of the color filter 2102 will be described later. A set of the color filter 2102, the photodiode 2104, and the transistor 2105 forms one pixel.

[0011] A microlens 2101 is provided corresponding to each pixel on the incident light side of the color filter 2102. The microlens 2101 condenses the incident light toward the corresponding photodiode 2104.

[0012] The wiring layer 2108 has wiring 2107 that transmits pixel signals from the PD layer 2106 to the signal processing chip 2111. The wiring 2107 may be multi-layered, and may include passive elements and active elements.

[0013] A plurality of bumps 2109 are arranged on the surface of the wiring layer 2108. The plurality of bumps 2109 are aligned with a plurality of bumps 2109 provided on the opposing surface of the signal processing chip 2111, and the imaging chip 2113 and the signal processing chip 2111 are pressed, etc., so that the aligned bumps 2109 are joined and electrically connected to each other.

[0014] Similarly, a plurality of bumps 2109 are arranged on the opposing surfaces of the signal processing chip 2111 and the memory chip 2112. These bumps 2109 are aligned with each other, and the signal processing chip 2111 and the memory chip 2112 are pressed together, etc., so that the aligned bumps 2109 are bonded and electrically connected to each other.

[0015] The bonding between the bumps 2109 is not limited to Cu bump bonding by solid-phase diffusion, and micro-bump bonding by solder melting may be adopted. Also, it is sufficient to provide about one bump 2109 for each block described later. Therefore, the size of the bump 2109 may be larger than the pitch of the photodiodes 2104. Also, in a peripheral region other than the pixel region where the pixels are arranged, a bump larger than the bump 2109 corresponding to the pixel region may be provided.

[0016] The signal processing chip 2111 has TSVs (through silicon vias) 2110 that connect the circuits provided on the front and back surfaces of the chip to each other. The TSVs 2110 are preferably provided in the peripheral region. The TSVs 2110 may also be provided in the peripheral region of the imaging chip 2113 and the memory chip 2112.

[0017] FIG. 26 is a diagram for explaining the pixel arrangement of the imaging chip 113. In particular, the imaging chip 2113 is shown as observed from the back side. For example, 8 million or more pixels are arranged in a matrix in the pixel region. In this embodiment, for example, four pixels, 2 pixels by 2 pixels adjacent to each other, form one block 2131. Then, four blocks, 2 blocks by 2 blocks adjacent to each other, form one unit group 32. The grid lines in the figure show the concept of grouping adjacent pixels to form the block 2131 and the unit group 32. The number of pixels forming the block 2131 and the number of blocks 2131 forming the unit group 32 are not limited to the above example, and may be more or less than that.

[0018] As shown in the partially enlarged view of the pixel region, the block 2131 contains four so-called Bayer arrays, arranged vertically and horizontally, each consisting of four pixels: green pixels Gb, Gr, blue pixel B, and red pixel R. The green pixels are pixels having a green filter as the color filter 2102 and receive light in the green wavelength band of the incident light. Similarly, the blue pixels are pixels having a blue filter as the color filter 2102 and receive light in the blue wavelength band, and the red pixels are pixels having a red filter as the color filter 2102 and receive light in the red wavelength band.

[0019] In this embodiment, a plurality of blocks 2131 are defined so that each block 2131 includes at least one of four pixels of Gb, Gr, B, and R. Each block 2131 can control the four pixels in the block 2131 with a control parameter determined for each block 2131. That is, imaging signals with different imaging conditions can be acquired for a pixel group included in a certain block 2131 and a pixel group included in another block 2131. That is, imaging operations can be performed under imaging conditions for a pixel group included in a certain block 2131 that are different from imaging conditions for a pixel group included in another block 2131. Examples of the control parameters include a frame rate, a gain, a thinning rate, the number of summation rows or columns for adding pixel signals, the accumulation time or number of accumulations of electric charges, the number of bits for digitization, and the like. Furthermore, the control parameters may be parameters for image processing after acquisition of image signals from pixels. In other words, the frame rate, gain, thinning rate, number of addition rows or number of addition columns for adding pixel signals, charge accumulation time, number of times charge is accumulated, and number of bits for digitization of a pixel group included in a certain block 2131 can be different from the frame rate, gain, thinning rate, number of addition rows or number of addition columns for adding pixel signals, charge accumulation time, number of times charge is accumulated, and number of bits for digitization of a pixel group included in another block 2131.

[0020] Fig. 27 is a circuit diagram corresponding to one unit group 32 in the imaging chip 2113. In Fig. 27, a rectangle surrounded by a dotted line representatively indicates a circuit corresponding to one pixel. Also, a rectangle surrounded by a dashed line corresponds to one block 2131. Note that at least a part of each transistor described below corresponds to the transistor 2105 in Fig. 25.

[0021] As described above, the unit group 32 is formed of four blocks 2131. The reset transistors 2303 of the pixels included in the unit group 32 are turned on / off in units of blocks 2131. In addition, the transfer transistors 2302 of the pixels included in the unit group 32 are also turned on / off in units of blocks 2131. In the example shown in Fig. 27, a reset wiring 2300-1 is provided for turning on / off the four reset transistors 2303 corresponding to the upper left block 2131-1, and a TX wiring 2307-1 is also provided for supplying a transfer pulse to the four transfer transistors 2302 corresponding to the block 2131-1.

[0022] Similarly, a reset wiring 2300-3 for turning on / off the four reset transistors 2303 corresponding to the lower left block 2131-3 is provided separately from the reset wiring 2300-1. Also, a TX wiring 2307-3 for supplying transfer pulses to the four transfer transistors 2302 corresponding to the same block 2131-3 is provided separately from the TX wiring 2307-1.

[0023] Similarly, in the upper right block 2131-2 and the lower right block 2131-4, a reset line 2300-2 and a TX line 2307-2, and a reset line 2300-4 and a TX line 2307-4 are provided in each block 2131, respectively.

[0024] The 16 photodiodes 2104 corresponding to each pixel are connected to a corresponding transfer transistor 2302. A transfer pulse is supplied to the gate of each transfer transistor 2302 via the TX wiring for each block 2131. The drain of each transfer transistor 2302 is connected to the source of the corresponding reset transistor 2303, and a so-called floating diffusion FD between the drain of the transfer transistor 2302 and the source of the reset transistor 2303 is connected to the gate of the corresponding amplification transistor 2304.

[0025] The drains of the reset transistors 2303 are commonly connected to a Vdd wiring 2310 to which a power supply voltage is supplied. A reset pulse is supplied to the gates of the reset transistors 2303 via the reset wiring for each block 2131.

[0026] The drains of the amplification transistors 2304 are commonly connected to a Vdd wiring 2310 to which a power supply voltage is supplied. The sources of the amplification transistors 2304 are connected to the drains of the corresponding selection transistors 2305. The gates of the selection transistors 2305 are connected to a decoder wiring 2308 to which a selection pulse is supplied. In this embodiment, the decoder wiring 2308 is provided independently for each of the 16 selection transistors 2305. The sources of the selection transistors 2305 are connected to a common output wiring 2309. The load current source 2311 supplies a current to the output wiring 2309. That is, the output wiring 2309 for the selection transistors 2305 is formed by a source follower. The load current source 2311 may be provided on the imaging chip 2113 side or on the signal processing chip 2111 side. When the unit group 32 includes one pixel, the selection transistor 2305 may be omitted.

[0027] Here, the flow from the start of charge accumulation to pixel output after accumulation is completed will be described. When a reset pulse is applied to the reset transistor 2303 through the reset wiring for each block 2131, and at the same time a transfer pulse is applied to the transfer transistor 2302 through the TX wiring for each block 2131, the potentials of the photodiode 2104 and floating diffusion FD are reset for each block 2131.

[0028] When the application of the transfer pulse is released, each photodiode 2104 converts the received light into an electric charge and accumulates it. After that, when the transfer pulse is applied again in a state where the reset pulse is not applied, the electric charge accumulated in the photodiode 2104 is transferred to the floating diffusion FD. The electric potential of the floating diffusion FD changes from the reset electric potential to the signal electric potential after the electric charge accumulation. That is, the electric potential of the floating diffusion FD becomes an electric potential according to the electric charge accumulated in the photodiode 2104. Then, when a selection pulse is applied to the selection transistor 2305 through the decoder wiring 2308, the fluctuation of the signal electric potential of the floating diffusion FD is transmitted to the output wiring 2309 through the amplification transistor 2304 and the selection transistor 2305. That is, when a selection pulse is applied to the selection transistor 2305 through the decoder wiring 2308, a signal generated by the electric potential of the floating diffusion FD is transmitted to the output wiring 2309. As a result, a pixel signal corresponding to the reset electric potential and the signal electric potential is output from the unit pixel to the output wiring 2309.

[0029] As described above, in this embodiment, the reset wiring and the TX wiring are common to the four pixels forming the block 2131. That is, the reset pulse and the transfer pulse are each applied simultaneously to the four pixels in the same block 2131. Therefore, all the pixels forming a certain block 2131 start charge accumulation at the same timing, and end charge accumulation at the same timing. However, pixel signals corresponding to the accumulated charges are selectively output from the output wiring 2309 by sequentially applying a selection pulse to each selection transistor 2305.

[0030] In this manner, in this embodiment, the charge accumulation start timing can be controlled for each block 2131. In other words, imaging can be performed at different timings between different blocks 2131.

[0031] 28 is a block diagram showing a functional configuration of the image sensor 22. An analog multiplexer 2411 sequentially selects the 16 photodiodes 2104 forming a unit group 32, and outputs the respective pixel signals to output wiring 2309 provided corresponding to the unit group 32. The multiplexer 2411 is formed in the image sensor chip 2113 together with the photodiodes 2104.

[0032] The pixel signals output via the multiplexer 2411 undergo correlated double sampling (CDS) and analog-to-digital (A / D) conversion by a signal processing circuit 2412 that performs CDS and A / D conversion and is formed in the signal processing chip 2111. The A / D converted pixel signals are passed to a demultiplexer 2413 and stored in pixel memories 2414 corresponding to the respective pixels. The demultiplexer 2413 and pixel memories 2414 are formed in the memory chip 2112.

[0033] The arithmetic circuit 2415 processes the pixel signals stored in the pixel memory 2414 and passes them to a downstream image processing unit. The arithmetic circuit 2415 may be provided in the signal processing chip 2111 or in the memory chip 2112. Note that although the connection for one unit group 32 is shown in Fig. 28, in reality, these exist for each unit group 32 and operate in parallel. However, it is not necessary for there to be a arithmetic circuit 2415 for each unit group 32. For example, one arithmetic circuit 2415 may process sequentially while referring to values ​​of the pixel memories 2414 corresponding to each unit group 32 in order.

[0034] As described above, output wiring 2309 is provided corresponding to each unit group 32. Since the imaging element 22 has the imaging chip 2113, the signal processing chip 2111, and the memory chip 2112 stacked on top of each other, by using electrical connection between the chips using bumps 2109 for these output wiring 2309, it is possible to route wiring without increasing the size of each chip in the planar direction.

[0035] FIG. 1 is a block diagram showing the configuration of an electronic device according to a first embodiment of the present invention. In this embodiment, an information terminal device 10 having an imaging function will be described as an example of an electronic device. The information terminal device 10 is a device that is thin, has a large liquid crystal monitor on the front side, and has an imaging optical system on the back side, such as a tablet-type information terminal device. The information terminal device 10 includes an imaging optical system 21, an imaging element 22, a control unit 23, a liquid crystal monitor 24, a memory card 25, an operation unit 26, a DRAM 27, a flash memory 28, and a posture detection unit 29. In the following description, of the various functions and configurations of the information terminal device 10, functions and configurations related to imaging will be described, and descriptions of other functions and configurations will be omitted.

[0036] The imaging optical system 21 is composed of a plurality of lenses, and forms a subject image on the imaging surface of the imaging element 22. Note that in Fig. 1, the imaging optical system 21 is illustrated as a single lens.

[0037] The imaging section, i.e., the imaging element 22, captures the subject image formed by the imaging optical system 21 and outputs an imaging signal. The imaging element 22 has an imaging surface 30 on which a plurality of imaging pixels are arranged. The control section 23 is an electronic circuit that controls each section of the information terminal device 10, and is composed of a CPU and its peripheral circuits. A predetermined control program is written in advance in a flash memory 28, which is a non-volatile recording medium. The control section 23 controls each section by reading and executing the control program from the flash memory 28. This control program uses a DRAM 27, which is a volatile recording medium, as a working area.

[0038] The liquid crystal monitor 24 as a display unit is a display device using a liquid crystal panel. The control unit 23 causes the imaging element 22 to repeatedly capture an image of the subject at a predetermined cycle (for example, 1 / 60th of a second). Then, various image processes are performed on the imaging signal output from the imaging element 22 to create an image of the subject, which is displayed on the liquid crystal monitor 24. In addition to the above-mentioned subject image, the liquid crystal monitor 24 displays, for example, captured image data (described later) and a setting screen for setting imaging conditions. The display device may be, for example, a display device using an organic EL panel other than the liquid crystal monitor 24.

[0039] The attitude detection unit 29 is, for example, an acceleration sensor, and detects the attitude of the information terminal device 10. Based on the signal output from the attitude detection unit 29, the control unit 23 determines whether the attitude of the information terminal device 10 is in a portrait position for capturing a portrait-oriented image or in a landscape position for capturing a landscape-oriented image.

[0040] The control unit 23 performs various image processing on the imaging signal output from the imaging element 22 to generate captured image data, and records the image data on a memory card 25, which is a portable recording medium. The operation unit 26 has various operation members such as push buttons, and outputs operation signals to the control unit 23 in response to the operation of these operation members. The liquid crystal monitor 24 is a touch panel type display device, and constitutes a part of the operation unit 26. The control unit 23 performs display control to display the subject image captured by the imaging element 22 on the liquid crystal monitor 24.

[0041] The control unit 23 includes an object recognition unit 23a, a weight setting unit 23b, an area division unit, i.e., an area partition unit 23c, an imaging condition setting unit 23d, an image processing unit 23e, and an imaging control unit 23f in the form of software. Each of these units is realized in the form of software by the control unit 23 executing a predetermined control program stored in the flash memory 28. Each of these units can also be configured as an electronic circuit.

[0042] The object recognition unit 23a recognizes or detects people, their faces, animals such as dogs and cats, vehicles such as bicycles, cars and trains, food on a plate, toys and other predetermined specific subject objects, i.e., subject elements, from the subject image captured by the image sensor 22, for example by a well-known method such as so-called template matching processing.

[0043] The weight setting unit 23b weights the subject objects recognized by the object recognition unit 23a. Specifically, the weight setting unit 23b assigns a predetermined weight to each of the subject objects recognized by the object recognition unit 23a, such as people, their faces, animals, vehicles, food, toys, etc. For example, the weight setting unit 23b assigns the highest weight to people and their faces, the next highest weight to animals, the next highest weight to vehicles, and the smaller weight to food, toys, etc. In this way, the weight setting unit 23b ranks the subject objects by assigning a predetermined weight to the subject objects recognized by the object recognition unit 23a.

[0044] The area division unit 23c judges whether the number of the subject objects recognized by the object recognition unit 23a and weighted by the weight setting unit 23b exceeds a predetermined number, and if the number of the subject objects is within the predetermined number, determines all of the subject objects recognized by the object recognition unit 23a as main subjects, and if the number of the subject objects exceeds the predetermined number, selects subject objects from those with relatively high ranks up to a predetermined number and determines the predetermined number of subject objects as main subjects. Note that this predetermined number can be set in advance by the user via the operation unit 26.

[0045] The area dividing unit 23c further specifies the position, i.e., the area, in the subject image in which the main subject determined as described above exists, and divides (divides) the area of ​​the main subject in the subject image into partial areas. As described later, when the subject image is displayed on the liquid crystal monitor 24, the area of ​​the main subject divided by the area dividing unit 23c, specifically, a frame representing the outline or contour of the main subject, is displayed superimposed on the subject image. Note that dividing the area of ​​the main subject on the subject image by the area dividing unit 23c is nothing other than specifying, i.e., dividing, the imaging area in which the main subject is imaged on the imaging surface of the imaging element 22.

[0046] The imaging condition setting unit 23d sets the imaging conditions of the imaging element 22, and also sets image processing conditions for the main subject, as described in detail later. The image processing unit 23e performs a plurality of image processing operations on the subject image. The imaging control unit 23f controls each of a plurality of imaging regions of the imaging element 22, each of which includes at least one imaging pixel, independently of one another. That is, as described above, the imaging control unit 23f controls imaging conditions such as exposure time, amplification factor, and frame rate for each unit group 32 of the imaging element 22. Note that the function of the imaging control unit 23f may be provided in the imaging element 22.

[0047] FIG. 2(a) is a plan view showing a schematic image pickup surface 30 of the image pickup element 22, and FIG. 2(b) is an enlarged plan view of a partial region 30a of the image pickup surface 30. As shown in FIG. 2(b), a large number of image pickup pixels 31 are two-dimensionally arranged on the image pickup surface 30. Each of the image pickup pixels 31 has a color filter (not shown). The color filters are of three types, red (R), green (G), and blue (B), and the notations "R", "G", and "B" in FIG. 2(b) indicate the type of color filter that the image pickup pixel 31 has. As shown in FIG. 2(b), the image pickup pixels 31 having such color filters are arranged on the image pickup surface 30 of the image pickup element 22 according to a so-called Bayer array.

[0048] The imaging pixel 31 having a red filter photoelectrically converts light in the red wavelength band of the incident light and outputs a light reception signal (photoelectrically converted signal). Similarly, the imaging pixel 31 having a green filter photoelectrically converts light in the green wavelength band of the incident light and outputs a light reception signal. Moreover, the imaging pixel 31 having a blue filter photoelectrically converts light in the blue wavelength band of the incident light and outputs a light reception signal.

[0049] The imaging element 22 of the present embodiment is configured to be able to individually control each imaging region consisting of a total of four imaging pixels 31, i.e., each unit group 32, which are adjacent 2 pixels by 2 pixels. For example, when charge accumulation is started simultaneously for two different unit groups 32, one unit group 32 can read out charges, i.e., read out light reception signals, 1 / 30 seconds after the start of charge accumulation, and the other unit group 32 can read out charges 1 / 15 seconds after the start of charge accumulation. In addition, one unit group 32 may start charge accumulation at a timing different from the start of charge accumulation for the other unit group 32. In other words, the imaging element 22 can set a different exposure time (so-called shutter speed) for each unit group 32 in one imaging operation.

[0050] In addition to the exposure time described above, the image sensor 22 can vary the amplification factor of the light reception signal (so-called ISO sensitivity) for each unit group 32. The image sensor 22 can change the timing for starting charge accumulation and the timing for reading out the light reception signal for each unit group 32. The image sensor 22 can change the frame rate at which a moving image is captured for each unit group 32. The image capture control unit 23f of the control unit 23 shown in FIG. 1 controls imaging conditions such as the exposure time, amplification factor, and frame rate for each unit group 32.

[0051] In summary, the image pickup element 22 is configured so that the image pickup conditions such as the exposure time, the amplification factor, and the frame rate can be made different for each unit group 32.

[0052] The information terminal device 10 thus configured is configured so that the imaging conditions for an important subject object detected by the object recognition unit 23a, i.e., a main subject, can be manually set separately from imaging conditions for other subject parts, for example, the subject background, etc. This will be described in detail below.

[0053] 3(a) is a schematic diagram showing a state in which a subject image 110 is displayed on the liquid crystal monitor 24 of the information terminal device 10 whose orientation has been determined to be vertical by the orientation detection unit 29. In the following description, it is assumed that, as an example of a subject, two people 101 and 102, a small animal 103, a table 105, and a cake 104 placed on a plate as food are present in the imaging range of the information terminal device 10. It is assumed that, as viewed from the information terminal device 10, the two people 101 and 102 are on the other side of the table 105, the small animal 103 is under the table 105, and the cake 104 on a plate is placed on the table 105. For convenience of explanation, it is assumed that no subject that can be extracted as another subject element is present in the background portion 106 of the subject image 110. It should be noted that this subject image 110 was captured under the same imaging conditions (exposure time, amplification rate, frame rate) for all unit groups 32, i.e., for the entire imaging surface 30, based on exposure conditions based on the photometric value of the subject luminance or exposure conditions manually set by the user.

[0054] The object recognition unit 23a recognizes specific subject objects, that is, two people 101 and 102, a small animal 103, a cake 104 on a plate, and a table 105, from the subject image 110 displayed on the liquid crystal monitor 24. The object recognition unit 23a performs subject detection using image data indicating the subject image 110 output from the imaging element 22. As a result of the subject detection, the object recognition unit 23a recognizes the specific subject objects, that is, the two people 101 and 102, the small animal 103, the cake 104 on a plate, and the table 105.

[0055] When the object recognition unit 23a detects a plurality of objects, the weight setting unit 23b assigns weights to each of the plurality of objects detected by the object recognition unit 23a. Here, the weight setting unit 23b assigns the largest weight to the two people 101 and 102, the second largest weight to the small animal 103, the third largest weight to the cake 104 on the plate, and the fourth largest weight to the table 105.

[0056] The area division unit 23c determines, among the five subject objects detected by the object recognition unit 23a, for example, the four objects with the highest ranks weighted by the weight setting unit 23b as main subjects, and specifies or partitions the positions of the four main subjects in the subject image, i.e., the areas of the four main subjects, as main subject areas. In the following description, specifying or partitioning the main subject area is also referred to as area division.

[0057] When the main subject regions are specified or divided by the region dividing section 23c, the control section 23 causes the four main subject regions to be superimposed and explicitly displayed on the subject image 110A displayed in the display area 111 as shown in Fig. 3(b). For example, the control section 23 displays the contours 101a, 102a, 103a, 104a of the four main subject regions as indicated by dotted lines on the subject image 110A. Instead of displaying the contours by dotted lines as shown in the figure, the contour displays 101a, 102a, 103a, 104a can also be displayed by highlighting each main subject region as a whole brighter than other regions or by highlighting it with a color.

[0058] The control unit 23 generates partial images 181 of the main subjects 101-104 from the subject image 110A. In addition to displaying the contours 101a, 102a, 103a, and 104a of the main subject region on the subject image 110A, the control unit 23 provides a partial image display area 190 below the subject image 110A, specifically at the bottom of the liquid crystal monitor 24, as shown in FIG. 3(b), and displays the partial images 181 of the main subjects 101-104. That is, the control unit 23 displays the partial regions specified by the region dividing unit 23c as the contours 101a, 102a, 103a, and 104a of the main subject region, and displays the partial images 181 corresponding to the region of the main subject in the subject image. The control unit 23 also displays a setting screen 130 for the imaging conditions of the main subject in the display area 114 at the top right of the display screen of the liquid crystal monitor 24. In this way, the partial images 181 are displayed at the bottom of the display screen where the user's finger can easily select the partial images, etc. The setting screen 130 is displayed on the side of the user's dominant hand so that the user can easily set the imaging conditions. In the embodiment shown in FIG. 3(b), it is assumed that the user's dominant hand is right-handed, and if the user's dominant hand is left-handed, the screen is displayed on the left side of the display screen. Furthermore, even if the setting screen 130 and the partial image display area 190 are provided within the subject image 110A, the control unit 23 controls the display so that the aspect ratio (the ratio of the vertical length to the horizontal length of the subject image 110A) of the subject image 110A is not changed. Furthermore, the control unit 23 displays the partial image 181 on the lower side of the display screen, so that the subject image 110A is less likely to be hidden by the user's finger, and the visibility of the subject image 110A is improved.

[0059] The setting screen 130 for the imaging conditions of the main subject lists, for example, from the top, "shutter speed," "ISO sensitivity," and "image processing effect" as examples of imaging conditions, i.e., setting items 131 of the imaging conditions. The image processing effect is a setting item that is set when performing image processing for obtaining, for example, skin correction with whitening, a monochrome image, a sepia image, or the like. Thus, the setting items 131 of the setting screen 130 include "shutter speed" and "ISO sensitivity" as imaging conditions, and "image processing effect" as an image processing condition, but in the following description, they are also simply referred to as "imaging conditions." When capturing a moving image, "frame rate" is additionally displayed on the setting screen 130 as a setting item 131 of the imaging conditions. This makes it possible to change the setting of the frame rate for the main subject.

[0060] For each setting item 131, an item display 131a, a setting status display 131b, an up button display 131c, and a down button display 131d are displayed. For example, in the case of "shutter speed", the item display 131a displays the characters "shutter speed" indicating the shutter speed, which is a setting item, and the setting status display 131b displays the characters "1 / 125 seconds" indicating the current setting status of the shutter speed. The up button display 131c and the down button display 131d are display buttons for changing the current setting status. When the up button display 131c or the down button display 131d is touched, the setting status is changed sequentially. In addition, a decision button 132 for confirming the set imaging conditions is displayed at the bottom right of the display area 114. The same applies to "ISO sensitivity". Note that the setting status is the same for all unit groups 32, that is, the entire imaging surface 30, until a main subject area is selected by the user, which will be described later, and the imaging conditions of the main subject area are changed.

[0061] When any of the partial images 181 is selected by the user, the control unit 23 displays an enlarged image of the main subject related to the selected partial image 181 on the lower side of the liquid crystal monitor 24, as shown in FIG. 3(c). That is, the control unit 23 displays the partial image corresponding to the area of ​​the main subject in the subject image as an enlarged image. The control unit 23 displays the image in the partial image display area 190 enlarged more than the image showing the main subject of the subject image 110A displayed in the display area 111. FIG. 3(c) shows a state in which the partial image 181 of the person 101 is selected in FIG. 3(b) and the enlarged image 121 of the person 101 is displayed in the partial image display area 190. FIG. 3(d) shows a state in which the partial image 181 of the cake 104 is selected in FIG. 3(b) and the enlarged image 124 of the cake 104 is displayed in the partial image display area 190. That is, the selection of cake 104 is indicated by enlarging the image of cake 104 displayed in partial image display area 190 more than the image showing the main subject of subject image 110A displayed in display area 111. By displaying an enlarged image of the main subject on liquid crystal monitor 24 in this manner, even if the display of the main subject in subject image 110A is small, the visibility of the enlarged image of the main subject is good, making it easier for the user to make the desired setting.

[0062] The control unit 23 displays an enlarged image in the partial image display area 190, and also displays a back button 182 at the bottom right of the liquid crystal monitor 24. When the user operates the back button 182, the control unit 23 returns the display form of the partial image display area 190 to the display form shown in FIG.

[0063] As shown in FIG. 3(c), while an enlarged image of the main subject 101 is displayed in the partial image display area 190, the user touches the up button display 131c and the down button display 131d for various setting items 131a of the main subject setting 130 to set imaging conditions, such as the shutter speed.

[0064] For example, FIG. 3(c) shows an example in which the shutter speed setting is changed from 1 / 125 seconds to 1 / 250 seconds because there is a possibility that person 101 may move during capture, and the image processing effect is set to whitening mode so that the person appears pale.

[0065] When the decision button 132 at the bottom right of the display area 114 is operated, the imaging condition setting unit 23d sets the imaging conditions displayed in the display area 114 as the imaging conditions for the person 101. Specifically, the imaging condition setting unit 23d sets the exposure time (charge accumulation time) and amplification factor of the multiple unit groups 32 of the image sensor 22 corresponding to the person 101 as the main subject divided by the region dividing unit 23c, in the imaging control unit 23f according to the set shutter speed and ISO sensitivity. More specifically, the imaging condition setting unit 23d sets the exposure time and amplification factor for the multiple unit groups 32 that output image signals related to the person 101 as the main subject in the imaging surface 30 of the image sensor 22, in the imaging control unit 23f. At the same time, the imaging condition setting unit 23d sets a whitening mode in the image processing unit 23e for the main subject.

[0066] The imaging element 22 creates a subject image of the main subject captured under the imaging conditions set by the imaging condition setting unit 23d. The image processing unit 23e applies whitening processing to the main subject set by the imaging condition setting unit 23d. The control unit 23 displays the subject image of the main subject subjected to the whitening processing on the liquid crystal monitor 24. The user can check the effects of the imaging conditions and image processing conditions set by the user by looking at the subject image.

[0067] Thereafter, when a release button (not shown) or a button instructing capture of an image is operated, the control unit 23 generates subject image data that has been captured and image-processed for each of the main subjects under the imaging conditions and image processing conditions respectively set by the imaging condition setting unit 23d, and records the captured image data on the memory card 25.

[0068] In main subject setting 130, when the user sets the imaging conditions for the main subject, he or she selects one partial image from the multiple partial images 181, and control unit 23 enlarges and displays the selected partial image before setting the imaging conditions for that main subject. Alternatively, when one partial image is selected by the user, the imaging conditions for that main subject can be set without displaying the enlarged image, or even imaging conditions can be set for a main subject selected with a finger that is highlighted with a dotted line in the through image in Figure 3(b) without displaying multiple partial images.

[0069] In the above description, the case where there are many main subjects has been described, but in the case where there are a small number of main subjects, such as one or two, the control unit 23 controls each unit as follows with regard to the setting of the imaging conditions for the main subjects. Fig. 4(a) and Fig. 5(a) are schematic diagrams showing a state in which a subject image of one person 106 is displayed on the liquid crystal monitor 24 in the information terminal device 10 in which the orientation of the information terminal device has been determined to be vertical by the orientation detection unit 29. Fig. 4(a) shows a state in which the person 106 is displayed large on the liquid crystal monitor 24, that is, a state in which the proportion of the display area of ​​the liquid crystal monitor 24 occupied by the person 106 is larger than a predetermined proportion. Fig. 5(a) shows a state in which the person 106 displayed on the liquid crystal monitor 24 is displayed small, that is, a state in which the proportion of the display area of ​​the liquid crystal monitor 24 occupied by the person 106 is smaller than a predetermined proportion.

[0070] 4(a) and 5(a), object recognition unit 23a extracts person 106 as the subject object, and weight setting unit 23b assigns the largest weight to person 106. Region division unit 23c determines subject object 106 to be the main subject. Furthermore, when subject images of people 106 and 107 are displayed on liquid crystal monitor 24, person 106 and person 107 are each determined to be the main subject.

[0071] As shown in FIG. 4(a), when there is only one main subject and the proportion of the main subject in the imaging range is greater than a predetermined proportion, the control unit 23 performs the above-mentioned area division, and then, as shown in FIG. 4(b), the control unit 23 displays the outline 106a of the main subject area on the subject image 110B. The control unit 23 also displays a setting screen 130A for setting imaging conditions for the person 106 who is the main subject on the liquid crystal monitor 24. For example, the control unit 23 displays the setting screen 130A for setting imaging conditions for the person 106 who is the main subject on the upper right of the liquid crystal monitor 24. That is, the control unit 23 does not display the setting screen 130 and the partial image display area 190 shown in FIG. 3(b) and the like on the subject image 110B.

[0072] When the setting state display 131b of each setting item 131 is changed and the decision button 132 is touched, the image capturing condition setting section 23d of the control section 23 sets the changed image capturing condition as the image capturing condition for the person 106.

[0073] As a result, person 106 in subject image 110B is displayed according to the set imaging conditions and image processing conditions. Although not shown, the same applies to a case where there are two main subjects and the proportion of the imaging range occupied by the main subjects is greater than a predetermined proportion. In this case, when an image of person 106, who is the main subject, is selected, imaging conditions for person 106 can be set, and when an image of person 107 is selected, imaging conditions for person 107 can be set.

[0074] When a release button (not shown) or a button instructing capture of an image is operated in the state shown in FIG. 4(b), the control unit 23 generates subject image data that has been captured and image-processed for the main subject under the imaging conditions and image processing conditions set by the imaging condition setting unit 23d, and records the data on the memory card 25.

[0075] As shown in Fig. 5(a), when there is only one main subject and the proportion of the main subject occupying the imaging range is equal to or less than a predetermined proportion, after the above-mentioned region division, as shown in Fig. 5(b), control unit 23 displays an enlarged image 125 of person 106, who is the main subject, in partial image display area 190 below subject image 110C, specifically in the lower part of liquid crystal monitor 24, in addition to displaying outline 106a of the main subject region on subject image 110C. A setting screen 130A for setting imaging conditions for person 106, who is the main subject, is displayed in display area 114 at the upper right of the screen.

[0076] When the setting state display 131b of each setting item 131 is changed and the decision button 132 is operated, the image capturing condition setting section 23d of the control section 23 sets the changed image capturing condition as the image capturing condition for the person 106.

[0077] As a result, control unit 23 displays person 106 in subject image 110C and person 106 in enlarged image 125 according to the set imaging conditions and image processing conditions.

[0078] 5(b), when a release button (not shown) or a button for instructing imaging is operated, control unit 23 generates subject image data that has been imaged and image-processed for the main subject under the imaging conditions and image processing conditions set by imaging condition setting unit 23d, and records the data in memory card 25. Although not shown, the same applies when there are two main subjects and the proportion of the imaging range that the main subjects occupy is smaller than a predetermined proportion.

[0079] ---When the information terminal device 10 is in the horizontal position--- When the orientation detection unit 29 determines that the orientation of the information terminal device 10 is landscape, the control unit 23 displays, for example, a subject image 110E as shown in FIG. 6(a) on the liquid crystal monitor 24. In the following description, it is assumed that, as an example of a subject, two people 101 and 102, a small animal 103, tables 105A, 105B, and 105C, and cakes 104A and 104B placed on plates as food are present in the imaging range of the information terminal device 10. The object recognition unit 23a recognizes the two people 101 and 102, the small animal 103, the tables 105A, 105B, and 105C, and the cakes 104A and 104B as subject objects, and the weight setting unit 23b assigns a predetermined weight to each of the eight subject objects to rank each subject object. From the ranked eight object subjects, the region dividing unit 23c determines five object subjects, namely the people 101 and 102, the small animal 103, and the cakes 104A and 104B, as main objects.

[0080] FIG. 6(b) is a display screen of the liquid crystal monitor 24 including a subject image in which five subject objects 101, 102, 103, 104A, and 104B determined to be main subjects are highlighted with dotted lines. FIG. 6(b) is a diagram corresponding to FIG. 3(b) in a case where the orientation detection unit 29 has determined that the orientation of the information terminal device 10 is vertical. When the orientation detection unit 29 has determined that the orientation of the information terminal device 10 is horizontal, three partial images 126 to 128 of the five main subjects are displayed in the display areas 146 to 148 on the left side of the liquid crystal monitor 24. A setting screen 130 for setting the imaging conditions of the main subject is displayed on the right side of the screen. When the operation marks 115 and 116 in the display areas 146 and 148 are operated, the control unit 23 sequentially switches the images of the three main subjects displayed in the display areas 146 to 148 to the partial images of the remaining two main subjects.

[0081] Here, for example, when image 128 of cake 104A displayed in display area 148 in the state shown in Fig. 6(b) is selected, control unit 23 causes enlarged image 129 of cake 104A to be displayed in partial image display area 149 on the left side of liquid crystal monitor 24, as shown in Fig. 6(c). This allows the user to appropriately change the imaging conditions for cake 104A, which is the main subject, from setting screen 130. Because enlarged image 129 of cake 104A is displayed, the visibility of enlarged image 129 is good, making it easy for the user to make desired settings.

[0082] In addition, the control unit 23 may appropriately change the display range of the subject image 110D as shown in FIG. 6(d) so that the subject image 110D of the cake 104A for which the imaging conditions are set is not hidden by the partial image display area 149 and the setting screen 130. As shown in FIG. 6(d), an example is shown in which the control unit 23 changes the display range of the subject image 110D on the liquid crystal monitor 24 so that the subject image of the cake 104A is displayed in a display area 152 that displays the subject image between the partial image display area 149 and the setting screen 130. Also, as shown in FIG. 6(d), the visibility of the setting screen 130 may be improved by preventing the subject image 110E from being displayed in the display area of ​​the setting screen 130 for the imaging conditions.

[0083] When the decision button 132 at the bottom right of the display area 151 is operated, the image capturing condition setting unit 23d sets the image capturing conditions displayed in the display area 151 as the image capturing conditions for the cake 104A. As a result, the cake 104A in the subject image 110E and the cake 104A in the enlarged image 129 in the display area 149 are displayed according to the set image capturing conditions and image processing conditions. The image capturing conditions and image processing conditions for the other main subjects are set in the same manner. The display of the subject image and the release operation after the image capturing conditions and image processing conditions for the main subject are set in the same manner as in the above-mentioned example. Note that, when the enlarged image 129 in the display area 149 in FIG. 6(c) is operated twice in a short time, the display form of the liquid crystal monitor 24 may be returned to the state shown in FIG. 6(b).

[0084] The display form on the liquid crystal monitor 24 is also the same when there is one main subject and the proportion of the main subject that occupies the imaging range is equal to or less than a predetermined proportion. In this case, the enlarged image on the left side of the screen is displayed in the center in the vertical direction. Also, when there are a small number of main subjects, such as one or two, and the proportion of the main subject that occupies the imaging range is greater than a predetermined proportion, the display form on the liquid crystal monitor 24 is the same as that shown in Figures 4(a) and 4(b).

[0085] ---flowchart--- 7 is a flow chart of the process of setting imaging conditions for each of the main subjects and capturing an image of the subject. When the information terminal device 10 is powered on, the program shown in FIG. 7 is started and executed by the control unit 23. In step S1, the control unit 23 starts an application program for imaging and proceeds to step S3. In step S3, the imaging control unit 23f controls the imaging pixels 31 of the imaging element 22 so that the charge accumulation time and amplification factor of the imaging pixels 31 are the same values ​​for all unit groups 32, which are determined based on the luminance of the subject, for example, and the subject image based on the imaging signals from the imaging pixels 31 controlled in this way is displayed on the liquid crystal monitor 24.

[0086] In step S7, object recognition unit 23a recognizes or extracts the subject object, i.e., the subject element, from subject image 110 displayed on liquid crystal monitor 24 as subject image 110, and proceeds to step S9. Object recognition unit 23a recognizes or extracts the subject object, i.e., the subject element, from image data representing subject image 110 output from imaging element 22. In step S9, weight setting unit 23b weights the subject elements extracted by object recognition unit 23a, and proceeds to step S11. In step S11, region division unit 23c determines a predetermined number of subject elements having the highest importance set by weight setting unit 23b out of the subject elements extracted by object recognition unit 23a as main subjects, performs region division, and proceeds to step S15.

[0087] In step S15, the control unit 23 controls each unit to explicitly display the main subject on the subject image, for example, displaying the outline of the main subject with a dotted line or highlighting it, and then proceeds to step S17. In step S17, the control unit 23 judges whether or not there are many main subjects. In this embodiment, if the number of main subjects is a predetermined number, for example, three or more, it is determined that there are many main subjects. If the judgment in step S17 is affirmative, in step S19, the control unit 23 displays the subject image 110A, the partial image 181 of the main subject, etc. on the liquid crystal monitor 24, as shown in FIG. 3(b), and proceeds to step S21. In step S21, the control unit 23 sets the imaging conditions for each main subject in response to the user's operation, and proceeds to step S23. The display transition of the liquid crystal monitor 24 and the setting of the imaging conditions in response to the user's operation in step S21 are as described above.

[0088] In step S23, the control unit 23 judges whether or not a user has issued an instruction to capture an image, i.e., whether or not a release button (not shown) or a button for instructing capture has been operated. If the result of step S23 is affirmative, the process proceeds to step S25, and if the result of step S23 is negative, the process returns to step S21.

[0089] In step S25, the control unit 23 controls the image sensor 22 to capture each main subject under the imaging conditions set for that main subject as described above, and performs a predetermined image processing on the main subject of the subject image obtained by imaging as described above to generate captured image data and record it on the memory card 25.

[0090] In step S27, the control unit 23 judges whether or not to continue imaging. For example, the control unit 23 judges whether or not an instruction to end the imaging application program has been given. If it is judged in step S27 that imaging is to be continued, the process returns to step S7, and if it is judged that imaging is to be ended, the application program for imaging is ended.

[0091] If step S17 is judged to be negative, that is, if the number of main subjects is judged to be one or two, the process proceeds to step S31. In step S31, control unit 23 judges whether the proportion of the main subject occupying the imaging range is greater than a predetermined proportion, that is, whether the image of the main subject is larger than a predetermined size. If step S31 is judged to be negative, that is, if the proportion of the main subject occupying the imaging range is equal to or less than the predetermined proportion, the process proceeds to step S19.

[0092] If step S31 is judged to be positive, that is, if the proportion of the main subject in the imaging range is greater than a predetermined proportion, the process proceeds to step S35. In step S35, the control unit causes the liquid crystal monitor 24 to display the subject image 110B and the imaging condition setting screen 130, for example, as shown in FIG. 4(b), and proceeds to step S21.

[0093] The above-described first embodiment provides the following advantageous effects. (1) The area of ​​the main subject in the subject image is divided into partial areas, and the partial images and enlarged images of the main subject and the subject image are displayed on the liquid crystal monitor 24. This allows the user to easily recognize the main subject, improving convenience and providing an easy-to-use electronic device.

[0094] (2) A partial image or an enlarged image of the main subject and the subject image are displayed on the liquid crystal monitor 24, and the imaging conditions of the imaging area corresponding to the partial image can be manually set. The imaging conditions of the multiple unit groups 32 of the image sensor 22 corresponding to the main subject are set to the manually set imaging conditions. This makes it possible to provide a user interface that allows the user to appropriately set imaging conditions for each imaging area.

[0095] (3) The control unit 23 is configured to explicitly display the main subject area in the subject image. This allows the user to easily recognize the main subject even if the main subject is displayed small on the LCD monitor 24, which is highly convenient. (4) The image of the main subject is configured to be displayed on the liquid crystal monitor 24 as a partial image 181 separately from the subject image. This improves the visibility of the images of the main subject, making it easier for the user to select the main subject for which he or she wishes to change the shooting conditions.

[0096] (5) The image of the main subject is configured to be displayed as an enlarged image separately from the subject image on the LCD monitor 24. This improves the visibility of the main subject, makes it easier to change the shooting conditions, and makes it easier to check the effects of the shooting conditions and image processing conditions set by the user.

[0097] (6) The object recognition unit 23a is configured to recognize a specific object from the object image displayed on the liquid crystal monitor 24 as the object image 110. Among the object detected by the object recognition unit 23a, the object with the highest weighted rank set by the weight setting unit 23b is determined as the main object, and is made the target of manual setting of the imaging conditions. This makes it possible to limit the number of main objects to a predetermined number even if the object recognition unit automatically recognizes a large number of objects. (7) The image processing effect for the main subject can also be changed. This allows users to easily obtain subject images with the quality they desire.

[0098] ---Second embodiment--- A second embodiment of the present invention will be described with reference to Figs. 8 to 9(b). In the following description, the same components as those in the first embodiment are denoted by the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as those in the first embodiment. The object recognition unit 23a in the first embodiment automatically recognizes a predetermined object as a main object from a subject image, or recognizes a predetermined object as a main object from a subject image. That is, it detects the area or range of the object. The object recognition unit 23a in the second embodiment, when a user designates one point or one location in a subject image as a part of the main subject, automatically detects the area or range of the main subject based on color information of the subject image at the designated location, or detects the area or range of the main subject based on color information of the subject image at the designated location.

[0099] 8 is a block diagram showing the configuration of an information terminal device 10A according to the second embodiment. The information terminal device 10A according to the second embodiment differs from the information terminal device 10 according to the first embodiment in that it does not include a weight setting unit 23b. The rest of the information terminal device 10A is the same as the information terminal device 10 according to the first embodiment.

[0100] FIG. 9(a) is a diagram showing a state where a user touches a person 101 in a subject image 110 displayed on the liquid crystal monitor 24 with a finger 161. In this embodiment, when a user selects one point or one place in the subject image 110 as a part of the main subject as shown in FIG. 9(a), the object recognition unit 23a automatically detects an area or range of an image having substantially the same color information as the color information of the selected part of the subject image 110 based on the color information of the selected part. For example, when a user touches a part of the face of the person 101 in the subject image 110 as shown in FIG. 9(a), the object recognition unit 23a detects the face area of ​​the person 101 by detecting the spread of the skin color part based on the skin color information of the face. When a part of the subject image 110 displayed on the liquid crystal monitor 24 is specified by the user, the object recognition unit 23a recognizes a specific area including the specified area as the main subject. For example, when a part of a person area displayed in the subject image 110 is specified by the user, the object recognition unit 23a recognizes the person area as a specific area. Furthermore, when a user designates a part of a person area of ​​one person in a case where a plurality of people are displayed on the liquid crystal monitor 24 as the subject image 110, the object recognition unit 23a may recognize the plurality of person areas as specific areas. That is, even when the user designates one area, the object recognition unit 23a may recognize the plurality of areas as specific areas based on, for example, color information or brightness information of the designated area or subject information (person, face, etc.) of the designated area.

[0101] When the object recognition unit 23a detects the area of ​​the main subject, the area division unit 23c partitions the main subject area within the subject image 110A. As a result, the control unit 23 highlights the person's face, i.e., the main subject, on the subject image 110A as shown by a dotted line in Fig. 9(b). Similarly, when the user touches another place on the subject image 110A, another main subject area is detected based on the color information thereof, and the area is highlighted on the subject image 110A.

[0102] Subsequent changes to the imaging conditions from the setting screen 130 for the imaging conditions of the main subject are the same as those in the first embodiment, and therefore will not be described in detail.

[0103] In addition to the effects of the first embodiment, the second embodiment described above has the following effects: (1) When the user touches a point or a location in the subject image as part of the main subject, the main subject is automatically detected. This allows the user to arbitrarily specify the main subject, so that the user can easily obtain a subject image with the intended subject image quality.

[0104] ---Third embodiment--- A third embodiment of the present invention will be described with reference to Figs. 10(a) to 11(b). In the following description, the same components as in the first embodiment are given the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as in the first and second embodiments. The third embodiment differs mainly from the first and second embodiments in that, when a user specifies a range of subjects, subjects within the specified range are set as the main subjects.

[0105] The configuration of the information terminal device 10A of the third embodiment is the same as that of the information terminal device 10A of the second embodiment shown in Fig. 8. Fig. 10(a) is a diagram showing a state in which a user touches a person 101 in a subject image 110 displayed on a liquid crystal monitor 24 with two fingers 161. In this embodiment, for example, when the subject image 110 is displayed on the liquid crystal monitor 24 as shown in Fig. 10(a) and two arbitrary positions on the liquid crystal monitor 24 are designated by the user, the control unit 23 causes the liquid crystal monitor 24 to display a rectangular frame 171 having diagonals passing through the designated two positions as shown in Fig. 10(b).

[0106] The control unit 23 enlarges the frame 171 when a pinch-out operation is performed as shown in Fig. 10(c), and reduces the frame 171 when a pinch-in operation is performed as shown in Fig. 10(d). In addition, when the relative positions of the two fingers 161 are changed, the control unit 23 changes the aspect ratio of the length and width of the frame 171 according to the position touched by the finger 161. In addition, when the frame 171 is selected by a long press operation of the finger 161 on the frame 171 or an area inside the frame 171, and the finger 161 moves on the display surface of the liquid crystal monitor 24, the control unit 23 moves the position of the frame 171 according to the movement of the finger 161 as shown in Fig. 10(e).

[0107] Furthermore, when the control unit 23 determines that the display surface of the liquid crystal monitor 24 is touched with one finger 161 and the finger 161 is traced on the display surface in the direction of the arrow A as shown in FIG. 11(a), the control unit 23 displays a line 172 along the traced path on the liquid crystal monitor 24. When the display surface of the liquid crystal monitor 24 is then traced with the finger 161 as shown by the arrow B in FIG. 11(b), the control unit 23 displays the line 172 along the path of the finger 161. Then, when the start point of the line 172 and the path of the finger 161 overlap, the control unit 23 sets the closed line 172 as a frame 171 (not shown). Note that even if the finger 161 is removed from the display surface of the liquid crystal monitor 24 before the start point of the line 172 and the path of the finger 161 overlap, if the distance between the start point and the position where the finger 161 is removed is close, the control unit 23 automatically closes the line 172 and sets it as the frame 171.

[0108] When frame 171 is specified in this manner, object recognition unit 23a detects the subject object within frame 171 from subject image 110 displayed on liquid crystal monitor 24. Then, area division unit 23c determines the subject object detected by object recognition unit 23a as the main subject, and specifies or divides the position of the main subject in subject image 110, i.e., the area of ​​the main subject, as the main subject area. The subsequent change of imaging conditions from setting screen 130 for imaging conditions of the main subject is the same as in the first embodiment, and therefore a detailed description will be omitted.

[0109] In the above-described third embodiment, in addition to the effects of the first and second embodiments, the following effects are achieved. (1) It is configured so that a frame 171 or a line 172 can be drawn by touching the display surface of the liquid crystal monitor 24. It is also configured so that a subject object within an area specified by the frame 171 or the line 172 can be detected. This allows the user to specify the main subject as intended, and therefore the subject image as intended by the user can be easily obtained.

[0110] ---Variations--- (1) In the above description, the object recognition unit 23a is configured to recognize a specific object, thereby making it possible to change the imaging conditions for the area inside the outline of the main object. However, the present invention is not limited to this. For example, the imaging conditions for the entire area inside the frame 171 in the third embodiment described above may be configured to be changeable. (2) The display mode may be changed according to the size of the screen of the liquid crystal monitor 24. For example, in the case of a tablet terminal device with a large screen size, the main subject is displayed larger than in a terminal device such as a smartphone, so that, for example, unless the main subject is very small, it is not necessary to display an enlarged image. In other words, the control unit 23 may determine whether the size of the main subject to be displayed is a predetermined size or not, and control the display mode according to the result of the determination.

[0111] (3) The display order of the setting items 131 of the imaging conditions may be changed depending on the type of subject. For example, if the main subject is a person, there is a high possibility that the settings related to the brightness of the person will be changed, so the control unit 23 may be configured to preferentially display setting items related to changing the brightness as the setting items 131. This improves the operability of changing the settings. (4) In the above description, the liquid crystal monitor 24 is a touch panel type display device and constitutes part of the operation unit 26, but the present invention is not limited to this. For example, the operation unit 26 may be provided with operation devices such as operation keys, an operation dial, a mouse, etc., so that an arbitrary position on the display screen can be specified by moving a cursor, a pointer, etc. on the display screen of the liquid crystal monitor 24 to a desired position.

[0112] (5) When displaying an enlarged image of the main subject, for example, the aspect ratio of the area displaying the subject image may differ from that of the original subject image. In such a case, the subject image may be reduced in size and displayed in its entirety, or an appropriate image may be cropped from the subject image and displayed. (6) In the above description, for example, "shutter speed," "ISO sensitivity," "frame rate," and "image processing effect" are given as examples of the setting items 131 of the imaging conditions, but these are merely examples and the present invention is not limited to these. For example, it is sufficient that at least one of "shutter speed," "ISO sensitivity," and "frame rate" as imaging conditions is included in the setting items 131. Also, for example, "image processing effect" as an image processing condition does not have to be included in the setting items 131. Items other than those mentioned above may be included as the setting items 131.

[0113] (7) In the above description, the information terminal device 10 is described as being a device that is thin, has a large liquid crystal monitor on the front side, and has an imaging optical system on the rear side, such as a tablet-type information terminal device, but the present invention is not limited to this. For example, the information terminal device 10 may be an imaging device, that is, a camera with an integrated lens, or a camera with an interchangeable lens that allows an interchangeable lens to be attached to and detached from a camera body.

[0114] (8) In the above description, the information terminal device 10 is a single electronic device having the image sensor 22 and the control unit 23, but the present invention is not limited to such an embodiment. For example, the present invention can also be applied to an electronic device that controls an externally provided image sensor 22. Below, a detailed description is given of an embodiment in which the image sensor 22 is provided externally.

[0115] FIG. 12 is a block diagram showing a schematic configuration of an imaging system according to this modified example. The imaging system 1000 shown in FIG. 12 includes an imaging unit 1001 and an electronic device 1002. The imaging unit 1001 includes the photographing optical system 21 and the imaging element 22 described in the first embodiment, and further includes a first communication unit 1003. The electronic device 1002 includes the control unit 23, the liquid crystal monitor 24, the memory card 25, the operation unit 26, the DRAM 27, the flash memory 28, and the attitude detection unit 29 described in the first embodiment, and further includes a second communication unit 1004. The first communication unit 1003 and the second communication unit 1004 can perform bidirectional data communication, for example, by well-known wireless communication technology or optical communication technology. The imaging unit 1001 and the electronic device 1002 may be connected by a wired cable or the like, and the first communication unit 1003 and the second communication unit 1004 may perform bidirectional data communication.

[0116] In the imaging system 1000 according to this modification, the control unit 23 controls the image sensor 22 by data communication via the second communication unit 1004 and the first communication unit 1003. For example, by transmitting and receiving predetermined control data to and from the imaging unit 1001, different imaging conditions are set for each unit group 32, and imaging signals are read out from each unit group 32.

[0117] As described above, in the imaging system 1000, it is the control unit 23 that controls each unit group 32. The electronic device 1002 does not include an imaging element 22, but performs the same control as in the first embodiment by controlling the imaging element 22 (imaging unit 1001) provided outside the electronic device 1002. In other words, the present invention can be applied to electronic devices that do not include an imaging element 22.

[0118] (9) The display form of the enlarged image described above is an example, and the present invention is not limited to this. For example, as shown in FIG. 13(a), a speech bubble 201 may be displayed next to a main subject (person 101), and an enlarged image of the main subject may be displayed in the speech bubble 201. The position of the speech bubble is not limited to the side of the main subject, but may be above or below the main subject, or may be in an oblique position. The shape of the speech bubble may not be rectangular as shown in FIG. 13(a), but may be circular or polygonal. The outline of the speech bubble may not be displayed. An image showing the imaging conditions currently set for the main subject may be displayed together with or instead of the enlarged image of the main subject.

[0119] (10) As a slide bar for changing the setting of each setting item 131, for example, as shown in FIG. 13(b), a slider (knob) 201a may be displayed that moves along the contour of a speech bubble 201 for enlarging and displaying a main subject. In the example shown in FIG. 13(b), a scale line 201b is provided along the contour of the circular speech bubble 201. The user may be configured to be able to change the setting value of a specific setting item 131 by moving the slider 201a along the contour of the speech bubble 201. Note that the main subject is not shown in FIG. 13(b). The slide bar and slider 201a may be displayed together with the speech bubble 201 or instead of the speech bubble 201.

[0120] (11) As another example of a display form in which the main subject is explicitly displayed in the subject image, the main subject may be displayed so as to appear to stand out from the screen. Fig. 14(a) is a diagram showing an example of a subject image 110 displayed on the liquid crystal monitor 24. For example, an image of a person 101 who is the main subject in this subject image 101 may be enlarged slightly (for example, about 1.2 times) and displayed, as in image 101b of person 101 shown in Fig. 14(b), so that person 101 appears to stand out from the screen.

[0121] (12) In the above description, the display area of ​​the partial image 181 or the enlarged image and the display area of ​​the subject image are separated by straight lines. However, the present invention is not limited to this. For example, the separation may be by a curved line. (13) For example, as shown in Fig. 15(a), when the image of person 101, who is the main subject in subject image 110, is touched, icons 202 for selecting setting items of the image capture conditions may be displayed around the image of person 101. Also, as shown in Fig. 15(b), when the image of person 101, who is the main subject in subject image 110, is touched, slide bar 203 for changing the settings of each setting item of the image capture conditions may be displayed around the image of person 101. In this way, it is possible to directly change the setting of the image capture conditions for the main subject. In this case, the slide bar 203 may display the image capture conditions currently set for the main subject.

[0122] (14) For example, as shown in FIG. 16, a diagram 204 showing where in the subject image 110 the main subject area for which the imaging conditions are to be changed is located may be displayed together with the subject image 110. That is, as shown in FIG. 16, a diagram (area display diagram) 204 showing the main subject area for which the imaging conditions are to be changed may be displayed, for example, in the upper right area of ​​the liquid crystal monitor 24. In the example shown in FIG. 16, for example, it is assumed that a table 105, a cake 104 placed on a plate, and a small box 109 are present in the imaging range of the information terminal device 10. It is assumed that the cake 104 placed on the plate and the small box 109 are placed on the table 105. It is also assumed that the main subject area for which the imaging conditions are to be changed, i.e., the main subject for which the imaging conditions are to be changed, is the table 105.

[0123] In the area display diagram 204, an area corresponding to the background portion 108 in the subject image 110 and an area corresponding to an area other than the table 105 in the subject image 110 are, for example, masked or displayed dark. As a result, in the area display diagram 204, the table 105, the imaging condition of which is to be changed, is displayed more clearly than other areas. Therefore, by visually checking the area display diagram 204, the user can easily recognize that the main subject, the imaging condition of which is to be changed, is the table 105. The imaging conditions for the table 105 can be changed by operating the slide bar 203 displayed below the area display diagram 204. In this case, the imaging conditions currently set for the main subject may be displayed by the slide bar 203.

[0124] (15) For example, as shown in Fig. 17, one of two display areas 194, 195 (for example, display area 194) may be configured to display an image of the main subject (person 101) before changing the settings of the imaging conditions, and the other (for example, display area 195) may be configured to display an image of the main subject (person 101) after changing the settings of the imaging conditions. By configuring in this way, for example, it is possible to check the subtle changes in the image caused by changing settings such as the shutter speed or ISO sensitivity by looking at images before and after the setting change.

[0125] (16) For example, as shown in Fig. 18, icons for setting a main subject for each motif may be displayed on liquid crystal monitor 24 together with subject image 110, and when the icon is selected, a subject that matches the motif corresponding to the icon may be set as the main subject. In Fig. 18, icons 205a, 205b, and 205c for setting a main subject for each motif are displayed at the bottom of the screen of liquid crystal monitor 24.

[0126] For example, when icon 205a is touched, control unit 23 controls each unit to set people 106, 107 in subject image 110 as the main subject. For example, when icon 205b is touched, control unit 23 controls each unit to set vehicle 206 in subject image 110 as the main subject. For example, when icon 205c is touched, control unit 23 controls each unit to set background portion 108 in subject image 110 as the main subject. This configuration makes it easy to set a main subject for each motif and change imaging conditions.

[0127] (17) For example, as shown in Fig. 19, icons for setting each person in the subject image 110 as a main subject may be displayed on the liquid crystal monitor 24 together with the subject image 110, and when the icon is selected, the person corresponding to the icon is set as the main subject. In Fig. 19, icons 221-223 for setting each person as a main subject are displayed at the bottom of the screen of the liquid crystal monitor 24.

[0128] That is, the control unit 23 controls each unit to detect a human face from the subject image displayed as the subject image. Then, when a human face is detected, an icon for setting that person as a main subject is generated based on the detected facial image, and displayed on the liquid crystal monitor 24. In the example shown in FIG. 19, the control unit 23 detects the faces of three people 211-213 from the subject image displayed as the subject image. Then, based on the facial images of the people 211-213, icons 221-223 are generated and displayed on the liquid crystal monitor 24.

[0129] For example, when icon 221 corresponding to person 211 is touched, control unit 23 controls each unit to set person 211 in subject image 110 as the main subject. For example, when icon 222 corresponding to person 212 is touched, control unit 23 controls each unit to set person 212 in subject image 110 as the main subject. For example, when icon 223 corresponding to person 213 is touched, control unit 23 controls each unit to set person 213 in subject image 110 as the main subject. This configuration makes it easy to select whether or not to set each person in subject image 110 as the main subject.

[0130] (18) In the above description, the orientation of information terminal device 10 detected by orientation detection unit 29 and the display direction of the subject image are the same, that is, if the orientation of information terminal device 10 detected by orientation detection unit 29 is horizontal, the display direction of the subject image is horizontal, but this is not limited to the above. Even if the orientation detection unit 29 detects that the orientation of information terminal device 10 is horizontal, if the user has set the display direction of the subject image to be vertical in advance, control unit 23 sets the display direction of the subject image to vertical and performs subsequent control.

[0131] (19) In the above explanation, no specific reference was made to the minimum unit of division into partial regions. However, for example, the minimum unit of division into partial regions may be one unit group 32, or may be, for example, one pixel. (20) The above-described embodiments and modifications may be combined with each other. ---Fourth embodiment---

[0132] 20 is a block diagram showing the configuration of an imaging device according to a fourth embodiment of the present invention. The imaging device 3010 is a lens-integrated camera. The imaging device 3010 includes an imaging optical system 21, an imaging element 22, a control unit 33, a liquid crystal monitor 24, a memory card 25, an operation unit 26, a DRAM 27, a flash memory 28, and a sound recording unit 34.

[0133] The imaging optical system 21, imaging element 22, liquid crystal monitor 24, memory card 25, operation unit 26, DRAM 27 and flash memory 28 are the same as those in the first embodiment, and therefore description thereof will be omitted.

[0134] The control unit 33 creates an image file (described later) based on the imaging signal output from the imaging element 22, and records the image file on a memory card 25, which is a removable recording medium. The operation unit 26 has various operation members such as push buttons, and outputs operation signals to the control unit 33 in response to the operation of these operation members. The sound recording unit 34 is composed of, for example, a microphone, and converts environmental sounds into audio signals and inputs them to the control unit 33.

[0135] Next, the imaging process executed by the control unit 33 will be described in detail.

[0136] (1) Preliminary imaging The control unit 33 performs a preliminary imaging before the main imaging. In the following description, this preliminary imaging is called a preliminary imaging. In the preliminary imaging, the control unit 33 sets the imaging conditions for each unit group 32 to predetermined imaging conditions determined by a known automatic exposure control. One subject image is obtained by the preliminary imaging.

[0137] (2) Identifying the darkest, brightest, and main areas The control unit 33 measures the subject luminance for each unit group 32, that is, the luminance distribution of the subject, from the subject image obtained by the preliminary imaging. Hereinafter, a description will be given with reference to FIG.

[0138] Now, assume that an image of a subject as shown in FIG. 21(a) is to be captured. A person 1104, a building 1102, a ground 1101, a sky 1103, and a sun 1105 are captured in an image capture screen 1100 shown in FIG. 21(a). A shadow 1106 of the building 1102 is present on the ground 1101. An example of the luminance distribution of the subject obtained by capturing an image of the subject through preliminary capturing is shown in FIG. 21(b). In the luminance distribution map 67 shown in FIG. 21(b), one small rectangle represents one unit group 32. Also, a numerical value written in the rectangle represents a luminance value (Bv value) detected in one unit group 32.

[0139] The control unit 33 identifies the darkest part (darkest part) and the brightest part (brightest part) of the subject from the brightness distribution map 67 of the subject illustrated in FIG. 21(b). In FIG. 21(b), the minimum brightness value of "4" is detected in the region 1109 where the shadow 1106 of the building 1102 is located. That is, the darkest part of the subject illustrated in FIG. 21(a) is the region 1109 where the shadow 1106 is located (hereinafter, referred to as the darkest part 1109). Also, in FIG. 21(b), the maximum brightness value of "8" is detected in the region 1108 where the sun 1105 is located. That is, the brightest part of the subject illustrated in FIG. 21(a) is the region 1108 where the sun 1105 is located (hereinafter, referred to as the brightest part 1108).

[0140] Next, the control unit 33 executes a predetermined main subject detection process on the subject image obtained by the preliminary capture. The main subject detection process is a process for detecting the face of the subject person, for example, by so-called face recognition technology, and identifying the position and range in which the subject person exists. For example, in the subject shown in FIG. 21(a), the face 1107 of the person 1104 is first detected, and based on this, the area 1110 in which the person 1104 is located is identified. The area 1110 in which the detected person 1104 exists is referred to as the main part of the subject (hereinafter, referred to as the main part 1110).

[0141] Note that the main subject detection process may be a process other than the process of detecting a subject person based on the face recognition technology described above. For example, the main part of the subject may be identified based on the focus position of the imaging optical system 21 or color information of the subject. Also, the main part of the subject may be identified by well-known techniques such as feature detection and pattern matching. Alternatively, all parts other than the darkest part 1109 and the brightest part 1108 may be regarded as the main part.

[0142] In the above-described embodiment, the control unit 33 specifies the brightest part 1108, the darkest part 1109, and the main part 1110, but depending on the subject image, the main part 1110 may be the brightest part 1108, or the main part 1110 may be the darkest part 1109. For example, in a backlit scene, the main subject becomes dark and the main part 1110 and the darkest part 1109 become the same region. In such a case, the control unit 33 does not specify the three parts, the brightest part 1108, the darkest part 1109, and the main part 1110, but may specify only two parts, the main part 1110 (= the brightest part 1108) and the darkest part 1109, or the main part 1110 (= the darkest part 1109) and the brightest part 1108.

[0143] (3) Detection of blackout and whiteout Next, the control unit 33 checks whether or not there is crushed black in the darkest part 1109. In other words, the control unit 33 determines whether or not the luminance value (pixel value) of the subject image obtained by preliminary imaging in any one of the unit groups 32 belonging to the darkest part 1109 falls below a predetermined lower limit value.

[0144] Similarly, control unit 33 checks whether there is blown out highlights in brightest portion 1108. In other words, control unit 33 determines whether the luminance value of the subject image obtained by preliminary imaging exceeds a predetermined upper limit value in any of unit groups 32 located in brightest portion 1108. For example, when the luminance value is expressed as an integer from 0 to 255, the lower limit value is set to 1 and the upper limit value is set to 254.

[0145] The detection results of shadow crush and highlight blowout by the control unit 33 will be one of the following: (A) neither shadow crush nor highlight blowout is present, (B) shadow crush is present but highlight blowout is not present, (C) highlight blowout is present but shadow crush is not present, or (D) both shadow crush and highlight blowout are present.

[0146] (4) Setting the initial conditions Based on the detection results of blocked up shadows and blown out highlights, the control unit 33 determines initial values ​​(initial conditions) of imaging conditions to be set for the unit groups 32 belonging to the darkest part 1109 in actual imaging, initial values ​​(initial conditions) of imaging conditions to be set for the unit groups 32 belonging to the brightest part 1108 in actual imaging, and initial values ​​(initial conditions) of imaging conditions to be set for the unit groups 32 belonging to the main part 1110 in actual imaging. A method of determining the initial conditions by the control unit 33 will be described in detail later.

[0147] (5) Acceptance of changes to imaging conditions Next, the control unit 33 displays the subject image obtained by the preliminary imaging on the liquid crystal monitor 24. Then, the control unit 33 gives the user an opportunity to change the imaging conditions for each unit group 32 from the above-mentioned initial conditions. Specifically, the control unit 33 accepts a predetermined change operation on an operating member (not shown).

[0148] In this embodiment, to prevent operations from becoming complicated, only the imaging conditions of the main part 1110 are accepted for change. In other words, only the imaging conditions of the main part 1110 can be explicitly changed by the user. The imaging conditions of the darkest part 1109 and the brightest part 1108 are basically automatically changed by the control unit 33 in response to an operation for changing the imaging conditions of the main part 1110. In other words, the imaging conditions of the darkest part 1109 and the brightest part 1108 are linked to the imaging conditions of the main part 1110. A method for changing the imaging conditions by the control unit 33 will be described in detail later.

[0149] The control unit 33 highlights the main part 1110 in the subject image displayed on the liquid crystal monitor 24. For example, a dashed line is displayed surrounding the main part 1110, or the main part 1110 is displayed blinking. Furthermore, when at least one of the imaging conditions of the darkest part 1109, the brightest part 1108, and the main part 1110 is changed, the control unit 33 displays the subject image reflecting the change in the imaging conditions on the liquid crystal monitor 24. In other words, the change in the imaging conditions is reflected on the liquid crystal monitor 24. For example, when the imaging conditions of the main part 1110 are changed to imaging conditions such that the main part 1110 is imaged brighter, the main part 1110 is displayed brighter in the subject image displayed on the liquid crystal monitor 24.

[0150] (6) Actual Imaging When a user who has set the imaging conditions to desired values ​​(conditions) performs a predetermined imaging operation on an operating member (not shown) (for example, full pressing of a release switch), the control unit 33 causes the image sensor 22 to capture an image of the subject and creates image data of the subject (main imaging). At this time, imaging conditions are set independently for the unit group 32 belonging to the darkest part 1109, the unit group 32 belonging to the brightest part 1108, and the unit group 32 belonging to the main part 1110. The control unit 33 records the image data created by the main imaging, for example, in the memory card 25 or displays it on the liquid crystal monitor 24.

[0151] The above is the imaging process by the control unit 33. Note that the imaging conditions described above may be exposure conditions such as shutter speed, ISO sensitivity, and aperture value, or imaging conditions such as a video frame rate.

[0152] Next, (4) setting of initial conditions and (5) acceptance of operation for changing imaging conditions among the above-mentioned imaging processes will be described by dividing them into cases according to the results of (3) detection of blocked up shadows and blown out highlights.

[0153] (A) When there is no crushed shadow or blown highlights (normal image) Fig. 22(a) shows a histogram of luminance for each unit group 32 when there is neither crushed shadows nor blown highlights. In Fig. 22(a), the horizontal axis represents luminance (pixel value) and the vertical axis represents the number (frequency) of unit groups 32 having that luminance. Note that Fig. 22(a) shows a schematic diagram of a set 1121 of unit groups 32 belonging to the darkest part 1109, a set 1123 of unit groups 32 belonging to the brightest part 1108, and a set 1122 of unit groups 32 belonging to the main part 1110.

[0154] 22(a), the sets 1121, 1122, and 1123 are illustrated so that their luminance does not overlap, but the sets 1121, 1122, and 1123 may overlap when, for example, the luminance values ​​of the sets 1121, 1122, and 1123 have a range, or when there is little difference in luminance between the set 110 of the darkest part 1109 and the set 1122 of the main part luminance. Even in such a case, the control unit 33 can change the sets 1121, 1122, and 1123 independently to the left or right on the luminance axis, as described later.

[0155] When there is no crushed shadow or blown highlights, that is, when the luminance of the entire screen is within a predetermined range (a range from a lower limit value to an upper limit value), the above three sets 1121, 1122, and 1123 all fall within a predetermined lower limit value 1120 and a predetermined upper limit value 1130. In this case, the control unit 33 sets the initial conditions of the darkest part 1109, the brightest part 1108, and the main part 1110 to the same imaging conditions as those of the preliminary imaging, that is, the predetermined imaging conditions determined by the control unit 33. For example, the exposure conditions such as the shutter speed (charge accumulation time), the aperture value, and the ISO sensitivity (amplification rate) are set to the predetermined exposure conditions determined by the automatic exposure control.

[0156] When the user performs a change operation (e.g., an operation to increase the shutter speed of the main part 1110) to change the imaging conditions of the main part 1110 so that the main part 1110 becomes darker (the brightness of the main part 1110 becomes lower), the control unit 33 changes the imaging conditions of the darkest part 1109 so that the brightness difference D1 between the group 1121 of the darkest part 1109 and the group 1122 of the main part 1110 is maintained. In other words, the imaging conditions of the darkest part 1109 are changed to imaging conditions (e.g., a faster shutter speed) such that the darkest part 1109 is imaged darker. In other words, in FIG. 22(a), when the user tries to move the group 1122 of the main part 1110 to the left on the brightness axis, the control unit 33 moves the group 1121 of the darkest part 1109 to the left on the brightness axis in conjunction with this. That is, when the control unit 33 receives an operation to change the imaging conditions of the main part 1110 from the user, the control unit 33 changes the imaging conditions of the main part 1110 and the imaging conditions of the darkest part 1109. The control unit 33 changes the imaging conditions of the main part 1110, thereby changing the imaging conditions of the darkest part 1109.

[0157] However, the control unit 33 changes the imaging conditions of the darkest part 1109 so that the luminance of the darkest part 1109 does not fall below a predetermined lower limit 1120. That is, the control unit 33 stops (prohibits) changing the imaging conditions of the darkest part 1109 when the luminance of the darkest part 1109 reaches the predetermined lower limit 1120. In other words, in FIG. 22(a), when the user tries to move the set 1122 of the main part 1110 leftward on the luminance axis, if the set 1121 of the darkest part 1109 exceeds the predetermined lower limit 1120, the control unit 33 moves only the set 1122 of the main part 1110 leftward on the luminance axis, and the set 1121 of the darkest part 1109 remains stopped at the lower limit 1120 without moving. In this case, the luminance difference D1 between the set 1121 of the darkest part 1109 and the set 1122 of the main part 1110 changes.

[0158] Furthermore, the control unit 33 changes the imaging conditions of the main part 1110 so that the luminance of the main part 1110 does not fall below a predetermined lower limit 1120. That is, the control unit 33 stops changing the imaging conditions of the main part 1110 when the luminance of the main part 1110 reaches the predetermined lower limit 1120. In other words, in Fig. 22(a) , when the user attempts to move the set 1122 of the main part 1110 leftward on the luminance axis, if the set 1122 of the main part 1110 exceeds the predetermined lower limit 1120, the control unit 33 stops the set 1122 of the main part 1110 and does not move it.

[0159] It should be noted that the above-described change operation for making the main part 1110 darker (lowering the luminance of the main part 1110) does not change the imaging conditions of the brightest part 1108. In other words, in Fig. 22(a), when the user attempts to move the group 1122 of the main part 1110 to the left on the luminance axis, the group 1123 of the brightest part 1108 remains stationary and does not move.

[0160] Furthermore, when the user performs a change operation (e.g., an operation to slow down the shutter speed of the main part 1110) to change the imaging conditions of the main part 1110 so that the main part 1110 becomes brighter (the brightness of the main part 1110 becomes higher), the control unit 33 changes the imaging conditions of the brightest part 1108 so that the brightness difference between the set 1123 of the brightest part 1108 and the set 1122 of the main part 1110 is maintained. In other words, the imaging conditions of the brightest part 1108 are changed to imaging conditions (e.g., a slower shutter speed) such that the brightest part 1108 is imaged brighter. In other words, in FIG. 22(a), when the user tries to move the set 1122 of the main part 1110 to the right on the brightness axis, the control unit 33 moves the set 1123 of the brightest part 1108 to the right on the brightness axis in conjunction with this. That is, when the control unit 33 receives an operation from the user to change the imaging conditions of the main part 1110, the control unit 33 changes the imaging conditions of the main part 1110 and the imaging conditions of the brightest part 1108. The control unit 33 changes the imaging conditions of the main part 1110, thereby changing the imaging conditions of the brightest part 1108.

[0161] However, the control unit 33 changes the imaging conditions of the brightest part 1108 so that the luminance of the brightest part 1108 does not exceed a predetermined upper limit value 1130. That is, the control unit 33 stops changing the imaging conditions of the brightest part 1108 when the luminance of the brightest part 1108 reaches the predetermined upper limit value 1130. In other words, in FIG. 22(a), when the user tries to move the set 1122 of the main part 1110 to the right on the luminance axis, if the set 1123 of the brightest part 1108 exceeds the predetermined upper limit value 1130, the control unit 33 moves only the set 1122 of the main part 1110 to the right on the luminance axis, and the set 1123 of the brightest part 1108 remains stationary and does not move. In this case, the luminance difference D2 between the set 1123 of the brightest part 1108 and the set 1122 of the main part 1110 changes.

[0162] Furthermore, the control unit 33 changes the imaging conditions of the main part 1110 so that the luminance of the main part 1110 does not exceed a predetermined upper limit value 1130. That is, the control unit 33 stops changing the imaging conditions of the main part 1110 when the luminance of the main part 1110 reaches the predetermined upper limit value 1130. In other words, in Fig. 22(a) , when the user attempts to move the set 1122 of the main part 1110 to the right on the luminance axis, if the set 1122 of the main part 1110 exceeds the predetermined upper limit value 1130, the control unit 33 stops the set 1122 of the main part 1110 and does not move it.

[0163] In the above-described change operation for making the main part 1110 brighter (increasing the luminance of the main part 1110), the imaging conditions of the darkest part 1109 are not changed. In other words, in Fig. 22(a), when the user attempts to move the group 1122 of the main part 1110 to the right on the luminance axis, the group 1121 of the darkest part 1109 remains stationary and does not move.

[0164] Note that the control unit 33 may change the imaging conditions of the darkest part 1109 in response to a change in the imaging conditions of the main part 1110. For example, when it is considered that a certain amount or more of high-sensitivity noise is generated in the darkest part 1109, the imaging conditions of the darkest part 1109 may be changed so as to eliminate the noise.

[0165] (B) When there is crushed shadows but no blown highlights (crushed shadows image) 22B shows a histogram of the luminance for each unit group 32 in the case where there is blocked-up shadows and there is no blown-out highlights. In the case where there is blocked-up shadows and there is no blown-out highlights, that is, the luminance of the brightest part 1108 and the main part 1110 is within a predetermined range (a range from a lower limit value to an upper limit value) but the luminance of the darkest part 1109 is outside that range, a set 1123 of the brightest part 1108 and a set 1122a of the main part 1110 are within a predetermined lower limit value 1120 and a predetermined upper limit value 1130, and at least a part of a set 1121a of the darkest part 1109 is located to the left beyond the predetermined lower limit value 1120. In this case, the control unit 33 sets the initial conditions of the brightest part 1108 to the same imaging conditions as the preliminary imaging, that is, the predetermined imaging conditions determined by automatic exposure control.

[0166] The control unit 33 sets the initial condition of the darkest part 1109 to an imaging condition where there is no crushed black. In other words, the imaging condition is set so that the darkest part 1109 is imaged brighter than the predetermined imaging condition determined by the automatic exposure control. As a result, the set 1121a of the darkest part 1109 shown in FIG. 22(b) moves to a position adjacent to the predetermined lower limit value 1120 and to the right of the predetermined lower limit value 1120 (set 1121 shown by a solid line in FIG. 22(b)).

[0167] Furthermore, the control unit 33 sets the initial condition of the main part 1110 to an imaging condition corresponding to the initial condition of the darkest part 1109 and the initial condition of the brightest part 1108. Specifically, assuming that the absolute value of the difference between the mode of luminance of the darkest part 1109 and the mode of luminance of the main part 1110 under predetermined imaging conditions determined by automatic exposure control is D1, the absolute value of the difference between the mode of luminance of the main part 1110 under the same imaging conditions and the mode of luminance of the brightest part 1108 is D2, and the absolute value of the difference between the mode of luminance of the darkest part 1109 under predetermined imaging conditions determined by automatic exposure control and the mode of luminance of the darkest part 1109 under the above-mentioned initial conditions is da, the control unit 33 sets the initial condition of the main part 1110 to an imaging condition in which the luminance of the main part 1110 becomes larger (brighter) by D3 defined by the following formula (1). D3 = (da × D2) / (D1 + D2) (1)

[0168] In other words, the initial conditions of the darkest part 1109 and the main part 1110 are set so that crushed black is eliminated while maintaining the relative relationship between the luminance of the darkest part 1109, the luminance of the main part 1110, and the luminance of the brightest part 1108 under predetermined imaging conditions determined by automatic exposure control. As a result, the set 1122a of the main part 1110 shown in Figure 5 moves by D3 to the right on the luminance axis (set 1122 shown by a solid line in Figure 22(b)).

[0169] When the user performs a change operation to change the imaging conditions of the main part 1110 so that the main part 1110 becomes darker (the brightness of the main part 1110 becomes lower), the control unit 33 changes the imaging conditions of the brightest part 1108 so that the brightness difference D2' between the set 1123 of the brightest part 1108 and the set 1122 of the main part 1110 is maintained. That is, the imaging conditions of the brightest part 1108 are changed to imaging conditions such that the brightest part 1108 is imaged darker. In other words, in FIG. 22(b), when the user tries to move the set 1122 of the main part 1110 to the left on the brightness axis, the control unit 33 moves the set 1123 of the brightest part 1108 to the left on the brightness axis in conjunction with this. That is, when the control unit 33 accepts an operation to change the imaging conditions of the main part 1110 from the user, it changes the imaging conditions of the main part 1110 and the imaging conditions of the brightest part 1108. The control unit 33 changes the imaging conditions for the main part 1110, thereby changing the imaging conditions for the brightest part 1108.

[0170] However, the control unit 33 changes the imaging conditions of the main part 1110 so that the luminance of the main part 1110 does not fall below a predetermined lower limit 1120. In other words, the control unit 33 stops (prohibits) the change of the imaging conditions of the main part 1110 when the luminance of the main part 1110 reaches the predetermined lower limit 1120. In other words, in Fig. 22(b), when the user attempts to move the set 1122 of the main part 1110 to the left on the luminance axis, if the set 1122 of the main part 1110 exceeds the predetermined lower limit 1120, the control unit 33 stops and does not move the set 1122 of the main part 1110.

[0171] Furthermore, when the user performs a change operation to change the imaging conditions of the main part 1110 so that the main part 1110 becomes brighter (the luminance of the main part 1110 becomes higher), the control unit 33 changes the imaging conditions of the brightest part 1108 so that the luminance difference between the set 1123 of the brightest part 1108 and the set 1122 of the main part 1110 is maintained. That is, the imaging conditions of the brightest part 1108 are changed to imaging conditions such that the brightest part 1108 is imaged brighter. In other words, in FIG. 22(b), when the user tries to move the set 1122 of the main part 1110 to the right on the luminance axis, the control unit 33 moves the set 1123 of the brightest part 1108 to the right on the luminance axis in conjunction with this. That is, when the control unit 33 accepts an operation to change the imaging conditions of the main part 1110 from the user, it changes the imaging conditions of the main part 1110 and the imaging conditions of the brightest part 1108. The control unit 33 changes the imaging conditions for the main part 1110, thereby changing the imaging conditions for the brightest part 1108.

[0172] However, the control unit 33 changes the imaging conditions of the brightest part 1108 so that the luminance of the brightest part 1108 does not exceed a predetermined upper limit value 1130. That is, the control unit 33 stops changing the imaging conditions of the brightest part 1108 when the luminance of the brightest part 1108 reaches the predetermined upper limit value 1130. In other words, in FIG. 22(b), when the user tries to move the set 1122 of the main part 1110 to the right on the luminance axis, if the set 1123 of the brightest part 1108 exceeds the predetermined upper limit value 1130, the control unit 33 moves only the set 1122 of the main part 1110 to the right on the luminance axis, and the set 1123 of the brightest part 1108 remains stationary and does not move. In this case, the luminance difference D2' between the set 1123 of the brightest part 1108 and the set 1122 of the main part 1110 changes.

[0173] Furthermore, the control unit 33 changes the imaging conditions of the main part 1110 so that the luminance of the main part 1110 does not exceed a predetermined upper limit value 1130. That is, the control unit 33 stops changing the imaging conditions of the main part 1110 when the luminance of the main part 1110 reaches the predetermined upper limit value 1130. In other words, in Fig. 22(b), when the user attempts to move the set 1122 of the main part 1110 to the right on the luminance axis, if the set 1122 of the main part 1110 exceeds the predetermined upper limit value 1130, the control unit 33 stops the set 1122 of the main part 1110 and does not move it.

[0174] The above change operation does not change the imaging conditions of the darkest part 1109. In other words, in Fig. 22(a), no matter whether the user tries to move the group 1122 of the main part 1110 to the left or right on the luminance axis, the group 1121 of the darkest part 1109 does not move.

[0175] (C) When there is whiteout but no blackout (whiteout image) 22(c) shows a histogram of the luminance for each unit group 32 in the case where blown-out highlights are present but no crushed shadows are present. In the case where blown-out highlights are present but no crushed shadows are present, that is, the luminance of the darkest part 1109 and the main part 1110 is within a predetermined range (a range from a lower limit value to an upper limit value) but the luminance of the brightest part 1108 is outside that range, a set 1121 of the darkest part 1109 and a set 1122a of the main part 1110 are within a predetermined lower limit value 1120 and a predetermined upper limit value 1130, and at least a part of a set 1123a of the brightest part 1108 is located to the right exceeding the predetermined upper limit value 1130. In this case, the control unit 33 sets the initial conditions of the darkest part 1109 to the same imaging conditions as those of the preliminary imaging, that is, the predetermined imaging conditions determined by automatic exposure control.

[0176] The control unit 33 sets the initial conditions of the brightest part 1108 to imaging conditions that do not include blown-out highlights. In other words, the imaging conditions are set so that the brightest part 1108 is imaged darker than the predetermined imaging conditions determined by the automatic exposure control. As a result, the set 1123a of the brightest part 1108 shown in FIG. 22(c) moves to a position adjacent to the predetermined upper limit value 1130 and to the left of the predetermined upper limit value 1130 (set 1123 shown by a solid line in FIG. 22(c)).

[0177] Furthermore, the control unit 33 sets the initial conditions of the main part 1110 to imaging conditions corresponding to the initial conditions of the darkest part 1109 and the brightest part 1108. Specifically, assuming that the absolute value of the difference between the mode of luminance of the darkest part 1109 and the mode of luminance of the main part 1110 under predetermined imaging conditions determined by automatic exposure control is D1, the absolute value of the difference between the mode of luminance of the main part 1110 under the same imaging conditions and the mode of luminance of the brightest part 1108 is D2, and the absolute value of the difference between the mode of luminance of the brightest part 1108 under predetermined imaging conditions determined by automatic exposure control and the mode of luminance of the brightest part 1108 under the above-mentioned initial conditions is db, the control unit 33 sets the initial conditions of the main part 1110 to imaging conditions under which the luminance of the main part 1110 becomes smaller (darker) by D4 defined by the following formula (2). D4 = (db × D1) / (D1 + D2) (2)

[0178] In other words, the initial conditions for the brightest part 1108 and the main part 1110 are set so as to prevent blown-out highlights while maintaining the relative relationship between the luminance of the darkest part 1109, the luminance of the main part 1110, and the luminance of the brightest part 1108 under predetermined imaging conditions determined by automatic exposure control. As a result, the set 1122a of the main part 1110 shown in Fig. 22(c) moves to the left on the luminance axis by D4 (set 1122 shown by a solid line in Fig. 22(c)).

[0179] When the user performs a change operation to change the imaging conditions of the main part 1110 so that the main part 1110 becomes darker (the brightness of the main part 1110 becomes lower), the control unit 33 changes the imaging conditions of the darkest part 1109 so that the brightness difference D1' between the group 1121 of the darkest part 1109 and the group 1122 of the main part 1110 is maintained. That is, the imaging conditions of the darkest part 1109 are changed to imaging conditions such that the darkest part 1109 is imaged darker. In other words, in FIG. 22(c), when the user tries to move the group 1122 of the main part 1110 to the left on the brightness axis, the control unit 33 moves the group 1121 of the darkest part 1109 to the left on the brightness axis in conjunction with this. That is, when the control unit 33 accepts an operation to change the imaging conditions of the main part 1110 from the user, it changes the imaging conditions of the main part 1110 and the imaging conditions of the darkest part 1109. The control unit 33 changes the imaging conditions for the main part 1110, thereby changing the imaging conditions for the darkest part 1109.

[0180] However, the control unit 33 changes the imaging conditions of the darkest part 1109 so that the luminance of the darkest part 1109 does not fall below a predetermined lower limit 1120. That is, the control unit 33 stops changing the imaging conditions of the darkest part 1109 when the luminance of the darkest part 1109 reaches the predetermined lower limit 1120. In other words, in FIG. 22(c), when the user tries to move the set 1122 of the main part 1110 leftward on the luminance axis, if the set 1121 of the darkest part 1109 exceeds the predetermined lower limit 1120, the control unit 33 moves only the set 1122 of the main part 1110 leftward on the luminance axis, and the set 1121 of the darkest part 1109 remains stationary and does not move. In this case, the luminance difference D1' between the set 1121 of the darkest part 1109 and the set 1122 of the main part 1110 changes.

[0181] Furthermore, the control unit 33 changes the imaging conditions of the main part 1110 so that the luminance of the main part 1110 does not fall below a predetermined lower limit 1120. That is, the control unit 33 stops changing the imaging conditions of the main part 1110 when the luminance of the main part 1110 reaches the predetermined lower limit 1120. In other words, in Fig. 22(c) , when the user attempts to move the set 1122 of the main part 1110 to the left on the luminance axis, if the set 1122 of the main part 1110 exceeds the predetermined lower limit 1120, the control unit 33 stops the set 1122 of the main part 1110 and does not move it.

[0182] Furthermore, when the user performs a change operation to change the imaging conditions of the main part 1110 so that the main part 1110 becomes brighter (the brightness of the main part 1110 becomes higher), the control unit 33 changes the imaging conditions of the darkest part 1109 so that the brightness difference between the group 1121 of the darkest part 1109 and the group 1122 of the main part 1110 is maintained. That is, the imaging conditions of the darkest part 1109 are changed to imaging conditions such that the darkest part 1109 is imaged brighter. In other words, in FIG. 22(c), when the user tries to move the group 1122 of the main part 1110 to the right on the brightness axis, the control unit 33 moves the group 1121 of the darkest part 1109 to the right on the brightness axis in conjunction with this. That is, when the control unit 33 accepts an operation to change the imaging conditions of the main part 1110 from the user, it changes the imaging conditions of the main part 1110 and the imaging conditions of the darkest part 1109. The control unit 33 changes the imaging conditions for the main part 1110, thereby changing the imaging conditions for the darkest part 1109.

[0183] However, the control unit 33 changes the imaging conditions of the main part 1110 so that the luminance of the main part 1110 does not exceed a predetermined upper limit value 1130. In other words, the control unit 33 stops changing the imaging conditions of the main part 1110 when the luminance of the main part 1110 reaches the predetermined upper limit value 1130. In other words, in Fig. 22(c), when the user attempts to move the set 1122 of the main part 1110 to the right on the luminance axis, if the set 1122 of the main part 1110 exceeds the predetermined upper limit value 1130, the control unit 33 stops and does not move the set 1122 of the main part 1110.

[0184] The above change operation does not change the imaging conditions of the brightest part 1108. In other words, in Fig. 22(c), regardless of whether the user attempts to move the set 1122 of the main part 1110 to the left or right on the luminance axis, the set 1123 of the brightest part 1108 does not move.

[0185] (D) When both black and white areas are present 22(d) shows a histogram of luminance for each unit group 32 in the case where both blocked up shadows and blown out highlights are present. In other words, in the case where the luminance of the main part 1110 is within a predetermined range (between the lower limit and the upper limit) but the luminance of the darkest part 1109 and the brightest part 1108 are outside that range, the set 1122a of the main part 1110 will be within the predetermined lower limit 1120 and the predetermined upper limit 1130, at least a part of the set 1121a of the darkest part 1109 will be located to the left beyond the predetermined lower limit 1120, and at least a part of the set 1123a of the brightest part 1108 will be located to the right beyond the predetermined upper limit 1130.

[0186] In this case, the control unit 33 sets the initial conditions of the darkest part 1109, the brightest part 1108, and the main part 1110 to imaging conditions that maintain the relative positional relationship (luminance relationship) and prevent crushed shadows and blown out highlights. That is, the control unit 33 sets the initial conditions of the darkest part 1109 to imaging conditions that make the darkest part 1109 brighter than the predetermined imaging conditions determined by automatic exposure control, and sets the initial conditions of the brightest part 1108 to imaging conditions that make the brightest part 1108 darker than the predetermined imaging conditions determined by automatic exposure control. As a result, the set 1121a of the darkest part 1109 and the set 1123a of the brightest part 1108 in the initial conditions shown by the dashed lines in FIG. 22(d) move within the ranges from the lower limit 1120 to the upper limit 1130 (sets 1121 and 1123 shown by solid lines in FIG. 22(d)). The initial conditions of the main portion 1110 are set to imaging conditions according to the initial conditions of the darkest portion 1109 and the initial conditions of the brightest portion 1108 .

[0187] Specifically, assuming that the absolute value of the difference between the mode of luminance of the darkest part 1109 and the mode of luminance of the main part 1110 under specified imaging conditions determined by automatic exposure control is D1, the absolute value of the difference between the mode of luminance of the main part 1110 and the mode of luminance of the brightest part 1108 under the same imaging conditions is D2, the absolute value of the difference between the mode of luminance of the darkest part 1109 under specified imaging conditions determined by automatic exposure control and the mode of luminance of the darkest part 1109 under the above-mentioned initial conditions is da, and the absolute value of the difference between the mode of luminance of the brightest part 1108 under specified imaging conditions determined by automatic exposure control and the mode of luminance of the brightest part 1108 under the above-mentioned initial conditions is db, the imaging conditions such that the luminance of the main part 1110 changes by D5 defined by the following equation (3) are set as the initial conditions of the main part 1110. D5=(db×D1−da×D2) / (D1+D2) ···(3)

[0188] In other words, initial conditions for the darkest part 1109, the main part 1110, and the brightest part 1108 are set so that the relative relationship between the luminance of the darkest part 1109, the luminance of the main part 1110, and the luminance of the brightest part 1108 under specified imaging conditions determined by automatic exposure control is maintained, and neither black crush nor white blowout occurs.

[0189] When the user performs a change operation to change the imaging conditions of the main part 1110, the control unit 33 changes only the imaging conditions of the main part 1110, without changing the imaging conditions of the darkest part 1109 or the brightest part 1108. In other words, in Fig. 22(d), when the user attempts to move the set 1122 of the main part 1110 in the left-right direction on the luminance axis, the control unit 33 moves only the set 1122 of the main part 1110 in the direction of the luminance axis desired by the user, while keeping the set 1121 of the darkest part 1109 and the set 1123 of the brightest part 1108 fixed.

[0190] However, the control unit 33 changes the imaging conditions of the main part 1110 so that the luminance of the main part 1110 does not fall below a predetermined lower limit 1120 or exceed a predetermined upper limit 1130. In other words, the control unit 33 prohibits changing the imaging conditions of the main part 1110 such that the luminance of the main part 1110 falls below the predetermined lower limit 1120, and prohibits changing the imaging conditions of the main part 1110 such that the luminance of the main part 1110 exceeds a predetermined upper limit 1130. In other words, in FIG. 22(d), when the user tries to move the set 1122 of the main part 1110 to the left on the luminance axis, if the set 1122 of the main part 1110 exceeds the predetermined lower limit 1120, the control unit 33 stops the set 1122 of the main part 1110 and does not move it. In addition, when the user attempts to move the set 1122 of the main part 1110 to the right on the brightness axis, if the set 1122 of the main part 1110 exceeds a predetermined upper limit value 1130, the control unit 33 will keep the set 1122 of the main part 1110 stationary and not move it.

[0191] In the above description, the initial values ​​of the imaging conditions are automatically determined by the control unit 33 using known automatic exposure control, but these may be manual values ​​designated by the user.

[0192] According to the imaging device according to the above-described fourth embodiment, the following advantageous effects can be obtained. (1) The image sensor 22 (imaging section) is configured to be able to set different imaging conditions for each of the multiple unit groups 32 (imaging regions). The control section 33 causes the unit group 32 for which a first imaging condition is set to image the main part 1110 of the subject, and causes a second unit group 32 for which a second imaging condition is set to image the darkest part 1109 and the brightest part 1108 of the subject. The control section 33 changes the first imaging condition in response to a user's operation, and changes the second imaging condition based on the change. In this way, it is possible to provide a user-friendly imaging device 3010 in which desired imaging conditions can be easily set for each imaging region.

[0193] (2) The control unit 33 sets initial values ​​of the first imaging condition and the second imaging condition such that the imaging signal of the unit group 32 capturing the main part 1110 and the imaging signal of the unit group 32 capturing the darkest part 1109 and the brightest part 1108 are both within the range of the lower limit value 1120 to the upper limit value 1130, prior to changing the first imaging condition by the user operation. This improves the dynamic range of the captured image. In other words, a captured image with a wide dynamic range can be obtained.

[0194] (3) The control unit 33 prohibits a change in the second imaging condition that would cause the imaging signal of the unit group 32 that captures the darkest part 1109 and the brightest part 1108 to deviate from the range between the lower limit value 1120 and the upper limit value 1130. Since this is done, a change in the imaging condition will not cause blocked up shadows or blown out highlights, and the user can freely set the imaging condition without worrying about blocked up shadows or blown out highlights.

[0195] (4) The control unit 33 prohibits a change in the second imaging condition that would result in an exchange of magnitude between the imaging signal of the main portion 1110, the imaging signal of the darkest portion 1109, and the imaging signal of the brightest portion 1108. This prevents the creation of an image with unnatural brightness.

[0196] (5) The control unit 33 determines the amount of change in the imaging conditions for the darkest part 1109 and the brightest part 1108, based on the amount of change in the imaging conditions for the main part 1110. In this manner, desired imaging conditions can be easily set for each imaging region.

[0197] The following modifications are also within the scope of the present invention, and one or more of the modifications may be combined with the above-described embodiment.

[0198] (Variation 1) We have explained how the imaging conditions of the darkest part 1109 and the brightest part 1108 are changed and what changes are prohibited in response to a change operation that changes the imaging conditions of the main part 1110, but the above explanation is just one example, and the imaging conditions may be changed in a different pattern or changes to the imaging conditions may be prohibited.

[0199] For example, when the user performs a change operation to change the imaging conditions of the main part 1110 so that the main part 1110 is imaged brighter, the control unit 33 may change the imaging conditions of the darkest part 1109 and the brightest part 1108 so that both the darkest part 1109 and the brightest part 1108 are imaged brighter (or darker). That is, when the control unit 33 accepts an operation to change the imaging conditions of the main part 1110 from the user, it changes the imaging conditions of the main part 1110, the imaging conditions of the brightest part 1108, and the imaging conditions of the darkest part 1109. The control unit 33 changes the imaging conditions of the brightest part 1108 and the imaging conditions of the darkest part 1109 by changing the imaging conditions of the main part 1110. At this time, the brightness step (e.g., the number of shutter speed steps) changed by the user may be different from the brightness step (e.g., the number of shutter speed steps) changed in the darkest part 1109 and the brightest part 1108.

[0200] However, whatever the pattern, imaging conditions that cause crushed shadows in the darkest part 1109 or blown highlights in the brightest part 1108 should be prohibited. Also, it is desirable to avoid a pattern in which, in conjunction with a change in the imaging conditions for the main part 1110 (for example, increasing the shutter speed of the main part 1110 by one step), the same change is made to the imaging conditions for both the darkest part 1109 and the brightest part 1108 (for example, increasing the shutter speed of both the darkest part 1109 and the brightest part 1108 by one step).

[0201] (Variation 2) Although it has been described above that when the imaging conditions are changed, the changes are reflected in the subject image displayed on the liquid crystal monitor 24, a difference may be set in the timing at which the changes are reflected between the main portion 1110 and the other portions (the darkest portion 1109 and the brightest portion 1108). For example, when the imaging conditions are changed, the changes may be reflected first only in the main portion 1110, and then several seconds later, the changes may be reflected in the darkest portion 1109 and the brightest portion 1108.

[0202] (Variation 3) In the change of the imaging conditions, the explanation has been given based on the most frequent value of the luminance of the darkest part 1109, the most frequent value of the luminance of the brightest part 1108, and the most frequent value of the luminance of the main part 1110, but these values ​​do not have to be the most frequent value. For example, they may be the median value or the average value of the luminance.

[0203] (Variation 4) In the above-mentioned (C) case where blown-out highlights are present but no crushed shadows are present, the amount of change in the imaging conditions may be varied depending on the type of subject located in the brightest part 1108. That is, when the control unit 33 receives an operation to change the imaging conditions of the main part 1110 from the user, it changes the imaging conditions of the main part 1110 and the imaging conditions of the brightest part 1108. In such a case, the imaging conditions of the brightest part 1108 may be changed by an amount different from the amount of change in the imaging conditions of the main part 1110. For example, the control unit 33 is caused to determine the subject located in the brightest part 1108 from information such as the imaging distance. Here, when it is determined that the brightest part 1108 includes sky, in response to a change operation to change the imaging conditions of the main part 1110 so that the main part 1110 is imaged brighter, the control unit 33 changes the imaging conditions of the sky part (brightest part 1108) by the same amount so that the brightness of the sky, i.e., the brightness of the brightest part 1108 is imaged smaller (darker).

[0204] Similarly, when it is determined that a building is present in the brightest part 1108, in response to a change operation for changing the imaging conditions of the main part 1110 so that the main part 1110 is imaged brighter, the control unit 33 changes the imaging conditions of the part of the building (the brightest part 1108) so that the luminance of the building, i.e., the luminance of the brightest part 1108, is imaged smaller (darker). At this time, the amount of change is made smaller than the amount of change of the main part 1110. For example, when an operation for slowing down the shutter speed of the main part 1110 by one step is performed, the shutter speed of the brightest part 1108 is made faster by 1 / 3 step. However, it is desirable to change the imaging conditions of the brightest part 1108 so that image blurring does not occur. In other words, when the possibility of image blurring is higher than a certain level, the change of the imaging conditions of the brightest part 1108 is prohibited.

[0205] By doing so, it is possible to obtain a vivid imaging result in which, for example, the blue of the sky is more emphasized.

[0206] (Variation 5) In the above-mentioned (B) case where there is crushed shadows and there is no blown highlights, the amount of change in the imaging conditions may be varied depending on the type of subject located in the darkest part 1109. That is, when the control unit 33 receives an operation to change the imaging conditions of the main part 1110 from the user, it changes the imaging conditions of the main part 1110 and the imaging conditions of the darkest part 1109. In such a case, the imaging conditions of the darkest part 1109 may be changed by an amount different from the amount of change in the imaging conditions of the main part 1110. For example, the control unit 33 is caused to determine the subject located in the darkest part 1109 from information such as the imaging distance. Here, when it is determined that a building is present in the darkest part 1109, in response to a change operation to change the imaging conditions of the main part 1110 so that the main part 1110 is imaged brighter, the control unit 33 changes the imaging conditions of the building part (darkest part 1109) by the same amount so that the luminance of the building, i.e., the luminance of the darkest part 1109 is imaged larger (brighter). However, it is desirable to change the imaging conditions of the darkest part 1109 so as not to cause image blurring. In other words, when the possibility of image blurring becoming higher than a certain level, changing the imaging conditions of the darkest part 1109 is prohibited.

[0207] Similarly, when it is determined that the darkest part 1109 includes the sky (night sky), in response to a change operation for changing the imaging conditions of the main part 1110 so that the main part 1110 is imaged brighter, the control unit 33 changes the imaging conditions of the sky (darkest part 1109) so that the brightness of the sky, i.e., the brightness of the darkest part 1109, is imaged larger (brighter). At this time, the amount of change is made smaller than the amount of change of the main part 1110. For example, when an operation for slowing down the shutter speed of the main part 1110 by one step is performed, the shutter speed of the darkest part 1109 is slowed by 1 / 3 step. However, it is desirable to change the imaging conditions of the darkest part 1109 so that high-sensitivity noise and image blur do not occur. In other words, when the possibility of high-sensitivity noise and image blurring is higher than a certain level, the change of the imaging conditions of the darkest part 1109 is prohibited.

[0208] In this way, it is possible to prevent an unnatural captured image from appearing too bright, for example, in the night sky.

[0209] (Variation 6) In the above-described embodiment, the main part of the subject and the darkest and brightest parts, which are the background of the subject, are distinguished based on the brightness distribution of the subject, and the imaging conditions of the darkest and brightest parts, which are the background, are changed according to the change in the imaging conditions of the main part. The main part of the subject and the background of the subject may be distinguished based on information other than the brightness distribution of the subject. For example, the main part of the subject and the background may be distinguished based on the speed distribution (movement distribution) of the subject.

[0210] For example, the part of the subject that moves faster can be regarded as the main part, and the part that moves slower can be regarded as the background part (such as when capturing an image of a vehicle). In this case, when the control unit 33 receives an operation from the user to change the imaging conditions for the main part, it changes the imaging conditions for the main part and the imaging conditions for the background part. In such a case, the imaging conditions for the background part may be changed by a different amount from the amount of change in the imaging conditions for the main part. For example, by changing the imaging conditions to speed up the shutter speed of the main part, the control unit 33 also speeds up the shutter speed of the background part accordingly, but the amount of change is smaller than that of the main part. Also, by changing the imaging conditions to slow down the shutter speed of the main part, the shutter speed of the background part is also slowed down accordingly, but the amount of change is larger than that of the main part.

[0211] As another example, the part of the subject that moves slowly can be regarded as the main part, and the part that moves quickly can be regarded as the background part. In this case, when the control unit 33 receives an operation from the user to change the imaging conditions for the main part, it changes the imaging conditions for the main part and the imaging conditions for the background part. In such a case, the imaging conditions for the background part may be changed by a different amount from the amount of change in the imaging conditions for the main part. For example, when the control unit 33 increases the shutter speed for the main part, it also increases the shutter speed for the background part accordingly, but the amount of change is smaller than that for the main part. Also, when the control unit 33 decreases the shutter speed for the main part, it also decreases the shutter speed for the background part accordingly, but the amount of change is larger than that for the main part.

[0212] In either case, it is desirable to set a lower limit for the shutter speed of the background to prevent image blurring. It is also desirable to prohibit any change that would result in the shutter speed of the main part being higher than that of the background.

[0213] (Variation 7) In the above-described embodiment, the imaging conditions are changed by accepting a predetermined change operation on the operation member, but the change operation of the imaging conditions is not limited to this. For example, an operation screen used to change the imaging conditions may be displayed on the liquid crystal monitor 24 having a touch sensor that accepts an input operation from the user. Specifically, as shown in FIG. 23, an operation image 1111 for accepting an operation to change the imaging conditions is displayed on the liquid crystal monitor 24. The operation image 1111 includes an adjustment bar 1112 and a slider 1113. The setting of the imaging conditions is determined by the position of the slider 1113 relative to the adjustment bar 1112. The change operation of the imaging conditions is performed by the user operating the slider. For example, when the brightness of the main part is to be changed as the change operation of the imaging conditions, the position of the slider 1113 relative to the adjustment bar 1112 is moved. When the user moves the slider 1113 to the left end side of the adjustment bar, the imaging conditions are set so that the main part is darker than the current imaging conditions. In this case, the imaging conditions of the imaging area other than the main part are changed by changing the imaging conditions of the main part. Also, when the user moves the slider 1113 to the right end of the adjustment bar, the imaging conditions of the main part are set so that the imaging conditions are brighter than the current imaging conditions. In this case, the imaging conditions of the imaging area other than the main part are changed by changing the imaging conditions of the main part.

[0214] (Variation 8) In the above-described embodiment, the imaging conditions are changed for the main part of the subject and the darkest and brightest parts that are the background of the subject, but the subject for which the imaging conditions are changed is not limited to this. For example, as shown in the first embodiment, the second embodiment, and the third embodiment, when multiple subjects are extracted by the object recognition unit 23a, the imaging conditions may be changed for the selected main subject and the other subjects among the multiple subjects. That is, by changing the imaging conditions for one subject among the multiple subjects extracted by the object recognition unit 23a, the imaging conditions for the other extracted subjects may be changed. In this case, the amount of change in the imaging conditions may be different between the selected subject and the other subjects among the subjects extracted by the object recognition unit 23a. Also, when one subject is selected by the user, the imaging conditions for the selected subject may be changed to change the imaging conditions for the other subjects (for example, a person, a face, a building, the sky, etc.) included in the subject. In this way, the fourth embodiment can be applied to the first embodiment, the second embodiment, and the third embodiment. This allows the time required for changing the imaging conditions to be reduced even when multiple subjects are present, since changing the imaging conditions for one subject can change the imaging conditions for the other subjects, and also reduces the time required for selecting the subject for which the imaging conditions are to be changed.

[0215] (Variation 9) In each of the above-described embodiments, an imaging device that is a single electronic device having the imaging element 22 and the control unit 33 has been described, but the present invention is not limited to such embodiments. For example, the present invention can also be applied to an electronic device that controls an externally provided imaging element 22. Below, a detailed description is given of an embodiment in which the imaging element 22 is provided externally.

[0216] FIG. 24 is a block diagram showing a schematic configuration of an imaging system according to the ninth modification. The imaging system 4000 shown in FIG. 24 includes an imaging unit 4001 and an electronic device 4002. The imaging unit 4001 includes the imaging optical system 21 and the imaging element 22 described in the fourth embodiment, and further includes a first communication unit 4003. The electronic device 4002 includes the control unit 33, the liquid crystal monitor 24, the memory card 25, the operation unit 26, the DRAM 27, the flash memory 28, and the sound recording unit 34 described in the fourth embodiment, and further includes a second communication unit 4004. The first communication unit 4003 and the second communication unit 4004 can perform bidirectional data communication, for example, by known wireless communication technology or optical communication technology.

[0217] In the imaging system 4000 according to the ninth modification, the control unit 33 controls the imaging element 22 by data communication via the second communication unit 4004 and the first communication unit 4003. For example, by transmitting and receiving predetermined control data to and from the imaging unit 4001, different imaging conditions are set for each unit group 32, and imaging signals are read out from each unit group 32.

[0218] The present invention is not limited to the above-mentioned embodiment, and includes electronic devices of various structures that include a display control unit that displays a subject image captured by an imaging unit controlled under predetermined imaging conditions for each of a plurality of imaging regions on a display unit, and a division unit that divides the subject image into at least one partial region, and the display control unit displays the partial image captured by the imaging unit of the imaging region corresponding to the partial region and the subject image on the display unit. The present invention is not limited to the above-mentioned embodiment, and includes electronic devices of various structures that include the above-mentioned electronic device, an imaging unit controlled under predetermined imaging conditions for each of a plurality of imaging regions, an imaging control unit that controls under predetermined imaging conditions for each of a plurality of imaging regions, and a recording unit that records the subject image captured by the imaging unit on a recording medium, and the imaging control unit controls the corresponding imaging region under the imaging conditions set by the setting unit to generate a subject image, and the recording unit records the subject image on a recording medium. Furthermore, the present invention is not limited to the above-mentioned embodiments, and includes imaging systems of various structures that include the above-mentioned electronic device and an imaging device, the electronic device further including a first communication unit capable of communicating with the imaging device, the imaging device further including an imaging unit controlled under predetermined imaging conditions for each of a plurality of imaging areas, an imaging control unit that controls under predetermined imaging conditions for each of the plurality of imaging areas to generate a subject image, and a second communication unit capable of communicating with the electronic device, the second communication unit transmitting the subject image to the electronic device via the first communication unit, and the first communication unit transmitting to the imaging device the imaging conditions set by the setting unit for each partial area in the subject image displayed by the display control unit.

[0219] As described above, in the imaging system 4000, it is the control unit 33 that controls each unit group 32. The electronic device 4002 does not include an imaging element 22, but performs the same control as in the first embodiment by controlling the imaging element 22 (imaging unit 4001) provided outside the electronic device 4002. In other words, the present invention can be applied to electronic devices that do not include an imaging element 22.

[0220] As long as the characteristics of the present invention are not impaired, the present invention is not limited to the above-described embodiments, and other forms that are conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention.

[0221] The disclosures of the following priority applications are incorporated herein by reference: Japanese Patent Application No. 202104, 2014 (filed September 30, 2014) Japanese Patent Application No. 202105, 2014 (filed September 30, 2014) [Explanation of symbols]

[0222] 10 information terminal device, 21 imaging optical system, 22 imaging element, 23, 33 control unit, 23a object recognition unit, 23b weighting setting unit, 23c area division unit (area division unit), 23d imaging condition setting unit, 23e image processing unit, 23f imaging control unit, 24 liquid crystal monitor, 25 memory card, 26 operation unit, 27 DRAM, 28 flash memory, 29 attitude detection unit, 34 sound recording unit, 3010 imaging device

Claims

1. a control unit that causes a display unit to display a first image of a subject captured in a first region of an imaging element and a second image of the subject captured in a second region of the imaging element; an operation unit that receives a first operation for selecting an area for setting an imaging condition for controlling the imaging element from the first area and the second area; Equipped with The operation unit accepts the first operation on the display unit, the control unit causes the display unit to display a third image used to set an area selected by the first operation out of the first area and the second area to an imaging condition different from an area not selected by the first operation out of the first area and the second area; and The third image is a function of selecting a setting item for setting the imaging condition of the area selected by the first operation out of the first area and the second area; a function of changing settings of the imaging conditions of the area selected by the first operation out of the first area and the second area; a function of displaying settings of the imaging conditions of the area selected by the first operation out of the first area and the second area; having at least one function of electronic equipment.

2. 2. The electronic device according to claim 1, The control unit causes the display unit to display the first image and the second image side by side, The operation unit is an electronic device that accepts, as the first operation, an operation of selecting one of the first image and the second image displayed on the display unit.

3. 2. The electronic device according to claim 1, the control unit causes the display unit to sequentially display the first image and the second image; The operation unit is an electronic device that accepts, as the first operation, an operation of selecting one of the first image and the second image displayed on the display unit.

4. 4. The electronic device according to claim 2, The operation unit is an electronic device that displays, on the display unit, an enlarged image of an image selected by the first operation between the first image and the second image when the third image is displayed on the display unit.

5. 5. The electronic device according to claim 1, The operation unit is an electronic device that accepts a second operation for setting the imaging conditions of an area selected by the first operation out of the first area and the second area.

6. 6. The electronic device according to claim 5, The control unit sets the imaging conditions for an area selected by the first operation out of the first area and the second area based on the second operation.

7. 7. The electronic device according to claim 5, The operation unit is an electronic device that accepts an operation using the third image as the second operation.

8. 8. The electronic device according to claim 7, The control unit is an electronic device that displays the third image on the display unit around an image of an area selected by the first operation out of the first area and the second area.

9. 9. The electronic device according to claim 8, The control unit is an electronic device that displays, on the display unit, an icon image for setting the imaging conditions of the area selected by the first operation between the first area and the second area as the third image.

10. 10. The electronic device according to claim 1, a photoelectric conversion unit that converts light into an electric charge is disposed in each of the first region and the second region; The control unit is an electronic device that controls, as the imaging condition, a storage time of the electric charges converted by the photoelectric conversion unit.

11. 11. The electronic device according to claim 1, a photoelectric conversion unit that converts light into an electric charge is disposed in each of the first region and the second region; The control unit is an electronic device that controls, as the imaging condition, a frame rate for reading out a signal based on the electric charge converted by the photoelectric conversion unit.

12. 12. The electronic device according to claim 10, an amplifier that amplifies a signal based on the charge converted by the photoelectric conversion unit, The control unit is an electronic device that controls an amplification rate of the amplifier unit as the imaging condition.

13. 13. The electronic device according to claim 1, a photoelectric conversion unit that converts light into an electric charge is disposed in each of the first region and the second region; An electronic device in which the photoelectric conversion unit arranged in the first region and the photoelectric conversion unit arranged in the second region are arranged side by side in a row direction.

14. 14. The electronic device according to claim 13, The first region and the second region each include a plurality of the photoelectric conversion units arranged therein; An electronic device in which the plurality of photoelectric conversion units arranged in the first region and the plurality of photoelectric conversion units arranged in the second region are arranged side by side in the row direction.

Images