Imaging device and its control method

Abstract

To adaptively create a high-quality image with a wider dynamic range even when a subject has a large contrast difference and a large amount of motion.SOLUTION: An electronic apparatus has: an image pickup device; driving means that can drive the image pickup device by a plurality of driving methods including a first driving method and a second driving method by which a plurality of images different in dynamic range from each other can be acquired; selection means that selects any one of the plurality of driving methods; and creation means that, from the plurality of images acquired by driving the image pickup device by the first driving method or the second driving method selected by the selection means, creates one image with a wider dynamic range than those of the images.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention Imaging device relates to an image sensor and its control method, and particularly to a technique for generating a high-dynamic range image.

Background Art

[0002] Conventionally, techniques for expanding the dynamic range related to gradation have been proposed using image sensors such as CCD image sensors and CMOS image sensors used in general digital cameras.

[0003] In the imaging device of Patent Document 1, a technique for expanding the dynamic range is disclosed by applying a plurality of different amplification gains to the signals of the same pixel, simultaneously reading them out, and synthesizing them by signal processing. Further, in Patent Document 2, a technique for expanding the dynamic range is disclosed by switching the number of shootings according to the detected state of the subject, the shooting mode, etc., performing a plurality of shootings, and synthesizing them by signal processing.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] [[ID=Y45]] If the difference in amplification gain is increased for shooting in order to further expand the dynamic range by the method disclosed in Patent Document 1, black crushing of pixels where the image signal is below a predetermined output level and white blooming of pixels exceeding the predetermined output level are less likely to occur. On the other hand, the gradation of pixels at the intermediate output level is lost.

[0006] Furthermore, if the number of shots is increased to further expand the dynamic range using the method disclosed in Patent Document 2, an unnatural composite image may be generated when the subject is moving.

[0007] This invention was made in view of the above-mentioned problems, and aims to adaptively generate high-quality images with a wider dynamic range, even when there is a large difference in brightness or motion of the subject. [Means for solving the problem]

[0008] To achieve the above objective, the present invention provides an imaging device comprising: an image sensor; a control means that, when a first shooting mode is set, controls the image sensor to acquire multiple images by taking multiple images while changing the exposure; when a second shooting mode different from the first shooting mode is set, controls the image sensor to acquire multiple images by amplifying the image signal obtained by taking one image with multiple different gains within the image sensor; and when a third shooting mode different from the first and second shooting modes is set, controls the image sensor to acquire multiple images by multiplying one of multiple image signals obtained by taking images a fewer number of times than the number of images taken in the first shooting mode by multiple different gains; A generation means for generating a single image with a wider dynamic range than each of the multiple images acquired in any of the first to third shooting modes, It has. [Effects of the Invention]

[0009] According to the present invention, even when there is a large difference in brightness or motion of the subject, it is possible to adaptively generate high-quality images with a wider dynamic range. [Brief explanation of the drawing]

[0010] [Figure 1] A block diagram showing the schematic configuration of an imaging device according to the first embodiment of the present invention. [Figure 2] A diagram showing the schematic configuration of an image sensor according to the first embodiment. [Figure 3] A flowchart of the HDR shooting mode setting process according to the first embodiment. [Figure 4] A diagram showing an example of a user interface display according to the first embodiment. [Figure 5] A flowchart of the HDR shooting process according to the first embodiment. [Figure 6] A block diagram showing the schematic configuration of an imaging device according to the second embodiment. [Figure 7] A flowchart of the HDR shooting mode setting process according to the second embodiment. [Figure 8] A diagram showing an example of a user interface display according to the second embodiment. [Figure 9] A flowchart of the HDR shooting mode automatic setting process according to the second embodiment. [Figure 10A] A diagram showing an example of an HDR shooting mode table according to the second embodiment. [Figure 10B] A diagram showing another example of an HDR shooting mode table according to the second embodiment. [Figure 11] A flowchart of the HDR shooting process according to the second embodiment. [Figure 12] A diagram showing an example of a user interface display according to the third embodiment. [Figure 13] A flowchart of the process for changing the HDR shooting mode according to the third embodiment. [Figure 14] A diagram showing an example of a user interface display according to the third embodiment. [Figure 15] A diagram showing an example of a user interface display according to the third embodiment. [[ID=三十六]] [Figure 16] A diagram showing an example of a user interface display according to the third embodiment.

Modes for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0012] <First Embodiment> [Configuration of Imaging Device] FIG. 1 is a block diagram showing a schematic configuration of an imaging device 100 according to the first embodiment of the present invention. The imaging device 100 may be an electronic device having a camera function. For example, it may be a camera such as a digital camera and a digital video camera, or may be a mobile phone with a camera, a computer with a camera, a game machine, or the like.

[0013] In FIG. 1, the photographing lens 101 is an interchangeable lens unit that can be attached to the main body of the imaging device 100, or a lens unit incorporated in the main body, and is composed of a plurality of lens groups such as a focus lens and a zoom lens, and an aperture and the like.

[0014] The imaging element 102 is composed of a CMOS image sensor, a CCD image sensor, or the like having a plurality of pixels, and performs photoelectric conversion on each pixel for the optical image of the subject imaged by the photographing lens 101 to generate charges corresponding to the incident light amount. Although details will be described later, the imaging element 102 of the present embodiment can be driven by a driving method that outputs one image signal by applying a single gain to the voltage signal corresponding to the charges generated in each pixel, and a driving method that outputs a plurality of image signals by applying a plurality of different gains to the charges of each pixel. 。 [[ID=IP19]]

[0015] The image acquisition unit 103 acquires the image signal output from the imaging element 102, temporarily holds the acquired image signal, and performs photometry using the acquired image signal.

[0016] The image synthesis unit 104 generates an HDR (High Dynamic Range) image from the image signal based on the output of the image sensor 102, which is temporarily held in the image acquisition unit 103, using a known synthesis method. For example, for low-luminance image portions below a predetermined signal level, a high-gain image signal or an image signal obtained by overexposure is used, and for high-luminance image portions exceeding a predetermined signal level, a low-gain image signal or an image signal obtained by underexposure is used for synthesis. It is preferable that the normal image used for the signal in the dark areas of the synthesized image has suppressed random noise in the dark areas. If HDR synthesis is not performed, the image signal held in the image acquisition unit 103 is output as is.

[0017] The image processing unit 105 performs various signal processing, such as gamma processing, color signal processing, and exposure correction processing, on the image signal output from the image synthesis unit 104, and outputs the processed image signal.

[0018] The image recording unit 106 records the image signal processed by the image processing unit 105 onto a storage device or storage medium. As the storage device or storage medium, for example, a memory device that can be mounted on the imaging device 100 can be used.

[0019] The control unit 107 includes operating elements such as a release button, a mode switching dial, a zoom control lever, and a touch panel. By operating the control unit 107, the user can input various instructions to the imaging device 100. User input via the control unit 107 is notified to the system control unit 110. The user can also set the imaging device 100 to the HDR shooting mode, which will be described later, by operating the control unit 107. This process will be described later with reference to Figures 3 and 4.

[0020] The memory unit 108 is a memory unit that stores the content of the user's instructions to the imaging device 100, and is composed of an electrically erasable and recordable non-volatile memory.

[0021] The display unit 109 is for displaying captured images, information at the time of shooting, and a user interface for operation by the operation unit 107, and is composed of, for example, a TFT-LCD. Furthermore, by providing a touch panel on the front of the display unit 109, it together with the display unit 109 forms part of the operation unit 107. Alternatively, it may be composed of an EVF with eye-tracking functionality, and by detecting the user's gaze, it may also form part of the operation unit 107.

[0022] The system control unit 110 controls the image sensor control unit 111 and the lens control unit 112 based on the image signal and photometric results held in the image acquisition unit 103, as well as input from the user via the operation unit 107.

[0023] The image sensor control unit 111 controls the drive of the image sensor 102 according to the control signal from the system control unit 110. The lens control unit 112 controls the drive of the photographic lens 101 according to the control signal from the system control unit 110.

[0024] [Image sensor configuration] Next, the configuration of the image sensor 102 will be described. Figure 2 is a block diagram showing the schematic configuration of the image sensor 102 according to this embodiment, and will be described here as a CMOS image sensor.

[0025] As shown in Figure 2, multiple unit pixels 1100 are arranged in a matrix within the pixel region 1108. Note that in Figure 2, for simplicity, n+1 pixels are shown horizontally and 4 pixels vertically; however, in reality, a very large number of pixels are arranged vertically as well. Furthermore, each unit pixel 1100 is provided with one of several optical color filters (in this embodiment, as an example, red, green, and blue). In Figure 2, a unit pixel 1100 equipped with a red color filter and primarily receiving red light is labeled R, a unit pixel 1100 equipped with a green color filter and primarily receiving green light is labeled G, and a unit pixel 1100 equipped with a blue color filter and primarily receiving blue light is labeled B. Each unit pixel 1100 having one of the three color filters is arranged, for example, according to a Bayer array.

[0026] The vertical scanning circuit 1103 sends drive pulses to the unit pixels 1100 of each row through drive signal lines 1102 that are wired commonly for each row. For the sake of simplicity in the illustration, only one drive signal line 1102 is shown for each row, but in reality, multiple drive signal lines are wired for each row.

[0027] Each unit pixel 1100 in the same column is connected to a common vertical output line (column output line) 1101, and the signal from each pixel is input to a common column circuit 1104 for each column via this vertical output line 1101. The column circuit 1104 is connected to the vertical scanning circuit 1103 via a column circuit signal line 1105, and performs processing to apply a gain set according to the instructions of the system control unit 110 to the signal from each pixel. As a result, the column circuit 1104 can perform processing to output the signal from each pixel with a single gain, and processing to output the signal from each pixel with multiple different gains. Furthermore, the column circuit 1104 performs A / D conversion processing on the gain-applied analog signal and outputs a digital signal.

[0028] The digital signal output from the column circuit 1104 is input to the horizontal transfer circuit 1107. The horizontal transfer circuit 1107 outputs the input digital image signal to the image acquisition unit 103.

[0029] [Drive method] In the first embodiment, there is a first driving method and a second driving method as a method for driving the imaging device 100 to acquire multiple images used for dynamic range expansion processing. Then, depending on the HDR shooting mode described later, the first driving method and the second driving method are switched to drive the imaging device 100 with either driving method, and an HDR image is generated by combining the multiple images obtained.

[0030] ● First driving method First, let's explain the first driving method. Here, we will describe the case where the image sensor 102 takes three (multiple) consecutive shots to acquire three images with different dynamic ranges.

[0031] The system control unit 110 controls the aperture of the photographic lens 101 via the lens control unit 112, and the vertical scanning circuit 1103 of the image sensor 102 via the image sensor control unit 111. For the first image acquisition, it controls the aperture of the photographic lens 101 and / or the charge accumulation time of each unit pixel 1100 so that the brightness of the main subject is appropriate. Furthermore, it controls the column circuit 1104 to apply an appropriate gain to the pixel signal obtained from each unit pixel 1100.

[0032] In the second image acquisition, the aperture of the imaging lens 101 and / or the charge accumulation time of each unit pixel 1100 are controlled so that the brightness of the dark areas of the subject is appropriate. At this time, the column circuit 1104 may be configured to apply a larger gain to the pixel signal than in the first image. In the third image acquisition, the aperture of the imaging lens 101 and / or the charge accumulation time of each unit pixel 1100 are controlled so that the brightness of the bright areas of the subject is appropriate. At this time, the column circuit 1104 may be configured to apply a smaller gain to the pixel signal than in the first image.

[0033] As described above, the system control unit 110 controls the image sensor 102 and the photographic lens 101 via the image sensor control unit 111 and the lens control unit 112, thereby enabling the acquisition of three images with different dynamic ranges in three shooting sessions.

[0034] ●Second drive method Next, we will describe a second driving method. Here, we will describe a case in which two images with different dynamic ranges are acquired by applying two types of gains to the image signal obtained by taking one shot with the image sensor 102.

[0035] The system control unit 110 controls the aperture of the photographic lens 101 via the lens control unit 112 and the vertical scanning circuit 1103 of the image sensor 102 via the image sensor control unit 111 to acquire a pixel signal from each unit pixel 1100. The system control unit 110 then controls the column circuit 1104 via the column circuit signal line 1105 so that it applies an appropriate gain to the acquired pixel signal, and outputs the first image. Next, the system control unit 110 controls the column circuit 1104 via the column circuit signal line 1105 so that it applies a different gain to the same pixel signal as the first image, and outputs the second image. In this way, by passing a single pixel signal through the column circuit 1104 twice with different gains applied, it is possible to acquire two images with different dynamic ranges.

[0036] Although the above example uses three shots and two types of gain, the present invention is not limited to three shots and two types of gain. For example, four or more images with different dynamic ranges may be obtained by taking four or more consecutive shots, or three or more images with different dynamic ranges may be obtained by applying three or more different gains to the image signal obtained in a single shot.

[0037] Furthermore, the first and second driving methods described above have an advantage in terms of image quality because the number of pixels in each resulting image is the same as in normal shooting. On the other hand, since the column circuit 1104 and the horizontal transfer circuit 1107 are driven for a longer time than in normal shooting, the readout time is extended and power consumption increases, which can lead to a decrease in frame rate during continuous shooting and an impact on recording time during video recording. In contrast, the second driving method can also be driven as follows.

[0038] Specifically, the system control unit 110 controls the vertical scanning circuit 1103 to control the charge storage time for even-numbered rows of unit pixels 1100 so that the brightness of the main subject is appropriate, and to control the column circuit 1104 to apply an appropriate gain to the pixel signal. For odd-numbered rows of unit pixels 1100, the system control unit 110 controls the charge storage time so that the dark areas of the subject are brightened, and to control the column circuit 1104 to apply a different gain to the pixel signal than that of the even-numbered rows. In this way, the system control unit 110 instructs the image sensor 102 and the imaging lens 101 via the image sensor control unit 111 and the lens control unit 112 to set the controls, thereby outputting two images with different dynamic ranges in a single shot.

[0039] While this method halves the vertical pixel count of the resulting image, it allows for shooting with the same readout time as normal shooting, thus minimizing the impact on frame rates during continuous shooting or video recording. Alternatively, the pixel count can be restored by interpolating pixels from two images using image processing.

[0040] [HDR shooting mode setting process] Next, the HDR shooting mode setting process by the system control unit 110 of the imaging device 100 according to the first embodiment will be described with reference to Figures 3 and 4.

[0041] Figure 3 shows a flowchart of the HDR shooting mode setting process, which is performed in response to user operations on the operation unit 107 based on the content displayed on the display unit 109. Figure 4 shows the content displayed on the display unit 109 at that time. This process is initiated when the HDR shooting mode setting process is instructed via a menu button (not shown) or touch panel included in the operation unit 107.

[0042] First, in the flowchart of Figure 3, when the user starts setting the HDR shooting mode, in S201, the system control unit 110 displays the user interface (UI) 300 shown in Figure 4 on the display unit 109. The UI 300 shows the HDR shooting mode setting screen displayed on the display unit 109. Option 301 (Brightness Priority) indicates a mode that prioritizes the brightness of the subject during HDR shooting, resulting in a wider dynamic range. Option 302 (Motion Priority) is a mode that is effective when the subject is moving a lot or fast, and option 303 (OFF) indicates the setting when HDR shooting is not performed. Arrow 304 indicates the currently selected setting. Memo 305 also indicates the characteristics of the currently selected mode.

[0043] In S202, the user operates the control unit 107 to move the arrow 304 displayed on the UI 300, thereby selecting the desired option from options 301 to 303. If the user selects option 303 (OFF), the process proceeds to S203, and the system control unit 110 stores the HDR shooting OFF setting in the memory unit 108. If the user selects option 301 (Brightness Priority), the process proceeds to S204, and the system control unit 110 stores the Brightness Priority HDR shooting mode in the memory unit 108. If the user selects option 302 (Motion Priority), the process proceeds to S205, and the system control unit 110 stores the Motion Priority HDR shooting mode in the memory unit 108.

[0044] In S206, if the user does not instruct the system to end the HDR shooting mode setting process, the system returns to S202 and continues the HDR shooting mode setting process. When the user instructs the system to end the HDR shooting mode setting process, the system terminates with the selected HDR shooting mode setting stored in the storage unit 108.

[0045] [HDR shooting processing] Next, the processing in the system control unit 110 of the imaging device 100 according to the first embodiment during HDR shooting will be described with reference to Figure 5.

[0046] Figure 5 is a flowchart of the HDR shooting process performed based on the HDR shooting mode stored in the memory unit 108. This process starts when HDR shooting is instructed via the release button (not shown) or touch panel included in the operation unit 107. Furthermore, if the process is performed via S203 in Figure 3, i.e., if the HDR shooting OFF setting is selected, this process is not performed.

[0047] When HDR shooting is started, in S401, the system control unit 110 reads the HDR shooting mode stored in the memory unit 108.

[0048] In S402, the system control unit 110 checks the HDR shooting mode read from the memory unit 108 in S401. If it is the brightness-priority HDR shooting mode, it proceeds to S403; if it is the motion-priority HDR shooting mode, it proceeds to S404.

[0049] In S403, the system control unit 110 instructs the image sensor control unit 111 and the lens control unit 112 to take pictures using a drive method associated with the brightness-priority HDR shooting mode. Specifically, it instructs the system control unit 110 to drive the shooting lens 101 and the image sensor 102 using the first drive method to take three consecutive shots and output three images with different dynamic ranges.

[0050] Meanwhile, in S404, the system control unit 110 instructs the image sensor control unit 111 and the lens control unit 112 to shoot using a drive method associated with motion-priority HDR shooting mode. Specifically, it instructs the system control unit 110 to drive the shooting lens 101 and the image sensor 102 using a second drive method, and to set up the system to output two images with different dynamic ranges by applying two different types of gain to the pixel signal obtained in a single shot.

[0051] In S405, the image sensor control unit 111 and the lens control unit 112 control the photographic lens 101 and the image sensor 102 according to the settings instructed in S403 or S404, and output the obtained image signal to the image acquisition unit 103.

[0052] In S406, the image synthesis unit 104 generates an HDR image using the image signal temporarily held in the image acquisition unit 103 in S405 through a known synthesis process, and then terminates HDR shooting.

[0053] As described above, according to the first embodiment, by adaptively changing the image shooting method for HDR according to the difference in brightness and the amount of movement of the subject, it is possible to generate high-quality images with a wider dynamic range even when the difference in brightness and the amount of movement of the subject are large.

[0054] <Second Embodiment> Next, a second embodiment of the present invention will be described. In the first embodiment described above, an example was explained in which the user sets the HDR shooting mode in advance and takes pictures according to the set HDR shooting mode. However, it is conceivable that there may be situations in which it is difficult for the user to select the appropriate HDR shooting mode. Therefore, in the second embodiment, an automatic setting (Auto) is added to the HDR shooting mode selection, and an example is described in which the system control unit 110 of the imaging device 100 automatically selects the appropriate HDR shooting mode according to the subject being photographed by the user.

[0055] [Configuration of the imaging device] Figure 6 is a block diagram showing the schematic configuration of the imaging device 100 according to the second embodiment. Note that components similar to those in Figure 1 are given the same reference numerals and their descriptions are omitted. In this embodiment, the system control unit 110 includes a brightness difference detection unit 501, a motion detection unit 502, and an exposure control unit 503.

[0056] The luminance difference detection unit 501 uses the image signal temporarily stored in the image acquisition unit 103 to calculate the luminance difference between the bright and dark areas of the subject. For example, the entire screen is divided into multiple areas, the brightness is measured for each area, and the luminance difference of the subject is calculated from the bright and dark areas. The motion detection unit 502 detects the amount of motion of the subject from the image signal temporarily stored in the image acquisition unit 103 and the image signal captured before that image signal. The exposure control unit 503 uses the image signal temporarily stored in the image acquisition unit 103 to calculate the exposure conditions at the time of shooting.

[0057] When the system control unit 110 selects the automatic selection mode (Auto) for setting the HDR shooting mode based on a determination described later, it determines the HDR shooting mode and the shooting conditions for HDR shooting based on at least one of the following: the brightness difference of the subject obtained from the brightness difference detection unit 501, the amount of motion of the subject obtained from the motion detection unit 502, and the exposure conditions calculated by the exposure control unit 503.

[0058] [HDR shooting mode setting process] Next, the HDR shooting mode setting process by the system control unit 110 of the imaging device 100 according to the second embodiment will be described with reference to Figures 7 to 10B.

[0059] Figure 7 shows a flowchart of the HDR shooting mode setting process, which is performed in response to user operations on the operation unit 107 based on the content displayed on the display unit 109. Figure 8 shows the content displayed on the display unit 109 at that time. This process is initiated when the HDR shooting mode setting process is instructed via a menu button (not shown) or touch panel included in the operation unit 107.

[0060] First, in the flowchart of Figure 7, when the user starts setting the HDR shooting mode, in S601, the system control unit 110 displays the UI800 shown in Figure 8 on the display unit 109. Note that in Figure 8, the same reference numbers are used for components similar to those in Figure 4, and their explanations are omitted.

[0061] Option 701 (Auto) indicates a setting that automatically selects the HDR shooting mode best suited to the subject being photographed. Specifically, it automatically determines and sets either brightness priority (option 301), motion priority (option 302), or hybrid mode (described later).

[0062] Options 703-705 indicate the degree of dynamic range expansion in HDR shooting, expressed in stops of exposure value. Option 702 is a setting where the system control unit 110 automatically determines the degree of dynamic range expansion based on the detection result. Option 703 indicates a setting that expands the dynamic range by 1 stop of exposure value in both the positive and negative directions. Similarly, option 704 indicates a setting that expands the dynamic range by 2 stops of exposure value in both the positive and negative directions, and option 705 indicates a setting that expands the dynamic range by 3 stops of exposure value in both the positive and negative directions. Arrow 706 indicates the currently selected dynamic range setting. Memo 305 also displays the characteristics of the currently selected setting.

[0063] In S602, the user operates the control unit 107 to access the UI 8 By moving the arrow 304 displayed on 00, the user selects the HDR shooting mode from options 301-303 and 701. If the user selects option 303 (OFF), the process proceeds to S606, and the system control unit 110 stores the HDR shooting OFF setting in the memory unit 108. If the user selects an option other than 303, the process proceeds to S603.

[0064] In S603, the system determines the option selected by the user, and if the user selects option 701 (Auto), the system proceeds to S609, and the system control unit 110 is H DR shooting mode automatic setting process Perform Details regarding the automatic HDR shooting mode setting process will be described later, referring to Figures 9 to 10B.

[0065] Furthermore, when the user selects option 301 (brightness priority), the process proceeds to S604, where the user further operates the control unit 107 to move the arrow 706 displayed on the UI 300, thereby selecting the desired dynamic range expansion from options 702 to 705. In the subsequent S607, the system control unit 110 stores the brightness priority HDR shooting mode selected by the user in S603 and the exposure value stop setting selected by the user in S604 in the memory unit 108.

[0066] Figure 8 shows an example where the user has selected options 301 and 703. In this case, at S607, the brightness-priority HDR shooting mode is selected, and the number of exposure stops is stored in the memory unit 108 so that a total of three images are output by taking three consecutive shots: an image with appropriate exposure for the main subject, an image with an exposure one stop brighter than this exposure, and an image with an exposure one stop darker than this exposure.

[0067] On the other hand, the user selects Limb 3 Selecting 02 (Motion Priority) proceeds to S605, where the user further operates the control unit 107, UI 8 By moving the arrow 706 displayed on 00, the user selects the desired dynamic range expansion from the options 702 to 705. In the subsequent S608, the system control unit 110 stores the motion-priority HDR shooting mode selected by the user in S603 and the exposure value stop setting selected by the user in S605 in the memory unit 108.

[0068] In S610, if the user does not instruct the system to end the HDR shooting mode setting process, the system returns to S602 and continues the HDR shooting mode setting process. When the user instructs the system to end the HDR shooting mode setting process, the system terminates with the selected HDR shooting mode and number of stops setting stored in the storage unit 108.

[0069] Next, the automatic HDR shooting mode setting process performed in S609 of Figure 7 will be explained with reference to Figures 9, 10A, and 10B.

[0070] Figure 9 is a flowchart of the automatic HDR shooting mode setting process, showing the process of automatically selecting the HDR shooting mode based on at least one of the following: the brightness difference of the subject obtained from the brightness difference detection unit 501, the amount of motion of the subject obtained from the motion detection unit 502, and the exposure conditions calculated by the exposure control unit 503. Figures 10A and 10B show HDR shooting mode tables that define HDR shooting modes according to the brightness difference of the subject, the amount of motion of the subject, and the exposure conditions. Hereinafter, the brightness difference of the subject, the amount of motion of the subject, and the exposure conditions will be collectively referred to as "evaluation values".

[0071] When the HDR shooting mode automatic setting process is started, in S801, the system control unit 110 performs pre-shooting to detect the state of the subject and temporarily stores the signal output from the image sensor 102 in the image acquisition unit 103. The system control unit 110 reads the image temporarily stored in the image acquisition unit 103 and calculates an evaluation value used for selecting the HDR shooting mode.

[0072] In S802, the system control unit 110 reads the corresponding HDR shooting mode from the HDR shooting mode table shown in Figure 10A or Figure 10B, according to the evaluation value calculated in S801. The HDR shooting mode tables and the contents of the HDR shooting modes in Figures 10A and 10B will be described in detail later.

[0073] In S803, the system control unit 110 stores the HDR shooting mode selected in S802 in the memory unit 108.

[0074] In S804, the system control unit 110 determines whether or not to continue the HDR shooting mode automatic setting process. For example, if HDR shooting is instructed by a release button (not shown) included in the operation unit 107, or if the release button is no longer pressed, the HDR shooting mode automatic setting process is terminated; otherwise, the process returns to S801 and the above-described process is repeated.

[0075] Next, we will explain the HDR shooting mode tables shown in Figures 10A and 10B, which are referenced in S802. In Figures 10A and 10B, column 901 shows the setting number assigned to each HDR shooting mode, and column 902 shows the selection conditions for brightness difference, motion amount, and brightness, which represent the state of the subject, and corresponds to the evaluation value. Column 903 shows the types of HDR shooting modes: brightness priority HDR shooting mode, motion priority HDR shooting mode, and hybrid HDR shooting mode.

[0076] Similar to the first embodiment, HDR images are captured using the first driving method in brightness-priority HDR shooting mode and the second driving method in motion-priority HDR shooting mode. In hybrid HDR shooting mode, shooting is performed using a third driving method that is intermediate between the first and second driving methods. Here, for example, two shots are taken: the first shot outputs a single image signal with a single gain applied, and the second shot outputs two image signals with different dynamic ranges by applying two different types of gains.

[0077] Furthermore, column 905 in Figure 10A shows the combinations of ISO sensitivity, exposure time, and number of shots, and column 906 in Figure 10B shows the combinations of ISO sensitivity, aperture value, and number of shots.

[0078] As shown in Figures 10A and 10B, settings A1, A2, D1, D2, G1, and G2 indicate motion-priority HDR shooting mode, which is selected when the amount of subject movement exceeds a predetermined range (large). In other words, these HDR shooting modes are selected when the amount of subject movement is large and generating a composite image from multiple shots would result in an unnatural appearance. Furthermore, the gain value used differs depending on the brightness difference and brightness. By using these settings, it becomes possible to shoot moving subjects that are difficult to handle with brightness-priority HDR shooting mode while maintaining a comparable dynamic range.

[0079] Similarly, settings C1, C2, F1, F2, I1, and I2 indicate brightness-priority HDR shooting modes, which are selected when the subject's movement falls below a predetermined range (small). In other words, these HDR shooting modes are selected when the subject's movement is small and generating a composite image from multiple shots will not result in an unnatural appearance. The gain value used is also changed according to the brightness.

[0080] Settings B1, B2, E1, E2, H1, and H2 indicate the Hybrid HDR shooting mode, which is selected when the subject's movement falls within a predetermined range (medium). This predetermined range refers to the range of motion where, for example, taking three shots as shown in setting C1 would result in an unnatural composite image, but taking two shots would produce a natural composite image. Furthermore, the gain value used varies depending on the brightness difference.

[0081] Here, as an example, we will explain the settings for the Hybrid HDR shooting mode with setting number B1. First, two shots are taken. In the first shot, the obtained image signal is multiplied by a gain equivalent to ISO 800 so that the brightness of the main subject is appropriate. Then, in the second shot, different gains equivalent to ISO 100 and ISO 6400 are multiplied compared to the first shot. As a result, three image signals—ISO 800, ISO 100, and ISO 6400—are obtained for use in generating the HDR image through the image synthesis process in the image synthesis unit 104.

[0082] [HDR shooting processing] Next, the processing in the system control unit 110 of the imaging device 100 according to the second embodiment during HDR shooting will be described with reference to Figure 11.

[0083] Figure 11 is a flowchart of the HDR shooting process performed based on the HDR shooting mode stored in the memory unit 108. This process starts when HDR shooting is instructed via the release button (not shown) or touch panel included in the operation unit 107. Furthermore, the same step numbers are used for processes similar to those described in the first embodiment with reference to Figure 5, and explanations are omitted as appropriate. Also, if the process is accessed via S606 in Figure 7, that is, if the HDR shooting OFF setting is selected, this process is not performed.

[0084] When HDR shooting is started, in S1101, the system control unit 110 reads the HDR shooting mode stored in the memory unit 108.

[0085] In S1102, the system control unit 110 checks the HDR shooting mode read from the memory unit 108 in S1101. Then, it proceeds to S1103 if it is the brightness-priority HDR shooting mode, to S1104 if it is the motion-priority HDR shooting mode, and to S1107 if it is the hybrid HDR shooting mode.

[0086] The shooting operation in S1103 is basically the same as the shooting operation in S403 in Figure 5 (i.e., shooting using the first drive method), but in the second embodiment, the dynamic range expansion amount is set. Therefore, the shooting lens 101 and image sensor 102 are controlled according to this setting. Also, if the brightness-priority HDR shooting mode is selected by the HDR shooting mode automatic setting process, the shooting lens 101 and image sensor 102 are controlled using the combination of ISO sensitivity, exposure time, and exposure value corresponding to Figure 10A or Figure 10B.

[0087] Furthermore, the shooting operation in S1104 is basically the same as the shooting operation in S404 in Figure 5 (i.e., shooting using the second drive method), but in the second embodiment, the dynamic range expansion amount is set. Therefore, the shooting lens 101 and image sensor 102 are controlled according to this setting. Also, if the motion-priority HDR shooting mode is selected by the HDR shooting mode automatic setting process, the shooting lens 101 and image sensor 102 are controlled using the combination of ISO sensitivity, exposure time, and exposure value corresponding to Figure 10A or Figure 10B.

[0088] Furthermore, in S1107, the system control unit 110 instructs the image sensor control unit 111 and the lens control unit 112 to perform shooting using a drive method associated with the hybrid HDR shooting mode. Specifically, it instructs the system control unit 110 to drive the shooting lens 101 and the image sensor 102 using a third drive method to perform two consecutive shots, apply two different types of gain to the image signal obtained from the second shot, and output three images with different dynamic ranges. Since the hybrid HDR shooting mode is selected by the HDR shooting mode automatic setting process, the shooting lens 101 and the image sensor 102 are controlled using a combination of ISO sensitivity, exposure time, and exposure value corresponding to Figure 10A or Figure 10B.

[0089] In S1105, the image sensor control unit 111 and the lens control unit 112 control the image sensor 102 and the shooting lens 101 according to the settings instructed in S1103, S1104, or S1107 to perform HDR shooting and output the obtained image signal to the image acquisition unit 103.

[0090] In S406, the image synthesis unit 104 generates an HDR image using the image signal temporarily held by the image acquisition unit 103 in S1005 through a known synthesis process, and then terminates HDR shooting.

[0091] As described above, according to the second embodiment, even in situations where it is difficult for the user to select an appropriate HDR shooting mode, it is possible to generate high-quality images with a wider dynamic range, even when there is a large difference in brightness or motion of the subject.

[0092] In the example described above, the HDR shooting mode is automatically selected when the user selects the automatic HDR shooting mode setting (Auto). However, the present invention is not limited to this. For example, the HDR shooting mode may always be automatically selected when the user selects HDR shooting.

[0093] <Third Embodiment> Next, a third embodiment of the present invention will be described. In the third embodiment, an example will be described in which the system control unit 110 sets the HDR shooting mode when displaying a live view (LV) display that shows a real-time captured image of the subject on the display unit 109. Note that the configuration of the imaging device 100 in the third embodiment is the same as that described with reference to Figure 6 in the second embodiment, so the description will be omitted here.

[0094] [HDR settings displayed when using LV display] Figure 12 shows the display for setting the HDR shooting mode, which is displayed on the display unit 109 of the imaging device 100 according to the third embodiment. In Figure 12, UI1200 is the LV image displayed on the display unit 109 during HDR shooting. The LV image displays the main subject 1201 being shot in HDR, along with the HDR shooting mode stored in the memory unit 108 in area 1202. In the example shown in Figure 12, "Motion Priority" is displayed in area 1202, which indicates that the HDR shooting mode stored in the memory unit 108 is the motion priority HDR shooting mode. If the HDR shooting mode stored in the memory unit 108 is the brightness priority HDR shooting mode, then "Brightness Priority" will be displayed in area 1202.

[0095] This configuration allows the user to always know what HDR shooting mode the imaging device 100 is in during HDR shooting.

[0096] [Suggested HDR shooting modes when viewing LV] Next, the display that suggests an HDR shooting mode suitable for the shooting conditions, and the process of changing the HDR shooting mode from the suggested display, in the LV display of the system control unit 110 of the imaging device 100 according to the third embodiment, will be explained with reference to Figures 13 to 16. In Figures 14 to 16, the same reference numerals are used for components that are the same as those in Figure 12, and explanations are omitted as appropriate.

[0097] Figure 13 shows a flowchart illustrating how the user operates the control unit 107 to change the HDR shooting mode in the LV display during HDR shooting, according to the content displayed on the display unit 109. Figure 14 shows the content displayed on the display unit 109. This process is initiated when the control unit 107 instructs the system to suggest an HDR shooting mode in the LV display during HDR shooting.

[0098] First, in the flowchart in Figure 13, the HDR shooting mode when displaying LV during HDR shooting is change When processing begins, in S1301, the system control unit 110 reads the current HDR shooting mode stored in the memory unit 108.

[0099] In S1302, the system control unit 110 acquires an image at a constant frame rate by controlling the photographic lens 101 and the image sensor 102 via the image sensor control unit 111 and the lens control unit 112 in order to display an LV on the display unit 109. The image acquisition unit 103 outputs the image signal input at a constant frame rate to the image synthesis unit 104, while the exposure control unit 503 uses the image signal to perform photometering of the subject.

[0100] In S1303, the system control unit 110 displays the LV image of the subject and the current HDR shooting mode read out in S1301 on the display unit 109. Figure 14 shows an example of the display content at this time. In Figure 14, subject 1401 is a subject image with a higher brightness than the main subject 1201, that is, a subject image such as the sun.

[0101] In S1304, the system control unit 110 calculates the luminance difference between regions with different luminances, which were photometrically measured by the luminance difference detection unit 501 in S1302. For example, it calculates the luminance difference between a subject 1201 with normal luminance and a subject 1401 with high luminance.

[0102] In S1305, the system control unit 110 determines whether the luminance difference calculated in S1304 is greater than or equal to a predetermined value. If it is greater than or equal to the predetermined value, the process proceeds to S1306; otherwise, it proceeds to S1310.

[0103] In S1306, the system control unit 110 determines whether the current HDR shooting mode read from the memory unit 108 in S1301 is motion-priority HDR shooting mode or not. If it is motion-priority HDR shooting mode, proceed to S1307; otherwise, proceed to S1314.

[0104] In S1307, the system control unit 110 displays a simplified menu on the display unit 109 superimposed on the LV image for changing the HDR shooting mode to brightness-priority HDR shooting mode. Figure 15 shows the display content at this time. In Figure 15, the UI 1200 displays a simplified menu 1501 for changing from the current HDR shooting mode, which is motion-priority HDR shooting mode, displayed in area 1202, to brightness-priority HDR shooting mode. The user can select whether to change to brightness-priority HDR shooting mode or remain in motion-priority HDR shooting mode by operating the operation unit 107 according to this simplified menu 1501.

[0105] In step S1308, if the user selects to switch to brightness-priority HDR shooting mode, the process proceeds to S1309; otherwise, it proceeds directly to S1314.

[0106] In S1309, the system control unit 110 stores the brightness-priority HDR shooting mode set by the user in the memory unit 108, and proceeds to S1314.

[0107] This allows the camera to easily switch to brightness-priority HDR shooting mode at the appropriate time by prompting the user to change the setting when motion-priority HDR shooting mode is set while a subject with a large difference in brightness is being photographed.

[0108] On the other hand, if the brightness difference calculated in S1304 is not equal to or greater than a predetermined value in S1305, the process proceeds to S1310, where the system control unit 110 determines whether the current HDR shooting mode read from the memory unit 108 in S1301 is the brightness-priority HDR shooting mode. If it is the brightness-priority HDR shooting mode, the process proceeds to S1311; otherwise, it proceeds to S1314.

[0109] In S1311, the system control unit 110 displays a simple menu on the display unit 109 superimposed on the LV image for changing the HDR shooting mode to motion-priority HDR shooting mode. Figure 16 shows the display content at this time. In Figure 16, area 1601 indicates that the current HDR shooting mode is brightness-priority HDR shooting mode, and the UI 1200 displays a simple menu 1602 for changing from the current HDR shooting mode, brightness-priority HDR shooting mode, to motion-priority HDR shooting mode. The user can select whether to change to motion-priority HDR shooting mode or remain in brightness-priority HDR shooting mode by operating the operation unit 107 according to this simple menu 1602.

[0110] In step S1312, if the user selects to change to motion-priority HDR shooting mode, the process proceeds to S1313; otherwise, it proceeds directly to S1314.

[0111] In S1313, the system control unit 110 stores the motion-priority HDR shooting mode set by the user in the memory unit 108, and then proceeds to S1314.

[0112] This allows the camera to easily switch to motion-priority HDR shooting mode at the appropriate time by prompting the user to change the setting when it is determined that brightness-priority HDR shooting mode is set while a fast-moving subject is being photographed.

[0113] In S1314, if the system control unit 110 determines that metering for LV display is complete, it terminates the HDR shooting mode suggestion process. Otherwise, it returns to S1301 and proposes the optimal HDR shooting mode by metering and calculating the brightness difference again.

[0114] As described above, according to the third embodiment, it becomes possible to easily and appropriately set the HDR shooting mode in a timely manner according to the state of the subject being photographed. <Other Embodiments> Furthermore, this invention relates to multiple devices (e.g., host computer, interface device). , Even when applied to a system consisting of a single device (such as a video camera), Place It may be applied.

[0115] Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0116] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0117] 100: Imaging device, 101: Imaging lens, 102: Image sensor, 103: Image acquisition unit, 104: Image synthesis unit, 105: Image processing unit, 106: Image recording unit, 107: Operation unit, 108: Memory unit, 109: Display unit, 110: System control unit, 111: Image sensor control unit, 112: Lens control unit, 501: Brightness difference detection unit, 502: Motion detection unit, 503: Exposure control unit

Claims

1. Image sensor and A control means that, when a first shooting mode is set, controls the image sensor to acquire multiple images by taking multiple images while changing the exposure; when a second shooting mode different from the first shooting mode is set, controls the image sensor to acquire multiple images by amplifying the image signal obtained by taking one image with multiple different gains within the image sensor; and when a third shooting mode different from the first and second shooting modes is set, controls the image sensor to acquire multiple images by multiplying one of the multiple image signals obtained by taking images a fewer number of times than in the first shooting mode by multiple different gains. A generation means for generating a single image with a wider dynamic range than each of the multiple images acquired in any of the first to third shooting modes, An imaging device characterized by having the following features.

2. The imaging apparatus according to claim 1, characterized in that, in the second imaging mode, the image signal obtained by performing the single imaging is divided into a plurality of regions, and different gains are applied to the image signal for each of the plurality of regions.

3. The imaging device according to claim 1 or 2, further comprising setting means for setting to any of the first to third shooting modes.

4. The imaging device according to claim 3, characterized in that the setting means sets the first shooting mode when prioritizing the dynamic range of the captured image, and sets the second shooting mode when the subject is moving fast.

5. It further has an operating means for inputting user instructions, The imaging device according to claim 3 or 4, characterized in that the setting means is set to one of the first to third shooting modes in response to the operation of the operating means.

6. The imaging apparatus according to any one of claims 3 to 5, characterized in that the setting means allows selection of the range of dynamic range expansion.

7. A luminance difference detection means for detecting the luminance difference of the subject to be photographed, Motion detection means for detecting the movement of the subject, A brightness detection means for detecting the brightness of the subject, The imaging apparatus according to any one of claims 3 to 6, further comprising the above.

8. The imaging device according to claim 7, characterized in that the setting means is set to one of the first to third shooting modes according to at least one of the brightness difference, movement, and brightness of the subject.

9. The system further includes a storage means that stores a table in which the luminance difference, movement, and brightness of the subject are associated with the first to third shooting modes. The imaging device according to claim 7 or 8, characterized in that the setting means is set to one of the first to third shooting modes by referring to the table.

10. The imaging apparatus according to any one of claims 1 to 9, further comprising a suggestion means for suggesting any of the first to third shooting modes.

11. A control step is provided to control the system so that when a first shooting mode is set, multiple images are acquired by taking multiple images with varying exposures using the image sensor; when a second shooting mode different from the first shooting mode is set, multiple images are acquired by amplifying the image signal obtained by taking one image using the image sensor with multiple different gains within the image sensor; and when a third shooting mode different from the first and second shooting modes is set, multiple images are acquired by multiplying one of the multiple image signals obtained by taking images a fewer number of times than in the first shooting mode by multiple different gains. A generation step of generating a single image with a wider dynamic range than each of the multiple images acquired in any of the first to third shooting modes, A control method for an imaging device, characterized by having the following features.

12. A program for causing a computer to perform each step of the control method for the imaging apparatus described in claim 11.

13. A computer-readable storage medium storing the program described in claim 12.

Images