Photographing control device, imaging apparatus, photographing control method, and program

Abstract

To provide a technique to create at least three images different in exposure that can be used for creation of a HDR composite image, by using a DGO image pick-up device.SOLUTION: A photographing control device is provided comprising: photographing control means that performs control to perform photographing at least twice according to a photographing instruction, by using an image pick-up device configured to create a first image and a second image different in exposure in single photographing; and exposure control means that alternately performs first exposure control and second exposure control in each photographing of the photographing performed at least twice, wherein the first exposure control is exposure control to be performed so that the first image is created with first exposure and the second image is created with second exposure lower than the first exposure, and the second exposure control is exposure control to be performed so that the first image is created with the first exposure and the second image is created with third exposure higher than the first exposure.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an imaging control device, an imaging device, an imaging control method, and a program. [Background technology]

[0002] There is known an imaging element that has two input memories in a column AMP section and can output two images with different gains in response to one shooting (see Patent Document 1). In this specification, such an imaging element is called a Dual Gain Output (DGO) imaging element. Since the gain of the imaging element is related to the ISO sensitivity, it is possible to generate two images with different gains using a DGO imaging element, thereby generating two images with different exposures. Patent Document 1 discloses a technology for generating a High Dynamic Range (HDR) composite image by synthesizing two images generated in one shooting using a DGO imaging element. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-178186 A Summary of the Invention [Problem to be solved by the invention]

[0004] In general, the more images with different exposures there are, the better the gradation of the HDR composite image can be generated. However, Patent Document 1 does not consider increasing the number of images used to generate the HDR composite image.

[0005] The present invention has been made in view of these circumstances, and aims to provide a technique for generating at least three images with different exposures using a DGO imaging element that can be used to generate an HDR composite image. [Means for solving the problem]

[0006] In order to solve the above problem, the present invention provides an imaging control device comprising: an imaging control means for controlling at least two imaging operations in response to an imaging instruction using an imaging element configured to generate a first image and a second image having different exposures in one imaging operation; and an exposure control means for alternately performing a first exposure control or a second exposure control in each of the at least two imaging operations, wherein the first exposure control is exposure control such that the first image is generated at a first exposure and the second image is generated at a second exposure lower than the first exposure, and the second exposure control is exposure control such that the first image is generated at the first exposure and the second image is generated at a third exposure higher than the first exposure. Effect of the Invention

[0007] According to the invention, it is possible to use the DGO imager to generate at least three images with different exposures that can be used to generate an HDR composite image.

[0008] Other features and advantages of the present invention will become apparent from the accompanying drawings and the following detailed description of the preferred embodiments of the present invention. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing the configuration of an imaging device 100. [Diagram 2] FIG. 2 is a diagram showing the configuration of an image sensor 102. [Diagram 3] FIG. 1 is a schematic diagram of a circuit showing one of the columns AMP204. [Figure 4] 4 is a flowchart of HDR shooting processing executed by the imaging device 100 according to the first and second embodiments. [Diagram 5] 13A is a flowchart showing details of the process of S409 according to the first embodiment; FIG. 13B is a flowchart showing details of the process of S409 according to the second embodiment; FIG. [Figure 6]FIG. 4 is a diagram showing an example of a program diagram used in HDR shooting processing. [Figure 7] 5A and 5B are diagrams for explaining the addition ratios of each image in HDR synthesis processing. [Figure 8] 13 is a flowchart of HDR shooting processing executed by the imaging device 100 according to the third embodiment. [Figure 9] 9A shows the structure of a RAW image file 901; FIG. 9B shows an example of the structure of ImageData 912; and FIG. 9C shows another example of the structure of ImageData 912. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] [First embodiment] 1 is a block diagram showing the configuration of an image capture device 100 having a function as an image capture control device. In FIG. 1, an optical lens 101 is an optical lens for taking in light from a subject and forming an image of the light on an image sensor .

[0012] The imaging element 102 receives incident light from the optical lens 101, converts it into an electrical signal, and outputs it. Examples of the imaging element 102 include a Charge Coupled Device (CCD) image sensor and a Complementary Metal Oxide Semiconductor (CMOS) image sensor. An example of the imaging element 102 is a type that directly outputs a generated analog video signal. Another example of the imaging element 102 is a type that internally performs analog-to-digital (AD) conversion on the generated analog video signal and outputs digital data such as low voltage differential signaling (LVDS).

[0013] Here, the configuration of the image sensor 102 will be described with reference to Fig. 2. In the example of Fig. 2, the image sensor 102 is of a type that outputs digital data. In Fig. 2, a timing pulse control unit 201 controls the operation of the image sensor 102 by supplying an operating clock (CLK) to each block of the image sensor 102 and by supplying a timing signal to each block.

[0014] A vertical scanning circuit 202 performs timing control for sequentially reading out pixel signal voltages from a pixel section 203, in which a plurality of pixels are arranged two-dimensionally, during one frame. In general, video signals are sequentially read out row by row from the top row to the bottom row during one frame.

[0015] The pixel unit 203 is a photoelectric conversion element that photoelectrically converts incident light and outputs a voltage according to the amount of incident light. The pixel unit 203 converts the captured light into electric charge and accumulates the electric charge in a floating diffusion (FD) capacitance. The size of the FD capacitance can be changed, and the signal-to-noise ratio (SN ratio) can be improved by changing the capacitance according to the ISO sensitivity. In general, the capacitance is set to "large" at low ISO sensitivity, and the capacitance is set to "small" at high ISO sensitivity. Note that when outputting images with two different gains, which will be described later, the capacitance for accumulating electric charge is common to the two gains. In addition, the size of the capacitance that can be set is not limited to two types, "large" and "small", and a configuration in which three or more stages of capacitance can be set may be adopted.

[0016] The column AMP 204 is used to electrically amplify a signal read from the pixel unit 203. By amplifying the signal with the column AMP 204, the signal level of the pixel relative to the noise generated by the downstream column ADC 205 can be amplified, and the signal-to-noise ratio can be equivalently improved. In addition, the gain of the column AMP can be changed from the timing pulse control unit 201.

[0017] The image sensor 102 has two input memories in the column AMP 204, and by changing the gain of the column AMP 204, it is possible to output signals with two types of gain for generating an HDR composite image. By having two input memories, it is possible to apply two types of gain separately to a signal at a specific time read from the FD and output the signal. Therefore, although the amount of data increases, it is possible to obtain two types of images with different gains that are simultaneous. Note that the image sensor 102 may be configured to output three or more types of images with different gains that are simultaneous.

[0018] The column ADC 205 performs AD conversion on the read signals from the column AMP 204. The digitized signals are sequentially read out by the horizontal transfer circuit 206. The output of the horizontal transfer circuit 206 is input to the signal processing circuit 207.

[0019] The signal processing circuit 207 is a circuit that performs signal processing digitally. The signal processing circuit 207 can easily perform gain calculation by adding a certain amount of offset value through digital processing of the signal and by performing shift calculation and multiplication. Furthermore, a pixel area that is intentionally shielded from light may be provided in the pixel unit 203. In this case, the signal processing circuit 207 can perform a digital black level clamp operation using the shielded pixel area. The output of the signal processing circuit 207 is passed to an external output circuit 208.

[0020] The external output circuit 208 has a serializer function and converts a multi-bit input parallel signal from the signal processing circuit 207 into a serial signal. The external output circuit 208 also converts this serial signal into, for example, an LVDS signal, and outputs it to the outside of the image sensor 102.

[0021] Returning to FIG. 1, the image acquisition unit 103 captures the video signal output from the image sensor 102 and performs various processes. If the image sensor 102 is of a type that outputs an analog video signal (if AD conversion is not performed inside the image sensor 102), the image acquisition unit 103 includes an analog front end that performs AD conversion. The image acquisition unit 103 removes fixed pattern noise of the image sensor 102, performs black level clamping processing, and the like. The image acquisition unit 103 also has a role of separating the video signal output from the image sensor 102 into an image signal for recording and an evaluation signal for controlling the image sensor 102.

[0022] The signal processing unit 104 performs various image processing such as a pixel addition function, noise reduction, gamma correction, knee correction, digital gain, etc., which are typical image processing functions of the imaging device 100, as well as defect correction. The image acquisition unit 103 and the signal processing unit 104 also include a memory circuit (not shown) that stores setting values ​​required for each correction and image processing.

[0023] The image synthesis unit 105 generates an HDR synthetic image using an arbitrary synthesis method from the HDR generation signal output from the image sensor 102. For example, there is a synthesis method in which a high-gain image is used as a normal image, and a low-gain image is used for bright, blown-out parts of the normal image. However, the synthesis method of this embodiment is not particularly limited, and any synthesis method (synthesis algorithm) can be used as long as it is a method for synthesizing two images with different gains.

[0024] The signal recording unit 106 records the video signal received from the image synthesis unit 105 in a storage device or storage medium.

[0025] The exposure control unit 107 can calculate an optimal amount of exposure based on information on the video signal received from the image acquisition unit 103. Then, the exposure control unit 107 determines the operation of the image sensor control unit 108, and transmits the determined operation to the image sensor control unit 108.

[0026] The control unit 109 includes a ROM and a RAM, and performs overall control of the imaging device 100 by executing a control program stored in the ROM using the RAM as a work memory.

[0027] Next, the operation of the image sensor 102 when generating an HDR composite image will be described with reference to Fig. 3. Fig. 3 is a conceptual diagram of a circuit in which one column of the column AMP204 is extracted. As described above, the image sensor 102 can change the gain of the column AMP204 to generate an HDR composite image.

[0028] OP305 is an operational amplifier, and an input capacitance and a feedback capacitance are connected to OP305. C303 and C304 are input capacitances, and SW301 and SW302 are switches. A signal read from the pixel unit 203 is connected to C303 and C304 via SW301 and SW302.

[0029] C306 and C308 are feedback capacitances, and SW307 is a switch. The connection of C308 can be controlled by SW307. Since the column AMP204 uses capacitance, the amplification factor is (input capacitance / feedback capacitance).

[0030] As described above, the column AMP204 has two input capacitances. First, the column AMP204 applies a gain based on C303 and C306 to the input signal by turning SW301 ON and SW302 and SW307 OFF, and outputs the signal to the column ADC205. Next, the column AMP204 applies a gain based on C304, C306, and C308 to the input signal by turning SW301 OFF and SW302 and SW307 ON, and outputs the signal. This makes it possible to output two images that have been applied with different gains.

[0031] 4 is a flowchart of the HDR shooting process executed by the imaging device 100. Unless otherwise specified, the process of each step of this flowchart is executed under the control of the control unit 109 that operates according to a control program. When the power switch of the imaging device 100 is turned on and the operation mode is set to the HDR shooting mode, and then the release button of the imaging device 100 is pressed, the process of this flowchart starts.

[0032] In S401, the control unit 109 initializes N to 0. N is a variable indicating the number of times photography has been performed.

[0033] In S402, the control unit 109 adds 1 to N. When the process of S402 is performed for the first time, N becomes 1, indicating that the first shooting is to be performed.

[0034] In S403, the control unit 109 determines whether or not N is an odd number. If N is odd (i.e., the number of times shooting is odd), the process proceeds to S404, and if not (i.e., the number of times shooting is even), the process proceeds to S405.

[0035] In S404, the exposure control unit 107 obtains an appropriate exposure value (EV) by automatic exposure (AE) processing, and determines the exposure conditions (ISO sensitivity, aperture, and time) corresponding to each of appropriate exposure and underexposure.

[0036] Fig. 6 is a diagram showing an example of a program diagram used in the HDR shooting process of Fig. 4. In Fig. 6, the right vertical axis indicates the aperture value (AV), the left and upper parts indicate the EV value, and the lower part indicates the time value (TV) and ISO sensitivity value. The exposure control unit 107 determines appropriate exposure conditions (ISO sensitivity, aperture, time) by referring to the program diagram based on the control value corresponding to the previously obtained EV value, and performs exposure control according to the conditions. In Fig. 6, line 602 indicates the appropriate exposure diagram, line 601 indicates the overexposure diagram, and line 603 indicates the underexposure diagram.

[0037] Consider the case where a DGO imaging element such as the imaging element 102 is used to capture both optimally exposed and underexposed images at the same time. Since different TV values ​​cannot be set for optimal and underexposed images when using a DGO imaging element, it is necessary to perform imaging with a combination of exposure conditions that allows underexposure to fall within the minimum ISO sensitivity. For example, when underexposure is performed at -2EV, the combination of TV and ISO values ​​shown in the box 605 is used. For example, when the Brightness Value (BV value) is 10 and the AV value is 2.8, a combination of TV value 250 and ISO value 400 is used for optimal exposure, and a combination of TV value 250 and ISO value 100 is used for underexposure. Therefore, in this example, the exposure control unit 107 determines the exposure conditions corresponding to optimal exposure to be ISO value 400, AV value 2.8, and TV value 250, and the exposure conditions corresponding to underexposure to be ISO value 100, AV value 2.8, and TV value 250.

[0038] In S405, the exposure control unit 107 obtains an appropriate exposure value (Exposure Value (EV)) by automatic exposure (AE) processing, and determines the exposure conditions (ISO sensitivity, aperture, seconds) corresponding to each of the appropriate exposure and overexposure in the same manner as in S404. Note that, since image noise increases when the ISO sensitivity for overexposure increases, it is desirable for the exposure control unit 107 to control the ISO sensitivity for appropriate exposure determined in S404 to match the ISO sensitivity for overexposure determined in S405. Therefore, for example, when shooting overexposure at +2EV, the combination shown in the frame 604 is used as the TV value and ISO value. For example, when the BV value is 10 and the AV value is 2.8, a combination of TV value 60 and ISO value 100 is used for the appropriate exposure, and a combination of TV value 60 and ISO value 400 is used for the overexposure. Therefore, in this example, the exposure control unit 107 determines the exposure conditions corresponding to optimal exposure to be ISO value 100, AV value 2.8, and TV value 60, and determines the exposure conditions corresponding to overexposure to be ISO value 400, AV value 2.8, and TV value 60.

[0039] In S406, the control unit 109 controls so that shooting is performed according to the two types of exposure conditions (ISO value, AV value, and TV value) determined in S404 or S405. At this time, under the control of the control unit 109, the exposure control unit 107 notifies the image sensor control unit 108 of the two types of ISO values ​​under the two types of exposure conditions determined in S404 or S405. The image sensor control unit 108 sets two types of gain for the image sensor 102 based on the notified two types of ISO values. Since shooting is performed according to such control, two images with different exposures are generated.

[0040] In S407, the image synthesis unit 105 synthesizes the two images acquired in S406 to generate an intermediate synthesis image. The two images synthesized here are two images obtained by one shooting (one exposure) using the imaging element 102, which is a DGO imaging element. Therefore, there is no need to perform a registration process for the two images during the synthesis process.

[0041] In general, when generating an HDR composite image from three images, namely, properly exposed, overexposed, and underexposed, a Mix table that determines the addition ratio of each image as shown in FIG. 7(a) is used to perform image synthesis. In the Mix table in FIG. 7(a), an addition ratio 701 corresponding to overexposure, an addition ratio 702 corresponding to properly exposed, and an addition ratio 703 corresponding to underexposure are set as weighted addition ratios that divide the luminance range into roughly three parts. In order to obtain an HDR composite image, an overexposed image (hereinafter abbreviated as "over image") is used for the dark area, a properly exposed image (hereinafter abbreviated as "proper image") is used for the midtone luminance area, and an underexposed image (hereinafter abbreviated as "under image") is used for the bright area according to the Mix table. In addition, for the boundary area between the dark area and the midtone area and the midtone area and the bright area, the synthesis ratio is gradually changed according to the weighted addition ratio of the Mix table, thereby obtaining the effect of smoothing the image switching.

[0042] However, in this embodiment, at the time of S407, the two images obtained by the current shooting are combined. Therefore, when the two images are an appropriate image and an underexposed image, the image combination unit 105 performs the combination process according to the addition ratio 712 corresponding to the appropriate exposure and the addition ratio 703 corresponding to the underexposed image, as shown in FIG. 7(b). When the two images are an appropriate image and an overexposed image, the image combination unit 105 performs the combination process according to the addition ratio 722 corresponding to the appropriate exposure and the addition ratio 701 corresponding to the overexposed image, as shown in FIG. 7(c). The addition ratio 712 in FIG. 7(b) is a combination of the addition ratio 701 and the addition ratio 702 in FIG. 7(a), and the addition ratio 722 in FIG. 7(c) is a combination of the addition ratio 702 and the addition ratio 703 in FIG. 7(a).

[0043] In S408, the control unit 109 determines whether or not N is equal to or greater than 2. If N is equal to or greater than 2 (i.e., if there is an image captured prior to the current capture), the process proceeds to S409, and if N is not equal to or greater than 2 (i.e., if the current capture is the first capture), the process returns to S402.

[0044] In S409, a process of generating an HDR blended image from the intermediate blended image and another image is performed. Details of the process in S409 will be described later with reference to FIG.

[0045] In S410, the control unit 109 determines whether or not to end the shooting. For example, if the release button is still pressed, the control unit 109 determines not to end the shooting, and if the release button is not pressed, the control unit 109 determines to end the shooting. If it is determined to end the shooting, the process of this flowchart ends, and if not, the process returns to S402.

[0046] Next, the details of the process of S409 will be described with reference to Fig. 5(a). In S501, the control unit 109 judges whether or not N = 2. If N = 2 (i.e., in the case of the first process of S409), the process proceeds to S502, and if not, the process proceeds to S504.

[0047] In S502, the image synthesis unit 105 generates an HDR synthesis image by synthesizing the previous intermediate synthesis image and the current unsuitable image. The "previous intermediate synthesis image" refers to the intermediate synthesis image generated in S407 from the two images obtained by the previous shooting (N-1th shooting). The "current unsuitable image" refers to the image that is not the suitable image of the two images obtained by the current shooting (Nth shooting), and is an underexposed image when N is an odd number, and an overexposed image when N is an even number. Since the processing of S502 is performed when N=2, the current unsuitable image is an overexposed image.

[0048] The previous intermediate composite image and the current unsuitable image to be composited in S502 have different shooting (exposure) timings. Therefore, when compositing, the image composition unit 105 aligns the previous intermediate composite image and the current unsuitable image. As a method for the alignment process, any known method can be used, such as a method for detecting the amount of deviation between two images by frequency analysis. After the alignment process, the image composition unit 105 composites the previous intermediate composite image (an image generated by combining the previous suitable image and the underexposed image) and the current unsuitable image (the current overexposed image) according to the Mix table in FIG. 7(c). As a result, an HDR composite image is generated by combining the three images of the suitable image, the underexposed image, and the overexposed image.

[0049] In S503, the signal recording unit 106 stores the HDR composite image generated in S502 in a recording medium.

[0050] In S504, the image synthesis unit 105 generates an HDR synthesis image by synthesizing the current intermediate synthesis image and the previous unsuitable image. The "current intermediate synthesis image" refers to the intermediate synthesis image generated in S407 from the two images obtained by the current shooting (Nth shooting). The "previous unsuitable image" refers to the image that is not the suitable image of the two images obtained by the previous shooting (N-1th shooting), and is an underexposed image when N-1 is an odd number, and an overexposed image when N-1 is an even number. The image synthesis unit 105 performs position adjustment processing in the same manner as in S502 during synthesis processing. In addition, when the current intermediate synthesis image is based on the suitable image and the overexposed image, the image synthesis unit 105 performs synthesis processing according to the Mix table in FIG. 7(b). In the case where the current intermediate synthesis image is based on the suitable image and the underexposed image, the image synthesis unit 105 performs synthesis processing according to the Mix table in FIG. 7(c). As a result, an HDR synthesis image is generated by synthesizing three images, the suitable image, the underexposed image, and the overexposed image.

[0051] In S505, the signal recording unit 106 stores the HDR composite image generated in S504 in a recording medium.

[0052] 4 and 5(a), every time the second (N=2) or subsequent (N>2) shooting is performed, in S504, an HDR composite image is generated from the current intermediate composite image generated from the current suitable image and unsuitable image, and the previous unsuitable image. Also, during the second (N=2) shooting, an HDR composite image is generated from the previous intermediate composite image generated from the previous suitable image and unsuitable image, and the current unsuitable image. Therefore, for each of the shooting times of N=2 to 5, an HDR composite image is generated by the combination of the following images, and for shooting times of N=6 or more, the generation of an HDR composite image is repeated in a similar manner (the number in parentheses indicates the number of shootings). N=2: Optimal image (1) + Underexposed image (1) + Overexposed image (2) N=2: Optimal image (2) + Overexposed image (2) + Underexposed image (1) N=3: Optimal image (3) + Underexposed image (3) + Overexposed image (2) N=4: Optimal image (4) + Overexposed image (4) + Underexposed image (3) N=5: Optimal image (5) + Underexposed image (5) + Overexposed image (4)

[0053] As described above, according to the first embodiment, the imaging device 100 controls the imaging element 102 to perform at least two imaging operations in response to an imaging instruction. The imaging device 100 may control the imaging element 102 to perform three or more imaging operations as the at least two imaging operations. The imaging element 102 is a DGO imaging element configured to generate two images (a first image and a second image) with different exposures in one imaging operation. The imaging device 100 alternately performs a first exposure control (S404) or a second exposure control (S405) in each of the at least two imaging operations. The first exposure control is an exposure control in which the first image is generated at the first exposure and the second image is generated at the second exposure that is lower than the first exposure. The second exposure control is an exposure control in which the first image is generated at the first exposure and the second image is generated at the third exposure that is higher than the first exposure.

[0054] Thus, according to the first embodiment, it is possible to generate, using the DGO imager, at least three images with different exposures that can be used to generate an HDR composite image.

[0055] In the example of Fig. 4, the first exposure is an appropriate exposure, the second exposure is an underexposure, and the third exposure is an overexposure, so that an appropriate image, an underexposure image, and an overexposure image that can be used to generate an HDR composite image are obtained. However, in this embodiment, the three types of exposure corresponding to the three images for generating an HDR composite image are not limited to an appropriate exposure, an underexposure, and an overexposure. As long as the relationship that the second exposure is lower than the first exposure and the third exposure is higher than the first exposure is satisfied, they can be used as the first exposure, the second exposure, and the third exposure.

[0056] In the example of FIG. 4, when N (number of times of shooting) is an odd number, exposure control (first exposure control) is performed to generate an image with appropriate exposure (an example of the first exposure) and an image with underexposure (an example of the second exposure). In addition, when N (number of times of shooting) is an even number, exposure control (second exposure control) is performed to generate an image with appropriate exposure (an example of the first exposure) and an image with overexposure (an example of the third exposure). Therefore, the alternating between the first exposure control (S404) and the second exposure control (S405) in each shooting starts with the first exposure control. However, a configuration may be adopted in which the alternating between the first exposure control (S404) and the second exposure control (S405) in each shooting starts with the second exposure control. In this case, a configuration may be adopted in which the process transitions from S403 to S405 when N is an odd number in FIG. 4, and the process transitions from S403 to S404 when N is an even number.

[0057] 7, two tables are provided for combinations of TV values ​​and ISO values ​​in the program diagram, and the ISO sensitivity of the optimum exposure is different for the cases of optimum exposure and underexposure (first exposure control) and optimum exposure and overexposure (second exposure control). However, a configuration may be adopted in which the ISO sensitivity of the optimum exposure is not changed between the first exposure control and the second exposure control, and a single table is used in which the ISO sensitivity of the underexposure and overexposure is set to ±2EV from the optimum exposure.

[0058] [Second embodiment] In the first embodiment, the details of the process of S409 are described with reference to FIG. 5(a). In the second embodiment, a modified example of the process of S409 is described with reference to FIG. 5(b). In this embodiment, the basic configuration of the imaging device 100 is similar to that of the first embodiment. Below, differences from the first embodiment will be mainly described.

[0059] FIG. 5B is a flowchart showing details of the process of S409 according to the second embodiment.

[0060] In S511, the image synthesis unit 105 generates an HDR synthesis image by synthesizing the previous intermediate synthesis image and the current unsuitable image. In the synthesis process, the image synthesis unit 105 performs a position adjustment process in the same manner as in S502. Furthermore, when the previous intermediate synthesis image is based on the suitable image and the over-exposed image, the image synthesis unit 105 performs the synthesis process according to the Mix table in FIG. 7(b). When the previous intermediate synthesis image is based on the suitable image and the under-exposed image, the image synthesis unit 105 performs the synthesis process according to the Mix table in FIG. 7(c). As a result, an HDR synthesis image is generated by synthesizing three images, the suitable image, the under-exposed image, and the over-exposed image.

[0061] In S512, the signal recording unit 106 stores the HDR composite image generated in S504 in a recording medium.

[0062] 4 and 5(b), every time the second (N=2) or subsequent (N>2) photographing is performed, in S511, an HDR composite image is generated from the previous intermediate composite image generated from the previous suitable and unsuitable images, and the current unsuitable image. Therefore, for each of the photographing times of N=2 to 5, an HDR composite image is generated by the combination of the images listed below, and for the photographing times of N=6 or more, the generation of an HDR composite image is repeated in a similar manner (the number in parentheses indicates the number of photographing times). N=2: Optimal image (1) + Underexposed image (1) + Overexposed image (2) N=3: Optimal image (2) + Overexposed image (2) + Underexposed image (3) N=4: Optimal image (3) + Underexposed image (3) + Overexposed image (4) N=5: Optimal image (4) + Overexposed image (4) + Underexposed image (5)

[0063] According to the second embodiment, the number of generated HDR composite images is one less than the number of shots, but all HDR composite images are generated by combining the previous suitable image, the previous unsuitable image, and the current unsuitable image. Therefore, when multiple HDR composite images generated in succession are used as frames of a video, the shooting timing between frames becomes uniform.

[0064] On the other hand, in the first embodiment, the same number of HDR composite images as the number of shots are generated, but as can be understood from a comparison of S502 and S504, the first and second HDR composite images are generated at the same timing (first and second shots). Therefore, when multiple HDR composite images generated in succession are used as frames of a video, the shooting timing between frames becomes uniform from the second frame onwards, but a timing discrepancy occurs between the first and second frames.

[0065] Therefore, a configuration may be adopted in which the process in S409 is switched between the process in FIG. 5(a) (first generation process) and the process in FIG. 5(b) (second generation process) depending on whether the shooting instruction is a still image shooting instruction or a video shooting instruction. By configuring the imaging device 100 to execute the process in FIG. 5(b) in S409 when a video shooting instruction is given, it becomes possible to generate a high-quality video with reduced timing deviation. Since a large number of frames are generally shot in the case of a video, the fact that the number of frames is one less than the number of shots is not a major disadvantage. In addition, by configuring the imaging device 100 to execute the process in FIG. 5(a) in S409 when a still image shooting instruction is given, it becomes possible to obtain an HDR composite image that is one more than the case of FIG. 5(b).

[0066] [Third embodiment] In the third embodiment, a configuration for reducing noise by synthesizing two suitable images when generating an HDR composite image will be described. In this embodiment, the basic configuration of the imaging device 100 is similar to that of the first embodiment. Below, differences from the first embodiment will be mainly described.

[0067] Fig. 8 is a flowchart of HDR shooting processing executed by the imaging device 100 according to the third embodiment. In the first and second embodiments, the processing shown in Fig. 4 is executed as the HDR shooting processing, but in the third embodiment, the processing shown in Fig. 8 is executed as the HDR shooting processing. In Fig. 8, the same reference numerals as in Fig. 4 are used for steps in which the same or similar processing as in Fig. 4 is executed.

[0068] In S807, the control unit 109 determines whether or not N is equal to or greater than 2. If N is equal to or greater than 2 (i.e., if there is an image captured prior to the current capture), the process proceeds to S808, and if N is not equal to or greater than 2 (i.e., if the current capture is the first capture), the process returns to S402.

[0069] In S808, the image synthesis unit 105 synthesizes the previous suitable image and the current suitable image to generate a suitable image with reduced noise (noise-reduced suitable image). The two suitable images synthesized in S808 have different shooting (exposure) timings. Therefore, in the synthesis process, the image synthesis unit 105 performs a positioning process similar to the case of S502. After the positioning process, the image synthesis unit 105 performs a noise reduction process to synthesize the previous suitable image and the current suitable image. As a noise reduction process method, any known method such as cyclic noise reduction can be used.

[0070] In S809, the image synthesis unit 105 generates an HDR synthesis image by synthesizing the current unsuitable image and the previous unsuitable image with the noise-reduced image generated in S808. When performing the synthesis process, the image synthesis unit 105 performs a positioning process similar to that in S502. Since the images to be synthesized here are an appropriate image (strictly speaking, an appropriate noise-reduced image), an underexposed image, and an overexposed image, the image synthesis unit 105 performs synthesis according to the Mix table in Fig. 7(a). As a result, an HDR synthesis image is generated by synthesizing three images, the appropriate image (strictly speaking, an appropriate noise-reduced image), an underexposed image, and an overexposed image.

[0071] In S810, the signal recording unit 106 stores the HDR composite image generated in S809 in a recording medium.

[0072] In S811, the control unit 109 determines whether or not to end the shooting. For example, if the release button is still pressed, the control unit 109 determines not to end the shooting, and if the release button is not pressed, the control unit 109 determines to end the shooting. If it is determined to end the shooting, the process of this flowchart ends, and if not, the process returns to S402.

[0073] As described above, according to the third embodiment, the imaging device 100 can generate an HDR composite image in which noise in pixels corresponding to optimal exposure is reduced by combining a suitable image generated in the current shooting, a suitable image generated in the previous shooting, an unsuitable image generated in the current shooting, and an unsuitable image generated in the previous shooting, for each shooting except the first shooting out of at least two shootings.

[0074] [Fourth embodiment] In the first to third embodiments, a configuration for saving an HDR composite image has been described, but a configuration for saving a suitable image and an unsuitable image before synthesis in addition to the HDR composite image may be adopted. Therefore, in the fourth embodiment, a configuration for saving a suitable image and an unsuitable image (an underexposed image or an overexposed image) obtained by one shooting as one RAW image file will be described.

[0075] 9(a) is a diagram showing the structure of a RAW image file 901. The container file format of the RAW image file described in this embodiment is the ISO base media file format defined in ISO / IEC14496-12. Therefore, the container format of the RAW image file has a tree structure, with each node called a box. Furthermore, each box can have multiple boxes as child elements.

[0076] The RAW image file 901 has at its beginning a box ftyp 902 for describing the file type. The RAW image file 901 also has a box moov 903 including all metadata, a box uuid 909 including an image for display, a box mdat 911 including the media data body (image data) of the track, and other boxes 908.

[0077] Box moov 903 has, as child elements, box uuid 904 that stores MetaData 905 and a composite THM image for display 906, and trak box 907 that stores information that references ImageData.

[0078] Metadata of the image is described in MetaData 905. MetaData 905 includes, for example, the date and time the image was created, the shooting conditions, information indicating whether the image was shot in HDR or SDR, information about the included RAW image (optimized image, underexposed image, or overexposed image), and other shooting information.

[0079] The box uuid 909 has, as a child element, a composite large THM image for display 910. The box mdat 911 has, as a child element, an ImageData 912.

[0080] 9B is a diagram showing an example of the configuration of ImageData 912 recorded as a still image when the image is kept within the dynamic range of SDR. In this case, ImageData 912 has a composite image for display 920, an image suitable for display 921, an image unsuitable for display 922, an image suitable for display 923, an image unsuitable for display 924, and RAW development parameters 925, which are JPEG compressed using SDR development. By recording multiple frames of this ImageData 912 as one frame in one file, multiple frames obtained by continuous shooting can be saved together in one file. Alternatively, multiple frames can be saved as a video RAW file.

[0081] When multiple frames are saved in one file, the underexposed image and the overexposed image may be saved as separate boxes, as shown in FIG. 9(c).

[0082] FIG. 9C is a diagram showing another example of the configuration of ImageData 912 recorded as a still image when the image is kept within the dynamic range of SDR. In this case, ImageData 912 has a composite image for display 920, an appropriate image for display 921, an underexposed image for display 926, and an overexposed image for display 927, which are JPEG compressed in SDR development. ImageData 912 also has an appropriate image 923 (first storage area), an underexposed image 928 (second storage area), an overexposed image 929 (third storage area), and RAW development parameters 925. By saving multiple frames in one file with this ImageData 912 as one frame, multiple RAW frames obtained by continuous shooting can be saved together in one file. Alternatively, multiple RAW frames can be saved as a video RAW file. When recording images using this method, a display suitable image 921 and a suitable image 923 are recorded every frame, and a pair of a display under-image 926 and an under-image 928 and a pair of a display over-image 927 and an over-image 929 are recorded alternately once every two frames.

[0083] The above describes the case where RAW images captured in SDR are recorded. When RAW images captured in HDR are recorded, the image data is HDR-developed and HEVC-compressed to be displayed, and then the image data is recorded as a display image (reference numerals 906, 910, 920, 921, 922, 926, 927).

[0084] It should be noted that the file format described above is merely an example, and a file format having other boxes may be used as necessary.

[0085] As described above, according to the fourth embodiment, the imaging device 100 saves in one RAW image file a suitable image and an unsuitable image generated by one shooting using the imaging element 102. Furthermore, the imaging device 100 may save in one RAW image file each image generated by two or more shootings using the imaging element 102.

[0086] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0087] [summary] The above-described embodiment discloses at least the inventions shown in the following items, but is not limited to these inventions. [Item 1] a photographing control means for controlling the photographing to be performed at least two times in response to a photographing instruction, using an image sensor configured to generate a first image and a second image having different exposures in one photographing; An exposure control means for alternately performing a first exposure control or a second exposure control in each of the at least two photographing operations, the first exposure control is an exposure control such that the first image is generated at a first exposure and the second image is generated at a second exposure lower than the first exposure; the second exposure control is an exposure control such that the first image is generated at the first exposure and the second image is generated at a third exposure that is higher than the first exposure. An exposure control means; An imaging control device comprising: [Item 2] a generating unit configured to generate a High Dynamic Range (HDR) composite image by synthesizing the first image generated by the current shooting, the second image generated by the current shooting, and the second image generated by the previous shooting, for each of the at least two shootings except for a first shooting. 2. The imaging control device according to item 1, [Item 3] The generating means generates an HDR composite image by combining the first image generated by the first shooting, the second image generated by the first shooting, and the second image generated by the second shooting. 3. The imaging control device according to item 2, [Item 4] a generating unit configured to generate an HDR composite image by synthesizing the first image generated by the previous shooting, the second image generated by the previous shooting, and the second image generated by the current shooting, for each of the at least two shootings except for a first shooting. 2. The imaging control device according to item 1, [Item 5] a generation unit that performs a first generation process when the shooting instruction is an instruction to shoot a still image, and performs a second generation process when the shooting instruction is an instruction to shoot a video, The first generation process includes: For each of the at least two photographings except for a first photographing, an HDR composite image is generated by combining the first image generated by the current photographing, the second image generated by the current photographing, and the second image generated by a previous photographing; a process of generating an HDR composite image by combining the first image generated by the first shooting, the second image generated by the first shooting, and the second image generated by the second shooting, The second generation process includes: A process of generating an HDR composite image by synthesizing the first image generated by the previous shooting, the second image generated by the previous shooting, and the second image generated by the current shooting, for each shooting except for the first shooting of the at least two shootings. 2. The imaging control device according to item 1, [Item 6] a generating unit configured to generate an HDR composite image in which noise of pixels corresponding to the first exposure is reduced by synthesizing the first image generated by the current shooting, the first image generated by the previous shooting, the second image generated by the current shooting, and the second image generated by the previous shooting, for each of the at least two shootings except for a first shooting. 2. The imaging control device according to item 1, [Item 7] The generating means includes: generating an image with reduced noise by synthesizing the first image generated by the current shooting and the first image generated by the previous shooting; The noise-reduced image, the second image generated by the current shooting, and the second image generated by the previous shooting are combined to generate an HDR combined image in which noise in pixels corresponding to the first exposure is reduced. 7. The imaging control device according to item 6, [Item 8] a storage unit that stores the first image and the second image in one RAW image file in each of the at least two shootings. 8. The imaging control device according to claim 1, wherein: [Item 9] a storage unit that generates a RAW image file including a first storage area corresponding to the first exposure, a second storage area corresponding to the second exposure, and a third storage area corresponding to the third exposure, stores each first image obtained by each of the at least two shootings in the first storage area, and stores each second image obtained by each of the at least two shootings in the second storage area or the third storage area depending on the exposure of each second image. 8. The imaging control device according to claim 1, wherein: [Item 10] In each of the at least two photographings in which the second exposure control is performed, except for the first photographing, the exposure control means controls the ISO sensitivity for the second image in the current photographing so as to match the ISO sensitivity for the first image in the previous photographing. 10. The imaging control device according to any one of items 1 to 9, [Item 11] the first exposure is a moderate exposure, the second exposure is an underexposure; The third exposure is an overexposure 11. The imaging control device according to any one of items 1 to 10, [Item 12] In the first of the at least two photographing operations, the exposure control means performs the first exposure control. 12. The imaging control device according to any one of items 1 to 11, [Item 13] The photographing control means controls the photographing so that the photographing is performed three or more times as the photographing at least two times. 13. The imaging control device according to any one of items 1 to 12, [Item 14] An imaging control device according to any one of items 1 to 13, The imaging element; An imaging device comprising: [Item 15] A photographing control method executed by a photographing control device, a photographing control step of controlling the photographing to be performed at least two times using an image sensor configured to generate a first image and a second image having different exposures in one photographing in response to a photographing instruction for generating a high dynamic range (HDR) composite image; An exposure control step of alternately performing a first exposure control or a second exposure control in each of the at least two photographing operations, the first exposure control is an exposure control such that the first image is generated at a first exposure and the second image is generated at a second exposure lower than the first exposure; the second exposure control is an exposure control such that the first image is generated at the first exposure and the second image is generated at a third exposure that is higher than the first exposure. an exposure control step; A shooting control method comprising: [Item 16] A program for causing a computer to function as each of the means of the imaging control device according to any one of items 1 to 13.

[0088] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0089] 100: imaging device, 101: optical lens, 102: imaging element, 103: image acquisition section, 104: signal processing section, 105: image synthesis section, 106: signal recording section, 107: exposure control section, 108: imaging element control section, 109: control section

Claims

1. an image capturing control means for controlling the image capturing device to capture at least two images in response to an image capturing instruction, the image capturing device being configured to generate a first image and a second image having different exposures in one image capturing operation; an exposure control unit that alternately performs a first exposure control or a second exposure control in each of the at least two photographing operations, the first exposure control is an exposure control such that the first image is generated at a first exposure and the second image is generated at a second exposure lower than the first exposure; the second exposure control is an exposure control such that the first image is generated at the first exposure and the second image is generated at a third exposure that is higher than the first exposure. An exposure control means; An imaging control device comprising:

2. a generating unit configured to generate a High Dynamic Range (HDR) composite image by synthesizing the first image generated by the current shooting, the second image generated by the current shooting, and the second image generated by the previous shooting, for each of the at least two shootings except for a first shooting.

2. The imaging control device according to claim 1 .

3. The generating means generates an HDR composite image by combining the first image generated by the first shooting, the second image generated by the first shooting, and the second image generated by the second shooting.

3. The imaging control device according to claim 2.

4. a generating unit configured to generate an HDR composite image by synthesizing the first image generated by the previous shooting, the second image generated by the previous shooting, and the second image generated by the current shooting, for each of the at least two shootings except for a first shooting.

2. The imaging control device according to claim 1 .

5. a generation unit that performs a first generation process when the shooting instruction is an instruction to shoot a still image, and performs a second generation process when the shooting instruction is an instruction to shoot a video, The first generation process includes: generating an HDR composite image by combining the first image generated by the current shooting, the second image generated by the current shooting, and the second image generated by the previous shooting, for each of the at least two shootings except for a first shooting; a process of generating an HDR composite image by combining the first image generated by the first shooting, the second image generated by the first shooting, and the second image generated by the second shooting, The second generation process includes: A process of generating an HDR composite image by synthesizing the first image generated by the previous shooting, the second image generated by the previous shooting, and the second image generated by the current shooting, for each shooting except for the first shooting of the at least two shootings.

2. The imaging control device according to claim 1 .

6. a generating unit configured to generate an HDR composite image in which noise of pixels corresponding to the first exposure is reduced by synthesizing the first image generated by the current shooting, the first image generated by the previous shooting, the second image generated by the current shooting, and the second image generated by the previous shooting, for each of the at least two shootings except for a first shooting.

2. The imaging control device according to claim 1 .

7. The generating means includes: generating an image in which noise is reduced by synthesizing the first image generated by the current shooting and the first image generated by the previous shooting; The noise-reduced image, the second image generated by the current shooting, and the second image generated by the previous shooting are combined to generate an HDR combined image in which noise in pixels corresponding to the first exposure is reduced.

7. The imaging control device according to claim 6,

8. a storage unit that stores the first image and the second image in one RAW image file in each of the at least two shootings.

2. The imaging control device according to claim 1 .

9. a storage unit that generates a RAW image file including a first storage area corresponding to the first exposure, a second storage area corresponding to the second exposure, and a third storage area corresponding to the third exposure, stores each first image obtained by each of the at least two shootings in the first storage area, and stores each second image obtained by each of the at least two shootings in the second storage area or the third storage area depending on the exposure of each second image.

2. The imaging control device according to claim 1 .

10. In each of the at least two photographings in which the second exposure control is performed, except for the first photographing, the exposure control means controls the ISO sensitivity for the second image in the current photographing so as to match the ISO sensitivity for the first image in the previous photographing.

2. The imaging control device according to claim 1 .

11. the first exposure is a moderate exposure; the second exposure is an underexposure; The third exposure is an overexposure.

2. The imaging control device according to claim 1 .

12. In the first of the at least two photographing operations, the exposure control means performs the first exposure control.

2. The imaging control device according to claim 1 .

13. The photographing control means controls the photographing so that the photographing is performed three or more times as the photographing at least two times.

2. The imaging control device according to claim 1 .

14. The imaging control device according to any one of claims 1 to 13, The imaging element; An imaging device comprising:

15. A photographing control method executed by a photographing control device, an image capturing control step of controlling the image capturing device to capture at least two images in response to an image capturing instruction, the image capturing device being configured to generate a first image and a second image having different exposures in one image capturing; an exposure control step of alternately performing a first exposure control or a second exposure control in each of the at least two photographing operations, the first exposure control is an exposure control such that the first image is generated at a first exposure and the second image is generated at a second exposure lower than the first exposure; the second exposure control is an exposure control such that the first image is generated at the first exposure and the second image is generated at a third exposure that is higher than the first exposure. an exposure control step; A shooting control method comprising:

16. A program for causing a computer to function as each of the means of the imaging control device according to any one of claims 1 to 13.

Images