Photographing control apparatus, imaging apparatus, storage medium, computer program product, and photographing control method

Abstract

The invention provides an imaging control apparatus, an imaging apparatus, a storage medium, a computer program product, and an imaging control method. The photographing control apparatus includes a photographing control unit and an exposure control unit. The photographing control unit controls at least two photographs to be performed in response to a photographing instruction using an image sensor configured to generate a first image and a second image having different exposures in a single photographing. The exposure control unit alternately performs the first exposure control and the second exposure control between the imaging in each of the at least two imaging. The first exposure control is an exposure control that generates a first image with a first exposure and generates a second image with a second exposure lower than the first exposure. The second exposure control is an exposure control that generates a first image with a first exposure and generates a second image with a third exposure that is higher than the first exposure.

Description

Technical Field

[0001] The present invention relates to a shooting control device, a camera device, a storage medium, a computer program product and a shooting control method. Background Art

[0002] It is known that an image sensor in which two input memories are set in a column amplification unit and two images with different gains can be output in the case of a single shot (see Japanese Patent Laid-Open No. 2020-178186). In this specification, such an image sensor will be referred to as a dual gain output (DGO) image sensor. Since the gain of the image sensor is related to the ISO sensitivity, the DGO image sensor can be used to generate two images with different exposure levels by generating two images with different gains. Japanese Patent Laid-Open No. 2020-178186 discloses a technology for generating a high dynamic range (HDR) composite image by synthesizing two images generated from a single shot using a DGO image sensor.

[0003] In general, by using a greater number of images with different exposures, the tone characteristics of an HDR composite image can be improved. However, Japanese Patent Laid-Open No. 2020-178186 does not mention increasing the number of images used to generate an HDR composite image. Summary of the invention

[0004] The present invention has been made in view of such circumstances, and provides a technique for using a DGO image sensor to generate at least three images having different exposures, which can then be used to generate an HDR composite image.

[0005] According to a first aspect of the present invention, a shooting control device is provided, comprising: a shooting control unit, which is configured to control at least two shootings to be performed using an image sensor in response to a shooting instruction, the image sensor being configured to generate a first image and a second image with different exposures in a single shooting; and an exposure control unit, which is configured to perform first exposure control and second exposure control alternately between shootings in each of the at least two shootings, wherein the first exposure control is an exposure control for generating the first image with a first exposure and generating the second image with a second exposure lower than the first exposure, and the second exposure control is an exposure control for generating the first image with the first exposure and generating the second image with a third exposure higher than the first exposure.

[0006] According to a second aspect of the present invention, there is provided an image pickup apparatus including: the shooting control apparatus according to the first aspect; and an image sensor.

[0007] According to a third aspect of the present invention, there is provided a computer-readable storage medium storing a computer program for causing a computer to function as the photographing control device according to the first aspect.

[0008] According to a fourth aspect of the present invention, there is provided a computer program product comprising a computer program for causing a computer to function as the photographing control apparatus according to the first aspect.

[0009] According to a fifth aspect of the present invention, there is provided a shooting control method performed by a shooting control device, comprising: controlling at least two shootings to be performed using an image sensor in response to a shooting instruction, the image sensor being configured to generate a first image and a second image with different exposures in a single shooting; and in each of the at least two shootings, alternating between shootings a first exposure control and a second exposure control, wherein the first exposure control is an exposure control for generating the first image with a first exposure and generating the second image with a second exposure lower than the first exposure, and the second exposure control is an exposure control for generating the first image with the first exposure and generating the second image with a third exposure higher than the first exposure.

[0010] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram showing the configuration of the imaging apparatus 100 .

[0012] Figure 2 is a diagram showing the configuration of the image sensor 102 .

[0013] Figure 3 2 is a conceptual diagram showing a circuit focusing on one column of the column amplifiers 204 .

[0014] Figure 4 : is a flowchart showing the HDR shooting process performed by the image pickup apparatus 100 according to the first embodiment and the second embodiment.

[0015] Figure 5A is a flowchart illustrating the processing of step S409 according to the first embodiment in detail.

[0016] Figure 5B is a flowchart illustrating the processing of step S409 according to the second embodiment in detail.

[0017] Figure 6 is a diagram showing an example of a program diagram used in HDR photographing processing.

[0018] FIG. 7A to FIG. 7CIt is a diagram showing the addition ratio of each image in the HDR synthesis process.

[0019] Figure 8 : is a flowchart showing the HDR shooting process performed by the image pickup apparatus 100 according to the third embodiment.

[0020] Fig. 9A is a diagram showing the structure of a RAW image file 901 .

[0021] Fig. 9B is a diagram showing an example of the configuration of the image data 912 .

[0022] Fig. 9C is a diagram showing another example of the configuration of the image data 912 . DETAILED DESCRIPTION

[0023] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but are not limited to inventions requiring all such features, and a plurality of such features may be appropriately combined. In addition, in the accompanying drawings, the same reference numerals are given to the same or similar configurations, and their redundant descriptions are omitted.

[0024] First embodiment

[0025] Figure 1 1 is a block diagram showing the configuration of an image pickup apparatus 100 that can function as a shooting control apparatus. Figure 1 In FIG. 1 , the optical lens 101 is an optical lens for capturing light from a subject and forming the light into an image on the image sensor 102 .

[0026] The image sensor 102 receives incident light from the optical lens 101, converts the light into an electrical signal, and outputs the electrical signal. As examples of the image sensor 102, a charge coupled device (CCD) image sensor and a complementary metal oxide semiconductor (CMOS) image sensor, etc., can be given. One example of the image sensor 102 is a sensor that directly outputs a generated analog video signal. Another example of the image sensor 102 is a sensor that performs analog-to-digital (AD) conversion on a generated analog video signal internally and outputs digital data such as low voltage differential signal (LVDS) data.

[0027] Here we will refer to Figure 2 The configuration of the image sensor 102 is described below. Figure 2 In the example shown, image sensor 102 is a sensor that outputs digital data. Figure 2In the present invention, the timing pulse control unit 201 controls the operation of the image sensor 102 by supplying an operation clock (CLK) to each block in the image sensor 102, and supplying a timing signal to each block, etc.

[0028] The vertical scanning circuit 202 controls the timing for sequentially reading out the pixel signal voltage of the pixel unit 203 in which a plurality of pixels are arranged two-dimensionally during a single frame. In a single frame, the video signal is typically read out sequentially in row units from the top row to the bottom row.

[0029] The pixel unit 203 is a photoelectric conversion device for photoelectrically converting incident light and outputting a voltage corresponding to the intensity of the incident light. The pixel unit 203 converts the captured light into electric charge and accumulates the electric charge in a capacitive floating diffusion (FD). The capacity of the FD can be changed, and the signal-to-noise ratio (SN ratio) can be improved by changing the capacity according to the ISO sensitivity. Generally, when the ISO sensitivity is low, the capacity is set to "large", and when the ISO sensitivity is high, the capacity is set to "small". Note that in the case of outputting an image with two different gains (described later), the capacity for accumulating charge is the same for the two gains. In addition, the capacity that can be set is not limited to two types, namely "large" and "small", and a configuration in which the capacity can be set to three or more levels is also possible.

[0030] The column amplifier 204 is used to electrically amplify the signal read out from the pixel unit 203. Amplifying the signal using the column amplifier 204 makes it possible to amplify the pixel signal level relative to the noise generated in the column ADC 205 in the subsequent stage, which substantially improves the SN ratio. The gain of the column amplifier can also be changed from the timing pulse control unit 201.

[0031] The image sensor 102 includes two input memories in the column amplifier 204, and by changing the gain in the column amplifier 204, it is possible to output signals with two gains to generate an HDR composite image. Having two input memories makes it possible to output two gains respectively for the signal read out from the FD at a specific moment. Although this increases the amount of data, it also makes it possible to obtain two images with different gains with simultaneity. Note that a configuration in which three or more images with different gains with simultaneity are output can be used as the configuration of the imaging sensor 102.

[0032] The column ADC 205 performs analog-to-digital conversion on the readout signal from the column amplifier 204. The digitized signal is sequentially read out by the horizontal transfer circuit 206. The output of the horizontal transfer circuit 206 is input to the signal processing circuit 207.

[0033] The signal processing circuit 207 is a circuit for digitally processing a signal. The signal processing circuit 207 can easily perform gain calculation by adding a set amount of offset value to the signal through digital processing, and performing shift calculation and multiplication, etc. The pixel unit 203 may also be provided with a pixel area that is intentionally shielded from light. In this case, the signal processing circuit 207 can use the shielded pixel area to perform a digital black level clamping operation. The output of the signal processing circuit 207 is transmitted to the external output circuit 208.

[0034] The external output circuit 208 has a serialization function, and converts a multi-bit input parallel signal from the signal processing circuit 207 into a serial signal. The external output circuit 208 converts the serial signal into, for example, an LVDS signal, and outputs the signal to the outside of the image sensor 102.

[0035] Return to Figure 1 , the image acquisition unit 103 acquires the video signal output by the image sensor 102 and performs various types of processing. In the case where the image sensor 102 is a type that outputs an analog video signal (in the case where AD conversion is not performed in the image sensor 102), the image acquisition unit 103 includes an analog front end for performing AD conversion. The image acquisition unit 103 removes the fixed pattern noise generated by the image sensor 102, and performs black level clamping processing, etc. The image acquisition unit 103 also has the function of separating the video signal output by the image sensor 102 into an image signal for recording and an evaluation signal for controlling the image sensor 102.

[0036] The signal processing unit 104 performs various types of image processing, including a pixel addition function, noise reduction, gamma correction, knee correction, digital gain, and other types of defect correction, which are typical image processing functions of the imaging apparatus 100. The image acquisition unit 103 and the signal processing unit 104 also include a storage circuit (not shown) for storing setting values ​​necessary for the correction and image processing performed by each of these units.

[0037] The image synthesis unit 105 generates an HDR synthesized image using any desired synthesis method according to the HDR generation signal output from the image sensor 102. As an example, there is a synthesis method that uses a high-gain image as a normal image and a low-gain image for a bright and blown-out portion in the normal image. However, the synthesis method of the present embodiment is not particularly limited, and any synthesis method (synthesis algorithm) can be used as long as two images with different gains are used for synthesis.

[0038] The signal recording unit 106 records the video signal received from the image synthesis unit 105 into a storage device or a storage medium.

[0039] The exposure control unit 107 can calculate the optimum exposure amount based on information in the video signal received from the image obtaining unit 103. Then, the exposure control unit 107 determines an operation to be performed by the image sensor control unit 108, and communicates the determined operation to the image sensor control unit 108.

[0040] The control unit 109 includes a ROM and a RAM, and controls the imaging apparatus 100 as a whole by executing a control program stored in the ROM, using the RAM as a working memory.

[0041] Next, we will refer to Figure 3 Operations performed by the image sensor 102 in the case of generating an HDR composite image are described. Figure 3 is a conceptual diagram showing a circuit focusing on one column in column amplifier 204. As previously described, image sensor 102 may vary the gain in column amplifier 204 to generate an HDR composite image.

[0042] OP305 is an operational amplifier, and an input capacitor and a feedback capacitor are connected to OP305. C303 and C304 are input capacitors, and SW301 and SW302 are switches. A signal read out from the pixel unit 203 is input to C303 and C304 via SW301 and SW302.

[0043] C 306 and C 308 are feedback capacitors, and SW 307 is a switch. The connection of C 308 can be controlled by SW 307. The column amplifier 204 uses capacitors, so the amplification ratio is (input capacitance / feedback capacitance).

[0044] As described above, the column amplifier 204 has two input capacitors. First, the column amplifier 204 applies a gain based on C303 and C306 to the input signal by turning on SW301 and turning off SW302 and SW307, and outputs the signal to the column ADC 205. Next, the column amplifier 204 applies a gain based on C304, C306, and C308 to the input signal by turning off SW301 and turning on SW302 and SW307, and outputs the signal. This makes it possible to output two images with different gains applied.

[0045] Figure 4 1 is a flowchart showing the HDR shooting process performed by the image pickup apparatus 100. Unless otherwise specified, the processing of each step in this flowchart is performed under the control of the control unit 109 operating according to the control program. When the power switch of the image pickup apparatus 100 is turned on, the operation mode is set to the HDR shooting mode, and when the release button of the image pickup apparatus 100 is pressed, the processing of this flowchart is started.

[0046] In step S401, the control unit 109 resets N to 0. N is a variable indicating the number of times shooting is performed.

[0047] In step S402, the control unit 109 adds 1 to N. When the process of step S402 is performed for the first time, N is set to 1, indicating that the first shooting is performed.

[0048] In step S403, the control unit 109 determines whether N is an odd number. If N is an odd number (ie, the number of shots is an odd number), the sequence moves to step S404, and if N is not an odd number (ie, the number of shots is an even number), the sequence moves to step S405.

[0049] In step S404 , the exposure control unit 107 obtains an appropriate exposure value (EV) through automatic exposure (AE) processing, and determines exposure conditions (ISO sensitivity, aperture, and time value) corresponding to both appropriate exposure and underexposure.

[0050] Figure 6 It is shown in Figure 4 FIG. 1 is a diagram showing an example of a program diagram used in an HDR shooting process. Figure 6 In FIG. 1 , the vertical axis on the right side represents the aperture value (AV), the left side and the upper part represent the EV value, and the lower part represents the time value (TV) and the ISO sensitivity value. The exposure control unit 107 refers to the program chart to determine appropriate exposure conditions (ISO sensitivity, aperture, time value) based on the control value corresponding to the EV value obtained earlier, and controls the exposure according to these conditions. Figure 6 , line 602 represents proper exposure, line 601 represents overexposure, and line 603 represents underexposure.

[0051] Consider the case where a DGO image sensor such as the image sensor 102 is used to capture images at both proper exposure and underexposure. In the case of capturing using the DGO image sensor, it is not possible to set different TV values ​​for proper exposure and underexposure, so it is necessary to capture using a combination of exposure conditions in which the underexposure falls within the lowest ISO sensitivity. Therefore, for example, in the case of capturing underexposure at -2EV, the combination represented in the box 605 is used as the TV value and the ISO value. For example, in the case where the brightness value (BV value) is 10 and the AV value is 2.8, a combination of a TV value of 250 and an ISO value of 400 is used for proper exposure, and a combination of a TV value of 250 and an ISO value of 100 is used for underexposure. Therefore, in this example, the exposure control unit 107 sets the exposure condition corresponding to proper exposure to: ISO value 400, AV value 2.8, TV value 250, and sets the exposure condition corresponding to underexposure to: ISO value 100, AV value 2.8, TV value 250.

[0052] In step S405, the exposure control unit 107 obtains an appropriate exposure value (EV) by automatic exposure (AE) processing, and determines exposure conditions (ISO sensitivity, aperture, and time value) corresponding to both appropriate exposure and overexposure by the same method as that used in step S404. Note that as the ISO sensitivity increases at the time of overexposure, image noise increases, and therefore, it is desirable that the exposure control unit 107 controls the ISO sensitivity for appropriate exposure determined in step S404 and the ISO sensitivity for overexposure determined in step S405 so that these sensitivities are consistent. Therefore, for example, in the case of shooting overexposure at +2eV, the combination indicated in the frame 604 is used as the TV value and the ISO value. For example, in the case of a BV value of 10 and an AV value of 2.8, a combination of a TV value of 60 and an ISO value of 100 is used for appropriate exposure, and a combination of a TV value of 60 and an ISO value of 400 is used for overexposure. Therefore, in this example, the exposure control unit 107 sets the exposure conditions corresponding to proper exposure to ISO value 100, AV value 2.8, and TV value 60, and sets the exposure conditions corresponding to overexposure to ISO value 400, AV value 2.8, and TV value 60.

[0053] In step S406, the control unit 109 controls shooting to be performed according to the two exposure conditions (ISO value, AV value, and TV value) determined in step S404 or step S405. At this time, the exposure control unit 107 communicates the two ISO values ​​under the two exposure conditions determined in step S404 or step S405 to the imaging element control unit 108 under the control of the control unit 109. The image sensor control unit 108 sets the two gains of the image sensor 102 based on the two ISO values ​​communicated here. Because shooting is performed according to such control, two images with different exposures are generated.

[0054] In step S407, the image synthesis unit 105 generates an intermediate synthesis image by synthesizing the two images obtained in step S406. The two images synthesized here are two images obtained from a single shot (single exposure) using the image sensor 102 as a DGO image sensor. Therefore, there is no need to perform alignment processing on the two images during the synthesis process.

[0055] Generally, when an HDR composite image is generated from three images at proper exposure, overexposure, and underexposure, a method such as Fig. 7A The image synthesis is performed using a mixing table for determining the addition ratio of the images, such as the mixing table shown in FIG. Fig. 7AIn the mixing table, the addition ratio 701 corresponding to overexposure, the addition ratio 702 corresponding to proper exposure, and the addition ratio 703 corresponding to underexposure are set as weighted addition ratios that roughly divide the brightness range into three parts. Based on the mixing table, in order to obtain an HDR composite image, an overexposed image (hereinafter abbreviated as "over-image") can be used for a dark area, a properly exposed image (hereinafter abbreviated as "proper image") can be used for a medium brightness area, and an underexposed image (hereinafter abbreviated as "under-image") can be used for a bright area. For the boundary area between the dark part and the middle part and the boundary area between the bright part and the middle part, the effect of smoothing the transition between images can be obtained by gradually changing the synthesis ratio according to the weighted addition ratio in the mixing table.

[0056] However, in this embodiment, the synthesis process of the two images obtained by this shooting is performed at the time of step S407. Therefore, in the case where the two images are a proper image and an under-exposure image, the image synthesis unit 105 synthesizes the two images according to the addition ratio 712 corresponding to the proper exposure and the addition ratio 703 corresponding to the under-exposure (such as Figure 7B Similarly, in the case where the two images are a proper image and an over-exposure image, the image synthesis unit 105 performs synthesis processing according to the addition ratio 722 corresponding to the proper exposure and the addition ratio 701 corresponding to the over-exposure (as shown in FIG. Figure 7C ) to perform the synthesis process. Note that Figure 7B The addition ratio 712 is Fig. 7A The synthesis of the addition ratio 701 and the addition ratio 702 in Figure 7C The addition ratio 722 is Fig. 7A The synthesis of the addition ratio 702 and the addition ratio 703.

[0057] In step S408, the control unit 109 determines whether N is at least 2. In the case where N is at least 2 (i.e., in the case where there is an image captured before the current shooting), the sequence moves to step S409, and in the case where N is not at least 2 (i.e., in the case where the current shooting is the first shooting), the sequence returns to step S402.

[0058] In step S409, a process is performed to generate an HDR composite image from the intermediate composite image and another image. Figure 5A The processing of step S409 is described in detail.

[0059] In step S410, the control unit 109 determines whether to end shooting. For example, if the release button continues to be pressed, the control unit 109 determines that shooting is not to be ended, and if the release button is not pressed, the control unit 109 determines that shooting is to be ended. If the control unit 109 determines that shooting is to be ended, the process of this flowchart ends, and if the control unit 109 does not determine that shooting is to be ended, it returns to step S402.

[0060] Next, we will refer to Figure 5A The processing of step S409 is described in detail. In step S501, the control unit 109 determines that N = 2. In the case of N = 2 (ie, the first example of the processing for step S409), the sequence moves to step S502, and in the case of N≠2, the sequence moves to step S504.

[0061] In step S502, the image synthesis unit 105 generates an HDR composite image by synthesizing the previous intermediate composite image with the current inappropriate image. The "previous intermediate composite image" refers to the intermediate composite image generated in step S407 based on the two images obtained in the previous shooting (N-1th shooting). The "current inappropriate image" refers to an image that is not a proper image among the two images obtained in the current shooting (Nth shooting), which is an under-image when N is an odd number, and an over-image when N is an even number. When N=2, the processing of step S502 is performed, so the current inappropriate image is an over-image.

[0062] The previous intermediate composite image and the current inappropriate image composited in step S502 have different shooting (exposure) timings. Therefore, when composited, the image composite unit 105 aligns the previous intermediate composite image with the current inappropriate image. Any known method such as a method of detecting the offset between the two images by frequency analysis can be used as an alignment processing method. After the alignment processing, the image composite unit 105 performs the alignment processing according to Figure 7C The previous intermediate composite image (the image generated by combining the previous appropriate image with the under-image) and the current inappropriate image (the current over-image) are composited using the blending table in. As a result, an HDR composite image is generated by composite three images consisting of the appropriate image, the under-image, and the over-image.

[0063] In step S503 , the signal recording unit 106 stores the HDR composite image generated in step S502 in a recording medium.

[0064] In step S504, the image synthesis unit 105 generates an HDR composite image by synthesizing the intermediate synthetic image of this time with the inappropriate image of the previous time. The "intermediate synthetic image of this time" refers to the intermediate synthetic image generated in step S407 based on the two images obtained from this shooting (Nth shooting). The "previous inappropriate image" refers to an image that is not a proper image among the two images obtained in the previous shooting (N-1th shooting), which is an under-image when N-1 is an odd number, and an over-image when N-1 is an even number. The image synthesis unit 105 performs alignment processing in the same manner as step S502 when performing synthesis processing. In the case where the intermediate synthetic image of this time is based on the proper image and the over-image, the image synthesis unit 105 performs alignment processing in accordance with step S502. Figure 7B In the case where the intermediate synthesized image of this time is based on the appropriate image and the under-image, the image synthesizing unit 105 performs synthesis processing according to Figure 7C As a result, an HDR composite image obtained by synthesizing three images consisting of a proper image, an under-image, and an over-image is generated.

[0065] In step S505 , the signal recording unit 106 stores the HDR composite image generated in step S504 in a recording medium.

[0066] In this way, according to Figure 4 and Figure 5A , whenever the second shooting (N=2) and subsequent shooting (N>2) are performed, in step S504, an HDR composite image is generated based on the intermediate composite image of this time (which is generated based on the appropriate image and the inappropriate image of this time) and the inappropriate image of the previous time. In the case where the second shooting (N=2) is performed, an HDR composite image is also generated based on the intermediate composite image of the previous time (which is generated based on the appropriate image and the inappropriate image of the previous time) and the inappropriate image of this time. Therefore, for each shooting of N=2 to 5, an HDR composite image is generated by combining the following images, and for each shooting of N=6 or more, the HDR composite image is repeatedly generated in the same manner (the number in the brackets indicates the number of shootings).

[0067] N = 2: proper image (1) + under-image (1) + over-image (2)

[0068] N = 2: appropriate image (2) + over-image (2) + under-image (1)

[0069] N = 3: proper image (3) + under-image (3) + over-image (2)

[0070] N=4: proper image (4) + over-image (4) + under-image (3)

[0071] N=5: proper image (5) + under-image (5) + over-image (4)

[0072] As described above, according to the first embodiment, the camera device 100 controls at least two shots to be performed using the image sensor 102 in response to a shooting instruction. The camera device 100 can control more than three shots to be performed as at least two shots. The image sensor 102 is a DGO image sensor configured to generate two images (a first image and a second image) with different exposures in a single shot. The camera device 100 performs the first exposure control (step S404) and the second exposure control (step S405) alternately between shots in each of the at least two shots. The first exposure control is an exposure control that generates a first image with a first exposure and generates a second image with a second exposure lower than the first exposure. The second exposure control is an exposure control that generates a first image with a first exposure and generates a second image with a third exposure higher than the first exposure.

[0073] Therefore, according to the first embodiment, a DGO image sensor may be used to generate at least three images with different exposures that may be used to generate an HDR composite image.

[0074] Note that Figure 4 In the example of , the first exposure is proper exposure, the second exposure is underexposure, and the third exposure is overexposure, so a proper image, underimage, and overimage that can be used to generate an HDR composite image are obtained. However, in this embodiment, the three types of exposure corresponding to the three images used to generate the HDR composite image are not limited to proper exposure, underexposure, and overexposure. As long as the relationship that the second exposure is lower than the first exposure and the third exposure is higher than the first exposure is satisfied, any exposure can be used as the first exposure, the second exposure, and the third exposure.

[0075] exist Figure 4 In the example, when N (the number of shots) is an odd number, exposure control (first exposure control) is performed to generate an image with proper exposure (an example of the first exposure) and underexposure (an example of the second exposure). In addition, when N (the number of shots) is an even number, exposure control (second exposure control) is performed to generate an image with proper exposure (an example of the first exposure) and overexposure (an example of the third exposure). Therefore, starting from the first exposure control, the first exposure control (step S404) and the second exposure control (step S405) are alternately performed between shots. However, it is also possible to alternatively adopt a configuration in which starting from the second exposure control, the first exposure control (step S404) and the second exposure control (step S405) are alternately performed between shots. In this case, in Figure 4In the embodiment, a configuration may be adopted in which the sequence moves from step S403 to step S405 when N is an odd number, and the sequence moves from step S403 to step S404 when N is an even number.

[0076] In addition, FIG. 7A to FIG. 7C In the example of , two tables are set for the combination of TV value and ISO value in the program diagram, and a configuration is adopted in which the ISO sensitivity for proper exposure is different between the combination of proper exposure and underexposure (first exposure control) and the combination of proper exposure and overexposure (second exposure control). However, a configuration using a single table (in which the ISO sensitivity for underexposure and overexposure differs by ±2EV from proper exposure) may be adopted instead of changing the ISO sensitivity for proper exposure between the first exposure control and the second exposure control.

[0077] Second embodiment

[0078] First Embodiment Reference Figure 5A The processing of step S409 is described in detail. Figure 5B A modification of the process of step S409 is described. In this embodiment, the basic configuration of the image pickup apparatus 100 is the same as that of the first embodiment. Details different from the first embodiment will be mainly described below.

[0079] Figure 5B is a flowchart illustrating the processing of step S409 according to the second embodiment in detail.

[0080] In step S511, the image synthesis unit 105 generates an HDR synthesized image by synthesizing the previous intermediate synthesized image with the current inappropriate image. The image synthesis unit 105 performs alignment processing in the same manner as step S502 when performing the synthesis processing. In the case where the previous intermediate synthesized image is based on the appropriate image and the over-image, the image synthesis unit 105 generates an HDR synthesized image based on the appropriate image and the over-image. Figure 7B In the case where the previous intermediate synthesized image is based on the appropriate image and the under-image, the image synthesizing unit 105 performs synthesis processing according to Figure 7C As a result, an HDR composite image obtained by synthesizing three images consisting of a proper image, an under-image, and an over-image is generated.

[0081] In step S512 , the signal recording unit 106 stores the HDR composite image generated in step S504 in a recording medium.

[0082] In this way, according to Figure 4 and Figure 5B, when the second shooting (N=2) and subsequent shooting (N>2) are performed, in step S511, an HDR composite image is generated based on the previous intermediate composite image (which is generated based on the previous appropriate image and inappropriate image) and the inappropriate image of this time. Therefore, for each shooting of N=2 to 5, an HDR composite image is generated by combining the following images, and for each shooting of N=6 or more, the HDR composite image is repeatedly generated in the same manner (the number in the brackets indicates the number of shootings).

[0083] N = 2: proper image (1) + under-image (1) + over-image (2)

[0084] N=3: proper image (2) + over-image (2) + under-image (3)

[0085] N=4: proper image (3) + under-image (3) + over-image (4)

[0086] N=5: proper image (4) + over-image (4) + under-image (5)

[0087] According to the second embodiment, the number of generated HDR composite images is one less than the number of shooting times, but all HDR composite images are generated based on a combination of the previous appropriate image, the previous inappropriate image, and the current inappropriate image. Therefore, in the case where a plurality of HDR composite images generated continuously are used as frames of a moving image, the shooting timing is equal between the frames.

[0088] On the other hand, in the first embodiment, the same number of HDR composite images as the number of shooting times are generated, but it can be understood by comparing steps S502 and S504 that the first HDR composite image and the second HDR composite image are generated at the same timing (the initial shooting and the second shooting). In this way, in the case where a plurality of HDR composite images generated continuously are used as frames of a moving image, the shooting timing is equal between frames from the second frame onwards, but a timing shift occurs for the interval between the first frame and the second frame.

[0089] Therefore, it is possible to adopt a method of selecting the shooting command according to whether the shooting command is a still image shooting command or a moving image shooting command. Figure 5A Processing in (first generation processing) or Figure 5B The processing in (second generation processing) is used as the configuration of the processing performed in step S409. By configuring the imaging apparatus 100 to execute in step S409 when a moving image shooting instruction is made Figure 5BBy performing the processing, a high-quality moving image with suppressed timing deviation can be generated. A large number of frames are usually captured for moving images, so there is no significant disadvantage if the number of frames is one less than the number of shots performed. In addition, by configuring the imaging device 100 to execute in step S409 when a still image shooting instruction is issued, Figure 5A processing, so that we can obtain Figure 5B There is one more HDR composite image.

[0090] Third embodiment

[0091] The third embodiment will describe a configuration for reducing noise by synthesizing two appropriate images when generating an HDR composite image. In this embodiment, the basic configuration of the image pickup apparatus 100 is the same as that of the first embodiment. Details different from the first embodiment will be mainly described below.

[0092] Figure 8 1 is a flowchart showing the HDR shooting process performed by the image pickup apparatus 100 according to the third embodiment. Figure 4 The processing shown is HDR shooting processing, but in the third embodiment, Figure 8 The processing shown is HDR shooting processing. Figure 8 In the implementation and Figure 4 The steps of processing the same or similar processing are given in Figure 4 Same reference numerals.

[0093] In step S807, the control unit 109 determines whether N is at least 2. In the case where N is at least 2 (i.e., in the case where there is an image captured before the current shooting), the sequence moves to step S808, and in the case where N is not at least 2 (i.e., in the case where the current shooting is the first shooting), the sequence returns to step S402.

[0094] In step S808, the image synthesis unit 105 generates a suitable image with reduced noise (a noise-reduced suitable image) by synthesizing the previous suitable image with the current suitable image. The two suitable images synthesized in step S808 have different shooting (exposure) timings. In this way, the image synthesis unit 105 performs alignment processing in the same manner as step S502 when performing the synthesis processing. After the alignment processing, the image synthesis unit 105 performs noise reduction processing by synthesizing the previous suitable image with the current suitable image. As a noise reduction processing method, any known method such as cyclic noise reduction can be used.

[0095] In step S809, the image synthesis unit 105 generates an HDR synthesized image by synthesizing the inappropriate image of this time and the inappropriate image of the previous time with the appropriate image with reduced noise generated in step S808. The image synthesis unit 105 performs alignment processing in the same manner as step S502 when performing synthesis processing. The synthesized images here are the appropriate image (and strictly speaking, the appropriate image with reduced noise), the under-image, and the over-image, so the image synthesis unit 105 selects the appropriate image according to the noise reduction. Fig. 7A As a result, an HDR composite image obtained by synthesizing three images consisting of a proper image (strictly speaking, a proper image with noise reduced), an under-image, and an over-image is generated.

[0096] In step S810 , the signal recording unit 106 stores the HDR composite image generated in step S809 in a recording medium.

[0097] In step S811, the control unit 109 determines whether to end shooting. For example, if the release button continues to be pressed, the control unit 109 determines that shooting is not to be ended, and if the release button is not pressed, the control unit 109 determines that shooting is to be ended. If the control unit 109 determines that shooting is to be ended, the process of this flowchart ends, and if the control unit 109 does not determine that shooting is to be ended, it returns to step S402.

[0098] As described above, according to the third embodiment, the camera device 100 can generate an HDR composite image with reduced noise of pixels corresponding to appropriate exposure by synthesizing an appropriate image generated in this shooting, an appropriate image generated in the previous shooting, an inappropriate image generated in this shooting, and an inappropriate image generated in the previous shooting for each shooting except the initial shooting in at least two shootings.

[0099] Fourth embodiment

[0100] Although the first to third embodiments describe a configuration in which an HDR composite image is saved, a configuration in which a proper image and an inappropriate image before synthesis are saved in addition to the HDR composite image may be adopted. Therefore, the fourth embodiment will describe a configuration in which a proper image and an inappropriate image (under-image or over-image) obtained in a single shot are saved as a single RAW image file.

[0101] Fig. 9A901. The container file format of the RAW image file described in this embodiment is an ISO-based media file format defined in ISO / IEC 14496-12. Therefore, the container format of the RAW image file has a tree structure having nodes called "boxes". In addition, each box can have multiple boxes as child elements.

[0102] The RAW image file 901 has a box ftyp 902 for describing the file type at the beginning. The RAW image file 901 also includes a box moov 903 containing all metadata, a box uuid 909 containing a display image, a box mdat 911 containing the body of the media data (image data) of the track, and other boxes 908.

[0103] The moov box 903 has, as child elements, a box uuid 904 holding metadata (MetaData) 905 and a synthesized THM image 906 for display, and a trak box 907 holding information of reference image data (ImageData).

[0104] The metadata 905 includes image metadata. The metadata 905 includes, for example, information indicating the date / time when the image was created, the conditions when the image was shot, whether the image was shot in HDR or SDR, information of the RAW image contained in the metadata 905 (proper image, under-image, or over-image), and other shooting information.

[0105] The box uuid 909 has a synthetic large THM image for display 910 as a child element. The box mdat 911 has image data 912 as a child element.

[0106] Fig. 9B 9 is a diagram showing an example of the configuration of image data 912 recorded as a still image when the image falls within the SDR dynamic range. In this case, the image data 912 includes: a composite image 920 for display compressed using JPEG in SDR development, a proper image 921 for display, an inappropriate image 922 for display, a proper image 923, an inappropriate image 924; and RAW development parameters 925. Multiple frames obtained as burst can be saved together in a single file by recording the multiple frames in a single file using the image data 912 as a single frame. Alternatively, the multiple frames can be saved as a moving image RAW file.

[0107] In the case of saving multiple frames in a single file, such as Fig. 9C As shown, the under-image and over-image may be saved in separate frames.

[0108] Fig. 9C is a diagram showing another example of the configuration of image data 912 recorded as a still image when the image falls within the SDR dynamic range. In this case, the image data 912 includes a composite image 920 for display compressed using JPEG in SDR development, an appropriate image 921 for display, an under-image 926 for display, and an over-image 927 for display. The image data 912 also includes an appropriate image 923 (first storage area), an under-image 928 (second storage area), an over-image 929 (third storage area), and RAW development parameters 925. By using the image data 912 as a single frame to save multiple frames in a single file, multiple RAW frames obtained as continuous shooting can be saved together in a single file. Alternatively, multiple RAW frames can be saved as a moving image RAW file. In the case of recording images by this method, appropriate images 921 and appropriate images 923 for display are recorded for each frame, and a pair including an under-image 926 and an under-image 928 for display, and a pair including an over-image 927 and an over-image 929 for display are recorded alternately every two frames.

[0109] The above describes the case of recording a RAW image captured in SDR. In the case of recording a RAW image captured in HDR, the image data can be developed as a display image in HDR (reference numerals 906, 910, 920, 921, 922, 926, and 927), and then the HEVC compressed display image can be recorded.

[0110] Note that the above file formats are only examples, and file formats with other boxes may be used as needed.

[0111] As described above, according to the fourth embodiment, the imaging apparatus 100 saves in a single RAW image file the appropriate image and the inappropriate image generated from a single shooting using the image sensor 102. In addition, the imaging apparatus 100 may save in a single RAW image file the images generated from two or more shootings using the image sensor 102.

[0112] Other embodiments

[0113] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0114] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A shooting control device, comprising: a photographing control unit configured to control at least two photographs to be performed using an image sensor configured to generate a first image and a second image having different exposures in a single photographing in response to a photographing instruction; as well as an exposure control unit configured to perform first exposure control and second exposure control alternately between shots in each of the at least two shots, wherein the first exposure control is exposure control that generates the first image at a first exposure and generates the second image at a second exposure lower than the first exposure, The second exposure control is exposure control for generating the first image at the first exposure and generating the second image at a third exposure higher than the first exposure.

2. The shooting control device according to claim 1, further comprising: A generating unit is configured to generate a high dynamic range composite image, i.e., an HDR composite image, by synthesizing a first image generated in this shooting, a second image generated in this shooting, and a second image generated in a previous shooting for each shooting except an initial shooting in the at least two shootings.

3. The shooting control device according to claim 2, in, The generation unit generates an HDR composite image by synthesizing a first image generated in an initial shooting, a second image generated in the initial shooting, and a second image generated in a second shooting.

4. The photographing control device according to claim 1, further comprising: A generating unit is configured to generate an HDR composite image by synthesizing a first image generated in a previous shooting, a second image generated in a previous shooting, and a second image generated in this shooting for each shooting except an initial shooting in the at least two shootings.

5. The photographing control device according to claim 1, further comprising: a generating unit configured to perform a first generating process when the shooting instruction is an instruction to shoot a still image, and to perform a second generating process when the shooting instruction is an instruction to shoot a moving image, The first generation process is a process of performing the following operations: For each shooting except the initial shooting in the at least two shootings, generating an HDR composite image by synthesizing a first image generated in the current shooting, a second image generated in the current shooting, and a second image generated in a previous shooting; and generating an HDR composite image by combining a first image generated in a first photographing, a second image generated in a first photographing, and a second image generated in a second photographing, and The second generation process is a process of performing the following operations: For each shooting except the initial shooting among the at least two shootings, an HDR composite image is generated by synthesizing the first image generated in the previous shooting, the second image generated in the previous shooting, and the second image generated in the current shooting.

6. The photographing control device according to claim 1, further comprising: A generation unit is configured to generate an HDR composite image with reduced noise from pixels corresponding to the first exposure by synthesizing a first image generated in this shooting, a first image generated in a previous shooting, a second image generated in this shooting, and a second image generated in the previous shooting for each of the at least two shootings except an initial shooting.

7. The photographing control device according to claim 6, in, The generating unit is configured to: generating an image with reduced noise by synthesizing a first image generated in the current shooting and a first image generated in the previous shooting; as well as An HDR composite image in which noise of pixels corresponding to the first exposure is reduced is generated by synthesizing the image with reduced noise, the second image generated in the current shooting, and the second image generated in the previous shooting.

8. The photographing control device according to claim 1, further comprising: A saving unit is configured to save the first image and the second image in each of the at least two shots as a single RAW image file.

9. The photographing control device according to claim 1, further comprising: A saving unit is configured to generate a RAW image file including a first storage area corresponding to the first exposure, a second storage area corresponding to the second exposure, and a third storage area corresponding to the third exposure, save each first image obtained in each of the at least two shots in the first storage area, and save each second image obtained in each of the at least two shots in the second storage area or the third storage area according to the exposure of the corresponding second image.

10. The photographing control device according to claim 1, in, The exposure control unit is configured to control, in each shooting in the at least two shootings except the initial shooting in which the second exposure control is performed, so that the ISO sensitivity used for the second image in the current shooting is consistent with the ISO sensitivity used for the first image in the previous shooting.

11. The photographing control device according to claim 1, in, The first exposure is a proper exposure, The second exposure is an underexposure, and The third exposure is overexposure.

12. The photographing control device according to claim 1, in, The exposure control unit is configured to perform the first exposure control in an initial photographing of the at least two photographings.

13. The photographing control device according to claim 1, in, The photographing control unit is configured to perform control so that at least three photographs are performed as the at least two photographs.

14. A camera device, comprising: The shooting control device according to any one of claims 1 to 13; as well as Image sensor. 15 . A computer-readable storage medium storing a computer program for causing a computer to function as the photographing control device according to claim 1 . 16 . A computer program product comprising a computer program for causing a computer to function as the photographing control device according to claim 1 .

17. A shooting control method performed by a shooting control device, comprising: controlling at least two shots to be performed using an image sensor in response to a shooting instruction, the image sensor being configured to generate a first image and a second image having different exposures in a single shot; as well as In each of the at least two shots, the first exposure control and the second exposure control are performed alternately between shots, wherein the first exposure control is exposure control that generates the first image at a first exposure and generates the second image at a second exposure lower than the first exposure, The second exposure control is exposure control for generating the first image at the first exposure and generating the second image at a third exposure higher than the first exposure.

Images