Imaging device, image processing device, and image processing program

Abstract

Ensure that the brightness of the displayed image is maintained even in darkness below the controlled exposure limit. [Solution] The imaging device includes an imaging unit that images a subject and outputs a first frame and a second frame following the first frame; a correction unit that corrects the brightness value of the second frame by adding a value based on the brightness value of the first frame and a value based on the brightness value of the second frame based on predetermined weight values; a determination unit that determines the correction conditions of the correction unit; and an output unit that outputs the second frame corrected by the correction unit. The value based on the brightness value of the first frame is a value obtained by correcting the brightness value of the first frame using a frame prior to the first frame by the correction unit, and the determination unit determines a first weight value to be given to the value based on the brightness value of the first frame and a second weight value to be given to the value based on the second frame.

Description

【Technical Field】 【0001】 The present invention relates to an imaging device, an image processing device, and an image processing program. 【Background Art】 【0002】 When the brightness of a captured image of a subject is insufficient as a result, there is a still camera that increases and changes the gain of an amplification circuit that amplifies at least the captured image signal (see, for example, Patent Document 1 below). However, in Patent Document 1 described above, correction using image signals of consecutive frames is not considered. 【Prior Art Documents】 【Patent Documents】 【0003】 【Patent Document 1】 Japanese Patent Application Laid-Open No. 11-41515 【Summary of the Invention】 【0004】 The imaging device of the first disclosed technology includes an imaging unit that captures a subject and outputs a first frame and a second frame after the first frame, a value based on the luminance value of the first frame, and a value based on the luminance value of the second frame are added based on a predetermined weight value to correct the luminance value of the second frame, a determination unit that determines a correction condition of the correction unit, and an output unit that outputs the second frame corrected by the correction unit. The value based on the luminance value of the first frame is a value obtained by correcting the luminance value of the first frame using a frame before the first frame by the correction unit, and the determination unit determines a first weight value given to the value based on the luminance value of the first frame and a second weight value given to the value based on the second frame. 【0005】 The imaging device of the second disclosed technology includes: an imaging unit that images a subject and outputs a first frame and a second frame following the first frame; a determination unit that determines a correction condition for the second frame using the first frame based on the brightness value of the second frame; a correction unit that corrects the brightness value of the second frame based on the brightness value of the second frame and the correction condition for the second frame using the first frame determined by the determination unit; and an output unit that outputs the second frame after correction by the correction unit. The determination unit determines a first weight value to be given to the brightness value of the corrected first frame used for correcting the second frame, and a second weight value to be given to the second frame. 【0006】 The image processing apparatus of the third disclosed technology includes: an acquisition unit that acquires a first frame and a second frame following the first frame; a correction unit that corrects the brightness value of the second frame by adding a value based on the brightness value of the first frame and a value based on the brightness value of the second frame based on predetermined weight values; a determination unit that determines the correction conditions of the correction unit; and an output unit that outputs the second frame corrected by the correction unit. The determination unit determines a first weight value to be assigned to the value based on the brightness value of the first frame and a second weight value to be assigned to the value based on the second frame. 【0007】 The image processing program of the fourth disclosed technology causes the processor to perform an acquisition process to acquire a first frame and a second frame following the first frame; a correction process to correct the brightness value of the second frame by adding a value based on the brightness value of the first frame and a value based on the brightness value of the second frame based on predetermined weight values; a determination process to determine the correction conditions for the correction process; and an output process to output the second frame corrected by the correction process, wherein in the determination process, the processor determines a first weight value to be assigned to the value based on the brightness value of the first frame and a second weight value to be assigned to the value based on the second frame. [Brief explanation of the drawing] 【0008】 [Figure 1]Figure 1 is a block diagram showing an example of the hardware configuration of the imaging device according to Example 1. [Figure 2] Figure 2 is a block diagram showing an example of the functional configuration of the imaging device and image processing device according to Example 1. [Figure 3] Figure 3 is a flowchart showing an example of the correction processing procedure by the imaging device and image processing device. [Figure 4] Figure 4 is a graph showing the relationship between ΔEV and the moving average. [Figure 5] Figure 5 is an explanatory diagram showing the first correction example using a moving average. [Figure 6] Figure 6 is an explanatory diagram showing a second correction example using a moving average. [Figure 7] Figure 7 is an explanatory diagram showing a comparative example of display frame updates between correction without using a moving average and correction with a moving average. [Figure 8] Figure 8 is an explanatory diagram showing examples of how each corrected frame shown in Figure 7 is displayed. [Figure 9] Figure 9 is an explanatory diagram showing an example of correction using alpha blending. [Figure 10] Figure 10 is a graph showing the grayscale curve. [Modes for carrying out the invention] 【0009】 For example, digital cameras have a live view video display function that displays image frames continuously output from the image sensor on the display unit, allowing the user to shoot while checking the image on the display unit. By using the live view video display function, users can shoot while checking the position of the subject and the overall composition of the photograph in real time. 【0010】 However, when the brightness of the subject decreases, the display of the live view video becomes darker, making it difficult for the user to confirm the position and composition of the subject. In conventional live view video, for example, as the brightness of the subject decreases, it was necessary to increase the ISO sensitivity of the image sensor to ensure the brightness of the display frame of the live view video. Furthermore, if the brightness of the subject decreases further when the ISO sensitivity has reached its upper limit, it was necessary to increase the exposure time (for example, up to 1 second) to ensure the brightness of the display frame of the live view video. However, when the exposure time becomes long (for example, more than 1 second), the update frequency (frame rate) of the display frame of the live view video also slows down with the exposure time, which could reduce the user's operability. 【0011】 Examples 1 to 3 described below provide an imaging device, an image processing device, and an image processing program that enable the display of live view video at a brightness level recognizable by humans, even when photographing subjects that are darker than the controlled exposure limit of live view video (hereinafter referred to as the "controlled exposure limit"). The following explanation will be given with reference to the attached drawings. [Examples] 【0012】 <Example of hardware configuration for imaging device> Figure 1 is a block diagram showing an example of the hardware configuration of an imaging device according to Embodiment 1. The imaging device 100 is a device capable of shooting video, and specifically, it is, for example, a digital camera, a digital video camera, a smartphone, a tablet, a personal computer, or a game console. In Figure 1, a digital camera is used as an example of an imaging device for explanation. 【0013】 The imaging device 100 includes a processor 101, a memory device 102, a drive unit 103, an optical system 104, an image sensor 105, an AFE (Analog Front End) 106, an LSI (Large Scale Integration) 107, an operating device 108, a sensor 109, a display device 110, a communication IF (Interface) 111, and a bus 112. The processor 101, memory device 102, drive unit 103, LSI 107, operating device 108, sensor 109, display device 110, and communication IF 111 are connected to the bus 112. The lens barrel, which is composed of at least the optical system 104, may be integrated with the imaging device 100 or may be detachable. 【0014】 The processor 101 controls the imaging device 100. The memory device 102 serves as the work area for the processor 101. The memory device 102 is a non-temporary or temporary recording medium that stores various programs and data. Examples of the memory device 102 include ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and flash memory. Multiple memory devices 102 may be mounted on the imaging device 100, and at least one of them may be detachable from the imaging device 100. 【0015】 The drive unit 103 drives and controls the optical system 104. The drive unit 103 has a drive circuit 103a and a drive source 103b. The drive circuit 103a controls the drive source 103b by instructions from the processor 101. The drive source 103b is, for example, a motor, and under the control of the drive circuit 103a, it moves the zooming lens 141b and focusing lens 141c in the optical system 104 in the optical axis direction, and controls the opening and closing of the aperture 142. 【0016】 The optical system 104 includes a plurality of lenses (lens 141a, zoom lens 141b, and focusing lens 141c) arranged in the optical axis direction and an aperture 142. The optical system 104 condenses subject light and emits it to the imaging device 105. 【0017】 The imaging device 105 receives the subject light from the optical system 104 and converts it into an electrical signal. The imaging device 105 may be, for example, a solid-state imaging device of the XY address type (e.g., a CMOS (Complementary Metal-Oxide Semiconductor) sensor) or a solid-state imaging device of the sequential scanning type (e.g., a CCD (Charge Coupled Device)). 【0018】 A plurality of light-receiving elements (pixels) are arranged in a matrix on the light-receiving surface of the imaging device 105. A plurality of types of color filters that transmit light of different color components are arranged in a predetermined color arrangement (e.g., a Bayer arrangement) for each pixel of the imaging device 105. Therefore, each pixel of the imaging device 105 outputs an analog electrical signal corresponding to each color component by color separation using the color filter. 【0019】 The AFE 106 is an analog front-end circuit that performs signal processing on the analog electrical signal from the imaging device 105. The AFE 106 sequentially executes gain adjustment of the electrical signal, analog signal processing (correlated double sampling, black level correction, etc.), A / D conversion processing, and digital signal processing (defective pixel correction, etc.) to generate RAW image data and output it to the LSI. The above-described drive unit 103, optical system 104, imaging device 105, and AFE 106 constitute the imaging unit 120. 【0020】 LSI107 is an integrated circuit that performs specific processing on RAW image data from AFE106, such as color interpolation, white balance adjustment, edge enhancement, gamma correction, and gradation conversion, as well as encoding, decoding, and compression / decompression. Specifically, LSI107 may be implemented by a PLD (Programmable Logic Device) such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array). 【0021】 The operating device 108 is used to input commands and data. Examples of the operating device 108 include various buttons including a shutter release button, switches, dials, and a touch panel. Sensors are devices that detect information, such as AF (Automatic Focus) sensors, AE (Automatic Exposure) sensors, gyro sensors, accelerometers, and temperature sensors. The display device 110 displays image data and setting screens. The display device 110 includes a rear monitor located on the back of the imaging device 100 and an electronic viewfinder. The communication IF 111 connects to external devices or a network to send and receive data. 【0022】 <Example of functional configuration of an imaging device> Figure 2 is a block diagram showing an example of the functional configuration of the imaging device and image processing device according to Example 1. Figure 3 is a flowchart showing an example of the correction processing procedure by the imaging device and image processing device. 【0023】 In Figure 2(A), the imaging device 100 includes an imaging unit 120, a determination unit 202, a correction unit 203, and an output unit 204. In Figure 2(B), the image processing device 200 includes an acquisition unit 201, a determination unit 202, a correction unit 203, and an output unit 204. Specifically, the acquisition unit 201, the determination unit 202, the correction unit 203, and the output unit 204 are realized, for example, by having the processor 101 execute a program stored in the storage device 102 shown in Figure 1. The image processing device 200 is the imaging device 100 of Figure 2(A) with the imaging unit 120 removed. When the imaging unit 120 is attached to the image processing device 200, it becomes the imaging device 100. Since the image processing device 200 has the same configuration as the imaging device 100, except that the imaging unit 120 is replaced by the acquisition unit 201, the following explanation will use the imaging device 100 as an example. 【0024】 The imaging unit 120 captures an image of the subject and outputs a series of frames that are consecutive in time. If one frame in the series is designated as the first frame, then frames that are later in time than the first frame are referred to as the second frame. The second frame may be the frame immediately following the first frame, or it may be the second frame or later. 【0025】 Furthermore, the frame includes frames output from the imaging unit 120 or the storage device 102 and acquired (hereinafter referred to as "acquired frames") and frames displayed on the display device 110 (hereinafter referred to as "displayed frames"). In the correction using the moving average described later, the first and second frames are acquired frames. 【0026】 The acquisition unit 201 acquires a series of frames (step S301). Since the image processing device does not have an imaging unit 120, instead, a series of frames acquired from an external source are stored in the storage device 102. Therefore, the acquisition unit 201 reads the series of frames stored in the storage device 102 (step S301). 【0027】 The determination unit 202 determines the correction conditions for the second frame using the first frame based on the brightness value of the second frame (step S302). Specifically, for example, the determination unit 202 determines the correction conditions based on the exposure value of the imaging device 100 when the second frame is output. Specifically, for example, the determination unit 202 determines the correction conditions when the exposure value of the imaging device 100 when the second frame is output falls below a predetermined exposure value. 【0028】 First, let's explain the predetermined exposure values ​​for live view video. The predetermined exposure value is, for example, the control exposure limit. The control exposure limit is determined by the exposure control parameters: shutter speed, aperture, and ISO sensitivity. Each parameter is limited in its possible values ​​by the performance or specifications of the imaging device 100. 【0029】 If the shutter speed range of the imaging device 100 is, for example, 30 to 1 / 8000 [sec], then the upper limit (maximum speed) of that range is 1 / 8000 [sec] and the lower limit (minimum speed) is 30 [sec]. Also, if the aperture range of the imaging device 100 is, for example, f1 to f32, then the upper limit (wide open) of that range is f1 and the lower limit (small aperture) is f32. Furthermore, if the ISO sensitivity range of the imaging device 100 is, for example, 100 to 25600, then the upper limit (maximum sensitivity) of that range is 25600 and the lower limit (minimum sensitivity) is 100. 【0030】 The controlled exposure limit for live view video is, for example, the exposure value when the shutter speed is at its fastest, the aperture is wide open, and the ISO sensitivity is at its highest. However, the controlled exposure limit is not limited to this exposure value. 【0031】 For example, the controllable exposure limit shutter speed can be 1 sec or 1 / 2 sec, rather than 30 sec (the fastest), within the controllable range of the imaging device 100, for example, 30 to 1 / 8000 sec. In live view video, the fastest speed is not used as in still image shooting, and the limit is generally set to a higher speed than the controllable range for still images. 【0032】 Furthermore, for example, the aperture of the controllable exposure limit may be f1 (wide open) or f1.4, within the range controllable by the imaging device 100, for example, f1 to f32. Similarly, for example, the ISO sensitivity of the controllable exposure limit may be 25600 (maximum sensitivity) or 12800, within the range controllable by the imaging device 100, for example, 100 to 25600. In this way, a predetermined controllable exposure limit value is determined. 【0033】 Furthermore, the correction conditions are the conditions under which the correction unit 203 corrects the second frame using the first frame. The correction conditions vary depending on the type of correction. Details of the correction conditions will be described later. 【0034】 The correction unit 203 corrects the second frame using a moving average based on the brightness value of the second frame and the correction conditions for the second frame using the first frame determined by the determination unit 202 (step S303). Correction using a moving average is a correction process that corrects the second acquired frame by a moving average of one or more first acquired frames and second acquired frames (described later in Figures 4 to 8). 【0035】 The output unit 204 outputs the second frame after correction by the correction unit 203 (step S304). The output destination of the output unit 204 may be, for example, the display device 110, or a computer connected via the communication IF 111 for communication. When output to the display device 110, the sequentially generated second frames are displayed as a live view video. 【0036】 <Correction using moving average> Figure 4 is graph 400 showing the relationship between ΔEV and the moving average. The vertical axis of the graph is the target EV, and the horizontal axis is the control EV. The characteristics of ΔEV and the moving average can be divided into, for example, a region 401 where ΔEV ≥ 0 and a region 402 where ΔEV > ΔEV. In this embodiment, the boundary values ​​that divide each region of ΔEV (for example, ΔEV=0, ΔEV=-1, ΔEV=-1.585, ΔEV=-2, etc.) are assumed to be included in the upper region of ΔEV. However, the boundary values ​​are not limited to these. ΔEV is the difference value obtained by subtracting the control EV from the target EV, and is calculated by the following equation (1). 【0037】 ΔEV = Target EV - Control EV ... (1) 【0038】 The control EV on the right-hand side of equation (1) is calculated by equation (2) below, and the target EV is calculated by equation (3) below. 【0039】 Control EV = AV + TV ... (2) Target EV=Target BV+SV...(3) 【0040】 In equation (2), AV is, for example, a value within the range up to the upper limit of the aperture that determines the controlled exposure limit as described above, and TV is, for example, a value within the range up to the lower limit of the shutter speed that determines the controlled exposure limit as described above. Target BV is the metering value of the subject or the exposure amount of the still image determined by the photographer, and SV is, for example, a value within the range up to the upper limit of the ISO sensitivity that determines the controlled exposure limit as described above. 【0041】 In region 401 where ΔEV is ΔEV≧0, frames captured with proper exposure are obtained, so the imaging device 100 outputs acquired frames without performing correction by moving average or digital gain. 【0042】 In region 402 where ΔEV is 0 > ΔEV, the acquired frame is captured at a level below the control exposure limit, resulting in insufficient brightness. The correction unit 203 amplifies the brightness of the acquired frame using a moving average value M(n) corresponding to ΔEV and a digital gain G. 【0043】 Furthermore, G is not limited to the brightness at which ΔEV ≥ 0 in region 402 where ΔEV > ΔEV, but may also be an amplification amount that brings the brightness to an arbitrarily defined lower limit of ΔEV (for example, ΔEV = -0.5). In other words, the magnitude of G should be an amplification amount that can make dark and hard-to-see subjects visible. 【0044】 The correction unit 203 corrects the second acquired frame according to the correction condition 403. The correction condition is the total number of frames n, which is the sum of the first acquired frame and the second acquired frame (1 frame). The number of first acquired frames changes according to ΔEV. The number of first acquired frames used for the moving average is (n-1), which is obtained by subtracting the number of second acquired frames (1 frame) from n. 【0045】 When ΔEV is greater than or equal to the control exposure limit, such as in region 401 where ΔEV≧0, the number of acquisition frames n for taking the moving average is n=1. Therefore, the determination unit 202 determines the number of first acquisition frames to be 0 as a correction condition, the correction unit 203 does not correct the second acquisition frames because the number of first acquisition frames is 0, and the output unit 204 outputs the acquired frames as display frames. 【0046】 In region 402 where ΔEV is 0 > ΔEV, ΔEV falls below the control exposure limit. Therefore, the number of acquisition frames n for taking the moving average increases as ΔEV decreases. 【0047】 If ΔEV is in region 402a where 0 > ΔEV ≥ -1, the determination unit 202 determines the number of first acquired frames to be 1 as a correction condition, and the correction unit 203, since the number of first acquired frames is 1, sets the number of acquired frames for which the moving average is taken to n=2 and performs correction using the moving average. If the moving average value M at this time is M(2), then the brightness M × G of the corrected display frame of the second acquired frame will be M(2) × G(ΔEV). 【0048】 If ΔEV is in region 402b where -1 > ΔEV ≥ -1.585, the determination unit 202 determines the number of first acquired frames to be 2 as a correction condition, and the correction unit 203, since the number of first acquired frames is 2, sets the number of acquired frames for which the moving average is taken to n=3 and performs correction using the moving average. If the moving average value M at this time is M(3), then the brightness M × G of the corrected display frame of the second acquired frame will be M(3) × G(ΔEV). 【0049】 If ΔEV is in region 402c where -1.585 > ΔEV ≥ -2, the determination unit 202 determines the number of first acquired frames to be 3 as a correction condition, and the correction unit 203, since the number of first acquired frames is 3, sets the number of acquired frames n for taking the moving average to n=4 and performs correction using the moving average. If the moving average value M at this time is M(4), then the brightness M × G of the corrected display frame of the second acquired frame will be M(4) × G(ΔEV). 【0050】 Note that since the moving average values ​​M(2) to M(4) are average values, the brightness of the corrected frame is approximately the same as that of a single acquired frame. 【0051】 Figure 5 is an explanatory diagram showing the first correction example using a moving average. Figure 5(A) shows the change in ΔEV over time. Figure 5(B) shows the moving average of the acquired frame and the displayed frame over time. Figure 5(C) shows the setting value of the digital gain G over time. Figure 5(D) shows the change in brightness of the displayed frame over time. As an example, let time t0 be the time when ΔEV falls below the control exposure limit. The time before time t0 is the time period when ΔEV=0, and the time after time t0 is the time period when ΔEV falls below the control exposure limit and becomes ΔEV=-2. 【0052】 In Figure 5 (and subsequent figures), {(Fx~Fy) / n} (where x and y are frame numbers) represents the frame after moving average, calculated by dividing the sum of pixels at the same position in n frames from frame Fx to Fy by n. 【0053】 In Figure 5(A), the ΔEV before time t0 is ΔEV≧0. In this case, the acquired frames F1 and F2 are captured with proper exposure, so the display frames F1 and F2 before time t0 are displayed as acquired frames F1 and F2 (Figure 5(B)). As shown in Figure 5(C), there is no digital gain amplification correction before time t0, and as shown in Figure 5(D), the display frames F1 and F2 are displayed with proper brightness. 【0054】 In Figure 5(A), the ΔEV immediately after time t0 is ΔEV = -2, but the acquired frame F3 immediately before time t0 is captured with proper exposure, so the acquired frame F3 is displayed as is as the display frame F3 immediately after time t0 (Figure 5(B)). For this reason, as shown in Figure 5(C), there is no amplification correction of the digital gain, and as shown in Figure 5(D), the display frame F3 is displayed with proper brightness. 【0055】 In Figure 5(A), the ΔEV of the acquired frame F4 immediately after time t0 is ΔEV = -2. The correction unit 203 corrects the acquired frame F4 according to the correction condition 403 in Figure 4 and outputs the display frame {(F1~F4) / 4}. In this example, since ΔEV = -2, the number of frames n for which the moving average is taken according to the correction condition 403 is n = 4. Accordingly, the display frame {(F1~F4) / 4} is the moving average of the four most recent acquired frames F1~F4 (Figure 5(B)). 【0056】 The correction unit 203 determines the digital gain G(ΔEV) based on the ΔEV at the time of acquisition of acquisition frame F4 before the generation of display frame {(F1~F4) / 4} (Figure 5(C)). In Example 1, since the number of acquisition frames n for taking the moving average was set to n=4, the acquisition frame that determines the ΔEV for determining the digital gain G(ΔEV) is, for example, one of the most recent acquisition frames F1~F4 before the generation of display frame {(F1~F4) / 4}. Since it is preferable to have a small time lag between the acquisition of an acquisition frame and the display of display frame {(F1~F4) / 4}, the most recent acquisition frame is preferably F4. By determining the digital gain G(ΔEV) based on ΔEV in this way, the brightness of the display frame {(F1~F4) / 4} becomes brighter initially and then gradually reaches the appropriate brightness (Figure 5(D)). The same applies to subsequent display frames {(F2~F5) / 4}, {(F3~F6) / 4}, ... 【0057】 Figure 6 is an explanatory diagram showing a second correction example using a moving average. Figures (A) to (D) in Figure 6 show changes over time similar to Figures (A) to (D) in Figure 5. 【0058】 In Figure 6(A), the ΔEV before time t0 is ΔEV≧0. In this case, the acquired frames F1 and F2 are captured with proper exposure, so the display frames F1 and F2 before time t0 are displayed as acquired frames F1 and F2 (Figure 6(B)). As shown in Figure 6(C), before time t0, there is no amplification correction of the digital gain, and as shown in Figure 6(D), the display frames F1 and F2 are displayed with proper brightness. 【0059】 In Figure 6(A), the ΔEV immediately after time t0 is ΔEV = -2, but the acquired frame F3 immediately before time t0 is captured with proper exposure, so the acquired frame F3 is displayed as is as the display frame F3 immediately after time t0 (Figure 6(B)). For this reason, as shown in Figure 6(C), there is no amplification correction of the digital gain, and as shown in Figure 6(D), the display frame F3 is displayed with proper brightness. 【0060】 In Figure 6(A), the ΔEV of the acquired frame F4 immediately after time t0 is ΔEV = -2. The correction unit 203 corrects the acquired frame F4 according to the correction condition 403 in Figure 4 and outputs the display frame {(F1~F4) / 4}. In this example, since ΔEV = -2, the number of frames n for which the moving average is taken according to the correction condition 403 is n = 4. Accordingly, the display frame {(F1~F4) / 4} is the moving average of the four most recent acquired frames F1~F4 (Figure 6(B)). 【0061】 The correction unit 203 determines the digital gain based on the statistical values ​​of each ΔEV at the time of acquisition of the four acquisition frames F1 to F4 immediately preceding the moving average (for example, the mean, median (if n is even, further averaged or the larger value is selected), minimum, maximum, or other statistical values). This gradually increases the amplification factor (Figure 6(C)). Therefore, the brightness of the display frame {(F1~F4) / 4} is always maintained at the appropriate brightness (Figure 6(D)). The same applies to subsequent display frames {(F2~F5) / 4}, {(F3~F6) / 4}, ... 【0062】 Figure 7 is an explanatory diagram showing a comparison of display frame updates between correction without using a moving average and correction with a moving average. Figure 8 is an explanatory diagram showing display examples of each corrected frame shown in Figure 7. Figure 8 shows a pendulum swinging from left to right. Figure 8(A) is a specific display image example of display frames F1 to F6 in Figure 7(A), Figure 8(B) is a specific display image example of display frames f1 to f3 in Figure 7(B), and Figure 8(C) is a specific display image example of display frames (F1+F2) / 2 to (F5+F6) / 2 in Figure 7(C). 【0063】 Figure 7(A) shows an example of display frame updates when the frame rate is N [fps]. In Figure 7(A), frames F1, F2, ... are acquired sequentially for each exposure with an exposure time of 1 / N [sec], and when frames F2, F3, ... are acquired at the timing of the next exposure, frames F1, F2, ... are displayed sequentially as shown in Figure 8(A). 【0064】 Figure 7(B) shows an example of display frame updates when the frame rate is 0.5N [fps]. For each exposure with an exposure time of 2 / N [sec], frames f1, f2, ... are acquired sequentially, and when frames f2, f3, ... are acquired at the timing of the next exposure, frames f1, f2, ... are displayed sequentially as shown in Figure 8(B). Thus, because the frame rate is lower than in Figure 7(A), the shutter speed becomes slower (exposure time becomes longer), and the display screen can be made brighter, but the frequency of display screen updates decreases compared to Figure 7(A). 【0065】 Figure 7(C) shows an example of display frame updates when correction using a moving average is performed, with a frame rate of N [fps] and the number of acquired frames n for moving average calculation being n=2. Similar to Figure 7(A), frames F1, F2, ... are acquired sequentially for each exposure with an exposure time of 1 / N [sec]. After acquiring two consecutive frames, when frames F3, F4, ... are acquired at the timing of the next exposure, the corrected frames {(F1+F2) / 2}, {(F2+F3) / 2}, ... are displayed sequentially, as shown in Figure 8(C), using the moving average. 【0066】 In Figure 7(C), the correction unit 203 applies digital gain to the frames {(F1+F2) / 2}, {(F2+F3) / 2}, ... after moving average, while maintaining the same update frequency as in Figure 7(A), thereby ensuring the brightness of the display screen is equivalent to the exposure time (2 / N) in Figure 7(B). 【0067】 Thus, according to Example 1, the brightness of the displayed image can be ensured even when the darkness falls below the control exposure limit. Specifically, for example, the imaging device 100 limits the update frequency of the display screen (i.e., exposure time) to a certain extent, for example, to a maximum of 1 / 4 [sec], in order to prevent a decrease in user operability. Instead, when the darkness falls below the control exposure limit, the imaging device 100 ensures brightness by applying a digital gain to the moving average value of the acquired frames. Furthermore, simply applying a digital gain to each acquired frame would amplify noise as well, but by applying a digital gain to the moving average value of the acquired frames, the effect of reducing noise can be obtained. In this way, by applying a digital gain to the frame after the moving average, the insufficient brightness is increased, and the noise caused by applying the digital gain is suppressed. 【0068】 As a result, even when photographing subjects that are darker than the control exposure limit, the imaging device 100 can display live view video at a brightness level that is recognizable to humans. This ensures that the brightness of the live view video display frame is maintained even when photographing the night sky or animals in a pitch-black jungle, preventing a decrease in operability. [Examples] 【0069】 Next, Example 2 will be described. In Example 1, the correction unit 203 performed correction using a moving average, but in Example 2, the correction unit 203 performs correction using α blend instead of correction using a moving average. Also, the amplification correction using the digital gain G of the correction unit 203 is the same as in Example 1, so a detailed explanation will be omitted in Example 2. In Example 2, only the correction content of the correction unit 203 is different, so the configuration in Figures 1 to 3 is the same in Example 2 as well. Also, in correction using α blend, the first frame is the display frame and the second frame is the acquisition frame. Note that in Example 2, the explanation will focus on correction using α blend, so the same reference numerals will be used for components identical to those in Example 1, and their explanations will be omitted. 【0070】 <Correction using alpha blending> Figure 9 is an explanatory diagram showing an example of correction using alpha blending. In Figure 9, the YUV pixels of the latest acquisition frame N (where Y is the luminance signal, U is the chrominance signal obtained by subtracting the luminance component Y from the blue component, and V is the chrominance signal obtained by subtracting the luminance component Y from the red component) are Ny, Nu, and Nv, the YUV pixels of the intermediate frame S, which is the acquired frame corrected, are Sy, Su, and Sv, and the YUV pixels of the display frame, which is displayed by applying a digital gain G to the intermediate frame S, are Dy, Du, and Dv. Alpha blending is a process that superimposes two image data and combines them based on a transparency α (0 ≤ α ≤ 1) set for each frame. 【0071】 The two image data sets referred to here are the acquired frame (second frame) and the display frame (first frame) output based on the acquired frame immediately preceding the first acquired frame. In Figure 9, the exposure time is set to 1 / 4 [sec]. 【0072】 For example, let time t0 be the time when ΔEV reached the control exposure limit. The period before time t0 is defined as the time when ΔEV was brighter than the control exposure limit, and the period after time t0 is defined as the time when ΔEV fell below the control exposure limit. 【0073】 Let frame N0 be the frame acquired at time t0. At acquisition time t0 of frame N0, ΔEV is not below the control exposure limit, and prior to acquisition time t0, ΔEV was brighter than the control exposure limit, so there is no display frame S to use for correction. Therefore, the output unit 204 outputs the acquired frame N0 as the intermediate frame S0. That is, as shown in equation (4) of Table 900, (Ny(0),Nu(0),Nv(0))=(Sy(0),Su(0),Sv(0)). 【0074】 【number】 【0075】 The correction unit 203 then amplifies the intermediate frame S0 using a digital gain G (ΔEV), and the output unit 204 outputs the amplified display frame D0. As a result, the display frame D0 is displayed on the display device 110. 【0076】 Frame N1 is the acquired frame at time t1, after the exposure time has elapsed from time t0. The ΔEV at acquisition time t1 of frame N1 is below the control exposure limit. Therefore, the determination unit 202 determines that the first frame to be used for correction is the intermediate frame S0 from the previous time t0. Then, the correction unit 203 generates the intermediate frame S1 by α blending the intermediate frame S0 and the acquired frame N1. Specifically, for example, the correction unit 203 generates the intermediate frame S1 using equation (5) (assuming k=1) in Table 900. 【0077】 【number】 【0078】 The correction unit 203 then amplifies the intermediate frame S1 using a digital gain G (ΔEV) to generate a display frame D1, and the output unit 204 outputs the amplified display frame D1. As a result, the display frame D1 is displayed on the display device 110. 【0079】 Frame N2 is the frame acquired at time t2, after the exposure time has elapsed from time t1. The ΔEV at acquisition time t2 of frame N2 is below the control exposure limit. Therefore, the determination unit 202 determines that the first frame to be used for correction is the intermediate frame S1 from the previous time t1. Then, the correction unit 203 generates an intermediate frame S2 by α blending the intermediate frame S1 and the acquired frame N2. Specifically, for example, the correction unit 203 generates the intermediate frame S2 using equation (5) (assuming k=2) in Table 900. 【0080】 The correction unit 203 then amplifies the intermediate frame S2 using a digital gain G (ΔEV) to generate a display frame D2, and the output unit 204 outputs the amplified display frame D2. As a result, the display frame D2 is displayed on the display device 110. 【0081】 Frame N3 is the frame acquired at time t3, after the exposure time has elapsed from time t2. The ΔEV at acquisition time t3 of frame N3 is below the control exposure limit. Therefore, the determination unit 202 determines that the first frame to be used for correction is the intermediate frame S2 from the previous time t2. Then, the correction unit 203 generates the intermediate frame S3 by α blending the intermediate frame S2 and the acquired frame N3. Specifically, for example, the correction unit 203 generates the intermediate frame S3 using equation (5) (assuming k=3) in Table 900. 【0082】 The correction unit 203 then amplifies the intermediate frame S3 using a digital gain G (ΔEV) to generate a display frame D3, and the output unit 204 outputs the amplified display frame D3. As a result, the display frame D3 is displayed on the display device 110. 【0083】 The value of α is set to be proportional to the ΔEV at the acquisition time tk of the second frame, acquisition frame Nk. In other words, as ΔEV increases, the value of α increases, and as the value of ΔEV decreases, the value of α decreases. 【0084】 While correction using a moving average is applied based on the number of acquired frames n used for the moving average, correction using alpha blending allows for stepless adjustment of "how many frames the moving average is equivalent to" simply by changing the value of alpha, rather than being discrete like the number of frames n. Furthermore, since the first frame required for correction is only the display frame that was added one frame earlier, the calculation is simpler compared to correction using a moving average, and the processing load can be reduced. [Examples] 【0085】 Example 3 will now be described. Example 3 is an example in which, in addition to correction using a moving average and correction using α blending as in Examples 1 and 2, the correction unit 203 corrects according to a grayscale curve. Specifically, for example, two types of grayscale curves, which will be described later, are used to switch between the two curves when outputting the display frame based on ΔEV. In Example 3, the explanation will focus on correction based on the grayscale curve, so the same reference numerals are used for components identical to those in Examples 1 and 2, and their explanations will be omitted. 【0086】 Figure 10 is a graph showing the grayscale curve. The horizontal axis of graph 1000 represents the brightness obtained from the image input signal from the image sensor. The vertical axis represents the brightness output to the display device 110 (for example, a standard display device such as an sRGB monitor). The dotted line is the normal grayscale curve 1001, and the solid line is the grayscale curve 1002 of Example 3. Both grayscale curves 1001 and 1002 show the grayscale characteristics of the display. Grayscale curves 1001 and 1002 are stored in the storage device 102. 【0087】 In the imaging device 100 of Embodiment 3, in order to improve the visibility of dark areas, the gradation curve 1002 is set so that the brightness of the display output value corresponding to the sensor input value of the midtone or lower from the imaging unit 120 becomes brighter than the gradation curve 1001, in order to display the dark areas of subjects with midtones or lower brighter. 【0088】 Specifically, for example, if ΔEV is not below a preset brightness level that is brighter than the control exposure limit, the imaging device 100 outputs a display frame as a display output using the gradation curve 1001 without performing correction by moving average or α blending. On the other hand, if ΔEV is below the aforementioned brightness level, the imaging device 100 outputs a display frame as a display output using the gradation curve 1002, regardless of whether the correction unit 203 performs correction by moving average or α blending. 【0089】 Thus, according to Example 3, if the sensor input value from the imaging unit 120 is below the midtone level, the display output can be made brighter when ΔEV is below a certain brightness level compared to when ΔEV is not below that brightness level. 【0090】 It should be noted that the present invention is not limited to the above, and may be combined in any way. Furthermore, other embodiments that can be conceivable within the scope of the technical idea of ​​the present invention are also included in the scope of the present invention. [Explanation of Symbols] 【0091】 100 Imaging device, 101 Processor, 102 Storage device, 110 Display device, 120 Imaging unit, 201 Acquisition unit, 202 Determination unit, 203 Correction unit, 204 Output unit

Claims

[Claim 1] An imaging unit that captures an image of a subject and outputs a first frame and a second frame that follows the first frame, A correction unit that corrects the luminance value of the second frame by adding a value based on the luminance value of the first frame and a value based on the luminance value of the second frame based on a predetermined weight value, A determination unit for determining the correction conditions of the correction unit, It has an output unit that outputs the second frame corrected by the correction unit, The value based on the luminance value of the first frame is a value obtained by correcting the luminance value of the first frame using frames prior to the first frame by the correction unit. The determination unit determines a first weight value to be assigned to the value based on the luminance value of the first frame, and a second weight value to be assigned to the value based on the second frame. Imaging device. [Claim 2] The determination unit determines the first weight value and the second weight value based on the brightness value of the second frame. The imaging apparatus according to claim 1. [Claim 3] The determination unit determines the first weight value and the second weight value such that the second weight value decreases as the brightness value of the second frame decreases. The imaging apparatus according to claim 1 or 2. [Claim 4] The determination unit determines the first weight value and the second weight value based on the photometric values ​​of the subject. The imaging apparatus according to claim 1. [Claim 5] The correction unit corrects the luminance value of the second frame by amplifying the value obtained by adding the value based on the luminance value of the first frame and the value based on the luminance value of the second frame, based on the first weight value and the second weight value, with a predetermined gain. The imaging apparatus according to any one of claims 1 to 4. [Claim 6] The determination unit determines the predetermined gain based on the brightness value of the second frame. The imaging apparatus according to claim 5. [Claim 7] The determination unit determines the predetermined gain based on the photometric value of the subject. The imaging apparatus according to claim 5. [Claim 8] The correction unit corrects the brightness value of the second frame when the exposure value of the imaging device, according to the photometric value of the subject, falls below a predetermined exposure value. The imaging apparatus according to any one of claims 1 to 7. [Claim 9] The second weight value is α (0 ≤ α ≤ 1), and the first weight value is (1 - α). The imaging apparatus according to any one of claims 1 to 8. [Claim 10] The value based on the luminance value of the first frame is a value obtained by the correction unit by adding a value based on the luminance value of a frame prior to the first frame and a value based on the luminance value of the first frame, based on a predetermined weight value. The imaging device according to any one of claims 1 to 9. [Claim 11] The first frame and the second frame are frames for live view video. The imaging apparatus according to any one of claims 1 to 10. [Claim 12] The system includes a storage unit that stores first grayscale information and second grayscale information that define the output value from the output unit corresponding to the input value from the imaging unit, The output value of the second grayscale information is higher than the output value of the first grayscale information for input values ​​smaller than a predetermined input value. Based on the exposure value of the imaging device corresponding to the photometric value of the subject, either the first grayscale information or the second grayscale information is selected and used. The imaging apparatus according to any one of claims 1 to 11. [Claim 13] When the exposure value of the imaging device corresponding to the photometric value of the subject falls below the predetermined exposure value, the second grayscale information is used. The imaging apparatus according to claim 12. [Claim 14] An acquisition unit that acquires a first frame and a second frame that follows the first frame, A correction unit that corrects the luminance value of the second frame by adding a value based on the luminance value of the first frame and a value based on the luminance value of the second frame based on a predetermined weight value, A determination unit for determining the correction conditions of the correction unit, It has an output unit that outputs the second frame corrected by the correction unit, The determination unit is an image processing apparatus that determines a first weight value to be assigned to the value based on the brightness value of the first frame, and a second weight value to be assigned to the value based on the second frame. [Claim 15] In the processor, A process to acquire the first frame and the second frame that follows the first frame, A correction process is performed to correct the luminance value of the second frame by adding a value based on the luminance value of the first frame and a value based on the luminance value of the second frame based on a predetermined weight value, A determination process for determining the correction conditions of the correction process, The system executes an output process that outputs the second frame corrected by the correction process, In the determination process, a first weight value to be assigned to the value based on the brightness value of the first frame and a second weight value to be assigned to the value based on the second frame are determined. Image processing program. [Claim 16] An imaging unit that captures an image of a subject and outputs a first frame and a second frame that follows the first frame, A determination unit that determines the correction conditions for the second frame using the first frame based on the brightness value of the second frame, A correction unit corrects the brightness value of the second frame based on the brightness value of the second frame and the correction conditions for the second frame using the first frame determined by the determination unit, It has an output unit that outputs a second frame after correction by the correction unit, The determination unit determines a first weight value to be assigned to the luminance value of the corrected first frame used for correcting the second frame, and a second weight value to be assigned to the second frame. Imaging device. [Claim 17] An acquisition unit that acquires a first frame and a second frame that follows the first frame, A determination unit that determines the correction conditions for the second frame using the first frame based on the brightness value of the second frame, A correction unit corrects the brightness value of the second frame based on the brightness value of the second frame and the correction conditions for the second frame using the first frame determined by the determination unit, It has an output unit that outputs a second frame after correction by the correction unit, The determination unit determines a first weight value to be assigned to the luminance value of the corrected first frame used for correcting the second frame, and a second weight value to be assigned to the second frame. Image processing device. [Claim 18] In the processor, A process to acquire the first frame and the second frame that follows the first frame, A determination process that determines the correction conditions for the second frame using the first frame based on the brightness value of the second frame, A correction process that corrects the brightness value of the second frame based on the brightness value of the second frame and the correction conditions for the second frame using the first frame determined by the determination process, The system then performs an output process that outputs the second frame after correction by the correction process described above. In the aforementioned determination process, a first weight value to be assigned to the luminance value of the corrected first frame used for correcting the second frame, and a second weight value to be assigned to the second frame are determined. Image processing program.

Images