Imaging device, imaging method, computer program, and storage medium

Abstract

The present invention provides an imaging device that reduces image quality degradation before and after insertion and removal of an ND filter. [Solution] The imaging device includes an exposure control means for controlling at least two or more exposure parameters, a filter insertion / removal means for inserting and removing an ND filter on the optical axis, and a scene analysis means for analyzing the depth of the shooting range, the movement of the subject, or the importance of the scene. The exposure control means is characterized in that, when the ND filter is inserted or removed, it controls the exposure using a combination of the exposure parameters and a control amount determined based on the analysis results obtained by the scene analysis means, so as to cancel out the change in brightness of the output image associated with the insertion or removal of the ND filter.

Description

Technical Field

[0001] The present invention relates to an imaging device, an imaging method, a computer program, a storage medium, etc. suitable for applications such as monitoring.

Background Art

[0002] Conventionally, there are cameras with built-in ND (Neutral Density) filters to expand the proper exposure range. However, mounting multiple ND filters causes the camera body to become larger and more expensive, so a single high-density ND filter may be adopted. Also, there are cameras that automatically insert and remove the ND filter under certain illuminance conditions during automatic exposure control to expand the proper range of automatic exposure control.

[0003] When a high-density ND filter is inserted or removed during automatic exposure control, other exposure parameters (aperture, shutter, gain) automatically change to cancel out the luminance change associated with the insertion and removal of the ND filter. As a result, image quality such as depth of field, motion resolution, and noise level also changes significantly. Thus, using a fixed program diagram preset in the camera may have an adverse effect on the visibility of the scene.

[0004] For example, in Patent Document 1, in order to shorten the period during which the exposure varies in response to the insertion and removal of the ND filter, the number of frames with image quality degradation is reduced by discontinuously controlling the exposure simultaneously with the insertion and removal of the ND filter.

[0005] Also, when the ND filter is inserted or removed, the state of the insertion and removal of the ND filter is reflected in the moving image being captured, and the luminance change is drastic, resulting in frames with low visibility. Therefore, if the insertion and removal of the ND filter are performed based only on the illuminance condition, it may reduce the evidentiary ability of the images of important scenes in a camera for monitoring purposes.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

[0007] Furthermore, in the technique disclosed in Patent Document 1, the insertion and removal of the ND filter is performed at a predetermined illumination level, which means that when the predetermined illumination level is reached in a scene important for surveillance purposes, frames with degraded image quality will occur.

[0008] Furthermore, because the exposure parameters used to compensate for the brightness changes associated with inserting and removing ND filters are fixed, there is a problem in that changes in image quality become more pronounced.

[0009] Therefore, one of the objectives of the present invention is to provide an imaging device that can reduce image quality degradation before and after inserting or removing an ND filter. [Means for solving the problem]

[0010] To achieve the above objective, the imaging apparatus according to the present invention is Exposure control means for controlling at least two or more exposure parameters, A filter insertion / removal means for inserting and removing an ND filter on the optical axis, It includes a scene analysis means for analyzing the depth of the shooting range, the movement of the subject, or the importance of the scene, The exposure control means is characterized in that, when the ND filter is inserted or removed, it performs exposure control using a combination of exposure parameters and a control amount determined based on the analysis results obtained by the scene analysis means, so as to cancel out the change in brightness of the output image that occurs when the ND filter is inserted or removed. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide an imaging device that can reduce image quality degradation before and after inserting and removing an ND filter. [Brief explanation of the drawing]

[0012] [Figure 1] This is a functional block diagram showing an example of the configuration of an imaging device 100 according to Embodiment 1 of the present invention. [Figure 2] This figure shows an example of a program diagram that serves as a reference when inserting an ND filter according to Embodiment 1. [Figure 3] This flowchart shows an example of ND filter insertion / removal and associated exposure control according to Embodiment 1. [Figure 4] (A) and (B) are diagrams illustrating use cases for situations with a small and large depth of field in a shooting scene. [Figure 5] (A) and (B) are diagrams showing examples of program diagrams that are regenerated in step S305. [Figure 6] This figure shows an example of a program diagram that serves as a reference when the ND filter is removed according to Embodiment 1. [Figure 7] (A) and (B) are diagrams showing examples of program diagrams that are regenerated when the ND filter is removed according to Embodiment 1. [Figure 8] This flowchart shows an example of ND filter insertion / removal and associated exposure control according to Embodiment 2. [Figure 9] (A) and (B) are diagrams showing examples of program diagrams that are regenerated when an ND filter is inserted according to Embodiment 2. [Modes for carrying out the invention]

[0013] Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same reference numeral is used for the same member or element, and redundant explanations are omitted or simplified.

[0014] <Embodiment 1> FIG. 1 is a functional block diagram showing a configuration example of an imaging device 100 according to Embodiment 1 of the present invention. Note that some of the functional blocks shown in FIG. 1 are realized by causing a CPU or the like as a computer (not shown) included in the imaging device 100 to execute a computer program stored in a memory as a storage medium (not shown).

[0015] However, part or all of them may be realized by hardware. As the hardware, a dedicated circuit (ASIC), a processor (reconfigurable processor, DSP), or the like can be used. Also, each of the functional blocks shown in FIG. 1 does not have to be built in the same housing, and may be constituted by separate devices connected to each other via signal paths.

[0016] As shown in FIG. 1, the imaging device 100 includes a lens 101, an aperture 102, an ND filter 103, an imaging element 104, a gain control circuit 105, an A / D converter 106, an image signal processing unit 107, a camera control unit 108, an aperture drive unit 109, and an ND filter drive unit 110.

[0017] The lens 101 is an optical system that forms an image of light incident from a subject on the imaging element 104. The lens 101 includes a focus lens that focuses on a subject, a zoom lens that adjusts the angle of view, and the like.

[0018] The aperture 102 (aperture stop) is driven by the aperture drive unit 109 and is used to control the amount of light incident on the light receiving surface of the imaging element 104 via the lens 101.

[0019] The ND filter 103 is an achromatic Neutral Density filter for reducing the amount of incident light, and is configured to be selectively inserted and removed into the optical path of the lens 101 by the ND filter drive unit 110. Here, the ND filter drive unit 110 functions as filter insertion / removal means for inserting and removing the ND filter on the optical axis.

[0020] The image sensor 104 is, for example, a CMOS image sensor or a CCD image sensor, and converts the subject image formed by the lens 101 into an analog image signal.

[0021] The gain control circuit 105 controls the gain of the electrical signal output from the image sensor 104 and adjusts the brightness of the analog image signal.

[0022] The A / D converter 106 converts the analog image signal, which has been amplified by the gain control circuit 105, into a digital image signal.

[0023] The processing performed by the image signal processing unit 107 and the camera control unit 108 is executed by the CPU, which acts as a computer within the camera control unit 108. Specifically, the CPU within the camera control unit 108 controls the image signal processing unit 107 and the entire imaging device by executing a computer program read from a memory device (not shown).

[0024] The image signal processing unit 107 performs various processes on the digital image signal, such as color conversion, gamma correction, and digital gain addition. It also performs predetermined calculations using the digital image signal and sends the calculation results to the camera control unit 108.

[0025] The camera control unit 108 performs exposure control, focus detection, focus adjustment control, white balance control, etc., based on the calculation results supplied from the image signal processing unit 107. This enables AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, etc.

[0026] Furthermore, the camera control unit 108 controls the image sensor 104, gain control circuit 105, image signal processing unit 107, and aperture drive unit 109, and also changes the aperture value and shutter speed, and inserts and removes ND filters via the ND filter drive unit 110. In addition, the camera control unit 108 changes the exposure by adjusting the analog and digital gains.

[0027] Thus, the camera control unit 108 functions as an exposure control means that performs an exposure control step that controls at least two exposure parameters. The exposure control means controls at least two of the exposure parameters: an aperture diaphragm for controlling the amount of light incident on the light-receiving surface, shutter speed, and gain. In this embodiment, the shutter speed may be the exposure time by the mechanical shutter or the charge accumulation time by the image sensor.

[0028] The operation of the imaging device according to Embodiment 1 of the present invention will be described below with reference to Figures 2 to 7. Figure 2 is a diagram showing an example of a reference program diagram when an ND filter is inserted according to Embodiment 1.

[0029] As a premise for this explanation, the imaging device is assumed to have the ND filter removed from the optical path in its initial state, and the AE processing is being performed according to the program diagram shown in Figure 2 when the absolute brightness value (Bv) calculated from the image signal of the subject is less than the ND filter insertion brightness x.

[0030] In this embodiment, if, in this state, an instruction is given to remove the ND filter of the imaging device via an external interface (not shown) and fix it in that state, the program diagram in Figure 2 will be used. Therefore, the program diagram in Figure 2 will be used as the reference diagram from now on.

[0031] Furthermore, the program diagram (reference diagram) in Figure 2 functions as a first program diagram that pre-defines the transitions of multiple exposure parameters corresponding to the brightness of the subject.

[0032] Here, the program diagram is a table that shows the distribution of aperture Av, shutter speed Tv, and gain Sv with respect to the absolute brightness value Bv, and the relationship between each value is given by Equation 1 shown below. Note that ΔY is the exposure difference (difference between the correct exposure value and the current exposure value) calculated by the image signal processing unit 107.

[0033] Bv = Av + ​​Tv - Sv + ΔY ... (Equation 1) The camera control unit 108 controls each part according to the program diagram shown in Figure 2, and maintains the appropriate brightness of the image by reducing the exposure difference ΔY.

[0034] Figure 3 is a flowchart illustrating an example of ND filter insertion / removal and associated exposure control according to Embodiment 1. The CPU and other components within the camera control unit 108 execute computer programs stored in memory, sequentially performing each step of the flowchart in Figure 3.

[0035] Here, the memory described above functions as a computer-readable storage medium that stores the program that causes the computer to execute each step of the computer program.

[0036] Note that the flowchart in Figure 3 shows an example of a processing flow that applies to both inserting and removing an ND filter. However, we will first explain the case where an ND filter is inserted from a state where no ND filter is present.

[0037] In step S301, the camera control unit 108 estimates the depth of the shooting range. That is, it may estimate the subject distance by using the contrast evaluation value calculated by the image signal processing unit 107 for AF processing to estimate the amount of defocus of the subject, and then estimate the depth of the shooting range.

[0038] Alternatively, the range of distance to the subject may be estimated based on the results of generating a depth map from a 2D image using machine learning. Instead of contrast evaluation values, phase difference detection output from the image sensor or distance measurement values ​​obtained by a TOF (Time of Flight) sensor may be used.

[0039] Furthermore, by recognizing predetermined subjects such as people or cars from the image, the depth may be estimated not by the total distance of the shooting range, but by the distance between the front and rear of the predetermined subject. Additionally, the predetermined subject may be selected via an external interface (not shown) according to the user's focus. For example, only people, only cars, or both people and cars may be selectable as predetermined subjects.

[0040] Furthermore, step S301 functions as a scene analysis step (scene analysis means) that analyzes the depth of the shooting range. Here, the depth includes distance information between at least multiple subjects from the imaging device.

[0041] Figures 4(A) and 4(B) illustrate use cases for situations where the depth of the shooting range is small and large in a shooting scene, with a car being used as an example of the predetermined subject. As in Figure 4(A), if only one subject is present in the shooting range, the required depth of field for the shooting scene is relatively small, and even if the depth of field changes from d1 to d2 due to the change in aperture, there is no significant difference in the visibility of the subject of interest.

[0042] On the other hand, as shown in Figure 4(B), when there are multiple subjects and differences in distance from the imaging device, the required depth of field relative to the shooting range is relatively large, and when the depth of field changes from d1 to d2, some subjects will fall outside the depth of field, reducing visibility. In such cases, it becomes necessary to maintain a small aperture setting.

[0043] In step S302, the camera control unit 108 estimates the illuminance (luminance) based on the exposure difference ΔY calculated by the image signal processing unit 107 and updates the absolute luminance value Bv. Note that the order of processing in steps S301 and S302 may be reversed.

[0044] In step S303, a decision is made as to whether to insert or remove the ND filter. That is, if an ND filter is to be inserted from a state where no ND filter is initially installed, the camera control unit 108 compares the current absolute brightness value Bv with the ND filter insertion brightness x and decides whether or not to insert the ND filter.

[0045] If the absolute luminance value Bv exceeds the luminance x when the ND filter is inserted, the system determines "Yes" in step S303 and proceeds to step S304. On the other hand, if the system determines "No" in step S303, it proceeds to step S301.

[0046] In step S304, the camera control unit 108 calculates the exposure parameters, namely aperture, shutter speed, and gain, after, for example, the insertion of an ND filter.

[0047] Here, before inserting the ND filter, let Avout be the aperture, Tvout be the shutter speed, and Svout be the gain. After inserting the ND filter, let Avin be the aperture, Tvin be the shutter speed, and Svin be the gain. Furthermore, if Nvin is the change in absolute luminance value Bv caused by inserting the ND filter, then the above exposure parameters satisfy the relationship shown in Equation 2 below.

[0048] Avin+Tvin-Svin+Nvin=Avout+Tvout-Svout…(Formula 2)

[0049] Equation 2 shows that the change in absolute brightness value Bv Nvin caused by inserting an ND filter can be offset by at least one of the aperture, shutter speed, and gain. That is, for example, if inserting an ND filter makes the image 3 stops darker, the change in absolute brightness value Bv Nvin can be offset by opening the aperture 3 stops.

[0050] Alternatively, you could slow down the shutter speed by three stops. Or, you could open the aperture by one stop, slow down the shutter speed by one stop, and increase the gain by one stop. You can freely adjust the distribution of the changes in each of these parameters.

[0051] In this embodiment, in step S304, the camera control unit 108 first determines the aperture Avin after inserting the ND filter based on the depth estimated in step S301. Specifically, depending on the accuracy and likelihood of the depth estimation result obtained in step S301, the camera control unit 108 determines the aperture Avin after inserting the ND filter based on at least one of the following: subject distance, allowable scattering circle diameter, focal length, etc.

[0052] For example, if it is impossible to estimate the specific depth of the subject, you can pre-set the aperture value based on the depth, such as F16 for greater depth and F2.8 for smaller depth.

[0053] On the other hand, if it is possible to estimate the specific depth of the subject, as mentioned above, it is also possible to calculate the aperture value required to keep multiple subjects within the depth of field based on at least one of the following: subject distance, allowable scattering circle diameter, focal length, etc. In addition, the focus lens position may be adjusted based on the difference between the front depth of field and the rear depth of field.

[0054] Next, the camera control unit 108 calculates Tvin and Svin based on Equation 2 and the reference line diagram in Figure 2. Here, we will explain how to calculate Tvin and Svin when, for example, x=16, Avout=8 (F16), Tvout=8 (1 / 250s), Svout=0 (0dB), and Nvin=6 after inserting the ND filter.

[0055] If the required depth of field for the shooting scene is relatively small, and Avin = 4 (F4), then Tvin - Svin = 6 based on Equation 2.

[0056] Furthermore, in the reference diagram of Figure 2, when the absolute brightness value Bv decreases, the camera control unit 108 first decreases Tv to Tv=6 (1 / 60s), then decreases Av to Av=3 (F2.8), and finally increases Sv to Sv=3 (18dB). Therefore, following the transition order of each parameter in the reference diagram, we set Tvin=6 (1 / 60s). Also, as mentioned above, Tvin - Svin = 6, so Svin = 0 is calculated.

[0057] On the other hand, if the required depth of field for the shooting scene is relatively large, and the aperture Avin after inserting the ND filter is Avin=7 (F11), then Tvin - Svin = 3 based on Equation 2. Therefore, following the transition order of the reference diagram, we set Tvin = 6 (1 / 60s).

[0058] According to the baseline diagram in Figure 2, Av would then be 3 (F2.8), but since Avin has already been determined to be 7, we then focus on the transition of Sv and calculate the value in the range up to Sv=3 (18dB) where Tvin - Svin = 3, which gives us Svin = 3.

[0059] As described above, in this embodiment, after determining Avin, Tvin and Svin are uniquely determined according to the value of Nvin (the amount of change in the absolute luminance value Bv caused by the insertion of the ND filter) and the transition order of each parameter in the reference diagram.

[0060] In step S305, the camera control unit 108 regenerates the program diagram. This is because the Avin, Tvin, and Svin calculated in step S304 do not conform to the reference diagram in Figure 2, making continuous AE processing impossible when the absolute luminance value Bv is updated next time.

[0061] The range in which the diagram needs to be regenerated is the luminance range from (xy) to (x+Nvin). Here, xy is the absolute luminance value Bv used as the threshold for removing the ND filter again, and Nvin is the change in absolute luminance value Bv caused by inserting the ND filter, as described above.

[0062] To prevent repeated insertion and removal of the ND filter after it has been inserted beyond the ND filter insertion brightness x, in response to minute changes in the shooting environment, we define xy as the value of the ND filter. That is, once the ND filter is inserted, it is set so that it will not be removed until the brightness decreases by a predetermined amount y from the ND filter insertion brightness x. In this embodiment, we will explain using y=2.

[0063] When the absolute luminance value Bv falls below xy, the ND filter is removed, and Avout, Tvout, and Svout are recalculated, and the program diagram is regenerated. Therefore, a program diagram corresponding to an absolute luminance value Bv below xy is basically unnecessary.

[0064] However, if, after inserting the ND filter, instructions are given to fix the imaging device in the inserted state via an external interface (not shown), the reference diagram shall be the alternative diagram described later. This makes it possible to set exposure parameters corresponding to absolute luminance values ​​(Bv) below xy.

[0065] Thus, in step S305, after inserting or removing the ND filter, a second program diagram is generated based on the combination of the first program diagram (reference diagram) and the exposure parameters.

[0066] Figures 5(A) and 5(B) show examples of program diagrams regenerated in step S305. Figure 5(A) shows the program diagram regenerated when Avin=4 (F4), and Figure 5(B) shows the program diagram regenerated when Avin=7 (F11). Similar to step S304, the regeneration of the program diagram follows the transition order of the reference diagram.

[0067] That is, as shown in Figure 5(A), if we start with Avin=4(F4) when Bv=x, we first set Tv=6(1 / 60s) according to the transition order of each parameter in the baseline diagram in the range from x to (xy). However, as mentioned above, since Tvin=6(1 / 60s) is already present, no transition is necessary.

[0068] Next, if the brightness decreases by one step, we set Av=3 (F2.8), but since we can only regenerate a diagram that is one step lower than Avin=4 (F4), if the brightness decreases by another step, we continue to increase Sv. That is, since y=2, for the remaining step, we just need to generate a diagram that transitions to increase Sv up to 1 (6dB).

[0069] In Figure 5(A), conversely, the range from x to (x+Nvin) follows the transition order of each parameter when the absolute luminance value Bv increases in the reference diagram. That is, as shown in Figure 5(A), when Bv=x, Sv first becomes 0 (0dB), then transitions to increase Av=8 (F16), and thereafter transitions to increase Tv.

[0070] Specifically, first we set Sv=0 (0dB), but since Sv is already 0 (0dB), no transition is necessary. Next, as the brightness increases by 4 levels, we transition to Av=8 (F16), but since we can only regenerate transitions from Av=4 (F4) to Av=8 (F16), which is 4 levels higher, if the brightness increases further we transition to increase Tv.

[0071] However, since Nvin = 6 in Figures 5(A) and 5(B), for the remaining two stages, we just need to generate a diagram that increases Tv by two stages up to Tv = 8 (1 / 250 s). The diagram for the range beyond x + Nvin is the same as the reference diagram.

[0072] On the other hand, as shown in Figure 5(B), if we start with Avin=7(F11) at Bv=x, then in the range from x to (xy), we first set Tv=6(1 / 60s) according to the transition order of each parameter in the baseline diagram. However, since Tvin=6(1 / 60s) is already present, no transition is necessary.

[0073] Next, the graph transitions so that Av decreases towards Av=3 (F2.8), but we only need to generate a graph that shows the transition from Avin=7 (F11) down to Avin=5 (F5.6), where Av decreases by two steps.

[0074] For the range from x to (x+Nvin), the transition is initially made to decrease Sv, but since it is only possible to regenerate the transition from Svin=3 (18dB) down to Sv=0 by decreasing Sv by 3 steps, the transition is then made to increase Av.

[0075] That is, Av is increased to 8 (F16), but since only one transition can be reproduced from Avin=7 (F11), the transition is continued to increase Tv. Since Nvin=6, for the remaining two steps of brightness increase, a diagram should be generated that transitions so that Tv increases up to Tv=8 (1 / 250s). The shape of the diagram in the range beyond x+Nvin is the same as the reference diagram.

[0076] As described above, after determining Avin, Tvin, and Svin after inserting the ND filter in step S304, the diagram is regenerated according to the rules for the transition order of the control parameters of the reference diagram. At that time, the diagram can be regenerated for the range from (xy), which is the luminance after removing the ND filter, to (x+Nvin), where the absolute luminance value Bv is.

[0077] Next, in step S306, the camera control unit 108 inserts the ND filter and, at the same time, controls each part based on the second program diagram calculated in step S304 to set the exposure parameters after inserting the ND filter to Avin, Tvin, and Svin.

[0078] In other words, after inserting or removing the ND filter, in step S306, automatic exposure control is performed based on a second program diagram generated based on a combination of the first program diagram and exposure parameters.

[0079] Here, step S306 functions as a filter insertion / removal step, where the ND filter is inserted into and removed from the optical axis. The second program diagram, regenerated in step S305, is set as the program diagram to be referenced in subsequent AE processing. Then, the process returns to step S301.

[0080] In other words, in the exposure control step shown in Figure 3, when an ND filter is inserted or removed, exposure control is performed to cancel out the change in brightness of the output image associated with the insertion or removal of the ND filter, using a combination of exposure parameters and a control amount determined based on the analysis results obtained by the scene analysis means.

[0081] Specifically, in the exposure control step, the aperture diaphragm is controlled based on the depth of the subject, and at least one of the shutter speed and the gain is controlled to cancel out the change in brightness on the light-receiving surface caused by inserting or removing the ND filter.

[0082] The above example, using Figure 3, illustrates how to insert an ND filter from a state without one, and how to perform exposure control accordingly. Conversely, the following example, also using Figure 3, illustrates how to remove an ND filter from a state with one inserted, and how to perform exposure control accordingly.

[0083] In other words, with the ND filter installed, after processing steps S301 and S302, step S303 determines whether or not to remove the ND filter. The threshold value used in this case is Bv=xy, as described above.

[0084] In other words, in step S303, when determining whether to remove the ND filter, a program diagram like the one shown in Figure 6 is used as an alternative to the reference diagram, for example, as shown in Figure 5(A), where x and y are thresholds for removing the ND filter. Specifically, Figure 6 is a diagram showing an example of a program diagram that serves as a reference when removing the ND filter according to Embodiment 1.

[0085] Then, if it is determined in step S303 that the absolute luminance is below Bv=xy, the process proceeds to step S304. If it is determined to be No in step S303, the process returns to step S301.

[0086] In step S304, the exposure parameters after removing the ND filter are calculated, and in step S305, the program diagram is regenerated.

[0087] Figures 7(A) and 7(B) show examples of program diagrams that are regenerated when the ND filter is removed according to Embodiment 1.

[0088] Figure 7(A) shows an example of a program diagram regenerated when Avout=4 (F4) is started when the ND filter is removed (Bv=xy), and Figure 7(B) shows an example of a program diagram regenerated when Avout=7 (F11) is started when the ND filter is removed. Note that Nvout is the amount of change in the absolute luminance value Bv caused by the removal of the ND filter, and the relationship is Nvout = -Nvin.

[0089] Furthermore, in the program diagrams shown in Figures 5(A) and (B) after inserting the ND filter, when the absolute luminance value Bv exceeds x+Nvin and then falls below x+Nvin again, the settings may be switched to the program diagram shown in Figure 6.

[0090] This is because the regenerated diagram based on the scene depth when the ND filter is inserted is not necessarily suitable when the ND filter is removed, and the program diagram in Figure 6 can be said to be the shape that best adheres to the design philosophy of the reference diagram. In addition, the elapsed time since the last insertion or removal of the ND filter, or the depth estimation result when it falls below x+Nvin again, may be added as conditions for switching.

[0091] Subsequently, in step S306, the ND filter is removed and the exposure parameters are changed before returning to step S301.

[0092] <Embodiment 2> The operation of the imaging device according to Embodiment 2 of the present invention will now be described. In Embodiment 1, the luminance x, which is the threshold value when inserting the ND filter, and the luminance xy, which is the threshold value when removing the ND filter, are fixed values, and the ND filter is inserted or removed when the absolute luminance value Bv reaches either of these values.

[0093] However, even if the absolute luminance value Bv satisfies the conditions, it is desirable to avoid inserting or removing the ND filter at moments of high scene importance. Therefore, in this embodiment, the ND filter insertion luminance x (first luminance) and the ND filter removal luminance xy (second luminance) can be changed according to the importance of the scene. That is, the first luminance set as the ND filter insertion condition and the second luminance set as the removal condition are changed based on the scene analysis results.

[0094] The configuration of the imaging device according to Embodiment 2 is the same as that of Embodiment 1, so a description of the configuration will be omitted.

[0095] Figure 8 is a flowchart illustrating an example of ND filter insertion / removal and associated exposure control according to Embodiment 2. The CPU and other components within the camera control unit 108 execute computer programs stored in memory, sequentially performing the operations of each step in the flowchart in Figure 8.

[0096] Note that the steps with the same reference numerals as in Embodiment 1 perform the same processing, so their explanation will be omitted. Also, as in Embodiment 1, the case where an ND filter is inserted will be explained.

[0097] In step S307, before determining whether to insert or remove the ND filter, a determination is made as to whether the scene is important as surveillance footage. In this embodiment, for example, the presence or absence of movement of the subject is used as an indicator of importance as surveillance footage. Here, step S307 functions as a scene analysis step (scene analysis means) that analyzes the movement of the subject or the importance of the scene, together with the depth estimation in step S301.

[0098] The presence or absence of motion in the subject is determined by referring to the rate of change per unit time of the depth estimation result acquired in step S301, or the motion estimation result on the image by the image signal processing unit 107. That is, motion includes velocity information of the subject. Motion estimation can be performed using known techniques such as the calculation of motion vectors or optical flow. Also, similar to step S301, the area or target for detecting motion may be limited to predetermined subjects set in advance using known subject recognition techniques.

[0099] If movement of the subject is detected, i.e., if it is determined to be an important scene in step S307, the process returns to step S301 and the ND filter is not inserted or removed. If no movement of the subject is detected, i.e., if it is determined to be "No" in step S307, the process proceeds to step S303.

[0100] However, if the illuminance estimation result in step S302 determines that it is impossible to maintain proper exposure without inserting or removing an ND filter, that is, if the absolute brightness is above a predetermined threshold (x+A), the process will exceptionally proceed to step S303, even if it is a critical scene.

[0101] In step S303, the luminance x, which is the threshold for determining whether to insert an ND filter, is compared with the current absolute luminance value Bvnow. If the current absolute luminance value Bvnow exceeds the ND filter insertion luminance x, the process proceeds to step S304 and then to step S305, where the program diagram is regenerated in the range (xy)~(Bvnow+Nvin).

[0102] For example, this section explains how to regenerate the program diagram when Bvnow=18, it is not a critical scene, and an ND filter is inserted.

[0103] Figures 9(A) and 9(B) show examples of program diagrams that are regenerated when an ND filter is inserted according to Embodiment 2. Figure 9(A) shows the program diagram that is regenerated when Avout=4 (F4) is started when the ND filter is removed, and Figure 9(B) shows the program diagram that is regenerated when Avout=7 (F11) is started when the ND filter is removed.

[0104] Furthermore, the program diagram is regenerated in the same manner as in Embodiment 1, for each range of Bvnow~(xy) and Bvnow~(Bvnow+Nvin), according to the transition order of each parameter in the reference diagram of Figure 2.

[0105] Furthermore, if the exposure setting is reached to the level immediately before insertion or removal without further insertion or removal of the ND filter, automatic exposure control may be performed by referring to the first program diagram, or a third program diagram generated based on the first program diagram and the density of the ND filter.

[0106] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible in accordance with the spirit of the present invention, and these are not excluded from the scope of the present invention.

[0107] Furthermore, the present invention includes, for example, a system that realizes the functions of the above embodiment using at least one processor such as a CPU, memory, and circuitry (e.g., an ASIC). Alternatively, multiple processors may be used for distributed processing.

[0108] Furthermore, in order to realize some or all of the control in the above embodiment, a computer program that realizes the functions of the above embodiment may be supplied to the imaging device, etc., via a network or various storage media. The computer (or CPU, MPU, etc.) in the imaging device, etc., may then read and execute the program. In that case, the program and the storage medium that stores the program constitute the present invention. Furthermore, the present invention includes the following combinations.

[0109] (Configuration 1) An imaging device comprising: exposure control means for controlling at least two or more exposure parameters; filter insertion / removal means for inserting and removing an ND filter on the optical axis; and scene analysis means for analyzing the depth of the shooting range, the movement of the subject, or the importance of the scene, wherein when the ND filter is inserted or removed, the exposure control means performs exposure control to cancel out the change in brightness of the output image associated with the insertion or removal of the ND filter, using a combination of the exposure parameters and a control amount determined based on the analysis results obtained by the scene analysis means.

[0110] (Configuration 2) The imaging apparatus according to Configuration 1, characterized in that the exposure control means controls at least two of the exposure parameters: an aperture diaphragm for controlling the amount of light incident on the light-receiving surface, shutter speed, and gain.

[0111] (Configuration 3) The imaging apparatus according to Configuration 2, characterized in that the exposure control means controls the aperture diaphragm based on the depth of the subject and controls at least one of the shutter speed and the gain to cancel out the change in brightness on the light-receiving surface due to the insertion and removal of the ND filter.

[0112] (Configuration 4) The imaging device according to any one of Configurations 1 to 3, wherein the exposure control means comprises a first program diagram that predefines the transitions of a plurality of exposure parameters corresponding to the brightness of the subject, and after the insertion or removal of the ND filter, automatic exposure control is performed based on a second program diagram generated based on a combination of the first program diagram and the exposure parameters.

[0113] (Configuration 5) An imaging device according to any one of Configurations 1 to 4, characterized in that a first brightness set as the insertion condition for the ND filter and a second brightness set as the removal condition are changed based on the analysis results.

[0114] (Configuration 6) An imaging device according to any one of Configurations 1 to 5, characterized in that, after inserting or removing the ND filter, if the exposure setting is reached to the setting immediately before insertion or removal without further insertion or removal of the ND filter, automatic exposure control is performed by referring to a first program diagram, or a third program diagram generated based on the first program diagram and the density of the ND filter.

[0115] (Configuration 7) The imaging device according to any one of Configurations 1 to 6, characterized in that the depth includes distance information between at least multiple subjects.

[0116] (Configuration 8) The imaging device according to any one of Configurations 1 to 7, characterized in that the motion includes velocity information of the subject.

[0117] (Method) An imaging method comprising: an exposure control step of controlling at least two or more exposure parameters; a filter insertion / removal step of inserting and removing an ND filter on the optical axis; and a scene analysis step of analyzing the depth of the shooting range, the movement of the subject, or the importance of the scene, wherein when the ND filter is inserted or removed, the exposure control step controls the exposure so as to cancel out the change in brightness of the output image associated with the insertion or removal of the ND filter, using a combination of the exposure parameters and a control amount determined based on the analysis results obtained by the scene analysis step.

[0118] (Program) A computer program that causes a computer to execute each of the means of the imaging device described in any one of configurations 1 to 8.

[0119] (Medium) A computer-readable storage medium that stores a program that causes the computer to execute each step of the computer program described in Configuration 10. [Explanation of Symbols]

[0120] 101...Lens, 102...Aperture, 103...ND filter, 104...Image sensor, 105...Gain control circuit, 106...A / D converter, 107...Image signal processing unit, 108...Camera control unit, 109...Aperture drive unit, 110...ND filter drive unit

Claims

1. Exposure control means for controlling at least two or more exposure parameters, A filter insertion / removal means for inserting and removing an ND filter on the optical axis, It includes a scene analysis means for analyzing the depth of the shooting range, the movement of the subject, or the importance of the scene, The imaging apparatus is characterized in that, when the ND filter is inserted or removed, the exposure control means performs exposure control using a combination of exposure parameters and a control amount determined based on the analysis results obtained by the scene analysis means, so as to cancel out the change in brightness of the output image that occurs when the ND filter is inserted or removed.

2. The imaging apparatus according to claim 1, characterized in that the exposure control means controls at least two of the exposure parameters: an aperture diaphragm for controlling the amount of light incident on the light-receiving surface, shutter speed, and gain.

3. The imaging apparatus according to claim 2, characterized in that the exposure control means controls the aperture diaphragm based on the depth of the subject and controls at least one of the shutter speed and the gain to cancel out the change in brightness on the light-receiving surface due to the insertion and removal of the ND filter.

4. The imaging apparatus according to claim 1, wherein the exposure control means includes a first program diagram that predefines the transitions of a plurality of exposure parameters corresponding to the brightness of the subject, and after the insertion or removal of the ND filter, automatic exposure control is performed based on a second program diagram generated based on a combination of the first program diagram and the exposure parameters.

5. The imaging apparatus according to claim 1, characterized in that a first brightness set as the insertion condition for the ND filter and a second brightness set as the removal condition are changed based on the analysis results.

6. The imaging device according to claim 1, characterized in that, after inserting or removing the ND filter, if the exposure setting is reached to the setting immediately before insertion or removal without further insertion or removal of the ND filter, automatic exposure control is performed by referring to a first program diagram, or a third program diagram generated based on the first program diagram and the density of the ND filter.

7. The imaging device according to claim 1, characterized in that the depth includes distance information between at least a plurality of subjects.

8. The imaging device according to claim 1, characterized in that the aforementioned movement includes speed information of the subject.

9. An exposure control step that controls at least two or more exposure parameters, A filter insertion / removal step involves inserting and removing an ND filter onto the optical axis, It includes a scene analysis step that analyzes the depth of the shooting range, the movement of the subject, or the importance of the scene, The imaging method is characterized in that, when the ND filter is inserted or removed, the exposure control step performs exposure control using a combination of exposure parameters and a control amount determined based on the analysis results obtained by the scene analysis step, so as to cancel out the change in brightness of the output image that occurs when the ND filter is inserted or removed.

10. A computer program for causing a computer to execute each means of an imaging apparatus according to any one of claims 1 to 8.

11. A computer-readable storage medium storing a program that causes a computer to execute each step of the computer program described in claim 10.

Images