Control device, method, and program for imaging device
The control device for imaging devices estimates defocus variation and adjusts settings like shutter speed and ISO sensitivity to maintain focus on subjects in low-contrast or low-light conditions, addressing focus detection issues and improving image clarity.
Patent Information
- Application Number
- JP2025078099
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-08
- Publication Date
- 2026-11-19
AI Technical Summary
Existing imaging technologies struggle to maintain focus on a main subject effectively when contrast is low or in low-light conditions due to variations in focus detection results caused by noise, leading to difficulty in appropriate imaging control.
A control device for an imaging device that estimates the range of defocus variation using image data and a defocus map, determines setting values such as shutter speed and ISO sensitivity based on this estimation, and adjusts imaging settings to minimize defocus variation, utilizing machine learning models for accurate focus control.
Enables appropriate imaging control by reducing the influence of noise and motion blur, resulting in sharper images by adjusting settings like shutter speed and ISO sensitivity to match the subject's conditions and environment.
Smart Images

Figure 2026182137000001_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to a control apparatus, method, and program for an imaging device. BACKGROUND ART
[0002] When photographing a main subject detected by an imaging device, it is desirable to keep focusing on the main subject without being affected by other subjects, the background, or focus detection errors, even if the movement of the main subject is large. Patent Document 1 discloses a method of adjusting focus on the main subject using focus detection results from a plurality of regions as a method of continuously focusing on the main subject. PRIOR ART DOCUMENTS PATENT DOCUMENTS
[0003] PATENT DOCUMENT 1 Japanese Unexamined Patent Publication No. 2022-137760 SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION
[0004] However, when the contrast of the subject is low, or when the shooting environment is dark such as at night and the ISO sensitivity is set high, variations in focus detection results may occur due to the influence of noise. Originally, it is necessary to correctly adjust the focus on the main subject while taking into account the variations in the focus detection results, but the technique of Patent Document 1 cannot acquire variation information of the focus detection results, making appropriate imaging control difficult when variations in focus detection results occur. MEANS FOR SOLVING THE PROBLEMS
[0005] The control device for an imaging device according to the present disclosure is a control device for an imaging device equipped with imaging means for imaging a subject, and is characterized by comprising: estimation means for estimating the range of variation of the defocus range of the subject based on image data and a defocus map acquired from the imaging means; determination means for determining a set value for the imaging means based on the range of variation of the defocus range estimated by the estimation means; and control means for controlling the setting of the imaging means based on the set value set by the determination means. [Effects of the Invention]
[0006] According to this disclosure, appropriate imaging control becomes possible when variations occur in focus detection results. [Brief explanation of the drawing]
[0007] [Figure 1] This is a block diagram showing an example of the hardware configuration of a digital camera according to the first embodiment. [Figure 2] This diagram illustrates the amount of defocusing. [Figure 3] This diagram shows the functional configuration of a digital camera according to the first embodiment. [Figure 4] This diagram illustrates the defocus range and the range of variation in the defocus range. [Figure 5] This is a flowchart showing the process performed by the digital camera according to the first embodiment. [Figure 6] This diagram shows the functional configuration of an information processing device for training machine learning models. [Figure 7] This flowchart shows the process for determining the variation range of the correct defocus range. [Figure 8] This diagram illustrates the determination of the range of variation in the correct defocus range. [Figure 9] This is a flowchart showing the process performed by the digital camera according to the second embodiment. [Figure 10] This diagram illustrates the aperture value and focus position control for the maximum range of defocus. [Figure 11] This diagram illustrates the aperture value and focus position control for the closest focusing point. [Modes for carrying out the invention]
[0008] Preferred embodiments of the present invention will be described below with reference to the attached drawings. <First Embodiment> Figure 1 is a block diagram showing an example of the hardware configuration of a digital camera (hereinafter simply referred to as "camera") 10 according to the first embodiment. In this embodiment, the camera 10 is an interchangeable-lens digital camera and comprises a camera body 100 and a lens unit 150 that can be attached to or removed from the camera body 100.
[0009] Let me explain the camera body 100. The camera body 100 includes an image sensor 101, a system control unit 102, a shutter 103, a memory 104, a power switch 105, a mode switching unit 106, and an AE (Auto Exposure) unit 107. The camera body 100 also includes a rear monitor 108, a touch panel 109, a viewfinder display unit 110, an eyepiece lens 111, an eyepiece detection unit 112, and a shutter control unit 114.
[0010] The image sensor 101 is composed of a CMOS (Complementary Metal Oxide Semiconductor) type imaging sensor, which converts optical signals, which are optical images, into electrical signals. Light rays incident on the photographic lens 151 are formed as optical images on the image sensor 101 through the aperture 152 and shutter 103 of the lens unit 150. The system control unit 102, composed of a CPU and the like, controls the camera body 100. The system control unit 102 includes an image processing unit for the video signal obtained by the image sensor 101. The system control unit 102 also includes a phase-difference AF unit that performs focus detection processing using a phase-difference detection method based on focus detection image data (signals for phase-difference AF (Auto Focus)) obtained from the image sensor 101 and the image processing unit. More specifically, the image processing unit generates a pair of image data formed by light beams passing through a pair of pupil regions of the imaging optical system as focus detection image data. The phase-difference AF unit detects the amount of focus shift (defocus amount) based on the amount of shift of the pair of image data. In this way, the phase-difference AF unit of the camera 10 performs phase-difference AF (image plane phase-difference AF) based on the output of the camera 10 without using a dedicated AF sensor.
[0011] Memory 104 stores programs, variables, constants, etc., for the operation of the system control unit. Memory 104 also includes electrically erasable and restorable non-volatile memory, which stores various parameters, ISO sensitivity settings, shooting modes, various correction data, etc. The power switch 105 switches between the power on and power off modes of the camera body 100. The mode switching section 106 is a switch for switching and setting various shooting modes such as still image shooting, live view shooting, and video shooting. Modes included in still image shooting mode include auto shooting mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). The AE unit 107 performs photometry based on AE signals obtained from the image sensor 101 and the image processing unit, thereby executing exposure adjustment processing to optimize the shooting conditions. Specifically, it performs photometry based on the AE signals, and calculates the exposure amount under the currently set aperture value (F-number), shutter speed, and ISO sensitivity. From the difference between the calculated exposure amount and a predetermined appropriate exposure amount, it computes the appropriate aperture value, shutter speed, and ISO sensitivity to be set at the time of shooting, and sets them as shooting conditions to perform the exposure adjustment processing. The AE unit 107 functions as an exposure adjustment unit that calculates the exposure conditions at the time of shooting using the photometry result, and controls the aperture value of the diaphragm 152, the shutter speed of the shutter 103, and the ISO sensitivity.
[0012] The rear monitor 108 is configured by a liquid crystal device, an LED, or the like that displays shooting information such as operation states including characters, images, and audio, and messages under the control of the system control unit 102. The touch panel 109 is disposed in an area substantially equivalent to that of the rear monitor 108, detects contact of a finger or a pen, notifies the system control unit 102 of the contact position on the rear monitor 108, and executes an operation or function associated with the contact. The viewfinder display unit 110, like the rear monitor 108, displays shooting information under the control of the system control unit 102, and constitutes an electronic viewfinder (EVF) together with the eyepiece lens 111. The eye proximity detection unit 112 detects the eye proximity state of the photographer. The system control unit 102 selectively displays the shooting information on the rear monitor 108 or the viewfinder display unit 110 in accordance with the eye proximity state of the photographer detected by the eye proximity detection unit 112. The shutter control unit 114 controls the shutter 103.
[0013] Next, the lens unit 150 will be described. The lens unit 150 is mechanically and electrically coupled to the camera body 100 via the lens mount mechanism 120, and is detachable. The lens unit 150 includes a shooting lens 151, a diaphragm 152, a lens drive circuit 153, an aperture control circuit 154, and a lens control unit 155. The lens control unit 155 controls the lens unit 150. The lens control unit 155 includes a memory (not shown) that stores various constants, variables, and programs for lens operation. The lens control unit 155 also includes a non-volatile memory that stores information specific to the lens unit, such as maximum and minimum aperture values and focal length.
[0014] The system control unit 102 of the camera body 100 calculates the amount of defocus using the output information of the image sensor 101. Then, based on the calculated amount of defocus, the system control unit 102 communicates via the lens control unit 155 of the lens unit 150 and controls the lens drive circuit 153 to adjust the focus.
[0015] <Defocus amount> Here, we will explain the amount of defocus. Figure 2 is a diagram illustrating the amount of defocus, showing the relationship between the amount of defocus of the imaging optical system and the phase difference (image shift) between the first focus detection signal and the second focus detection signal acquired from the image sensor 101. An image sensor 101 (not shown in Figure 2) is positioned on the imaging plane 200, and the exit pupil of the imaging optical system is divided into a first pupil region 211 and a second pupil region 212. The amount of defocus d is defined such that |d| is the distance (size) from the imaging position C of the light beam from the subject (221, 222) to the imaging plane 200, and a negative sign (d<0) represents a front-focus state where the imaging position C is on the subject side of the imaging plane 200, and a positive sign (d>0) represents a back-focus state where the imaging position C is on the opposite side of the subject from the imaging plane 200. In the in-focus state where the imaging position C is on the imaging plane 200, d=0. The imaging optical system is in focus (d=0) with respect to subject 221 and in a front-focus state (d<0) with respect to subject 222. The front-focus state (d<0) and the back-focus state (d>0) together are called a defocus state (|d|>0).
[0016] In the front-focused state (d<0), the light beam from the subject 222 that passes through the first pupil region 211 (second pupil region 212) is focused and then spreads out with a width Γ1 (Γ2) centered on the centroid position G1 (G2) of the light beam, forming a blurred image on the imaging surface 200. This blurred image is received by each first focus detection pixel (each second focus detection pixel) on the image sensor, and a first focus detection signal (second focus detection signal) is generated. In other words, the first focus detection signal (second focus detection signal) is a signal that represents a subject image in which the subject 222 is blurred by a blur width Γ1 (Γ2) at the centroid position G1 (G2) of the light beam on the imaging surface 200.
[0017] The blur width Γ1 (Γ2) of the subject image increases roughly in proportion to the increase in the magnitude of the defocus amount d |d|. Similarly, the magnitude of the image shift amount p (=difference in the centroid position of the light beam G1-G2) between the first focus detection signal and the second focus detection signal |p| also increases roughly in proportion to the increase in the magnitude of the defocus amount d |d|. This is the opposite of the back-focus condition, but the principle is the same. Thus, as the magnitude of the defocus amount d increases, the magnitude of the image displacement amount p also increases. In this embodiment, a phase difference detection method of focus detection is performed, in which the defocus amount is calculated from the image displacement amount p obtained using the image sensor 101. Therefore, the phase-difference AF unit of the system control unit 102 converts the image shift amount p into a detected defocus amount using a conversion coefficient calculated based on the baseline length, given the relationship that the magnitude of the image shift amount p increases as the magnitude of the defocus amount d increases. In this embodiment, the unit of the defocus amount is the product of the aperture value in the imaging optical system during shooting and the allowable circle of confusion diameter δ [Fδ].
[0018] <Camera Functional Configuration> Figure 3 is a diagram showing the functional configuration of the camera 10 according to the embodiment. The camera 10 comprises an imaging unit 301, a defocus range variation estimation unit 302, a set value determination unit 303, and a setting control unit 304. The defocus range variation estimation unit 302, the set value determination unit 303, and the setting control unit 304 are realized by the CPU or other components constituting the system control unit 102 executing a predetermined program. The imaging unit 301 captures an image of the subject and outputs image data and a defocus map, which will be described later. The defocus range variation estimation unit 302 estimates the defocus range variation of the subject based on the image data and defocus map acquired from the imaging unit 301. The defocus range variation estimation unit 302 uses a trained machine learning model that takes the image data and defocus map as input and outputs the defocus range variation. The setting value determination unit 303 determines the setting value of the imaging unit 301 based on the defocus range variation range estimated by the defocus range variation range estimation unit 302. The setting control unit 304 controls the settings of the imaging unit 301 based on the setting value determined by the setting value determination unit 303. In this embodiment, the AE unit 107 is described as performing exposure adjustment processing based on the setting value determined by the setting value determination unit 303. In this way, the camera 10 can perform a combination of estimating the range of defocus variation and exposure adjustment processing, and can change the settings to values suitable for the subject's condition and shooting environment according to the estimated range of defocus variation.
[0019] <Defocus range variation> The defocus range variation estimation unit 302 estimates the variation range of the defocus range of the subject. Here, the defocus range is the range of values for the amount of defocus that the subject has, and the variation range is the range of values that represents the variation in the defocus range. Figure 4 is a diagram illustrating the defocus range and the range of variation in the defocus range. Figure 4(a) shows the process of photographing person 401 using camera 10. 402 represents the pupil of person 401, 403 represents the face of person 401, and 404 represents the torso of person 401, each displayed and visualized in terms of their spatial extent as seen from camera 10. 405 indicates the focus position of the camera 10's shooting lens 151, showing that it is focused on the pupil 402 of person 401. Figure 4(b) shows the defocus range of the pupil 402, face 403, and torso 404 of person 401. The horizontal axis represents the magnitude of the defocus amount, with the side closer to the camera 10 defined as near and the side further away as far. The length of the line segment represents the defocus range, which is the range of values for the defocus amount. Figure 4(c) shows the range of defocus variation for the pupil 402, face 403, and torso 404 of person 401. The length of the line segment represents the range of the closest and furthest values of the defocus range, and represents the range of defocus range variation, which is the range of values that represents the variability of the defocus range.
[0020] In Figure 4(a), for example, the extent of the object's spread in the depth direction of the torso 404 of person 401 as seen from camera 10 is such that the closest point is the tip of person 401's nose and the furthest point is the tip of person 401's shoulder. Therefore, the closest value of the defocus amount of the torso 404 of person 401 is the defocus amount indicating the tip of person 401's nose, and the furthest value of the defocus amount is the defocus amount indicating the tip of person's shoulder. The range of values defined by these values is the defocus range of the torso 404 of person 401. However, if the defocus amount indicating the tip of the nose or the defocus amount indicating the tip of person's shoulder varies and cannot be uniquely determined, the range of values will be such that as shown in Figure 4(c), the defocus range variation of the torso 404 of person 401. The torsos of the figures in Figures 4(b) and 4(c) illustrate these relationships. In Figure 4(b), the nearest point is, for example, 0.2Fδ, and the farthest point is, for example, -1.4Fδ. In Figure 4(c), the nearest point is, for example, 0.6Fδ to 0.1Fδ, and the farthest point is, for example, -1.2Fδ to -1.8Fδ. The defocus range variation estimation unit 302 estimates the defocus range variation, which is a range of values representing the variation in the defocus range, by taking into account the distance relationship in the depth direction of the estimated object, such as the pupils and torso of the subject, and the variation on the far side and near side.
[0021] In the first embodiment, the setting values determined by the setting value determination unit 303 are assumed to be the shutter speed and ISO sensitivity of the imaging optical system during shooting. However, the setting values are not limited to shutter speed and ISO sensitivity, but can be any values that can control the range of defocus variation. <Determining Shutter Speed and ISO Sensitivity> The flow for determining shutter speed and ISO sensitivity in this embodiment will be described below. Figure 5 is a flowchart showing the process performed by camera 10. In step S501, the phase-detection AF unit starts autofocus (AF), and the AE unit 107 starts autofocus (AE). Generally, the phase-detection AF unit performs AF using the smallest part of the defocus range variation estimation unit 302 as the focus position, but the part used for AF is not limited to the smallest part of the estimation unit. The conditions for starting AF and AE may be pressing a button assigned to start AF and AE, or half-pressing the shutter button.
[0022] In step S502, imaging is performed by the imaging unit 301 under the control of the system control unit 102. In step S503, under the control of the system control unit 102, the imaging unit 301 outputs a defocus map based on the image captured in step S502. The defocus map is information about the defocus amount distribution to which a certain amount of defocus is assigned to a certain number of pixels on the imaging surface 200. In step S504, the defocus range variation estimation unit 302 estimates the defocus range variation for each part of the subject to be estimated. The defocus range variation estimation unit 302 takes the image data and defocus map acquired from the imaging unit 301 as input and outputs the defocus range of the subject. As an estimation result, the defocus range variation estimation unit 302 outputs the defocus range variation for the entire subject and for each part such as the eyes, face, and torso.
[0023] In step S505, the system control unit 102 determines whether the defocus range variation range estimated in step S504 is greater than or equal to a threshold, that is, whether the defocus range variation range of the closest or furthest point of the area where AF is performed is greater than or equal to a threshold. If it is determined that the defocus range variation range is greater than or equal to a threshold, the process proceeds to step S506. If it is determined that the defocus range variation range is less than a threshold, the process is terminated because the defocus range variation range is small enough that there is no need to change the set value. In step S506, the system control unit 102 determines whether the number of times the shutter speed and ISO sensitivity have been determined is less than or equal to the set value. If it determines that the number of determinations is less than or equal to the set value, the system proceeds to step S507. If the number of determinations exceeds the set value, the system determines that further changes to the settings will not reduce the range of defocus variation, and terminates the change of the shooting settings.
[0024] In step S507, the system control unit 102 calculates the average value of the defocus range variation at the closest point and the defocus range variation at the furthest point of the area where AF is performed, from the defocus range variation range estimated in step S504. In step S508, the setting value determination unit 303 determines the shutter speed based on the average value of the defocus range variation of the AF area calculated in step S507. The shutter speed is set to a value such that the estimated result of the defocus range variation of the AF area, which is then estimated by the defocus range variation estimation unit 302, becomes smaller. Further details will be described later. In step S509, the AE unit 107 performs exposure adjustment processing based on the shutter speed determined in step S508 and determines the ISO sensitivity. In step S510, the setting control unit 304 adjusts to the setting value determined in steps S508 and S509. In step S511, the system control unit 102 increments the decision count by +1 and stores it in memory 104. Then, the process returns to step S502, where the shutter speed and ISO sensitivity are re-determined. Once the determination of the shutter speed and ISO sensitivity is complete, the decision count is cleared.
[0025] Next, as a specific example, we will explain the flow for determining the settings when performing autofocus (AF) with the pupil 402 of person 401 as the focus point, as shown in Figure 4. In step S504, the defocus range variation estimation unit 302 estimates the defocus range variation of the target area. If, in step S505, it is determined that the range of change in the defocus range of the closest or furthest point of the pupil 402 of person 401 is greater than or equal to a threshold, the process proceeds to step S506. If, in step S506, it is determined that the number of times the shutter speed and ISO sensitivity have been determined is less than or equal to the set value, the process proceeds to step S507. In step S507, the average value ER ̄ ( ̄ is superimposed on ER) of the defocus range variation range of pupil 402 is calculated from the defocus range variation range estimated in step S504 using the following formula (1). E Max max : Maximum value of the closest point of the pupil E Max min : The minimum value of the closest point of the pupil EMin max : Maximum distance of the pupil EMin min : Minimum value of the furthest point of the pupil
[0026]
number
[0027] In step S508, the shutter speed is determined based on the average value of the defocus range variation of the pupil 402 calculated in step S507. The shutter speed is set to a value such that the estimated result of the defocus range variation of the pupil 402, which is then estimated by the defocus range variation estimation unit 302, approaches the threshold value of step S505. In this embodiment, it is assumed that the defocus range variation is caused by noise in the image sensor. In this case, the effect of noise in the image sensor can be suppressed by slowing down the shutter speed (increasing the value of the shutter speed), so the shutter speed is set to a value as shown in equation (2) below. The shutter speed reduction factor α is α > 1.0, for example 1.2. SS1: Changed shutter speed SS0: Changed shutter speed
[0028]
number
[0029] In step S509, the AE unit 107 performs exposure adjustment processing based on the shutter speed determined in step S508, as shown in equation (3) below, and determines the ISO sensitivity to lower the ISO sensitivity (make the ISO sensitivity value smaller). S1: Changed ISO sensitivity S0: Current ISO sensitivity
[0030]
number
[0031] In step S511, the number of decisions is incremented by 1 and stored in memory 104. Then, the process returns to step S502, where the shutter speed and ISO sensitivity are re-determined. One effect of this embodiment is that by slowing down the shutter speed and lowering the ISO sensitivity based on the shutter speed, the influence of noise from the image sensor is reduced. As a result, the quality of the defocus map output from the imaging unit is improved, and the range of variation in the defocus range of the estimated target is reduced. By reducing the range of variation in the defocus range, it becomes possible to take pictures that are more in focus on the subject.
[0032] <Machine learning model for estimating the range of defocus variation> Figure 6 shows the functional configuration of the information processing device 20 for training machine learning models. The information processing device 20 is a learning means for learning the parameters of the machine learning model used by the defocus range variation estimation unit 302, and comprises a data storage unit 601, an inference unit 605, a loss calculation unit 607, and an update unit 608. The data storage unit 601 stores multiple sets of training data, each set consisting of three types of data: an image 603, a defocus map 604, and the ground truth defocus range variation 602. The ground truth defocus range variation 602 includes the defocus range variation for each part of the subject in the image. The data storage unit 601 also stores model parameters 609, which are the parameters of the machine learning model to be trained. Model parameters 609 are, for example, the connection weighting coefficients between nodes in each neural network. The information processing device 20 is, for example, a PC (personal computer) or a server, and the parameters of the machine learning model learned by the information processing device 20 are transferred to the camera 10 for use. Alternatively, the camera 10 may be configured to include a data storage unit 601, a loss calculation unit 607, and an update unit 608, and the camera 10 may learn the parameters of the machine learning model.
[0033] One specific machine learning algorithm is deep learning, which uses neural networks to generate features and connection weights for learning. This section will explain learning using neural networks. The inference unit 605 reads the model parameters 609 stored in the data storage unit 601 and estimates the defocus range variation. The input data for the inference unit 605 is the training image 603 and the defocus map 604 associated with the training image 603, both stored in the data storage unit 601, and outputs the defocus range variation inference result 606. The loss calculation unit 607 calculates the error between the defocus range variation inference result 606 output by the inference unit 605 and the correct defocus range variation 602, which is the training data, based on the loss function. The update unit 608 updates the neural network model parameters 609 based on the loss function obtained by the loss calculation unit 607, so as to reduce the error between the output of the inference unit 605 and the training data. This update process for the model parameters 609 is performed, for example, by using backpropagation. Backpropagation is a method for adjusting the model parameters 609 of each neural network so as to reduce the above error.
[0034] By applying the model parameters 609, which have been trained using the learning method described above, to the system control unit 102 of the camera 10, the defocus range variation estimation unit 302 can estimate the defocus range variation. Here, we will describe the case where the trained machine learning model is applied to a lens-interchangeable digital camera as the camera 10. The input data is, for example, an image captured by the camera 10 and a defocus map. These are used as input data for the trained machine learning model. The output data is the inference result, and an estimated value of the defocus range variation of the subject is output.
[0035] <Determination of the range of variation in the correct defocus range> Next, we will explain the determination flow for the ground truth defocus range variation, which is the training data used to learn the model parameters. Figure 7 is a flowchart showing the process for determining the variation range of the correct defocus range. Note that the operation in Figure 7 is performed by both human decision-making and mechanical processes. In step S701, the image output from camera 10 and the defocus map are superimposed. This is performed mechanically. In step S702, the nearest and furthest positions of each part of the subject in the image are identified. In step S703, candidate defocus amounts corresponding to the nearest and furthest positions of each part identified in step S702 are listed. Steps S702 and S703 are performed by human visual inspection. In step S704, the range of variation in the defocus range for the closest and furthest values of each part is calculated from the candidate defocus amounts for the closest and furthest values of each part.
[0036] Next, as a concrete example, we will explain by using the case of determining the range of variation in the correct defocus range of a person's face. Figure 8 is a diagram illustrating the determination of the range of variation in the correct defocus range. Figure 8(a) shows the process of photographing person 401 using camera 10. Figure 8(b) is a superimposed image of the face of person 401, taken from the left at an oblique angle in Figure 8(a), and the corresponding defocus map. Figure 8(c) is a diagram illustrating the method for calculating the range of variation of the nearest and farthest values. In the decision flow shown in Figure 7, in step S701, the image output from camera 10 and the defocus map are superimposed, as shown in Figure 8(b). In step S702, the nearest and furthest points from the face of person 401 in the image are identified. In Figure 8(b), the nearest point is to the right of the face, and the furthest point is to the left of the face. In step S703, candidates for the nearest and farthest defocus amounts identified in step S702 are listed. The nearest candidates are, for example, the defocus amounts in regions (1,2), (2,2), and (3,2) of the defocus map in Figure 8(b), which are 0.9Fδ, 0.3Fδ, and 0.4Fδ, respectively. The farthest candidates are, for example, the defocus amounts in regions (2,4) and (3,4) of the defocus map, which are -2.2Fδ and -2.5Fδ, respectively. In step S704, the range of variation in the defocus range for the nearest and farthest values is calculated from the candidates for the nearest and farthest defocus amounts. The method for calculating the range of variation in the defocus range is to use the range from the minimum value to the maximum value among the candidates as the range of variation in the defocus range. Specifically, the range of variation in the defocus range for the nearest value is 0.3Fδ to 0.9Fδ, and the range of variation in the defocus range for the farthest value is -2.5Fδ to -2.2Fδ.
[0037] As described above, when variations in focus detection results occur, appropriate imaging control becomes possible by estimating the defocus range corresponding to the subject area, including the variation information.
[0038] [Example 1] In the first embodiment, an example was described in which it was assumed that the defocus range fluctuation was caused by noise in the image sensor. In contrast, in Modification 1, an example is described in which it was assumed that the fluctuation was caused by motion blur. The processing performed by camera 10 is the same as in Figure 5, and the processing content of steps S508 and S509 differs from that of the first embodiment, so steps other than steps S508 and S509 will be omitted from the explanation. In step S508, the setting value determination unit 303 determines the shutter speed based on the average value of the defocus range variation of the part where AF is performed, which was calculated in step S507. In the modified example 1, it is assumed that the defocus range variation is caused by subject blur. In this case, subject blur can be suppressed by increasing the shutter speed (decreasing the value of the shutter speed), so the shutter speed setting value is determined as shown in equation (4) below. The shutter speed increase ratio β is β < 1.0, for example, 0.7.
[0039]
number
[0040] In step S509, the AE unit 107 performs exposure adjustment processing based on the shutter speed determined in step S508, as shown in equation (5) below, and determines the ISO sensitivity to increase the ISO sensitivity (increase the ISO sensitivity value).
[0041]
number
[0042] [Differentiation 2] In the first embodiment, in step S504, the defocus range variation was calculated from the minimum and maximum values of the closest and furthest defocus amount candidates listed in steps S501 to S503. In contrast, in the second modification, the focus range variation is calculated using an equation that represents the distribution of data used in statistics. Here, we will explain the method for calculating the closest defocus range variation of the face of person 401 mentioned above as an example. The range of variation in the closest defocus area of the face can be expressed using the following equations (6) and (7). N: Number of closest candidates for the faces of 401 people F Max i : Closest candidate values for the faces of person 401 FMax ̄ ( ̄ is placed above FMax): Average value of the nearest candidate for 401 faces. σ: Standard deviation of the nearest candidate face for person 401 K: constant
[0043]
number
[0044] If the closest candidate faces of person 401 are 0.9Fδ, 0.3Fδ, and 0.4Fδ, then using equation (6), we get 0.263, and the range of variation in the closest defocus range of the face is 0.27Fδ to 0.80Fδ using equation (7). Here, the constant is set to 1, but to increase the proportion of the range of variation that includes the closest candidate faces, the constant may be set to a value greater than 1 (for example, 2).
[0045] <Second Embodiment> A second embodiment will be described with reference to Figures 9 to 11. The camera configuration in the second embodiment is the same as that of the camera 10 in the first embodiment. Hereinafter, components similar to those in the first embodiment will be denoted by the same reference numerals and their descriptions will be omitted. Differences from the first embodiment will be described in detail. In the second embodiment, the setting values determined by the setting value determination unit 303 are described as the aperture value and focus position in the imaging optical system during shooting. <Determining aperture value and focus position> The flow for determining the aperture value and focus position in this embodiment will be described. Figure 9 is a flowchart showing the process performed by camera 10. In the flowchart of Figure 9, steps S501 to S505, S509, and S510 are the same as in the flowchart of Figure 5, and their explanation is omitted. If it is determined in step S505 that the defocus range variation is greater than or equal to the threshold, the process proceeds to step S904; if it is determined that the defocus range variation is less than the threshold, the process proceeds to step S910.
[0046] In step S904, the system control unit 102 determines whether multiple parts of the subject have been detected. If it determines that multiple parts of the subject have been detected, it proceeds to step S910. If it determines that multiple parts of the subject have not been detected, it proceeds to step S905. Multiple parts of the subject include, for example, the torso and face of a person. In step S910, the system control unit 102 determines the maximum range of defocus for the smallest part of the subject, such as the eyes of a person, as the width of the target to be focused. The maximum range of defocus is the range from the maximum value of the closest point to the minimum value of the furthest point of the subject. In step S905, the system control unit 102 reads the focus setting, which has been pre-set by the user, from the memory 104 and determines the target width based on that setting. The focus setting is a parameter that determines where the focus should be set when no subject area is detected, and can be selected from three values: "entire," "foreground," and "background." When the focus setting is "entire," the target width is the maximum range of defocus for the entire subject. When the focus setting is "foreground" or "background," the target width is the range of variation at the closest point and the range of variation at the furthest point, respectively.
[0047] In step S906, the system control unit 102 calculates the average value of the aperture value and target width from the target width determined in step S905 or S910. It determines values such that the aperture value and target width are each included within the unit depth determined by the allowable circle of confusion. In step S907, the system control unit 102 determines the focus position based on the average value of the target width calculated in step S906. In this embodiment, the defocus amount, which is the average value of the target width, is used as the focus position to determine the lens drive amount required to control the photographic lens 151.
[0048] Next, as concrete examples, we will explain the setting value determination flow using two scenarios: one where the torso and face of a person can be detected (e.g., kendo), and another where the face is visible at close range but cannot be detected due to water splashes (e.g., a dolphin jumping scene). Figure 10 is a diagram illustrating the aperture value and focus position control for the maximum range of defocus. In an example where the torso and face of a person have been detected, since multiple body parts have been detected in step S904, the process proceeds to step S910. In step S910, as shown in Figure 10(a), the maximum range of facial defocus is determined as the width of the target. In step S906, as shown in Figure 10(b), the average value of the target width and the aperture value are determined such that the target width is contained within the unit depth determined by the acceptable circle of confusion. In step S907, as shown in Figure 10(c), the amount of lens drive required to control the photographic lens 151 is determined using the defocus amount, which is the average value of the target width, as the focus position.
[0049] Figure 11 is a diagram illustrating the aperture value for the closest focusing position and the control of the focusing position. In an example where the face is visible at close range but cannot be detected due to water splashes, the system proceeds to step S905 because multiple parts have not been detected in step S904. In step S905, assuming the focus setting is pre-set to "Foreground" to focus on the dolphin's face, which is at the closest point, the overall closest range of variation shown in Figure 11(a) is determined as the target width. Steps S906 and S907 are similar to the process in the example where the torso and face of a person have been detected, as shown in Figures 11(b) and 11(c).
[0050] As an effect of this embodiment, by determining the center position of the defocus range variation of the subject as the focus position, and by determining the aperture value such that the range obtained from the defocus range variation of the subject is included within the unit depth determined by the allowable circle of confusion, imaging control that takes defocus range variation into account can be performed, and as a result it becomes possible to take pictures that are more in focus on the subject.
[0051] Although the present invention has been described above along with its embodiments, these embodiments are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features. In this embodiment, the digital camera itself, which is the imaging device, is shown as an example of a control device to which the present invention is applied. However, an external information processing device that can communicate with the imaging device may also be configured to function as a control device to which the present invention is applied. (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.
[0052] This embodiment includes the following configurations, methods, and programs. [Configuration 1] A control device for controlling an imaging device equipped with imaging means for imaging a subject, An estimation means for estimating the range of variation in the defocus range of the subject based on the image data and defocus map acquired from the imaging means, A determination means for determining the set value of the imaging means based on the defocus range variation range estimated by the estimation means, A control device for an imaging device, characterized by comprising: a control means for controlling the settings of the imaging means based on the setting value set by the determination means. [Configuration 2] The control device for an imaging device according to Configuration 1, characterized in that the estimation means uses a trained machine learning model that takes image data and a defocus map as input and outputs a defocus range variation. [Configuration 3] The control device for an imaging apparatus according to configuration 1 or 2, characterized in that the defocus range variation range is the range of the closest value and the range of the furthest value of the defocus range. [Structure 4] The control device for an imaging device according to any one of configurations 1 to 3, characterized in that the setting value is at least one of shutter speed, ISO sensitivity, aperture value, and focus position. [Composition 5] The control device for an imaging device according to any one of configurations 1 to 4, characterized in that the determination means determines the set value such that the variation range of the defocus range estimated by the estimation means becomes smaller. [Composition 6] The aforementioned settings are shutter speed and ISO sensitivity. The control device for an imaging device according to any one of configurations 1 to 5, characterized in that the determination means increases the value of the shutter speed and decreases the value of the ISO sensitivity. [Composition 7] The aforementioned setting value is the shutter speed. The control device for an imaging device according to any one of configurations 1 to 5, characterized in that the determination means reduces the value of the shutter speed. [Structure 8] The aforementioned setting value is the aperture value, The control device for an imaging apparatus according to any one of configurations 1 to 7, characterized in that the determination means determines the aperture value such that the range obtained from the defocus range variation range estimated by the estimation means is included within the unit depth determined by the allowable circle of confusion. [Composition 9] The aforementioned setting value is the focus position, The control device for an imaging apparatus according to any one of configurations 1 to 8, characterized in that the determination means determines the center position of the defocus range variation width estimated by the estimation means as the focus position. [Configuration 10] The control device for an imaging apparatus according to any one of configurations 1 to 9, characterized in that the estimation means estimates the range of variation in the defocus range for each part of the subject. [Method 1] A control method for controlling an imaging device equipped with imaging means for capturing images of a subject, A step of estimating the range of variation in the defocus range of the subject based on the image data and defocus map acquired from the imaging means, The steps include determining the set value of the imaging means based on the estimated defocus range variation, A method for controlling an imaging device, characterized by comprising the step of controlling the settings of the imaging means based on the set setting value. [program] A program for controlling an imaging device equipped with imaging means for capturing images of a subject, An estimation means for estimating the range of variation in the defocus range of the subject based on the image data and defocus map acquired from the imaging means, A determination means for determining the setting value of the imaging means based on the defocus range variation range estimated by the estimation means, A program for causing a computer to function as a control means for controlling the settings of the imaging means based on the setting value set by the determination means. [Explanation of Symbols]
[0053] 10: Camera, 20: Information processing device, 100: Camera body, 102: System control unit, 150: Lens unit, 301: Imaging unit, 302: Defocus range variation estimation unit, 303: Set value determination unit, 304: Setting control unit
Claims
1. A control device for controlling an imaging device equipped with imaging means for imaging a subject, An estimation means for estimating the range of variation in the defocus range of the subject based on the image data and defocus map acquired from the imaging means, A determination means for determining the setting value of the imaging means based on the defocus range variation range estimated by the estimation means, A control device for an imaging device, characterized by comprising: a control means for controlling the settings of the imaging means based on the setting value set by the determination means.
2. The control device for an imaging device according to claim 1, characterized in that the estimation means uses a trained machine learning model that takes image data and a defocus map as input and outputs a defocus range variation.
3. The control device for an imaging apparatus according to claim 1 or 2, characterized in that the defocus range variation range is the range of the closest value and the range of the furthest value of the defocus range.
4. The control device for an imaging device according to claim 1 or 2, characterized in that the setting value is at least one of the following: shutter speed, ISO sensitivity, aperture value, and focus position.
5. The control device for an imaging apparatus according to claim 1 or 2, characterized in that the determination means determines the set value such that the variation range of the defocus range estimated by the estimation means becomes smaller.
6. The aforementioned settings are shutter speed and ISO sensitivity. The control device for an imaging apparatus according to claim 1 or 2, characterized in that the determination means increases the value of the shutter speed and decreases the value of the ISO sensitivity.
7. The aforementioned setting value is the shutter speed. The control device for an imaging apparatus according to claim 1 or 2, characterized in that the determination means reduces the value of the shutter speed.
8. The aforementioned setting value is the aperture value, The control device for an imaging apparatus according to claim 1 or 2, characterized in that the determination means determines the aperture value such that the range obtained from the defocus range variation range estimated by the estimation means is included within the unit depth determined by the allowable circle of confusion.
9. The aforementioned setting value is the focus position, The control device for an imaging apparatus according to claim 1 or 2, characterized in that the determination means determines the center position of the defocus range variation width estimated by the estimation means as the focus position.
10. The control device for an imaging apparatus according to claim 1 or 2, characterized in that the estimation means estimates the range of variation in the defocus range for each part of the subject.
11. A control method for controlling an imaging device equipped with imaging means for capturing images of a subject, A step of estimating the range of variation in the defocus range of the subject based on the image data and defocus map acquired from the imaging means, The steps include determining the set value of the imaging means based on the estimated defocus range variation, A method for controlling an imaging device, characterized by comprising the step of controlling the settings of the imaging means based on the set setting value.
12. A program for controlling an imaging device equipped with imaging means for capturing images of a subject, An estimation means for estimating the range of variation in the defocus range of the subject based on the image data and defocus map acquired from the imaging means, A determination means for determining the setting value of the imaging means based on the defocus range variation range estimated by the estimation means, A program for causing a computer to function as a control means for controlling the settings of the imaging means based on the setting value set by the determination means.
Citation Information
Patent Citations
JP137760A
