Image capturing control apparatus, control method, and storage medium

Abstract

To provide a technique for estimating accuracy of dimension measurement performed by using a plurality of parallax images having parallax with each other photographed under a specific photographing condition.SOLUTION: An accuracy estimation device includes acquisition means for acquiring photographing conditions of a plurality of parallax images mutually having parallax, and estimation means for estimating accuracy of dimension measurement performed using the plurality of parallax images on the basis of the photographing conditions.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an accuracy estimation device, an imaging device, an accuracy estimation method, a control method, and a program. [Background technology]

[0002] There is a technology that acquires distance information from a captured image or the state of an imaging device at the time of capturing, and calculates dimensional information of a subject. Patent Document 1 proposes a method for determining a capturing condition that enables optimal distance measurement within a specified distance measurement range by specifying a desired distance measurement range when measuring distance from multiple images captured under different capturing conditions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6091228 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 relates to distance measurement from multiple images taken under different shooting conditions, and does not relate to dimension measurement performed using multiple parallax images with mutual parallax taken under specific shooting conditions.

[0005] The present invention has been made in consideration of the above circumstances, and aims to provide a technology for estimating the accuracy of dimension measurement performed using multiple disparity images with mutual disparity that are captured under specific shooting conditions. [Means for solving the problem]

[0006] In order to solve the above problem, the present invention provides an accuracy estimation device comprising: an acquisition means for acquiring shooting conditions of a plurality of disparity images having mutual disparity; and an estimation means for estimating the accuracy of dimension measurement performed using the plurality of disparity images based on the shooting conditions. [Effects of the Invention]

[0007] According to the present invention, it is possible to estimate the accuracy of dimension measurement performed using a plurality of parallax images having mutual parallaxes that are captured under specific capturing conditions.

[0008] Other features and advantages of the present invention will become more apparent from the accompanying drawings and the following detailed description of the preferred embodiments of the present invention. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing the functional configuration of a digital camera 100. [Figure 2] FIG. 2 is a diagram showing the configuration of an image sensor 11. [Figure 3] 1A and 1B are diagrams for explaining the principle of distance measurement using an imaging surface phase difference method. [Figure 4] 10 is a flowchart of a photographing process including a process of determining and notifying photographing conditions suitable for dimension measurement. [Figure 5] FIG. 2 is a diagram showing a user interface according to the first embodiment. [Figure 6] 5A is a diagram showing an example of a warning display regarding dimension measurement accuracy, and FIG. 5B is a diagram showing an example of a notification in S413 of FIG. 4. [Figure 7] FIG. 10 is a diagram showing a user interface according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] In the following embodiments, a digital camera (image capture device) capable of acquiring depth information related to the distance distribution of a subject is used as an accuracy estimation device for estimating the accuracy of dimension measurement. It is also possible to apply the accuracy estimation device to a digital camera that outputs the distance to one or more points on a focused subject as depth information. The configurations of the following embodiments are also applicable to any device capable of calculating the dimensions of a subject based on a captured image and the depth information corresponding to the captured image. Depth information is information in the depth direction of a captured image, and a depth image or depth map is information representing the depth distribution. Examples of depth information that can be used include an image shift amount map calculated from multiple viewpoint images (multiple parallax images with mutual parallax), a defocus amount map calculated by multiplying the image shift amount by a predetermined conversion coefficient, and a distance map in which the defocus amount is converted into distance information of the subject.

[0012] [First embodiment] FIG. 1 is a block diagram showing the functional configuration of a digital camera 100. The imaging optical system 10 is composed of a lens unit included in the digital camera 100 or a lens device that can be attached to the camera body, and forms an optical image of a subject on an image sensor 11. The imaging optical system 10 includes multiple lenses (not shown) arranged on an optical axis 102 and has an exit pupil 101 located a predetermined distance away from the image sensor 11. In this specification, the direction parallel to the optical axis 102 is defined as the z direction (depth direction). In other words, the depth direction is the direction in which the subject exists relative to the position of the digital camera 100. The direction perpendicular to the optical axis 102 and parallel to the horizontal direction of the image sensor 11 is defined as the x direction, and the direction perpendicular to the optical axis 102 and parallel to the vertical direction of the image sensor 11 is defined as the y direction.

[0013] The image sensor 11 is, for example, a CCD (charge-coupled device) image sensor or a CMOS (complementary metal-oxide semiconductor) image sensor. The image sensor 11 performs photoelectric conversion on the subject image formed on the imaging surface via the imaging optical system 10, and outputs an image signal related to the subject image. As will be described later, the image sensor 11 of this embodiment has an imaging surface phase difference distance measurement function, and is capable of generating and outputting distance information indicating the distance from the front focal position of the digital camera 100 to the focused subject (subject distance) in addition to the captured image.

[0014] The control unit 12 includes a CPU (Central Processing Unit) or a microprocessor, a memory for storing a control processing program, and the like, and controls the operation of each component included in the digital camera 100. In this embodiment, the control unit 12 includes a shooting mode determination unit 120, a focus control unit 121, an exposure control unit 122, a measurement accuracy estimation unit 123, a notification control unit 124, and an optical image stabilization drive amount confirmation unit 125. Optical image stabilization is also expressed as OIS (Optial Image Stabilizer). The control unit 12 also controls the image processing unit 13, the storage unit 14, the input unit 15, the display unit 16, and the communication unit 17.

[0015] The imaging mode determination unit 120 determines whether the current imaging mode is a mode for performing normal imaging (normal imaging mode) or a mode for performing imaging for dimension measurement (measurement mode). If the current imaging mode is the measurement mode, the imaging mode determination unit 120 further determines which of the multiple measurement modes the current imaging mode is, and performs processing such as restricting the imaging conditions according to the determination result.

[0016] The focus control unit 121 performs focus control on a desired subject, and calculates distance information to the subject based on position information of the focusing lens during focusing and optical design information of the imaging optical system 10 .

[0017] The exposure control unit 122 calculates the optimum exposure conditions in consideration of the fixed photographing conditions according to the measurement mode, and sets photographing parameters such as sensitivity, aperture value, and exposure time.

[0018] The measurement accuracy estimation unit 123 estimates an error that may occur during dimension measurement based on the set or calculated focal length, subject distance, sensitivity, aperture value, and exposure time. If the estimated error exceeds a preset error range, the measurement accuracy estimation unit 123 calculates shooting conditions (recommended shooting conditions) that will bring the error within the set range.

[0019] The notification control unit 124 notifies the photographer (user) via the display unit 16 of estimated errors in dimension measurement, recommended imaging conditions, maximum setting values ​​of various imaging parameters, and the like.

[0020] The optical image stabilization drive amount confirmation unit 125 confirms the position of the optical image stabilization lens driven by optical image stabilization during shooting, and estimates the amount of field curvature corresponding to the position of the optical image stabilization lens.The optical image stabilization drive amount confirmation unit 125 then determines whether the drive amount of the optical image stabilization lens has reached a drive amount at which the amount of field curvature becomes greater than a predetermined amount.

[0021] The image processing unit 13 executes various types of image processing. The image processing unit 13 has an image generation unit 130, a depth generation unit 131, and a dimension measurement unit 132. The image processing unit 13 also has a memory 133 used as a work area for image processing. The functions of the image processing unit 13 can be configured using a logic circuit, or can be configured using a CPU and a memory that stores a calculation processing program.

[0022] The image generation unit 130 generates an image for viewing by performing various signal processing such as noise removal, demosaicing, luminance signal conversion, aberration correction, white balance adjustment, and color correction on the image signal output from the image sensor 11. The captured image data output from the image generation unit 130 is temporarily stored in a memory 133.

[0023] The depth generation unit 131 generates a depth image that represents the depth distribution based on signals obtained by the ranging pixels of the image sensor 11. In the depth image, the value indicated by each pixel represents the distance to a subject that exists in the area of ​​the captured image that corresponds to that pixel.

[0024] The dimension measurement unit 132 measures the dimension between positions (at least two points) specified by the user on the image displayed on the display unit 16. The dimension to be measured may be any of the dimension in pixel units on the image, the dimension on the image plane converted from the pixel size, and the dimension on the object side converted based on the shooting magnification.

[0025] The storage unit 14, input unit 15, display unit 16, and communication unit 17 are all connected to a bus. The storage unit 14 includes a non-volatile storage medium. For example, captured image data, intermediate data generated during the processing of each unit of the digital camera 100, parameters referenced in the operation of the image processing unit 13 and the digital camera 100, etc. are stored in the storage unit 14. The storage unit 14 may be any storage unit as long as it ensures processing performance acceptable for realizing the processing. A storage medium capable of high-speed reading and writing and with a large capacity may be used, such as a flash memory.

[0026] The input unit 15 is a user interface unit that includes a device that detects user operation input. For example, input of information to the digital camera 100, operation to change settings, etc. are detected by a dial, button, switch, touch panel, etc., and the input unit 15 outputs a signal corresponding to the operation input to the control unit 12.

[0027] The display unit 16 includes a display device such as a liquid crystal display or an organic electroluminescence (EL) display. The display unit 16 is used for checking the composition of the image during shooting by displaying a through image of the captured image, displaying various setting screens, notifying message information, etc. Furthermore, the display unit 16 displays information output by the notification control unit 124 of the control unit 12. In addition, in an embodiment using a touch panel, the display unit 16 can have both a display function and an input function.

[0028] The communication unit 17 is a communication interface unit that transmits and receives information between the digital camera 100 and an external device. The communication unit 17 can transmit the acquired captured image data, depth information, dimensions, coordinate information, dimension measurement accuracy, etc. to the external device.

[0029] Next, the configuration of the image sensor 11 will be described with reference to Fig. 2. Fig. 2(A) is a schematic diagram showing the arrangement of a pixel group. The direction perpendicular to the plane of Fig. 2(A) is the z direction, and two mutually orthogonal directions within the plane of Fig. 2(A) are the x direction and the y direction. Fig. 2(B) is a schematic diagram explaining the pixel configuration. The direction perpendicular to the plane of Fig. 2(B) is the y direction, and two mutually orthogonal directions within the plane of Fig. 2(B) are the x direction and the z direction.

[0030] As shown in FIG. 2A, the image sensor 11 is composed of a pixel group in which a large number of pixel units 110 are arranged. Each pixel unit 110 has a 2-row, 2-column configuration to which different color filters are applied. As shown in the enlarged view, red (R), green (G), and blue (B) color filters are arranged, and an image signal corresponding to any one of the R, G, and B color information is output from the photoelectric conversion element that constitutes the pixel unit 110. Note that, although FIG. 2A shows an example in which R color filters are arranged at the top left, B color filters at the bottom right, and G color filters at the bottom left and top right, the color filter arrangement is not limited to this.

[0031] The image sensor 11 has a distance measurement function using an imaging surface phase difference method. A cross-sectional view of a unit pixel taken along line II' in FIG. 2A is shown in FIG. 2B. The unit pixel includes a light guide layer 113 including a microlens 111 and a color filter 112, and a light receiving layer 114 including a first photoelectric conversion unit 115 and a second photoelectric conversion unit 116. In the light guide layer 113, the microlens 111 efficiently guides incident light to the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116. The color filter 112 transmits light in a predetermined wavelength band. The color filter 112 transmits only light in a wavelength band corresponding to R, G, or B, and guides it to the subsequent first photoelectric conversion unit 115 and second photoelectric conversion unit 116.

[0032] The light receiving layer 114 is provided with a first photoelectric conversion unit 115 and a second photoelectric conversion unit 116, which photoelectrically convert received light and output analog image signals. Two types of signals output from these two photoelectric conversion units are used for distance measurement. In distance measurement, of the two photoelectric conversion units arranged in a predetermined direction (horizontal direction) in the image sensor 11, an image signal formed by a signal output from the first photoelectric conversion unit 115 is used as an A image signal, and an image signal formed by a signal output from the second photoelectric conversion unit 116 is used as a B image signal. Depth information or distance information can be obtained from the phase difference between the A image signal and the B image signal. In other words, the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116 each partially receive the light beam incident via the microlens 111. Therefore, the image signal A and the image signal B are pupil-divided image signals relating to light beams that have passed through different pupil partial regions of the exit pupil 101 of the imaging optical system 10. In each pixel unit, the image signal (so-called added image signal) obtained by photoelectric conversion by both the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116 is used as the captured image. In other words, the signal obtained by combining the image signal A and the image signal B is equivalent to the image signal for viewing that is output from the photoelectric conversion unit in a configuration in which the unit pixel has only one photoelectric conversion unit.

[0033] The image sensor 11 of this embodiment can output an image signal for viewing and image A and B signals (pupil-divided images) for distance measurement. All of the pixel units constituting the image sensor 11 are equipped with multiple photoelectric conversion units, making it possible to acquire high-density depth information. Note that while FIG. 2(B) illustrates a configuration in which two photoelectric conversion units are arranged in one pixel, this is not limiting, and a configuration in which three or more photoelectric conversion units are provided in one pixel is also possible.

[0034] The distance measurement principle of the image pickup surface phase difference method will be described with reference to Fig. 3. The subject distance can be calculated based on the outputs (pupil-divided images) of the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116. Fig. 3(A) is a schematic diagram showing the exit pupil 101 of the image pickup optical system 10 and the light beam received by the first photoelectric conversion unit 115. Fig. 3(B) is a schematic diagram showing the exit pupil 101 of the image pickup optical system 10 and the light beam received by the second photoelectric conversion unit 116. In Figs. 3(A) and 3(B), the direction perpendicular to the paper surface is defined as the y direction, and two directions orthogonal to each other within the paper surface are defined as the x direction and the z direction.

[0035] The microlens 111 shown in FIGS. 3(A) and 3(B) is arranged so that the exit pupil 101 and the light receiving layer 114 are optically conjugate. A light beam that has passed through the exit pupil 101 of the imaging optical system 10 is collected by the microlens 111 and guided to the first photoelectric conversion unit 115 or the second photoelectric conversion unit 116. At this time, the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116 mainly receive light that has passed through different pupil partial regions, as shown in FIGS. 3(A) and 3(B), respectively. The first photoelectric conversion unit 115 receives light that has passed through the first pupil partial region 310, and the second photoelectric conversion unit 116 receives light that has passed through the second pupil partial region 320.

[0036] The plurality of first photoelectric conversion units 115 included in the image sensor 11 output first image signals corresponding to the A image signals. The plurality of second photoelectric conversion units 116 included in the image sensor 11 output second image signals corresponding to the B image signals. The intensity distribution of an image formed on the image sensor 11 by light passing through the first pupil partial region 310 can be obtained from the first image signals. The intensity distribution of an image formed on the image sensor 11 by light passing through the second pupil partial region 320 can be obtained from the second image signals. The relative positional shift amount between the first and second image signals (so-called parallax amount) is a value that corresponds to the defocus amount. The relationship between the parallax amount and the defocus amount will be described with reference to FIGS. 3(C), 3(D), and 3(E).

[0037] 3(C), 3(D), and 3(E) show a first light beam 311 passing through a first pupil partial region 310 and a second light beam 321 passing through a second pupil partial region 320. FIG. 3(C) shows a state during focusing, in which the first light beam 311 and the second light beam 321 converge on the light receiving surface of the image sensor 11. At this time, the amount of parallax between the first image signal and the second image signal is zero.

[0038] Figure 3(D) shows a defocused state where the image is focused in the negative direction (leftward) of the z-axis (optical axis). The parallax between the first and second image signals is negative. Figure 3(E) shows a defocused state where the image is focused in the positive direction of the z-axis. The parallax between the first and second image signals is positive. Comparing Figures 3(D) and 3(E) reveals that the direction of the misalignment changes depending on whether the defocus amount is positive or negative. It also reveals that misalignment occurs according to the imaging relationship (geometric optical relationship) of the imaging optical system depending on the defocus amount. The parallax corresponding to the misalignment between the first and second image signals can be detected using a region-based matching method.

[0039] The flow of the photographing process, including the determination of photographing conditions suitable for dimension measurement and the notification process, will be described with reference to Fig. 4. The process of each step in Fig. 4 can be realized by the control unit 12 reading out a corresponding processing program stored in the storage unit 14, for example, and loading it into a volatile memory (not shown) for execution.

[0040] In S401, the control unit 12 determines whether or not the photographer (user) has pressed the shutter button halfway. A halfway press of the shutter button is denoted as "SW1." The control unit 12 repeats the determination in S401 until the shutter button is pressed halfway. When the shutter button is pressed halfway, the process proceeds to S402.

[0041] In S402a, the photographing mode determination unit 120 determines whether the current photographing mode set by the user is the measurement mode. If it is the measurement mode, the processing step proceeds to S402b, and if not, the processing step proceeds to S403.

[0042] In S402b, the shooting mode determination unit 120 determines the type of measurement mode that has been set. The exposure control unit 122 sets restrictions on the shooting conditions depending on the type of measurement mode. There are multiple types of measurement modes as shown in Table 1, and the shooting parameters that are preferentially set according to the user's intentions differ depending on the type of measurement mode. The digital camera 100 is configured to be able to set at least one of the multiple types of measurement modes shown in Table 1.

[0043] [Table 1]

[0044] Here, we will explain the relationship between shooting conditions and distance resolution of distance measurement. The baseline length W in the image plane phase difference method corresponds to the distance between the centers of gravity of the light intensity distribution of the light beams at the exit pupil, which corresponds to the baseline length W in triangulation. The baseline length W varies depending on the aperture value, being longest when the aperture is fully open and becoming shorter when the aperture is closed. If the aperture size is smaller than a certain value, the two images cannot be distinguished and distance measurement becomes impossible. The aperture value (limit value) corresponding to the smallest aperture size for which distance measurement is possible depends on the characteristics of the imaging optical system and image sensor, and the parallax calculation method, but is often set to an F-number of F11, for example.

[0045] The aperture value (baseline length converted F-number) calculated according to the base line length W is defined as F' = f / W, where f is the focal length. If the amount of parallax on the image plane is r and the focal length on the image side is s', the amount of defocus on the image side, def, can be calculated using Equation 1. def=rs' / (Wr) (1) The base line length W is in mm and the parallax r is in μm, so W>>r. def≒rs' / W ··· (2) Here, if macro photography is excluded, s' can be approximated as f, so def≒rf / W=rF' (3) It can be expressed as:

[0046] The distance resolution depends on the resolution of detectable defocus, and the defocus resolution depends on the resolution Δr of the parallax r and the aperture value F′ (baseline length converted F value) calculated according to the base line length W. Δdef≒ΔrF' (4) Therefore, the larger the aperture size (the smaller the base line length converted F-number), the more improved the distance resolution becomes.

[0047] Consider the conversion from the image-side distance resolution Δdef to the object-side distance resolution Δd. If the subject distance is s, the distance resolution Δd is given by the following formula: Δd=Δrs(sf) / {fW+r(sf)} ··· (5) If the imaging magnification β=1 / m and macro photography is excluded, the above formula 5 can be approximated by the following formula 6. Δd≒ΔrF′m 2 (6) The smaller the imaging magnification, that is, when the focal length is constant, the farther the subject distance becomes, the worse the distance resolution becomes.

[0048] If the distance resolution is taken as the distance error range and the error (%) of the dimension measurement is calculated using this, it can be approximated by the following formula. Δh(%)≒±mΔrF' / (f±mΔrF')×100 ··· (7) From Equation 7, we can see that, assuming a constant parallax resolution, the dimension measurement error will be smaller when the reciprocal of the magnification, m, is small, the aperture value (F' value) is small, and the focal length is long. In other words, the longer the focal length and the closer the subject distance, the higher the accuracy of measurement.

[0049] Note that the object resolution R can be used as an index of measurement accuracy instead of the shooting distance or shooting magnification. The object resolution R can be calculated using the shooting magnification β and the size c of the unit pixel of the pixel unit 110 using the following formula: R=c / β (8) As can be seen from Equation 8, the resolution R can be obtained by dividing the size of the unit pixel of the image sensor 11 used for shooting by the shooting magnification.

[0050] The parallax resolution Δr depends on factors such as the parallax calculation method and the window size used during calculation, but when the parallax calculation method is the same, it depends on the SNR of the image, so the parallax resolution decreases as the ISO sensitivity increases. Δr can be expressed as a function of the ISO sensitivity, but it is also possible to set an ISO sensitivity that satisfies a certain Δr and consider it a constant value to perform the error calculation using Equation 7. As mentioned above, the parallax resolution Δr depends on the parallax calculation method and window size, but good results can be obtained by shooting at ISO 1600 or lower, for example. Note that, although it also depends on the desired measurement accuracy, measurements are possible even when the sensitivity is increased to around ISO 6400.

[0051] As explained above, dimension measurement errors are related to focal length and magnification, and magnification is determined by focal length and subject distance. In the case of a zoom lens, focal length can also be set as a changeable parameter. Therefore, in addition to the measurement mode settings shown in Table 1, a setting can be provided to select whether to prioritize focal length or subject distance when taking photographs. Setting which to prioritize can be used as an indicator for reporting, as explained later.

[0052] Next, we will explain the relationship between shooting conditions and the distance measurement range. If blur occurs in the image used to calculate the parallax between two images, it becomes difficult to distinguish between the two images, and the accuracy of the parallax calculation decreases. Therefore, the distance range that can be measured is related to the depth of field of the image. The depth of field is expressed on the image side by the circle of least confusion δ and the aperture value F, and the distance measurement range is within the range of ±αFδ. Here, α is a coefficient determined by the parallax calculation method, etc.

[0053] Next, each measurement mode will be described in detail. In fully automatic mode, the exposure control unit 122 automatically sets the focusing resolution, focusing range, shutter speed (exposure time), and ISO sensitivity to a setting that balances the focal length and subject distance during shooting. Taking into account the focusing resolution and focusing range, for example, the aperture value is set to approximately F4, and the ISO sensitivity is set to approximately 400, ultimately resulting in a shutter speed that achieves proper exposure. The lower limit of the shutter speed is set to, for example, (1 / focal length) plus the number of correction steps provided by the image stabilization function. If the shutter speed is longer than the set limit, the exposure control unit 122 adjusts the ISO sensitivity setting to 1600 to achieve proper exposure. If the exposure is even lower, the exposure control unit 122 adjusts the aperture value if it can be set to F4 or lower. However, depending on the focal length, subject distance, and ambient light level, the focusing accuracy may not meet the desired level. However, the exposure control unit 122 adopts the optimal setting within these constraints.

[0054] The ranging resolution priority mode is a mode that minimizes ranging resolution at the focal length and subject distance during shooting. The exposure control unit 122 sets the aperture to full aperture, sets the ISO sensitivity to a low sensitivity, and finally determines the shutter speed. More specifically, for example, the ISO sensitivity is set to about 100 and is limited to a range not exceeding 400. The exposure control unit 122 determines the shutter speed to achieve appropriate exposure according to the amount of ambient light so as to satisfy these conditions.

[0055] The depth measurement priority mode is a mode that maximizes the range of distance measurement possible at the focal length and subject distance during shooting. The exposure control unit 122 sets the aperture to the smallest aperture value that allows distance measurement. As mentioned above, F11 is generally the limit value, so the exposure control unit 122 sets it to, for example, between F8 and F11. Next, the exposure control unit 122 determines the ISO sensitivity and shutter speed so that shooting can be performed with optimal exposure. However, each setting value is determined within the range of the maximum ISO sensitivity suitable for distance measurement and the minimum shutter speed suitable for distance measurement.

[0056] In the camera shake reduction priority mode, the exposure control unit 122 sets the shutter speed so that the effects of camera shake are not induced even without using optical image stabilization at the focal length and subject distance during shooting. For example, the shutter speed is set to an exposure time shorter than (1 / focal length). Next, the exposure control unit 122 determines the aperture value and ISO sensitivity so that shooting can be performed with optimal exposure. The respective setting values ​​are determined within the range of the maximum F-number suitable for distance measurement and the maximum ISO sensitivity suitable for distance measurement.

[0057] In aperture priority mode, the exposure control unit 122 fixes the aperture value to the one set by the user for the focal length and subject distance at the time of shooting, and then determines the ISO sensitivity and shutter speed so that shooting can be performed with optimal exposure. However, each setting value is determined within the range of the maximum ISO sensitivity suitable for distance measurement and the minimum shutter speed suitable for distance measurement.

[0058] In the shutter speed priority mode, the exposure control unit 122 fixes the shutter speed to the one set by the user for the focal length and subject distance at the time of shooting, and then determines the aperture value and ISO sensitivity so that shooting can be performed with optimal exposure. However, each setting value is determined within the range of the maximum F-number suitable for distance measurement and the maximum ISO sensitivity suitable for distance measurement.

[0059] In S403, the focus control unit 121 executes a focus operation. The focus control unit 121 executes a focus adjustment operation on a subject to be photographed using an AF (autofocus) function or an MF (manual focus) function. In the AF function, the focus control unit 121 drives a focusing lens, which is part of the imaging optical system 10, and controls the target subject to be optimally focused using a phase difference method and a contrast method. The focus control unit 121 also estimates the subject distance based on position information of the focusing lens and design information of the imaging optical system 10.

[0060] In S404, the exposure control unit 122 fixes the prioritized shooting parameters according to the constraints of the shooting conditions set in S402b (i.e., according to the type of measurement mode), and then sets other shooting parameters to achieve optimal exposure. The detailed control procedure for the shooting parameters is as explained for each type of measurement mode in S402b. Note that when the shooting mode is the normal shooting mode, there are no constraints on the shooting conditions according to the type of measurement mode, so the exposure control unit 122 sets the shooting parameters without being subject to such constraints.

[0061] In S405, the photographing mode determination unit 120 determines whether the current photographing mode set by the user is the measurement mode. If it is the measurement mode, the processing step proceeds to S406, and if not, the processing step proceeds to S409.

[0062] In S406, the measurement accuracy estimation unit 123 estimates the measurement accuracy. Specifically, the measurement accuracy estimation unit 123 estimates the measurement accuracy (dimension measurement error) according to Equation 7 using the shooting parameters set by the previous processes, the estimated subject distance, and the set focal length. If the measurement accuracy does not satisfy the required threshold (predetermined standard), the measurement accuracy estimation unit 123 selects parameters that can be changed depending on the type of measurement mode and determines whether there are settings that will make the measurement accuracy satisfy the threshold when those parameters are changed. If there are settings that satisfy the measurement accuracy threshold, the measurement accuracy estimation unit 123 changes the shooting parameters to settings that satisfy the measurement accuracy threshold. If there are no settings that satisfy the measurement accuracy threshold, the measurement accuracy estimation unit 123 calculates setting values ​​for each shooting parameter that are recommended when the type of measurement mode is changed.

[0063] Here, individual settings such as aperture value, ISO sensitivity, and shutter speed are set to optimize exposure within the range of limits suitable for measurement, but the final measurement accuracy must be confirmed and the settings adjusted. Since optimal exposure conditions may not be achieved when settings within the limits of each parameter are used, the measurement accuracy estimation unit 123 also determines whether optimal exposure conditions are met. Alternatively, a separate setting may be selected to prioritize settings for optimal exposure for viewing images or settings for dimension measurement, and the recommended values ​​may be calculated according to this setting.

[0064] Furthermore, after determining the optimal aperture value, ISO sensitivity, and shutter speed for measurement, the measurement accuracy estimation unit 123 calculates a combination of focal length and subject distance at which the measurement accuracy satisfies a threshold value. The measurement accuracy estimation unit 123 performs calculations for two cases: when the focal length is fixed and when the subject distance is fixed. In other words, the measurement accuracy estimation unit 123 calculates what subject distance satisfies the accuracy at the focal length set by the user, and what focal length satisfies the measurement accuracy threshold when the subject distance is fixed.

[0065] In S407, the notification control unit 124 notifies the user of the estimated value of the measurement accuracy (dimension measurement error) obtained in S406, information indicating whether the error is smaller than a threshold value, the set shooting parameters, the recommended shooting parameters, the focal length, the subject distance, the shooting magnification, and the like. This notification is performed, for example, by displaying the necessary information on the display unit 16. For example, if the accuracy of the dimension measurement does not meet a predetermined standard, the notification control unit 124 notifies the user that the accuracy of the dimension measurement does not meet the predetermined standard. An example of the case where the accuracy of the dimension measurement does not meet the predetermined standard is when the error is larger than a threshold value. In addition, the notification control unit 124 notifies the user of information related to error and accuracy, as well as information related to setting, recommending, and changing shooting conditions, such as limit values ​​for the shooting parameters that can be set during measurement and a notification of whether to change to recommended shooting parameters.

[0066] If the desired measurement accuracy is not achieved due to the focal length or subject distance, the notification control unit 124 notifies the user of a change in focal length and subject distance, depending on whether the focal length or subject distance is prioritized. In the case of focal length priority (fixed), the notification control unit 124 calculates and notifies the user of a subject distance that will result in a shooting magnification that will achieve the desired measurement accuracy according to the focal length set at the time of shooting. In this case, the user changes the distance to the subject to achieve the desired shooting magnification and then takes the photograph. In the case of subject distance priority (fixed), the notification control unit 124 automatically sets or notifies the user of a focal length that will result in a shooting magnification that will achieve the desired measurement accuracy according to the subject distance at the time of shooting. If there is no focal length that satisfies the shooting magnification, the notification control unit 124 notifies the user of this fact and simultaneously notifies the user of a recommended subject distance.

[0067] Examples of user interfaces are shown in Fig. 5. Fig. 5(A) shows the user interface when priority is given to focal length. Fig. 5(B) shows the user interface when priority is given to subject distance. The user interfaces in Fig. 5(A) and Fig. 5(B) display constraints (such as a minimum shutter speed indicated by reference numeral 503a, described later) that are required for the accuracy of dimension measurement to meet a predetermined standard.

[0068] In FIGS. 5A and 5B, 500 is a display provided on the display unit 16, which displays a captured image acquired by the image sensor 11 during shooting. 501 is an area displaying a symbol indicating the type of measurement mode during shooting, and displays a symbol corresponding to the measurement mode, such as those shown in Table 1. 502 is an area displaying the shooting magnification or the reciprocal of the shooting magnification. The shooting magnification or the reciprocal of the shooting magnification is highlighted by changing the display color, etc., so that the user can recognize whether the shooting magnification is set to achieve the desired measurement accuracy. 503a indicates the minimum shutter speed that can be set in the current measurement mode. 503b is an area displaying the shutter speed during shooting. 504a indicates the maximum aperture value (F-number) that can be set in the current measurement mode. 504b is an area displaying the aperture value during shooting. 505 indicates the degree of exposure under the current shooting conditions relative to the correct exposure. 506a indicates the maximum ISO sensitivity that can be set in the current measurement mode. 506b is an area displaying the ISO sensitivity during shooting. The shutter speed, aperture value, and ISO sensitivity may be fixed depending on the type of measurement mode. In this case, the fixed setting values ​​are displayed in a different color to indicate that they are fixed values.

[0069] 5A, 507a is object distance information that is displayed when it becomes necessary to change the object distance so that the measurement accuracy satisfies the threshold value when focal length priority is set. 507b indicates the current object distance, and 507c is the object distance that is recommended so that the measurement accuracy satisfies the threshold value. In order for the measurement accuracy to satisfy the threshold value under the current shooting conditions, the user needs to move to the object distance indicated by 507c.

[0070] In FIG. 5B, 508a denotes focal length information. The focal length information is displayed when a variable focal length photographing lens is used and the setting is set to prioritize subject distance, and it becomes necessary to change the focal length so that the measurement accuracy satisfies the threshold. 508b denotes the current focal length, and 508c denotes the focal length recommended for satisfying the threshold. To satisfy the threshold for measurement accuracy under the current photographing conditions, the user must change the focal length of the photographing lens to the focal length indicated by 508c. If the measurement accuracy does not satisfy the threshold within the range of changeable focal lengths at the subject distance during photographing, the notification control unit 124 notifies the user that there is no setting that satisfies the threshold for measurement accuracy and recommends changing the subject distance.

[0071] Furthermore, if the measurement accuracy does not satisfy the threshold, the notification control unit 124 may not only display a numerical value but also display warning displays such as those indicated by reference numerals 601 to 605 in Fig. 6(A) on the user interface shown in Fig. 5(A) and Fig. 5(B). Note that the expected value of the measurement accuracy (measurement error) may be displayed by popping up a warning display such as that indicated by reference numeral 604, or may be constantly displayed on the user interface shown in Fig. 5(A) and Fig. 5(B).

[0072] In S408, the control unit 12 determines whether or not the user has changed the imaging conditions via the input unit 15. If the imaging conditions have not been changed, the processing proceeds to S409. If the imaging conditions have been changed, the control unit 12 changes the imaging conditions according to the user's instructions and returns the processing to S401.

[0073] In S409, the control unit 12 determines whether or not the user has fully pressed the shutter button. Full pressing of the shutter button is denoted as "SW2." If the user has fully pressed the shutter button, the process proceeds to S410; if not, the process returns to S401.

[0074] In S410, the control unit 12 performs various control operations and executes a series of imaging controls such as exposure and readout. The image processing unit 13 generates an ornamental image and a depth image based on the captured image read out from the imaging element 11.

[0075] The image for viewing is generated by the image generation unit 130. The image generation unit 130 generates one Bayer array image by adding the pixel values ​​of each pixel of the A image signal and the B image signal, and performs demosaicing processing of the R, G, and B color images on the generated Bayer array image to generate an image for viewing. In addition, processing such as noise removal or reduction, luminance signal conversion, aberration correction, white balance adjustment, and color correction is performed, and data of the generated image for viewing is stored in memory 133.

[0076] The depth image is generated by the depth generation unit 131. First, the depth generation unit 131 performs a light intensity correction process on the A image signal and the B image signal to correct the light intensity balance between the two images. Then, the depth generation unit 131 performs band limitation using a bandpass filter to remove low SNR components. Then, the depth generation unit 131 derives the amount of parallax at each pixel between the two images using a correlation calculation. Methods such as NCC, SSD, and SAD are used to derive the degree of correlation. NCC is an abbreviation for "Normalized Cross-Correlation." SSD is an abbreviation for "Sum of Squared Difference," and SAD is an abbreviation for "Sum of Absolute Difference." The calculated amount of parallax (denoted as d) is converted into a defocus amount using a predetermined conversion coefficient. The defocus amount corresponds to the distance from the image sensor 11 to the focal point of the imaging optical system 10. Here, the predetermined conversion coefficient is denoted as K, and the defocus amount is denoted as ΔL. The parallax amount d is converted into the defocus amount ΔL by the following equation 9. ΔL=K×d (9) Furthermore, the defocus amount ΔL can be converted into the subject distance using the lens formula in geometrical optics shown in Equation 10 below. 1 / A+1 / B=1 / F (10) In Equation 10, A represents the distance from the object plane to the principal point of the imaging optical system 10 (subject distance), B represents the distance from the principal point of the imaging optical system 10 to the image plane, and F represents the focal length of the imaging optical system 10. In Equation 10, the value of B can be derived from the defocus amount ΔL, so that the distance A can be derived based on the setting of the focal length F during imaging. The depth generation unit 131 generates two-dimensional information in which the derived subject distance is used as a pixel value, and stores depth image data based on the two-dimensional information in memory 133.

[0077] In S411, the shooting mode determination unit 120 determines whether the current shooting mode set by the user is the measurement mode. If it is the measurement mode, the processing proceeds to S412; if not, the processing of this flowchart ends. The determination processing of S411 and subsequent processing can be executed in parallel while the captured image is read from the image sensor 11 and image processing is being executed.

[0078] In S412, the optical stabilization drive amount confirmation unit 125 acquires data recording how the optical stabilization unit was driven during exposure when optical stabilization was performed during image capture. In particular, the optical stabilization drive amount confirmation unit 125 acquires information about the average drive position during exposure. Based on the design values ​​of the imaging optical system 10, the optical stabilization drive amount confirmation unit 125 stores information about how field curvature changes depending on the drive position of the optical stabilization unit, and calculates the amount of field curvature corresponding to the acquired average drive position. While it is possible to correct distance measurement errors due to field curvature based on the calculated amount of field curvature when generating a distance image, there are cases where the correction effect is insufficient, such as when the amount of field curvature is greater than a predetermined amount. Therefore, the optical stabilization drive amount confirmation unit 125 sets a limit drive amount according to the imaging optical system 10 and the desired measurement error amount, and determines that the measurement error will increase if the optical stabilization unit is driven beyond that drive amount.

[0079] In S413, if the drive amount of the image stabilization optical system is large and the impact on measurement accuracy cannot be ignored based on the result obtained in S412, the notification control unit 124 notifies the photographer of a decrease in measurement accuracy and a recommendation to re-shoot. Fig. 6(B) shows an example of the notification in S413. The notification in S413 is a warning displayed on a user interface such as that shown in Fig. 5(A) or 5(B). Notification is not limited to a warning display, and it is also possible to notify the image user by, for example, adding a warning as meta information to the captured image.

[0080] As described above, according to the first embodiment, the digital camera 100 estimates the accuracy of dimension measurement performed using a plurality of parallax images based on the shooting conditions. Furthermore, the digital camera 100 issues a notification based on the estimated accuracy.

[0081] The timing when the digital camera 100 estimates the accuracy and notifies the user based on the estimated accuracy may be before or after capturing multiple parallax images. When the accuracy estimation and notification based on the estimated accuracy are performed before capturing images, the user can know in advance whether the accuracy of the dimension measurement will meet a predetermined standard if the image is captured under the current capturing conditions. Furthermore, the user can search for recommended capturing conditions (changes in the capturing conditions required for the accuracy to meet the predetermined standard) that will make the accuracy of the dimension measurement meet the predetermined standard while reflecting the user's intention. Furthermore, the digital camera 100 can notify the user of the recommended capturing conditions (changes in the capturing conditions required for the accuracy to meet the predetermined standard) in a format such as that shown by reference numerals 601 to 603 and 605 in FIG. 6(A), for example.

[0082] When the accuracy is estimated and a notification based on the estimated accuracy is performed after the image is captured, the user can know whether the image was captured under the shooting conditions required for the dimension measurement accuracy to satisfy a predetermined condition, even after the image is captured. In this case, a device other than the digital camera 100 (e.g., a personal computer) may estimate the accuracy and provide a notification based on the estimated accuracy. For example, the personal computer can acquire the shooting conditions corresponding to the captured multiple parallax images and determine whether the dimension measurement accuracy satisfies a predetermined standard based on the acquired shooting conditions. The personal computer can then notify the user of the estimation results.

[0083] The notification content includes measurement accuracy, shooting conditions that need to be changed (changes to shooting conditions), optimal shooting conditions, and a determination result as to whether measurement is possible with an accuracy that meets a predetermined standard. The notification method may be to display the notification content on a computer display screen or to add the notification content to the image meta information. Alternatively, the notification method may be to classify images into different folders depending on the measurement accuracy, shooting conditions, or whether measurement is possible with an accuracy that meets a predetermined standard.

[0084] Here, the subject of notification of the photography conditions is not necessarily limited to the user, but may be the photography system or the dimension measurement application.

[0085] In this embodiment, a method based on an imaging plane phase difference method has been described as an example of dimension measurement. However, the dimension measurement of this embodiment is not limited to this method. For example, a method may be adopted in which the length and area of ​​an object to be measured on an image are determined using image recognition technology such as machine learning, and then the dimensions are calculated by converting the length and area of ​​the object based on the distance information, imaging magnification, and resolution information described above. Alternatively, a method may be adopted in which whether an object with a predetermined length or area is captured in an image is determined based on the distance information, imaging magnification, and resolution information described above. Even in such measurements, accuracy varies depending on noise, blur, resolution, and the like contained in the image. By estimating whether measurement is possible with the desired measurement accuracy (accuracy that meets a predetermined standard) based on the imaging conditions and notifying the user as described above, it is possible to inform the user whether measurement is possible with the desired measurement accuracy.

[0086] [Second embodiment] In the first embodiment, a configuration was described in which, in adjusting the magnification ratio so that the dimension measurement accuracy satisfies a threshold, information about the item that is not given priority, focal length or subject distance, is displayed depending on which item is given priority, thereby assisting in the adjustment of the magnification ratio. In contrast, in the second embodiment, a user interface will be described that allows the user to adjust the magnification ratio without setting a priority for focal length or subject distance.

[0087] In this embodiment, the basic configuration of the digital camera 100 is the same as in the first embodiment. Below, differences from the first embodiment will be mainly described.

[0088] Fig. 7 is a diagram showing a user interface according to the second embodiment. The user interface in Fig. 7 is a user interface for assisting the user in adjusting the imaging magnification so that it does not fall below the minimum imaging magnification required for measurement with the set dimension measurement accuracy.

[0089] The object-side area captured by the digital camera 100 is determined based on the size and magnification of the image sensor 11. Therefore, a rectangular area in the XY directions at distance Z that satisfies the minimum magnification is determined based on the size of the image sensor 11. The control unit 12 superimposes this rectangular area on the user interface as rectangular area 701. That is, the control unit 12 displays information indicating the area corresponding to the minimum magnification among the areas corresponding to the current magnification. Reference numeral 702 indicates the minimum magnification. In the shooting situation shown in FIG. 7, the current magnification displayed in area 502 is 1 / 100, which is smaller than the minimum magnification of 1 / 50. If the image sensor size is 36 mm × 24 mm, a rectangular area of ​​1800 mm × 1200 mm is captured at the minimum magnification. When the magnification is 1 / 100, the area of ​​the screen 500 shown in FIG. 7 corresponds to an area of ​​3600 mm × 2400 mm. Therefore, when the current shooting magnification is smaller than the minimum shooting magnification, the rectangular area 701 is smaller than the screen 500, and therefore the rectangular area 701 is displayed on the screen 500. On the other hand, when the current shooting magnification is equal to or greater than the minimum shooting magnification, the size of the rectangular area 701 is equal to or greater than the size of the screen 500, and therefore the rectangular area 701 is not displayed.

[0090] Therefore, the user can adjust the magnification to be equal to or greater than the minimum magnification by adjusting the focal length and subject distance so that the rectangular area 701 is not displayed. When the magnification is equal to or greater than the minimum magnification, the control unit 12 may notify the user that the magnification condition is met by, for example, changing the display color of the magnification in the area 502.

[0091] As described above, according to the second embodiment, when the current shooting magnification is smaller than the minimum shooting magnification (the minimum shooting magnification that satisfies the required dimension measurement accuracy), the digital camera 100 displays a rectangular area 701 indicating the area to be shot at the minimum shooting magnification on the screen 500. This allows the user to set the shooting magnification to equal to or greater than the minimum shooting magnification, without considering the specific values ​​of the focal length and the subject distance, by intuitively adjusting the focal length or the subject distance so that the size of the rectangular area 701 becomes equal to or greater than the size of the screen 500 and is no longer displayed.

[0092] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0093] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0094] 100 digital camera, 12 control unit, 120 shooting mode determination unit, 121 focus control unit, 122 exposure control unit, 123 measurement accuracy estimation unit, 124 notification control unit, 125 optical image stabilization drive amount confirmation unit

Claims

1. an acquisition means for acquiring photographing conditions for a plurality of parallax images having mutual parallax; an estimation means for estimating accuracy of dimension measurement performed using the plurality of parallax images based on the photographing conditions; An accuracy estimation device comprising:

2. The photographing conditions include at least one of sensitivity, aperture value, exposure time, focal length, subject distance, and drive amount of an optical vibration isolation means.

2. The accuracy estimation device according to claim 1.

3. The photographing conditions include resolution of the plurality of parallax images.

2. The accuracy estimation device according to claim 1.

4. The acquisition means acquires the resolution by dividing the size of a unit pixel of an image sensor used to capture the plurality of parallax images by the imaging magnification of the plurality of parallax images.

4. The accuracy estimation device according to claim 3.

5. The device further includes a notification unit that notifies the user based on the estimated accuracy.

5. The accuracy estimation device according to claim 1, wherein the accuracy estimation device is a device for estimating accuracy of a plurality of pixels.

6. The notification means notifies information indicating the estimated accuracy.

6. The accuracy estimation device according to claim 5.

7. When the estimated accuracy does not satisfy a predetermined standard, the notifying means notifies that the estimated accuracy does not satisfy the predetermined standard.

7. The accuracy estimation device according to claim 5 or 6.

8. a determination unit that determines, when the estimated accuracy does not satisfy a predetermined standard, changes to the photographing conditions that are required to make the dimension measurement accuracy satisfy the predetermined condition, The notification means notifies the user of the change in the imaging conditions when the estimated accuracy does not satisfy the predetermined standard.

7. The accuracy estimation device according to claim 5 or 6.

9. an imaging means for capturing a plurality of parallax images having mutual parallax; an estimation means for estimating accuracy of dimension measurement performed using the plurality of parallax images based on imaging conditions of the plurality of parallax images before capturing the plurality of parallax images; a notification means for performing a notification based on the estimated accuracy before capturing the plurality of parallax images; An imaging device comprising:

10. The notification means notifies a constraint required for the accuracy of the dimension measurement to satisfy a predetermined standard for at least one item included in the photographing conditions.

10. The imaging device according to claim 9.

11. The photographing conditions include a photographing magnification, When the current magnification is smaller than the minimum magnification required for the accuracy of the dimension measurement to satisfy a predetermined standard, the notification means displays information indicating the photographing area corresponding to the minimum magnification among the photographing areas corresponding to the current magnification.

10. The imaging device according to claim 9.

12. When the driving amount of the optical image stabilization means during the photographing of the plurality of parallax images is greater than a threshold value, the notification means notifies the recommendation to photograph again.

12. The imaging device according to claim 9, wherein the imaging device is a lens.

13. An accuracy estimation method executed by an accuracy estimation device, an acquisition step of acquiring photographing conditions for a plurality of parallax images having mutual parallax; an estimation step of estimating accuracy of dimension measurement performed using the plurality of parallax images based on the photographing conditions; An accuracy estimation method comprising:

14. A control method for an imaging device, comprising: an imaging step of capturing a plurality of parallax images having mutual parallax; an estimation step of estimating accuracy of dimension measurement performed using the plurality of parallax images based on imaging conditions of the plurality of parallax images before capturing the plurality of parallax images; a notification step of issuing a notification based on the estimated accuracy before capturing the plurality of disparity images; A control method comprising:

15. A program for causing a computer to function as each of the means of the accuracy estimation device according to any one of claims 1 to 8.

16. A program for causing a computer to function as each of the means of the imaging device according to any one of claims 9 to 12.

Images