Distance measuring device and distance detection technology, imaging device

Abstract

To provide an art that enables a subject distance to be accurately measured even if a temperature changes.SOLUTION: A distance measuring device comprises: acquisition means that acquires an image obtained from an imaging device imaging a subject image image-formed via an optical system by an image pickup element, and a plurality of pieces of temperature information measured upon imaging the image. and within the imaging device at a time in the vicinity of a time when imaging the image; calculation means that calculates distance information corresponding to the image; and distance calibration means that calibrates the distance information. The distance calibration means is configured to calibrate the distance information on the basis of the plurality of pieces of temperature information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a distance measuring device and a distance detection technique, and particularly to a distance measuring device and a distance detection technique used in a digital still camera, a digital video camera, or the like.

Background Art

[0002] Conventionally, as a distance measuring device and a distance detection technique applicable to a digital still camera or a digital video camera, a distance detection technique using a phase difference method is known. In this method, the defocus amount and the distance to the subject can be calculated based on the phase difference between images with different viewpoints.

[0003] In recent years, techniques have been disclosed for correcting the deviation of the calculated subject distance due to heat transmitted from the heat generated by the camera itself or the environmental temperature to the lens, resulting in a deviation in the defocus amount. For example, Patent Document 1 describes performing temperature correction of the distance based on the temperature of a thermometer and a temperature table provided in the lens in order to display an accurate subject distance. Further, Patent Document 2 describes a technique of refocusing by moving the focus lens in a predetermined direction because the focus position may deviate when there is a temperature change of a predetermined amount or more.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

[0006] Patent Document 2 corrects defocus shifts by focus drive, therefore it is not possible to correct distance information.

[0007] This invention was made in consideration of the above-mentioned problems, and its purpose is to provide a technology that can accurately calculate the subject distance even when there are temperature changes depending on the shooting conditions. [Means for solving the problem]

[0008] To achieve the above objective, the distance measuring device of the present invention comprises an acquisition means for acquiring an image obtained from an imaging device that captures an image of a subject formed through an optical system using an image sensor, and a plurality of temperature information measured in the imaging device at the time of image acquisition and in the vicinity thereof, a calculation means for calculating distance information corresponding to the image, and a distance calibration means for calibrating the distance information, wherein the distance calibration means calibrates the distance information based on the plurality of temperature information. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a technology that can accurately calculate the subject distance even when there are temperature changes depending on the shooting conditions. [Brief explanation of the drawing]

[0010] [Figure 1] Figure illustrating the principle of the present invention. [Figure 2] A diagram showing distance measurement errors due to temperature changes. [Figure 3] A diagram showing an example of the positional relationship between a thermometer and an optical system. [Figure 4] A diagram illustrating an example of the relationship between temperature changes in a thermometer and an optical system. [Figure 5]A diagram showing an example of the difference and derivative values ​​of the temperature between a thermometer and an optical system. [Figure 6] Block diagram showing the functional configuration of a digital camera 100, which is an example of a distance measuring device of the present invention. [Figure 7] Diagram illustrating the image sensor 11 of the digital camera 100. [Figure 8] A diagram illustrating the distance measurement principle of the image plane phase-difference distance measurement method. [Figure 9] A flowchart illustrating the processes performed by digital camera 100. [Figure 10] A flowchart illustrating the process according to Example 1 of the present invention. [Figure 11] A flowchart illustrating the process according to Example 2 of the present invention. [Modes for carrying out the invention]

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] <Basic flow of distance measurement> Figure 1 shows the imaging relationship of the lens relating to the principle of the present invention.

[0013] In this specification, the direction parallel to the optical axis is defined as the z-direction or defocus direction, the direction perpendicular to the optical axis and parallel to the horizontal direction of the imaging plane is defined as the x-direction, and the direction parallel to the vertical direction of the imaging plane is defined as the y-direction, or axes are defined accordingly.

[0014] In the present invention, information on the subject distance is obtained by using a method of performing distance measurement by obtaining a parallax amount from an image formed through an imaging optical system, such as an imaging plane phase difference distance measurement method or a DFD (Depth from Defocus) method, and converting the parallax amount into a defocus amount. Here, factors causing deviation of the defocus amount described above include changes in the lens barrel mechanism, elongation of the flange back, glass thickness, interval, lens refractive index, and lens curvature, which result in changes in the optical arrangement. The present invention is characterized in having a configuration for correcting the deviation of the defocus amount.

[0015] In FIG. 1, OBJ represents the object plane, IMG represents the image plane, H represents the front principal point, H' represents the rear principal point, f represents the focal length of the lens, S represents the distance from the object plane to the front principal point, and S' represents the distance from the rear principal point to the image plane. Also, ΔS' represents the defocus amount, and ΔS represents the relative distance on the object side corresponding to the defocus amount. The dashed-dotted line 100 is the optical axis, the dotted line is the imaging light beam, and the broken line is the defocus light beam.

[0016] It is known that the following mathematical formula 1 holds in the imaging of a lens.

[0017]

Equation

[0018] Also, when defocused, mathematical formula 2 obtained by transforming mathematical formula 1 clearly holds.

[0019]

Equation

[0020] Since S and f read from photographing information such as the lens state are known, S' is obtained from mathematical formula 1, and ΔS is obtained from mathematical formula 2 using ΔS' obtained by the above-described imaging plane phase difference distance measurement method or DFD (Depth from Defocus) method.

[0021] As described above, by solving equations 1 and 2 simultaneously, subject distance information can be generated using the shooting information and the amount of defocus. Since the amount of defocus can be determined for a portion or the entire area of ​​the screen, the subject distance information obtained corresponds to the area for which the amount of defocus was calculated.

[0022] <Distance measurement error due to temperature-induced defocus shift> Figure 2 illustrates the temperature-induced distance measurement error in distance measurement.

[0023] In distance measurement methods that measure distance from an image formed through an imaging optical system, such as the aforementioned image plane phase difference distance measurement method and DFD (Depth from Defocus) method, errors occur due to the defocus shift of the imaging optical system.

[0024] Figures 2(a) and 2(b) are schematic diagrams illustrating the temperature focus shift of the imaging optical system 101. As shown in Figure 2(a), light emitted from an object point 102 located on the optical axis 100 is ideally imaged by the imaging optical system 101 at a single point on the optical element 103 arranged on the image plane.

[0025] However, when there is a temperature change in the imaging optical system 101, the optical arrangement changes due to the expansion of the lens barrel mechanism and flange back, as well as changes in glass thickness, spacing, lens refractive index, and lens curvature. As a result, as shown in Figure 2(b), the position where the light emitted from object point 102 forms an image shifts from the optical element 103, causing a defocus shift.

[0026] In typical imaging devices, this defocusing is corrected by driving the focus lens, reducing the degradation of the captured image. However, when performing distance measurement, driving the focus lens to correct the defocusing changes the object distance S in Equation 1 mentioned above, leading to distance measurement errors. Therefore, correction of the defocusing must be done without driving the focus lens.

[0027] <Relationship between thermometer position and optical system temperature> Figure 3 shows an example of the positional relationship between the thermometer and the optical system.

[0028] In digital cameras, the main heat source other than ambient temperature is the image processing IC on the circuit board, which performs image processing using signals obtained from the sensor (image sensor). Nowadays, the number of pixels in captured images has increased, and the load on complex image processing tends to increase. For this reason, digital cameras often have a thermometer mounted near the image processing IC to monitor the circuit board temperature and prevent it from exceeding a predetermined value, in order to prevent overheating.

[0029] The relationship between the temperature indicated by a thermometer near the image processing IC and the temperature change of the optical system during continuous shooting is explained using Figure 4(a). Hereafter, the temperature of the optical system will be assumed to be proportional to the defocus.

[0030] In Figure 4(a), the horizontal axis represents the image sensor operating time, and the vertical axis represents the temperature. The solid line represents the temperature obtained from a thermometer near the image processing IC, and the dashed line represents the temperature of the optical system. The zero position on the horizontal axis is the temperature at the start of shooting, i.e., the ambient temperature. As shown in Figure 4(a), especially immediately after the start of shooting, the rise in temperature near the image processing IC and the image sensor operating time change rapidly with a steep slope, while the temperature of the optical system changes relatively slowly.

[0031] The reason for this is that there is a time lag between the temperature rise near the image processing IC due to continuous shooting and its transmission to the optical system, due to heat dissipation by the digital camera body and the lens barrel of the optical system. If there were no time lag, simply correcting the defocus based on the temperature near the image processing IC would suffice. However, the occurrence of this time lag leaves challenges in correcting the measurement distance. These challenges will be explained using Figure 4(b).

[0032] Figure 4(b) shows an example of different ambient temperatures in a thermometer near an image processing IC. Similar to Figure 4(a), the horizontal axis represents the image sensor drive time, and the vertical axis represents the temperature. The solid line represents ambient temperature A, and the dashed line represents ambient temperature B. When the temperature of the thermometer near the image processing IC is the same, two different image sensor drive times can be obtained depending on the ambient temperature. If the image sensor drive times differ, the temperature of the optical system cannot be accurately estimated. Therefore, defocus correction may not be performed accurately, and defocus errors may occur. This defocus error causes errors in distance measurement and becomes a problem when performing temperature correction of distance information. This is a problem that can occur when there is some correlation between the temperature of the optical system and defocus, not just a proportional relationship. Solutions to the above problems will be discussed later.

[0033] <Solutions to the correction challenges> In this invention, in order to solve the problems that arise when performing temperature correction of distance information, the temperature of the optical system at the time the distance measurement image was taken is estimated using temperature information and time information near the time the distance measurement image was taken. This enables highly accurate subject distance correction.

[0034] Figure 5(a) is an example graph showing the difference between the temperature of the optical system and the temperature of a thermometer near the image processing IC. The horizontal axis represents the image sensor operating time, and the vertical axis represents the temperature difference. The solid line represents ambient temperature A, and the dashed line represents ambient temperature B. As shown in Figure 5(a), the relationship obtained by subtracting the thermometer temperature from the optical system temperature is determined by the heat dissipation of the optical system and the heat generation and heat dissipation of the imaging system, and does not depend on the ambient temperature. In other words, it can be seen that the temperature difference can be estimated if the operating time from the start of shooting can be estimated.

[0035] Figure 5(b) shows the time derivative of the temperature measured by a thermometer near the image processing IC. The horizontal axis represents the image sensor operating time, and the vertical axis represents the derivative of the temperature obtained from the thermometer near the image processing IC. As shown in Figure 5(b), the derivative is the largest immediately after the start of imaging and decreases with time, exhibiting a monotonically decreasing function. After sufficient time has elapsed, i.e., in a steady state, the derivative becomes zero.

[0036] Since the derivative is a monotonically decreasing function, there is a one-to-one relationship between the derivative and elapsed time. In other words, it is possible to estimate the operating time by evaluating the derivative from temperature and time information from thermometers near multiple image processing ICs.

[0037] By using the relationship shown in Figure 5(b) between the temperature information at the time the distance measurement image was captured, the temperature and time information around the time the distance measurement image was captured, and the image sensor drive time information in the distance measurement image, it becomes possible to estimate the image sensor drive time information. By estimating the image sensor drive time information, it becomes possible to estimate the temperature information of the optical system from the relationship in Figure 5(a), and as mentioned above, the temperature of the optical system is proportional to the defocus, so highly accurate correction of the subject distance can be performed.

[0038] <Image Information> Figure 6 shows a block diagram illustrating an example of the functional configuration of a distance measuring device according to the present invention. A digital camera 100 is shown as an example of the distance measuring device according to the present invention.

[0039] The imaging optical system 10 is a photographic lens in the digital camera 100, and it forms an optical image of the subject on the image sensor 11. The imaging optical system 10 is composed of a plurality of lenses 112 arranged on the optical axis and has an exit pupil 111 at a predetermined distance from the image sensor 11.

[0040] The image sensor 11 is, for example, a CCD (charge-coupled device) or a CMOS sensor (complementary metal-oxide-semiconductor). The image sensor 11 converts the subject image formed on the imaging surface via the imaging optical system 10 into photoelectric signals and outputs an image signal related to the subject image.

[0041] The control unit 12 is a control device such as a CPU or microprocessor, and controls the operation of each block of the digital camera 100. For example, the control unit 12 controls autofocus (AF) during image capture, changes in focus position, changes in F-number (aperture), image acquisition, and the storage unit 15, input unit 16, display unit 17, and communication unit 18.

[0042] The measurement unit 13 calculates the distance to the focused subject. As shown in the figure, the measurement unit 13 includes a lens drive information acquisition unit 130, a temperature information acquisition unit 131, a correction information acquisition unit 132, and a correction unit 133. The roles of each component of the measurement unit 13 will be described later.

[0043] The image processing unit 14 is a block that implements various image processing functions of the digital camera 100. It performs various signal processing functions such as noise reduction, demosaicing, luminance signal conversion, aberration correction, white balance adjustment, and color correction of the image signal output from the image sensor 11. The image data (captured image) output from the image processing unit 14 is stored in a memory (not shown) and used by the display unit 17. The output image data is also stored in the storage unit 15. The image processing unit 14 can be configured using logic circuits. Alternatively, it may be configured with a central processing unit (CPU) and a memory that stores the processing program.

[0044] Next, the storage unit 15 is a non-volatile recording medium on which captured image data, intermediate data generated during the operation of each block, and parameters referenced in the operation of the image processing unit 14 and the digital camera 100 are recorded. The storage unit 15 can be any recording medium that can read and write at high speed and has a large capacity, as long as it ensures the processing performance that is acceptable for realizing the processing. For example, flash memory is preferred.

[0045] The input unit 16 is a user interface that detects information input and setting change operation inputs to the digital camera 100, such as dials, buttons, switches, and touch panels. When the input unit 16 detects an operation input, it outputs a corresponding control signal to the control unit 12.

[0046] The display unit 17 is, for example, a display device such as a liquid crystal display or an organic EL display. The display unit 17 is used for confirming the composition during shooting by displaying the captured image, and for displaying various setting screens and message information. Furthermore, by using a touch panel, it can combine display and input functions.

[0047] The communication unit 18 is a communication interface provided by the digital camera 100 that enables the transmission and reception of information with the outside world. The communication unit 18 may be configured to send captured images, shooting information, etc., to other devices.

[0048] Sensor 19 consists of sensors that monitor the status of the imaging device, and typical sensors installed include acceleration sensors, gyroscopes, and temperature sensors.

[0049] <Example of image sensor configuration> A detailed example of the configuration of the image sensor 11 described above will be explained with reference to Figures 7(a) and 7(b).

[0050] As shown in Figure 7(a), the image sensor 11 is composed of multiple 2x2 pixel groups 210, each with different color filters applied, arranged in a linked configuration. As shown in the enlarged view, the pixel groups 210 have red (R), green (G), and blue (B) color filters, and each pixel (photoelectric conversion element) outputs an image signal indicating one of the R, G, or B color information. In this embodiment, the color filters are described as having the distribution shown in the figure as an example, but the implementation of the present invention is not limited to this.

[0051] To realize the ranging function of the image plane phase-difference ranging method, one pixel (photoelectric conversion element) is composed of multiple photoelectric conversion units arranged in a line in the I-I' cross section of Figure 7(a) relating to the horizontal direction of the image sensor 11. More specifically, as shown in Figure 7(b), each pixel is composed of a light guide layer 213 including a microlens 211 and a color filter 212, and a first photoelectric conversion unit 215 and a second photoelectric conversion unit 216.

[0052] In the light guide layer 213, the microlens 211 is configured to efficiently guide the light beam incident on the pixel to the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216. The color filter 212 allows light in a predetermined wavelength band to pass through, and only allows light in one of the R, G, or B wavelength bands mentioned above to pass through, guiding it to the subsequent first photoelectric conversion unit 215 and the second photoelectric conversion unit 216.

[0053] The light-receiving layer 214 is provided with two photoelectric conversion units (a first photoelectric conversion unit 215 and a second photoelectric conversion unit 216) that convert the received light into an analog image signal, and two types of signals output from these two photoelectric conversion units are used for distance measurement. That is, each pixel of the image sensor 11 similarly has two photoelectric conversion units arranged horizontally, and an image signal composed of the signal output from the first photoelectric conversion unit 215 of all pixels and an image signal composed of the signal output from the second photoelectric conversion unit 216 are used. In other words, the first photoelectric The conversion unit 215 and the second photoelectric conversion unit 216 each partially receive the light beam that enters the pixel through the microlens 211. Therefore, the two types of image signals ultimately obtained are pupil-splitting images corresponding to the light beam that has passed through different regions of the exit pupil of the imaging optical system 10. Here, the combined image signals obtained by photoelectric conversion by the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 at each pixel are equivalent to the image signal (for viewing) output from a single photoelectric conversion unit in an embodiment where only one photoelectric conversion unit is provided at each pixel.

[0054] With this structure, the image sensor 11 of this embodiment is capable of outputting both an image signal for viewing and an image signal for distance measurement (two types of pupil-splitting images).

[0055] In this embodiment, it is described that all pixels of the image sensor 11 are equipped with two photoelectric conversion units, but the present invention is not limited to this. For example, the structure shown in Figure 7(b) may be arranged not only horizontally but also vertically, that is, each pixel may be equipped with four photoelectric conversion units, enabling pupil division not only horizontally but also vertically. By using a four-division configuration, detection becomes possible for subjects in any direction, horizontal or vertical, and more accurate distance measurement can be performed.

[0056] <Distance measurement principle of imaging plane phase difference distance measurement method> Here, the principle for deriving the subject distance based on the aforementioned pupil-reducing image set will be explained with reference to Figures 8(a) to (e).

[0057] Figure 8(a) is a schematic diagram showing the exit pupil 111 of the imaging optical system 10 and the light beam received by the first photoelectric conversion unit 215 of the pixel in the image sensor 11. Figure 8(b) is a schematic diagram similarly showing the light beam received by the second photoelectric conversion unit 216.

[0058] The microlenses 211 shown in Figures 8(a) and (b) are arranged so that the exit pupil 111 and the light-receiving layer 214 are optically conjugate. The light beam that passes through the exit pupil 111 of the imaging optical system 10 is focused by the microlenses 211 and guided to the first photoelectric conversion unit 215 or the second photoelectric conversion unit 216. At this time, the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 mainly receive the light beam that has passed through different pupil regions, as shown in Figures 8(a) and (b). The first photoelectric conversion unit 215 receives the light beam that has passed through the first pupil region 510, and the second photoelectric conversion unit 216 receives the light beam that has passed through the second pupil region 520.

[0059] Multiple first photoelectric conversion units 215 in the image sensor 11 primarily receive light beams that have passed through the first pupil region 510 and output a first image signal. Simultaneously, multiple second photoelectric conversion units 216 in the image sensor 11 primarily receive light beams that have passed through the second pupil region 520 and output a second image signal. From the first image signal, the intensity distribution of the image formed on the image sensor 11 by the light beams that have passed through the first pupil region 510 can be obtained. Similarly, from the second image signal, the intensity distribution of the image formed on the image sensor 11 by the light beams that have passed through the second pupil region 520 can be obtained.

[0060] The relative positional shift between the first image signal and the second image signal (the so-called parallax amount) is a value corresponding to the defocus amount. The relationship between the parallax amount and the defocus amount will be explained using Figures 8(c), (d), and (e). Figures 8(c), (d), and (e) are schematic diagrams illustrating the image sensor 11 and the imaging optical system 10. In the figures, reference numeral 511 indicates the first light beam passing through the first pupil region 510, and reference numeral 521 indicates the second light beam passing through the second pupil region 520.

[0061] Figure 8(c) shows the state when in focus, where the first light beam 511 and the second light beam 521 converge on the image sensor 11. At this time, the parallax between the first image signal formed by the first light beam 511 and the second image signal formed by the second light beam 521 is 0. Figure 8(d) shows the state where the image side is defocused in the negative z-axis direction. At this time, the parallax between the first image signal formed by the first light beam and the second image signal formed by the second signal is not 0, but has a negative value. Figure 8(e) shows the state where the image side is defocused in the positive z-axis direction. At this time, the parallax between the first image signal formed by the first light beam and the second image signal formed by the second light beam has a positive value. Comparing Figure 8(d) and Figure 8(e), it can be seen that the direction of the positional shift changes depending on whether the amount of defocus is positive or negative. It can also be seen that the positional shift occurs according to the imaging relationship (geometric relationship) of the imaging optical system, depending on the amount of defocus. The amount of disparity, which is the positional shift between the first image signal and the second image signal, can be detected by the region-based matching method described later.

[0062] <Defocused Image Generation Process> The image processing unit 14 generates a defocused image (defocus distribution information) from the two obtained image signals. The defocus generation unit 141 within the image processing unit 14 performs the generation process for the defocused image. The process related to defocused image generation will now be explained using the flowchart in Figure 9.

[0063] In step S001, the defocus generation unit 131 performs light intensity correction processing on image signals S1 and S2. Due to vignetting (aperture vignetting) at the peripheral angle of view of the imaging optical system 10, the shapes of the first pupil region 510 and the second pupil region 520 are different, resulting in an imbalance in light intensity between image signals S1 and S2. Therefore, in this step, the defocus generation unit 141 performs light intensity correction on image signals S1 and S2 using, for example, a light intensity correction value pre-stored in a memory (not shown).

[0064] In S002, the defocus generation unit 141 performs processing to reduce noise generated during conversion in the image sensor 11. Specifically, the defocus generation unit 141 achieves noise reduction by applying filter processing to image signals S1 and S2. Generally, the higher the spatial frequency in the high-frequency region, the lower the signal-to-noise ratio and the relatively larger the noise component. Therefore, the defocus generation unit 141 applies a low-pass filter whose pass-through rate decreases as the spatial frequency increases. Note that the light intensity correction in S001 may not yield satisfactory results depending on the manufacturing errors of the imaging optical system 10, so it is preferable for the defocus generation unit 141 to apply a band-pass filter that blocks the DC component and has a low pass-through rate for high-frequency components.

[0065] In S003, the defocus generation unit 141 derives the amount of disparity between the images based on the image signals S1 and S2. Specifically, the defocus generation unit 141 sets a point of interest corresponding to representative pixel information and a matching region centered on the point of interest within the image signal S1. The matching region may be a rectangular region, such as a square region with sides of a predetermined length centered on the point of interest. Next, the defocus generation unit 141 sets a reference point within the image signal S2 and sets a reference region centered on the reference point. The reference region has the same size and shape as the matching region described above. The defocus generation unit 141 derives the correlation between the image contained within the matching region of the image signal S1 and the image contained within the reference region of the image signal S2 while sequentially moving the reference point. The reference point with the highest correlation among the derived correlations is identified as the corresponding point in the image signal S2 that corresponds to the point of interest. The relative positional shift between the corresponding point identified in this way and the point of interest becomes the amount of disparity at the point of interest.

[0066] The defocus generation unit 141 calculates the amount of parallax by sequentially changing the point of interest according to the representative pixel information, thereby deriving the amount of parallax at multiple pixel positions determined by the representative pixel information. In this embodiment, for simplicity, in order to obtain defocus information at the same resolution as the viewing image, the number of pixel positions (the group of pixels included in the representative pixel information) for calculating the amount of parallax is set to be the same as that of the viewing image. Methods such as NCC (Normalized Cross-Correlation), SSD (Sum of Squared Difference), and SAD (Sum of Absolute Difference) may be used to derive the correlation.

[0067] Furthermore, the derived parallax amount d can be converted into a defocus amount, which is the distance from the image sensor 11 to the focal point of the imaging optical system 10, by using a predetermined conversion coefficient. Here, if the predetermined conversion coefficient K and the defocus amount ΔL are, the parallax amount d is, ΔL = K × d This allows it to be converted into a defocus amount.

[0068] The defocus generation unit 141 constructs two-dimensional information using the defocus amount derived in this way as pixel values ​​and stores it as a defocus image in a memory (not shown). The defocus image obtained in this way is used as a map of defocus amounts for measuring the subject distance. In this embodiment, the defocus image is mainly generated, output, and recorded. However, it may also be generated, output, and recorded in the form of disparity amount distribution information indicating phase difference, or in the form of subject distance distribution information (subject distance image) obtained by further converting the defocus amount to subject distance based on the lens state of the imaging optical system. Hereafter, the disparity amount distribution, defocus image, and subject distance image will be collectively referred to as distance information. [Examples]

[0069] The basic flow of a distance measuring device according to the first embodiment of the present invention will be described below with reference to Figure 10.

[0070] Figure 10 is a flowchart of the present invention, in which each step of the measurement unit 13 is executed based on the instructions of the control unit 12 in Figure 6.

[0071] In step S101, the shooting information reading process, the measurement unit 13 acquires data from the image (distance measurement image) stored in the memory unit 15. The lens drive information acquisition unit 130 reads parameters such as focal length, focus position, aperture value, and shooting magnification, which are stored in a memory area such as the memory unit 15 in the camera as additional information to the distance measurement image. The temperature information acquisition unit 131 also reads temperature information near the image processing IC and time information when the distance measurement image was taken. The parameters are used in post-processing, such as when converting from defocus to object distance, and are stored in the memory unit 15. If there is no temperature information or time information to acquire, the calibration process is terminated because temperature-based distance calibration is not performed.

[0072] In the temperature reading process in the vicinity of step S102, the correction information acquisition unit 132 reads the temperature information near the image processing IC for a time period close to the time the distance measurement image was taken. This reading may also be performed by the temperature information acquisition unit 131. If temperature information for a close time period cannot be obtained, the time calibration of the temperature by the correction unit 133, which will be described later, is not performed.

[0073] In the temperature reading process, temperature information from a shooting file taken in a nearby time period may be read, or temperature information from a time close to when the distance measurement image was taken may be saved in the storage unit 15 or other memory at the time the distance measurement image was taken, and the saved temperature information may be read.

[0074] In step S103, the temperature correction amount calculation process, the correction unit 133 takes the temperature of the distance measurement image and information from a shooting file taken at a similar time as input, and calculates the temperature correction amount using the temperature coefficient table stored in the camera's memory unit 15. The temperature coefficient table may also be stored in the lens and transmitted during communication between the camera and the lens. Alternatively, for example, the temperature coefficient table may be stored on the cloud, and when performing temperature correction on the distance measurement image, the temperature coefficient table may be downloaded via the network and used for correction.

[0075] In the distance calibration process of step S104, the correction unit 133 performs distance calibration using the temperature after time correction. Distance calibration is performed using the distance coefficient table stored in the memory unit 15 of the camera. As with step S103, the distance coefficient table may be stored in the lens or in the cloud. That is, based on a command from the control unit 12, the communication unit 18 may communicate with an external network, obtain the distance coefficient table stored in the cloud, and provide the distance coefficient table to the correction unit 133. The correction unit 133 calculates the distance calibration value based on the corrected temperature information and calibrates the distance data in the distance measurement image by adding it to the distance value obtained by the method described above. As a result of the distance calibration process of step S104, the correction unit 133 can calculate distance information that has been appropriately temperature corrected.

[0076] In step S105, the distance information calculated in S104 is saved to the storage unit 15.

[0077] As explained above, the present invention provides a technology that can accurately calculate the subject distance even when there are temperature changes depending on the shooting conditions.

[0078] In step S104, the method of calibrating distance was described as calculating and correcting the distance calibration value, but this is not the only method. For example, the distance may be calibrated by calculating and correcting the calibration value of the image-side defocus amount. Alternatively, the distance may be calibrated by calculating and correcting the calibration values ​​of the focal length or the conversion coefficient K mentioned above. By using these methods, it is possible to calculate the subject distance with high accuracy using the same principle as described above. [Examples]

[0079] The following describes the flow of a distance measuring device according to a second embodiment of the present invention, with reference to Figure 11.

[0080] Figure 11 is a flowchart of Embodiment 2 of the present invention, which is performed in the measurement unit 13 in Figure 6, similar to Embodiment 1. In this flowchart as well, each step is executed in each part of the measurement unit 13 based on instructions from the control unit 12.

[0081] The flowchart in Example 2 adds lens replacement detection processing, image sensor drive time calculation processing, and multiple thermometer calibration processing compared to Example 1.

[0082] Steps S201 to S202 are the same process as steps S101 to S102 in Example 1.

[0083] Step S203 is the process of calculating the temperature calibration coefficient based on the image sensor operating time.

[0084] When the image sensor is not operating, the image processing IC, which is the main source of heat, is not operating, so the only temperature change is heat dissipation from the digital camera body and the lens barrel of the optical system. Therefore, by saving the operating time of the image sensor in a log file inside the camera, it becomes possible to accurately estimate the temperature changes. The log file is stored in the storage unit 15, and the correction unit 133 refers to the operating time of the image sensor from the log file when calculating the temperature calibration coefficient.

[0085] Instead of saving the image sensor operating time itself to a log file, the temperature trend can be estimated by comparing the interval between image captures with the sleep time set for the camera, assuming that only heat dissipation occurs during the sleep time. Here, the sleep time set for the camera is the time during which the camera enters a sleep state when no operation is performed for a certain period of time. Even without recording the image sensor operating time in a log file, the time during which the image sensor was not operating can be estimated by referring to the interval between image captures, the sleep time, and the time when the power was turned on again. This allows for accurate estimation of the temperature trend. The interval between image captures and the sleep time set for the camera are acquired by the correction unit 133 from the storage unit 15 and used in the calculation process of the temperature calibration coefficient.

[0086] Furthermore, under the above conditions, in situations where the subject is continuously being viewed through the viewfinder, it is conceivable that no operations on the camera will be performed for a certain period of time. In such cases, for example, the similarity of subjects being continuously photographed can be determined, and if that similarity is above a predetermined threshold, it can be determined that the camera was not in sleep mode (the image sensor was operating). This is because, when photographing a moving subject, if it can be determined that the type of subject has not changed, it can be inferred that the image sensor was operating.

[0087] Step S204 is the process of calculating the temperature calibration coefficient based on the lens replacement determination.

[0088] The distance measuring device and the digital camera including it in this embodiment may have a configuration in which the lens, which is part of the optical system, can be replaced. When the lens is changed during shooting, it is expected that the digital camera body will be hot, while the lens barrel of the optical system will be at the same temperature as the ambient temperature.

[0089] At this time, a time lag occurs before the heat from the digital camera is transferred to the lens barrel of the optical system, so it is necessary to change the temperature calibration coefficient. Alternatively, the time immediately after lens replacement (exchange information) can be saved in the camera's log file, and the temperature calibration coefficient can be changed by correcting the way heat is transferred. The log file containing the exchange information is stored in a storage area such as the storage unit 15, and is used by the correction unit 133 when changing the temperature calibration coefficient.

[0090] Alternatively, the temperature of the lens barrel of the optical system immediately after lens replacement can be estimated from the ambient temperature. The ambient temperature (external temperature, the temperature of the surroundings outside the device) can be obtained, for example, by the communication unit 18 communicating with an external network to acquire temperature information of the shooting location. Specifically, the communication unit 18 can be equipped with a GPS function to acquire location information, and based on the acquired location information, the communication unit 18 can communicate with an external network to acquire temperature information at that location. The acquired temperature information is recorded in a storage area such as the storage unit 15 and used by the correction unit 133 when changing the temperature calibration coefficient.

[0091] Alternatively, the system may determine whether the lighting is artificial or sunlight based on the color temperature of the image, inferring that the image is indoors if it is artificial lighting, and outdoors if it is sunlight, thereby estimating the external temperature. The color temperature determination is performed by the image processing unit 14, and the correction unit 133 refers to the determination result. The system may also allow the user to select whether the image is indoors or outdoors, and this can be specified via the input unit 16 or the display unit 17.

[0092] Furthermore, the system may include an item that allows the user to set external temperature information at the time of shooting, and may also perform an estimation of the external temperature. Specifically, it may be configured as a distance measuring device or digital camera in which the user can input external temperature information via the input unit 16 and the display unit 17.

[0093] Step S205 is the process of calculating the temperature calibration coefficient using multiple thermometers.

[0094] By equipping the optical system with multiple thermometers and using the temperatures at each location, the temperature of the lens barrel can be estimated with greater accuracy, and the temperature calibration coefficient can be corrected. In this case, the temperature information acquisition unit 131 (or the correction information acquisition unit 132) acquires temperature information from multiple thermometers (sensors 19). Then, the correction unit 133 corrects the temperature calibration coefficient using the obtained temperature information.

[0095] Step S206 is a temperature correction calculation process. Unlike in Example 1, in step S206, the correction unit 133 can calculate more accurate temperature correction information by using the temperature calibration coefficient corrected in steps S203 to S205.

[0096] Steps S207 to S208 are the same process as steps S104 to S105 in Example 1.

[0097] According to Example 2, it is possible to provide a technology that can accurately calculate the subject distance even when there are temperature changes depending on the shooting conditions.

[0098] [Other embodiments] The calculation of subject distance was explained in Examples 1 and 2. The calculated subject distance information can be converted into a two-dimensional distance information map (distance image) by a map generation unit (not shown in Figure 6). The generated distance image is recorded in a storage unit 15 or the like, and can be displayed on the display unit 17 by the user's instruction from the input unit 16. When the distance image generated by the map generation unit (not shown) is displayed on the display unit 17, it is preferable that the user can easily see the differences in distance information. For example, color coding or shading may be adjusted according to the distance. This allows the user to confirm a distance image that accurately reflects the calculated subject distance.

[0099] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0100] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. [Explanation of Symbols]

[0101] 10 Imaging optical system 11 Image sensor 12 Control Unit 13 Measurement Unit 14 Image Processing Unit 15 Storage section 16 Input section 17 Display 18 Communications Department 19 sensors 110 Digital Cameras 111 Exit pupil 112 lenses 130 Lens drive information acquisition unit 131 Temperature information acquisition section 132 Correction Information Acquisition Unit 133 Correction section 210 pixel group 211 Microlenses 212 Color Filters 213 Light guide layer 214 Light-receiving layer 215 First photoelectric conversion unit 216 Second photoelectric conversion unit

Claims

1. An imaging device that captures an image of a subject formed through an optical system using an image sensor, and an acquisition means that acquires a plurality of temperature information measured near an image processing IC in the imaging device at the time of image acquisition and at a time close to that time, and the time information at which the plurality of temperature information was acquired. A calculation means for calculating distance information corresponding to the aforementioned image, The system comprises distance calibration means for calibrating the distance information, The distance calibration means is a distance measuring device characterized by calibrating the distance information by estimating the driving time by evaluating the differential value from the plurality of temperature information and time information.

2. The calculation means estimates the driving time of the image sensor, The distance measuring device according to claim 1, characterized in that the distance calibration means calibrates the distance information using the driving time.

3. The imaging device is equipped with a storage means for storing the sleep time, which is the time until the device enters a sleep state, in a storage area. The distance measuring device according to claim 1 or 2, characterized in that the distance calibration means calibrates the distance information using the sleep time.

4. The distance measuring device according to any one of claims 1 to 3, characterized in that the optical system is replaceable, the acquisition means acquires replacement information indicating that the optical system has been replaced, and the distance calibration means calibrates the distance information using the replacement information.

5. The system includes an external temperature acquisition means for acquiring external temperature information at the time of image acquisition, The distance measuring device according to claim 4, characterized in that the distance calibration means calibrates the distance information using the external temperature information.

6. The distance measuring device according to claim 5, characterized in that the external temperature acquisition means acquires the external temperature information by communication with an external source.

7. The system includes an input means that allows the user to input the external temperature information, The distance measuring device according to claim 5 or 6, characterized in that the external temperature acquisition means acquires the input external temperature information.

8. The distance measuring device according to any one of claims 1 to 7, characterized in that the distance calibration means calibrates the temperature information using a temperature calibration coefficient calculated using the external temperature information, and calibrates the subject distance based on the calibrated temperature information.

9. The distance measuring device according to any one of claims 1 to 8, characterized in that the calculation means acquires the distance information based on a first image based on a light beam that has passed through the first pupil region of the optical system and a second image based on a light beam that has passed through the second pupil region of the optical system.

10. The distance measuring device according to any one of claims 1 to 9, characterized in that it has a generation means for generating a distance image using the distance information.

11. An imaging device having a distance measuring device according to any one of claims 1 to 10.

12. An acquisition step which involves obtaining an image from an imaging device that captures an image of a subject formed through an optical system using an image sensor, and acquiring a plurality of temperature information measured near an image processing IC in the imaging device at the time of image acquisition and at a time close to that time, and the time information at which the plurality of temperature information was acquired. A calculation step for calculating distance information corresponding to the aforementioned image, The distance calibration step includes calibrating the distance information, A distance measurement method characterized in that, in the distance calibration step, the distance information is calibrated by estimating the driving time by evaluating the differential value from the plurality of temperature information and time information.

13. A program for causing a computer to perform each of the steps described in claim 12.

Images