Distance measurement device, moving device, distance measurement method, moving device control method, and computer program

Abstract

To achieve a distance measuring device that can reduce the influence of errors.SOLUTION: A distance measuring device has: first acquisition means 310 that acquires first distance information including errors through an imaging optical system; second acquisition means 320 that acquires second distance information including smaller errors than those in the first distance information; correction information generation means 330 that, based on the second distance information, calculates a correction value for correcting the errors in the first distance information; and correction means 340 that corrects the first distance information by using the correction value.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a distance measurement device, a moving device, a distance measurement method, a method for controlling a moving device, a computer program, and the like. [Background technology]

[0002] In imaging devices, a device equipped with a distance measurement function has been proposed that can acquire distance information such as the defocus state of a subject or the distance from the imaging device to the subject (hereinafter referred to as subject distance) based on image signals with parallax captured from different viewpoints. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. WO2010 / 010707 Summary of the Invention [Problem to be solved by the invention]

[0004] In an imaging device with such a distance measurement function, the relative positional relationship between the optical system and the imaging element may shift due to factors such as expansion and contraction of the lens or lens barrel caused by heat, resulting in distance measurement errors. In Patent Document 1, a correction coefficient for correcting these distance measurement errors is created using the temperature detected by a temperature sensor. However, since Patent Document 1 requires a separate temperature sensor and only measures the temperature near the temperature sensor, it may not be possible to accurately detect temperature changes when there is a distribution (variation) in the temperature on the sensor surface or in the lens, and distance measurement errors may not be accurately corrected.

[0005] In view of the above problems, an object of the present invention is to provide a distance measurement device or the like that can reduce the influence of errors. [Means for solving the problem]

[0006] The distance measurement device of the present invention comprises: The distance between the imaging unit and the subject via the imaging optical system Changes over time a first acquisition means for acquiring first distance information including an error; a second acquiring means for acquiring second distance information having an error smaller than that of the first distance information; used to acquire the first distance information Tase a first defocus amount corresponding to the deviation in the optical axis direction between the sensor surface and the image plane; 、 the second distance information Related to The second defocus amount 、 a generation means for calculating a correction value for correcting the first defocus amount based on the difference between the a calculation unit that calculates the distance between the imaging unit and the subject using the first defocus amount corrected using the correction value. death, the second acquisition means acquires the second distance information by calculating an optical flow of feature points of the target from a plurality of images. It is characterized by: [Effects of the Invention]

[0007] According to the present invention, it is possible to realize a distance measurement device or the like that is capable of reducing the influence of errors. [Brief explanation of the drawings]

[0008] [Figure 1] 1A is a diagram illustrating a schematic configuration of an imaging device according to a first embodiment of the present invention, FIG. 1B is an xy cross-sectional view of the imaging element 101 of FIG. 1A, and FIG. 1C is a diagram illustrating a schematic I-I' cross-section of the pixel group 150 of FIG. 1B. [Figure 2] (A) is a diagram showing the relationship between the exit pupil of the imaging optical system and the light receiving part of the image sensor, (B) is a diagram showing the state when in focus, (C) is a diagram showing the state when defocused in the negative direction of the z axis on the image side, and (D) is a diagram showing the state when defocused in the positive direction of the z axis on the image side. [Figure 3] 1A is a functional block diagram showing a schematic configuration of a distance measurement device 110 according to a first embodiment, FIG. 1B is a flowchart showing the operation of the distance measurement device 110, and FIG. 1C is a flowchart showing part of the operation of FIG. [Figure 4] (A) is a diagram showing the two-dimensional distribution of the amount of field curvature of the imaging optical system 120 within the effective pixel range of the image sensor 101, (B) is a diagram showing the amount of field curvature along I-I' in Figure 4(A), and (C) is a diagram showing the amount of change on the image side along I-I' in Figure 4(A). [Figure 5] 1A is a flowchart showing the operation of the second acquisition process performed by the second acquisition means 320, and FIGS. 1B, 1C, and 1D are diagrams explaining the calculation of the optical flow. Also, FIG. 1B shows feature points 501 calculated for the image signal S21 at time t2, FIG. 1C shows feature points 502 calculated for the first image signal S11 at time t1, and FIG. 1D shows the calculated optical flow. [Figure 6] 6A is a flowchart showing the operation of the correction information generation process performed by the correction information generation means 330 in step S330. Also, (B) is a diagram showing the defocus amount D1, which is the first distance information Idist1 acquired by the first acquisition means 310, relative to the assumed image plane. Also, (C) is a diagram showing the defocus amount D2, which is the second distance information Idist2 acquired by the second acquisition means 320, relative to the assumed image plane. (D) is a diagram showing the image-side defocus amount along I-I' in FIGS. 6B and 6C, and (E) is a diagram showing the image-side change amount calculated by fitting, indicated by a dashed line. [Figure 7] 10A is a block diagram illustrating a schematic configuration of a distance measurement system according to a second embodiment of the present invention, and FIG. 10B is a diagram illustrating an example of the configuration of a second distance measurement device 730 according to the second embodiment. [Figure 8] 10A is a functional block diagram showing a schematic configuration of a distance measurement device 720 according to a second embodiment, and FIG. 10B is a diagram illustrating second distance information Idist2 acquired by a second distance measurement device 730. FIG. [Figure 9] 9A is a flowchart illustrating a process for determining whether or not to perform the correction process according to the third embodiment, and FIG. 9B is a flowchart illustrating step S900 in FIG. 9A. [Figure 10]FIG. 10(A) is a flowchart showing the overall flow of performing correction value processing involving area division according to the fourth embodiment, FIG. 10(B) is a diagram showing the state in which the area for calculating the defocus amount is divided, and FIG. 10(C) is a flowchart showing the details of step S1030. [Figure 11] FIG. 11(A) is a flowchart showing the entire process of looping correction by feeding back the calculated correction value, and FIG. 11(B) is a flowchart showing part of FIG. 11(A). [Figure 12] 10(A) is a schematic diagram showing the overall configuration of a moving device according to a fifth embodiment, and FIG. 10(B) is a block diagram of the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, preferred embodiments of the present invention will be described by way of example with reference to the accompanying drawings. In each drawing, the same members or elements are designated by the same reference numerals, and duplicated descriptions will be omitted or simplified.

[0010] In the embodiments, the present invention will be described as being applied to an in-vehicle camera as an imaging device, but imaging devices also include electronic devices with imaging functions, such as digital still cameras, digital movie cameras, smartphones with cameras, tablet computers with cameras, network cameras, drone cameras, and cameras mounted on robots. In addition, in the embodiments, an imaging device mounted on a vehicle is used as an example of a mobile device, but the mobile device is not limited to a vehicle and includes any mobile device, such as an AGV (Automatic Guided Vehicle), an AMR (Autonomous Mobile Robot), a cleaning robot, or a drone.

[0011] Example 1 In this embodiment, distance values obtained by different methods are converted into image plane defocus amounts and compared, and the amount of change in the amount of field curvature over time is calculated as a correction value, thereby reducing the influence of distance measurement errors over time. This will be explained in detail below. <Configuration of imaging device> FIG. 1A is a diagram showing a schematic configuration of an imaging device according to an embodiment of the present invention.

[0012] 1A, an imaging device 100 includes an imaging optical system 120, an imaging element 101, a distance measurement device 110, and information storage means 170. The distance measurement device 110 can be configured using a logic circuit. Alternatively, the distance measurement device 110 may have a central processing unit (CPU) and a memory for storing a processing program, and the CPU as a computer may execute the computer program stored in the memory.

[0013] The imaging optical system 120 is a photographing lens of the imaging device 100 or the like, and has the function of forming an image of a subject on the imaging element 101. The imaging optical system 120 is composed of a plurality of lens groups (not shown), and has an exit pupil 130 at a predetermined distance from the imaging element 101. In this specification, the z-axis is parallel to the optical axis 140 of the imaging optical system 120. Furthermore, the x-axis and y-axis are perpendicular to each other and to the optical axis. <Image sensor configuration>

[0014] The image sensor 101 is composed of a CMOS (complementary metal-oxide semiconductor) or a CCD (charge-coupled device) and has a distance measurement function using an imaging surface phase difference distance measurement method. The subject image formed on the image sensor 101 via the imaging optical system 120 is photoelectrically converted by the image sensor 101 to generate an image signal based on the subject image. A color image can be generated by subjecting the acquired image signal to development processing by an image generation unit. The generated color image can also be stored in an image storage unit.

[0015] Fig. 1(B) is an xy cross-sectional view of the image sensor 101 in Fig. 1(A). The image sensor 101 is configured by arranging multiple pixel groups 150 in a 2-row x 2-column array. The pixel group 150 is configured by arranging green pixels 150G1 and 150G2 in the diagonal direction, and a red pixel 150R and a blue pixel 150B in the other two pixels.

[0016] Fig. 1(C) is a diagram schematically illustrating the II' cross section of the pixel group 150 in Fig. 1(B). Each pixel is composed of a light receiving layer 182 and a light guide layer 181. The light receiving layer 182 has two photoelectric conversion units (a first photoelectric conversion unit 161 and a second photoelectric conversion unit 162) arranged therein for photoelectrically converting received light.

[0017] The light guide layer 181 is provided with microlenses 183 for efficiently guiding light beams incident on the pixels to the photoelectric conversion units, color filters (not shown) that transmit light in a predetermined wavelength band, wiring (not shown) for reading images and driving pixels, etc. This is an example of a photoelectric conversion unit divided into two in one pupil division direction (x-axis direction), but depending on the specifications, an image sensor having photoelectric conversion units divided into two pupil division directions (x-axis direction and y-axis direction) may also be used. The pupil division direction and the number of divisions are arbitrary.

[0018] Furthermore, the combination of colors received by each pixel is not limited to the arrangement shown in the figure, and is configured in a desired arrangement with desired spectral characteristics according to the wavelength band to be detected, such as IR (Infra Red) or White, which does not impose restrictions on the transmitted wavelength band, etc. The light receiving layer 182 has a photoelectric conversion unit formed using a semiconductor or the like that is sensitive to the wavelength band to be detected, and when the wavelength band to be detected is in the visible range, Si or the like is used, but is not limited to this and can be formed of any material according to the target wavelength band. <Principle of distance measurement using the split-pupil image plane phase difference ranging method> The light beams received by the first photoelectric conversion section 161 and the second photoelectric conversion section 162 included in the image sensor 101 of this embodiment will be described with reference to FIGS. 2(A) to 2(D).

[0019] 2A is a diagram showing the relationship between the exit pupil of the imaging optical system and the light receiving portion of the image sensor. Only the exit pupil 130 of the imaging optical system 120 and a green pixel 150G1 are shown as a representative example of pixels arranged in the image sensor 101. The exit pupil 130 and the light receiving layer 182 are optically conjugate with each other due to the microlens 183 in the pixel 150G1 shown in FIG. 2A. 2A, the light beam that has passed through the first pupil region (210) in the exit pupil 130 is incident on the first photoelectric conversion unit 161. On the other hand, the light beam that has passed through the second pupil region (220) is incident on the second photoelectric conversion unit 162.

[0020] A plurality of first photoelectric conversion units 161 provided in each pixel photoelectrically convert the received light beams to generate a first image signal. Similarly, a plurality of second photoelectric conversion units 162 provided in each pixel photoelectrically convert the received light beams to generate a second image signal. From the first image signal, the intensity distribution of an image formed on the image sensor 101 by the light beam that has mainly passed through the first pupil region 210 can be obtained, and from the second image signal, the intensity distribution of an image formed on the image sensor 101 by the light beam that has mainly passed through the second pupil region 220 can be obtained.

[0021] The amount of parallax between the first image signal and the second image signal corresponds to the amount of defocus. The relationship between the amount of parallax and the amount of defocus will be explained using FIGS. 2(B), (C), and (D). FIGS. 2(B), (C), and (D) are schematic diagrams showing the relationship between the image sensor 101 and the imaging position of the imaging optical system 120 in this embodiment. In the diagrams, 211 denotes a first light beam passing through the first pupil region 210, and 221 denotes a light beam passing through the second pupil region 220.

[0022] 2(B) is a diagram showing the state when in focus, in which the first light beam 211 and the second light beam 221 converge on the image sensor 101. At this time, the relative positional shift amount between the first image signal formed by the first light beam 211 and the second image signal formed by the second light beam 221 is zero. FIG. 2(C) is a diagram showing the state when the image side is defocused in the negative direction of the z axis.

[0023] At this time, the amount of relative positional shift between the first image signal formed by the first light beam and the second image signal formed by the second light beam is not 0 but has a negative value. Figure 2(D) is a diagram showing a state where the image side is defocused in the positive direction of the z axis. At this time, the amount of relative positional shift between the first image signal formed by the first light beam and the second image signal formed by the second light beam is not 0 but has a positive value.

[0024] A comparison of Figures 2(C) and (D) reveals that the direction of positional misalignment changes depending on whether the defocus amount is positive or negative. Furthermore, the geometric relationship reveals that parallax occurs depending on the defocus amount. Therefore, the amount of parallax between the first image signal and the second image signal can be detected using a region-based matching technique (described later), and the detected amount of parallax can be converted into a defocus amount using a predetermined conversion coefficient. In this manner, in this embodiment, multiple parallax image signals having parallax output from light beams passing through different pupil regions of a monocular optical system are acquired by a single image sensor, and first distance information is obtained based on these multiple signals. However, acquisition is not limited to a single image sensor, and multiple image sensors may be used.

[0025] Furthermore, by using the imaging relationship of the imaging optical system 120, which will be explained using Equation 2 below, the defocus amount on the image side can be converted into the distance from the subject to the imaging device 100 (hereinafter referred to as subject distance). <Explanation of distance measurement device>

[0026] The distance measurement device of this embodiment will be described. Fig. 3(A) is a functional block diagram showing the general configuration of the distance measurement device 110 of this embodiment, Fig. 3(B) is a flowchart showing the operation of the distance measurement device 110, and Fig. 3(C) is a flowchart showing part of the operation of Fig. 3(B). A central processing unit (CPU) in the distance measurement device 110 executes a computer program stored in memory to perform the processes of Fig. 3(B) and (C), thereby realizing the function of each block in Fig. 3(A).

[0027] Distance measurement device 110 acquires first distance information Idist1 using first acquisition means 310, and acquires second distance information Idist2 using second acquisition means 320. First acquisition means 310 acquires first distance information including errors (errors over time of the imaging device, manufacturing errors) via an imaging optical system, and second acquisition means 320 acquires second distance information having smaller errors than the first distance information. Errors include errors over time and manufacturing errors, but the following examples mainly describe errors over time.

[0028] The correction information generating means 330 obtains a correction value Ic from the first distance information Idist1 and the second distance information Idist2. That is, the correction information generating means 330 calculates the correction value Ic for correcting the time error of the first distance information based on the second distance information. Furthermore, the correcting means 340 corrects the first distance information Idist1 using the correction value Ic to generate and output corrected distance information IdistC. That is, the correcting means 340 corrects the first distance information using the correction value.

[0029] The following describes the processing performed by the first acquisition means 310, the second acquisition means 320, the correction information generation means 330, and the correction means 340. <Acquisition of first distance information> In step S310, the first acquisition means 310 performs a first acquisition process using the first image group Sg1 acquired from the image sensor 101, and acquires first distance information Idist1 representing the distance to the subject (first acquisition step).

[0030] The first image group Sg1 includes a first image signal S11 generated by the first photoelectric conversion unit 161 and a second image signal S12 generated by the second photoelectric conversion unit 162. The specific processing content of the first acquisition process in step S310 will be described below with reference to FIG.

[0031] In step S311, the first acquisition means 310 corrects the difference in light intensity between the first image signal S11 and the second image signal S12. The balance of light intensity between the first image signal S11 and the second image signal S12 is disrupted due to the imaging optical system 120. In step S311, the first acquisition means 310 uses the light intensity correction value stored in the information storage means 170 to perform light intensity correction processing between the first image signal S11 and the second image signal S12. It is not necessary to use the light intensity correction value stored in the information storage means. For example, the light intensity correction may be performed after generating the light intensity correction value from the area ratio between the first pupil region 210 and the second pupil region 220.

[0032] In step S312, the first acquisition means 310 performs noise reduction processing to reduce noise generated in the image sensor 101. Specifically, filter processing using a band-pass filter is performed on the first image signal S11 and the second image signal S12. Generally, the higher the spatial frequency band, the lower the SN ratio (ratio of signal component to noise component) and the relatively larger the noise component.

[0033] Therefore, a low-pass filter whose pass rate decreases as the frequency increases is used. Note that even after the light intensity correction process in step S311, the results may not always match the design values due to manufacturing errors in the imaging optical system 120. For this reason, a band-pass filter whose pass rate in the frequency band when spatial frequency = 0 is 0 and whose pass rate in the high frequency band is low is used.

[0034] In step S313, the first acquisition means 310 performs a parallax detection process to calculate the amount of parallax between the first image signal S11 and the second image signal S12. Specifically, a point of interest is set in the first image signal S11, and a matching region is set with the point of interest at its center. The matching region may be, for example, a rectangle with a side of a predetermined length and centered at the point of interest, but is not limited to this.

[0035] Next, a reference point is set in the second image signal S12, and a reference region centered on the reference point is set. The reference region has the same size and shape as the matching region. While sequentially moving the reference point, the degree of correlation between the first image signal S11 contained in the matching region and the second image signal S12 contained in the reference region is calculated, and the reference point with the highest correlation is set as the corresponding point that corresponds to the attention point. The amount of relative positional deviation between the attention point and the corresponding point is the amount of parallax at the attention point.

[0036] The first obtaining means 310 can calculate the amount of parallax at a plurality of pixel positions by calculating the amount of parallax while sequentially changing the point of interest. A known method can be used to calculate the degree of correlation. For example, a method called NCC (Normalized Cross-Correlation) that evaluates the normalized cross-correlation between image signals, SSD (Sum of Squared Difference) that evaluates the sum of squares of the differences between image signals, or SAD (Sum of Absolute Difference) that evaluates the absolute value of the differences can be used.

[0037] When these correlation calculation methods are used, reliability information that indicates the reliability of the calculated distance information can be generated using the texture amount and frequency component information of each pixel block. In step S314, a distance conversion process is performed. That is, the first acquisition unit 310 converts the amount of parallax into the distance (defocus amount) from the image sensor 101 to the image formation point of the imaging optical system 120 using a predetermined conversion coefficient. Hereinafter, the coefficient for converting the amount of parallax into the amount of defocus will be referred to as the BL value. When the BL value is BL, the amount of defocus is ΔL, and the amount of parallax is d, the amount of parallax d can be converted into the amount of defocus ΔL using the following equation 1.

[0038] (Number 1) ΔL=BL×d The first acquiring means 310 converts the parallax amounts at a plurality of pixel positions into defocus amounts, thereby acquiring first distance information Idist1 including the defocus amounts at a plurality of pixel positions as distance information. Furthermore, to convert this defocus amount into the subject distance, the following formula 2, which is the lens formula in geometric optics, can be used:

[0039] (Number 2) 1 / A+1 / B=1 / f A: distance from the object surface to the principal point of the imaging optical system 120, B: distance from the principal point of the imaging optical system 120 to the image plane, f: focal length of the imaging optical system 120.

[0040] In Equation 2, the focal length is a known value. Furthermore, the value of B can be calculated using the defocus amount. Therefore, the distance A to the object surface, i.e., the subject distance, can be calculated using the focal length and defocus amount. This method of obtaining subject distance information is called a phase difference ranging method, and in particular, the method of obtaining distance information from signals obtained from different pupil regions of a single optical system, as described above, is called a split-pupil phase difference method.

[0041] In this case, if the distance B from the principal point of the imaging optical system 120 to the image plane, which is calculated using the calculated defocus amount, contains an error, then according to equation 2, the calculated distance A to the object plane, i.e., the subject distance, will also contain an error. The above errors can be caused by changes over time in the image capture device 100 due to the effects of changes in the ambient temperature and humidity, vibrations, etc. Specifically, these changes include changes in optical characteristics caused by changes in the refractive index and curvature of each lens in the imaging optical system 120 due to changes in the ambient environment, and deformations such as bending of the image capture element 101 due to changes in the ambient environment. Additionally, errors can also occur due to manufacturing errors.

[0042] Due to a difference between the assumed image plane position of imaging optical system 120 and the actual image position, the conversion relationship between the defocus amount and the distance from the object plane to the principal point of imaging optical system 120 breaks down, causing an error in the value of distance B from the principal point of imaging optical system 120 to the image plane. Hereinafter, the difference between the assumed image plane position of imaging optical system 120 and the actual image plane position is referred to as the image-side change amount. Even when the image-side change amount is not 0, the relationship between the defocus amount and the object distance follows equation 2, so that if the defocus amount is corrected using a correctly estimated image-side change amount, it is possible to determine distance A, which is the subject distance with reduced error.

[0043] As described above, the first acquisition means 310 acquires first distance information based on the first signal and the second signal having parallax output from the image sensor 101 via the imaging optical system. Note that there may be two or more signals having parallax. When the first distance information is acquired using such a pupil division phase difference method, as described above, the first distance information includes errors over time due to the characteristics of the imaging optical system and the image sensor.

[0044] There are several factors that cause the image side change amount to become 0 due to a change over time. For example, a case where the optical characteristics of the imaging optical system 120 change over time due to a temperature change will be described. Figure 4(A) is a diagram showing the two-dimensional distribution of the amount of field curvature of the imaging optical system 120 within the effective pixel range of the image sensor 101, and Figure 4(B) is a diagram showing the amount of field curvature along II' in Figure 4(A).

[0045] In FIG. 4(B), m1 represents the amount of field curvature when no change occurs over time, and m2 represents the amount of field curvature when a change occurs over time. The difference between m1 and m2 in the figure is the aforementioned amount of image-side change. FIG. 4(C) is a diagram showing the amount of image-side change along I-I' in FIG. 4(A). As shown in FIG. 4(C), when a change occurs over time in the optical characteristics of the imaging optical system 120, the amount of image-side change changes depending on the angle of view. If the amount of image-side change is a constant value independent of the angle of view, correction is possible with at least one piece of correction information.

[0046] However, when the amount of image-side change varies depending on the angle of view, it is difficult to estimate the amount of image-side change using only one piece of correction information. In this case, it is necessary to estimate a two-dimensional distribution of the amount of image-side change. In order to generate this correction value Ic, in this embodiment, second distance information acquired by an acquisition means different from the first acquisition means is used to generate the correction value Ic.

[0047] In this embodiment, as described above, first distance information Idist1 indicating the distance to the subject is acquired using the first image group Sg1 acquired from the image sensor 101 of the first acquisition means 310. Furthermore, the first image group Sg1 acquired by the image sensor 101 is temporarily stored in the information storage means 170, and the first image group Sg1 stored in the information storage means 170 is used as the second image group Sg2 in the second acquisition means to acquire second distance information Idist2. <Acquisition of second distance information>

[0048] The second acquisition means 320 acquires the first image group Sg1 and the second image group Sg2 from the information storage means 170. Here, the second image group Sg2 is an image group in which image signals captured at a time earlier than that of the first image group Sg1 are stored in the information storage means 170. The second acquisition means 320 acquires second distance information Idist2 using the first image group Sg1 and the second image group Sg2. The second image group Sg2 includes a first image signal S21 and a second image signal S22 captured using the imaging device 100 at a different time from that of the first image group Sg1.

[0049] In this way, in this embodiment, the second acquisition means acquires the second distance information based on the image signals of two frames, but it may also be configured to acquire the second distance information based on the image signals of three or more frames. As described above, the first image signal S21 is an image signal generated by the first photoelectric conversion unit 161, and the second image signal S22 is an image signal generated by the second photoelectric conversion unit 162.

[0050] The second acquisition means 320 acquires the second distance information Idist2 using a known SfM (Structure from Motion) method. The specific content of the second acquisition process performed by the second acquisition means 320 in step S320 will be described below with reference to FIG. 5(A). FIG. 5(A) is a flowchart showing the operation of the second acquisition process (second acquisition step) performed by the second acquisition means 320 in step S320. A central processing unit (CPU) in the distance measurement device 110 executes a computer program stored in memory to perform each process shown in FIG. 5(A).

[0051] In step S321, optical flow calculation processing is performed. That is, an image signal S11 is obtained from a first image group Sg1 captured at time t1 stored in the information storage means 170, and an image signal S21 is obtained from a second image group Sg2 captured at a different time t2. Then, optical flow is calculated from each image signal using a known method. The time relationship is t1>t2, and t2 is in the time series earlier than t1.

[0052] The optical flow may be calculated from the second image signals S12 and S22 as long as they are image signals from the same viewpoint, or may be calculated from the sum of the first and second image signals of the first image group Sg1 (S11+S12) and the second image signal of the second image group Sg2 (S21+S22).

[0053] The calculation of optical flow will be specifically described using Figures 5(B), (C), and (D). Feature points are calculated for the acquired image signal S11 and image signal S21 using the well-known Harris corner detection algorithm. Figure 5(B) shows feature points 501 calculated for the image signal S21 at time t2, Figure 5(C) shows feature points 502 calculated for the first image signal S11 at time t1, and Figure 5(D) shows the calculated optical flow.

[0054] Feature points 501 calculated for image signal S21 at time t2 are indicated by stars in Fig. 5(B), and feature points 502 calculated for first image signal S11 at time t1 are indicated by stars in Fig. 5(C). Fig. 5(D) shows optical flow 500 calculated by associating the calculated feature points between image signal S21 and image signal S11 using a well-known method, the Kanade-Lucas-Tomasi (KLT) feature tracking algorithm.

[0055] The algorithms used to calculate feature points, feature amounts, and optical flow are not limited to the above methods. FAST (Features from Accelerated Segment Test), BRIEF (Binary Robust Independent Elementary Features), ORB (Oriented FAST and Rotated BRIEF), etc. may also be used.

[0056] In step S322, a distance calculation process is performed. That is, the calculated optical flow 500 is used to calculate the distance to the subject using a known method. The coordinates in the image coordinate system of the subject for which the distance is to be calculated are (u, v), the optical flow of the subject for which the distance is to be calculated is (Δu, Δv), and the distance to the subject is z. Of the camera movements between images used when calculating the optical flow, the rotational movement amount is (ωx, ωy, ωz) and the translational movement amount is (tx, ty, tz). Furthermore, if the focal length of the camera is f, the following relationship holds:

[0057]

number

[0058]

number

[0059] The camera movement amount between the image signal S21 and the image signal S11 used to calculate the optical flow 500 is calculated using a known method. Specifically, the camera fundamental matrix F is acquired using the 8-point algorithm so as to satisfy the epipolar constraint using the feature point 501 at time t2, the feature point 502 at time t1, and the optical flow 500 representing the correspondence between them. At this time, it is also preferable to use the RANSAC (Random Sample Consensus) method to efficiently remove outliers and perform calculation using a stable method.

[0060] The camera fundamental matrix F is decomposed into the camera fundamental matrix E using a known method, and the camera extrinsic parameters, rotational movement amount R(ωx,ωy,ωz) and translational movement amount T(tx,ty,tz), are calculated from the camera fundamental matrix E. Here, the calculated camera extrinsic parameters are relative displacements of the camera movement amount from time t2 to time t1, and since the scaling is indefinite, the translational movement amount T(tx,ty,tz) in particular is a normalized relative value.

[0061] This is scaled to obtain the translational movement amount T(tx, ty, tz), which is the actual movement amount. Specifically, a second image signal S12 corresponding to an image signal taken from a different viewpoint at the same time as the first image signal S11 from which the feature point 502 was obtained is obtained from the information storage means 170. Similarly, an image signal S22 corresponding to an image signal taken from a different viewpoint at the same time as the image signal S21 from which the feature point 501 was obtained is obtained from the information storage means 170.

[0062] Then, in step S310, a first acquisition process is performed on each image pair to acquire distance values from the first distance information based on the amount of parallax. After that, the amount of translational movement T, which is parallel to the optical axis of the camera, is acquired from the difference in distance values between feature point 502 and feature point 501, which are associated by the optical flow.

[0063] The actual translational movement amount T(tx, ty, tz) from time t2 to t1 is obtained by scaling other components from the obtained scaled actual movement amount tz. Then, the distance z from the camera at each coordinate on the image of the feature point 502 in the first image signal S11 is calculated using Equations 3 and 4.

[0064] The method for scaling the camera movement amount is not limited to this method, and the camera movement amount may be obtained from various measuring devices, specifically an IMU (inertial measurement unit) or a GNSS (Global Navigation Satellite System), or in the case of an in-vehicle camera, vehicle speed information or GPS information, and then scaling may be performed.

[0065] Note that bundle adjustment, a well-known method, may be used to calculate the camera movement amount and the positional relationship between the subject and the camera. The relationships between variables such as the camera fundamental matrix and optical flow, including internal camera parameters such as focal length, can be analytically calculated using the nonlinear least squares method to improve consistency.

[0066] Of the feature points used to calculate the camera movement amount, feature points calculated from a subject that is not a stationary object with respect to the world coordinate system to which the image capture device belongs may be excluded from the processing. The known method of camera movement amount estimation calculates various parameters assuming that the subject is a stationary object, and therefore becomes a source of error when the subject is a moving object.

[0067] Therefore, the accuracy of calculating various parameters can be improved by excluding feature points calculated from moving objects. Moving objects are determined by classifying the subject using image recognition technology or by comparing the relative value of the amount of change in the time series of acquired distance information with the amount of movement of the imaging device.

[0068] In step S323, a second distance information conversion process is performed. That is, the distance from the camera at each coordinate on the image of feature point 502 in the first image signal S11 acquired in this manner is converted into an image-side defocus amount using Equation 2 to acquire second distance information Idist2. As described above, the second acquisition means acquires the second distance information by calculating the optical flow of the target feature point from multiple images. <Calculation of correction information and correction principle>

[0069] The correction information generating means 330 generates and acquires a correction value Ic corresponding to the amount of change on the image side from the first distance information Idist1 and the second distance information Idist2. A method for generating the correction value Ic corresponding to the amount of change on the image side from the first distance information Idist1 and the second distance information Idist2 will be described below.

[0070] The first distance information Idist1 is derived based on the amount of parallax calculated by the first acquisition means. As shown in Figures 2B, 2C, and 2D, the amount of parallax corresponds to the distance between the centers of gravity of the first and second light beams, which is determined based on the positional relationship between the imaging surface of the imaging optical system and the image sensor. Therefore, if the shape of the imaging surface of the imaging optical system or the image sensor changes due to changes in the surrounding environment, such as temperature and humidity, the first distance information Idist1 will be affected and will differ from its value before and after the change. In other words, the first distance information Idist1 calculated by the first acquisition means is easily affected by changes in the optical system over time.

[0071] On the other hand, the second distance information Idist2 is calculated from an optical flow that associates a group of images acquired at a time interval much shorter than the time interval at which environmental changes such as temperature and humidity occur. Therefore, the first distance information used to calculate this distance information also uses relative changes over a short time interval as the camera movement amount, and does not contain any error over time due to environmental changes such as temperature and humidity. In other words, the second distance information Idist2 calculated by the second acquisition means is much less affected by changes in the optical system over time than the first distance information Idist1.

[0072] However, the second distance information Idist2 is information that depends on the coordinates at which the feature points are calculated, and is therefore distance information that is sparse with respect to the angle of view. Therefore, it is necessary to calculate a correction value Ic corresponding to the amount of image-side change caused by changes in the surrounding environment using the second distance information Idist2. By correcting the first distance information Idist1 and calculating the corrected distance information IdistC, it is possible to reduce the amount of error over time and obtain precise distance information for the angle of view. Furthermore, if precise distance information can be obtained, it becomes possible to measure the detailed shape of an object.

[0073] 6A is a flowchart showing the operation of the correction information generation process performed in step S330 by the correction information generation means 330. The central processing unit (CPU) in the distance measurement device 110 executes a computer program stored in memory to perform each process in FIG. In step S331, an image side change amount calculation process is performed to calculate an image side change amount using the first distance information Idist1 acquired by the first acquisition means 310 and the second distance information Idist2 acquired by the second acquisition means 320.

[0074] FIG. 6(B) is a diagram showing the defocus amount D1, which is the first distance information Idist1 acquired by the first acquisition unit 310, relative to the assumed image plane. FIG. 6(C) is a diagram showing the defocus amount D2, which is the second distance information Idist2 acquired by the second acquisition unit 320, relative to the assumed image plane. FIG. 6(C) shows a sparse data group corresponding to each coordinate on the image of feature point 502, since distance information is acquired at the pixel corresponding to feature point 502. FIG. 6(D) is a diagram showing the image-side defocus amount along I-I' in FIGS. 6(B) and 6(C).

[0075] The discontinuous line segment p1 in Figure 6(D) represents the defocus amount D1, which is the first distance information Idist1, and the point data p2 in Figure 6(D) represents the defocus amount D2, which is the second distance information Idist2. As described above, the first distance information Idist1 contains an error due to changes over time in the image pickup device 100. On the other hand, the second distance information Idist2 is hardly affected by changes over time in the image pickup device 100. Therefore, if the difference between the two is calculated, the difference corresponds to the amount of change on the image side that has been affected by changes over time.

[0076] Fig. 6(E) is a diagram showing the image-side change amount calculated by fitting using a dashed line. This shows the image-side change amount obtained by subtracting p2, which is the defocus amount D2, which is the second distance information Idist2, from p1, which is the defocus amount D1, which is the first distance information Idist1 in Fig. 6(D). In the example shown in Fig. 6, the p1-p2 difference data 602 can actually be acquired from three points on the data acquisition coordinate system 601 shown in Fig. 6(E), which correspond to the coordinates of the feature point 502 in Fig. 5(C).

[0077] In contrast, the amount of change on the image side that is affected by changes over time is a change in the focus position at each pixel, that is, it corresponds to a change in the amount of field curvature. Because the amount of field curvature itself has a continuous and smooth shape, the amount of change on the image side between angles of view, that is, between pixels, is continuous and smooth. Therefore, it is possible to interpolate between angles of view by fitting using polynomial approximation using each difference data 602 acquired on each data acquisition coordinate 601.

[0078] The image side change amount 603 calculated from the polynomial approximation in this way is shown by a dashed line in Figure 6(E). For the sake of explanation, the image side change amount calculated from the polynomial approximation is expressed one-dimensionally along the line segment I-I', but the actual image side change amount is two-dimensional data on the xy plane. Therefore, using the acquired difference data that is discrete with respect to the angle of view, surface fitting is performed on the xy plane using polynomial approximation to estimate the image side change amount. The approximate surface data, which is the calculated image side change amount, is set as the correction value Ic.

[0079] In this way, the distance values and defocus amounts acquired by each acquisition method may take discrete values depending on the distance of the subject being photographed. However, by converting the distance information acquired by each method into an image-side defocus amount and obtaining a difference value, these can be treated as continuous values, and their values and shapes can be estimated and acquired using various approximations.

[0080] When performing surface fitting by polynomial approximation using the difference data to estimate the amount of change on the image side, it is also preferable to remove outliers from the difference data based on the amount of change on the image side estimated from the design values of the imaging optical system, image sensor, etc. In other words, the amount of change on the image side can be predicted in advance by simulation using the design values of the apparatus and changes in the environmental temperature and humidity.

[0081] The predicted values are stored as initial values for correction in an information storage means as a lookup table or the like, and are compared with the acquired difference data to remove outliers by threshold judgment, thereby improving fitting accuracy. That is, the initial values include the amount of image-side change due to the environmental temperature, and the correction information generation means may obtain initial values for correction related to the imaging optical system or the image sensor from a lookup table and calculate the correction values.

[0082] 3B, the first distance information Idist1 is corrected using the generated correction value Ic to generate corrected distance information IdistC (correction step). Specifically, when the first acquisition means converts the defocus amount ΔL generated in the distance conversion process of step S314 into B in equation 2 (the distance from the principal point of the imaging optical system 120 to the image plane), the defocus amount ΔL' corrected using the following equation 5 is used.

[0083] (Number 5) ΔL'=ΔL-Ic The distance to the subject is calculated based on Equation 2 using the distance B from the principal point of the imaging optical system 120 to the image plane calculated using this corrected defocus amount ΔL′, thereby calculating corrected distance information IdistC.

[0084] As described above, according to this embodiment, it is possible to realize a distance measurement device that can acquire distance values with reduced influence of errors over time. It should be noted that the above correction process does not need to be performed every time distance is calculated. The time interval over which environmental changes occur over time is much longer than the time interval over which this device calculates distance, so the correction value Ic calculated here can continue to be used for a while after the next frame. The determination of the timing to update the correction information will be described later.

[0085] Furthermore, because changes over time occur as image plane changes due to the principles of occurrence, it is desirable to correct the distance information using the defocus amount on the image side rather than the subject distance on the object side, as in the embodiment of the present invention. Furthermore, in this embodiment, the correction information generating means compares (calculates the difference between) the first distance information and the second distance information as the image-side defocus amount. Therefore, it is possible to calculate correction information by approximation for the entire angle of view at the same scale regardless of whether the subject is close or far away, without being affected by the longitudinal magnification of the optical system. As a result, it is possible to secure a large number of data points to be used for approximation, which has the effect of enabling calculation of correction information with high precision.

[0086] If the characteristics of the imaging optical system 120 change over time due to changes in temperature and humidity or vibration, not only will the amount of change on the image side change, but the focal length and BL value will also change, which can be a cause of distance measurement errors. However, even if the focal length or BL value changes, as is clear from Equations 1 and 2, the amount of change is swamped in the amount of change in B, and therefore can be corrected by the correction process according to this embodiment.

[0087] In the above embodiment, a set of two images taken at two different times is used as the set of images used to calculate the optical flow by the second acquisition means, but any number of sets of images may be used. By increasing the number of sets, the number of combinations of images for calculating the optical flow increases, and the calculation accuracy of the camera movement amount and the second distance information can be improved.

[0088] Furthermore, the image groups at time t1 and time t2 may be acquired by a person carrying a camera and walking around. The various arithmetic processing steps do not have to be performed inside the imaging device such as a camera. For example, the data may be sent to an external server or terminal via a network, and the data may be executed using a CPU, GPU, or IC installed on the server or terminal.

[0089] Example 2 Next, a second embodiment of the present invention will be described with reference to FIG. <Configuration of distance measurement system>

[0090] FIG. 7A is a block diagram schematically illustrating the configuration of a distance measurement system according to a second embodiment of the present invention. In FIG. 7A, a distance measurement system 700 of the second embodiment includes an imaging device 710, a distance measurement device 720, and a second distance measurement device 730.

[0091] The second distance measurement device 730 receives the return light of the irradiated laser light to obtain second distance information indicating the distance to the subject. <Configuration of the second distance measurement device>

[0092] 7(B) is a diagram showing an example of the configuration of a second distance measurement device 730 of Example 2. The second distance measurement device 730 is configured with a light projection system consisting of a projection optical system 731, a laser 732, and a projection control unit 733, and a light receiving system consisting of a light receiving optical system 734, a detector 735, a distance measurement calculation unit 736, and an output unit 737.

[0093] The laser 732 includes a semiconductor laser diode that emits pulsed laser light. The light from the laser 732 is collected and irradiated by a projection optical system 731 having a scanning system. Although a semiconductor laser is mainly used as the laser light, there is no particular limitation. Laser light is a type of electromagnetic wave with good directivity and convergence, and there is no particular limitation on wavelength. From the viewpoint of safety, it is preferable to use laser light in the infrared wavelength band.

[0094] The emission of laser light from the laser 732 is controlled by a projection control unit 733. The projection control unit 733 generates, for example, a pulse signal for causing the laser 732 to emit light, and the drive signal is also input to a distance measurement calculation unit 736. The scanning optical system in the projection optical system 731 repeatedly scans the laser light emitted from the laser 732 in the horizontal direction at a predetermined cycle.

[0095] The laser light reflected from the object is incident on a detector 735 via a light receiving optical system 734. The detector 735 includes a photodiode or the like, and outputs an electrical signal having a voltage value corresponding to the light intensity of the reflected light.

[0096] The electrical signal output from the detector 735 is input to a distance measurement calculation unit 736, which measures the time from when a drive signal is output from the projection control unit 733 to the laser 732 until a light reception signal is generated. That is, the time difference between when the laser light is emitted and when the reflected light is received is measured, and the distance to the subject is calculated.

[0097] The calculated distance to the subject is output as second distance information via output unit 737. The scanning optical system in the projection optical system may use a polygon mirror, a galvanometer mirror, or the like. In this embodiment, for example, a laser scanner is used that has multiple polygon mirrors stacked vertically and that horizontally scans multiple laser beams arranged in a vertical line. By configuring and operating the scanner as described above, it is possible to obtain the distance to an object from which the irradiated electromagnetic waves are reflected.

[0098] The second distance measurement device 730 measures the subject distance based on the time it takes for emitted laser light to reach the subject and be detected by the detector. Therefore, even if the optical characteristics of the projection optical system 731 or the light receiving optical system 734 change due to changes in temperature and humidity, vibrations, etc., the effect on the time of flight of light is small. In other words, compared to the distance measurement device 720, the second distance measurement device 730 is less affected by changes (errors) in the distance measurement results over time caused by changes in temperature and humidity and vibrations.

[0099] The angle of view and coordinates of the second distance information acquired by the second distance measurement device 730 are associated with the angle of view and coordinates of the image acquired by the imaging device 710 by prior calibration. <Explanation of distance measurement device> The distance measurement device 720 of this embodiment will be described below. Fig. 8(A) is a functional block diagram showing the general configuration of the distance measurement device 720 of this embodiment. Blocks with the same reference numerals as those in Fig. 3(A) have the same functions.

[0100] In the distance measurement device 720, the second acquisition means 320 acquires second distance information Idist2 acquired by the second distance measurement device 730. That is, the second distance information is measured based on the time it takes for an electromagnetic wave to be emitted and the reflected electromagnetic wave to arrive. The correction information generation means 330 acquires a correction value Ic from the first distance information Idist1 and the second distance information Idist2. Furthermore, the correction means 340 corrects the first distance information Idist1 using the correction value Ic to generate and output corrected distance information IdistC.

[0101] 8(B) is a diagram illustrating the second distance information Idist2 acquired by the second distance measurement device 730. For the purpose of explanation, the coordinates 801 at which the second distance information was acquired by the second distance measurement device 730 are represented by diamonds and are superimposed on the captured image 800 acquired by the imaging device 710 at the same time that the distance information was acquired by the second distance measurement device 730. At each point on the coordinates 801, the second distance information Idist2 is acquired based on the time of flight of the irradiated laser light.

[0102] The correction information generating means 330 performs a process equivalent to the correction information generating process in step S330 described above using the first distance information Idist1 and the second distance information Idist2 from the same time to generate a correction value Ic. Furthermore, the correction information generating means 330 performs a process equivalent to the correction process in step S340 described above to calculate corrected distance information IdistC by correcting the first distance information Idist1.

[0103] In the second embodiment, the second acquisition means uses second distance information Idist2 acquired by a second distance measurement device 730 that employs an active ranging method using laser light. Therefore, the images acquired for distance measurement are only the image group that generates the first distance information Idist1, which reduces memory capacity.

[0104] The second distance information acquired by the second distance measurement device described in this embodiment is measured by LiDAR (Light detection and ranging) or ToF (Time of Flight), but is not limited thereto. It may also be measured by a millimeter wave radar using electromagnetic waves instead of laser light, or a similar active ranging device.

[0105] The reliability of the second distance information can be calculated based on the intensity of the received light when the laser light is reflected and returned, and only the coordinates of the distance measurement with high reliability may be used in the correction information generation process. Example 3 <Determining whether correction processing can be performed>

[0106] In the third embodiment, an execution determination is made before the start of the correction processing flow. That is, whether or not a satisfactory amount of correction can be obtained when the correction processing flow is executed is determined based on the captured scene, etc., before the correction processing flow is executed. This makes it possible to avoid execution of the flow in advance for scenes in which a satisfactory amount of correction cannot be obtained, thereby achieving effects such as reducing the amount of calculation, reducing memory required to store image data sets, and reducing heat generation. Furthermore, if it is determined that correction is not necessary, the correction processing flow is not executed, and uncorrected first distance information is output.

[0107] Fig. 9(A) is a flowchart illustrating the execution determination of the correction process according to the third embodiment, and Fig. 9(B) is a flowchart illustrating step S900 in Fig. 9(A). A central processing unit (CPU) in the distance measurement device 110 executes a computer program stored in memory to perform the processes in Figs. 9(A) and 9(B).

[0108] If it is determined in step S900 that the correction procedure flow is to be performed, the correction process described in the first embodiment is executed, and corrected distance information is output. If it is determined in step S900 that the correction procedure is not to be performed, the first acquisition process in step S310 described above is executed, and the obtained first distance information is output without correction.

[0109] The execution determination in step S900 will be described in detail with reference to FIG. 9(B). In step S910, the environmental change determination is made based on whether or not there is an environmental change around the imaging device (or distance measurement device) according to this embodiment. Specifically, temperature information and humidity information of the ambient environment are acquired at a predetermined cycle using a sensor, and if the value or the amount of change thereof exceeds a preset threshold, it is determined to perform (Yes), and if it does not exceed the threshold, it is determined not to perform (No).

[0110] Alternatively, the process may be based on the output of a vibration detection sensor such as a gyroscope to determine whether or not a vibration exceeding a predetermined threshold has been momentarily applied. If vibration is detected, the process may be determined to be "Yes," and if not, the process may be determined not to be performed (No). Alternatively, the accumulated value of vibration may be stored, and if it is determined that the accumulated value exceeds a predetermined level, the process may be determined to be "Yes," and if not, the process may be determined to be "No." In this way, the surrounding environment determined in step S910 includes at least one of weather, season, temperature, humidity, and vibration.

[0111] In step S920, it is determined whether or not the scene allows a satisfactory amount of correction to be obtained. That is, it is determined whether or not the second distance information is appropriate. If the second acquisition means is the SfM method described in the first embodiment, feature points are calculated from an image of the current surrounding environment captured by the imaging device in the same manner as in the optical flow calculation process in step S321 described above, and a judgment score is obtained based on the calculated feature points. Specifically, the judgment score S is calculated using the following equation 6 based on the number N of feature points within the angle of view, the distribution D of the feature points within the angle of view, and the reliability C of the calculated feature points.

[0112] (Number 6) S = α*N+β*D+γ*C α, β, and γ are coefficients that are set appropriately.

[0113] The distribution D of feature points within the field of view is expressed as D=δ*σ using the standard deviation σ based on a histogram of the coordinate values of the coordinates where the feature points were acquired. The reliability C of the feature points is calculated based on the strength of the feature amounts according to the method used to calculate the feature points, and the sum of the strength of the feature amounts of the feature points is taken as the reliability C of the feature points. If the determination score S calculated in this way exceeds an appropriately set threshold, it is determined that appropriate second distance information has been obtained, and the scene determination in step S920 is performed (Yes).

[0114] Alternatively, if the distance estimation device is installed in a mobile device, it may be determined whether the moving speed of the mobile device is below a predetermined value based on GPS data or, for example, the rotation speed of an axle. If the moving speed is below the predetermined value, the optical flow cannot be obtained sufficiently using the SfM method, and in that case, control may be exercised not to execute the second acquisition process using the optical flow calculation process in step S321. Alternatively, it may be determined whether the magnitude of the optical flow is below a predetermined threshold, and if it is, not to execute the second acquisition process using the optical flow calculation process in step S321.

[0115] When the second acquisition means is an active distance measuring device using electromagnetic waves such as laser light as described in the second embodiment, the number N of measurement points within the angle of view can be the number of points at which the reflected light intensity can be measured and distance information can be acquired, and the distribution D of the measurement points within the angle of view can be evaluated in the same manner as described above. Furthermore, the reliability C of the calculated measurement points can be calculated as the sum of the quantified magnitudes of the reflected light intensity at each point, and a judgment score S can be calculated in the same manner using Equation 6, and the execution judgment can be performed based on a threshold value. In this way, the scene judgment in step S920 or the judgment of the appropriateness of the second distance information can be performed based on any one of the number, distribution, reliability, moving speed, and magnitude of optical flow of the measurement points used in the second acquisition process.

[0116] If either the environmental change determination in step S910 or the scene determination in step S920 determines not to perform correction (No), an interval determination is made in step S930. The purpose of the interval determination in step S930 is to measure the time elapsed since the previous correction and to prevent the time interval between corrections from being too long. An interval threshold th is set appropriately, and if the time ti elapsed since the previous correction exceeds th, the interval determination in step S930 determines that correction should be performed (Yes).

[0117] The interval threshold th is set according to the surrounding environment. The surrounding environment includes driving conditions such as the season, weather, and road surface. Under environmental conditions where changes over time are likely to occur (e.g., high temperature, high humidity, etc.), the value of the interval threshold th may be set small so that correction processing is performed more frequently. Furthermore, rather than always performing correction when the interval determination is Yes, the thresholds in steps S910 and S920 may be lowered by a predetermined percentage to increase the likelihood that correction will be performed.

[0118] The timing for determining when to perform the correction process is set to coincide with the start-up of the distance measurement device or the moving device. Immediately after the device is started up, various electrical components such as the image sensor and circuit boards begin to generate heat, and the temperature inside the device begins to rise rapidly. Therefore, it is possible to set in advance the time when it is expected that the device will be able to warm up, perform the correction process once after that time has passed, and then determine whether to perform the correction process from the next time onwards.

[0119] It is also preferable to determine the timing of performing the correction process depending on the movement status of the device. In the case of an on-board camera, the pitch fluctuation of the vehicle body is large when traveling on a rough road, and there is a high possibility that a large error will occur in the calculation of the camera attitude used by the second acquisition means. In this way, depending on the movement status (scene) of the device, it may be determined not to perform the correction process under circumstances in which a large error will be superimposed on the calculation of the correction value.

[0120] It is also possible to set the system to determine whether correction processing should be performed, and to issue an alert (warning) if the determination that correction processing should be performed continues at short intervals. The environmental temperature and humidity, which are factors that cause changes over time, have relatively long time constants compared to the operation of the device. Furthermore, shocks such as vibrations that cause changes over time rarely occur frequently.

[0121] Therefore, if the system continues to judge that repair or maintenance is required at short intervals, there is a high possibility that the problem is due to other factors, such as a malfunction, in addition to changes over time. Therefore, it is desirable to set up the system to issue an alert to prompt the system to carry out repair or maintenance. <Estimating image-side change amount by dividing the area> Example 4

[0122] In the correction process according to the fourth embodiment, when estimating the amount of change on the image side from the difference data between the defocus amounts obtained by different methods, the angle of view (screen) is divided into regions to estimate the amount of change on the image side, and the estimated amounts are integrated to calculate the amount of correction.

[0123] As mentioned above, the amount of image-side change that accompanies changes in the device configuration over time has a continuous and smooth shape, but the shape of the image-side change may be asymmetric with respect to the xy plane or may include localized irregularities due to the influence of design values and assembly errors of the optical system, etc. When performing surface estimation using polynomial approximation for such image-side change, it may not be possible to fully express the shape even if higher-order terms are used.

[0124] In such a case, the difference between the actual image side change amount and the estimated plane using the approximation formula becomes the residual error. This residual error is included in the correction value Ic, and the corrected distance information IdistC, which is the correction result, also contains the residual error. Therefore, in the fourth embodiment, in order to accurately express the image side change amount using the estimated plane using the approximation formula, the screen is divided into multiple regions and an approximate calculation of the correction value is performed for each region. Then, the results of the approximate calculation for the multiple regions are combined to reduce distance measurement errors.

[0125] FIG. 10(A) is a flowchart showing the overall flow of performing correction value processing involving area division according to the fourth embodiment, FIG. 10(B) is a diagram showing the state in which the area for calculating the defocus amount has been divided, and FIG. 10(C) is a flowchart showing the details of step S1030. A central processing unit (CPU) in the distance measurement device 110 executes a computer program stored in memory to perform the processes shown in FIGS. 10(A) and 10(C). The flowchart of Figure 10(A) of the fourth embodiment is the operation flow of the distance measurement device 110 shown in Figure 3(B) in which the correction information generation process of step S330 is replaced with correction information generation process involving area division of step S1030.

[0126] 10(B), the screen area for calculating the defocus amount is divided into three in the x direction, for example. That is, it is divided into area 1001 and area 1002 with coordinate x1 as the boundary, and further divided into area 1003 with coordinate x2 as the boundary. The number of divided areas can be set to any value, and the division direction can be either horizontal or vertical, or any shape can be formed by division in both horizontal and vertical directions, or division independent of the horizontal and vertical axes, and the sizes of the areas can also be different.

[0127] After the first distance information Idist1 and the second distance information Idist2 are acquired by the first acquisition process in step S310 and the second acquisition process in step S320, the process proceeds to step S1030. In step S1030, the same process as the correction information generation process in step S330 described above is performed for each region.

[0128] The correction information generation process involving region division in step S1030 will be described in detail with reference to FIG. In step S1031, the image is divided into regions, and the first distance information Idist1 and second distance information Idist2 corresponding to each region are passed to step S331 for image-side change amount calculation processing. As described above, the image-side change amount calculation processing in step S331 and the correction information calculation processing in step S332 are performed for each region to calculate the correction value Ic for each region (correction information generation step).

[0129] In the integration process of step S1032, the boundary portions of the correction values Ic for each region are smoothly joined, and the correction value Ic for the entire angle of view is calculated. Specifically, as shown in FIG. 10B, a boundary section w is set for boundary coordinate x1. The correction value Ic1 calculated for region 1001 and the correction value Ic2 calculated for region 1002 included in the boundary section w are averaged: Ic12 = (Ic1 + Ic2) ÷ 2. A similar process is performed for boundary coordinate x2, and the correction value Ic after the averaging process is used for the correction value for boundary section w, while the correction value Ic calculated for other regions is used, ultimately creating a provisional correction value Ictmp corresponding to all pixels.

[0130] The provisional correction value Ictmp is simply a spliced image and is not smooth, so it is smoothed using a known method of combining scaling and interpolation. In this way, the correction value for each region is calculated and integrated to calculate the correction value Ic corresponding to the entire angle of view.

[0131] By dividing the image side change amount into regions and estimating the image side change amount, even if the actual image side change amount has local irregularities or an asymmetric shape, low-order approximation is possible within each divided region, and it is possible to express it using polynomial approximation or various functions. As a result, the correction value Ic after integration has a shape that is close to the actual image side change amount, and it is possible to reduce the remaining error.

[0132] 10C, the correction value Ic calculated by executing the integration process in step S1032 may be fed back to the process in step S1031 again. Then, loop processing may be performed in which the calculated correction value Ic is used as the initial value instead of the difference between the defocus amounts Idist1 and Idist2.

[0133] In this case, by increasing the number of region divisions N each time the loop is run and estimating the local shape from the global shape of the image side variation, it is possible to prevent the approximation of the surface estimation from falling into a local minimum solution or diverging. Next, the feedback loop processing will be described. <Correction value is fed back and correction process is looped>

[0134] In the correction process according to the fourth embodiment, a process for improving accuracy by feeding back calculated correction information and performing correction process or correction value calculation process again will be described. Fig. 11(A) is a flowchart of the entire process of looping correction by feeding back the calculated correction value, and Fig. 11(B) is a flowchart of a part of Fig. 11(A). The central processing unit (CPU) in the distance measurement device 110 executes a computer program stored in memory to perform each process of Fig. 10(A) and (B).

[0135] The presence or absence of the correction value Ic fed back is determined in the correction information presence determination in step S1101. If there is no feedback correction value Ic, such as when the correction process is performed for the first time, the process performs steps S310 to S340 in Fig. 10A assuming that there is no initial data for the correction value Ic (or a dummy file with a value of 0).

[0136] If it is determined in step S1101 that a feedback correction value Ic is present, the feedback correction value Ic is used. Then, in step S310, a first acquisition unit calculates first distance information Idist1 using the defocus amount ΔL′ corrected using equation 5. Then, step S320 is performed, and the new correction value calculated using the correction information in step S330 is used as the correction value Ic. Next, in step S340, corrected distance information is calculated using the updated correction value Ic. In step S1102, it is determined whether the correction value Ic has been sufficiently updated.

[0137] If it is determined that the correction value Ic has not been updated sufficiently (Yes), the current correction value Ic is fed back, and the entire correction process flow is executed again from the process in step S1101. If it is determined in the update determination in step S1102 that the correction value Ic has not been updated sufficiently (No), the current corrected distance information is output. In the update determination in step S1102, if the number of updates has not reached a preset number, the determination may be Yes, and if the number of updates has reached a preset number, the determination may be No.

[0138] That is, it is possible to predict to some extent the amount of change on the image side based on the design values of the apparatus and changes in ambient temperature and humidity through prior simulation studies, etc. Therefore, since it is possible to predict how many feedback loops will be required for the correction value to converge to the actual amount of change on the image side based on this prediction, it is also possible to set a threshold value for the number of updates.

[0139] Alternatively, an update determination may be made in step S1102 based on the corrected distance information IdistC output by the correction process in step S340. Alternatively, a recognition process may be performed on the acquired image to determine whether areas classified as the same subject have the same distance value, whether an area recognized as a road surface has a distance distribution that can be expressed as a surface, etc. Furthermore, the corrected distance information IdistC may be compared with the second distance information, and the difference value may be subjected to a threshold determination.

[0140] Alternatively, in the update determination in step S1102, the update determination may be performed based on the calculated correction value Ic. The calculated correction value Ic may be compared with the difference in the image-side defocus amounts Idist1 and Idist2 that were the original data for surface fitting, and the correlation between the two may be determined using a threshold value.

[0141] The correction information generation process in step S330 in Fig. 11A may be replaced with the correction information generation process in step S1130 as shown in the flowchart in Fig. 11B. Step S1130 is a step in which the calculated correction value is fed back and correction value calculation is looped.

[0142] In Fig. 11(B), the determination of the presence or absence of correction information in step S1131 is the same as the processing in step S1101 in Fig. 11(A). If there is no feedback correction value Ic, the processing of steps S310 to S340 in Fig. 10(A) is performed, and if there is a feedback correction value Ic, the processing of steps S331 and S332 in Fig. 11(B) is performed using the feedback correction value Ic. In the update determination in step S1132, it is determined whether the update of the correction value Ic is insufficient. In this case, specifically, the determination may be made based on the amount of update of the coefficients of the polynomial function during fitting.

[0143] If it is determined in step S1132 that the update of the correction value Ic is insufficient (Yes), the current correction value Ic is fed back, and the correction value calculation process is performed again from the process of S1131. If it is determined in the update determination of step S1132 that the update of the correction value Ic is sufficient (No), the current corrected correction value Ic is output. Example 5 <In-vehicle devices and vehicle control>

[0144] (Basic configuration, overall configuration) Fig. 12(A) is a schematic diagram showing the overall configuration of a mobile device according to Example 5, and Fig. 12(B) is a block diagram of Example 5. The mobile device is not limited to a car, but may be a train, an airplane, a ship, a small mobility vehicle, various robots such as an AGV (Automatic Guided Vehicle), a drone, or the like.

[0145] 12, a vehicle 1100 has an imaging device 1110, a millimeter wave radar device 1120, a LiDAR device 1130 (LiDAR: Light Detection and Ranging), and a vehicle information measuring instrument 1140. It also has a route generation ECU 1150 (ECU: Electronic Control Unit) and a vehicle control ECU 1160. As an alternative to the route generation ECU 1150 and the vehicle control ECU 1160, they may be configured by a central processing unit (CPU) as a computer, a memory that stores a processing program, and the like.

[0146] The imaging device 1110 has a first acquisition means 310, a second acquisition means 320, a correction information generation means 330, a correction means 340, etc. that perform the same operations as those described in the first to fourth embodiments. Furthermore, the control means, such as the route generation ECU 1150 and the vehicle control ECU 1160, have a control process that issues a warning or controls the movement operation (direction, movement speed, etc.) of the vehicle 1100 as a moving device, based on the first distance information corrected by the correction means.

[0147] The imaging device 1110 captures an image of the surrounding environment including the road on which the vehicle 1100 is traveling, generates image information representing the captured image, and distance image information having information representing the distance to the subject for each pixel, and outputs the image information to the route generation ECU 1150. The imaging device 1110 is disposed near the top edge of the windshield of the vehicle 1100 as shown in Fig. 12, and captures an image of an area in a predetermined angular range (hereinafter referred to as the imaging angle of view) facing forward of the vehicle 1100.

[0148] The information representing the distance to the subject may be information that can be converted into the distance from the image capture device 1110 to the subject within the imaging angle of view, and may be information that can be converted using a predetermined lookup table or a predetermined conversion coefficient and conversion formula. For example, the distance value may be assigned to a predetermined integer value and output to the route generation ECU 1150. Alternatively, information that can be converted into an optically conjugate distance value (the distance from the image capture element to the conjugate point (so-called defocus amount) or the distance from the optical system to the conjugate point (the distance from the image-side principal point to the conjugate point)) that can be converted into the distance to the subject may be output to the route generation ECU 1150.

[0149] As the vehicle information measuring instruments 1140, the vehicle 1100 is equipped with a traveling speed measuring instrument 1141, a steering angle measuring instrument 1142, and an angular velocity measuring instrument 1143. The traveling speed measuring instrument 1141 is a measuring instrument that detects the traveling speed of the vehicle 1100. The steering angle measuring instrument 1142 is a measuring instrument that detects the steering angle of the vehicle 1100. The angular velocity measuring instrument 1143 is a measuring instrument that detects the angular velocity of the vehicle 1100 in the turning direction.

[0150] The route generation ECU 1150 is configured using logic circuits. The route generation ECU 1150 receives as input measurement signals from vehicle information measuring instruments provided in the vehicle 1100, image information and distance image information from the imaging device 1110, distance information from the radar device 1120, and distance information from the LiDAR device 1130. Based on this information, the route generation ECU 1150 generates target route information relating to at least either a target travel trajectory or a target travel speed of the vehicle 1100, and outputs the information to the vehicle control ECU 1160 sequentially.

[0151] In addition, if the vehicle 1100 is equipped with an HMI 1170 (Human Machine Interface) that displays images or issues audio notifications or warnings to the driver 1101, the target route information generated by the route generation ECU 1150 may be notified or warned to the driver 1101 via the HMI 1170.

[0152] By applying the distance correction according to this embodiment to the imaging device 1110, the accuracy of the distance information output is improved, the accuracy of the target route information output from the route generation ECU 1150 is improved, and safer vehicle driving control is achieved. The second acquisition means in this embodiment may be an SfM method based on an image obtained from an imaging device 1110, a radar device 1120, a LiDAR device 1130, or SfM using a vehicle information measuring instrument 1140 or an imaging device 1110.

[0153] While the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention. As mentioned above, in the above embodiments, examples of time-dependent errors have been described, but the errors may also be manufacturing errors, etc. Note that a computer program that realizes part or all of the control in this embodiment and the functions of the above-described embodiment may be supplied to a distance measurement device, a moving device, etc. via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the distance measurement device, moving device, etc. may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0154] 100 Imaging device 101 Image sensor 170 Information storage means 310 First Acquisition Method 320 Secondary Acquisition Method 330 Correction information generation means 340 Corrective Measures

Claims

1. a first acquisition means for acquiring, via an imaging optical system, first distance information including an error due to a change over time in the distance between the imaging unit and the subject; a second acquiring means for acquiring second distance information having an error smaller than that of the first distance information; a generating means for calculating a correction value for correcting the first defocus amount based on a difference between a first defocus amount corresponding to a deviation in the optical axis direction between a sensor surface used to acquire the first distance information and an imaging surface, and a second defocus amount related to the second distance information; a calculation unit that calculates a distance between the imaging unit and the subject using the first defocus amount corrected using the correction value, the second acquisition means acquires the second distance information by calculating an optical flow of feature points of the target from a plurality of images. A distance measurement device characterized by:

2. A first acquisition means for acquiring first distance information including an error due to a change over time in the distance between the imaging unit and the subject via an imaging optical system; a second acquiring means for acquiring second distance information having an error smaller than that of the first distance information; a generating means for calculating a correction value for correcting the first defocus amount based on a difference between a first defocus amount corresponding to a deviation in the optical axis direction between a sensor surface used to acquire the first distance information and an imaging surface, and a second defocus amount related to the second distance information; a calculation unit that calculates a distance between the imaging unit and the subject using the first defocus amount corrected using the correction value, the second distance information is measured based on the arrival time of the reflected electromagnetic wave after irradiating the object with the electromagnetic wave; A distance measurement device characterized by:

3. 3. The distance measurement device according to claim 1, wherein the first acquisition means acquires the first distance information based on a plurality of signals having parallax output from an imaging element.

4. 4. The distance measurement device according to claim 3, wherein the first acquisition means acquires the first distance information based on a plurality of signals having parallax output from light beams that have passed through different pupil regions of a monocular optical system.

5. 5. The distance measurement device according to claim 4, wherein the first acquisition means acquires the first distance information based on a plurality of signals having parallax output from a single imaging element.

6. 2. The distance measurement device according to claim 1, wherein the second acquisition means acquires the second distance information by SfM.

7. The distance measurement device according to claim 2 , wherein the second distance information is measured using LiDAR, ToF, or millimeter wave radar.

8. 3. The distance measurement device according to claim 1, wherein the error includes a time-dependent error in the amount of curvature of field.

9. The distance measurement device described in claim 1 or 2, characterized in that the generation means determines at least one of the surrounding environment, a predetermined elapsed time, and the appropriateness of the second distance information, and controls whether or not to correct the error in the first distance information based on the second distance information.

10. 10. The distance measurement device according to claim 9, wherein the surrounding environment includes at least one of weather, season, temperature, humidity, and vibration.

11. The distance measurement device according to claim 9, characterized in that the suitability of the second distance information is determined based on one of the number, distribution, reliability, moving speed, and magnitude of optical flow of measurement points used in the second acquisition means.

12. 3. The distance measurement device according to claim 1, wherein the generating unit calculates the correction value based on a difference between the first distance information and the second distance information.

13. 13. The distance measurement device according to claim 12, wherein the generating means performs approximate calculation of the correction value for each of a plurality of regions within a screen, and combines the results of the approximate calculation for the plurality of regions.

14. 13. The distance measurement device according to claim 12, wherein the generating means performs loop processing by feeding back the correction value.

15. 4. The distance measurement device according to claim 3, wherein the generation unit calculates the correction value using an initial value for correction related to the imaging optical system or the image sensor.

16. 16. The distance measurement device according to claim 15, wherein the generating means acquires the initial value for the correction from a lookup table.

17. 16. The distance measurement device according to claim 15, wherein the initial value includes an image-side variation amount based on an environmental temperature.

18. 4. The distance measurement device according to claim 3, wherein the second acquisition means acquires the second distance information based on image signals of a plurality of screens from the imaging element.

19. A mobile device equipped with the distance measurement device according to any one of claims 1 to 18, A mobile device comprising a control means for issuing a warning or controlling a moving operation of the mobile device based on the distance between the imaging unit and the subject calculated by the calculation means.

20. a first acquisition step of acquiring first distance information, the first distance information including an error due to a change over time in the distance between the imaging unit and the subject, via an imaging optical system; a second acquiring step of acquiring second distance information having an error smaller than that of the first distance information; a generation step of calculating a correction value for correcting the first defocus amount based on a difference between a first defocus amount corresponding to a deviation in the optical axis direction between a sensor surface used to acquire the first distance information and an imaging surface, and a second defocus amount related to the second distance information; a calculation step of calculating a distance between the imaging unit and the subject using the first defocus amount corrected using the correction value, the second obtaining step obtains the second distance information by calculating an optical flow of feature points of the target from a plurality of images; A distance measurement method characterized by:

21. a first acquisition step of acquiring first distance information, the first distance information including an error due to a change over time in the distance between the imaging unit and the subject, via an imaging optical system; a second acquiring step of acquiring second distance information having an error smaller than that of the first distance information; a generation step of calculating a correction value for correcting the first defocus amount based on a difference between a first defocus amount corresponding to a deviation in the optical axis direction between a sensor surface used to acquire the first distance information and an imaging surface, and a second defocus amount related to the second distance information; a calculation step of calculating a distance between the imaging unit and the subject using the first defocus amount corrected using the correction value; a control step of issuing a warning or controlling a movement operation of the moving device based on the distance between the imaging unit and the subject calculated in the calculation step, the second obtaining step obtains the second distance information by calculating an optical flow of feature points of the target from a plurality of images; A method for controlling a mobile device.

22. A computer program for controlling each means of the distance measurement device according to any one of claims 1 to 18 or the moving device according to claim 19 by a computer.

Images