Distance measurement device, moving body, distance measurement method, and computer program

Abstract

To provide a distance measurement device capable of stably acquiring a highly accurate correction value.SOLUTION: A distance measurement device comprises: first acquisition means for acquiring first distance information; second acquisition means for acquiring second distance information; first correction value generation means for calculating a first correction value for correcting the first distance information on the basis of the second distance information; reliability score determination means for calculating a reliability score indicating the reliability of the first correction value and determining the reliability of the first correction value on the basis of the reliability score; second correction value generation means for generating a second correction value from the first correction value that has a reliability score equal to or higher than a predetermined threshold and that has been acquired within a predetermined effective time interval; and correction means for correcting the first distance information using the second correction value.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a distance measurement device, a moving object, a distance measurement method, a computer program, and the like. [Background technology]

[0002] A technology is known in which a camera capable of acquiring depth information, such as a stereo ranging system or an image plane phase difference ranging system, is mounted on a moving object such as an automobile to measure the distance to a subject in front of the vehicle and control the vehicle based on the distance information.

[0003] However, in cameras mounted on vehicles, errors in distance measurements occur due to deformation of the camera over time, and errors in distance measurements occur due to the influence of the environment around the camera, such as the ambient temperature or the temperature inside the vehicle.

[0004] Patent Document 1 proposes a method of correcting errors in the distance measurements of a distance measuring camera in a time series manner by using distance information obtained from a second distance measuring means separate from the distance measuring camera. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2022-154179 Summary of the Invention [Problem to be solved by the invention]

[0006] However, when using a second ranging means different from the ranging camera to correct the distance value of the ranging camera, the second ranging means is required to have higher distance accuracy than the ranging camera, but when installed in a vehicle, it is necessary to measure distance in a variety of scenes. Some ranging methods are not good at certain scenes, and in some scenes the accuracy may be lower than that of the ranging camera being corrected.

[0007] To solve the above problem, Patent Document 1 sets a reliability for the distance measurement value obtained by the second distance measurement means, and does not perform correction if the reliability is low. In this case, it is difficult to perfectly match the reliability of the distance measurement value with the distance accuracy, and even if the reliability of the distance measurement value is high, the actual distance accuracy may be low. Furthermore, if the threshold for determining the reliability is set high to avoid this, the probability of performing correction may be significantly reduced.

[0008] In view of the above-mentioned problems, an object of the present invention is to provide a distance measurement device that can stably acquire highly accurate correction values. [Means for solving the problem]

[0009] The present invention provides a distance measurement device, a first acquisition means for acquiring first distance information; a second acquisition means for acquiring second distance information; a first correction value generating means for calculating a first correction value for correcting the first distance information based on the second distance information; a reliability score determination means for calculating a reliability score indicating the reliability of the first correction value and determining the reliability of the first correction value based on the reliability score; a second correction value generating means for generating a second correction value from the first correction value when the reliability score is equal to or greater than a predetermined threshold and is acquired within a predetermined valid time interval; a correction means for correcting the first distance information using the second correction value; The present invention is characterized by having the following. [Effects of the Invention]

[0010] According to the present invention, it is possible to realize a distance measurement device that can stably acquire highly accurate correction values. [Brief explanation of the drawings]

[0011] [Figure 1]1A is a diagram illustrating a schematic configuration example of an imaging device according to a first embodiment of the present invention, FIG. 1B is an xy cross-sectional view of the imaging element 101 of FIG. 1A, and FIG. 1C is a diagram illustrating a schematic I-I' cross-section of the pixel group 150 of FIG. 1B. [Figure 2] (A) is a diagram showing the relationship between the exit pupil of the imaging optical system 120 of embodiment 1 and the light receiving part of the image sensor 101, and (B) to (D) are diagrams for explaining the relationship between the amount of parallax and the amount of defocus due to the image sensor 101 and the imaging optical system 120 of embodiment 1. [Figure 3] (A) is a functional block diagram showing an example of the configuration of the distance measurement device 110 of embodiment 1, (B) is a flowchart showing an example of the operation of the distance measurement device 110, and (C) is a flowchart showing an example of the operation of part of Figure 3(B). [Figure 4] (A) is a diagram showing an example of the two-dimensional distribution of the amount of field curvature of the imaging optical system 120 within the effective pixel range of the image sensor 101, (B) is a diagram showing an example of the amount of field curvature along I-I' in Figure 4(A), and (C) is a diagram showing an example of the amount of change on the image side along I-I' in Figure 4(A). [Figure 5] 1A is a flowchart illustrating an example of a second acquisition step in which the CPU of the distance measurement device 110 acquires second distance information using the second acquisition means 320 in step S320. FIG. 1B is a diagram showing, with stars, feature points 501 calculated for the first image signal S21 at time t2. FIG. 1C is a diagram showing, with stars, feature points 502 calculated for the first image signal S11 at time t1. FIG. 1D is a diagram showing the calculated optical flow. [Figure 6]6A is a flowchart illustrating an example of the operation of the first correction value generation process performed by the first correction value generation means 330 in step S330. FIG. 6B is a diagram illustrating an example of the defocus amount d1, which is the first distance information Idist1 acquired by the first acquisition means 310, relative to the assumed image plane. FIG. 6C is a diagram illustrating an example of the defocus amount d2, which is the second distance information Idist2 acquired by the second acquisition means 320, relative to the assumed image plane. FIG. 6D is a diagram illustrating an example of the image-side defocus amount along line I-I' in FIGS. 6B and 6C, and FIG. 6E is a diagram illustrating an example of the image-side change amount calculated by fitting, using a dashed line. [Figure 7] 10A is a flowchart illustrating an example of a process for determining the reliability score of a first correction value performed by the reliability score determination means 340 in step S340, and FIG. 10B is a flowchart illustrating an example of a process for generating a second correction value in step S350. FIG. 10C is a diagram illustrating an example of a method for generating a second correction value. [Figure 8] FIG. 10A is a flowchart showing an example of a flow for determining a reliability score in embodiment 2, FIG. 10B is a flowchart showing an example of a flow for determining an outlier, and FIG. 10C is a diagram for specifically explaining an example of a flow for determining an outlier. [Figure 9] 10A is a flowchart illustrating an example of the operation of the distance measurement device 110 according to the third embodiment, and FIG. 10B is a diagram illustrating the effects of the third embodiment. [Figure 10] FIG. 10 is a diagram schematically illustrating an example of the configuration of a distance measurement device according to a fourth embodiment. [Figure 11] 10A is a functional block diagram showing an example of the configuration of a distance measurement device 1010 according to a fourth embodiment, and FIG. 10B is a flowchart showing an example of the operation of the distance measurement device 1010. [Figure 12] 10A is a flowchart showing an example of the operation of a second acquisition process according to the fourth embodiment, and FIG. 10B is a flowchart showing an example of a process for calculating an optical flow. [Figure 13]13A is a flowchart showing an example of the operation of the second correction value generation process according to the first embodiment, and a diagram for explaining the same. (B) is a diagram showing an example of the amount of parallax D1, which is the first distance information Idist1 acquired by the first acquisition unit 1110, and (C) is a diagram showing an example of the amount of parallax D2, which is the second distance information Idist2 acquired by the second acquisition unit 1120. (D) is a diagram showing an example of the amount of parallax along I-I' in FIGS. 13B and 13C. (E) is a diagram showing an example of data 1003 interpolated between angles of view. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, preferred embodiments of the present invention will be described by way of example with reference to the accompanying drawings. In each drawing, the same members or elements are designated by the same reference numerals, and duplicated descriptions will be omitted or simplified.

[0013] In the following embodiments, an example of an imaging device mounted on a vehicle as a moving body will be described. However, the moving body is not limited to a vehicle but includes any movable device. That is, the moving body includes, for example, an automated guided vehicle (AGV), an autonomous mobile robot (AMR), a cleaning robot, a drone, etc.

[0014] The imaging device may also be an electronic device with an imaging function, such as a digital still camera, a digital movie camera, a smartphone with a camera, a tablet computer with a camera, a network camera, a drone camera, or a camera mounted on a robot.

[0015] <Embodiment 1> In the first embodiment, distance values ​​acquired by different methods are converted into image plane defocus amounts and compared to calculate the amount of change in the amount of field curvature over time as the first correction value. In addition, a reliability score is determined within the valid time interval, and the first correction value determined to be reliable is used to create the second correction value, thereby increasing the reliability of the correction value.

[0016] Fig. 1(A) is a diagram schematically illustrating an example of the configuration of an imaging device according to embodiment 1 of the present invention. In Fig. 1(A), the imaging device 100 includes an imaging optical system 120, an imaging element 101, a distance measurement device 110, and information storage means 170. The distance measurement device 110 can be configured using a logic circuit or the like.

[0017] Alternatively, the distance measurement device 110 may have a central processing unit (CPU) and a memory for storing a processing program, and the CPU as a computer may execute the computer program stored in the memory.

[0018] The imaging optical system 120 is a photographing lens of the imaging device 100 or the like, and forms an image of a subject on the light receiving surface of the image sensor 101. The imaging optical system 120 is composed of a plurality of lens groups (not shown), and has an exit pupil 200 at a predetermined distance from the image sensor 101. In this specification, the z-axis is parallel to the optical axis 140 of the imaging optical system 120. Furthermore, the x-axis and y-axis are perpendicular to each other and to the optical axis.

[0019] The image sensor 101 in this embodiment is an image sensor that is configured with a CMOS, a CCD, or the like, and has a distance measurement function using an image-sensing-plane phase difference distance measurement method. An object image formed on the image sensor 101 via the imaging optical system 120 is photoelectrically converted by the image sensor 101, and an image signal based on the object image is generated.

[0020] Furthermore, a color image can be generated by subjecting the acquired image signal to development processing by the image generation unit, and the generated color image can be stored in an image storage unit (not shown).

[0021] Fig. 1(B) is an xy cross-sectional view of the image sensor 101 of Fig. 1(A). The image sensor 101 is configured by arranging multiple pixel groups 150 in a 2-row x 2-column arrangement. In the pixel group 150, green pixels 150G1 and 150G2 are arranged diagonally, and a red pixel 150R and a blue pixel 150B are arranged as the other two pixels.

[0022] Fig. 1(C) is a diagram schematically illustrating the II' cross section of the pixel group 150 in Fig. 1(B). Each pixel is composed of a light receiving layer 182 and a light guide layer 181. The light receiving layer 182 has two photoelectric conversion units (a first photoelectric conversion unit 161 and a second photoelectric conversion unit 162) arranged therein for photoelectrically converting received light.

[0023] The light guide layer 181 is provided with microlenses 183 for efficiently guiding the light beam incident on the pixel to the photoelectric conversion section, color filters (not shown) that pass light in a predetermined wavelength band, wiring (not shown) for reading images and driving pixels, and the like.

[0024] In the examples shown in Figures 1(B) and 1(C), an example of a photoelectric conversion unit divided into two in one pupil division direction (x-axis direction) is shown, but a photoelectric conversion unit divided into two pupil division directions (x-axis direction and y-axis direction) may also be provided, and the pupil division direction and number of divisions are arbitrary.

[0025] Furthermore, the color combinations received by each pixel are not limited to the example shown in FIG. 1B. Some pixels may be able to detect IR (Infra Red), white, etc., and filters with desired spectral characteristics may be arranged for each pixel in a desired arrangement.

[0026] The light receiving layer 182 has a photoelectric conversion section formed using a semiconductor or the like that is sensitive to the wavelength band to be detected, and if the wavelength band to be detected is in the visible range, a material such as Si is used, but it is not limited to this and can be formed from any material depending on the target wavelength band.

[0027] Next, the principle of distance measurement by the split-pupil image pickup plane phase difference ranging method using the image pickup element 101 of this embodiment will be described.

[0028] 2A is a diagram showing the relationship between the exit pupil of the imaging optical system 120 of the first embodiment and the light receiving portion of the image sensor 101. Only the exit pupil 200 of the imaging optical system 120 and the green pixel 150G1 are shown as a representative example of pixels arranged in the image sensor 101. The exit pupil 200 and the light receiving layer 182 are optically conjugate with each other due to the microlens 183 in the green pixel 150G1 shown in FIG.

[0029] 2A, the light beam that has passed through the first pupil region 210 in the exit pupil 200 is incident on the first photoelectric conversion unit 161. On the other hand, the light beam that has passed through the second pupil region 220 is incident on the second photoelectric conversion unit 162.

[0030] The signals from the plurality of first photoelectric conversion units 161 provided in each pixel photoelectrically convert the received light beams to generate first image signals, and similarly, the signals from the plurality of second photoelectric conversion units 162 provided in each pixel photoelectrically convert the received light beams to generate second image signals.

[0031] From the first image signal, the intensity distribution of the image formed on the image sensor 101 by the light beam that has mainly passed through the first pupil region 210 can be obtained, and from the second image signal, the intensity distribution of the image formed on the image sensor 101 by the light beam that has mainly passed through the second pupil region 220 can be obtained.

[0032] The amount of parallax between the first image signal and the second image signal corresponds to the amount of defocus. Figures 2B to 2D are diagrams for explaining the relationship between the amount of parallax and the amount of defocus due to the image sensor 101 and the imaging optical system 120 of the first embodiment. In the diagrams, 211 denotes a first light beam passing through the first pupil region 210, and 221 denotes a light beam passing through the second pupil region 220.

[0033] 2(B) is a diagram showing the state during focusing, in which the first light beam 211 and the second light beam 221 converge on the image sensor 101. At this time, the relative positional deviation between the first image signal formed by the first light beam 211 and the second image signal formed by the second light beam 221 is zero.

[0034] 2C shows a state where the image side is defocused in the negative direction of the z axis. In this state, the relative positional shift between the first image signal formed by the first light beam and the second image signal formed by the second light beam is not zero but has a negative value.

[0035] 2(D) shows a state where the image side is defocused in the positive direction of the z-axis. In this state, the relative positional shift between the first image signal formed by the first light beam and the second image signal formed by the second light beam is not 0 but has a positive value.

[0036] 2(C) and (D), it can be seen that the direction of the positional shift changes depending on whether the defocus amount is positive or negative. It can also be seen that the amount of parallax changes depending on the amount of defocus. Therefore, the amount of parallax between the first image signal and the second image signal can be detected using a region-based matching method, as described below, and the detected amount of parallax can be converted into the amount of defocus using a predetermined conversion coefficient.

[0037] In this way, in this embodiment, multiple image signals having parallax are acquired by a single image sensor from two light beams that pass through different pupil regions of a single optical system, and first distance information is acquired based on these multiple image signals.

[0038] Furthermore, as will be described later, the image-side defocus amount can be converted into the distance from the subject to the imaging device 100 (hereinafter referred to as subject distance) using Equation 2. Note that, for example, like a stereo camera, a plurality of image signals having parallax may be acquired using a plurality of imaging elements, and the first distance information may be acquired based on the plurality of image signals.

[0039] The distance measurement device 110 of this embodiment will be described. Fig. 3(A) is a functional block diagram showing an example of the configuration of the distance measurement device 110 of embodiment 1. Note that some of the functional blocks shown in Fig. 3(A) are realized by causing a CPU or the like serving as a computer (not shown) included in the distance measurement device 110 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0040] However, some or all of these functions may be implemented by hardware, which may be a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP).

[0041] Furthermore, the functional blocks shown in Fig. 3(A) do not have to be built into the same housing, but may be configured as separate devices connected to each other via signal paths. The above explanation regarding Fig. 3(A) also applies to Fig. 11(A).

[0042] Distance measurement device 110 acquires first distance information Idist1 by first acquisition means 310, and acquires second distance information Idist2 by second acquisition means 320. In this embodiment, as described above, the first acquisition means acquires the first distance information using an image sensor having a distance measurement function using an image plane phase difference ranging method.

[0043] The first acquisition means 310 acquires first distance information including errors (errors over time of the imaging device, manufacturing errors) via the imaging optical system 120, and the second acquisition means 320 acquires second distance information having errors smaller than the errors in the first distance information. The errors include errors over time and manufacturing errors, but the following embodiments will mainly describe errors over time.

[0044] The first correction value generating means 330 obtains the first correction value Ic1 from the first distance information Idist1 and the second distance information Idist2. That is, the first correction value generating means 330 calculates the first correction value Ic1 for correcting the time error of the first distance information based on the second distance information.

[0045] The reliability score determination means 340 calculates a reliability score indicating the reliability of the first correction value Ic1, and determines the reliability of the first correction value based on the reliability score. If it is determined that the first correction value Ic1 is reliable, the first correction value Ic1 is registered as valid data.

[0046] The second correction value generation means 350 generates the second correction value Ic2 using the correction value within a predetermined valid time interval from the first correction value Ic1 registered in the valid data. That is, the second correction value generation means 350 generates the second correction value from the first correction value whose reliability score is equal to or greater than a predetermined threshold and which was acquired within the predetermined valid time interval.

[0047] Furthermore, the correcting means 360 corrects the first distance information Idist1 using the second correction value Ic2 to generate and output corrected distance information IdistC.

[0048] Next, using Figures 3(B) and (C), we will explain the processing performed by the first acquisition means 310, the second acquisition means 320, the first correction value generation means 330, the confidence score determination means 340, the second correction value generation means 350, and the correction means 360.

[0049] Fig. 3(B) is a flowchart showing an example of the operation of the distance measurement device 110, and Fig. 3(C) is a flowchart showing an example of part of the operation of Fig. 3(B). Note that the operation of each step in the flowcharts of Fig. 3(B) and (C) is performed sequentially by a CPU or the like serving as a computer in the distance measurement device 110 executing a computer program stored in memory.

[0050] 3B, the CPU of the distance measurement device 110 performs a first acquisition process using the first image group Sg1 acquired from the image sensor 101 by the first acquisition means 310. Then, first distance information Idist1 indicating the distance to the subject is acquired. Here, step S310 functions as a first acquisition step for acquiring the first distance information.

[0051] The first image group Sg1 includes a first image signal S11 generated by the first photoelectric conversion unit 161 and a second image signal S12 generated by the second photoelectric conversion unit 162. The specific processing content of the first acquisition process in step S310 in Fig. 3(B) will be described below with reference to Fig. 3(C).

[0052] In step S311, the CPU of the distance measurement device 110 corrects the difference in light intensity between the first image signal S11 and the second image signal S12 using the first acquisition means 310. That is, the imaging optical system 120 causes an imbalance in the light intensity between the first image signal S11 and the second image signal S12.

[0053] In step S311, the CPU of the distance measurement device 110 uses the light amount correction value stored in the information storage means 170 to perform light amount correction processing between the first image signal S11 and the second image signal S12.

[0054] It is not necessary to use the light intensity correction value stored in the information storage means 170. For example, it is also possible to generate a light intensity correction value from the area ratio between the first pupil region 210 and the second pupil region 220, and then perform light intensity correction.

[0055] In step S312, the CPU of the distance measurement device 110 causes the first acquisition means 310 to perform noise reduction processing to reduce noise generated in the image sensor 101. Specifically, the CPU performs filtering processing using a band-pass filter on the first image signal S11 and the second image signal S12.

[0056] Generally, the higher the spatial frequency band, the lower the signal-to-noise ratio (ratio of signal components to noise components) and the greater the relative noise components. Therefore, a low-pass filter with a lower pass rate is used for higher frequencies.

[0057] Note that even after the light intensity correction process in step S311, the optical characteristics of the imaging optical system 120 may not always be the same as the design values ​​due to manufacturing errors of the imaging optical system 120. For this reason, for example, a band-pass filter or the like is used, which has a pass rate of 0 for the frequency band when the spatial frequency is 0 and a low pass rate for the high frequency band.

[0058] In step S313, the CPU of the distance measurement device 110 performs a parallax calculation process to calculate the amount of parallax between the first image signal S11 and the second image signal S12 using the first acquisition means 310. Specifically, a point of interest is set within the first image signal S11, and a matching area is set with the point of interest at its center. The matching area may be, for example, a rectangle with a side of a predetermined length and centered at the point of interest, but is not limited to this.

[0059] Next, a reference point is set in the second image signal S12, and a reference region centered on the reference point is set. The reference region has the same size and shape as the matching region. The reference point is moved sequentially, and the degree of correlation between the first image signal S11 contained in the matching region and the second image signal S12 contained in the reference region is calculated. The reference point with the highest correlation is set as the corresponding point that corresponds to the point of interest. The amount of relative positional deviation between the point of interest and the corresponding point is the amount of parallax at the point of interest.

[0060] The first obtaining means 310 can calculate the amount of parallax at a plurality of pixel positions by calculating the amount of parallax while sequentially changing the point of interest. Note that a known method can be used to calculate the degree of correlation.

[0061] That is, the correlation can be calculated using, for example, a method called NCC (Normalized Cross-Correlation) that evaluates the normalized cross-correlation between image signals, SSD (Sum of Squared Difference) that evaluates the sum of squares of the differences between image signals, or SAD (Sum of Absolute Difference) that evaluates the absolute value of the differences.

[0062] Furthermore, in this embodiment, reliability information representing the reliability of the calculated distance information is generated using the texture amount and frequency component information of each pixel block obtained when using these correlation calculation methods.

[0063] In step S314, the CPU of the distance measurement device 110 performs distance conversion processing. That is, the first acquisition means 310 uses a predetermined conversion coefficient to convert the amount of parallax into the distance (defocus amount) from the image sensor 101 to the image formation point of the imaging optical system 120. Here, the coefficient for converting the amount of parallax into the amount of defocus is called a BL value. When the BL value is BL, the amount of defocus is ΔL, and the amount of parallax is d, the amount of parallax d can be converted into the amount of defocus ΔL using the following equation 1.

[0064] ΔL=BL×d...(Formula 1) The first acquiring means 310 converts the parallax amounts at a plurality of pixel positions into defocus amounts, thereby acquiring first distance information Idist1 including the defocus amounts at a plurality of pixel positions as distance information.

[0065] Furthermore, to convert this defocus amount into the subject distance, the following Equation 2, which is a lens formula in geometrical optics, can be used.

[0066] 1 / A+1 / B=1 / f (Equation 2) A: distance from the object surface to the principal point of the imaging optical system 120, B: distance from the principal point of the imaging optical system 120 to the image plane, f: focal length of the imaging optical system 120.

[0067] In Equation 2, the focal length is a known value. The value of B can be calculated using the defocus amount. Therefore, the distance A to the object surface, i.e., the subject distance, can be calculated using the focal length and defocus amount. This method of obtaining subject distance information is called a phase difference ranging method, and in particular, the method of obtaining distance information based on signals obtained from different pupil regions of a single optical system is called a split-pupil phase difference method.

[0068] In addition, in the above formula 2, if the distance B from the principal point of the imaging optical system 120 to the image plane obtained using the calculated defocus amount contains an error, the distance A to the object plane, i.e., the subject distance, calculated using formula 2 will also contain an error.

[0069] Factors that cause an error in the distance B include changes in the image capturing device 100 that occur due to the influence of changes in the ambient temperature and humidity, vibrations, etc. Specifically, these are changes in the optical characteristics caused by changes in the refractive index and curvature of each lens in the imaging optical system 120 depending on the ambient changes, and deformation such as bending of the image capturing element 101 caused by the ambient changes. In addition, errors in the distance B can also occur due to manufacturing errors and changes over time.

[0070] When a difference occurs between the assumed image plane position of the imaging optical system 120 and the actual image position, the conversion relationship between the defocus amount and the distance from the object plane to the principal point of the imaging optical system 120 is disrupted, and an error occurs in the value of the distance B from the principal point of the imaging optical system 120 to the image plane.

[0071] Hereinafter, the difference between the assumed image plane position and the actual image plane position of the imaging optical system 120 will be referred to as the image-side change amount. Even if the image-side change amount is not 0, the relationship between the defocus amount and the object distance follows Equation 2. Therefore, if the defocus amount is corrected using a correctly estimated image-side change amount, it is possible to obtain distance A, which is the object distance with reduced error.

[0072] As described above, the first acquisition means 310 acquires first distance information based on the first signal and the second signal having parallax output from the image sensor 101 via the imaging optical system. Note that there may be three or more signals having parallax. When the first distance information is acquired using such a pupil division phase difference method, as described above, the first distance information contains errors due to changes in the characteristics of the imaging optical system and the image sensor over time, temperature changes, etc.

[0073] Figure 4(A) is a diagram showing an example of the two-dimensional distribution of the amount of field curvature of the imaging optical system 120 within the effective pixel range of the image sensor 101, and Figure 4(B) is a diagram showing an example of the amount of field curvature along I-I' in Figure 4(A).

[0074] In Figure 4(B), m1 represents the amount of field curvature when no changes over time or temperature have occurred, and m2 represents the amount of field curvature when changes over time or temperature have occurred. The difference between m1 and m2 in the figure is the amount of change on the image side mentioned above. Figure 4(C) is a diagram showing an example of the amount of change on the image side along I-I' in Figure 4(A).

[0075] As shown in Figure 4(C), when the optical characteristics of the imaging optical system 120 change over time or due to temperature changes, the amount of image-side change changes depending on the angle of view. If the amount of image-side change is a constant value independent of the angle of view, correction is possible with at least one piece of correction information. However, if the amount of image-side change changes depending on the angle of view, as shown in Figure 4(C), it is difficult to estimate the amount of image-side change using only one piece of correction information.

[0076] In such a case, it is necessary to estimate a two-dimensional distribution of the image-side change amount as shown in Fig. 4(C) In order to generate this first correction value Ic1, in this embodiment, as described above, the first correction value Ic1 is generated using second distance information acquired by an acquisition means different from the first acquisition means.

[0077] That is, in this embodiment, as described above, the first acquisition unit 310 uses the first image group Sg1 acquired from the image sensor 101 to acquire first distance information Idist1 that indicates the distance to the subject.

[0078] Furthermore, the first acquisition means 310 temporarily stores the first image group Sg1 acquired from the image sensor 101 in the information storage means 170. The second acquisition means 320 acquires second distance information Idist2 using the first image group Sg1 stored in the information storage means 170 as a second image group Sg2. Here, the second image group Sg2 is an image group in which image signals at a time earlier than that of the first image group Sg1 have been accumulated in the information storage means 170.

[0079] The second acquisition means 320 performs a second acquisition process to acquire second distance information Idist2 using the first image group Sg1 of the first frame and the second image group Sg2 of the second frame. In this embodiment, the second acquisition means acquires the second distance information based on image signals of two frames. However, the second distance information may be acquired based on image signals of three or more frames.

[0080] The first image group Sg1 includes a first image signal S11 and a second image signal S12 captured by the imaging device 100. The second image group Sg2 includes a first image signal S21 and a second image signal S22 captured by the imaging device 100 at a timing different from that of the first image group Sg1.

[0081] Here, the first image signals S11 and S21 are image signals generated by the first photoelectric conversion unit 161, and the second image signals S12 and S22 are image signals generated by the second photoelectric conversion unit 162.

[0082] The second acquisition means 320 acquires the second distance information Idist2 using a known SfM (Structure from Motion) method. Hereinafter, the specific content of the second acquisition step in which the CPU of the distance measurement device 110 acquires the second distance information using the second acquisition means 320 in step S320 will be described with reference to FIG. 5(A).

[0083] 5(A) is a flowchart showing an example of a second acquisition step in which, in step S320, the CPU of the distance measurement device 110 acquires second distance information by the second acquisition means 320. Note that the CPU or the like as a computer in the distance measurement device 110 executes a computer program stored in memory, thereby sequentially performing the operations of the steps in the flowchart in FIG.

[0084] In step S321, the CPU of the distance measurement device 110 performs optical flow calculation processing using the second acquisition means 320. That is, the second acquisition means 320 acquires a first image signal S11 from a first image group Sg1 of a first frame acquired at time t1 stored in the information storage means 170. Also, the second acquisition means 320 acquires a first image signal S21 from a second image group Sg2 of a second frame captured at a different time t2.

[0085] Then, optical flow is calculated using a known method based on the first image signal S11 of the first image group Sg1 and the first image signal S21 of the second image group Sg2. The time relationship is t1>t2, and t2 is a time earlier than t1.

[0086] Furthermore, the multiple images for which the optical flow is calculated may be image signals from the same viewpoint, and the optical flow may be calculated based on the second image signal S12 of the first image group Sg1 and the second image signal S22 of the second image group Sg2.

[0087] Alternatively, the optical flow may be calculated based on a composite image signal obtained by adding together the first image signal S11 and the second image signal S12 of the first image group Sg1, and a composite image signal obtained by adding together the first image signal S21 and the second image signal S22 of the second image group Sg2.

[0088] Calculation of optical flow will be specifically described with reference to Figures 5(B) to 5(D). Feature points are calculated for the acquired first image signal S11 and first image signal S21 using the Harris corner detection algorithm, which is a well-known method.

[0089] FIG. 5(B) is a diagram showing feature points 501 calculated for the first image signal S21 at time t2 with stars, FIG. 5(C) is a diagram showing feature points 502 calculated for the first image signal S11 at time t1 with stars, and FIG. 5(D) is a diagram showing the calculated optical flow.

[0090] In FIG. 5(D), 500 denotes an optical flow calculated by associating feature points between the calculated first image signal S21 and the first image signal S11 using the KLT (Kanade-Lucas-Tomasi) feature tracking algorithm, which is a well-known method.

[0091] The algorithms used to calculate feature points, feature amounts, and optical flow are not limited to the above methods. Other algorithms, such as FAST (Features from Accelerated Segment Test) and BRIEF (Binary Robust Independent Elementary Features), may also be used. Alternatively, ORB (Oriented FAST and Rotated BRIEF) may also be used.

[0092] In step S322, the CPU of distance measurement device 110 performs distance calculation processing. That is, the distance to the subject is calculated by a known method using the calculated optical flow 500. The coordinates in the image coordinate system of the subject for which the distance is to be calculated are set to (u, v), the optical flow of the subject for which the distance is to be calculated is set to (Δu, Δv), and the distance to the subject is set to z.

[0093] The camera movement amounts between images used to calculate the optical flow are defined as rotational movement (ωx, ωy, ωz) and translational movement (tx, ty, tz). If the focal length of the camera is f, the following relationships in Equation 3 and Equation 4 hold.

[0094]

number

number

[0095] The focal length f of the camera is known in advance through prior calibration, etc. Therefore, if the camera movement amounts, that is, the rotational movement amount (ωx, ωy, ωz) and the translational movement amount (tx, ty, tz), are calculated, the distance z from the camera to the subject at the subject position (u, v) on the image can be calculated from the optical flow.

[0096] Then, a known method is used to calculate the amount of camera movement between the first image signal S21 and the first image signal S11 used in calculating the optical flow 500. Specifically, the camera fundamental matrix F is acquired using the 8Point algorithm so as to satisfy the epipolar constraint using the feature point 501 at time t2, the feature point 502 at time t1, and the optical flow 500 representing the correspondence between them.

[0097] In this case, it is desirable to use the RANSAC (Random Sample Consensus) method to efficiently remove outliers and perform calculations using a stable method.

[0098] The camera fundamental matrix F is decomposed into the camera fundamental matrix E using a known method, and the camera external parameters, rotational movement amount R(ωx, ωy, ωz) and translational movement amount T(tx, ty, tz), are calculated from the camera fundamental matrix E.

[0099] Here, the calculated camera extrinsic parameters are the relative displacement of the camera movement from time t2 to time t1, and since the scaling is indefinite, the translational movement T(tx, ty, tz) in particular is a normalized relative value. By scaling this, the translational movement T(tx, ty, tz) becomes the actual movement amount.

[0100] Specifically, a second image signal S12 corresponding to an image signal taken from a different viewpoint at the same time as the first image signal S11 from which the feature point 502 was obtained is obtained from the information storage means 170. Similarly, a second image signal S22 corresponding to an image signal taken from a different viewpoint at the same time as the first image signal S21 from which the feature point 501 was obtained is obtained from the information storage means 170.

[0101] Then, in step S310, a first acquisition process is performed on each image pair to acquire distance values ​​from the first distance information based on the amount of parallax. After that, the amount of translational movement T, which is parallel to the optical axis of the camera, is acquired from the difference in distance values ​​between feature point 502 and feature point 501, which are associated by the optical flow.

[0102] The actual translational movement amount T(tx, ty, tz) from time t2 to t1 is obtained by scaling other components from the actual movement amount tz, which has also been scaled. Then, using Formula 3 and Formula 4, the distance z from the camera to the feature point 502 of the first image signal S11 at each coordinate on the image is calculated.

[0103] The method for scaling the camera movement amount is not limited to this method. For example, scaling may be performed using a measuring device such as an IMU (inertial measurement unit) or a GNSS (Global Navigation Satellite System). Alternatively, in the case of an in-vehicle camera, scaling may be performed by obtaining the camera movement amount from vehicle speed information, GPS information, or the like.

[0104] Note that bundle adjustment, a well-known method, may be used to calculate the camera movement amount and the positional relationship between the subject and the camera. The relationships between variables such as the camera fundamental matrix and optical flow, including internal camera parameters such as focal length, can be analytically calculated together using the nonlinear least squares method to improve consistency.

[0105] Of the feature points used to calculate the camera movement amount, feature points calculated from a subject that is not a stationary object with respect to the world coordinate system to which the image capture device belongs may be excluded from the processing. The known method of estimating the camera movement amount calculates various parameters assuming that the subject is a stationary object, and therefore becomes a source of error when the subject is a moving object.

[0106] Therefore, by excluding feature points calculated from moving objects, the accuracy of calculating various parameters can be improved. Note that the determination of moving objects is based on the classification of subjects using image recognition technology. Alternatively, the determination of moving objects may be made by comparing the relative value of the amount of change in the acquired distance information over time with the amount of movement of the imaging device.

[0107] In step S323, the CPU of the distance measurement device 110 performs a second distance calculation process. That is, the CPU converts the distance from the camera at each coordinate on the image of the feature point 502 in the first image signal S11 acquired as described above into an image-side defocus amount using Equation 2, and calculates second distance information Idist2.

[0108] In this manner, the second obtaining means obtains second distance information by calculating the optical flow of the target feature points from the image signals of a plurality of frames.

[0109] The first correction value generation means 330 generates and acquires a first correction value Ic1 corresponding to the amount of change on the image side based on the first distance information Idist1 and the second distance information Idist2. A method for generating the first correction value Ic1 corresponding to the amount of change on the image side from the first distance information Idist1 and the second distance information Idist2 will be described below.

[0110] The first distance information Idist1 is derived based on the amount of parallax calculated by the first acquisition means, and the amount of parallax corresponds to the distance between the centers of gravity of the first and second light beams based on the positional relationship between the imaging surface of the imaging optical system and the image sensor, as shown in Figures 2(B) to 2(D).

[0111] Therefore, if the shape of the imaging surface of the imaging optical system or the image sensor changes due to changes in the surrounding environment such as temperature and humidity, the first distance information Idist1 calculated by the first acquisition unit is easily affected by changes in the optical system over time, environmental changes, etc.

[0112] On the other hand, the second distance information Idist2 is calculated from an optical flow based on a group of images acquired at a time interval that is much shorter than the time interval at which changes in the surrounding environment such as temperature and humidity or changes over time occur.

[0113] Therefore, the first distance information used in calculating the second distance information also uses relative changes over short time intervals as the camera movement amount, so the amount of error over time due to changes in the surrounding environment such as temperature and humidity or changes over time can be almost ignored. In other words, the influence of environmental changes and changes over time on the optical system is much smaller for the second distance information Idist2 calculated by the second acquisition means than for the first distance information Idist1.

[0114] However, the second distance information Idist2 is information that depends on the coordinates at which the feature points are calculated, and is therefore distance information that is sparse with respect to the angle of view.

[0115] Therefore, in this embodiment, the second distance information Idist2 is used to calculate a first correction value Ic1 corresponding to the amount of image-side change due to changes in the surrounding environment and changes over time. That is, by correcting the first distance information Idist1 to calculate corrected distance information IdistC, it is possible to reduce the amount of error over time and obtain precise distance information for the angle of view. Furthermore, if precise distance information can be obtained, it becomes possible to measure the detailed shape of an object.

[0116] 6A is a flowchart showing an example of the operation of the first correction value generation process performed by the first correction value generation means 330 in step S330. Here, step S330 functions as a first correction value generation step for calculating a first correction value for correcting the first distance information based on the second distance information.

[0117] The CPU or the like serving as a computer within the distance measurement device 110 executes a computer program stored in memory, thereby sequentially performing the operations of the steps in the flowchart of FIG. 6(A).

[0118] In step S331, the CPU of the distance measurement device 110 performs an image side change amount calculation process to calculate an image side change amount using the first distance information Idist1 acquired by the first acquisition means 310 and the second distance information Idist2 acquired by the second acquisition means 320.

[0119] Fig. 6(B) is a diagram showing an example of the defocus amount d1, which is the first distance information Idist1 acquired by the first acquisition means 310, relative to the assumed image plane. Fig. 6(C) is a diagram showing an example of the defocus amount d2, which is the second distance information Idist2 acquired by the second acquisition means 320, relative to the assumed image plane.

[0120] In Fig. 6(C), distance information is acquired from the pixel corresponding to the feature point 502, and therefore is a sparse data group corresponding to each coordinate on the image of the feature point 502. Fig. 6(D) is a diagram showing an example of the image-side defocus amount along I-I' in Figs. 6(B) and (C).

[0121] The discontinuous line segment p1 in FIG. 6D indicates the defocus amount d1, which is the first distance information Idist1, and the point data p2 in FIG. 6D indicates the defocus amount d2, which is the second distance information Idist2.

[0122] As described above, the first distance information Idist1 contains errors due to environmental changes and changes over time in the image capture device 100. On the other hand, the second distance information Idist2 is hardly affected by changes over time in the image capture device 100. Therefore, if the difference between the two is calculated, the difference corresponds to the amount of change on the image side that has been affected by changes over time.

[0123] Fig. 6(E) is a diagram showing the image-side change amount calculated by fitting using a dashed line, which is the image-side change amount obtained by subtracting p2, which is the defocus amount d2, which is the second distance information Idist2, from p1, which is the defocus amount d1, which is the first distance information Idist1 in Fig. 6(D).

[0124] 6(B) to (E), the p1-p2 difference data 602 can actually be acquired from three points on the data acquisition coordinate system 601 shown in Fig. 6(E) that correspond to the coordinates of the feature point 502 in Fig. 5(C). In contrast, the amount of change on the image side that is affected by environmental changes and changes over time is a change in the focus position at each pixel, that is, it corresponds to a change in the amount of field curvature.

[0125] Since the amount of field curvature itself has a continuous and smooth shape, the amount of change on the image side between angles of view, i.e., between pixels, is continuous and smoothly connected. Therefore, it is possible to interpolate between angles of view by fitting with polynomial approximation using each difference data 602 acquired on each data acquisition coordinate 601.

[0126] The image side change amount 603 calculated from the polynomial approximation is shown by the dashed line in Fig. 6(E). For the sake of explanation, the image side change amount calculated from the polynomial approximation is expressed one-dimensionally along the line segment I-I', but the actual image side change amount is two-dimensional data on the xy plane.

[0127] In step S332, the CPU of the distance measurement device 110 performs correction information calculation processing using the first correction value generation means 330. That is, in step S332, using the acquired difference data that is discrete with respect to the angle of view, surface fitting is performed by polynomial approximation on the xy plane to estimate the image side change amount. Here, the approximate surface data that is the calculated image side change amount is set as the first correction value Ic1.

[0128] In this way, the distance values ​​and defocus amounts acquired by each acquisition method may take discrete values ​​depending on the distance of the subject being photographed. However, by converting the distance information acquired by each method into an image-side defocus amount and obtaining a difference value, these can be treated as continuous values, and their values ​​and shapes can be estimated and acquired using various approximations.

[0129] In step S332, when surface fitting by polynomial approximation is performed using the difference data to estimate the amount of change on the image side, it is desirable to remove outliers from the difference data based on the amount of change on the image side estimated from the design values ​​of the imaging optical system, the image sensor, etc. In other words, the amount of change on the image side can be predicted in advance by simulation based on the design values ​​of the device and changes in environmental temperature and humidity, as well as changes over time.

[0130] This predicted value is stored in the information storage means as an initial value for correction as a lookup table or the like, and compared with the acquired difference data, and outliers are removed by threshold judgment, thereby improving fitting accuracy. That is, the initial value includes the amount of image-side change due to environmental changes and aging, and the first correction value generation means 330 may obtain an initial value for correction related to the imaging optical system or the image sensor from a lookup table and calculate the correction value.

[0131] The reliability score determination means 340 calculates a reliability score S for the first correction value generated by the first correction value generation means 330 and determines whether the first correction value can be used in the correction process. Here, a method for calculating the reliability score S and a method for determining the first correction value using the reliability score S will be described.

[0132] 7(A) is a flowchart showing an example of the reliability score determination process for the first correction value performed by the reliability score determination means 340 in step S340. Here, step S340 functions as a reliability score determination step that calculates a reliability score indicating the reliability of the first correction value and determines the reliability of the first correction value based on the reliability score.

[0133] The CPU or the like serving as a computer within the distance measurement device 110 executes a computer program stored in memory, thereby sequentially performing the operations of the steps in the flowchart of FIG. 7(A).

[0134] In step S341, the CPU of the distance measurement device 110 generates a reliability score S of the first correction value as an index for determining whether or not the first correction value generation means 330 can acquire a good correction amount.

[0135] Specifically, when the second acquisition means uses the SfM method as in this embodiment, feature points are calculated from an image of the current surrounding environment captured by an imaging device in the same manner as the optical flow calculation process in step S321 described above, and a reliability score is obtained based on the calculated feature points.

[0136] More specifically, the reliability score S is calculated using, for example, the following formula 5 based on the number N of feature points within the field of view, the distribution D of feature points within the field of view, and the reliability C of the calculated feature points. S=α*N+β*D+γ*C...(Formula 5) Incidentally, α, β, and γ are coefficients that are set appropriately.

[0137] The distribution D of feature points within the field of view is expressed as D=δ*σ using the standard deviation σ based on a histogram of the coordinate values ​​of the coordinates where the feature points were obtained. The reliability C of a feature point is calculated based on the strength of the feature amount according to the method used to calculate the feature point, and the sum of the strength of the feature amount of each feature point is taken as the reliability C of the feature point.

[0138] In step S342, the CPU of the distance measurement device 110 determines the reliability score S by comparing the reliability score S generated in step S341 with an appropriately set threshold. If the reliability score S exceeds the threshold, it determines that the first correction value is appropriate (True) and proceeds to step S343. On the other hand, if the reliability score S is equal to or less than the threshold in step S342, it determines that the first correction value is inappropriate (False) and ends the flow of FIG. 7.

[0139] If it is determined that the first correction value is appropriate, in step S343, the CPU of the distance measurement device 110 registers it as valid data. That is, in step S343, the first correction value is registered by linking it to the data acquisition time and the reliability score S and saving it in the information storage means 170.

[0140] Here, the first correction value may be stored in the form of approximation surface data of the image side change amount, but from the viewpoint of data volume, only the coefficients of the polynomial that are the basis for creating the approximation surface data may be stored in the information storage means 170. Then, in the second correction value generation process described later, an approximation surface may be generated again based on the coefficients of the polynomial that have been stored in the information storage means 170.

[0141] Fig. 7(B) is a flowchart illustrating an example of the process of generating the second correction value in step S350. Note that the operation of each step in the flowchart of Fig. 7(B) is performed sequentially by a CPU or the like serving as a computer in the distance measurement device 110 executing a computer program stored in a memory.

[0142] In step S350, the CPU of the distance measurement device 110 generates a second correction value that is more reliable than the first correction value, using one or more first correction values ​​that have been determined to be appropriate by the reliability score S in the previous step and have been registered in the valid data.

[0143] Here, step S350 functions as a second correction value generation step that generates a second correction value from a first correction value whose reliability score is equal to or greater than a predetermined threshold and which was obtained within a predetermined valid time interval.

[0144] Specifically, in step S351, the CPU of the distance measurement device 110 acquires data within the valid time interval from the information storage means 170, and in step S352, generates a more reliable second correction value from multiple first correction values.

[0145] Fig. 7(C) is a diagram for explaining an example of a method for generating a second correction value. In the graph of Fig. 7(C), the horizontal axis represents time and the vertical axis represents the first correction value. The plotted points represent the first correction value obtained at each time.

[0146] Here, the first correction value is two-dimensional data in the surface direction, but for convenience it is plotted on the graph as a scalar value. The density of the plotted points corresponds to the reliability score, with higher density indicating a higher reliability score S. The white dots indicate points that have a low reliability score and are judged as False by the reliability score judgment means 340.

[0147] Next, we will explain a specific example of how to generate a second correction value when the current time is t1. The double-headed arrow in the figure indicates the effective time interval te of the first correction value. The effective time interval te is a time determined from the predicted value of the change in the state of the distance measurement device over time.

[0148] That is, it is set shorter than the predicted value of the time interval during which a change over time occurs. That is, within the effective time interval te, it is possible to obtain multiple correction values ​​while keeping the expected amount of error over time below a certain value. That is, for a distance measurement device with a certain error, it is possible to expect results equivalent to those obtained by correcting using multiple data.

[0149] The effective time interval te may be set based on moving object information about the moving object. The moving object information may include, for example, at least one of the speed and direction change of the moving object. For example, if the speed and direction change of the moving object are greater than predetermined values, it is desirable to shorten the effective time interval te in order to ensure the validity of the first correction value.

[0150] If the reliability score determination of the first correction value is erroneously estimated in step S340, the first correction value may contain a large error even if it is determined to be reliable in step S340. Even in such a case, a more reliable second correction value can be generated by extracting only the first correction values ​​within the valid time interval te from the first correction values ​​obtained under multiple conditions and generating the second correction value.

[0151] For example, in Figure 7(C), seven first correction values ​​are created between the current time t1 and the valid time interval te. The points a4 and a5 have low reliability scores and are significantly different from the other first correction value points. Therefore, the second correction value is created from five points whose reliability score S is greater than the predetermined threshold and is data within the valid time interval.

[0152] For example, when the reliability scores S are compared and the maximum reliability score S is at point a2, the first correction value at point a2 is set as the second correction value. In this case, it is expected that a higher accuracy of correction can be achieved by using the first correction value at point a2 rather than using point a7 of the first correction value generated at the current time t1 as the second correction value.

[0153] In this way, the second correction value generation means may select the first correction value with the largest reliability score among multiple first correction values ​​determined to be reliable by the reliability score determination means within the valid time interval as the second correction value.

[0154] Next, we will explain the case where time advances to t2. At time t2, a second correction value is generated similarly using the first correction value at the point with the largest confidence score S within the valid time interval te. That is, the first correction value at the time of point a6 with the largest confidence score S is adopted as the second correction value. In this case, the confidence score S at the time of a6 is lower than the confidence score S at the time of a2, which was adopted as the second correction value at time t1.

[0155] However, since the time interval from time t2 to the time of point a2 is longer than the effective time interval te, there is a high possibility that an error has occurred over time. Therefore, even at time t2, there is a low possibility of change over time, and from among the multiple highly reliable first correction values, a more reliable first correction value can be generated as the second correction value, thereby enabling more reliable correction to be performed.

[0156] In the above example, the valid data within the valid time interval te was selected with the largest reliability score S and used as the second correction value, but the second correction value may also be generated using multiple first correction values.

[0157] For example, a correction value that is more resistant to noise can be generated by taking the average value of multiple first correction values ​​that are within the valid time interval te and have a reliability score S greater than a threshold as the second correction value. Alternatively, the second correction value may be generated by statistically treating multiple data, such as by multiplying each first correction value by the reliability score S as a weight and averaging them.

[0158] That is, the second correction value generation means may generate the second correction value by averaging or weighting the average by the reliability score of multiple first correction values ​​that are determined to be reliable by the reliability score determination means within the valid time interval.

[0159] 3B, the CPU of distance measurement device 110 executes a correction processing step. That is, step S360 functions as a correction step in which the first distance information Idist1 is corrected using the generated second correction value Ic2 to generate corrected distance information IdistC.

[0160] Specifically, when the first acquisition means converts the defocus amount ΔL generated in the distance conversion process of step S314 into B in equation 2 (the distance from the principal point of the imaging optical system 120 to the image plane), it uses the defocus amount ΔL' corrected by the following equation 6. ΔL'=ΔL-Ic2...(Formula 6)

[0161] The corrected distance information IdistC is calculated by calculating the distance to the subject based on Equation 2 using the distance B from the principal point of the imaging optical system 120 to the image plane, which is calculated using this corrected defocus amount ΔL′. That is, in this embodiment, the defocus amount is corrected using the second correction value Ic2.

[0162] It is not necessary to perform the above correction process every time distance calculation is performed. The time interval at which environmental changes occur over time is much longer than the time interval at which distance measurement device 110 performs distance calculations, so the second correction value Ic2 calculated as described above can continue to be used for a while from the next frame onwards.

[0163] Furthermore, because changes over time occur as changes in the image plane due to the principles of occurrence, it is desirable to correct the distance information using the defocus amount on the image side, rather than the subject distance on the object side, as in embodiment 1.

[0164] In this embodiment, the first correction value generation means 330 compares (calculates the difference between) the first distance information and the second distance information as the image-side defocus amount. Therefore, it is possible to calculate correction information by approximating the entire field of view at once on the same scale regardless of whether the subject is close or far away, without being affected by the longitudinal magnification of the optical system. As a result, it is possible to secure a large number of data points to be used for approximation, which has the effect of enabling calculation of correction information with high accuracy.

[0165] Furthermore, if the characteristics of the imaging optical system 120 change over time due to changes in temperature and humidity or vibration, not only will the amount of change on the image side change, but the focal length and BL value will also change, which can be a cause of distance measurement errors. However, even if the focal length or BL value changes, as is clear from Equation 1 and Equation 2, the amount of change is reflected in the amount of change in BP, and therefore, these can be corrected simultaneously by the correction process according to this embodiment.

[0166] In the above embodiment, two sets of image signals captured at two different times are used as the set of images used to calculate the optical flow by the second acquisition means, but any number of sets of image signals may be used. By increasing the number of sets, the number of combinations of images for calculating the optical flow increases, thereby improving the accuracy of calculating the camera movement amount and the second distance information.

[0167] Furthermore, the image groups at time t1 and time t2 may be acquired by, for example, a person carrying a camera and walking around.

[0168] Furthermore, various arithmetic processing steps do not have to be performed inside an imaging device such as a camera. For example, data may be sent to an external server or terminal via a network, and the data may be executed using a CPU, GPU, or IC installed on the server or terminal.

[0169] <Embodiment 2> A second embodiment of the present invention will be described. In the second embodiment, an outlier determination of the first corrected value is performed in determining the confidence score S. The flow of the process will be described below with respect to differences from the first embodiment.

[0170] Fig. 8(A) is a flowchart showing an example of a flow for determining a reliability score in embodiment 2. Note that the operation of each step in the flowchart of Fig. 8(A) is performed sequentially by a CPU or the like serving as a computer in the distance measurement device 110 executing a computer program stored in a memory.

[0171] Step S800 corresponds to step S340 in the first embodiment, and in step S810, the CPU of the distance measurement device 110 performs processing to generate a reliability score S of the first corrected value, similar to step S341.

[0172] In step S820, the CPU of the distance measurement device 110 determines the reliability score S in the same manner as in step S342, and if the reliability score exceeds a predetermined threshold, proceeds to step S830, and if the reliability score is equal to or less than the predetermined threshold, ends the flow of Figure 8.

[0173] In the second embodiment, in step S830, the CPU of the distance measurement device 110 further performs outlier determination on the first corrected value determined to be reliable in the reliability score determination.

[0174] An example of outlier determination in step S830 in embodiment 2 will be described with reference to Figures 8(B) and (C). Figure 8(B) is a flowchart showing an example of the flow of outlier determination, and (C) is a diagram specifically explaining the example of the flow of outlier determination.

[0175] The CPU or the like serving as a computer within the distance measurement device 110 executes a computer program stored in memory, thereby sequentially performing the operations of the steps in the flowchart of FIG. 8(B).

[0176] First, in step S831, the CPU of the distance measurement device 110 reads valid data stored in the information storage means 170. Here, it is sufficient to read only the data required in the next step.

[0177] When valid data is read, for example, in the example shown in Fig. 8(C), six valid points within a valid time interval te are read. t1 indicates the current time, and in Fig. 8(C), multiple first correction values ​​determined to be reliable in the reliability score determination are shown as plotted points.

[0178] Next, in S832, the CPU of the distance measurement device 110 generates a statistical representative correction value cr from the multiple first correction values ​​registered in the valid data. The representative correction value cr indicates a more likely correction value expected during the valid time interval te. In other words, it is a correction value that can become the second correction value as described in the first embodiment.

[0179] Therefore, the correction value can be generated using a method similar to that for generating the second correction value. In Fig. 8(C), the value of the representative correction value cr is indicated by a dotted line. Note that here, the representative correction value cr is generated as a statistical representative correction value cr by, for example, averaging a plurality of first correction values, which are valid data.

[0180] In step S833, the CPU of the distance measurement device 110 compares each of the first correction values, which are valid data, with the representative correction value cr. As a comparison method, a statistical error evaluation index such as the difference in average values ​​or the root mean square error may be used. In FIG. 8C, the difference dc (difference) between the representative correction value cr and the latest first correction value is used as the evaluation value.

[0181] In step S834, the CPU of the distance measurement device 110 compares the evaluation value of the difference calculated in step S833 with a preset threshold value. The threshold value here may also be referred to as an allowable error.

[0182] If the evaluation value is less than the threshold, i.e., the error can be estimated to be small, it is determined to be not an outlier (False), and if it is equal to or greater than the threshold, i.e., the error can be estimated to be large, it is determined to be an outlier (True). The outlier determination result in step S834 is output as the determination result in step S840.

[0183] As can be seen from the example of Fig. 8(C), for example, the first correction value at time t1 has a large evaluation value dc and is therefore determined to be an outlier in step S834. Therefore, even if the reliability score S is determined to be high in step S820 of Fig. 8(A), it is determined to be an outlier (True) in step S830, and is not registered as valid data, and the flow of Fig. 8(A) ends.

[0184] That is, if there is a difference of a predetermined value or more between the statistical representative correction value cr of multiple first correction values ​​within the valid time interval whose reliability scores are equal to or greater than a threshold and the latest first correction value, the first correction value is determined to be unreliable even if the reliability score is equal to or greater than the threshold.

[0185] On the other hand, if the evaluation value dc is small, it is determined in step S834 that it is not an outlier (False), the determination in step S830 is False, and the first correction value is registered as valid data in step S840. Thereafter, the process proceeds to step S350, where a second correction value is generated using the registered valid data.

[0186] As described above, by performing outlier determination using valid data within the valid time interval te, it is possible to exclude the first correction value of outliers that cannot be completely eliminated by the confidence score S, thereby generating a more stable correction value.

[0187] <Embodiment 3> In the third embodiment, if the reliability score S is determined to be False (unreliable), the second correction value is not updated. That is, if the reliability score determination means determines that the reliability score is unreliable, the second correction value generation means does not update the second correction value. Below, the flow of the process will be described regarding the differences from the first embodiment.

[0188] Fig. 9(A) is a flowchart showing an example of the operation of the distance measurement device 110 according to embodiment 3. Note that the operation of each step in the flowchart of Fig. 9(A) is performed sequentially by a CPU or the like serving as a computer in the distance measurement device 110 executing a computer program stored in a memory.

[0189] Steps S910 to S930 in Fig. 9(A) are the same as steps S310 to S330 in Fig. 3(B), and therefore description thereof will be omitted. In step S940 in Fig. 9(A), the CPU of the distance measurement device 110 performs reliability score determination processing. The processing content is equivalent to the reliability score determination processing in steps S342 and S820. Note that in the third embodiment, the reliability score determination is performed using valid data within the valid time interval te.

[0190] If the reliability score S is equal to or less than the threshold, it is determined that the first correction value is inappropriate (False). If it is determined as False in step S940, the process does not proceed to the second correction value generation process in step S950, but proceeds to the correction value process in step S960.

[0191] If the confidence score S exceeds the threshold, in step S940, the CPU of the distance measurement device 110 determines that the first correction value is appropriate (True). If the determination in step S940 is True, the process proceeds to step S950. In step S950, the CPU of the distance measurement device 110 generates a second correction value using a method similar to that in the first and second embodiments. Thereafter, the process proceeds to step S960.

[0192] In step S960, the CPU of the distance measurement device 110 performs the correction process in the same manner as in the embodiment 1 and embodiment 2. However, if the process proceeds to step S960 after the determination in step S940 is False, a new second correction value has not been generated in step S950, and therefore the correction process is performed using the second correction value that already exists at the time of step S940.

[0193] Fig. 9(B) is a diagram for explaining the effect of the third embodiment. In the graph of Fig. 9(B), the horizontal axis represents time and the vertical axis represents correction value, similar to Fig. 7(C). The plotted points represent the first correction values ​​acquired at each time. The density of the plotted points corresponds to the reliability score; the higher the density, the higher the reliability score S. The white dots represent points determined to have a low reliability score (False) in the reliability score determination process of step S940.

[0194] 9B, a second correction value is generated and correction processing is performed as in the previous embodiments. The first correction value at time t2 is a white circle, and therefore the reliability score is determined to be False in step S940.

[0195] Therefore, the second correction value used in the correction process in step S960 is the same as the second correction value used at time t1, which is essentially equivalent to generating a second correction value at the effective time interval te based on the first correction value at time t1.

[0196] In the example of Fig. 9(B), all first correction values ​​between time t2 and time t3 are indicated by white circles, and the reliability score S is determined to be False. Therefore, between time t2 and time t3, the first correction value at time t1 is used to generate the second correction value in the valid time interval te. Here, between time t2 and time t3, the second correction value has not been updated even once during the valid time interval te.

[0197] That is, the second correction value at time t3 was generated before the valid time interval te from the current time t3. Although it is unlikely that the second correction value generated using a correction value before the valid time interval te will deteriorate immediately after the valid time interval te has passed, it is possible to increase the probability that valid data will be generated by, for example, reducing the threshold value of the reliability score S.

[0198] As time progresses, at t4, a first correction value with a high reliability score is obtained, and the reliability score determination in step S940 is determined to be True. Therefore, in step S950, the second correction value is updated. At this time, the correction data is generated using only the first correction value at point a9.

[0199] As described above, in the third embodiment, the reliability score determination is performed using valid data within the valid time interval te in step S940. Therefore, the first correction value that cannot be completely eliminated by the reliability score can be excluded as an outlier, and a more stable correction value can be generated.

[0200] <Embodiment 4> A fourth embodiment of the present invention will be described. The fourth embodiment is an example in which a stereo camera is used as the first acquisition means 1110 in the first embodiment. The flow of processing will be described below with respect to differences from the first embodiment.

[0201] 10 is a diagram schematically illustrating an example of the configuration of a distance measurement device according to embodiment 4. In FIG. 10, an imaging device 1000 includes a first imaging optical system 1021, a first imaging element 1001, a second imaging optical system 1022, a second imaging element 1002, a distance measurement device 110, and information storage means 170.

[0202] The first imaging optical system 1021 has a first optical axis 1041, and the second imaging optical system 1022 has a second optical axis 1042. The first imaging optical system 1021 and the second imaging optical system 1022 are installed so that their optical axes are parallel. The first optical axis 1041 and the second optical axis 1042 are separated by a base length B1 of the stereo camera.

[0203] 11A is a functional block diagram showing an example of the configuration of a distance measurement device 1010 according to embodiment 4. In embodiment 4, a stereo camera is used, and therefore the device has two image sensors: a first image sensor 1001 and a second image sensor 1002.

[0204] That is, the first acquisition means 1110 acquires the first distance information using a stereo camera, but other than that, it is the same as in embodiment 1. That is, 1110 to 1160 and 1170 in Fig. 11(A) are the same functional blocks as 310 to 360 and 170 in Fig. 3, respectively.

[0205] Fig. 11(B) is a flowchart showing an example of the operation of the distance measurement device 1010. Note that the operation of each step in the flowchart of Fig. 11(B) is performed sequentially by a CPU or the like serving as a computer in the distance measurement device 110 executing a computer program stored in memory. Note that steps S1140 to S1160 are the same processes as steps S340 to S360 in Fig. 3(B), respectively, and therefore description thereof will be omitted.

[0206] In step S1110, the CPU of the distance measurement device 110 performs a first acquisition process using the first image group Sg1 acquired by the first image sensor 1001 and the second image sensor 1002 using the first acquisition means 1110. Then, the CPU acquires first distance information Idist1 representing the distance to the subject.

[0207] The first image group Sg1 includes a first image signal S11 generated by the first imaging element 1001 and a second image signal S12 generated by the second imaging element 1002. Specific processing details will be described below with reference to FIG.

[0208] Fig. 12(A) is a flowchart showing an example of the operation of the second acquisition process according to embodiment 4. Note that the operation of each step in the flowchart of Fig. 12(A) is performed sequentially by a CPU or the like serving as a computer in the distance measurement device 110 executing a computer program stored in a memory.

[0209] The processing in steps S1111 to S1113 in Fig. 12A is the same as the processing in steps S311 to S313 in Fig. 3C, respectively, and therefore description thereof will be omitted. The CPU of the distance measurement device 110 calculates the amount of parallax in step S1113, and performs distance conversion processing in step S1114.

[0210] In the first embodiment, the amount of parallax is converted into the amount of defocus, but in the present embodiment, the amount of parallax is not converted into the amount of defocus, and in step S1110, the amount of parallax at multiple pixel positions is acquired as first distance information Idist1, which includes the amount of parallax as distance information.

[0211] In the case of the stereo camera used in this embodiment, the amount of parallax can be directly converted into the subject distance using the following equation 7. Z = B × f / D (Equation 7) Note that D is the parallax, B is the base length, and f is the same focal length of the first imaging optical system 1021 and the second imaging optical system 1022. In Equation 7, the base length and focal length are known values.

[0212] Next, the second acquisition process in step S1120 will be described with reference to Fig. 12(B). Fig. 12(B) is a flowchart showing an example of a process for calculating an optical flow. Note that the operation of each step in the flowchart in Fig. 12(B) is performed sequentially by a CPU or the like serving as a computer in the distance measurement device 110 executing a computer program stored in memory.

[0213] The distance calculation process from step S1021 to step S1022 is the same as step S321 to step S322 in Fig. 5A, respectively, and thereby the distance z from the camera to each coordinate of the feature point 502 on the image is calculated.

[0214] In step S1023, the CPU of the distance measurement device 110 performs a second distance calculation process. That is, the distance from the camera at each coordinate on the image of the feature point 502, which has been calculated up to step S1022, is converted into a parallax amount using Equation 7, and the converted amount is set as second distance information Idist2.

[0215] Next, the first correction value generation process in step S1130 of Fig. 11(B) will be specifically described with reference to Fig. 13(A) to (E). Fig. 13(A) is a flowchart showing an example of the operation of the second correction value generation process according to the fourth embodiment.

[0216] The CPU or the like serving as a computer within the distance measurement device 110 executes a computer program stored in memory, thereby sequentially performing the operations of the steps in the flowchart of FIG. 13(A).

[0217] In step S1131, the CPU of the distance measurement device 110 performs a parallax change amount calculation process. That is, in step S1131, the CPU calculates a ratio of the parallax amounts as the parallax change amount using the first distance information Idist1 acquired by the first acquisition means 1110 and the second distance information Idist2 acquired by the second acquisition means 1120.

[0218] 13B is a diagram showing an example of a parallax amount D1, which is the first distance information Idist1 acquired by the first acquisition unit 1110. FIG. 13C is a diagram showing an example of a parallax amount D2, which is the second distance information Idist2 acquired by the second acquisition unit 1120.

[0219] As shown in Fig. 13(C), second distance information Idist2 is acquired for the pixel corresponding to feature point 502, and therefore the disparity amount D2 is a sparse data group corresponding to each coordinate on the image of feature point 502. Fig. 13(D) is a diagram showing an example of the disparity amount along I-I' in Figs. 13(B) and (C).

[0220] 13D indicates the amount of parallax D1, which is the first distance information Idist1, and point data p2 in FIG. 13D indicates the amount of parallax D2, which is the second distance information Idist2. As described above, the first distance information Idist1 contains errors due to changes in the image capturing device 1000 over time.

[0221] On the other hand, the second distance information Idist2 is hardly affected by changes over time in the image capturing device 1000. Here, by calculating the ratio D2 / D1 of the parallax amounts as the amount of parallax change, it is possible to create a correction coefficient that can correct the parallax amount D1, which includes the effects over time.

[0222] In step S1132, the CPU of the distance measurement device 110 performs correction information calculation processing. That is, in step S1132, similar to the first embodiment, surface fitting by polynomial approximation on the xy plane is performed using the acquired parallax amount ratio D2 / D1, which is discrete with respect to the angle of view, to estimate the continuous parallax amount ratio D2 / D1. Here, the approximate surface data of the calculated parallax amount ratio D2 / D1 is set as a first correction value Ic1 (first correction information).

[0223] 13(E) is a diagram showing an example of data 1003 obtained by interpolating between angles of view, and in Fig. 13(E), 1002 is the ratio D2 / D1 of the parallax amounts acquired at each data acquisition coordinate 601. 1003 indicated by a dashed line is data obtained by interpolating between angles of view by fitting with polynomial approximation using the difference of 1002, which is the ratio D2 / D1 of the parallax amounts, and corresponds to the first correction value Ic1 (first correction information).

[0224] Steps S1140 to S1160 are the same processes as steps S340 to S360 in Fig. 3(B). That is, the first correction value Ic1 (first correction information) calculated in step S1130 is subjected to a reliability score determination process in step S1140 in Fig. 11(B). Furthermore, a second correction value generation process is executed in step S1150, and a correction process is executed in step S1160 to correct the amount of parallax using the second correction value.

[0225] As described above, the distance measurement devices of the first to fourth embodiments are mounted on a moving body, or are connected wirelessly to the moving body to control the moving body. The moving body has a control means such as an ECU for issuing a warning or controlling the moving operation of the moving body based on the first distance information corrected by the correction means.

[0226] That is, when the control means of the moving body detects based on the first distance information that the distance between the moving body and an obstacle or the like is less than a predetermined value, it issues a warning or controls the moving body to change its direction of travel, slow down, or stop in order to avoid the obstacle.

[0227] The present invention has been described above in detail based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible based on the spirit of the present invention, and these are not excluded from the scope of the present invention.

[0228] The present invention also includes those that realize the functions of the above-described embodiments using at least one processor or circuit such as a CPU, etc. Also, it is possible to use multiple processors to perform distributed processing.

[0229] In order to realize part or all of the control in the above embodiments, a computer program that realizes the functions of the above embodiments may be supplied to a distance measurement device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the distance measurement device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. The present invention also includes the following combinations.

[0230] (Configuration 1) A distance measurement device comprising: a first acquisition means for acquiring first distance information; a second acquisition means for acquiring second distance information; a first correction value generation means for calculating a first correction value for correcting the first distance information based on the second distance information; a reliability score determination means for calculating a reliability score indicating the reliability of the first correction value and determining the reliability of the first correction value based on the reliability score; a second correction value generation means for generating a second correction value from the first correction value when the reliability score is equal to or greater than a predetermined threshold and the first correction value was acquired within a predetermined valid time interval; and a correction means for correcting the first distance information using the second correction value.

[0231] (Configuration 2) The distance measurement device according to configuration 1, wherein the effective time interval is set based on moving body information relating to the moving body.

[0232] (Configuration 3) The distance measurement device according to configuration 2, wherein the moving body information includes at least one of the speed and direction change of the moving body.

[0233] (Configuration 4) A distance measurement device described in any one of configurations 1 to 3, characterized in that the second correction value generation means sets the first correction value having the largest reliability score among multiple first correction values ​​determined to be reliable by the reliability score determination means within the valid time interval as the second correction value.

[0234] (Configuration 5) A distance measurement device described in any one of configurations 1 to 4, characterized in that the second correction value generation means generates the second correction value by averaging or weighting the first correction values ​​determined to be reliable by the reliability score determination means within the valid time interval.

[0235] (Configuration 6) A distance measurement device described in any one of configurations 1 to 5, characterized in that if there is a difference of a predetermined value or more between a statistically representative correction value of multiple first correction values ​​within the valid time interval whose reliability scores are equal to or greater than the threshold and the latest first correction value, the distance measurement device determines that the reliability score is unreliable even if it is equal to or greater than the threshold.

[0236] (Configuration 7) A distance measurement device described in any one of configurations 1 to 6, characterized in that the second correction value generation means does not update the second correction value if the reliability score determination means determines that the second correction value is unreliable.

[0237] (Configuration 8) A distance measurement device according to any one of configurations 1 to 7, characterized in that the first acquisition means acquires the first distance information using an imaging element having a distance measurement function using an imaging surface phase difference ranging method.

[0238] (Configuration 9) The distance measurement device according to configuration 8, wherein the second correction value corrects the amount of defocus.

[0239] (Configuration 10) The distance measurement device according to any one of configurations 1 to 9, wherein the first acquisition means acquires the first distance information using a stereo camera.

[0240] (Configuration 11) The distance measurement device according to configuration 10, wherein the second correction value corrects the amount of parallax.

[0241] (Configuration 12) A moving body characterized by having a control means for issuing a warning or controlling the moving operation of the moving body based on the first distance information corrected by the correction means of the distance measuring device described in any one of the configurations.

[0242] (Method) A distance measurement method comprising: a first acquisition step of acquiring first distance information; a second acquisition step of acquiring second distance information; a first correction value generation step of calculating a first correction value for correcting the first distance information based on the second distance information; a reliability score determination step of calculating a reliability score indicating the reliability of the first correction value and determining the reliability of the first correction value based on the reliability score; a second correction value generation step of generating a second correction value from the first correction value when the reliability score is equal to or greater than a predetermined threshold and the first correction value was acquired within a predetermined valid time interval; and a correction step of correcting the first distance information using the second correction value.

[0243] (Program) A computer program for controlling each means of the distance measurement device according to any one of configurations 1 to 11 by a computer. [Explanation of symbols]

[0244] 100: Imaging device 101: Image sensor 170: Information storage means 310: First acquisition method 320: Second acquisition method 330: First correction information generating means 340: Trust score determination means 350: Second correction information generating means 360: Correction means

Claims

1. a first acquisition means for acquiring first distance information; a second acquisition means for acquiring second distance information; a first correction value generating means for calculating a first correction value for correcting the first distance information based on the second distance information; a reliability score determination means for calculating a reliability score indicating the reliability of the first correction value and determining the reliability of the first correction value based on the reliability score; a second correction value generating means for generating a second correction value from the first correction value when the reliability score is equal to or greater than a predetermined threshold and the first correction value is acquired within a predetermined valid time interval; a correction means for correcting the first distance information using the second correction value; A distance measurement device comprising:

2. 2. The distance measurement device according to claim 1, wherein the effective time interval is set based on mobile unit information relating to the mobile unit.

3. 3. The distance measurement device according to claim 2, wherein the moving body information includes at least one of a speed and a change in direction of the moving body.

4. 2. The distance measurement device according to claim 1, characterized in that the second correction value generation means sets the first correction value having the largest reliability score among the plurality of first correction values ​​determined to be reliable by the reliability score determination means within the valid time interval as the second correction value.

5. 2. The distance measurement device according to claim 1, wherein the second correction value generation means generates the second correction value by averaging or weighting the first correction values ​​determined to be reliable by the reliability score determination means within the valid time interval.

6. 2. The distance measurement device of claim 1, characterized in that if there is a difference of a predetermined value or more between a statistically representative correction value of multiple first correction values ​​within the valid time interval whose reliability scores are equal to or greater than the threshold and the latest first correction value, the device determines that the reliability score is unreliable even if it is equal to or greater than the threshold.

7. 2. The distance measurement device according to claim 1, wherein the second correction value generation means does not update the second correction value when the reliability score determination means determines that the second correction value is unreliable.

8. 2. The distance measurement device according to claim 1, wherein the first acquisition means acquires the first distance information using an image sensor having a distance measurement function based on an image plane phase difference distance measurement method.

9. 9. The distance measurement device according to claim 8, wherein the second correction value corrects a defocus amount.

10. 2. The distance measurement device according to claim 1, wherein the first acquisition means acquires the first distance information using a stereo camera.

11. 11. The distance measurement device according to claim 10, wherein the second correction value corrects the amount of parallax.

12. A moving body characterized by having a control means for issuing a warning or controlling the movement of the moving body based on the first distance information corrected by the correction means of the distance measurement device described in any one of claims 1 to 11.

13. a first acquisition step of acquiring first distance information; a second acquisition step of acquiring second distance information; a first correction value generating step of calculating a first correction value for correcting the first distance information based on the second distance information; a reliability score determination step of calculating a reliability score indicating the reliability of the first correction value and determining the reliability of the first correction value based on the reliability score; a second correction value generation step of generating a second correction value from the first correction value when the confidence score is equal to or greater than a predetermined threshold and is acquired within a predetermined valid time interval; a correction step of correcting the first distance information using the second correction value; A distance measurement method comprising:

14. A computer program for controlling each means of the distance measurement device according to any one of claims 1 to 11 by a computer.

Images