Calibration of camera-based height evaluation

Abstract

The present disclosure provides calibration of camera-based height assessment. Distances between spatial features, such as the headroom height of a bridge above a road, are determined based on camera images captured in a moving vehicle. An image stream is captured. And identifying the existence of the bridge overpass in the image. A range between the vehicle capturing each respective image and the bridge overpass is detected. And determining the height of each corresponding bridge flyover in a photometric measurement mode according to the corresponding detection range, the corresponding position on the corresponding bridge flyover and the calibration factor. A marking height is identified in a respective image of a respective one of the bridge overpasses. The calibration factor is corrected as a function of a difference between the respective marked height and the respective photometric height. In a subsequent determination of the monitored distance, the corrected calibration factor is employed.

Description

Technical Field

[0001] The present invention relates generally to advanced driver assistance systems in transportation vehicles and, more particularly, to calibrating an image analysis process to recognize the size of an object in a camera image, such as the size of the clearance below a bridge overpass. Background Art

[0002] Advanced driver assistance systems may typically employ cameras or other image sensors to obtain visual information about the external environment surrounding a motor vehicle for use in identifying obstacles and / or assisting in guiding the vehicle along a path of travel. In some cases, camera images may be examined to extract posted text information, such as the clearance height under an upcoming bridge overpass.

[0003] The camera image can also be analyzed to derive the actual distance between a pair of spatial features depicted in the image. The pair of spatial features can be the relative edges of a particular object (i.e., for detecting the size of the object), or can be associated with different objects (i.e., for detecting the separation distance between objects). A known function involves detecting the clearance height (above the ground) of a bridge overpass, under which the vehicle will travel when there is no sign indicating bridge clearance. Once the bridge height is obtained, it can be compared to the height of the host vehicle so that the driver can be warned whenever there may be a lack of clearance. This applies to commercial trucks as well as passenger cars and trucks, which may also tow trailers. The vehicle and / or trailer may be loaded with cargo or have other high-rise structures, such as roof-mounted items that may be susceptible to hitting the overpass, such as air conditioning units, vents, antennas, or luggage racks. It is known in the art to obtain vehicle height using manual input from the driver or by automatic detection (which can be obtained on the vehicle or from road infrastructure).

[0004] A front-facing camera can be used to identify approaching bridges. When not otherwise available, the height of the bridge can be calculated from the camera image based on geometric analysis. For example, the apparent angular size of the clear space measured between the lowest edge of the bridge and the underlying road surface when the image was captured and the range (i.e., distance) between the vehicle and the bridge can be used to calculate the actual size (e.g., clear height). Other equivalent calculations are also used.

[0005] When determining the actual size based on the camera image, the calculation involves calibration factors related to the optical properties of the camera, such as focal length, magnification and focal plane. Some of these factors may drift over time or due to changing environmental conditions. For example, the assumed reference frame used to analyze the image (e.g., the image position nominally representing a horizontal extension from the vehicle on the ground) may change due to cargo load. The measured variable (e.g., distance) can be obtained from vehicle sensors (e.g., wheel rotation sensors) which also require additional calibration. However, known calibrations do not provide sufficient accuracy over sufficiently long periods of use. Summary of the invention

[0006] The present invention introduces continuous recalibration which improves the accuracy of the image processing used to detect the actual distance as depicted in the camera image. Whenever a sign is posted near a bridge overpass, reference data is obtained giving the actual bridge height of the corresponding bridge. Even if the bridge clearance height is already known from the sign, the value of the clearance height can be calculated from the apparent size in the camera image data using the currently active calibration value. When there is a difference between the posted height and the calculated height, automatic recalibration is performed by adjusting one or more calibration factors.

[0007] In one aspect of the invention, a method for monitoring distances between spatial features based on camera images in a moving vehicle is provided. A stream of images is captured. The presence of a bridge interchange is identified in the images. The range between the vehicle capturing each respective image and the bridge interchange is detected. The height of each respective bridge interchange is photometrically determined based on the respective detected range, the respective position on the respective bridge interchange, and a calibration factor. A marked height is identified in a respective image of a respective one of the bridge interchanges. The calibration factor is corrected based on a difference between the respective marked height and the respective photometrically determined height. The corrected calibration factor is employed in a subsequent determination of the monitored distance. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a schematic diagram showing a vehicle approaching a bridge overpass having a marked height.

[0009] Figure 2 is a schematic diagram showing the geometric relationship between apparent viewing angle, range and actual size.

[0010] Figure 3 The triangular relationship between viewing angle, range and actual size is shown.

[0011] Figure 4 Image data of a bridge overpass is shown.

[0012] Figure 5 is a block diagram showing a preferred apparatus of the present invention.

[0013] Figure 6 A table of data obtained by collecting estimated bridge heights is shown.

[0014] Figure 7 is a block diagram illustrating a vehicle configured to implement the present invention.

[0015] Figure 8 is a block diagram illustrating a preferred apparatus for updating calibration factors.

[0016] Fig. 9 is a flow chart illustrating a method for monitoring bridge height.

[0017] Fig.10 is a flow chart illustrating a method of updating a calibration factor during monitoring of bridge height. DETAILED DESCRIPTION

[0018] refer to Figure 1 , showing a vehicle 10 traveling on a road surface 11 approaching a bridge overpass 12. A traffic sign 13 marked on or near the bridge overpass 12 includes text information that informs the driver of the clearance height between the bottom edge of the bridge overpass 12 (a first point) and a location on the road surface 11 below the bridge overpass 12 (a second point). An automated system using the camera image can compare the indicated clearance height to a known vehicle height to determine if insufficient clearance height is likely. The present invention also utilizes the clearance height information from the traffic sign 13 to calibrate or recalibrate the camera-based scene analysis performed by an automated system within the vehicle 10.

[0019] Figure 2 How to use camera images captured within a vehicle 14 to photometrically determine bridge clearance height is shown. The photometric analysis can be based on geometric (e.g., trigonometric) techniques or equivalent techniques. The vehicle 14 moves on a road surface 15 toward a bridge overpass 16, with a clearance height (H) 17 between the road surface 15 and the bottom side of the bridge overpass 16. A camera 18 captures an image with a field of view that includes the bridge overpass 16, the road surface 15, and the surrounding area. Within the field of view, rays 20 and 21 extend from the vehicle 14 toward the bridge overpass 16. Ray 20 can be horizontal (i.e., parallel to the road surface 15), and ray 21 coincides with the bottom edge of the bridge overpass 16. The bottom edge is preferably the lowest edge so that it defines the most restrictive clearance (which is typically on the side of the bridge facing oncoming traffic).

[0020] When the corresponding image is captured, the vehicle 14 is at a range (R) as shown by the line 22 between the vehicle 14 and the entry point under the bridge overpass 16. Figure 3 Using the trigonometric relationship shown in FIG. 1 , the angle magnitude θ and the range R can be used to derive the height H from the tangent of the angle θ. Specifically, the height H is equal to the tangent of the angle θ multiplied by the range R. There are many ways to implement the use of this photometric determination. For example, the angle θ can be determined based on Figure 2 , which results in the height determination being measured from the height of the camera 18. Therefore, it will be necessary to increase the final value of the headroom height H by the height of the camera above the ground. In other embodiments, the rays 20 and 21 defining the optical angle θ may be determined based on the position of the lower end of the bridge overpass and the top surface of the roadway below the bridge overpass in the camera image. For example, Figure 4 A camera image 25 depicting a road surface 26, a bridge overpass 28, and an intermediate clearance height 29 is shown. A camera can typically provide a fixed field of view, where a specific location in the field of view (e.g., a specific row of pixels) corresponds to a fixed angular direction relative to the vehicle. The corresponding angular height can be mapped to a specific row of pixels in the image 25, as shown by scale 27. Thus, in some embodiments, the angle θ can be found by identifying which row of pixels contains the lowest edge of the bridge overpass and which row of pixels contains the road surface below the identified lowest edge. The image distance between two rows of pixels (e.g., the difference between the corresponding row values ​​read from scale 27) determines the angle θ.

[0021] Figure 5 A block diagram of an embodiment of a clearance notification system based on a camera image obtained from a camera 30 is shown. When a sign providing the bridge height is present, the clearance height is detected by a sign recognition block 31, which analyzes the scene to read the marked clearance height. When no sign is detected with any clearance height information, the image from the camera 30 is provided to an image analyzer 33 for determining the image size or angle corresponding to a point-to-point clearance feature (e.g., the ground and bottom edges of an overpass). The corresponding range from the vehicle to the bridge overpass is determined in block 34, and the range R is provided along with the image size or angle to a calculation block 35 to perform a photometric analysis to produce the clearance height. The clearance height collected in block 32 (based on the sign 31 or calculator 35) is provided to a comparison block 36. A known vehicle height 37, which may be manually or automatically derived, is also provided to the comparison block 36, and the comparison block 36 initiates a notification action or a control action whenever the difference between the clearance height and the vehicle height is less than a predetermined threshold.

[0022] When bridge monitoring is performed, a database of bridge clearance heights is compiled for calibration purposes, the database including both the corresponding marked heights and the corresponding estimated heights calculated for the corresponding bridge interchange using camera images. Thus, database 40 may include record entries 41 and 42 that compile data for the corresponding bridges (denoted as i and i+1). For bridge i, the marked height x1 is recorded, and the estimated clearance heights x2, x3, and x4 are recorded. Multiple height estimates for a single approach to a particular bridge interchange can be calculated using camera images obtained from different ranges at different times. Multiple estimates can be combined to improve accuracy (e.g., by using an average or median). For bridge i+1, the marked height y1 is recorded, and the estimated clearance height y2 is recorded. A single estimate is necessary to compare with the marked height in order to potentially update the calibration factor, as described below.

[0023] Figure 7 A vehicle 45 is shown that is configured to estimate headroom using data collected during vehicle operation. An image sensor, such as camera 46, is mounted on vehicle 45 to capture a stream of images directed toward the exterior of vehicle 45. A rear-facing camera 47 may also be used to capture images. Cameras 46 and 47 are coupled to a controller 50, which may include a microcontroller-based electronic module configured in accordance with the present invention.

[0024] A time-of-flight sensor 48 (e.g., a radar or a lidar) is mounted on the vehicle 45 for determining the range between the vehicle 45 and different objects or surfaces that may be depicted in the captured images. A human-machine interface 51 is coupled to the controller 50 to enable manual interaction and communication with a user (e.g., inputting the height of the topmost portion of the vehicle and providing a message to the driver when insufficient clearance is detected). Wheel sensors 52 are also coupled to the controller 50 for measuring the range between the location where the corresponding image was captured and the location of the object depicted in the captured image. A powertrain control module (PCM) 53 is coupled to the controller 50, which can also be used to measure the range. The PCM 53 can also be configured to take evasive action in response to the determination that the vehicle 50 is driving into a bridge overpass with insufficient clearance.

[0025] Figure 8is a block diagram showing an example structure for updating calibration factors for use in calculating clear height (or for use in calculating any other distance between two spatial features on a captured image). The estimated height in block 55 calculated for a particular bridge seen in the corresponding image is sent to one input of a difference block 57. The posted height 56 ​​identified from the sign found in the captured image is provided to another input of the difference block 57. Based on the difference between the estimated height and the posted height, an adjustment block 58 calculates the adjustment to be made to the calibration factor 60. The updated calibration factor 60 is input to a calculation block 61 which uses the image data and the range data to perform an actual height calculation to estimate the clear height of a bridge overpass that does not have a posted height (or for detecting the actual size of any other imaged object as desired).

[0026] Fig. 9 A flow chart of a preferred method for monitoring bridge overpass height and for providing information to the driver whenever clearance may be insufficient is shown. In step 65, the continuously captured camera images are checked for the appearance of a bridge overpass. If no bridge is detected, step 65 continues to monitor the image stream. When a bridge overpass is detected in step 65, the corresponding captured images are analyzed in step 66 to determine whether there are any signs indicating the clearance height for the bridge overpass. If there is no indicated height, the bridge clearance height is estimated in step 70 based on the measurement range between the vehicle and the bridge overpass and the relative position of the road surface and the bottom of the bridge overpass shown in the corresponding image data. In step 71, the estimated clearance height is compared with the nominal height of the vehicle to determine whether there is sufficient clearance margin. If the margin is sufficient, the process is completed in step 72 (and once the encounter with the current bridge overpass is completed, it can return to step 65). If the margin is found to be insufficient in step 71, the driver can be notified and / or automatic intervention can occur in step 73, such as slowing down or stopping the vehicle.

[0027] When the marked height is found in step 66, it is then identified and used in step 67. Specifically, the marked height can be used to perform a calibration feature (e.g., as described in step 67) while using the marked height instead of the estimated height to determine the margin in step 71. Figure 8 ).

[0028] Fig.10A flow chart of another preferred method for updating the calibration factor after the indicated height is detected in step 67 is shown. In step 75, the captured images are associated with corresponding timestamps so that they can be associated with the contemporaneous range detection in step 76. The range detection contemporaneous with the corresponding captured images can be determined at the same time (for example, using time of flight data), or can be obtained at a later time after the vehicle has actually traveled a certain distance from the image location and the bridge interchange. A decision is made in step 77 to determine whether there will be additional attempts to obtain captured images and range data for additional locations during the movement of the vehicle toward the same bridge interchange. If so, additional images and ranges can be captured at steps 75 and 76. Once there are no more images to be obtained, an estimated clearance height is calculated for each corresponding captured image using the current value of the calibration factor in step 78. A check is performed in step 79 to determine whether the difference between the indicated height clearance and the estimated clearance height is greater than a threshold. If not, no adjustment to the calibration factor is required, and the process ends at step 80. Otherwise, at least one new calibration factor is calculated in step 81 to reduce the difference. In step 82 , the new calibration factor is stored and then used in subsequent calculations to estimate bridge overpass clearance heights or other distances between spatial features in the captured image.

Claims

1. A method for monitoring distances between spatial features based on camera images in a moving vehicle, comprising the following steps: Capture image stream; identifying the presence of a bridge overpass in the image; detecting a range between the vehicle capturing each respective image and the bridge overpass; determining photometrically the height of each of the corresponding bridge interchanges based on the corresponding detected range, the corresponding position on the corresponding bridge interchange and the calibration factor; identifying in the corresponding image a marked height of a corresponding one of the bridge interchanges; correcting the calibration factor based on a difference between the corresponding registered altitude and the corresponding photometrically determined altitude; and The corrected calibration factor is employed for subsequent determination of the monitoring distance.

2. The method of claim 1, wherein said subsequent determination of said monitored distance comprises determining the height of a bridge overpass that has no corresponding marked height.

3. The method of claim 1, wherein the step of modifying the calibration factor is conditioned upon the difference being a height difference greater than a threshold.

4. The method of claim 1, wherein the step of photometrically determining the height comprises detecting a vertical pixel position of the corresponding location on the corresponding bridge interchange in the corresponding image.

5. The method of claim 4, wherein the vertical pixel position of the corresponding position is compared with a vertical position of the ground.

6. The method of claim 1, wherein the range is detected based on a signal from a wheel sensor during travel of the mobile vehicle between a location where the corresponding image is captured and a location where the mobile vehicle is located on one side of the corresponding bridge interchange.

7. The method of claim 1, wherein the range is detected based on a time-of-flight signal reflected between the moving vehicle and the corresponding bridge overpass when the corresponding image is captured.

8. A transport vehicle comprising: an image sensor for capturing an image stream directed from outside the transport vehicle; a position sensor for detecting a range between the transport vehicle and a corresponding bridge overpass depicted in a corresponding image in the image stream; as well as a controller configured to (A) identify the presence of the corresponding bridge interchange, (B) photometrically determine the height of the corresponding bridge interchange based on the corresponding detected range, the corresponding position on the corresponding bridge interchange, and a calibration factor, (C) identify a marked height of the corresponding bridge interchange in the corresponding image, (D) correct the calibration factor based on a difference between the corresponding marked height and the corresponding photometrically determined height, and (E) use the corrected calibration factor to determine point-to-point distances in subsequent images in the image stream.

9. The transport vehicle of claim 8, wherein the point-to-point distances in subsequent images include clearance heights of corresponding bridge overpasses.

10. The transport vehicle of claim 8, wherein the point-to-point distances in subsequent images include sizes of objects depicted in the subsequent images.

11. The transport vehicle of claim 8, wherein modifying the calibration factor is conditioned upon the difference being a height difference greater than a threshold.

12. The transport vehicle of claim 8, wherein photometrically determining the height comprises detecting a vertical pixel position of the corresponding location on the corresponding bridge interchange in the corresponding image.

13. The transport vehicle of claim 12, wherein the vertical pixel position of the corresponding position is compared to a vertical position of the ground.

14. The transport vehicle of claim 8, further comprising a wheel sensor for detecting the range between a position where the corresponding image is captured and a position where the mobile vehicle is located on one side of the corresponding bridge interchange.

15. The transport vehicle of claim 8, further comprising a time-of-flight sensor for detecting the range based on a signal reflected between the transport vehicle and the corresponding bridge overpass when the corresponding image is captured.

Images