CALIBRATION OF A CAMERA-BASED ALTITUDE ASSESSMENT

Abstract

Distances between spatial features, such as the clearance height of a bridge over a roadway, are determined based on camera images captured from a moving vehicle. A stream of images is acquired. The presence of bridges is detected in the images. A distance between the vehicle and the bridges is detected at which each image is captured. The height of each bridge is photometrically determined based on the detected distance, the location on the bridge, and a calibration factor. A posted height is identified in each image of a bridge. The calibration factor is adjusted according to any discrepancy between the posted height and the photometrically determined height.The revised calibration factor will be used in subsequent determinations of the monitored distances.

Description

FIELD OF TECHNOLOGY

[0001] The present invention relates generally to advanced driver assistance systems in transportation vehicles and, more particularly, to calibrating image analysis processes for detecting the sizes of objects in camera images, such as the size of a clearance under a bridge overpass. BACKGROUND OF THE INVENTION

[0002] Advanced driver assistance systems can often utilize cameras or other image sensors to obtain visual information about the external environment around a motor vehicle for the purpose of identifying obstacles and / or assisting in guiding the vehicle along a travel path. In some cases, camera images can be analyzed to extract signage text information, such as a clearance height under an upcoming bridge overpass.

[0003] Camera images can also be analyzed to derive an actual distance between a pair of spatial features depicted in the images. The pair of spatial features can be opposite edges of a specific object (e.g., to detect an object's size) or can be associated with different objects (e.g., to detect a separation distance between the objects). A well-known function involves detecting the clearance height (above ground) of bridge overpasses under which a vehicle will soon pass when there is no signage indicating the bridge's clearance. Once the bridge height is obtained, it can be compared to the height of the host vehicle, allowing the driver to be warned whenever insufficient clearance may be present.This applies to commercial trucks as well as passenger cars and trucks that may also tow a trailer. The vehicle and / or trailer may be loaded with cargo or have other tall structures, such as roof-mounted items like air conditioning units, vents, antennas, or luggage racks, that may be prone to striking an overpass. It is known in the art to obtain vehicle height using manual input from the driver or through automatic detection (which may be onboard the vehicle or obtained from roadway infrastructure).

[0004] A forward-facing camera can be used to identify an approaching bridge. If not otherwise available, the bridge height can be calculated from camera images based on geometric analysis. For example, an apparent angular clearance measured between the lowest edge of the bridge and the underlying road surface, along with the distance (i.e., clearance) between the vehicle and the bridge when the image is captured, can be used to calculate an actual measurement (e.g., clearance). Other equivalent calculations have also been used.

[0005] When determining an actual size based on a camera image, the calculations incorporate calibration factors related to the camera's optical properties, such as focal length, magnification, and focal plane. Some such factors may drift over time or as a result of varying environmental conditions. For example, an assumed reference frame for analyzing images (e.g., an image location nominally representing a horizontal extension from the vehicle at ground level) may change as a result of cargo loading. Measured variables (e.g., distance) may be obtained from vehicle sensors (e.g., a wheel rotation sensor), which also require additional calibrations. However, the known calibrations have not provided sufficient accuracy over sufficiently long periods of use. SUMMARY OF THE INVENTION

[0006] The present invention introduces continuous recalibration that improves the accuracy of image processing for detecting actual distances depicted in camera images. Reference data is obtained whenever signage indicating an actual bridge height for that bridge is posted near a bridge overpass. Although the bridge clearance height is already known from the signage, a value for the clearance height is calculated from the apparent quantity in the camera image data using the currently active calibration values. Automatic recalibration is performed by adjusting one or more calibration factors when a discrepancy exists between the posted height and the calculated height.

[0007] In one aspect of the invention, a method is provided for monitoring distances between spatial features based on camera images in a moving vehicle. A stream of images is acquired. The presence of bridge overpasses is detected in the images. A distance between the vehicle and the bridge overpasses is detected at which each respective image is acquired. A height of each respective bridge overpass is photometrically determined according to the respective detected distance, a respective location on the respective bridge overpass, and a calibration factor. A posted height is detected in a respective image of a respective one of the bridge overpasses. The calibration factor is revised according to a discrepancy between the respective posted height and the respective photometrically determined height.The revised calibration factor is used for subsequent determinations of the monitored distances. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a schematic diagram showing a vehicle approaching a bridge overpass having a posted height. Fig. Figure 2 is a schematic diagram showing a geometric relationship between an apparent visual angle, a distance, and an actual size. Fig. Figure 3 illustrates the trigonometric relationship between visual angle, distance, and actual size. Fig. 4 shows image data for a bridge overpass. Fig. Figure 5 is a block diagram showing a preferred apparatus of the invention. Fig. Figure 6 shows a data table obtained by collecting estimated bridge heights. Fig. 7 is a block diagram showing a vehicle configured to implement the invention. Fig. Figure 8 is a block diagram showing a preferred apparatus for updating a calibration factor. Fig. 9 is a flowchart showing a method for monitoring a bridge height. Fig. Figure 10 is a flowchart showing a method that updates calibration factors while monitoring a bridge height. DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0008] With reference to Fig. 1, a vehicle 10 is shown traveling on a road surface 11 while approaching a bridge overpass 12. A traffic sign 13 posted on or near the bridge overpass 12 includes text information informing a driver of the clearance between a bottom edge of the bridge overpass 12 (a first point) and a position on the road surface 11 below the bridge overpass 12 (a second point). An automated system using camera images can compare the posted clearance to a known vehicle height to determine if there is a possibility that the clearance is insufficient. The present invention further utilizes the clearance information from the traffic sign 13 to calibrate or recalibrate camera-based scene analysis performed by automated systems within the vehicle 10.

[0009] Fig. 2 illustrates how a bridge overpass height may be photometrically determined using camera images captured within a vehicle 14. Photometric analysis may be based on geometric (e.g., trigonometric) techniques or an equivalent. The vehicle 14 is traveling on a road surface 15 toward a bridge overpass 16 having a overpass height (H) 17 between the road surface 15 and an underside of the bridge overpass 16. A camera 18 captures images having a field of view including the bridge overpass 16, the road surface 15, and surrounding areas. Within the field of view, beams 20 and 21 extend from the vehicle 14 toward the bridge overpass 16. Beam 20 may be horizontal (i.e., parallel to the road surface 15), and beam 21 is coincident with a bottom edge of the bridge overpass 16.The bottom edge is preferably the lowest edge so that it defines the most restrictive clearance (which is usually located on the lateral side of the bridge facing incoming traffic).

[0010] When each image is captured, the vehicle 14 is at a distance (R) shown by a line 22 between the vehicle 14 and an entry point under the bridge overpass 16. Using trigonometric relationships as in Fig. 3, an angular quantity θ and a distance R can be used to derive the height H according to the tangent of the angle θ. In particular, the height H is equal to the tangent of the angle θ multiplied by the distance R. There are many ways to implement the use of this photometric determination. For example, the angle θ can be defined according to the horizontal ray 20, as in Fig. 2, which results in the height determination being measured from the height of the camera 18. Consequently, a final value H for the clearance height would have to be increased by the camera height above the ground. In other embodiments, the rays 20 and 21 defining the optical angle θ can be determined according to the locations in the camera image of the lower end of the bridge overpass and the upper surface of the road below the bridge overpass. For example, Fig. 4 shows a camera image 25 showing a road surface 26, a bridge overpass 28, and an intermediate clearance 29. A camera may typically provide a fixed field of view, with specific locations in the field of view (e.g., specific pixel rows) corresponding to fixed angular directions relative to the vehicle. Corresponding angular heights may be mapped to specific pixel rows in the image 25, as shown by scale 27. Thus, in some embodiments, the angle θ may be found by identifying which pixel row contains a bottom edge of the bridge overpass and which pixel row contains the road surface below the identified bottom edge. The image distance between the two pixel rows (e.g., the difference between the corresponding row values ​​read from scale 27) determines the angle θ.

[0011] Fig. Figure 5 shows a block diagram for an implementation of a clearance notification system based on camera images acquired by a camera 30. If signage providing a bridge height is present, the clearance is detected by a sign recognition block 31, which analyzes a scene to read the posted clearance. If no signage providing any clearance information is detected, images from the camera 30 are provided to an image analyzer 33 to determine an image size or angle of view corresponding to the point-to-point clearance features (e.g., the ground and the bottom edge of the overpass).The corresponding distance to the bridge overpass from the vehicle is determined in a block 34, and the distance R, along with the image size or angle of view, is provided to a calculation block 35 for performing the photometric analysis, from which a clearance height is derived. A clearance height collected in a block 32 (based on the signage 31 or a computer 35) is provided to a comparison block 36. A known vehicle height 37, which can be derived manually or automatically, is also provided to the comparison block 36, and whenever a difference between the clearance height and the vehicle height is less than a predetermined threshold, a notification action or a control action is initiated by the comparison block 36.

[0012] As bridge monitoring progresses, a database of bridge clearance heights, including both the respective posted heights and the respective estimated heights calculated for corresponding bridge overpasses using camera images, is compiled for calibration purposes. Thus, a database 40 may include recorded entries 41 and 42 compiling data for respective bridges (denoted i and i+1). For bridge i, a posted height x1 is recorded, and estimated clearance heights x2, x3, and x4 are recorded. Using camera images acquired at different times from different distances, multiple height estimates can be calculated for a single approach to a specific bridge overpass. The multiple estimates can be combined for improved accuracy (e.g., by using a mean or median value).For bridge i+1, a posted elevation y1 is recorded, and an estimated clearance height y2 is recorded. A single estimate is all that is needed to compare to the posted elevation, potentially updating a calibration factor, as described below.

[0013] Fig. Figure 7 shows a vehicle 45 configured to estimate clearance heights using data collected during vehicle operation. An image sensor, such as a camera 46, is mounted on the vehicle 45 to capture a stream of images directed outwardly forward from the vehicle 45. Images may also be captured using a rearward-facing camera 47. The cameras 46 and 47 are coupled to a controller 50, which may consist of a microcontroller-based electronic module configured in accordance with the present invention.

[0014] A time-of-flight sensor 48 (e.g., a radar or lidar) is mounted on the vehicle 45 for determining a distance between the vehicle 45 and various objects or surfaces that may be depicted in the captured images. A human-machine interface 51 is coupled to the controller 50 to enable manual interaction and communication with a user (e.g., inputting a height of a top portion of the vehicle and providing notifications to a driver when insufficient clearance is detected). A wheel sensor 52 is also coupled to the controller 50 for use in measuring a distance between a location where respective images are captured and the locations of objects depicted in the captured images. A powertrain control module (PCM) 53 is coupled to the controller 50, which may also be useful for measuring distance.The PCM 53 may also be configured to take evasive action in response to a determination that the vehicle 50 is approaching a bridge overpass with insufficient clearance.

[0015] Fig. 8 is a block diagram showing an example structure for updating calibration factors for use in calculating clearance heights (or for calculating any other distances between two spatial features on captured images). An estimated height in a block 55, calculated for a specific bridge visible in a corresponding image, is sent to an input of a difference block 57. A signposted height 56, detected from signage found in the captured image, is provided to another input of the difference block 57. Based on the difference between the estimated height and the signposted height, an adjustment block 58 calculates an adjustment to be made to a calibration factor 60.The updated calibration factor 60 is input to a calculation block 61, which performs actual height calculations using image data and distance data to estimate clearance heights of bridge overpasses for which there is no posted height (or to detect actual sizes of any other imaged objects, as desired).

[0016] Fig. Figure 9 shows a flowchart of a preferred method for monitoring bridge overpass heights and providing information to a driver whenever a clearance height may be insufficient. In step 65, continuously acquired camera images are checked for the presence of a bridge overpass. If no bridge is detected, step 65 continues to monitor the stream of images. If a bridge overpass is detected in step 65, each acquired image is analyzed in step 66 to determine if any signage with a clearance height posted for the bridge overpass is present.If no posted height is present, the bridge overpass clearance is estimated in step 70 according to a measured distance between the vehicle and the overpass and the relative positions of the road surface and the underside of the overpass shown in respective image data. In step 71, the estimated overpass clearance is compared to the nominal height of the vehicle to determine if sufficient clearance exists. If the clearance is sufficient, the process completes in step 72 (and once the encounter with the current overpass is complete, a return to step 65 may occur). If insufficient clearance is found in step 71, the driver may be notified in step 73 and / or automatic intervention, such as slowing or stopping the vehicle, may occur.

[0017] If a marked height is found in step 66, the marked height is recognized and used in step 67. In particular, the marked height can be used to determine a calibration feature (e.g., in Fig. 8) in parallel with using the posted altitude in step 71 to determine the margin instead of the estimated altitude.

[0018] Fig.10 shows a flow diagram of another preferred method for updating the calibration factors after detecting a posted elevation in step 67. In step 75, acquired images are associated with respective timestamps, allowing them to be associated with contemporaneous range detections in step 76. The range detection contemporaneous with a respective acquired image may be determined at the same time (e.g., using time-of-flight data) or may be obtained at a later time after the vehicle has actually traveled the distance from the image location and the bridge overpass. In step 77, a decision is made to determine whether there will be additional attempts to obtain acquired images and range data for additional locations during the vehicle's movement toward the same bridge overpass. If so, additional images and ranges may be acquired in steps 75 and 76.Once no more images are available, the estimated clearance heights are calculated in step 78 for each respective acquired image using the current value of the calibration factors. In step 79, a check is performed to determine if a difference between the posted clearance height and the estimated clearance heights is greater than a threshold. If not, no adjustment of the calibration factors is required and the process ends at step 80. Otherwise, in step 81, at least one new calibration factor is calculated to reduce the difference. In step 82, the new calibration factor is saved and then used in subsequent calculations to estimate clearance heights of bridge overpasses or other distances between spatial features in acquired images.

Claims

[1] A method for monitoring distances between spatial features based on camera images in a moving vehicle, comprising the following steps: Capturing a stream of images; Detecting the presence of bridge overpasses in the images; Detecting a distance between the vehicle and the bridge overpasses at which each respective image is captured; photometrically determining a height of each respective bridge overpass according to the respective detected distance, a respective location on the respective bridge overpass and a calibration factor; Recognizing a signposted height of each of the bridge overpasses in a respective image; Revising the calibration factor in accordance with any discrepancy between the respective signposted altitude and the respective photometrically determined altitude; and Use the revised calibration factor for subsequent determinations of the monitored distances. [2] The method of claim 1, wherein the subsequent determinations of the monitored distances include determining a height of bridge overpasses for which there is no respective posted height. [3] The method of claim 1, wherein the step of revising the calibration factor is contingent upon the discrepancy being a height difference greater than a threshold. [4] The method of claim 1, wherein the step of photometrically determining the height comprises detecting a vertical pixel position of the respective location on the respective bridge overpass in the respective image. [5] The method of claim 4, wherein the vertical pixel position of the respective location is compared with a vertical position of a floor covering. [6] The method of claim 1, wherein the distance is detected according to signals from a wheel sensor during travel of the moving vehicle between a position at which a respective image is captured and a position at which the moving vehicle is located on one side of the respective bridge overpass. [7] The method of claim 1, wherein the distance is detected according to time-of-flight signals reflected between the moving vehicle and the respective bridge overpass at a time at which a respective image is captured. [8] Transport vehicle comprising: an image sensor for capturing a stream of images directed outward from the transport vehicle; a location sensor for detecting a distance between the transport vehicle and a respective bridge overpass depicted in a respective image in the stream of images; and a controller configured to (A) detect a presence of the respective bridge overpass, (B) photometrically determine a height of the respective bridge overpass according to the respective detected distance, a respective location on the respective bridge overpass, and a calibration factor, (C) detect a posted height of the respective bridge overpass in the respective image, (D) revise the calibration factor according to a discrepancy between the respective posted height and the respective photometrically determined height, and (E) employ the revised calibration factor to make determinations of point-to-point distances in subsequent images in the stream of images. [9] A transport vehicle according to claim 8, wherein the point-to-point distances in subsequent images comprise clearance heights of respective bridge overpasses. [10] A transport vehicle according to claim 8, wherein the point-to-point distances in subsequent images comprise sizes of objects depicted in the subsequent images. [11] A transport vehicle according to claim 8, wherein revising the calibration factor is dependent on the discrepancy being a height difference greater than a threshold value. [12] The transport vehicle of claim 8, wherein photometrically determining the height comprises detecting a vertical pixel position of the respective location on the respective bridge overpass in the respective image. [13] Transport vehicle according to claim 12, wherein the vertical pixel position of the respective location is compared with a vertical position of a floor covering. [14] A transport vehicle according to claim 8, further comprising a wheel sensor for detecting the distance between a position at which a respective image is captured and a position at which the moving vehicle is located on one side of the respective bridge overpass. [15] A transport vehicle according to claim 8, further comprising a time-of-flight sensor for detecting the distance according to signals reflected between the transport vehicle and the respective bridge overpass at a time when the respective image is captured.

Images