Non-rigid Stereo Vision Camera System

Abstract

To provide a long-baseline and long depth-range stereo vision system that are suitable for use in non-rigid assemblies where relative motion between two or more cameras of the system does not degrade estimates of a depth map.SOLUTION: A stereo vision system may include a processor configured to track camera parameters as a function of time to rectify images from the cameras even during fast and slow perturbations to camera positions. Factory calibration of the system is not needed, and manual calibration during regular operation is not needed, thus simplifying manufacturing of the system.SELECTED DRAWING: Figure 16

Description

[Technical field]

[0001] The present technology relates to stereo vision systems, and more particularly to stereo vision systems (e.g., stereo camera systems) having properties for improved operation in the presence of fast and slow mechanical perturbations, and methods for automatic calibration of the systems. [Background technology]

[0002] Stereo camera systems are sometimes used to obtain three-dimensional information about objects within their field of view by triangulation techniques. Conventional stereo camera systems may use rigid mounting members to fix the position of the cameras relative to each other. For example, US Pat. No. 6,399,633 discloses the use of thick metal plates as rigid mounting members for direct attachment of the cameras and their associated CMOS sensors and lenses to mechanically stabilize the relative positions of the cameras. The thickness of the plate illustrated in FIG. 5 of this document is about one-fifth of the distance between the cameras, which makes such a structure very heavy, which may not be suitable for use in systems with wide baseline stereo cameras. For example, a stereo camera system with a 1 m baseline between the cameras would require a 20 cm thick metal plate. If made from aluminum, the 20 cm thick metal plate would weigh 108 kg (1 m x 0.2 m x 0.2 m). The terms "wide baseline" and "long baseline" may be used interchangeably herein.

[0003] A problem with such existing stereo camera systems is that the individual camera modules (sometimes referred to herein as “camera sensors” or simply “cameras”) may shift and / or rotate relative to one another over time, which may cause initial calibration parameters to become inaccurate over time. It is desirable for the fields of view of the camera modules of a stereo camera system not to move relative to one another in order for the stereo camera system to remain properly calibrated. In some stereo camera systems, a relative camera orientation rotation of only 0.05 degrees may destroy the calibration of the system. Additionally, in conventional stereo camera systems, for example, the lens holder may move relative to the circuit board and / or other camera components, the relative pointing between the camera modules may change, and also the position of the circuit board and frame itself may change over time. The multiple interfaces between components in conventional stereo camera systems increase the possibility that vibration, shock, and even thermal expansion between the components may cause the fields of view of the cameras in the stereo camera system to shift over time. Relative movement of the camera components can invalidate the initial factory calibration of the system, which can cause the stereo vision data acquired by the system to be unreliable.

[0004] Such calibration issues have not been addressed or even recognized in the prior art because prior art stereo vision systems are typically either laboratory systems that are not generally subject to shock or vibration, or are used in situations such as short-range indoor robotics where highly accurate calibration is not generally considered essential. Thus, the inherent susceptibility of prior art stereo vision systems to losing calibration has not been recognized as a specific problem to be solved, because prior systems are limited to short baseline lengths and utilize large mechanical stiffeners.

[0005] Conventional automatic calibration methods typically fall into two categories: sparse keypoint and vanishing point methods. A keypoint is a point of interest that can be easily recognized in an image, such as a corner or edge, and often has an associated image descriptor. For example, the Scale Invariant Feature Transform (SIFT) method developed by David Lowe is an example of an algorithm for finding keypoints and providing a descriptor that is invariant to translation, rotation, and scaling transformations. Vanishing points are used to represent points at infinite distances. Vanishing points are sensitive to camera module orientation but not to relative translation between the camera modules, and can be used to align the orientation of a pair of stereo camera modules.

[0006] Methods for automatically calibrating a stereo vision system using a sparse keypoint approach have also been disclosed (see US Pat. No. 6,399,623 and US Pat. No. 6,399,623). The method in US Pat. No. 6,399,623 extracts keypoints, matches the keypoints in images from a first camera and a second camera, and determines camera calibration based on singular value decomposition analysis of the vertical error. The method in US Pat. No. 6,399,623 is similar to the method in US Pat. No. 6,399,623, but tracks keypoints across multiple image frames and applies 3D constraints using structure-from-motion techniques. The sparse keypoint approach may fail when keypoints are mismatched between images from a first camera and images from a second camera, which may occur in image scenes with repeating structures (e.g., picket fences, building windows, etc.). Moreover, subpixel accuracy of feature points may be required to obtain camera parameters with sufficient accuracy for long baseline and long range operation, which is often not possible with smooth or round features, or slightly defocused or blurry images. Traditional motion-derived structure computations are generally not fast enough to compensate for calibration errors on a frame-by-frame basis, and therefore are typically suited to the slow perturbations of short-baseline stereo vision systems. Another issue with sparse keypoint approaches is the sparse sampling of images, which does not provide enough information to accurately determine the intrinsic or extrinsic camera parameters. For example, a typical road scene image may have most features clustered in the center of the image, away from the sky and away from the textureless road surface, and therefore sampling is generally done in the center of the image. This clustered sampling does not provide enough information to determine with high accuracy the relative orientation of the stereo cameras (extrinsic camera parameters), much less the lens distortion coefficients (intrinsic camera parameters). These parameters are highly sensitive to distortions at the edges of the image.In other words, these conventional techniques do not compensate for camera parameters that can be important for proper calibration of a stereo vision system.

[0007] A method for automatically calibrating a stereo vision device by searching for vanishing points and correcting pitch and yaw errors has been disclosed (see, for example, U.S. Pat. No. 5,333,933). This method does not correct roll errors or relative translation vectors between the two cameras of the device. Moreover, the method requires straight and reliable road pavement markings, which may not be available or may be obscured by snow or faded by wear or sun exposure, thus limiting the angular accuracy of the device calibration.

[0008] A method for stereo vision using components mounted on the wings of a model airplane has also been disclosed (see, for example, non-patent document 1). Because the wings are flexible and may move, the authors developed a system to compensate for relative camera motion. This system requires the use of an inertial measurement unit (IMU) mounted directly to the stereo cameras to measure the acceleration and angular velocity of the cameras. The system then uses an extended Kalman filter to calculate the relative orientation and position of the cameras. One drawback of this system is that it requires IMU hardware and cannot correct the camera position from the video stream alone. Another drawback is that the IMU can be used to compensate for fast perturbations but not slow perturbations (e.g., slow shifts in the camera position), which means that the cameras may need to be manually calibrated frequently (e.g., daily). [Prior art documents] [Patent documents]

[0009] [Patent Document 1] U.S. Patent No. 6,392,688 [Patent Document 2] U.S. Patent No. 8,797,387 [Patent Document 3] U.S. Patent No. 10,097,812 [Patent Document 4] Special Publication No. 2008-509619 [Non-patent literature]

[0010] [Non-Patent Document 1] A 2018 conference paper titled "Flexible Stereo: Constrained, Non-rigid, Wide-baseline Stereo Vision for Fixed-Wing Aerial Platforms" by authors T. Hinzmann, T. Taubner and R. Siegwart Summary of the Invention [Problem to be solved by the invention]

[0011] The calibration problems described above may be exacerbated in long baseline stereo vision systems, in systems where it is not possible to connect two or more camera modules with heavy structural beams or mounting members, when the distance between the camera modules is, for example, greater than about 20 cm. For styling and optimal viewing viewpoints, it may be desirable to mount independent cameras on a non-rigid structure (e.g., a flexible structure). For example, mounting a pair of stereo cameras in a vehicle headlight, at the upper corner of a vehicle windshield, or on a vehicle side mirror would be advantageous, as such locations may be convenient for establishing a long baseline for a stereo vision system in a vehicle (e.g., a car, truck, bus, etc.). However, these locations are not rigid enough to maintain calibration over hours or days, let alone the 15-year lifespan of a typical vehicle. In some cases, it may not be practical to add rigid mechanical support structures between the headlights, the upper corners of the windshield, or the side mirrors, because the shortest path for these support structures may be blocked by the engine block, which may block the driver's view, or by the driver himself, respectively. Moreover, even if support structures can be added, the thickness and weight of such structures required to provide adequate structural support may be impractical. That is, the longer the beam, the more it may displace for the same amount of force applied to it, since the deflection displacement of the beam scales proportionally to the cube of the length of the beam. For example, for the same end force, a stereo vision system with a 2m baseline will displace 1000 times more than a stereo vision system with a 20cm baseline. This scaling has resulted in commercially available stereo vision systems being physically limited for baselines less than 20cm.

[0012] There is a need for a stereo vision system that can operate in high vibration and shock environments, that can be mounted on non-rigid structures, that can support long baseline and long range stereo vision, and that can operate with high accuracy. [Means for solving the problem]

[0013] Automatic calibration techniques suitable for long baselines and non-rigid structures have not been achieved prior to the techniques disclosed herein for at least four reasons: (1) Magnitude: By physics, beam displacement increases as the cube of the beam length (see discussion above), and therefore the magnitude of relative camera displacement increases with beam length. Additional structural support to ensure rigidity and prevent camera displacement may not be practically achievable. (2) Bandwidth: The calibration speed of an automatic calibration system should increase with increasing baseline length to account for both slow and fast variations. The processor hardware required to achieve the required computation speed may be expensive and may add undesirable complexity to the system. (3) Precision: The calibration precision required for a long baseline system may be much greater than for a short baseline system, as greater angular precision is required to estimate targets at longer distances. and (4) completeness: Conventional techniques for automatic calibration of stereo cameras may compensate for a small number of camera parameters (e.g., the relative orientation of the stereo cameras), but a full calibration would require compensation for all extrinsic and intrinsic camera parameters. Prior to the techniques presented herein, a system that addresses these four reasons has not been achieved.

[0014] The present technology described and claimed herein is directed to a stereo vision system in which no rigid mounting members are required. Vision may be achieved via multiple vision sensors (e.g., camera modules). In some aspects of the technology, a first camera module (e.g., camera sensor) and a second camera module may be placed on or attached to a structure that flexes, shifts, bends, and / or moves. As noted elsewhere herein, the terms camera sensor, camera module, and camera may be used interchangeably. A stereo camera system according to aspects of the technology may comprise a processor configured to perform active tracking and compensation for movement of the first camera relative to the second camera to provide accurate depth maps over all motion frequencies, from slow material deformations due to, e.g., temperature changes, to high speed vibration motions due to, e.g., road or engine noise. Because there is no requirement for rigid mounting of the cameras, aspects of the technology may enable wide baseline stereo configurations for long range measurements without the need for periodic manual calibration.

[0015] According to one aspect of the present technology, a stereo vision system is provided. The stereo vision system may include a first camera sensor configured to sense a first reflected energy in a first image and generate a first sensor signal based on the first reflected energy, a second camera sensor configured to sense a second reflected energy in a second image and generate a second sensor signal based on the second reflected energy, and at least one processor configured to receive the first sensor signal from the first camera sensor and the second sensor signal from the second camera sensor and generate three-dimensional (3D) data from the first sensor signal and the second sensor signal. The at least one processor may be configured to generate rectified stereo images from the first sensor signal and the second sensor signal using stereo calibration parameters, perform stereo matching on the rectified images, and perform automatic system calibration using data from the stereo images acquired by the first camera sensor and the second camera sensor. The automatic system calibration may be based on minimizing a cost function. In some embodiments of this aspect, the first reflected energy and / or the second reflected energy may comprise light energy or photons emitted from at least one electronic device (e.g., a headlight, a street light, a laser, etc.). In some embodiments of this aspect, the first reflected energy and / or the second reflected energy may comprise solar energy or photons emitted from the sun. In some embodiments of this aspect, the first reflected energy and / or the second reflected energy may comprise a combination of solar energy and light energy emitted from an electronic device.

[0016] According to another aspect of the present technology, a computer-implemented method for detection of an object in an image is provided. The method may be performed by one or more processors and may comprise identifying connected component regions on the object by color, determining an aspect ratio of each of the connected component regions, determining a distance between the connected component regions, and identifying the object as a known object based on the aspect ratio and the distance between the connected component regions. Any one or any combination of the presence, location, and size of the known object may be determined from the image.

[0017] According to another aspect of the present technology, a computer-implemented calibration method for calibrating a stereo vision system is provided. The method may be implemented by one or more processors. The stereo vision system may include a first camera sensor configured to sense a first reflected energy of a first image and generate a first sensor signal based on the first reflected energy, and a second camera sensor configured to sense a second reflected energy of a second image and generate a second sensor signal based on the second reflected energy. The method may include generating a stereo image from the first sensor signal and the second sensor signal, correcting the stereo image using stereo calibration parameters to produce a corrected stereo image, performing stereo matching on the corrected stereo image, and performing an automatic system calibration using the stereo matching results. The automatic system calibration may be based on minimizing a cost function. In some embodiments of this aspect, the first reflected energy and / or the second reflected energy may comprise light energy or photons emitted from at least one electronic device (e.g., a headlight, a street light, a laser, etc.). In some embodiments of this aspect, the first reflected energy and / or the second reflected energy may comprise solar energy or photons emitted from the sun. In some embodiments of this aspect, the first reflected energy and / or the second reflected energy may comprise a combination of solar energy and light energy emitted from an electronic device.

[0018] According to another aspect of the present technology, a non-transitory computer-readable medium is provided that stores computer-executable code that, when executed by one or more processors, can cause the one or more processors to calibrate a stereo vision system that may include a first camera sensor configured to sense a first reflected energy in a first image and generate a first sensor signal based on the first reflected energy, and a second camera sensor configured to sense a second reflected energy in a second image and generate a second sensor signal based on the second reflected energy. The method may include generating a stereo image from the first sensor signal and the second sensor signal, correcting the stereo image using stereo calibration parameters to produce a corrected stereo image, performing stereo matching on the corrected stereo image, and performing an automatic system calibration using the stereo matching results. The automatic system calibration may be based on minimizing a cost function. In some embodiments of this aspect, the first reflected energy and / or the second reflected energy may comprise light energy or photons emitted from at least one electronic device (e.g., a headlight, a street light, a laser, etc.). In some embodiments of this aspect, the first reflected energy and / or the second reflected energy may comprise solar energy or photons emitted from the sun. In some embodiments of this aspect, the first reflected energy and / or the second reflected energy may comprise a combination of solar energy and light energy emitted from an electronic device. [Brief description of the drawings]

[0019] [Figure 1A] FIG. 13 illustrates locations for mounting cameras in or on a vehicle without using or requiring large and heavy structural support materials between the cameras, in accordance with some embodiments of the present technology. [Figure 1B]FIG. 13 illustrates locations for mounting cameras in or on a vehicle without using or requiring large and heavy structural support materials between the cameras, in accordance with some embodiments of the present technology. [Figure 1C] FIG. 13 illustrates locations for mounting cameras in or on a vehicle without using or requiring large and heavy structural support materials between the cameras, in accordance with some embodiments of the present technology. [Figure 2A] FIG. 1B is a transparent or partial perspective view of the car of FIG. 1A in accordance with some embodiments of the present technology. [Figure 2B] FIG. 1C is a transparent or partial perspective view of the car of FIG. 1B, in accordance with some embodiments of the present technology. [Figure 2C] FIG. 1D is a transparent or partial perspective view of the car of FIG. 1C, in accordance with some embodiments of the present technology. [Figure 3A] 1A-1D are diagrams of forward-facing stereo vision camera modules mounted in various locations within a vehicle, in accordance with some embodiments of the present technology. [Figure 3B] 1A-1D are diagrams of forward-facing stereo vision camera modules mounted in various locations within a vehicle, in accordance with some embodiments of the present technology. [Figure 3C] 1A-1D are diagrams of forward-facing stereo vision camera modules mounted in various locations within a vehicle, in accordance with some embodiments of the present technology. [Figure 3D] 1A-1D are diagrams of forward-facing stereo vision camera modules mounted in various locations within a vehicle, in accordance with some embodiments of the present technology. [Figure 3E] 1A-1D are diagrams of forward-facing stereo vision camera modules mounted in various locations within a vehicle, in accordance with some embodiments of the present technology. [Figure 3F] 1A-1D are diagrams of forward-facing stereo vision camera modules mounted in various locations within a vehicle, in accordance with some embodiments of the present technology. [Figure 4] FIG. 2 is a diagram of a forward-facing stereo vision camera sensor mounted on a truck, in accordance with some embodiments of the present technology. [Figure 5A]FIG. 1 is a perspective view of side-facing and rear-facing camera modules of a stereo vision system, with the camera modules mounted on the side mirrors of a truck, in accordance with some embodiments of the present technology. [Figure 5B] FIG. 2 is a top view of side-facing and rear-facing camera modules of a stereo vision system, with the camera modules mounted on the side mirrors of a truck, in accordance with some embodiments of the present technology. [Figure 5C] FIG. 2 is a front elevation view of side-facing and rear-facing camera modules of a stereo vision system, with the camera modules mounted on the side mirrors of a truck, in accordance with some embodiments of the present technology. [Figure 6] 1 is a schematic diagram of cameras of a stereo vision system mounted on a traffic signal, in accordance with some embodiments of the present technology; [Figure 7] FIG. 1 is a schematic diagram of cameras of a stereo vision system mounted on a lamppost, in accordance with some embodiments of the present technology. [Figure 8] FIG. 1 is a schematic diagram of cameras of a stereo vision system mounted on a construction crane, in accordance with some embodiments of the present technology. [Figure 9] FIG. 1 is a schematic diagram of cameras of a stereo vision system mounted on a fixed-wing aircraft, in accordance with some embodiments of the present technology; [Figure 10A] FIG. 1 is a front perspective view of a camera arrangement for a stereo vision system mounted on an off-road or all-terrain vehicle, in accordance with some embodiments of the present technology. [Figure 10B] FIG. 2 is a rear perspective view of a camera arrangement of a stereo vision system mounted on an off-road or all-terrain vehicle, in accordance with some embodiments of the present technology. [Figure 11] FIG. 1 is a schematic diagram of two stereo vision system cameras mounted on a combine harvester, in accordance with some embodiments of the present technology. [Figure 12]1A-C show side elevation (A), front elevation (B), and plan view (C) of cameras of a stereo vision system mounted on an automated forklift, according to some embodiments of the present technology. [Figure 13] 1 is a block diagram of a stereo vision system in accordance with some embodiments of the present technology. [Figure 14] 14 is a block diagram of processing components of the system of FIG. 13 in accordance with some embodiments of the present technology. [Figure 15A] 4 is a flow diagram of a correction algorithm utilized in some embodiments of the present technology. [Figure 15B] FIG. 13 illustrates geometric details of the correction procedure implemented by an algorithm, in accordance with some embodiments of the present technique. [Figure 16] 15 is a block diagram of an auto-calibration engine of the processing components of FIG. 14 in accordance with some embodiments of the present technology. [Figure 17A] 17 is a block diagram of a calibration engine of the auto-calibration engine of FIG. 16 in accordance with some embodiments of the present technology. [Figure 17B] 4 is a flow diagram of a fast optimization method in accordance with some embodiments of the present technique. [Figure 17C-1] 1 is a flow diagram of an intermediate optimization method in accordance with some embodiments of the present technology. [Figure 17C-2] 17C-1 in accordance with some embodiments of the present technology. [Figure 17D-1] 4 is a flow diagram of a slow optimization method in accordance with some embodiments of the present technology. [Figure 17D-2] 4 is a flow diagram of a slow optimization method in accordance with some embodiments of the present technology. [Figure 17D-3] 4 is a flow diagram of a slow optimization method in accordance with some embodiments of the present technology. [Figure 17D-4] 4 is a flow diagram of a slow optimization method in accordance with some embodiments of the present technology. [Figure 18] 17 is a flow diagram of a procedure for the stereo image stabilization engine of the auto-calibration engine of FIG. 16 in accordance with some embodiments of the present technology. [Figure 19A] FIG. 17 is a block diagram of an absolute range calibration engine procedure of the auto-calibration engine of FIG. 16, the procedure having a non-negative disparity constraint, in accordance with some embodiments of the present technology. [Figure 19B] 11 is a graph illustrating how the number of pixels with negative disparity can vary as a function of yaw, in accordance with some embodiments of the present technology. [Figure 20A] FIG. 17 is a flow diagram of a procedure of the absolute range calibration engine of the auto-calibration engine of FIG. 16, in accordance with some embodiments of the present technique, the procedure being for an object with known dimensions. [Figure 20B] 1 illustrates an example of an image of an object with known dimensions, in accordance with some embodiments of the present technique. [Figure 21A] 17 is a flow diagram of an absolute range calibration engine procedure of the auto-calibration engine of FIG. 16, which procedure utilizes vehicle odometer measurements, in accordance with some embodiments of the present technology. [Figure 21B] 11 is a graph illustrating how disparity can vary as a function of range, in accordance with some embodiments of the present technology. [Figure 22] 17 is a flow diagram of a calibration manager procedure of the auto-calibration engine of FIG. 16 in accordance with some embodiments of the present technology. [Figure 23] FIG. 23 is a time series input diagram of the calibration manager of FIG. 22, in accordance with some embodiments of the present technology. [Figure 24] 1 is a flow diagram of a stop sign detector procedure, in accordance with some embodiments of the present technology. [Diagram 25] 15 illustrates a procedure for a stereo correspondence engine of the processing component of FIG. 14 in accordance with some embodiments of the present technique. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The inventors have developed image processing techniques that can enable a stereo vision system to produce depth maps with high quality, even in the presence of vibrations and even when the mounting structure for the system's cameras is not rigid (e.g., the mounting structure can be easily deformed from external forces). The inventors recognize that a camera auto-calibration method that can compensate for both fast and slow perturbations can enable long baseline (also referred to herein as "wide baseline") stereo vision systems to be used on dynamic platforms such as robots, cars, trucks, light posts, construction cranes, and aircraft.

[0021] For vehicles such as cars and trucks, there are mounting locations that may be preferred or even ideal for aesthetic design, for manufacturing, and / or for providing optimal viewpoints for the stereo vision system's sensors or cameras. However, these mounting locations may not be rigid or stable enough to support the strict angular tolerances of the cameras used in traditional stereo vision systems. Perturbing the relative pointing directions of the stereo cameras by even 0.05 degrees in some traditional stereo vision systems can corrupt the depth map by providing unreliable range estimators.

[0022] 1A-1C illustrate three different locations for mounting cameras in or on a vehicle without the use or need for large and heavy structural support materials between the cameras, in accordance with some embodiments of the present technology.

[0023] FIG. 1A illustrates a mounting configuration of a stereo vision system according to some embodiments of the present technology, where the left camera 100 and the right camera 102 of the system may be mounted on the top portion of the front windshield of the vehicle. This location may be desirable because the cameras 100, 102 are inside the cabin of the vehicle and may therefore be protected from dust and rain. For example, windshield wipers may be used to sweep debris (e.g., rain, snow, leaves, etc.) from the field of view of the cameras 100, 102. In some embodiments, the right camera 100 and the left camera 102 may be mounted asymmetrically and positioned to be within the sweeping zone of the vehicle's windshield wipers. In some embodiments, a typical baseline width between the right camera 100 and the left camera 102 may be about 80 cm, a width that would require impractically thick and heavy rigid structural members to connect the two cameras 100, 102 in a conventional stereo vision system. As will be appreciated, such structural members may obstruct the vision of a human driver. Conventional autonomous test vehicles with stereo cameras with baselines greater than 50 cm typically use bulky slotted aluminum members with 45 mm x 45 mm or 90 mm x 90 mm profiles to provide sufficient rigidity to maintain camera alignment for a day or sometimes longer.

[0024] FIG. 1B illustrates a mounting configuration of a stereo vision system according to some embodiments of the present technology, where the left camera 104 and the right camera 106 of the system may be mounted on the roof portion of a vehicle. The roof portion may provide an elevated vantage point that may enable the vision system to look around objects in front of the vehicle and be located away from splashes and splashes from wet pavement and road debris. In some embodiments, the cameras 104, 106 may be mounted onto a roof headliner without using extra structural material to stabilize the cameras 104, 106. Moreover, in some embodiments, the camera housings used with the cameras 104, 106 may include one or more cleaning devices (e.g., heating elements configured to defrost and / or defog, sprayers for liquid and / or air configured to remove solid debris, etc.). In some embodiments, a typical baseline width between the left camera 104 and the right camera 106 may be 1.25 m. In some implementations, the camera modules 104, 106 on the roof portion of the vehicle may have little or no impact on the styling of the vehicle. For example, in some embodiments, the cameras 104, 106 may be placed consecutively along the roof rack of the vehicle.

[0025] FIG. 1C illustrates a mounting configuration of a stereo vision system according to some embodiments of the present technology, where the left camera 108 and the right camera 110 of the system may be mounted on the side mirror of a vehicle. Side mirrors may provide the widest baseline, typically 2.1 m, for a forward-facing stereo vision system, but may experience larger perturbations than the mounting configuration illustrated in FIG. 1A and FIG. 1B. For a typical vehicle, the side mirrors are at the ends of the vehicle. A typical side mirror is mounted against an articulating joint and is a lightweight structure that is highly sensitive to internal vibration sources (e.g., the vehicle's engine) as well as external perturbations (e.g., from wind, road vibrations, etc.). Cameras mounted on side mirrors generally experience high speed as well as low speed perturbations.

[0026] 2A-2C illustrate see-through or partial perspective views of the vehicle of FIGS. 1A-1C, respectively, in accordance with some embodiments of the present technology. In some embodiments, wires 200, 202, 204, 206, 208, 210 may connect the cameras 100, 102, 104, 106, 108, 110 to electronic control units 212, 214, 216. The wires 200, 202, 204, 206, 208, 210 may be configured to transmit data to the electronic control units 212, 214, 216 and to provide power to the cameras 100, 102, 104, 106, 108, 110.

[0027] 3A-12 illustrate various embodiments of stereo vision systems on different platforms that may move or flex, may be subject to vibration and shock, may be inconvenient or costly to manually calibrate, and / or may be required to operate for years without manual calibration and maintenance. In FIG. 3A-12, the cameras of the system may be illustrated as mounted on the platform, but the electronic control unit and wiring may not be illustrated, as they may be mounted within the chassis of the platform or within the platform itself, according to some embodiments of the present technology.

[0028] 3A-3F illustrate camera modules C1, C2 for a forward-facing stereo vision system mounted at various locations in or on a vehicle according to some embodiments of the present technology. These locations typically do not provide optical rigidity to ensure alignment of the stereo vision cameras over the life of the vehicle or over the temperature range experienced by a typical vehicle, for example, during seasonal changes and / or night-to-day temperature fluctuations. FIG. 3A illustrates cameras C1, C2 mounted on a side mirror. FIG. 3B illustrates cameras C1, C2 mounted on a roof. FIG. 3C illustrates cameras C1, C2 mounted behind the windshield glass of the vehicle. FIG. 3D illustrates cameras C1, C2 mounted in the grille of the vehicle. FIG. 3E illustrates cameras C1, C2 mounted in the headlight housing of the vehicle. FIG. 3F illustrates cameras C1, C2 mounted in the fog light housing of the vehicle.

[0029] 4 illustrates camera modules C1, C2 of a forward-facing stereo vision system according to some embodiments of the present technology, where the camera modules C1, C2 are mounted on the side mirrors of a truck. The truck may have a wide baseline length of about 3m from the left side mirror to the right side mirror, which may allow for very long range stereoscopic viewing. In some embodiments, the camera modules C1, C2 may be made of CMOS sensors.

[0030] 5A-5C illustrate arrangements for side-facing and rear-facing stereo vision systems according to some embodiments of the present technology, with camera modules 500, 502 shown mounted on the right side mirror of a truck in FIG. 5A. FIG. 5A is an isometric or perspective view of the truck, FIG. 5B is a top or plan view of the truck, and FIG. 5C is a front elevation view of the truck. In some embodiments, a redundant stereo vision system may also be mounted on the left side mirror of the truck to sense objects on the left side of the truck. The redundant stereo vision system is not shown in the drawings to avoid unnecessary confusion. The two camera modules 500, 502 of the stereo vision system may be mounted vertically and spaced about 1 m apart. As will be appreciated, stereo matching may be performed to match images from cameras oriented at any angle relative to each other. For example, stereo matching of images from cameras positioned vertically relative to one another may correlate horizontal lines or rows of pixels of the images, stereo matching of images from cameras positioned horizontally relative to one another may correlate vertical lines or columns of pixels of the images, etc. The depth range of a stereo vision system may be proportional to the distance between the cameras of the system. For the system illustrated in Figures 5A-5C, the system may have a horizontal field of view 504 of 135 degrees as depicted in Figure 5B and a vertical field of view 506 of 90 degrees as depicted in Figure 5C. As will be appreciated, in other embodiments of the present technology, other fields of view are possible, for example, via different camera lenses (e.g., standard lenses, different types of wide angle lenses, etc.). Side-looking and rear-looking stereo vision configurations such as those of Figures 5A-5C may be particularly useful for detecting and therefore avoiding pedestrians, detecting and therefore avoiding cyclists, and detecting vehicles passing through.Conventional stereo vision systems typically have very poor depth range, especially in wide field of view systems, because in wide field of view systems the ability to accurately estimate angles degrades as each pixel is mapped to a wider range of angles. As will be appreciated, the depth range resolution of a stereo vision system may be proportional to the field of view of the system. For wide baseline stereo vision systems, it may be advantageous to use wide field of view cameras, because the depth range resolution of the system is inversely proportional to the baseline length of the system's stereo cameras. Thus, embodiments of the present technology may enable long range, surround view three-dimensional ("3D") sensing of objects by enabling a long stereo vision baseline (e.g., 1 m or more) compared to only about 5 cm to 10 cm baselines in conventional stereo imaging systems.

[0031] FIG. 6 illustrates two cameras 600, 606 of a stereo vision system mounted on a traffic light pole 602, according to some embodiments of the present technology. The two cameras 600, 606 may be separated by a long baseline limited by the length of the boom 602a of the traffic light pole 602. Some traffic light poles may support a stereo vision baseline length of about 5m, which may enable extremely long range depth estimation (e.g., more than 300m), which may enable measurement of queue length of vehicles in a traffic queue for intelligent switching of the traffic light 604. As will be appreciated, the boom of the traffic light pole may flex (e.g., with high winds, movement of objects hanging from the boom, road vibrations from passing large trucks nearby, etc.). Such flexures may be experienced by the cameras 600, 606, and such flexures may be exaggerated by the long pole 608 holding the cameras 600, 606. Thus, a long baseline length may cause the two cameras 600, 606 to quickly become misaligned in the typical operating environment of a traffic light pole 602.

[0032] 7 illustrates two cameras 702, 704 of a stereo vision system mounted on a lamppost 700, according to some embodiments of the present technology. The two cameras 702, 704 may be mounted vertically and pointed slightly downward to view an area of ​​interest (e.g., a pedestrian crosswalk). For example, the top edge 706 of the field of view of the camera 702 may deviate from the horizontal line by less than about 30 degrees (e.g., 20 degrees, 10 degrees, 5 degrees). The baseline distance between the two cameras 702, 704 may be several meters or more (e.g., 3m, 3.5m, 4m, 4.5m, etc.). Similar vertical configurations of cameras may be mounted on sign poles, telephone / utility poles, etc.

[0033] FIG. 8 illustrates two cameras 804, 806 of a stereo vision system mounted on a construction crane 800, according to some embodiments of the present technology. The cameras 804, 806 are mounted on a trolley 802 and may be positioned to have a downward field of view, for example, to monitor the payload 808 and the surrounding construction environment near and below the payload 808. A typical field of view 810 may be about 90 degrees to about 120 degrees and may extend in a direction parallel or nearly parallel to the boom of the crane 800. The distance between the cameras 804, 806 may be about 2 m to 5 m, depending on the desired minimum depth range and depth range resolution. A boom length of about 50 m and a crane height of about 200 m make a wide baseline, long range stereo vision system an attractive solution for monitoring the environment of a construction crane, especially when structural shock and vibration make it difficult to maintain a constant relative position between the cameras 804, 806.

[0034] FIG. 9 illustrates two cameras 902, 904 of a stereo vision system mounted on a fixed-wing aircraft 900, according to some embodiments of the present technology. The aircraft 900 may be, for example, a crop duster, a unmanned aerial vehicle (UAV), a quadcopter or hexacopter, an airplane, etc. In some embodiments, the cameras 902, 904 may be mounted on the underside of the wings of the aircraft 900 and pointed downward to view a ground area 906. In some embodiments, the two cameras 902, 904 may be pointed in a forward direction or in a rearward direction. Because the wings may flex or move during operation of the aircraft 900, the cameras 902, 904 may oscillate back and forth relative to each other in a given position, and stereoscopic vision may not be possible using conventional stereo vision techniques.

[0035] 10A and 10B respectively illustrate front and rear perspective views of a camera arrangement of a stereo vision system mounted on an off-road vehicle 1000 according to some embodiments of the present technology. The stereo vision system may have a surround view configuration consisting of four stereo vision camera pairs: a front stereo vision camera pair 1002, a left stereo vision camera pair 1004, a rear stereo vision camera pair 1006, and a right stereo vision camera pair 1008. The four stereo vision camera pairs 1002, 1004, 1006, 1008 may provide four horizontal fields of view of at least 90 degrees, one each in the front, rear, left, and right directions. As will be appreciated, off-road vehicles encounter extreme amounts of vibration and shock when operating on rough terrain, which would quickly cause conventional stereo vision cameras to become misaligned. Examples of off-road vehicles include, for example, military vehicles, jeeps, all-terrain vehicles, dump trucks, mining vehicles, and any type of vehicle that may be driven over unpaved surfaces such as gravel, rocky terrain, boulders, excavation sites, etc. High-resolution 3D information about the surrounding environment of an off-road vehicle can be vital to navigate uneven surfaces that may be filled with rocks, boulders, tree branches, etc., and therefore a reliable system for obtaining such information is of high importance.

[0036] 11 illustrates two stereo vision systems mounted on a combine harvester 1100, according to some embodiments of the present technology. The forward-facing stereo vision system may include a front camera 1102 and a front camera 1104, which may be used to obtain information to, for example, measure the amount of harvested material before it is encountered by the harvester 1100 to enable informed control of the throttle of the harvester 1110, thus maximizing the feed rate. The forward cameras 1102, 1104 may be mounted, for example, on the headliner 1116 of the harvester 1100. The spout-mounted stereo vision system may include a spout camera 1108 and a spout camera 1110, which may be used to obtain information to, for example, control the position of the spout 1112 and measure the amount of material in the collection bin. As the harvester 1100 moves over the fields, plants, and vegetation, both the spout 1112 and headliner 1116 of the harvester 1100 may experience extreme amounts of vibration and shock. Moreover, as will be appreciated, the headliner 1116 and spout 1112 may not be an "optically robust" structure such that relative movement of the front cameras 1102, 1104 and relative movement of the spout cameras 1108, 1110 may be avoided. That is, it may not be possible to maintain a constant alignment of the camera pair over a farming season, much less over the entire life of the harvester 1100. Camera alignment issues associated with the harvester 1100 may also be a problem with other farm equipment (e.g., tractors, hay balers, earthmoving vehicles, etc.).

[0037] FIG. 12 illustrates a left elevation, a front elevation, and a plan view of cameras 1202, 1204 of a forward-facing stereo vision system mounted on an automated forklift 1200, according to some embodiments of the present technology. In some embodiments, the cameras 1202, 1204 may provide a wide field of view, e.g., 135 degrees, which may enable collision avoidance and object localization, which are desirable features when moving large objects that may obstruct the operator's view. A wide baseline of more than 1 m in width is possible on a typical forklift platform. Forklift systems typically require a depth range of about 30 m, which is well beyond the capabilities of conventional stereo vision solutions. Forklifts can also be subject to extreme shocks during operation, and it is highly desirable, if not essential, for any stereo vision system used with a forklift to have shock-resistant cameras as well as shock-resistant electronics (e.g., sensors) that can maintain calibration. Shock resistance is desirable not only for forklifts, but also for automated mobile vehicles (eg, smart unmanned guided vehicles) used, for example, in warehouses and factories.

[0038] As can be seen from the above discussion of Figures 3A through 12, a stereo vision system that may be mounted on a non-rigid platform and that may be automatically calibrated to adjust for slow and / or fast alignment perturbations may be desirable for many different situations.

[0039] FIG. 13 illustrates a block diagram of a stereo vision system according to some embodiments of the present technology. A processing component 1310 may be included as part of each of the electronic control units (e.g., electronic control units 212, 214, 216). A first camera 1300 (also referred to as “camera 1”) and a second camera 1302 (also referred to as “camera 2”) of the stereo vision system may send signals or data of raw images 1304 and 1306 to the processing component 1310. In some embodiments, the cameras 1300, 1302 may be monochrome CMOS cameras, color CMOS cameras, near-infrared cameras, short-wave infrared cameras, mid-wave infrared cameras, or long-wave infrared cameras, and may provide the raw images 1304 and 1306 to the processing component 1310 in real time or near real time. In some embodiments, the cameras 1300, 1302 may have memory and / or may be associated with a data storage device, and the cameras 1300, 1302 may provide a sequence of images (e.g., a sequence of raw images 1304 and 1306) to the processing component 1310 that are reproduced from the memory and / or data storage device. In some embodiments, the images may be two-dimensional (e.g., height and width). In some embodiments, the images may be three-dimensional (e.g., height and width and color). The processing component 1310 may be configured to be instructed by a main system controller 1316 through command and control lines 1312. As will be appreciated, the command and control lines 1312 may be a wired communication mechanism (e.g., data bus, communication lines) or a wireless communication mechanism using communication techniques known in the art. In some embodiments, the main system controller 1318 may comprise a computer configured to coordinate high-level functions (e.g., automatic emergency braking for an automobile) and to communicate with various sub-systems (e.g., braking systems, sensor systems, etc.).In some embodiments, a common communication protocol may be used for communication over the command and control lines 1312 (e.g., Ethernet, CAN (Controller Area Network), I2C (Inter Integrated Circuit), etc.). In some embodiments, the processing component 1310 may report on calibration status (e.g., system health status, information quality status, etc.) to the main system controller 1318 over the command and control lines 1312. In some embodiments, the command and control lines 1312 may be used to configure auto-calibration and stereo enablement settings of the processing component 1310. The command and control lines 1312 may also be used to start, stop, and record both input and output data streams within the processing component 1310. A depth map 1308 and a corresponding confidence map 1314 may be output from the processing component 1310 for each pair of raw images 1304 and 1306. In some embodiments, the depth map 1308 may be a two-dimensional matrix with the same width and height as the raw image 1304, but whose values ​​indicate ranges (e.g., distances) to targets (e.g., objects in the field of view). The depth map 1308 may be reported or output as an RGB image, and each 24-bit color value may be encoded to be a range to a target. For example, if the values ​​of the depth map 1308 have units of mm (millimeters), then a distance from 0 to 16,777.215 m (meters) may be represented using 24 bits. In some embodiments, the confidence map 1314 may be a two-dimensional matrix with the same height and width as the raw image 1304, and each element of the confidence map 1314 may represent a confidence value or confidence level for the depth estimation of each pixel of the raw image 1304. In some embodiments, the confidence values ​​may be 8-bit unsigned values ​​from 0 to 255, and relatively high values ​​may indicate a high level of confidence in the corresponding depth estimation.In some embodiments, using an 8-bit representation for the belief map 1314 may allow the belief map 1314 to be conveniently displayed as a grayscale image, or may allow the data for the belief map 1314 to be sent out as a monochrome video data stream. The information provided by the belief map 1314 may be important to a sensor fusion processing pipeline that serves to combine estimates from various sensors (e.g., radar, lidar, sonar, etc.) to provide a reliable estimate of the surrounding environment.

[0040] The illuminator 1316 may be used to project light for nighttime operation according to some embodiments of the present technology. The illuminator 1316 may consist of one or more lighting elements (e.g., lamps, LEDs, etc.). In some embodiments, the stereo vision system may be provided on a vehicle and the illuminator 1316 may consist of a vehicle headlight, which may emit visible or infrared radiation. In some embodiments, the illuminator 1316 may consist of an externally mounted lamp configured to emit visible or infrared radiation. In some embodiments, the illuminator 1316 may be configured to emit visible or infrared radiation in synchronization with the exposure interval of the first camera sensor and the exposure interval of the second camera sensor. With a modulated waveform, the peak optical power may be higher for the same average optical power, and therefore the number of photons registered by the camera sensor may be increased without blinding an approaching driver or pedestrian, which may be sensitive to the average optical power. In some embodiments, the illuminator 1316 may comprise a vertical cavity surface emitting laser (VCSEL) array that may be configured to project a pseudo-random dot pattern. The use of a randomized dot pattern may add texture to the scene being illuminated, which may improve the ability of the stereo enable engine 1420 to generate accurate depth maps.

[0041] In some embodiments of the present technology, the stereo vision system of FIG. 13 may be modified to be a three-camera system. The first camera and the second camera may form a short-range sensing stereo vision system, and the first camera and the third camera may form a long-range sensing stereo vision system. The baseline distance between the first camera and the second camera may be shorter than the baseline distance between the first camera and the third camera. The stereo vision system consisting of the first camera and the second camera may have a lower minimum distance range but a higher range resolution compared to the stereo vision system consisting of the first camera and the third camera.

[0042] In some embodiments of the present technology, the stereo vision system of FIG. 13 may be modified to be a four-camera system consisting of two stereo vision systems, a short-range system with the first and second cameras, and a long-range system with the third and fourth cameras. Compared to a three-camera system, the additional (fourth) camera may provide an additional degree of freedom of movement and may allow the short-range system to have a different (e.g., wider) field of view than the long-range system. In some embodiments, the first and second cameras may have a shorter baseline and a wider field of view than the third and fourth cameras.

[0043] In some embodiments of the present technology, the stereo vision system of FIG. 13 may be modified to include a time-of-flight camera that may be configured to measure objects closer to the cameras 1300, 1302 than the minimum distance of the stereo vision system.

[0044] FIG. 14 illustrates a block diagram of a processing component 1310 according to some embodiments of the present technology. As discussed above, the processing component 1310 may convert the raw images 1304 and 1306 into a depth map 1308 and a belief map 1314. The raw image 1304 from the first camera may be corrected by a correction engine 1416. The raw image 1306 from the second camera may be corrected by a correction engine 1418. As will be appreciated, in computer vision technology, correction is a transformation process used to warp images such that the images from camera 1 and camera 2 are aligned row by row. That is, features within a row of pixels in the corrected image 1406 resulting from the raw image 1304 from camera 1 are aligned to the same row of pixels as matching features in the corrected image 1408 from the raw image 1306 of camera 2. Such row alignment of pixels may allow the stereo correspondence engine 1420 to operate faster since it allows the search for corresponding features to be a one-dimensional search along the rows rather than a two-dimensional search along the rows and columns. In some embodiments, the correction engines 1416 and 1418 may utilize the camera parameters 1410 and 1412, respectively, to determine a mapping or warping operation. Figure 15A illustrates a flow diagram of a correction algorithm that may be utilized by the correction engines 1416, 1418 in some embodiments of the present technology. Figure 15B illustrates the coordinate system and geometric details of the correction procedure performed by the algorithm in accordance with some embodiments of the present technology.

[0045] According to some embodiments of the present technology, the camera parameters 1410, 1412 may consist of six extrinsic camera parameters that may describe the relative positions and orientations of the two cameras 1300, 1302, and eighteen intrinsic camera parameters that may characterize the transformation from the camera coordinate system to the pixel coordinate system of each camera 1300, 1302. The six extrinsic variables may consist of relative roll, relative pitch, relative yaw, relative translation x, relative translation y, and relative translation z. The eighteen intrinsic camera parameters for both cameras 1300 and 1302 may consist of focal length (Fx, Fy), principal point (Cx, Cy), radial lens distortion coefficients K1, K2, and K3, and tangential lens distortion coefficients P1 and P2 for each camera 1300, 1302. These camera parameters are well known and are accounted for by OpenCV (Open Source Computer Vision Library), an open source library that contains hundreds of computer vision algorithms. When wide field of view lenses and / or fisheye lenses are used, one or more additional high order lens distortion parameters may need to be used. In some embodiments, the camera parameters 1410, 1412 may not be constant and may change as a function of time to track actual changes in the cameras 1300, 1302. In some embodiments, the camera parameters 1410, 1412 may be updated for every frame.

[0046] According to some embodiments of the present technology, the auto-calibration engine 1414 may be configured to implement a method for maintaining stereo camera calibration of the cameras 1300, 1302. In some embodiments, the auto-calibration engine 1414 may utilize the raw images 1304, 1306 of the current frame and the corrected images 1406, 1408 of the previous frame as inputs. The auto-calibration engine 1414 may output camera parameters 1410 for the first camera 1300 (camera 1) and camera parameters 1412 for the second camera 1302 (camera 2), and may output translation parameters 1426 for the first camera 1300 and translation parameters 1428 for the second camera 1302.

[0047] According to some embodiments of the present technology, the translation engines 1422, 1424 may perform a transformation operation to shift the corrected images 1406, 1408 to compensate for very fast perturbations and stabilize the corrected images 1406, 1408 relative to each other. The translation operation may be performed by pixel row and / or pixel column, which may be quickly achieved using an affine transformation procedure. The translation parameters 1426, 1428 may be row-based translation offsets and column-based translation offsets. In some embodiments, to reduce calculation time, one of the translation engines 1422, 1424 may be discarded (e.g., unused or idle or non-existent) and the other receives the translation offset (e.g., as a sum). The images 1430, 1432 output by the translation engines 1422, 1424 may be the corrected and stabilized images 1430, 1432, which may be sent to the stereo correspondence engine 1420. The stereo correspondence engine 1420 may determine disparity between matching pixels in the images 1430, 1432 (e.g., the images 1430, 1432 may be left and right images), may calculate a disparity map, may calculate and output a depth map 1308 from the disparity map, and may calculate and output a confidence map 1314.

[0048] FIG. 16 illustrates a block diagram of an auto-calibration engine 1414 according to some embodiments of the present technology. The auto-calibration engine 1414 may consist of five calibration engines: a fast calibration engine 1600, an intermediate calibration engine 1602, a slow calibration engine 1604, an absolute range calibration engine 1606, and a stereo image stabilization engine 1608. In some embodiments, the fast calibration engine 1600, the intermediate calibration engine 1602, and the slow calibration engine 1604 may compensate for camera parameter perturbations that may occur on a slow to a fast time scale, and the method used by these calibration engines 1600, 1602, 1604 may be based on optimizing a cost function. In some embodiments, the absolute range calibration engine 1606 may calibrate the absolute range for the object by adjusting the relative yaw, which is an extrinsic camera parameter. In some embodiments, the calibration of the absolute range calibration may be one of the most important and most difficult parameters to adequately calibrate since epipolar geometry constraints are not sensitive to yaw changes. In some embodiments, a stereo image stabilization engine 1608 may be used to track and compensate for frame-to-frame pitch and yaw variations. The time history of the camera parameters may be stored in a storage device 1622. The current camera parameters 1620 may be used as a starting point for the calibration engines 1600, 1602, 1604, 1606, 1608.In some embodiments, the outputs 1610, 1612, 1614, 1616, 1626 of the calibration engines 1600, 1602, 1604, 1606, 1608 may be combined by a calibration manager 1624, which uses this information to determine best estimates of the camera parameters 1410, 1412 for the first camera 1300 and the second camera 1302. The command and control lines 1312 may be used to initialize values ​​for the camera parameters, which may result from manual factory calibration, manufacturing specifications, or guesswork. As discussed below, the absolute range calibration engine 1606 has different embodiments. In some embodiments, any one or any combination of the fast calibration procedure performed by the fast calibration engine 1600, the intermediate calibration procedure performed by the intermediate calibration engine 1602, and the slow calibration procedure performed by the slow calibration engine 1604 may include an intrinsic parameter calibration procedure to calibrate one or more intrinsic camera parameters. For example, for each camera, any one or any combination of the focal length, the principal point, at least one radial lens distortion coefficient, and at least one tangential lens distortion coefficient may be calibrated by the intrinsic parameter calibration procedure. In some embodiments, the intrinsic parameter calibration procedure may be performed by the fast calibration engine 1600 and / or the intermediate calibration engine 1602 when there is a priori knowledge about the first and second cameras 1300, 1302, e.g., when the lens mounting itself is known to be flexible, which is a rare case. In some embodiments, it may not be necessary for the intrinsic camera parameters to be calibrated at fast and / or intermediate time scales, for several reasons.The reason is: (1) small changes in the piercing point horizontal location (Cx) and focal length (Fx and Fy) may be corrected to first order only by changes in yaw, (2) small changes in the piercing point horizontal location (Cy) may be corrected to first order only by changes in pitch, and (3) because the lens shape generally does not change, the lens distortion coefficients typically do not have to change.

[0049] FIG. 17A illustrates block diagrams of a fast calibration engine 1600, an intermediate calibration engine 1602, and a slow calibration engine 1604 according to some embodiments of the present technology. The calibration engines 1600, 1602, 1604 may have the same block diagram structure but may utilize different optimization methods 1700, 1722, 1724, respectively. In some embodiments, the basic concept of the methods 1700, 1722, 1724 is that for each of the methods 1700, 1722, 1724, the optimal or accurate camera parameters may be associated with a minimized cost function 1704 appropriate for the corresponding method. For example, for a nominal case of 24 camera parameters (6 extrinsic parameters and 18 intrinsic parameters), a brute force search of parameters may be performed using a cost function 100. 24 This may require evaluating multiples of the time and assuming 100's of values ​​per parameter, which would take an unreasonably long time to compute using a conventional computer (which would take longer than the age of the universe). As will be appreciated, to be practical for a real-world setting, the optimization operation should be completed in real-time and implemented at a time scale matching or faster than the time scale of the source or cause of the perturbation. Thus, in some embodiments, it is highly desirable to efficiently search for camera parameters by splitting the search into different time scales.

[0050] According to some embodiments of the present technology, one or more of the optimization methods 1700, 1722, 1724 may iteratively evaluate the cost function 1704 with respect to a subset of the camera parameters for a bounded search range. In some embodiments, to speed up the evaluation of the cost function 1704, one or more of the optimization methods 1700, 1722, 1724 may dynamically adjust the pyramid levels 1703 of the raw images 1304, 1306 from the first and second cameras 1300, 1302. The term "pyramid" is a common computer vision term that is broadly understood to refer to the downsampling of an image to a smaller image. Since the amount of time to compute the cost function (i.e., determine the minimum cost function) may be proportional to the size of the image being processed by subsampling to half resolution (or pyramid level 1), the time to compute the cost function may be reduced by a factor of four. For coarse parameter searches, a high pyramid level may be used to speed up the search. For finer parameter searches, lower pyramid levels may be used for enhanced accuracy. In some embodiments, the output 1610, 1612, 1614 of each of the optimization methods 1700, 1722, 1724 may be the camera parameters that minimize the cost function 1704 of the corresponding optimization method. As will be appreciated, minimizing a cost function may be considered an equivalent approach, and therefore may also be considered to maximize a figure of merit.

[0051] According to some embodiments of the present technology, an indicator of calibration health may be reported to the main system controller 1316 over command and control lines 1312. In some embodiments, calibration health may be made to be the negative of the (minimized) cost function (or maximized figure of merit) corresponding to the optimized camera parameters.

[0052] According to some embodiments of the present technology, the cost function 1704 may comprise a pyramid down block 1701 configured to reduce the size of the raw images 1304, 1306, a correction block 1706 configured to warp and row align the reduced images according to the test camera parameters 1702, a stereo block matching block 1710, and a block 1714 configured to calculate the negative of the number of valid pixels. In some embodiments, the stereo block matching block 1710 may use the stereo block matching algorithm of OpenCV with parameters that may include a pixel window size of 9x9, a normalized response prefilter, a prefilter cap of 63, a prefilter size of 11, a speckle window size of 100, a speckle range of 64, a texture threshold of 10, a uniqueness ratio of 10, and a disparity search range of 0 to 255. Such a block matching algorithm may be chosen for its fast execution across many different types of processors. The texture threshold and uniqueness ratio may be set relatively high (e.g., values ​​between 10 and 50) to minimize the number of incorrect distance estimators and to replace corresponding pixels with invalid pixels. In some embodiments, each of the optimization methods 1700, 1722, 1724 may find optimal camera parameters that minimize the number of invalid pixels after stereo block matching is performed.

[0053] FIG. 17B illustrates a flow diagram of a fast optimization method 1700 of the fast calibration engine 1600, according to some embodiments of the present technology. The fast optimization method 1700 may adjust pitch and roll extrinsic camera parameters since these parameters may be responsible for most of the short time scale distortions. The fast optimization method 1700 may be used for the most frequent updates among the optimization methods 1700, 1722, 1724. The fast optimization method 1700 may be considered fast because it explores a relatively small number of camera parameters (two) over a small range.

[0054] In step 1726 of the fast optimization method 1700, a quick search is performed for an initial guess of the camera parameters 1620 by checking the cost within the chosen grid points. In some embodiments, the grid points may typically have nine values ​​for pitch from -0.4 degrees to 0.4 degrees and three values ​​for roll from -0.2 degrees to 0.2 degrees for an initial guess of the camera parameters 1620 for a total of 27 grid points. When evaluating the cost function 1704, a pyramid level of 1 may be used to speed up the evaluation.

[0055] In step 1728 of the fast optimization method 1700, the pitch and roll that correspond to the lowest cost (the minimum of the cost function 1704) may be selected among the grid points. In step 1730 of the fast optimization method 1700, a COBYLA (Constrained Optimization by Linear Approximations) algorithm for derivative-free optimization with nonlinear inequality constraints may be applied with the pitch and roll found in step 1728, other camera parameters of the initial guess of camera parameters 1620, and bounds of 0.1 degrees on both pitch and roll. The cost function 1704 used in step 1730 may have a pyramid level of 0. The camera parameters 1620 may be updated using the pitch and roll corresponding to the lowest cost found by COBYLA, and the updated camera parameters 1610 are returned to the fast optimization method 1700.

[0056] According to some embodiments of the present technology, the COBYLA algorithm may construct successive linear approximations of the objective function and constraints using a simplex of points, and optimize these approximations within a trust region at each step. In some embodiments, since there are only two search dimensions, namely the relative pitch and roll of the stereo camera pair, over a small search range of 0.1 degrees for pitch and 0.1 degrees for roll, the optimization can be calculated quickly (e.g., 100 ms on an Intel Core i7 CPU processor) and compensate every other frame at 20 FPS if needed. In some embodiments, the search range can be increased or decreased based on the expected excursion for a given platform. For example, for a suction cup mount on a car, 0.1 degrees is large enough, but for a loosely mounted camera on a mountain bike being ridden on a trail, 0.2 degrees may be required.

[0057] It should be noted that in some embodiments, the COBYLA search method may be replaced by other search methods, such as the Nelder-Mead simplex search method, the Broyden-Fletcher-Goldfar-Shanno (BFGS) method, the Powell method, the sequential quadratic programming (SLSQP) method, or the Monte Carlo method.

[0058] FIG. 17C-1 illustrates a flow diagram of an intermediate optimization method 1722 of the intermediate calibration engine 1602, according to some embodiments of the present technology. The intermediate optimization method 1722 may update more parameters than the fast optimization method 1700, but may update slower than the fast optimization method 1700. In some embodiments, the primary objective of the intermediate optimization method 1722 is to find the translation vector and then optimize the pitch and roll. FIG. 17C-2 illustrates a flow diagram of the angle search process of the intermediate optimization method of FIG. 17C-1, according to some embodiments of the present technology.

[0059] In step 1732 of the intermediate optimization method 1722, a grid search is performed on the translation vector angle, which is represented by two angles, ay and az. The angle ay may be the angle between the translation vector and the x-axis measured on the xz plane. The angle az may be the angle between the translation vector and the x-axis measured on the yz plane. For example, the search may test all of the grids of values ​​(ay, az) from -7 degrees to 7 degrees in 1 degree increments around the initial translation vector of the camera parameters 1620. For each translation vector, the step 1732 uses an angle search process 1736 (see FIG. 17C-2) to find the best tuple of pitch and roll angles that results in the lowest cost. In some embodiments of the present technology, the angle search process 1736 may decompose the search process into a coarse angle search 1738, an intermediate angle search 1740, and a fine angle search 1742.

[0060] The coarse angle search 1738 may apply nine COBYLA searches centered at (0,0,0), (-1,-1,-1), (-1,-1,1), (-1,1,-1), (1,-1,-1), (1,1,-1), (1,-1,1), (-1,1,1), and (1,1,1) degrees (pitch, yaw, and roll) from the initial pitch, yaw, and roll specified in the camera parameters 1620. The COBYLA search may use pyramid level 3 with the cost function 1704 for fast evaluation, and the search bounds for all angles may be [-1,1] degrees. The pitch, yaw, and roll corresponding to the lowest cost may be used as a starting point for the intermediate angle search 1740.

[0061] The intermediate angle search 1740 may apply a COBYLA search starting from the pitch, yaw, and roll returned from the coarse angle search 1738. This COBYLA search may have angle bounds of [-0.2, 0.2] degrees on pitch, yaw, and roll and may use pyramid level 3 for the cost function 1704. The pitch, yaw, and roll corresponding to the lowest cost may be used as a starting point for the fine angle search 1742.

[0062] The fine angle search 1742 may apply a COBYLA search starting from the pitch, yaw, and roll returned from the medium angle search 1740. This COBYLA search may have angle bounds of [-0.1, 0.1] degrees on pitch, yaw, and roll and may use pyramid level 0 in the cost function 1704. The pitch, yaw, and roll corresponding to the lowest cost may be returned to step 1732.

[0063] The best estimates for ay, az, pitch, yaw, and roll from step 1732 may be used as a starting point for a fine translation vector and angle search 1734, which may apply a COBYLA optimization to (ay, az) followed by a COBYLA optimization to (pitch, yaw, roll). Both COBYLA searches may use pyramid level 0 in the cost function 1704 to maximize sensitivity to the optimization parameters. The two COBYLA searches may be repeated until little improvement is seen in the cost function. The initial camera parameters 1620 may be updated with the optimal values ​​for ay, az, pitch, yaw, and roll, and then the updated camera parameters 1612 may be returned from the intermediate optimization method 1722.

[0064] 17D-1 through 17D-4 (collectively "FIG. 17D") illustrate a flow diagram of a slow-speed optimization method 1724 according to some embodiments of the present technology. In some embodiments, the slow-speed optimization method 1724 of FIG. 17D may be performed by the slow-speed calibration engine 1604 once at factory calibration and then when deemed necessary by the main system controller 1316 (e.g., an accelerometer or other sensor reports a large impact event). The slow-speed optimization method 1724 may optimize extrinsic and intrinsic camera parameters. In step 1744, a global grid search may be performed on pitch, yaw, and roll angles using a sampling interval (e.g., 0.2 degrees, 0.4 degrees, 0.7 degrees, etc.) over [-10, 10] degrees. The remaining steps 1746 through 1796, FIG. 17D, may apply a local search method, such as COBYLA, to the previous best estimates of the camera parameters 1602. As previously mentioned, the intrinsic camera parameters may be the focal length (FX, FY), pierce point (CX, CY), and lens distortion coefficients (K1, K2, K3, P1, P2). The extrinsic camera parameters may be the pitch, yaw, roll, and translation vector (TX, TY, TZ). In some embodiments, because complementary parameters (e.g., pitch and TX, roll and TY, yaw and TZ) may distort an image (e.g., raw images 1304, 1306) in a similar manner, optimization of pitch, yaw, and roll may be followed by optimization of the translation vectors from step 1746 to step 1748, from step 1768 to step 1770, from step 1772 to step 1774, from step 1784 to step 1786, and from step 1788 to step 1790.

[0065] FIG. 18 illustrates a flow diagram for the stereo image stabilization engine 1608 of the auto-calibration engine 1414, according to some embodiments of the present technology. The stereo image stabilization engine 1608 may determine the amount to shift horizontally and vertically the images acquired from camera 1 and camera 2 such that the relative displacement between these cameras is minimized. The stereo image stabilization engine 1608 may compensate for the fastest perturbations (e.g., engine noise and road noise) and may update every frame. Unlike traditional monocular image stabilization, which may stabilize an image relative to the Earth, the stereo image stabilization engine 1608 may stabilize two images relative to each other. In some embodiments, the stereo image stabilization engine 1608 receives as inputs the corrected image 1406 resulting from the raw image 1304 from camera 1 and the corrected image 1408 resulting from the raw image 1306 of camera 2. The stereo image stabilization engine 1608 may output the translation parameters 1426 for camera 1 and the translation parameters 1428 for camera 2. At 1800, a forward tracking process may calculate optical flow for a sparse feature set using an iterative Lucas-Kanade method with pyramids for the corrected images 1406, 1408, which may be left corrected and right corrected images, from the previous frame to the current frame. At 1802, a backward tracking process may calculate optical flow backwards from points in the current frame to points in the previous frame. At 1804, the process may determine a good trajectory from the backward track (e.g., a good trajectory is when the backward tracking process at 1802 reprojects points within one pixel of points in the previous trajectory). At 1806, the process may discard bad trajectories. At 1808, the process may add the remaining good points to the trajectory. At 1810, the process may update an estimate of the translation parameters. In some embodiments, the translation parameter for camera 1 1426 and the translation parameter for camera 2 1428 may be the amount to shift the rows and columns of the images from camera 1 and camera 2 relative to the previous frame.In some embodiments, the translation parameter may be equal to the average relative translation per frame between images from camera 1 and images from camera 2, with outliers removed from the average.

[0066] After a few frames, one or more trajectories may be discarded because the keypoints may disappear from the screen (e.g., out of view) or may be occluded. At 1812, the process may find new keypoints to create new trajectories. For example, every n frames, where n may typically be 5, the stereo image stabilization engine 1608 may search for new keypoints in the corrected images 1406, 1408. A keypoint detector (not shown) may look for good points to track, for example, using a Shi-Tomasi corner detector. At 1814, the process may translate the corrected images 1406, 1408 using translation parameters 1426, 1428 to remove relative motion between camera 1 and camera 2. At 1816, the process may compute a disparity map from the corrected images 1406, 1408 using a block matching algorithm. At 1818, a process may ignore the currently tracked keypoint for the rectified image 1406 of camera 1, and at 1820, a process may find a good feature to track in the rectified image 1406 of camera 1. At 1822, for each found keypoint for the rectified image 1406 of camera 1, a process may find a matching keypoint in the rectified image 1408 of camera 2 using the processes at 1824, 1826, 1828, 1830. At 1824, a process may be performed to create a Hanning window for the phasecorrelation function. In OpenCV, phasecorrelation refers to a method of checking the similarity of two images with equal size. Phasecorrelation is a fast implementation of template matching. At 1826, a process may be performed to find matching keypoints between camera 1 and camera 2. Images 1406 and 1408 are rectified so that the matching keypoints are aligned along the same row. Adding the disparity value 1816 to the camera 1 keypoint sequence value yields the predicted location for the camera 2 keypoint sequence value.At 1828, a process may be performed to verify the match of 1826 by computing a phasecorrelation function for the matched keypoints. The phasecorrelation function is computed for an image patch (typically 40x40 pixels) centered on the matching keypoint pair between camera 1 and camera 2. The phasecorrelation function returns a corrected coordinate shift and a response (typically a value between 0 and 1 indicating the similarity of the two image patches). At 1830, a process may be performed to record the matching keypoints, correct the keypoint coordinates according to the corrected coordinate shift from 1828, and update the trajectory information if the phasecorrelation response is greater than a threshold (typically a threshold of 0.3).

[0067] In some embodiments of the present technology, the absolute range calibration engine 1606 may have three different implementations (referred to as 1606a, 1606b, and 1606c), any or all of which may be incorporated into the auto-calibration engine 1414. The three implementations 1606a, 1606b, and 1606c may use different absolute distance references. The implementation 1606a may use a non-negative disparity constraint (see FIG. 19A, FIG. 19B), the implementation 1606b may use a known object such as a stop sign (see FIG. 20A, FIG. 20B), and the implementation 1606c may use vehicle odometry (see FIG. 21A, FIG. 21B). In some embodiments, the relative yaw between the cameras 1300, 1302 may have a large effect on the absolute range calibration of the stereo vision system. The absolute range calibration engine 1606 may be used to calibrate the yaw.

[0068] FIG. 19A illustrates a flow diagram of an implementation 1606a in which the absolute range calibration engine 1606 has a non-negative disparity constraint, according to some embodiments of the present technology. The implementation 1606a may find the yaw when the chief rays of the first and second cameras 1300, 1302 do not intersect. When the chief rays intersect in front of the cameras, the corresponding disparity values ​​may be negative at ranges farther than the intersection point. By adjusting the yaw until the rays do not intersect, the cameras 1300, 1302 may become parallel, some negative values ​​in the disparity map 1712 may be reduced (sometimes significantly) to a minimum value, and a suitable absolute range may be found.

[0069] FIG. 19B is a graph 1900 illustrating how the number of pixels with negative disparity values ​​in a disparity map (e.g., calculated from raw images 1304, 1306 using initial camera parameters 1620 after correction, but with various values ​​of yaw) may vary as a function of yaw, according to some embodiments of the present technology. For yaw values ​​less than about 1.8 degrees in the graph 1900, the number of negative disparity values ​​increases significantly, indicating that at long ranges, and then at increasingly closer ranges, the chief rays of the cameras 1300, 1302 may begin to intersect. For yaw values ​​less than 1.65 degrees, the number of negative disparity values ​​decreases. This may be because the block matcher used in this analysis may only search disparity values ​​up to -32 pixels. Pixels with disparity values ​​of -33 pixels or lower may not be counted. For yaw values ​​greater than 1.85 degrees, the number of pixels with negative values ​​is small, but not zero. This may be because unmatched pixels may sometimes have negative disparity values. The desired yaw is in a transition region where the graph 1900 transitions from a region where the number of pixels is low and generally constant (yaw > 1.85 degrees in graph 1900) to include a region where the number of pixels increases rapidly (e.g., the region indicated by the upward pointing arrow 1912). The graph 1900 also illustrates a region where the disparity value of a pixel may be more negative than the minimum disparity limit (e.g., the region indicated by the downward pointing arrow 1914). Returning to the flow diagram of the implementation 1606a illustrated in FIG. 19A, at 1920, a coarse search is performed from -0.5 degrees (see 1902) to 1.0 degrees (see 1906) around the initial guess 1904 for the yaw for which the disparity map is calculated, and the number of negative disparity values ​​is tallied. The search may be performed in 0.1 degree steps. The result of the coarse search may be an array of 16 values. Graph 1900 is an example of a plot of 16 data points for a coarse curve. The derivative of the coarse curve with respect to yaw may be calculated. The derivative of the coarse curve may be sensitive to changes and therefore may be useful in finding transitions in the coarse curve.In 1922, the quality of the coarse search is checked by ensuring that the transition region (e.g., the region from 1.65 degrees to 1.8 degrees in graph 1900) is greater than 5 times the tail noise 1908. The tail noise 1908 may be defined to be the standard deviation of the last 5 points of the derivative of the coarse curve. In other words, the transition region should have a significant change in the number of negative disparity values ​​compared to the region where the number of negative disparities should be zero. Next, in 1924, a fine search range is determined by searching around the yaw corresponding to the most negative slope (e.g., about 1.7 degrees yaw in graph 1900). Starting from the largest coarse yaw value and searching towards lower coarse yaw values, an upper limit is determined to be the value at which the derivative of the coarse curve is greater than -3 times the tail noise. Starting from the yaw corresponding to the most negative slope and increasing towards the maximum yaw, a lower limit is determined to be the value at which the derivative of the coarse curve is less than -3 times the tail noise. Next, at 1926, a fine search grid is created from the lower limit to the upper limit, and a fine search is performed. This grid may typically have 21 points. At 1928, the yaw value that corresponds to the smallest number of negative disparity pixels is determined. In some embodiments, a transition point in the curve may be estimated in this manner.

[0070] FIG. 20A illustrates a flow diagram of an implementation 1606b in which an absolute range calibration engine 1606 uses an object with known dimensions, according to some embodiments of the present technology. FIG. 20B illustrates an example of imaging optics for imaging an object 2006 with known dimensions, according to some embodiments of the present technology. In some embodiments, since the dimensions of the object 2006 are known, and since the focal length of the camera lens of the camera 1300, 1302 is known, the range or distance to the object (target) 2006 may be determined according to equation 2020. More specifically, according to equation 2020, the ratio of the width W (see 2012) of the object to the range R (see 2014) to the object is equal to the ratio of the focal length F (see 2016) of the camera lens 2008 to the width H (see 2018) of the image sensed by the camera sensor 2010 (e.g., a CMOS sensor). In some embodiments, the known object can be recognized in an automated manner using known detection technology (e.g., a traffic sign recognition system, a license plate detector, an object detector, etc.). In some embodiments, the known object can be manually specified by manual input of a bounding box and a distance to the known object. An example of an object detector for detecting a stop sign is illustrated in FIG. 24.

[0071] Returning to the flow diagram of the implementation 1606b illustrated in FIG. 20A, at 2000, the absolute range calibration engine 1600 finds a yaw where the average distance to the object is equal to the known distance. In some embodiments of the present technology, the average distance may be calculated over the region of interest of the object. At 2002, a good starting interval for the root-finding algorithm is found by first initializing an interval to ±0.1 degrees of the yaw of the initial camera parameters 1620, and then modifying the left and right ends by 0.1 degrees until the average depth over the region of interest is positive at the left end of the interval and negative at the right end of the interval. At 2004, the root-finding algorithm is run. In some embodiments, the root-finding algorithm may be Brent's method, where the search interval found at 2002 is used to determine the best estimate of the yaw. The initial camera parameters 1620 are then returned along with the updated yaw value 1616.

[0072] FIG. 21A illustrates a flow diagram of an implementation 1606c in which the absolute range calibration engine 1606 utilizes vehicle odometry measurements 1628, according to some embodiments of the present technology. FIG. 21B is a graph 2108 illustrating how disparity can vary as a function of range, according to some embodiments of the present technology. The odometry measurements (e.g., speed, distance, heading, etc.) may be information already available from in-vehicle ("on-board") sensors and may be used to help calculate the yaw of the stereo vision system. In some embodiments, the odometry measurements may be available from an on-board radar and / or lidar system. In the implementation 1606c, the change in range ΔR is calculated 2100 for each trajectory 1626 from the stereo image stabilization engine 1608. ODO is calculated from the odometry data from time t1 to time t2. The corresponding change in range ΔR is measured by the stereo vision system and is reduced by ΔR due to miscalibration. ODO In 2102, the disparity offset d offsetis calculated using the function d=fB / R, where f is the focal length of the lens and B is the distance between the corrected range ΔR' and ΔR. ODO The baseline width between the first camera 1300 and the second camera 1302 is equal to d offset gives the same or similar values ​​for , but a moving object may not. offset Outliers for are removed and the remaining values ​​are averaged. In 2104, the yaw is calibrated to be equal to the initial yaw plus the average parallax offset divided by the focal length of the camera. The initial camera parameters 1620 are then returned along with the updated yaw value 1616.

[0073] FIG. 22 illustrates a flow diagram of procedures performed by the calibration manager 1624 according to some embodiments of the present technology. At 2200, the camera parameters are initialized and the command and control interface 1312 configures the calibration engines 1600, 1602, 1604, 1606, 1608. The initial camera parameters may be extrinsic and intrinsic camera parameters obtained from blueprints, CAD drawings, manufacturer data sheets, etc., or may be obtained from a manual calibration procedure. At 2202, the calibration manager 1624 waits for output from the calibration engines 1600, 1602, 1604, 1606, 1608. Once the output of the estimator of the camera parameters is received, at 2204, this estimator is used in a tracking filter (e.g., a Kalman filter) that uses a series of measurements over time and produces a new estimate of the camera parameters, which may be more accurate than an estimate for only a single measurement. This may be done for each of the multiple camera parameters. At 2206, the new estimates for the camera parameters may be saved in a storage device 1622, and updated camera parameters 1410, 1412 for camera 1 and camera 2 may be output from the calibration manager 1624.

[0074] FIG. 23 illustrates a time series input diagram for the calibration manager 1624, according to some embodiments of the present technology. The time axis is illustrated with respect to different frame numbers, which may correspond to different frame rates (e.g., 10, 30, or 60 FPS). The up arrow indicates the initialization or initiation of the calibration methods 1600, 1602, 1604, 1606, 1608 by the calibration manager 1624, and the down arrow indicates the completion of the calibration method, with optimal parameters being returned to the calibration manager 1624. The execution times for the calibration methods in FIG. 23 are an example of one embodiment.

[0075] 24 illustrates a flow diagram of procedures that may be implemented by a stop sign detector 2400 that may be used for absolute range calibration, according to some embodiments of the present technology. The input to the detector 2400 may be a color image 2402. For example, the detector 2400 may look for an arrangement of red connected components within the letter "STOP." These components may include red regions within an oval shape (e.g., the letter "O" 2406), red regions inside certain shapes (e.g., the letter "P" 2408), and a red octagonal region 2404. The relative location and size of the connected components may provide a unique signature that may be quickly detected with little computational resources.

[0076] FIG. 25 illustrates a flow diagram of a procedure performed by a stereo correspondence engine 1420 according to some embodiments of the present technology. Inputs to the stereo correspondence engine 1420 may be the corrected and stabilized images 1430, 1432 from camera 1 and camera 2. The stereo correspondence engine 1420 may output a depth map 1308 and a confidence map 1314. In some embodiments, a matching window 2500 in the corrected and stabilized image 1430 may be compared to a corresponding row 2502 in the corrected and stabilized image 1432. For each pixel, the resulting matching value 2506 may be the negative of the sum of the absolute differences between the matching window and each offset along the corresponding row 2502. A matching value for each offset may be calculated where the best match 2510 may correspond to the highest matching value, and the second best match 2512 may correspond to the second highest matching value. The uniqueness ratio for a given pixel may be a ratio determined as the best match 2510 divided by the second best match 2512. Higher confidence corresponds to a higher uniqueness ratio. The depth map 1308 may be derived from the best match 2510. The confidence map may be derived from the ratio of the best match 2510 to the second best match 2512.

[0077] According to some embodiments of the present technology, the processing component 1310 may be implemented in hardware (e.g., a computer processor programmed to perform the procedures and methods described above). According to some embodiments of the present technology, the processing component 1310 may be implemented in software (e.g., computer executable code), which may be stored on a non-transitory computer-readable storage medium or on multiple non-transitory computer-readable storage media and may be accessed and executed by a computer processor. According to some embodiments of the present technology, the processing component 1310 may be implemented in a combination of hardware and software. In some embodiments, aspects of the processing component 1310 may be implemented as one or more software modules. For example, one or more of the engines of the processing component 1310 may be implemented as software modules stored on a non-transitory computer-readable storage medium.

[0078] The stereo imaging system according to the techniques described herein may be embodied in different configurations. Exemplary configurations include combinations of configurations (1) to (27) as follows: (1) A stereo vision system comprising: a first camera sensor configured to sense a first reflected energy in a first image and generate a first sensor signal based on the first reflected energy; a second camera sensor configured to sense a second reflected energy in a second image and generate a second sensor signal based on the second reflected energy; and at least one processor configured to receive a first sensor signal from the first camera sensor and a second sensor signal from the second camera sensor and generate three-dimensional (3D) data from the first sensor signal and the second sensor signal, the at least one processor configured to generate corrected stereo images from the first sensor signal and the second sensor signal utilizing stereo calibration parameters, perform stereo matching on the corrected images, and perform automatic system calibration using data from a plurality of stereo images acquired by the first camera sensor and the second camera sensor, wherein the automatic system calibration is based on minimizing a cost function.

[0079] (2) Automatic system calibration Compensating for high frequency perturbations based on tracking image points corresponding to features in the first image and the second image; and calibrating, for each camera, any one or any combination of: focal length, principal point, at least one radial lens distortion coefficient, and at least one tangential lens distortion coefficient; 2. A stereo vision system according to claim 1, further comprising: a fast calibration procedure configured to generate first stereo calibration parameters for implementing one or both of the above.

[0080] (3) Automatic system calibration Compensating for intermediate frequency perturbations based on optimizing a number of effective pixels for dense stereo block matching as a function of relative camera pitch and relative camera roll; and calibrating, for each camera, any one or any combination of: focal length, principal point, at least one radial lens distortion coefficient, and at least one tangential lens distortion coefficient; The stereo vision system of configuration (1) or (2), further comprising an intermediate calibration procedure configured to generate second stereo calibration parameters for implementing one or both of the above.

[0081] (4) Automatic system calibration Compensating for low frequency perturbations based on optimizing a number of effective pixels for dense stereo block matching as a function of any one or any combination of a relative camera pitch, a relative camera roll, a relative camera yaw, and a relative camera position; and calibrating, for each camera, any one or any combination of: focal length, principal point, at least one radial lens distortion coefficient, and at least one tangential lens distortion coefficient; The stereo vision system of any of configurations (1) to (3), further comprising a slow calibration procedure configured to generate a third stereo calibration parameter for implementing one or both of the above.

[0082] (5) The stereo vision system of any of configurations (1) to (4), wherein the automatic system calibration comprises an absolute range calibration procedure configured to generate stereo calibration parameters for relative camera yaw based on a non-negative disparity constraint.

[0083] (6) The stereo vision system of any of configurations (1) to (5), wherein the automatic system calibration comprises an absolute range calibration procedure configured to generate stereo calibration parameters for relative camera yaw based on dimensions of objects in the scene in the plurality of stereo images and focal lengths of the first camera sensor and the second camera sensor.

[0084] (7) The stereo vision system of (6), wherein the object is a traffic sign. (8) The stereo vision system of (6), wherein the object is a license plate.

[0085] (9) The stereo vision system of any of configurations (1) to (8), wherein the automatic system calibration comprises an absolute range calibration procedure configured to generate stereo calibration parameters for relative camera yaw based on vehicle odometry.

[0086] (10) The stereo vision system of any of configurations (1) to (9), wherein the automatic system calibration comprises a calibration management procedure configured to manage operating schedules of the slow calibration procedure, the intermediate calibration procedure, the fast calibration procedure, and the absolute range calibration procedure.

[0087] (11) The stereo vision system of any of configurations (1) to (10), wherein the automatic system calibration comprises a calibration health indicator that indicates accuracy of the automatic system calibration.

[0088] (12) A stereo vision system of any of configurations (1) to (11), wherein stereo matching consists of depth estimation confidence scores for each pixel of multiple stereo images.

[0089] (13) The stereo vision system of any of configurations (1) to (12), wherein the first camera sensor and the second camera sensor are mounted on any of a vehicle, car, truck, a traffic light, a light pole, the left and right side mirrors, respectively, of the vehicle, the roof line, the upper left and right sides, respectively, of the windshield, and the left and right headlamps, respectively, of the vehicle.

[0090] (14) A stereo vision system of any of configurations (1) to (13), where the cost function is based on the number of valid pixels in the disparity map. (15) A cost function is optimized based on intrinsic camera parameters of a stereo vision system of any of the configurations (1) to (14).

[0091] (16) A stereo vision system of any of the configurations (1) to (15), in which a cost function is optimized based on the extrinsic camera parameters. (17) The stereo vision system of any of configurations (1) to (16), wherein the at least one processor is configured to provide a calibration health indicator.

[0092] (18) The stereo vision system of any of configurations (1) to (17), further comprising a near-range sensor system configured to provide 3D data for objects within a range closer than a minimum depth range of the first camera sensor and the second camera sensor, wherein the at least one processor is configured to combine 3D data resulting from the first sensor signal and the second sensor signal with 3D data provided by the near-range sensor system.

[0093] (19) A stereo vision system of configuration (18), wherein the near range sensor system comprises a pair of camera sensors having a wider field of view and a shorter baseline distance than the first camera sensor and the second camera sensor.

[0094] (20) The stereo vision system of configuration (18), wherein the near range sensor system comprises a third camera sensor forming a trinocular stereo system with the first camera sensor and the second camera sensor such that the first camera sensor and the second camera sensor have a shorter baseline length than the first camera sensor and the third camera sensor.

[0095] (21) The near-range sensor system is a stereo vision system consisting of a time-of-flight camera (18). (22) The stereo vision system of any of configurations (1) to (21), further comprising an active illumination device configured to emit visible or infrared radiation toward the field of view of the first camera sensor and the second camera sensor.

[0096] (23) A stereo vision system of configuration (22), wherein the active lighting device is configured to alternate between emitting radiation and not emitting radiation, and to emit radiation in synchronization with an exposure interval of the first camera sensor and an exposure interval of the second camera sensor.

[0097] (24) The stereo vision system of configuration (22), wherein the active illumination component comprises any one or any combination of a vertical cavity surface emitting laser array, a lamp emitting radiation in the visible spectral range, and a lamp emitting radiation in the near infrared spectral range.

[0098] (25) The stereo vision system of any of configurations (1) to (24), wherein at least one processor is configured to calculate motion-derived structure data from the first sensor signal and from the second sensor signal and use the motion-derived structure data to estimate 3D positions of objects that are closer than a minimum depth range of the first camera sensor and the second camera sensor.

[0099] Methods for object detection according to the techniques described herein may include various processes. An exemplary method includes a combination of processes (26) and (27) as follows:

[0100] (26) A computer-implemented method performed by one or more processors for detection of an object in an image, comprising: identifying connected component regions on the object by color; determining an aspect ratio of each of the connected component regions; determining a distance between the connected component regions; and identifying the object as a known object based on the aspect ratio and the distance between the connected component regions, wherein the presence, location, and size of the known object are determined from the image.

[0101] (27) The method of process (26), wherein identifying the connected component regions identifies the color as being red and identifying the object identifies the object as being a stop sign. A method for calibrating a stereo vision system according to the techniques described herein may include various processes. An exemplary method includes a combination of processes (28) to (33) as follows:

[0102] (28) A computer-implemented calibration method performed by one or more processors to calibrate a stereo vision system including a first camera sensor configured to sense a first reflected energy in a first image and generate a first sensor signal based on the first reflected energy and a second camera sensor configured to sense a second reflected energy in a second image and generate a second sensor signal based on the second reflected energy, the computer-implemented calibration method comprising: generating stereo images from the first sensor signal and the second sensor signal; correcting the stereo images using stereo calibration parameters to result in corrected stereo images; performing stereo matching on the corrected stereo images; and performing an automatic system calibration using results of the stereo matching, wherein the automatic system calibration is based on minimizing a cost function.

[0103] (29) The method of process (28), wherein the automatic system calibration comprises any one or any combination of a fast calibration procedure configured to generate first stereo calibration parameters to compensate for high frequency perturbations based on tracking image points corresponding to features in the first image and the second image, an intermediate calibration procedure configured to generate second stereo calibration parameters to compensate for intermediate frequency perturbations based on optimizing a number of effective pixels for dense stereo block matching as a function of relative camera pitch and relative camera roll, and a slow calibration procedure configured to generate third stereo calibration parameters to compensate for low frequency perturbations based on optimizing a number of effective pixels for dense stereo block matching as a function of relative camera pitch, relative camera roll, relative camera yaw, and relative camera position.

[0104] (30) The method of process (28) or process (29), wherein the automatic system calibration comprises an absolute range calibration procedure configured to generate stereo calibration parameters for relative camera yaw based on a non-negative disparity constraint.

[0105] (31) The method of any one of processes (28) to (30), wherein the automatic system calibration comprises an absolute range calibration procedure configured to generate stereo calibration parameters for relative camera yaw based on dimensions of objects in the scene in the multiple stereo images and focal lengths of the first camera sensor and the second camera sensor.

[0106] (32) The method of any one of processes (28) to (31), wherein the automatic system calibration comprises an absolute range calibration procedure configured to generate stereo calibration parameters for relative camera yaw based on vehicle odometry measurements.

[0107] (33) The method of any one of processes (28) to (32), wherein the automatic system calibration comprises a calibration management procedure configured to manage an operating schedule of the slow calibration procedure, the intermediate calibration procedure, the fast calibration procedure, and the absolute range calibration procedure.

[0108] A non-transitory computer-readable medium storing computer-executable code for calibrating a stereo vision system according to the techniques described herein may be embodied in different configurations. Exemplary configurations include combinations of configurations (34) through (39), as follows:

[0109] (34) A non-transitory computer-readable medium storing computer-executable code that, when executed by one or more processors, causes the one or more processors to calibrate a stereo vision system including a first camera sensor configured to sense a first reflected energy in a first image and generate a first sensor signal based on the first reflected energy and a second camera sensor configured to sense a second reflected energy in a second image and generate a second sensor signal based on the second reflected energy, the method comprising: generating stereo images from the first sensor signals and the second sensor signals; correcting the stereo images using stereo calibration parameters to result in corrected stereo images; performing stereo matching on the corrected stereo images; and performing an automatic system calibration using results of the stereo matching, wherein the automatic system calibration is based on minimizing a cost function.

[0110] (35) The non-transitory computer-readable medium of configuration (34), wherein the automatic system calibration comprises any one or any combination of: a fast calibration procedure configured to generate first stereo calibration parameters to compensate for high frequency perturbations based on tracking image points corresponding to features in the first image and the second image; an intermediate calibration procedure configured to generate second stereo calibration parameters to compensate for intermediate frequency perturbations based on optimizing a number of effective pixels for dense stereo block matching as a function of relative camera pitch and relative camera roll; and a slow calibration procedure configured to generate third stereo calibration parameters to compensate for low frequency perturbations based on optimizing a number of effective pixels for dense stereo block matching as a function of relative camera pitch, relative camera roll, relative camera yaw, and relative camera position.

[0111] (36) A non-transitory computer-readable medium of configuration (34) or configuration (35), wherein the automatic system calibration comprises an absolute range calibration procedure configured to generate stereo calibration parameters for relative camera yaw based on a non-negative disparity constraint.

[0112] (37) The non-transitory computer-readable medium of any one of configurations (34) to (36), wherein the automatic system calibration comprises an absolute range calibration procedure configured to generate stereo calibration parameters for relative camera yaw based on dimensions of objects in the scene of the multiple stereo images and focal lengths of the first camera sensor and the second camera sensor.

[0113] (38) The non-transitory computer-readable medium of any one of (34) to (37), wherein the automatic system calibration comprises an absolute range calibration procedure configured to generate stereo calibration parameters for relative camera yaw based on vehicle odometry.

[0114] (39) The non-transitory computer-readable medium of any one of configurations (34) to (38), wherein the automatic system calibration comprises a calibration management procedure configured to manage operating schedules of a slow calibration procedure, an intermediate calibration procedure, a fast calibration procedure, and an absolute range calibration procedure.

[0115] It should be understood that the embodiments and examples described herein have been chosen and described to illustrate the principles, methods, and processes of the technology of the present invention and its practical applications, thereby enabling those skilled in the art to utilize the technology of the present invention in various embodiments, with various modifications suited to the particular use contemplated. Although specific embodiments of the technology of the present invention have been described, these embodiments should not be considered exhaustive. Other embodiments and variations not specifically described herein, which will be apparent to those skilled in the art, are within the scope of the technology.

[0116] Unless otherwise stated, the terms "approximately" and "about" are used in some embodiments to mean within ±20% of a target value, in some embodiments within ±10% of a target value, in some embodiments within ±5% of a target value, and even in some embodiments within ±2% of a target value. The terms "approximately" and "about" can include the target value. The term "essentially" is used to mean within ±3% of a target value.

[0117] The techniques described herein may be embodied as a method in which at least some of the acts are described. Acts performed as part of the method may be ordered in any suitable manner. Thus, embodiments may be implemented in which acts are performed in an order different from that described, which may include performing some acts simultaneously despite being described as sequential acts in an example embodiment. Furthermore, a method may include more acts than are described in some embodiments and fewer acts than are described in other embodiments.

[0118] Various aspects of the disclosure may be used alone or in combination, or in various arrangements not specifically discussed in the embodiments described in the foregoing content, and therefore are not limited in their application to the details and arrangements of components set forth in the foregoing description or illustrated in the drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0119] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify claim elements does not, by itself, imply a priority, precedence, or order of one claim element relative to another claim element, or the temporal order in which acts of a method are performed, but is merely used as a label (in the absence of ordinary terminology) to distinguish one claim element having a certain name from another element having the same name to distinguish between claim elements.

[0120] The phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof, as well as additional items.

[0121] The indefinite articles "a" and "an," as used in the specification and claims, unless clearly indicated to the contrary, should be understood to mean "at least one."

[0122] Any use of the phrase "at least one" in reference to a list of one or more elements should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of every element specifically listed in the list of elements, and not excluding any combination of elements in the list of elements. This definition also allows for the optional presence of elements other than those specifically identified in the list of elements to which the phrase "at least one" refers, whether or not related to those specifically identified elements.

[0123] Any use of the phrase "equal" or "the same" in reference to two values ​​(e.g., distance, width, etc.) means that the two values ​​are the same within manufacturing tolerances. Thus, two values ​​being equal or the same may mean that the two values ​​differ from each other by ±5%.

[0124] The phrase "and / or" as used in the specification and claims should be understood to mean "either or both" of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with "and / or" should be interpreted in the same manner, i.e., "one or more" of the elements so conjoined. Other elements may optionally be present other than the elements specifically identified by the "and / or" clause, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, a reference to "A and / or B", when used with open-ended language such as "comprising", can refer in one embodiment to only A (optionally including elements other than B), in another embodiment to only B (optionally including elements other than A), in yet another embodiment to both A and B (optionally including other elements), etc.

[0125] As used herein and in the claims, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" shall be interpreted as being inclusive, i.e., the inclusion of at least one of several elements or a list of elements, but also including a plurality, and optionally including additional items not listed. Only when used in a term clearly indicating the contrary, such as "only one of" or "exactly one of," or in the claims, "consisting of" shall refer to the inclusion of only one element of several elements or a list of elements. In general, the term "or" as used herein shall only be interpreted as indicating exclusive alternatives (i.e., "one or the other but not both") when preceded by a term of exclusivity, such as "either," "one of," "only one of," or "exactly one of." "Consisting essentially of," when used in the claims, shall have its ordinary meaning as used in the field of patent law.

[0126] The term "substantially," as used herein, may be taken to mean, in some embodiments, within 95% of a target value, in some embodiments, within 98% of a target value, in some embodiments, within 99% of a target value, and in some embodiments, within 99.5% of a target value. In some embodiments, the term "substantially" may equal 100% of a target value.

[0127] Also, some of the embodiments described above may be implemented as one or more methods, of which some examples are provided. The acts performed as part of a method may be ordered in any suitable manner. Thus, embodiments may be constructed in which acts are performed in an order different from that illustrated or described herein, which may include performing some acts simultaneously despite being shown as sequential acts in an example embodiment.

[0128] Additionally, although advantages of the present invention have been illustrated, it should be understood that not every embodiment of the present invention includes every described advantage. Some embodiments may not implement features described herein as advantageous. Accordingly, the foregoing description and accompanying drawings are by way of example only.

Claims

1. At least one computer processor in communication with the at least two camera sensors, the at least one computer processor comprising: receiving video signals from the at least two camera sensors in real time or near real time, each of the at least two camera sensors configured to sense reflected energy of a first scene and generate a video signal based on the reflected energy; Processing the video signal, generating three-dimensional (3D) data corresponding to the first scene; calibrating the at least two camera sensors by automatically updating camera parameters of the at least two camera sensors for each frame of the video signals from the at least two camera sensors; at least one computer processor configured to: a first camera sensor configured to sense first reflected energy of the first scene and generate a first video signal based on the first reflected energy; a second camera sensor configured to sense second reflected energy of the first scene and generate a second video signal based on the second reflected energy; The at least one computer processor comprises: generating a corrected stereo image from the first and second video signals utilizing stereo calibration parameters; performing stereo matching on the rectified stereo images; generating a disparity map based on the rectified stereo images; and calibrating the first and second camera sensors by performing an automatic system calibration based on optimizing a cost function, the automatic system calibration including an absolute range calibration procedure configured to generate stereo calibration parameters for calibrating a relative camera yaw of the first and second camera sensors; A stereo vision system for a vehicle, wherein the absolute range calibration procedure is configured to generate the stereo calibration parameters based on a non-negative disparity constraint.

2. At least one computer processor in communication with at least two camera sensors, the at least one computer processor comprising: The at least two camera sensors are connected in real time or near real time. receiving a video signal from a camera sensor, wherein each of the at least two camera sensors is configured to sense reflected energy of a first scene and generate a video signal based on the reflected energy; Processing the video signal, generating three-dimensional (3D) data corresponding to the first scene; calibrating the at least two camera sensors by automatically updating camera parameters of the at least two camera sensors for each frame of the video signals from the at least two camera sensors; at least one computer processor configured to: a first camera sensor configured to sense first reflected energy of the first scene and generate a first video signal based on the first reflected energy; a second camera sensor configured to sense second reflected energy of the first scene and generate a second video signal based on the second reflected energy; The at least one computer processor comprises: generating a corrected stereo image from the first and second video signals utilizing stereo calibration parameters; performing stereo matching on the rectified stereo images; generating a disparity map based on the rectified stereo images; and calibrating the first and second camera sensors by performing an automatic system calibration based on optimizing a cost function, the automatic system calibration including an absolute range calibration procedure configured to generate stereo calibration parameters for calibrating a relative camera yaw of the first and second camera sensors; The absolute range calibration procedure is configured to generate the stereo calibration parameters based on dimensions of objects in the corrected stereo images and focal lengths of the first and second camera sensors.

3. At least one computer processor in communication with at least two camera sensors, the at least one computer processor comprising: receiving video signals from the at least two camera sensors in real time or near real time, each of the at least two camera sensors configured to sense reflected energy of a first scene and generate a video signal based on the reflected energy; Processing the video signal, generating three-dimensional (3D) data corresponding to the first scene; calibrating the at least two camera sensors by automatically updating camera parameters of the at least two camera sensors for each frame of the video signals from the at least two camera sensors; at least one computer processor configured to: a first camera sensor configured to sense first reflected energy of the first scene and generate a first video signal based on the first reflected energy; a second camera sensor configured to sense second reflected energy of the first scene and generate a second video signal based on the second reflected energy; The at least one computer processor comprises: generating a corrected stereo image from the first and second video signals utilizing stereo calibration parameters; performing stereo matching on the rectified stereo images; generating a disparity map based on the rectified stereo images; and calibrating the first and second camera sensors by performing an automatic system calibration based on optimizing a cost function, the automatic system calibration including an absolute range calibration procedure configured to generate stereo calibration parameters for calibrating a relative camera yaw of the first and second camera sensors; The automatic system calibration includes: a fast calibration procedure for generating first stereo calibration parameters to calibrate the first and second camera sensors to compensate for high frequency perturbations; an intermediate calibration procedure for generating second stereo calibration parameters to calibrate the first and second camera sensors to compensate for intermediate frequency perturbations; and a slow calibration procedure for generating third stereo calibration parameters to calibrate the first and second camera sensors to compensate for low frequency perturbations.

4. the fast calibration procedure is based on tracking image points corresponding to features in the rectified stereo images; the intermediate calibration procedure is based on optimizing a number of effective pixels for dense stereo block matching as a function of relative camera pitch and relative camera roll; 4. The stereo vision system of claim 3, wherein the slow calibration procedure is based on optimizing a number of effective pixels for dense stereo block matching as a function of any one or any combination of relative camera pitch, relative camera roll, relative camera yaw, and relative camera position.

5. 4. The stereo vision system of claim 3, wherein the automatic system calibration calibrates any one or any combination of a focal length, a principal point, at least one radial lens distortion coefficient, and at least one tangential lens distortion coefficient for each of the first and second camera sensors.

6. 1. A computer-implemented calibration method executed by one or more computer processors for calibrating a stereo vision system including at least two camera sensors, comprising: receiving video signals from the at least two camera sensors in real time or near real time, each of the at least two camera sensors configured to sense reflected energy of a scene and generate a video signal based on the reflected energy; Processing the video signal, generating three-dimensional (3D) data corresponding to the scene; calibrating the at least two camera sensors by automatically updating camera parameters of the at least two camera sensors for each frame of the video signals from the at least two camera sensors; and processing the video signal; the at least two camera sensors include a first camera sensor configured to sense a first reflected energy of the scene and generate a first video signal based on the first reflected energy, and a second camera sensor configured to sense a second reflected energy of the scene and generate a second video signal based on the second reflected energy; The method further comprises: generating a corrected stereo image from the first and second video signals utilizing stereo calibration parameters; performing stereo matching on the rectified stereo images; generating a disparity map based on the rectified stereo images; and calibrating the first and second camera sensors by performing an automatic system calibration based on optimizing a cost function, wherein performing the automatic system calibration includes performing an absolute range calibration procedure to generate stereo calibration parameters for calibrating relative camera yaw of the first and second camera sensors; performing the absolute range calibration procedure to generate the stereo calibration parameters based on a non-negative disparity constraint.

7. A computer-implemented calibration method executed by one or more computer processors for calibrating a stereo vision system including at least two camera sensors, comprising: receiving video signals from the at least two camera sensors in real time or near real time, each of the at least two camera sensors configured to sense reflected energy of a scene and generate a video signal based on the reflected energy; Processing the video signal, generating three-dimensional (3D) data corresponding to the scene; calibrating the at least two camera sensors by automatically updating camera parameters of the at least two camera sensors for each frame of the video signals from the at least two camera sensors; and processing the video signal; the at least two camera sensors include a first camera sensor configured to sense a first reflected energy of the scene and generate a first video signal based on the first reflected energy, and a second camera sensor configured to sense a second reflected energy of the scene and generate a second video signal based on the second reflected energy; The method further comprises: generating a corrected stereo image from the first and second video signals utilizing stereo calibration parameters; performing stereo matching on the corrected stereo images; generating a disparity map based on the rectified stereo images; and calibrating the first and second camera sensors by performing an automatic system calibration based on optimizing a cost function, wherein performing the automatic system calibration includes performing an absolute range calibration procedure to generate stereo calibration parameters for calibrating relative camera yaw of the first and second camera sensors; performing the absolute range calibration procedure to generate the stereo calibration parameters based on dimensions of objects in the corrected stereo images and focal lengths of the first and second camera sensors.

8. A computer-implemented calibration method executed by one or more computer processors for calibrating a stereo vision system including at least two camera sensors, comprising: receiving video signals from the at least two camera sensors in real time or near real time, each of the at least two camera sensors configured to sense reflected energy of a scene and generate a video signal based on the reflected energy; Processing the video signal, generating three-dimensional (3D) data corresponding to the scene; calibrating the at least two camera sensors by automatically updating camera parameters of the at least two camera sensors for each frame of the video signals from the at least two camera sensors; and processing the video signal; the at least two camera sensors include a first camera sensor configured to sense a first reflected energy of the scene and generate a first video signal based on the first reflected energy, and a second camera sensor configured to sense a second reflected energy of the scene and generate a second video signal based on the second reflected energy; The method further comprises: generating a corrected stereo image from the first and second video signals utilizing stereo calibration parameters; performing stereo matching on the corrected stereo images; generating a disparity map based on the rectified stereo images; and calibrating the first and second camera sensors by performing an automatic system calibration based on optimizing a cost function, wherein performing the automatic system calibration includes performing an absolute range calibration procedure to generate stereo calibration parameters for calibrating relative camera yaw of the first and second camera sensors; Performing the automatic system calibration includes: performing a fast calibration procedure to generate first stereo calibration parameters to calibrate the first and second camera sensors to compensate for high frequency perturbations; performing an intermediate calibration procedure to generate second stereo calibration parameters to calibrate the first and second camera sensors to compensate for intermediate frequency perturbations; The first and second camera sensors are calibrated to compensate for low frequency perturbations. and performing a slow calibration procedure to generate third stereo calibration parameters to calibrate the stereo image.

9. the fast calibration procedure is based on tracking image points corresponding to features in the rectified stereo images; the intermediate calibration procedure is based on optimizing a number of effective pixels for dense stereo block matching as a function of relative camera pitch and relative camera roll; 9. The method of claim 8, wherein the slow calibration procedure is based on optimizing a number of effective pixels for dense stereo block matching as a function of any one or any combination of relative camera pitch, relative camera roll, relative camera yaw, and relative camera position.

10. 9. The method of claim 8, wherein performing the automatic system calibration includes calibrating any one or any combination of a focal length, a principal point, at least one radial lens distortion coefficient, and at least one tangential lens distortion coefficient for each of the first and second camera sensors.

11. 1. A non-transitory computer readable medium storing computer executable code that, when executed by one or more processors, causes the one or more processors to perform a calibration method for calibrating a stereo vision system including at least two camera sensors, the method comprising: receiving video signals from the at least two camera sensors in real time or near real time, each of the at least two camera sensors configured to sense reflected energy of a scene and generate a video signal based on the reflected energy; Processing the video signal, generating three-dimensional (3D) data corresponding to the scene; calibrating the at least two camera sensors by automatically updating camera parameters of the at least two camera sensors for each frame of the video signals from the at least two camera sensors; and processing the video signal; the at least two camera sensors include a first camera sensor configured to sense a first reflected energy of the scene and generate a first video signal based on the first reflected energy, and a second camera sensor configured to sense a second reflected energy of the scene and generate a second video signal based on the second reflected energy; The method further comprises: generating a corrected stereo image from the first and second video signals utilizing stereo calibration parameters; performing stereo matching on the corrected stereo images; generating a disparity map based on the rectified stereo images; and calibrating the first and second camera sensors by performing an automatic system calibration based on optimizing a cost function, wherein performing the automatic system calibration includes performing an absolute range calibration procedure to generate stereo calibration parameters for calibrating relative camera yaw of the first and second camera sensors; A computer-readable medium for generating the stereo calibration parameters based on a non-negative disparity constraint by performing the absolute range calibration procedure.

12. A non-transitory computer-readable medium storing computer-executable code that, when executed by one or more processors, causes the one or more processors to perform a calibration method for calibrating a stereo vision system including at least two camera sensors, the method comprising: receiving video signals from the at least two camera sensors in real time or near real time, each of the at least two camera sensors configured to sense reflected energy of a scene and generate a video signal based on the reflected energy; Processing the video signal, generating three-dimensional (3D) data corresponding to the scene; calibrating the at least two camera sensors by automatically updating camera parameters of the at least two camera sensors for each frame of the video signals from the at least two camera sensors; and processing the video signal; the at least two camera sensors include a first camera sensor configured to sense a first reflected energy of the scene and generate a first video signal based on the first reflected energy, and a second camera sensor configured to sense a second reflected energy of the scene and generate a second video signal based on the second reflected energy; The method further comprises: generating a corrected stereo image from the first and second video signals utilizing stereo calibration parameters; performing stereo matching on the corrected stereo images; generating a disparity map based on the rectified stereo images; and calibrating the first and second camera sensors by performing an automatic system calibration based on optimizing a cost function, wherein performing the automatic system calibration includes performing an absolute range calibration procedure to generate stereo calibration parameters for calibrating relative camera yaw of the first and second camera sensors; performing the absolute range calibration procedure to generate the stereo calibration parameters based on dimensions of objects in the corrected stereo images and focal lengths of the first and second camera sensors.

13. A non-transitory computer-readable medium storing computer-executable code that, when executed by one or more processors, causes the one or more processors to perform a calibration method for calibrating a stereo vision system including at least two camera sensors, the method comprising: receiving video signals from the at least two camera sensors in real time or near real time, each of the at least two camera sensors configured to sense reflected energy of a scene and generate a video signal based on the reflected energy; Processing the video signal, generating three-dimensional (3D) data corresponding to the scene; calibrating the at least two camera sensors by automatically updating camera parameters of the at least two camera sensors for each frame of the video signals from the at least two camera sensors; and processing the video signal; the at least two camera sensors include a first camera sensor configured to sense a first reflected energy of the scene and generate a first video signal based on the first reflected energy, and a second camera sensor configured to sense a second reflected energy of the scene and generate a second video signal based on the second reflected energy; The method further comprises: generating a corrected stereo image from the first and second video signals utilizing stereo calibration parameters; performing stereo matching on the corrected stereo images; generating a disparity map based on the rectified stereo images; and calibrating the first and second camera sensors by performing an automatic system calibration based on optimizing a cost function, wherein performing the automatic system calibration includes performing an absolute range calibration procedure to generate stereo calibration parameters for calibrating relative camera yaw of the first and second camera sensors; Performing the automatic system calibration includes: performing a fast calibration procedure to generate first stereo calibration parameters to calibrate the first and second camera sensors to compensate for high frequency perturbations; performing an intermediate calibration procedure to generate second stereo calibration parameters to calibrate the first and second camera sensors to compensate for intermediate frequency perturbations; and performing a slow calibration procedure to generate third stereo calibration parameters to calibrate the first and second camera sensors to compensate for low frequency perturbations.

14. the fast calibration procedure is based on tracking image points corresponding to features in the rectified stereo images; the intermediate calibration procedure is based on optimizing a number of effective pixels for dense stereo block matching as a function of relative camera pitch and relative camera roll; 14. The computer-readable medium of claim 13, wherein the slow calibration procedure is based on optimizing a number of effective pixels for dense stereo block matching as a function of any one or any combination of relative camera pitch, relative camera roll, relative camera yaw, and relative camera position.

15. 14. The computer-readable medium of claim 13, wherein performing the automatic system calibration includes calibrating any one or any combination of a focal length, a principal point, at least one radial lens distortion coefficient, and at least one tangential lens distortion coefficient for each of the first and second camera sensors.

Images