Automatic calibration of stereoscopic imaging devices

Abstract

Systems and methods for calibrating a stereo imaging device are disclosed herein. An example implementation includes receiving, at a processor, a plurality of image pairs from a stereo imaging device, and for each image pair of the plurality of image pairs, computing a respective line delta value indicative of a deviation along a horizontal axis between a first image captured by a first sensor and a second image captured by a second sensor. The implementation also includes determining a median line delta value based on each respective line delta value and subsequently determining a respective set of disparities in one or more features in the first image and the second image based on the median line delta value. The method also includes computing a column delta value and calibrating the stereo imaging device using the median line delta value and the column delta value.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. nonprovisional patent application No. 17 / 081,576, entitled “Automatic Calibration of a Stereo Imaging Device,” filed October 27, 2020, the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] This disclosure relates generally to imaging devices, and more specifically, to the automatic calibration of stereo imaging devices.

[0004] background

[0005] Stereo imaging devices have two lenses used to estimate depth in a captured view. Both lenses are calibrated to provide accurate imaging and measurements. However, we have found that factory-calibrated stereo imaging devices can lose calibration due to environmental factors such as temperature variations, vibrations, or some unknown external influence. In this case, the relative positions (including translation and orientation) between the two lenses and their respective sensors deviate from their original positions set during factory calibration. Since some calibration parameters obtained during calibration depend on the relative positions between the two lenses and the two sensors, this change reduces the accuracy of the calibration parameters and negatively impacts the acquired image, which in turn negatively affects the accuracy of subsequent processes such as image analysis and object counting.

[0006] Therefore, improvements to stereoscopic imaging equipment are needed. Summary of the Invention

[0007] The following is a simplified overview of one or more aspects to provide a basic understanding of such aspects. This invention is not a comprehensive review of all anticipated aspects, nor is it intended to identify key or essential elements of all aspects, nor to depict the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that follows.

[0008] An exemplary implementation includes a method for calibrating a stereo imaging device, comprising: receiving, at a processor, a plurality of image pairs from the stereo imaging device, wherein each image pair includes a first image of a scene captured by a first sensor of the stereo imaging device and a second image of the scene captured by a second sensor of the stereo imaging device. The method further includes: at the processor and for each of the plurality of image pairs, calculating a corresponding row increment value, the corresponding row increment value indicating a deviation along a horizontal axis between the first image captured by the first sensor and the second image captured by the second sensor. The method further includes: determining a median row increment value based on each corresponding row increment value. The method further includes: determining a corresponding set of disparities in one or more features of the first and second images based on the median row increment value. The method further includes: calculating a column increment value, the column increment value being equal to the difference between a median disparity value based on each disparity in the corresponding set of disparities and a predetermined disparity value, wherein the column increment value indicates a deviation along a vertical axis between the first image captured by the first sensor and the second image captured by the second sensor. The method further includes: calibrating the stereo imaging device using the median row increment value and the column increment value.

[0009] Another exemplary implementation includes an apparatus for calibrating a stereo imaging device, comprising a memory and a processor communicating with the memory. The processor is configured to: receive a plurality of image pairs from the stereo imaging device, wherein each image pair includes a first image of a scene captured by a first sensor of the stereo imaging device and a second image of a scene captured by a second sensor of the stereo imaging device. The processor is configured to: for each of the plurality of image pairs, calculate a corresponding row increment value indicating a deviation along a horizontal axis between the first image captured by the first sensor and the second image captured by the second sensor. The processor is configured to: determine a median row increment value based on each corresponding row increment value. The processor is configured to: determine a corresponding set of disparities in one or more features of the first and second images based on the median row increment value. The processor is configured to: calculate a column increment value equal to the difference between the median disparity value based on each of the corresponding disparity sets and a predetermined disparity value, wherein the column increment value indicates a deviation along a vertical axis between the first image captured by the first sensor and the second image captured by the second sensor. The processor is configured to: calibrate the stereo imaging device using the median row increment value and the column increment value.

[0010] Another exemplary implementation includes an apparatus for calibrating a stereo imaging device, comprising: means for receiving, at a processor, a plurality of image pairs from the stereo imaging device, wherein each image pair includes a first image of a scene captured by a first sensor of the stereo imaging device and a second image of the scene captured by a second sensor of the stereo imaging device. Additionally, the apparatus further includes: means for calculating, at the processor and for each of the plurality of image pairs, a corresponding row increment value, the corresponding row increment indicating a deviation along a horizontal axis between the first image captured by the first sensor and the second image captured by the second sensor. The apparatus further includes: means for determining a median row increment value based on each corresponding row increment value. The apparatus further includes: means for determining a corresponding set of disparities in one or more features of the first and second images based on the median row increment value. The apparatus further includes: means for calculating a column increment value equal to the difference between a median disparity value based on each of the corresponding disparity sets and a predetermined disparity value, wherein the column increment value indicates a deviation along a vertical axis between the first image captured by the first sensor and the second image captured by the second sensor. The device further includes: means for calibrating the stereo imaging device using median row increment values ​​and column increment values.

[0011] Another exemplary implementation includes a computer-readable medium for calibrating a stereo imaging device, executable by a processor to: receive at the processor a plurality of image pairs from the stereo imaging device, wherein each image pair includes a first image of a scene captured by a first sensor of the stereo imaging device and a second image of the scene captured by a second sensor of the stereo imaging device. Additionally, the instructions are further executable to: at the processor and for each of the plurality of image pairs, calculate a corresponding row increment value indicating a deviation along a horizontal axis between the first image captured by the first sensor and the second image captured by the second sensor. The instructions are further executable to: determine a median row increment value based on each corresponding row increment value. The instructions are further executable to: determine a corresponding set of disparities in one or more features of the first and second images based on the median row increment value. The instructions are further executable to: calculate a column increment value equal to the difference between the median disparity value based on each of the corresponding disparity sets and a predetermined disparity value, wherein the column increment value indicates a deviation along a vertical axis between the first image captured by the first sensor and the second image captured by the second sensor. This instruction can be further executed to: calibrate the stereo imaging device using the median row increment value and column increment value.

[0012] To achieve the foregoing and related objectives, one or more aspects include the features fully described below and specifically pointed out in the claims. The following description and drawings illustrate certain illustrative features of one or more aspects in detail. However, these features indicate only a few of the various ways in which the principles of each aspect can be employed, and this description is intended to encompass all such aspects and their equivalents. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate one or more exemplary aspects of this disclosure, wherein dashed lines may denote optional components and, together with the detailed description, serve to explain their principles and implementation.

[0014] Figure 1 This is an example of two misaligned images from a sensor in a stereo imaging device that requires calibration, according to an exemplary aspect of this disclosure.

[0015] Figure 2 Examples of images output from a calibrated stereoscopic imaging device according to exemplary aspects of this disclosure.

[0016] Figure 3 Two histograms are depicted that map image parallax to image features according to an exemplary aspect of this disclosure.

[0017] Figure 4 This is a block diagram of a computing device that performs an automatic calibration component according to an exemplary aspect of this disclosure.

[0018] Figure 5 This is a flowchart illustrating a method for calibrating a stereo imaging device according to an exemplary aspect of this disclosure.

[0019] Figure 6 This is a flowchart illustrating a method for selecting row increment values ​​for calibration according to an exemplary aspect of this disclosure.

[0020] Figure 7 This is a flowchart illustrating a method for clustering multiple non-moving features in a corresponding disparity set according to an exemplary aspect of this disclosure. Detailed Implementation

[0021] Various aspects will now be described with reference to the accompanying drawings. In the following description, numerous specific details are set forth for illustrative purposes in order to provide a thorough understanding of one or more aspects. However, it will be apparent that such aspects can be practiced without these specific details.

[0022] This disclosure describes an automatic calibration method for adjusting the optical center of the right / left sensor in a stereo imaging device based on multiple image pairs from two sensors. This allows the image positions of identical physical features in the calibrated images to be aligned and the parallax between them to accurately determine the distance from the physical feature to the image plane of the sensor.

[0023] Traditionally, manual calibration works on the assumption that most central areas of an image are free of objects above ground level, and that feature points extracted for automatic calibration are derived from ground level. Due to this limitation, images (e.g., snapshots, frames, etc.) are taken in clear traffic conditions, and the derived calibration adjustments need to be double-checked before being applied to the device. These limitations prevent the use of manual calibration in real-time on the device.

[0024] This disclosure proposes an improvement over conventional calibration techniques, enabling adjustments to be computed directly on the device without human intervention. The computation is performed in two separate phases. The first phase involves performing row increment computation only on a set of images captured within a first (e.g., relatively short) time period. Only a relatively short time is required because row increment computation may use features from all altitudes. The second phase involves performing column increment computation, where the row increment is set to the row increment value derived from the first phase. In some respects, the second phase captures more frames than the first phase within a second (e.g., relatively long) time period (e.g., ≥45 minutes) to reduce the influence of objects above ground in the scene. Column increment computation requires reliable feature extraction only on the ground. The ground feature set requires more frames.

[0025] Figure 1 Example 100 depicts two misaligned images from sensors in a stereo imaging device 106 requiring calibration, according to an exemplary aspect of this disclosure. Image 102 was captured by a left sensor (e.g., sensor 108) of the stereo imaging device 106, and image 104 was captured by a right sensor (e.g., sensor 110) of the stereo imaging device 106. Although both images show the same objects (e.g., doors, doorknobs, and carpets) from a top-down view, the objects are positioned much farther away relative to each image than they are expected. As shown, the planar patterns in the images are derived from the ground, and the parallax between the left and right images does not reflect the correct camera height.

[0026] Figure 2 Example 200 depicts image output (i.e., images 202, 204) from a calibrated stereo imaging device 106 according to exemplary aspects of this disclosure. This is achieved by using the automatic calibration component 415 of this disclosure (in... Figure 4As described in [the document], misalignment can be corrected to produce the correct image output, where all common features in images 202 and 204 are spaced out by appropriate distances (e.g., depending on the distance between sensors 108 and 110 and the parallax). Finally, the planar pattern in the image originates from the ground, and the parallax between the left and right sensors reflects the correct camera height.

[0027] Figure 3 Two histograms, 300 and 301, depict image disparity mapping to image features according to an exemplary aspect of this disclosure. To achieve proper alignment, the disparity between different features in the image is determined. These disparities are particularly helpful in calculating column increment values, and in... Figure 5 Further discussion is needed.

[0028] Figure 4 This is a block diagram of a computing device 400 performing an automatic calibration component 415 according to an exemplary aspect of this disclosure. Theoretically, the calibration parameters include four sets of parameters:

[0029] as well as

[0030]

[0031] in as well as These are inherent parameters of the left and right sensors. and Indicates the optical center of the left and right imaging systems. and α represents the focal length of the left and right lenses. L and α R This represents the skew factor of the left and right sensors, and is often set to zero for many modern sensors. and This indicates the distortion of the left and right lenses. and This indicates the relative translation and rotation of the right camera lens compared to the left camera lens.

[0032] Of the 26 calibration parameters mentioned above, focal length, lens distortion, and skew factor are characteristic parameters of the two lenses used and do not change with environmental conditions. Additionally, relative translation... It is considered unrelated to environmental changes because in stereo imaging devices, the lens and sensor are fixed to a circuit board. The remaining seven parameters... and It is susceptible to changes in the relative positions of the two lenses and the two sensors.

[0033] In one exemplary aspect, the optical axes of the two lenses are parallel to each other, the two sensors are located in the same plane, and the baseline connecting the optical centers of the two sensors is aligned with the rows of the pixel arrays of the two sensors. In this typical hardware setup, the effect of these seven calibration parameters can be... The variation is approximated by the change. In this disclosure, as implemented by the automatic calibration component 415, the variation is represented as... It is represented by two parameters: the column increment value of the optical center of the right lens. and row increment value These two parameters, in pixels of the original image, are set to zero during factory calibration.

[0034] Assume the original left and right images are represented as I. L and I R The corresponding corrected image is represented by J. L and This indicates that equation (1) can be used. Equation (2) is constructed using bilinear interpolation:

[0035]

[0036] in It is the focal length, and These are the optical centers of the two corrected images. It is set to a fixed point, and The selection is done programmatically so that the left and right corrected images share a common rectangular region, which is mapped to I. L and I R It is within the boundaries of both and is the same size as the original image.

[0037] For a given point in the 3D world Its X-axis is aligned with the baseline, its Y-axis lies in the image plane, and its Z-axis is perpendicular to the image plane. In J L The projection in is represented as (x l y l ), and it in The projection in is represented as (x r y r For those without any loss of calibration For stereoscopic imaging devices, the relationship in equation (3) holds:

[0038]

[0039] Equivalently, given the projections of a point in two calibrated images, the column offset (or parallax) of the projections can be used to infer the height of the corresponding point in the 3D world. Typically, the projections of the same point in two calibrated images are found by using some type of cost-minimizing process to search for matching points in the same row of the two images.

[0040] If the device loses calibration, one or both of the equality relationships in equation (3) will no longer hold. Therefore, the projection of the same point will not appear in the same row of the two calibrated images (row misalignment), and / or the offset of the projections in the two images cannot be used to correctly infer the height (column misalignment). Thus, the automatic calibration component 415 finds the optimal result. Make equation (3) hold as close as possible or equivalently as possible:

[0041]

[0042] Generally, the row alignment of a corrected image is mainly affected by... The column alignment is mainly affected by Influence. The effect on line alignment is independent of the distance from the point of interest in the scene to the stereo imaging device, and when Line alignment is considered to be achieved when the pixel size is less than half a pixel. On the other hand, The impact of column alignment depends on the distance of the object of interest in the scene from the camera. In the same... In this case, the error introduced by height measurement differs for objects at different heights. The farther the object of interest is from the camera, the greater the error introduced by the same number of measurements. The larger the introduced error, the greater the maximum permissible column increment. Determined by focal length and camera height. The maximum permissible error for a given ground level height measurement, and the maximum permissible column increment. It is inversely proportional to the square of the camera height.

[0043] Figure 5 This is a flowchart illustrating a method 500 for calibrating a stereoscopic imaging device according to an exemplary aspect of this disclosure. (See also:) Figure 4 and Figure 5 In operation, the computing device 400 can execute the method 500 for calibrating the stereoscopic imaging device via the automatic calibration component 415 executed by the processor 405 and / or memory 410. In some aspects, the computing device 400 is the stereoscopic imaging device 106. In other aspects, the computing device 400 is an external device capable of communicating with the stereoscopic imaging device 106 (e.g., wirelessly via the Internet or via a wired connection such as via a USB cable).

[0044] At block 502, method 500 includes receiving multiple image pairs from a stereo imaging device. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415, and / or receiving component 420 may be configured to, or may include, means for receiving multiple image pairs from a stereo imaging device (e.g., computing device 400), wherein each image pair includes a first image of a scene captured by a first sensor of the stereo imaging device (e.g., image 102) and a second image of a scene captured by a second sensor of the stereo imaging device (e.g., image 104). In some aspects, the scene includes one or more feature points, wherein the first image includes a first set of feature points corresponding to the one or more feature points, and wherein the second image includes a second set of feature points corresponding to the one or more feature points.

[0045] Because the images may be misaligned, the stereo imaging device 106 needs to be calibrated. As mentioned earlier, the automatic calibration component 415 finds the optimal value of the optimization equation (3). In one aspect, row and column increments can be calculated simultaneously based on a single image pair from both the left and right sensors. To increase the reliability of the estimation, the automatic calibration component 415 can perform calculations on a set of image pairs captured over time and can perform row and column increment calculations in two separate stages. The first stage is calculating the row increments because the calculation of row increments is unaffected by objects above ground in the scene and can be completed in a very short time.

[0046] At block 504, method package 500 includes calculating a corresponding row increment value for each of a plurality of image pairs, the corresponding row increment value indicating the deviation along the horizontal axis between a first image captured by a first sensor and a second image captured by a second sensor. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415, and / or row increment calculation component 425 may be configured to, or may include, means for calculating the corresponding row increment value for each of the plurality of image pairs. In some aspects, each of the plurality of image pairs is captured at time intervals.

[0047] let P represents a single image pair. i The row increment estimation. When performing a single pair estimation on N image pairs captured at 1-minute intervals, the following equation applies:

[0048]

[0049] To calculate a single A pair of corrected images J L and It is Each image is divided into a predetermined number of sub-regions. The row increment calculation component 425 can determine the row increment value for each iteration in up to two iterations, where the first iteration is applied to a subset of the sub-regions and the second iteration is applied to all sub-regions until at least three pairs of matching feature points are found.

[0050] In J L Detection of feature point set in relevant regions And in Detect another set of feature points in the same region

[0051] For FP L Each point in The row increment calculation component 425 repeats the following steps. First, the row increment calculation component 425 calculates the data from the FP according to equation (6). R Obtain a subset of points

[0052]

[0053] Where R 1 , and definition A rectangular region is used to collect a subset of feature points. By default, R... 1 =10, and The size can be expressed as

[0054] Secondly, for Each point in The row incremental computation component 425 applies feature extraction functions such as the Kanade-Lucas-Tomasi (KLT) feature tracker. Used as a reference point and Used as the starting point for the KLT feature tracker. The KLT tracker will iteratively update... Until it finds A point in J, whose neighboring pixels are J L In The best match is found among the surrounding neighboring pixels. The last point of the KLT tracker is represented as... Therefore, from The updated subset was obtained And removed Repeating points in the text.

[0055] Third, for Each point in Line incremental calculation component 425 calculates In The small window centered on J L In The cross-correlation between the central windows. If the correlation is less than 0.9, then from Remove from

[0056] Fourth, the row incremental calculation component 425 checks the final... The size of . If it is zero, then No matching points found. If there are more than one, then... There are multiple matching points, and It was ignored.

[0057] In response to FP L Each point in After repeating the above steps, the row increment calculation component 425 obtains a set of matching pairs with corresponding row offsets:

[0058]

[0059] In some respects, the row incremental calculation component 425 can then be derived from d y The outlier removal algorithm is applied to MP at different angles, and matching feature points that are close to regions with glare (i.e., saturated pixels in the image) are rejected. The final set of row offsets is obtained as follows:

[0060]

[0061] if The row increment is then calculated as follows:

[0062]

[0063] if The row increment calculation component 425 then recalculates the row increment for all sub-regions in the image during the second iteration. If for the second iteration... The algorithm then stops and the incremental calculation component 425 reports an error of no feature found on the stereo imaging device.

[0064] At block 506, method 500 includes determining a median row increment value based on each corresponding row increment value. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415, and / or median row increment determination component 430 may be configured to, or may include, means for determining the median row increment value based on each corresponding row increment value.

[0065] In some respects, the median row increment value is a minimum mean absolute error (MMAE) estimate for each corresponding row increment value. Therefore, the median row increment determination component 430 obtains the minimum mean absolute error (MMAE) estimate of the row increment based on a set of row increment estimates from the image pair set:

[0066]

[0067] At box 508, method 500 includes determining a corresponding set of disparities in one or more features in a first image and a second image based on a median row increment value. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415, and / or disparity determination component 435 may be configured to, or may include, means for determining a corresponding set of disparities in one or more features in a first image and a second image based on a median row increment value. With the median row increment value RD calculated according to equation (10) MMAE Another pair of corrected images J L and It is Created. Parallax determination component 435 then for J L and Stereo matching algorithms, such as efficient large-scale stereo matching algorithms, are applied to create a pair of dense disparity maps D. L and D R For disparity map D L and D R Each disparity map in the first and second images determines a corresponding set of disparities in one or more features. In some respects, the corresponding set of disparities is a disparity histogram H. L and H R They are created based on the following equation:

[0068] H L ={N d N d =Quantity(x,y) Therefore D L (x,y)=d,d=0,…,O} (11)

[0069] H R ={N d N d =Quantity(x,y) Therefore D R (x,y)=d,d=0,…,O} (12)

[0070] Where [0,…,O] is D L and D R The parallax range.

[0071] Assume that the pixels in the set of regions in the image under consideration are primarily projections of real-world points onto the ground horizontally. Two histograms H... L and H R Usually in and It has a single peak (see histogram 300 for example). Additionally, and There should be at most one pixel that is the same or different. However, when there are repeating patterns in the scene or objects above the ground level, the two histograms may have multiple peaks, as shown in histogram 301. Because there are multiple peaks in the histograms, the disparity determination component 435 cannot simply use the disparity with the largest number of pixels.

[0072] In some respects, to address multiple peaks in a histogram, the disparity determination component 435 can first apply a clustering algorithm to the histogram, such as a mean-shift-based clustering algorithm. If there is only one cluster (see 300), the disparity at the peak is used. If there are multiple clusters, then... In When within the predefined range (i.e. Using the disparity at the peak of cluster i Where d threshold The default value is 7, Z ground It is the distance from the ground to the camera, and This is the expected disparity at the ground level. If no clustering satisfies the above range check, the initial ground level disparity cannot be derived, and the algorithm fails in column increment calculation.

[0073] When the stereo imaging device is being calibrated It should be within one pixel. and If the camera is out of calibration, then It may be with and There are a few pixels that are different. and It is obtained at the pixel level because of the disparity map D L and D R Only pixel-level disparity is available. These can be used as initial ground-level disparity in the next step to estimate sub-pixel-level ground-level disparity.

[0074] In summary, once the median row increment value has been estimated as described in box 504, a high-quality dense disparity map should be readily available. In a dense disparity map, objects at different heights will produce different disparities, with the floor contributing the largest number of pixels with the same disparity. If a histogram of all disparities in the disparity map is created and the histogram has a single mode, the peaks will reflect the disparity at the floor level relative to the current column increment setting in histogram 300. If the histogram has multiple peaks as in histogram 301, the presence of multiple large planar objects at different heights in the scene should be considered, and the frame may not be suitable for column increment estimation.

[0075] For a histogram with a single peak, the peak is the perceived floor disparity, and it may differ from the expected disparity at the floor height. The difference between the peak and the expected value is due to the use of an incorrect column increment. Feature points in the image whose disparity is close to the peak can be collected as samples for the next step of estimating the correct column increment.

[0076] At block 510, method 500 includes calculating a column increment value equal to the difference between the median disparity value based on each disparity in a corresponding disparity set and a predetermined disparity value, wherein the column increment value indicates the deviation along the vertical axis between a first image captured by a first sensor and a second image captured by a second sensor. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415, and / or column increment calculation component 440 may be configured to, or may include, means for calculating a column increment value equal to the difference between the median disparity value based on each disparity in a corresponding disparity set and a predetermined disparity value (e.g., ground horizontal disparity).

[0077] In some aspects, certain feature points can be filtered (in Figure 7 (Discussed in the middle). In general, from Figure 7 To determine the retained feature points for the final estimate, column increment calculation component 440 defines the column disparity set D = {D1, ..., D2}. P}, where P is the number of feature points in the set. From there, the column increment calculation component 440 determines the minimum mean absolute error (MMAE)D of the current floor level parallax. floor =median(D), and then obtain the final column increment as follows.

[0078] CD MMAE = D floor – D desired (13)

[0079] Where D desired It is the expected parallax at the floor level at a given camera height.

[0080] In order to determine Consider the following. Once the initial ground parallax... and The column increment calculation component 440 estimates the ground horizontal disparity at the sub-pixel level to more accurately estimate the column increment value. To achieve this goal, the feature detection and tracking algorithm in box 504 is applied to the corrected image pair J created in box 508. L and Although the algorithm is the same as in box 504, its algorithm parameters are significantly different. For example, in box 504, the row incremental calculation component 425 may only detect 100 feature points that are at least 20 pixels apart in the relevant region. In box 510, the column incremental calculation component 440 can detect as many feature points as possible that are 7 pixels apart in the relevant region. For feature tracking, the column incremental calculation component 440 uses a much smaller search range for feature point matching. This step will be applied at most three iterations: the first iteration is applied to a subset of the sub-region, the second iteration is applied to a different subset of that sub-region, and the third iteration is applied to all sub-regions until at least one pair of matching feature points is found.

[0081] From the first iteration, the column increment calculation component 440 uses corner detection algorithms (such as the Harris corner detector) to detect only J. L The set of feature points in the subregion or and Another set of feature points in the same sub-region or

[0082] For FP L Each point in The column increment calculation component 440 repeats the following steps. First, the column increment calculation component 440 obtains the following from FP... R subset of points

[0083]

[0084] Where R 2 , and Define a rectangular area To collect a subset of feature points. By default, R... 2 =1.1, and The column incremental calculation component 440 will The size is represented as

[0085] Secondly, for Each point in The computational component applies feature extraction functions such as the Kanade-Lucas-Tomasi (KLT) feature tracker. Used as a reference point and Used as the starting point for the KLT feature tracker. The KLT tracker will iteratively update... Until it finds A point in J, whose neighboring pixels are J L In The best match is found among the surrounding neighboring pixels. The last point of the KLT tracker is represented as... Therefore, we start from The updated subset was obtained Removed Repeating points in the text.

[0086] Third, for Each point in Column incremental calculation component 440 calculates to In The small window centered on J L In The cross-correlation between the central windows. If the correlation is less than 0.9, then from Remove from

[0087] Fourth, the column incremental calculation component 440 checks the final... The size of . If it is zero, then No matching points found. If there are more than one, then... It has multiple matching points and is ignored by the column incremental calculation component 440.

[0088] against The remaining point Column incremental calculation component 440 verification whether and If verification fails, then no Matching feature points. If Images near areas with glare (i.e., saturated pixels in the image) should be rejected.

[0089] In response to FP L Each point in After repeating the above steps, the column increment calculation component 440 obtains a set of matching pairs with corresponding column offsets:

[0090]

[0091] Then, the column increment calculation component 440 will calculate the sub-pixel horizontal ground disparity without column increment adjustment. Estimated as

[0092]

[0093] Subsequently, column increment calculation component 440 determines the optimal column increment for k = 1…K. For each pair, the column incremental calculation component 440 first starts from J L China and Israel Extract a small patch from the center of the image Then from by A series of blocks centered on the image Where c m This represents a specific column increment from the range (Δx-2, Δx+2), and Δx is derived by the column increment calculation component 440.

[0094]

[0095] Since only needs to come from It's a small piece, so there's no need to build the entire corrected image. This enables an efficient reconstruction process. Then, the column incremental calculation component 440 calculates... and The cross-correlation between them is expressed as S(C m ), C m ∈(Δx-2,Δx+2). For the maximum S(C m Choose the optimal

[0096]

[0097] Find the optimal value for each pair in MP. Then, the optimal column increment. Obtain their average value

[0098]

[0099] At block 512, method 500 includes calibrating a stereo imaging device using median row increment values ​​and column increment values. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415, and / or calibration component 445 may be configured to, or may include, means for calibrating the stereo imaging device using median row increment values ​​and column increment values. For example, calibration component 445 may set an initial row increment value to the median row increment value and set an initial column increment value to the determined column increment value.

[0100] Figure 6 This is a flowchart illustrating a method 600 for selecting row increment values ​​for calibration according to an exemplary aspect of this disclosure. Method 600 is optional and may be performed by an automatic calibration component 415 after block 506 and before block 508.

[0101] At block 602, method 600 includes generating a second corresponding disparity set using initial row increment values. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415, and / or generation component 450 may be configured to, or may include, means for generating a second corresponding disparity set using initial row increment values.

[0102] At block 604, method 600 includes a first quality measurement for determining a corresponding set of disparities based on median row increment values. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415, and / or determination component 451 may be configured to, or may include, means for determining the first quality measurement for a corresponding set of disparities based on median row increment values. At block 606, method 600 includes a second quality measurement for determining a second corresponding set of disparities using initial row increment values. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415, and / or determination component 451 may be configured to, or may include means for determining the second quality measurement for a second corresponding set of disparities using initial row increment values.

[0103] In determining the quality measurement, the automatic calibration component 415 can determine the disparity map (i.e., the visual form of the first corresponding disparity set and the second corresponding disparity set). The quality of the disparity map is determined by the row alignment of the left and right corrected images. If the two corrected images are not row aligned, feature points from the same object in world space in the two images will not be on the same line. Feature matching will fail to find the correct disparity, thus failing to obtain the correct height information from the physical feature points to the counting device. The quantitative measurement of disparity quality (QoD) is defined as follows:

[0104] QoD = ∑A large / ∑A small (20)

[0105] Where A large It is the area of ​​large, uniform regions (e.g., those with the same black or white color) in the disparity map that are greater than a given threshold (default is 1000), and A small This refers to the region of small spots in the disparity map with an area smaller than the threshold. Therefore, the first quality measurement is QoD1, and the second quality measurement is QoD2.

[0106] At block 608, method 600 includes determining whether a first mass measurement is greater than a second mass measurement. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415, and / or determining component 451 may be configured to, or may include, means for determining whether the first mass measurement is greater than the second mass measurement. In response to determining that the first mass measurement is greater than the second mass measurement, method 600 proceeds to 610. Otherwise, method 600 proceeds to 612.

[0107] At block 610, method 600 includes calibrating a stereo camera using median line increment values. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415, and / or calibration component 445 may be configured to, or may include, means for calibrating a stereo camera using median line increment values. At block 612, method 600 includes calibrating a stereo camera using initial line increment values. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415, and / or calibration component 445 may be configured to, or may include, means for calibrating a stereo camera using initial line increment values. In other words, automatic calibration component 415 compares the parallax quality with the initial line increment value and the median line increment value and selects the one with the better quality measurement.

[0108] Figure 7 This is a flowchart illustrating a method 700 for clustering multiple non-moving features in a corresponding disparity set according to an exemplary aspect of the present disclosure. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415 and / or clustering component 452 may be configured to, or may include, means for clustering multiple non-moving features in a corresponding disparity set.

[0109] Feature points sampled from a single frame contain their pixel location and disparity. Since feature points are collected around the floor level and may originate from objects above the floor, not all disparity samples can be used to estimate the ground level disparity. To remove outliers, clustering component 452 samples feature points from multiple frames captured one minute apart (e.g., more than 45 frames) and performs location-based filtering on all collected feature points.

[0110] At block 702, method 700 includes selecting a subset of features from a corresponding set of disparities that are within a threshold disparity difference of the peak disparity value. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415, and / or selection component 453 may be configured to, or may include, means for selecting a subset of features from a corresponding set of disparities that are within a threshold disparity difference of the peak disparity value. For example, referring to histogram 300, selection component 453 may select features related to d max Features within a parallax difference of 500.

[0111] At box 704, method 700 includes determining the number of times a corresponding feature appears in multiple image pairs for a subset of corresponding features. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415, and / or determination component 454 may be configured to, or may include, means for determining the number of times a corresponding feature appears in multiple image pairs for a subset of corresponding features. This requires filtering out moving features that do not remain in the frame indefinitely.

[0112] At block 706, method 700 includes determining whether the number of occurrences exceeds a threshold number. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415, and / or determination component 454 may be configured to, or may include, means for determining whether the number of occurrences exceeds the threshold number. If the number of occurrences exceeds the threshold number, method 700 proceeds to 708. Otherwise, method 700 proceeds to 710.

[0113] At block 708, method 700 includes including the corresponding feature among a plurality of non-moving features. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415 and / or including component 455 may be configured to or may include means for including the corresponding feature among a plurality of non-moving features.

[0114] At block 710, method 700 includes not including the corresponding feature among a plurality of non-moving features. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415 and / or including component 455 may be configured to or may include means for not including the corresponding feature among a plurality of non-moving features.

[0115] At block 712, method 700 includes determining whether all features in the subset have been considered. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415, and / or determination component 454 may be configured to, or may include, means for determining whether all features in the subset have been considered. If all of these have been considered, method 700 proceeds to block 714. Otherwise, method 700 returns to block 704 and considers another corresponding feature.

[0116] At block 714, method 700 includes calculating a median disparity value based on a plurality of non-movement features. For example, in one aspect, computing device 400, processor 405, memory 410, automatic calibration component 415, and / or computing component 456 may be configured to, or may include, means for calculating a median disparity value based on a plurality of non-movement features.

[0117] For example, let F n i F is a feature point in the nth frame. n i =(X n i ,Y n i D n i )i∈[1,M n ], where (X n i ,Y n i ) is the location of the feature map, and D n i M is the disparity of the feature point. n This is the number of feature points collected from the nth frame. Filtering will be performed by clustering component 452 on the counting map C, where the value of C at a certain position is the count of feature points collected at that position over time:

[0118] C(i,j) = ∑ k ∑ f 1(i== X k f && j== Y k f ), f ∈[1, M n ], k ∈[1, N](21)

[0119] Where N is the number of frames, and 1 (...) represents the assertion i == X. k f and j == Y k f Indicator functions.

[0120] The neighbor clustering method is applied to the counting graph C, ensuring that the points in each cluster are transitively 8-connected. The total count of each cluster equals the sum of the counts of the points within that cluster. The total count of clusters containing non-moving feature points in the scene should be significantly greater than the total count of clusters containing moving feature points. The total count of each cluster is compared to a predefined threshold; if the total count of a cluster is less than the threshold, all feature points in that cluster are ignored. Otherwise, the feature points in that cluster are used for the final MMAE estimation.

[0121] Although illustrative aspects and / or embodiments have been discussed above, it should be noted that various changes and modifications may be made herein without departing from the scope of the described aspects and / or embodiments as defined by the appended claims. Furthermore, while elements of the described aspects and / or embodiments may be described or claimed in the singular, the plural form is contemplated unless expressly specified as singular. Additionally, all or part of any aspect and / or embodiment may be used in conjunction with all or part of any other aspect and / or embodiment unless otherwise stated.

Claims

1. A method for calibrating a stereo imaging device, comprising: receiving, at a processor, a plurality of image pairs from the stereo imaging device, wherein each image pair includes a first image of a scene captured by a first sensor of the stereo imaging device and a second image of the scene captured by a second sensor of the stereo imaging device; computing, at the processor and for each image pair of the plurality of image pairs, a respective row delta value indicative of a deviation along a horizontal axis between the first image captured by the first sensor and the second image captured by the second sensor; determining a median row delta value based on each respective row delta value; determining a respective set of disparities in one or more features in the first image and the second image based on the median row delta value; computing a column delta value equal to a difference between a median disparity value based on each of the respective set of disparities and a predetermined disparity value, wherein the column delta value is indicative of a deviation along a vertical axis between the first image captured by the first sensor and the second image captured by the second sensor; and calibrating the stereo imaging device using the median row delta value and the column delta value.

2. The method of claim 1, further comprising: generating a second respective set of disparities using an initial row delta value prior to determining the median row delta value; determining a first quality measure of the respective set of disparities based on the median row delta value; determining a second quality measure of the second respective set of disparities using the initial row delta value; and in response to determining that the first quality measure is greater than the second quality measure, calibrating the stereo imaging device using the median row delta value.

3. The method of claim 1, wherein each respective disparity in the respective set of disparities is a respective feature point in an image pair, further comprising: prior to computing the column delta value, clustering a plurality of non-moving features in the respective set of disparities; and computing the median disparity value based on disparities of the plurality of non-moving features.

4. The method of claim 3, wherein a peak disparity value in the respective set of disparities is a floor captured in the plurality of image pairs, and wherein clustering the plurality of non-moving features comprises: selecting, from the respective set of disparities, a subset of features within a threshold disparity difference of the peak disparity value; for each respective feature, determining a number of times the respective feature appears in the plurality of image pairs; and in response to determining that the number of times is greater than a threshold number of times, including the respective feature in the plurality of non-moving features.

5. The method of claim 1, wherein each image pair in the plurality of image pairs is captured a time period apart.

6. The method of claim 1, further comprising: determining whether the respective set of disparities includes more than one peak disparity value; and in response to determining that the respective set of disparities includes more than one peak disparity value, selecting a different image pair and determining another dense disparity map of the different image pair to compute the column delta value. ​ ​ ​ ​ ​ 7. The method of claim 1, wherein the median row delta value is a minimum mean absolute error (MMAE) estimate of each respective row delta value.

8. An apparatus for calibrating a stereo imaging device, comprising: a memory; and a processor in communication with the memory and configured to: receive, from the stereo imaging device, a plurality of image pairs, wherein each image pair includes a first image of a scene captured by a first sensor of the stereo imaging device and a second image of the scene captured by a second sensor of the stereo imaging device; for each image pair of the plurality of image pairs, compute a respective row delta value indicative of a deviation along a horizontal axis between the first image captured by the first sensor and the second image captured by the second sensor; determine a median row delta value based on each respective row delta value; determine a respective set of disparities in one or more features of the first image and the second image based on the median row delta value; compute a column delta value equal to a difference between a median disparity value based on each of the respective set of disparities and a predetermined disparity value, wherein the column delta value is indicative of a deviation along a vertical axis between the first image captured by the first sensor and the second image captured by the second sensor; and calibrate the stereo imaging device using the median row delta value and the column delta value.

9. The apparatus of claim 8, wherein the processor is further configured to: prior to determining the median row delta value, generate a second respective set of disparities using an initial row delta value; determine a first quality measure of the respective set of disparities based on the median row delta value; determine a second quality measure of the second respective set of disparities using the initial row delta value; and in response to determining that the first quality measure is greater than the second quality measure, calibrate the stereo imaging device using the median row delta value.

10. The apparatus of claim 8, wherein each respective disparity of the respective set of disparities is a respective feature point in an image pair, wherein the processor is further configured to: prior to computing the column delta value, cluster a plurality of non-moving features in the respective set of disparities; and compute the median disparity value based on disparities of the plurality of non-moving features.

11. The apparatus of claim 10, wherein a peak disparity value of the respective set of disparities is a floor captured in the plurality of image pairs, and wherein the processor is further configured to cluster the plurality of non-moving features by: selecting, from the respective set of disparities, a subset of features within a threshold disparity difference of the peak disparity value; for each respective feature, determining a number of times the respective feature appears in the plurality of image pairs; and in response to determining that the number of times is greater than a threshold number of times, including the respective feature in the plurality of non-moving features.

12. The apparatus of claim 8, wherein each image pair of the plurality of image pairs is captured a period of time apart. ​ ​ 13. The apparatus of claim 8, wherein the processor is further configured to: determine whether the respective set of disparities includes more than one peak disparity value; and in response to determining that the respective set of disparities includes more than one peak disparity value, select a different image pair and determine another dense disparity map for the different image pair to compute the column delta value.

14. The apparatus of claim 8, wherein the median row delta value is a minimum mean absolute error (MMAE) estimate of each respective row delta value.

15. A computer-readable medium for automatically configuring a region of interest (ROI) associated with a camera, executable by a processor to: receive a plurality of image pairs from a stereo imaging device, wherein each image pair includes a first image of a scene captured by a first sensor of the stereo imaging device and a second image of the scene captured by a second sensor of the stereo imaging device; for each image pair of the plurality of image pairs, compute a respective row delta value indicative of a deviation between the first image captured by the first sensor and the second image captured by the second sensor along a horizontal axis; determine a median row delta value based on each respective row delta value; determine a respective set of disparities in one or more features in the first image and the second image based on the median row delta value; compute a column delta value equal to a difference between a median disparity value based on each of the respective sets of disparities and a predetermined disparity value, wherein the column delta value is indicative of a deviation between the first image captured by the first sensor and the second image captured by the second sensor along a vertical axis; use the median row delta value and the column delta value to calibrate the stereo imaging device. ​

Images