Method and apparatus for efficient data processing of initial correspondence assignment for three-dimensional reconstruction of object

Abstract

To provide systems and methods for efficiently processing data of initial correspondence assignments for three-dimensional reconstruction of an object.SOLUTION: A method includes: receiving a first set of images of a scene and a second set of images of the scene; determining a first pixel fingerprint based on the first set of images and a second pixel fingerprint based on the second set of images; generating a first binary pixel fingerprint based on the first pixel fingerprint and a second binary pixel fingerprint based on the second pixel fingerprint; and determining whether there exists a stereo correspondence between the first pixel fingerprint and the second pixel fingerprint at least in part based on comparing the first binary pixel fingerprint and the second binary pixel fingerprint.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The techniques described herein relate generally to three-dimensional (3D) reconstruction from two-dimensional images of an object or scene, and more particularly to data-efficient techniques for initial correspondence assignments between pairs of two-dimensional images. [Background technology]

[0002] Advanced machine vision systems and their underlying software are increasingly being used in a variety of manufacturing and quality control processes. Machine vision enables faster, more accurate, and repeatable results in the production of both mass-produced and custom products. A typical machine vision system includes one or more cameras aimed at the area of ​​interest, an illumination source to direct appropriate illumination at the area of ​​interest, a frame grabber / image processing element to acquire and transmit the images, a computer or on-board processing device to run machine vision software applications and manipulate the acquired images, and a user interface for interaction.

[0003] One form of 3D vision system is based on stereo cameras, which employ at least two cameras aligned side-by-side on a baseline with a separation of one to several inches between the cameras. Stereo vision-based systems are typically based on epipolar geometry and image rectification. They use correlation-based methods or can be combined with relaxation techniques to find correspondences in the rectified images from two or more cameras. However, the computational effort required by traditional stereo vision systems limits their ability to rapidly produce accurate and fast 3D reconstructions of objects and / or scenes. Summary of the Invention

[0004] The disclosed subject matter provides devices, systems, and methods for efficiently processing data for initial correspondence assignments between pairs of two-dimensional images, e.g., for three-dimensional reconstruction of an object or scene from a pair of two-dimensional images. The inventors recognized that while conventional systems can apply some kind of normalized cross-correlation to pairs of two-dimensional images to determine stereo correspondence, these systems often must expend significant computational effort to perform the necessary calculations (e.g., the number of pixel values ​​that need to be processed, the size of those pixel values, etc.). As discussed further herein, the inventors have developed techniques to improve three-dimensional data reconstruction techniques by efficiently processing initial correspondence assignments for pairs of two-dimensional images, particularly pairs of time-lapse stereo image sequences. A time-lapse stereo image sequence includes a set of images of an object or scene acquired from different viewpoints over time. The inventors note that the described systems and methods are particularly novel in that they derive a binary pixel fingerprint for each pixel fingerprint in a time-lapse stereo image sequence.

[0005] In some aspects, the described systems and methods provide a system for establishing stereo correspondence between two images. The system may include two or more cameras (or sensors) arranged to acquire images of a scene in a manner suitable for generating stereo image correspondence, for example, by positioning two or more cameras to acquire different viewpoints of the scene. Alternatively, the system may include one or more cameras (or sensors) with one or more inverse cameras (or projectors) arranged to acquire images of the scene in a manner suitable for generating pairwise stereo image correspondence. One or more projectors can be used to project a series of light patterns onto the scene, acquiring a set of images from each camera. The set of images acquired from each camera can be referred to as time-lapse images. Each image in the set of images can correspond to one of the series of projected light patterns.

[0006] Thus, time-lapse images and binary time-lapse images can be determined for each camera. A pixel fingerprint at position (i,j) in the time-lapse images can include an ordered set of pixel values ​​collected at position (i,j) and / or different positions in space and / or time relative to position (i,j) from the set of images acquired by each camera. In some embodiments, a pixel fingerprint at position (i,j) in the time-lapse images can include an ordered set of pixel values ​​collected at position (i,j). Additionally or alternatively, in some embodiments, a pixel fingerprint at position (i,j) in the time-lapse images can include an ordered set of pixel values ​​collected at different positions in space relative to position (i,j) from the set of images acquired by each camera. Additionally or alternatively, in some embodiments, a pixel fingerprint at position (i,j) in the time-lapse images can include an ordered set of pixel values ​​collected at different positions in time relative to position (i,j) from the set of images acquired by each camera. Additionally or alternatively, in some embodiments, a pixel fingerprint at position (i,j) in the time-lapse images may include an ordered set of pixel values ​​collected at different positions in space and time relative to position (i,j) from a set of images acquired by each camera.

[0007] A binary pixel fingerprint at location (i,j) in the binary time-lapse images is determined by comparing each set value of the pixel fingerprint at location (i,j) to one or more thresholds and replacing each set value with zero or one based on the one or more thresholds. Optionally, the set values ​​of the pixel fingerprint can be normalized before comparing with the one or more thresholds to generate a corresponding binary pixel fingerprint. A search (e.g., a search along an epipolar line) can be used to determine correspondence between the binary pixel fingerprints from each camera. One or more binary comparisons between pixels (e.g., pixels on or near the epipolar line) can be performed from the binary time-lapse images of each camera, and pixel correspondences can be determined based on the results of the comparisons.

[0008] In some aspects, a system, method, and / or computer-readable storage medium may be provided for determining stereo correspondence between a first pixel fingerprint of a first set of images and a second pixel fingerprint of a second set of images.

[0009] The system can include a processor configured to perform operations of receiving, from one or more image sensors, a first set of images of a scene and a second set of images of the scene, the second set of images being acquired from a different perspective than the first set of images.

[0010] The processor can be configured to perform operations to determine a first pixel fingerprint based on a first set of images. The first pixel fingerprint can include a first set of pixel values ​​at a first pixel location within each image in the first time-lapse image set. The processor can be configured to perform operations to determine a second pixel fingerprint based on a second set of images. The second pixel fingerprint can include a second set of pixel values ​​at a second pixel location within each image in the second time-lapse image set.

[0011] The processor can be configured to perform operations of generating a first binary pixel fingerprint based on the first pixel fingerprint. The first binary pixel fingerprint can include a first set of binary values ​​generated by comparing each of the first set of pixel values ​​to a threshold. The processor can be configured to perform operations of generating a second binary pixel fingerprint based on the second pixel fingerprint. The second binary pixel fingerprint can include a second set of binary values ​​generated by comparing each of the second set of pixel values ​​to a threshold.

[0012] The processor may be configured to perform an operation of determining whether a stereo correspondence exists between a first pixel fingerprint of a first set of images and a second pixel fingerprint of a second set of images based at least in part on a comparison of the first binary pixel fingerprint and the second binary pixel fingerprint.

[0013] In some embodiments, the first and / or second set of images of the scene are received from a reverse camera / projector.

[0014] In some embodiments, the system can include a projector configured to project a set of light patterns onto the scene, where each image in the first set of images can be acquired using a different light pattern from the set of light patterns.

[0015] In some embodiments, the set of light patterns can include fixed patterns that are translated, rotated and / or transformed to project different light patterns onto the scene.

[0016] In some embodiments, the first pixel fingerprint may include a first set of pixel values ​​as an ordered set corresponding to the temporal order of the first image set, and the second pixel fingerprint may include a second set of pixel values ​​as an ordered set corresponding to the temporal order of the second image set.

[0017] In some embodiments, generating the first set of binary values ​​may include, for each pixel value in the first set of pixel values, assigning a zero if the pixel value exceeds a threshold value and assigning a one if the pixel value does not exceed the threshold value.

[0018] In some embodiments, comparing the first binary pixel fingerprint and the second binary pixel fingerprint may include comparing corresponding binary values ​​in the first binary pixel fingerprint and the second binary pixel fingerprint.

[0019] In some embodiments, the processor can be configured to perform an operation of normalizing a first pixel fingerprint. Normalizing the first pixel fingerprint may include normalizing a first set of pixel values ​​at a first pixel location in each image of the first set of images to create a first normalized pixel fingerprint encompassing the first set of normalized pixel values. The processor can be configured to perform an operation of normalizing a second pixel fingerprint. Normalizing the second pixel fingerprint may include normalizing a second set of pixel values ​​at a second pixel location in each image of the second set of images to create a second normalized pixel fingerprint encompassing the second set of normalized pixel values. Generating the first binary pixel fingerprint may include generating the first binary pixel fingerprint based on the first normalized pixel fingerprint. Generating the second binary pixel fingerprint may include generating the second binary pixel fingerprint based on the second normalized pixel fingerprint.

[0020] In some embodiments, the processor can be configured to perform operations of generating a third binary pixel fingerprint based on the first normalized pixel fingerprint. The third binary pixel fingerprint can include a third set of binary values ​​generated by comparing the absolute value of each of the first normalized pixel value set to a confidence threshold. The processor can be configured to perform operations of generating a fourth binary pixel fingerprint based on the second normalized pixel fingerprint. The fourth binary pixel fingerprint can include a fourth set of binary values ​​generated by comparing the absolute value of each of the second normalized pixel value set to a confidence threshold.

[0021] In some embodiments, comparing the first binary pixel fingerprint with the second binary pixel fingerprint may include comparing an OR value of corresponding binary values ​​in the first, third, and fourth binary pixel fingerprints with an OR value of corresponding binary values ​​in the second, third, and fourth binary pixel fingerprints.

[0022] In some embodiments, comparing the first binary pixel fingerprint and the second binary pixel fingerprint may include determining whether corresponding binary values ​​in the first binary pixel fingerprint and the second binary pixel fingerprint are within a Hamming distance threshold. Comparing the first binary pixel fingerprint and the second binary pixel fingerprint may include determining whether corresponding binary values ​​in the first binary pixel fingerprint and the second binary pixel fingerprint are within a Hamming distance threshold.

[0023] In some embodiments, the processor may be configured to apply an image filter to each image in the first set of images before determining the first pixel fingerprint for the first set of images.

[0024] In some embodiments, determining whether a stereo correspondence exists between a first pixel fingerprint of the first set of images and a second pixel fingerprint of the second set of images may include performing continuous correlation over time to produce correlation values ​​between the first pixel fingerprint and the second pixel fingerprint, and determining that a potential correspondence exists between the first pixel fingerprint and the second pixel fingerprint based on correlation values ​​that exceed a correlation threshold and previous correlation values ​​for previous pairs of pixel fingerprints.

[0025] In some embodiments, determining whether a stereo correspondence exists between a first pixel fingerprint of a first image set and a second pixel fingerprint of a second image set may include performing a hole-filling operation to interpolate initial correspondences for pixel fingerprints that have not yet been determined to have potential correspondences with other pixel fingerprints.

[0026] In some embodiments, the first pixel location and / or the second pixel location may be selected based on a skip parameter.

[0027] In some embodiments, the first pixel location (i1, j1) can be selected based on a skip parameter S such that i1 modulo S=0 and j1 modulo S=0.

[0028] In some aspects, a system, method, and / or computer-readable storage medium may be provided for determining stereo correspondence between a first pixel fingerprint of a first set of images and a second pixel fingerprint of a second set of images.

[0029] The system can include a processor configured to perform operations of receiving a first set of images of a scene and a second set of images of the scene from one or more image sensors, where the second set of images can be acquired from a different perspective than the first set of images.

[0030] The processor may be configured to perform operations to determine a first pixel fingerprint based on a first set of images. The first pixel fingerprint may include a first set of pixel values ​​at different locations in space and / or time relative to a first pixel location in each image of the first set of images. The processor may be configured to perform operations to determine a second pixel fingerprint based on a second set of images. The second pixel fingerprint may include a first set of pixel values ​​at different locations in space and / or time relative to a second pixel location in each image of the second set of images.

[0031] The processor can be configured to perform operations of generating a first binary pixel fingerprint based on the first pixel fingerprint. The first binary pixel fingerprint can include a first set of binary values ​​generated by comparing each of the first set of pixel values ​​to a threshold. The processor can be configured to perform operations of generating a second binary pixel fingerprint based on the second pixel fingerprint. The second binary pixel fingerprint can include a first set of binary values ​​generated by comparing each of the second set of pixel values ​​to a threshold.

[0032] The processor may be configured to perform an operation of determining whether a stereo correspondence exists between a first pixel fingerprint of a first set of images and a second pixel fingerprint of a second set of images based at least in part on a comparison of the first binary pixel fingerprint and the second binary pixel fingerprint.

[0033] The foregoing has outlined, rather broadly, the features of the disclosed subject matter in order that the detailed description that follows may be better understood, and in order that the present contribution to the art may be better appreciated. There are, of course, additional features of the disclosed subject matter that will be described hereinafter and which form the subject of the claims appended hereto. The phraseology and terminology employed herein should be understood to be for the purpose of description and not of limitation.

[0034] In the drawings, each identical or nearly identical component shown in various figures is represented by the same reference numeral. For clarity, not every component is shown in every drawing. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating various aspects of the techniques and apparatus described herein. [Brief explanation of the drawings]

[0035] [Figure 1] 1 illustrates an exemplary embodiment in which one projector and two cameras are arranged to capture images of a scene in a manner suitable for generating stereo image correspondence, according to some embodiments.

[0036] [Figure 2] 1 shows an exemplary pair of stereo images corresponding to one of a series of projected light patterns, according to some embodiments.

[0037] [Figure 3] 1 shows an exemplary pair of time-lapse stereo image sequences of a scene according to some embodiments.

[0038] [Figure 4] 1 illustrates an exemplary pair of stereo images and corresponding normalized images according to some embodiments.

[0039] [Figure 5] 1 illustrates an exemplary pair of stereo images of a scene, according to some embodiments.

[0040] [Figure 6] 1 shows an exemplary pair of time-lapse stereo image sequences corresponding to a series of light patterns projected onto a scene, according to some embodiments.

[0041] [Figure 7] 1 illustrates an exemplary process for generating a binary pixel fingerprint of a pixel fingerprint from a pair of images, according to some embodiments.

[0042] [Figure 8] 1 illustrates an example of a binary pixel fingerprint generated for an exemplary pixel fingerprint from a pair of images according to some embodiments.

[0043] [Figure 9] 1 illustrates an exemplary computerized method for efficiently processing data for initial correspondence assignments for time-lapse stereo image sequences, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0044] The techniques described herein can be used to efficiently process data for initial correspondence assignments between pairs of two-dimensional images. The inventors recognized that while conventional systems are known to apply normalized cross-correlation to pairs of two-dimensional images to determine stereo correspondence, these systems must expend significant computational effort to perform the necessary calculations. As further described herein, the inventors have developed techniques for improving three-dimensional data reconstruction by efficiently processing data for initial correspondence assignments for pairs of two-dimensional images, particularly pairs of time-lapse stereo image sequences. The inventors note that the described systems and methods are novel in that they derive a binary pixel fingerprint for each pixel fingerprint in a time-lapse stereo image sequence.

[0045] In the following description, numerous specific details are set forth regarding the systems and methods of the disclosed subject matter, as well as the environments in which such systems and methods may operate, in order to provide a thorough understanding of the disclosed subject matter. In addition, it will be understood that the examples set forth below are illustrative only, and that other systems and methods are contemplated to exist that are within the scope of the disclosed subject matter.

[0046] FIG. 1 illustrates an exemplary embodiment 100 in which one projector 104 and two cameras 106 are arranged to capture images of an object or scene 102 in a manner suitable for generating stereo image correspondence. In some embodiments, a statistical pattern projector is used to temporally encode an image sequence of an object captured using multiple cameras. For example, the projector can project a translating pattern onto the object, and each camera can capture an image sequence including 12-16 images (or some other number of images) of the object. Each image contains a set of pixels that make up the image. In some embodiments, the light pattern can move horizontally and / or vertically such that the pattern rotates over the object or scene (e.g., without the pattern itself rotating clockwise or counterclockwise). Each camera 106 can include a charge-coupled device (CCD) image sensor, a complementary metal-oxide semiconductor (CMOS) image sensor, or other suitable image sensor. In some embodiments, each camera 106 can have a rolling shutter, a global shutter, or other suitable shutter type. In some embodiments, each of the cameras 106 may have a GigE Vision interface, a Universal Serial Bus (USB) interface, a coaxial interface, a FIREWIRE® interface, or other suitable interface. In some embodiments, each of the cameras 106 may have one or more smart functions. In some embodiments, each of the cameras 106 may have a C-mount lens, an F-mount lens, an S-mount lens, or other suitable lens type. In some embodiments, each of the cameras 106 may have a spectral filter matched to a projector, e.g., projector 104, to block ambient light outside the spectral range of the projector.

[0047] 2 shows an example pair of stereo images 200 and 250 corresponding to one of a series of projected light patterns. For example, projector 104 can project different light patterns onto an object, and camera 106 can acquire stereo images 200 and 250. In some embodiments, to reconstruct three-dimensional data from a sequence of stereo images from two cameras, it may be necessary to find pairs of corresponding pixels, such as pixels 202 and 252, between the images from each camera.

[0048] FIG. 3 shows an exemplary pair of stereo image sequences of a scene over time. For example, projector 104 may sequentially project different light patterns onto the scene, and one of cameras 106 may capture images 300, 302, and 304, while the other of cameras 106 may capture images 350, 352, and 354. In some embodiments, the captured images may be normalized to apply techniques of the described systems and methods. FIG. 4 shows an exemplary pair of stereo images 400 and 450 and their corresponding normalized images 450 and 452. As shown, stereo images 400 and 450 may be processed to include the light patterns of stereo images 400 and 450 while reducing the actual scene and / or objects in images 400 and 450. In some embodiments, knowledge of the type of light patterns projected may not be necessary to reduce the actual scene and / or objects in images 400 and 450.

[0049] FIG. 5 illustrates an exemplary pair of stereo images 500 and 550 (and associated pixels), where corresponding pixels 502 and 552 represent the same portion of a pattern projected in the two images 500 and 550. For example, as described above, projector 104 can project a light pattern onto a scene, and camera 106 can capture stereo images 500 and 550. The captured stereo images 500 and 550 can be used to identify correspondence between the two pixels. In some embodiments, a sequence of stereo images captured over time is used to identify correspondence. Continuing from the single stereo image pair shown in FIG. 5 , as shown in FIG. 6 , when projector 104 successively projects different light patterns onto the scene over time, camera 106 can capture a time-lapse stereo image sequence 600 and 650 and corresponding pixel fingerprints 602 and 652. Each camera 106 can capture a series of images 1, 2, 3, 4, ... N over time. Pixel fingerprints 602 and 652 are based on pixels (i,j) and (i',j') across the time-lapse stereo image sequences 600 and 650, respectively. Each pixel fingerprint contains an ordered list of gray values ​​over time, G_i_j_t, where t denotes discrete time-lapse instances 1, 2, 3, 4, ... N.

[0050] In some embodiments, a normalized cross-correlation algorithm using only the time-lapse images or a subset of the time-lapse images can be applied to the two image sequences to determine pairs of corresponding pixels from each image (e.g., pairs having similar time-lapse gray values). However, such a process may require significant computational effort to perform the necessary calculations. This process can be improved by efficiently processing the data for the initial correspondence assignments, and in particular by deriving a binary pixel fingerprint for each pixel fingerprint in the time-lapse stereo image sequence, as described further below.

[0051] In some embodiments, for each pixel of the first camera, potential corresponding pixels are retrieved by performing a normalized cross-correlation with all feasible candidates along the epipolar line of the second camera, with a threshold value to compensate for deviations due to camera calibration, e.g., ±1 pixel or other suitable value. In one example, this approximates computing normalized cross-correlations for 3000 potential pairs, a computational effort of dimension x res ×y res For the number of images N, approximately x res ×y res × N × 3000 multiplications (for example, when N = 24, it is approximately 94 × 10 9 ).

[0052] In some aspects, the described systems and methods relate to technical improvements over existing techniques, utilizing algorithmic approaches to reduce the computational effort of initial correspondence assignments by, for example, a factor of 200 or other suitable factor (e.g., compared to the approaches described above). The computational effort can be reduced by performing computationally intensive cross-correlations on only a subset of the potential correspondences and / or by reducing the amount of data processed to calculate the correspondences.

[0053] 9 illustrates an exemplary computerized method 900 for efficiently processing data for initial correspondence assignments of time-lapse stereo image sequences, according to some embodiments. Method 900 may be performed on any suitable computing system (e.g., a general-purpose computing device (CPU), a graphics processing unit (GPU), a field-programmable gate array device (FPGA), an application-specific integrated circuit device (ASIC), an ARM-based device, or other suitable computing system), and aspects of the technology described herein are not limited in this respect. In some embodiments, one or more cameras may generate a binary pixel fingerprint for each pixel of the captured image. In this case, the bandwidth required to send data from the camera to the system may be reduced.

[0054] At operation 902, the system may receive a first set of images of a scene and a second set of images of the scene from one or more image sensors. The second set of images is acquired from a different perspective than the first set of images. For example, the system may receive a time-lapse sequence of 12 to 16 images of the scene, or any other suitable number of images, from two cameras as described with respect to FIG. 1. In some embodiments, the projector is configured to project a set of light patterns onto the scene. Each image in the first set of images and / or the second set of images may be acquired using a different light pattern from the set of light patterns. For example, as shown in FIG. 2, a light pattern may be projected onto the scene and two cameras may acquire respective images. In another example, as shown in FIG. 3, a set of light patterns may be projected onto the scene sequentially and two cameras may acquire respective time-lapse stereo image sequences. In some embodiments, the set of light patterns may include a fixed pattern that is translated, rotated, and / or transformed to change the light pattern projected onto the scene. In some embodiments, the set of light patterns may include a moving optical pattern. In some embodiments, the moving optical pattern may include an optical pattern that translates along a circular path. In some embodiments, the light pattern can be moved horizontally and / or vertically such that the pattern rotates over the object or scene (eg, without the pattern itself rotating clockwise or counterclockwise).

[0055] At operation 904, the system may determine a first pixel fingerprint based on the first image set. In some embodiments, the first pixel fingerprint includes a first set of pixel values ​​at a first pixel location in each image of the first time-lapse image set. In some embodiments, the first pixel fingerprint may include a first set of pixel values ​​at different locations in space and / or time relative to the first pixel location in each image of the first image set. FIG. 8 shows an example of such a pixel fingerprint. In some embodiments, the first pixel fingerprint includes the first set of pixel values ​​as an ordered set corresponding to the temporal order of the first image set. FIGS. 5-6 show examples of such temporal pixel locations and corresponding pixel values ​​in the ordered set of images. In some embodiments, the system applies a weak Gaussian filter to each image of the first image set before determining the first pixel fingerprint for the first image set. In some embodiments, the system selects the first pixel location based on a skip parameter. For example, the system can select the first pixel location (i1, j1) based on the skip parameter S such that i1 modulo S=0 and j1 modulo S=0.

[0056] At operation 906, the system may determine a second pixel fingerprint based on the second image set. In some embodiments, the second pixel fingerprint includes a second set of pixel values ​​at a second pixel location in each image of the second time-lapse image set. In some embodiments, the second pixel fingerprint may include a second set of pixel values ​​at different locations in space and / or time relative to the second pixel location in each image of the second image set. FIG. 8 shows an example of such a pixel fingerprint. In some embodiments, the second pixel fingerprint includes the second set of pixel values ​​as an ordered set corresponding to the temporal order of the second image set. FIGS. 5-6 show examples of such temporal pixel locations and corresponding pixel values ​​in the ordered set of images. In some embodiments, the system applies a weak Gaussian filter to each image of the second image set before determining the second pixel fingerprint for the second image set. In some embodiments, the system selects the second pixel location based on a skip parameter. For example, the system can select the second pixel location (i2, j2) based on the skip parameter S such that i2 modulo S=0 and j2 modulo S=0.

[0057] At operation 908, the system may generate a first binary pixel fingerprint based on the first pixel fingerprint. The first binary pixel fingerprint includes a first set of binary values ​​generated by comparing each of the first set of pixel values ​​to a threshold. In some embodiments, generating the first set of binary values ​​includes assigning a zero to each pixel value in the first set of pixel values ​​if the pixel value exceeds the threshold, and assigning a one if the pixel value does not exceed the threshold. Figure 8 shows an example of such a pixel fingerprint.

[0058] In some embodiments, the system may normalize the first pixel fingerprint by normalizing a first set of pixel values ​​at a first pixel location in each image of the first set of images to create a first normalized pixel fingerprint that encompasses the first set of normalized pixel values. FIG. 4 shows exemplary stereo images and corresponding normalized images. In such embodiments, the system may generate a first binary pixel fingerprint based on the first normalized pixel fingerprint in the manner described with respect to the first pixel fingerprint. FIG. 7 and the associated description illustrate an exemplary process corresponding to this technique, and FIG. 8 and the associated description illustrate an example of such a binary pixel fingerprint generated by this technique.

[0059] At operation 910, the system may generate a second binary pixel fingerprint based on the second pixel fingerprint. The second binary pixel fingerprint includes a second set of binary values ​​generated by comparing each of the second set of pixel values ​​to a threshold. In some embodiments, generating the set of second binary values ​​includes, for each pixel value in the second set of pixel values, assigning a zero if the pixel value exceeds the threshold and assigning a one if the pixel value does not exceed the threshold. Figure 8 shows an example of such a pixel fingerprint.

[0060] In some embodiments, the system may normalize the second pixel fingerprint by normalizing a second set of pixel values ​​at a second pixel location in each image of the second set of images to create a second normalized pixel fingerprint that encompasses the second set of normalized pixel values. FIG. 4 shows exemplary stereo images and corresponding normalized images. In such embodiments, the system may generate a second binary pixel fingerprint based on the second normalized pixel fingerprint in the manner described with respect to the second pixel fingerprint. FIG. 7 and the associated description illustrate an exemplary process corresponding to this technique, and FIG. 8 and the associated description illustrate an example of such a binary pixel fingerprint generated according to this technique.

[0061] At operation 912, the system may determine whether a stereo correspondence exists between a first pixel fingerprint of the first set of images and a second pixel fingerprint of the second set of images based at least in part on comparing the first binary pixel fingerprint and the second binary pixel fingerprint. In some embodiments, comparing the first binary pixel fingerprint and the second binary pixel fingerprint includes comparing corresponding binary values ​​in the first binary pixel fingerprint and the second binary pixel fingerprint. In some embodiments, comparing the first binary pixel fingerprint and the second binary pixel fingerprint includes determining whether corresponding binary values ​​in the first binary pixel fingerprint and the second binary pixel fingerprint are within a Hamming distance threshold.

[0062] In some embodiments, the system may generate a third binary pixel fingerprint based on the first normalized pixel fingerprint by comparing the absolute value of each of the first set of normalized pixel values ​​to a confidence threshold. In some embodiments, the system may generate a fourth binary pixel fingerprint based on the second normalized pixel fingerprint by comparing the absolute value of each of the second set of normalized pixel values ​​to a confidence threshold. In some embodiments, comparing the first binary pixel fingerprint and the second binary pixel fingerprint includes comparing (i) an OR value of corresponding binary values ​​in the first, third, and fourth binary pixel fingerprints and (ii) an OR value of corresponding binary values ​​in the second, third, and fourth binary pixel fingerprints.

[0063] In some embodiments, the system may determine whether a stereo correspondence exists between a first pixel fingerprint of a first set of images and a second pixel fingerprint of a second set of images by performing successive correlations over time to produce correlation values ​​between the first pixel fingerprint and the second pixel fingerprint, and determining that a potential correspondence exists between the first pixel fingerprint and the second pixel fingerprint based on correlation values ​​that exceed a correlation threshold and previous correlation values ​​for previous pairs of pixel fingerprints. In some embodiments, the system may perform a hole-filling operation to interpolate initial correspondences for pixel fingerprints that have not yet been determined to have potential correspondences with other pixel fingerprints.

[0064] In some embodiments, the described systems and methods provide processes suitable for implementation in a computing system. In some embodiments, a computing system (not shown in FIG. 1 ) receives a pair of time-lapse stereo image sequences from an image sensor or camera (e.g., as described with respect to operation 902). As described further herein, for example, the computing system may be a general-purpose computing device (CPU), a graphics processing unit (GPU), a field-programmable gate array device (FPGA), an application-specific integrated circuit device (ASIC), an ARM-based device, or other suitable computing system. In some embodiments, one or more cameras may generate a binary pixel fingerprint for each pixel in the captured images. This can reduce the bandwidth required to send data from the camera to the system.

[0065] The system may determine a pixel fingerprint for each image sequence (e.g., as described with respect to operations 904 and / or 906). In some embodiments, the system may smooth the image sequence, such as by using a weak Gaussian filter for each image. In some embodiments, the system may optionally normalize the image sequence for each pixel fingerprint. Based on the normalized image sequence or normalized pixel fingerprint, a binary sequence or binary pixel fingerprint is created for each pixel of each camera, e.g., BINARY1_i_j (e.g., as described with respect to operation 908) for the first camera, Camera 1, and BINARY1_i'_j' (e.g., as described with respect to operation 910) for the second camera, Camera 2. For a given pixel, a representative binary value for the pixel is determined. For example, the system may assign a 0 or 1 to the pixel value based on the normalized pixel value over time. For example, the system may assign a 0 or 1 by determining whether the normalized value is ≧0 or <0 (or other suitable threshold). For example, the system may assign a 1 to values ​​greater than or equal to 0 (eg, indicating lighter than average) and a 0 to values ​​less than 0 (eg, indicating darker than average).

[0066] The system can begin searching for potential correspondences using (normalized or non-normalized) binary pixel fingerprints (e.g., as described with respect to operation 912). For a given camera 1 pixel, the search space in camera 2 can first be restricted to an epipolar line with a defined threshold, e.g., SAFETY_THRESHOLD. For potential correspondences, the system can, for example, check whether pairs of binary sequences match (or are equivalent) to the potential correspondence, e.g., whether BINARY1_i_j==BINARY1_i'_j'.

[0067] Alternatively or additionally, for potential correspondences, the system can check whether pairs of binary sequences are within a particular distance threshold. For example, the system can check whether pairs of binary sequences are within a Hamming distance threshold HD_THRESHOLD. For example, the system can determine whether HAMMING_DISTANCE(BINARY1_i_j, BINARY1_i’_j’) < HD_THRESHOLD. In some embodiments, the Hamming distance between two binary sequences of equal length is the number of positions where the corresponding values are different. For example, the Hamming distance between 1011101 and 1001001 is 2.

[0068] Alternatively or additionally, based on the normalized image sequences, the system can generate another pair of binary sequences BINARY2_i_j and BINARY2_i’_j’ for each pixel of each camera. This pair of binary sequences can represent a confidence flag for the initial pair of binary sequences. For a given pixel, a representative binary value for that pixel is determined. For example, based on the normalized pixel values over time, the system can assign 0 or 1 depending on whether the absolute normalized value is below a confidence threshold CONFIDENCE_THRESHOLD. For potential correspondences, the system can check whether BINARY1_i_j|BINARY2_i_j|BINARY2_i’_j’ == BINARY1_i’_j’|BINARY2_i’_j’|BINARY2_i_j.

[0069] If a pair of binary sequences is similar (e.g., matching) according to one or more of the criteria described above, the system can perform successive correlations over time. For example, if the correlation produces a value that exceeds an initial correlation threshold INITIAL_CORRELATION_THRESHOLD and the correlation is greater than any previous potential correspondence for (i,j), then pixel pair (i,j) and (i',j') can be marked as a potential correspondence. In some embodiments, for a particular pixel (i,j) in camera 1, the system can use the respective binary sequence or fingerprint as a criterion to identify similar or matching pixels (i',j') in camera 2 (e.g., along an epipolar line, across the entire image of camera 2, or at other suitable locations). If the criteria are met, the system can store the resulting pairs (i,j) and (i',j') in a list, and later perform correlations over time. For example, the system can correlate each pair in the list and select the pair with the highest correlation score as a potential correspondence. Alternatively or additionally, the system may perform a temporal correlation such that the correlation of the current pair (i,j) and (i',j') exceeds an initial threshold and has a higher correlation than the previous correlation pair including (i,j), or vice versa. If the correlation of the current pair exceeds the initial threshold and has a higher correlation than the previous correlation pair, the current pair may replace the previous potential correspondence. In some embodiments, the temporal correlation may be performed by using normalized cross-correlation based on the temporal gray values ​​or normalized values ​​of the pair of pixels to be correlated.

[0070] In some embodiments, after the initial search is complete, a hole-filling process can be performed to interpolate initial correspondences for pixels not assigned by the process described herein. For example, because the search space is highly limited, approximately 85%-95% of all correspondences may be found initially. A hole-filling process (at the correspondence level) can be used to produce 100% correspondences (e.g., leveraging the initially identified correspondences). In some embodiments, the hole-filling process can identify specific points and / or pixels (e.g., oversaturated image points) that do not have correspondences and generate suitable correspondences by interpolating related correspondences near the identified points. In some embodiments, the system can apply a filter (e.g., a weak smoothing filter) to the image sequence before initial correspondence assignment, thereby achieving 100% correspondences (e.g., not requiring a hole-filling process). The filter can reduce noise and, to some extent, aggregate small spatial neighborhoods for each time-lapse image. This can reduce the number of pixel-accurate match intensity deviations due to perspective and sampling effects of different cameras or image sensors.

[0071] In some embodiments, the described systems and methods provide a process suitable for implementation on a GPU. The GPU receives a pair of time-lapse stereo image sequences from an image sensor or camera (e.g., as described with respect to operation 902). The GPU can use a comparison threshold during the correspondence search. For example, the GPU can use a Hamming distance threshold HD_THRESHOLD, which is either 0 (e.g., no difference is allowed) or greater than 0 (e.g., some difference is allowed). The GPU can store, determine, and / or receive the HD_THRESHOLD parameter. In some embodiments, the GPU can smooth the image sequence (e.g., using a weak Gaussian filter for each image). Additionally or alternatively, the GPU can rectify the image sequence. For example, the GPU can rectify the image sequence to the standard case, such that epipolar lines become horizon lines due to the rectification.

[0072] In some embodiments, the GPU can optionally normalize the image sequence for each pixel fingerprint. Based on the normalized image sequence or normalized pixel fingerprint, the GPU can create a binary sequence or binary pixel fingerprint BINARY1 for a subset of pixels for the first camera, camera 1 (e.g., as described with respect to operation 908), and another binary pixel fingerprint BINARY1 is created for the second camera, camera 2 (e.g., as described with respect to operation 910). The GPU can select the subset of pixels for camera 1 using various techniques. For example, in some embodiments, the GPU can select pixels using modulo arithmetic. For example, the GPU can use skip-grid skip such that the selected pixel (i, j) satisfies (i modulo SKIPPING=0, j modulo SKIPPING=0). For a given pixel, a representative binary value for the pixel is determined. For example, the GPU can assign a 0 or 1 to the pixel based on the normalized pixel value over time. For example, the GPU can assign a 0 or 1 by determining whether the normalized value is ≧0 or <0 (or other suitable threshold). For example, the system may assign a 1 to values ​​greater than or equal to 0 (eg, indicating lighter than average) and a 0 to values ​​less than 0 (eg, indicating darker than average).

[0073] The GPU can then generate a binary sequence for each pixel for each image sequence from each camera. For example, in the case where camera 1 and camera 2 each acquire an image sequence of 16 images, the GPU can generate a 16-bit binary sequence for each pixel for camera 2 and a 16-bit binary sequence for each pixel on the skip grid for camera 1. The system can use the (normalized or non-normalized) binary sequences or binary pixel fingerprints to initialize a search for potential correspondences (e.g., as described with respect to operation 912). For example, for each pixel on the skip grid for camera 1, the GPU can compare each pixel on camera 2 that is near the corresponding epipolar line for camera 2 with respect to the binary sequence or binary pixel fingerprint. For potential correspondences, the GPU can determine whether a pair of binary sequences matches a potential correspondence by checking whether the pair of binary sequences is within a certain distance threshold. For example, the system can check whether the pair of binary sequences is within a Hamming distance threshold. For example, the system can determine whether HAMMING_DISTANCE(BINARY1_i_j, BINARY1_i'_j')≦HD_THRESHOLD. If the pair of binary sequences meets the condition, the GPU can save the pair in a list of potential correspondences. After performing the potential correspondence step for each pixel on the skip grid in camera 1, the GPU can generate a list of potential correspondences. For each pixel on the skip grid in camera 1, the GPU can perform continuous correlation over time with the list of potential candidates. For example, if the correlation produces a value that exceeds an initial correlation threshold INITIAL_CORRELATION_THRESHOLD and / or if the correlation is greater than the previous potential correspondence in the list, the pair of binary sequences can be marked as the current initial correspondence.

[0074] FIG. 7 illustrates an exemplary process 700 for generating a binary pixel fingerprint for a pixel fingerprint from a pair of time-lapse images, according to some embodiments. In this exemplary process, a first set of images and a second set of images of a scene are received from one or more cameras or image sensors as described herein. The second set of images may be acquired from a different viewpoint than the first set of images to generate pairwise stereo image correspondence. For example, a temporal sequence of several images (e.g., 12-16 images) of a scene, or any other suitable number of images, may be received from the two cameras described with respect to FIG. 1. A first pixel fingerprint 702 for pixel (i,j) is determined from the first set of images, and a first pixel fingerprint 702 for pixel (i',j') is determined from the second set of images, as described above. The first pixel fingerprint 702 is calculated by determining values ​​v1 through v2. N , which represent the actual grayscale and / or color values ​​of each pixel in each image of the first image set. The second pixel fingerprint 752 includes values ​​v1′ to v N ', which represent the actual grayscale and / or color values ​​of each pixel in each image of the second image set. TIFF2025138855000002.tif18160 The binary pixel fingerprint has values ​​b1 to b2 corresponding to the first pixel fingerprint and the second pixel fingerprint. N and a first binary pixel fingerprint 706 including values ​​b1′ to b N ' in the form of a second binary pixel fingerprint 756, which is generated in the manner described above and with respect to Figure 9. Figure 8 provides an illustrative example of process 700.

[0075] 8 illustrates an example binary pixel fingerprint 800 generated for a pixel fingerprint from a pair of images according to some embodiments. In this example, a first image set of 19 images is received from a first camera, Camera 1, and a second image set of 19 images is received from a second camera, Camera 2 (e.g., where the image sets from Camera 1 and Camera 2 represent a stereo pair). For a given pixel location (i,j) in the first image set and a corresponding pixel location (i',j') in the second image set, a first pixel fingerprint 802 and a second pixel fingerprint 852 are generated, respectively. Each pixel fingerprint includes the gray pixel value for the given pixel location across the image sets. For example, first pixel fingerprint 802 includes gray values ​​for pixel locations (i,j) across the first image (e.g., pixel 802 includes grayscale values ​​63, 158, etc.), and second pixel fingerprint 852 includes gray values ​​for pixel locations (i',j') across the second image (e.g., pixel 852 includes grayscale values ​​81, 186, etc.). Table 1 below shows the gray values ​​for first pixel fingerprint 802 and second pixel fingerprint 852. Optionally, first pixel fingerprint 802 and second pixel fingerprint 852 are normalized, respectively, to generate first normalized pixel fingerprint 804 and second normalized pixel fingerprint 854. TIFF2025138855000003.tif13159Here, the t-th component v of vector v is t and the average value of the components of vector v avg is subtracted, and the resulting value is v from each component of the vector v. avg The length of the vector vv obtained by subtracting avg, where x is the number of pixels in the image, and y is the number of pixels in the image. Table 1 below shows normalization values ​​for first normalized pixel fingerprint 804 and second normalized pixel fingerprint 854. Finally, first binary pixel fingerprint 806 and second binary pixel fingerprint 856 are generated from first normalized pixel fingerprint 804 and second normalized pixel fingerprint 854, respectively, by comparing each normalization value to a threshold, e.g., zero, and assigning, e.g., zero, if the normalized value exceeds the threshold, and another binary value, e.g., one, if the normalized value is below the threshold. Table 1 below shows the binary values ​​for first binary pixel fingerprint 806 and second binary pixel fingerprint 856.

[0076] TIFF2025138855000004.tif176167

[0077] Alternatively, first binary pixel fingerprint 806 and second binary pixel fingerprint 856 can be generated from first pixel fingerprint 802 and second pixel fingerprint 852, respectively (without performing normalization) by comparing each value to a threshold and, for example, assigning a binary value, such as zero, if the value exceeds the threshold and another binary value, such as one, if the value is below the threshold. Table 1 shows values ​​for corresponding pixel pairs for the two cameras that, given a pixel from a camera and any pixel from another camera, result in the same binary sequence or fingerprint value, although the respective normalized values ​​and resulting binary pixel fingerprint values ​​may differ if the selected pixels do not correspond.

[0078] Techniques operating according to the principles described herein may be implemented in any suitable manner. The process and decision blocks in the flowcharts above represent steps and acts that may be included in algorithms that perform these various processes. Algorithms derived from these processes may be implemented as software integrated with and directing the operations of one or more special-purpose or general-purpose processors, as functionally equivalent circuitry such as digital signal processing (DSP) circuitry or application-specific integrated circuit devices (ASICs), or in any other suitable manner. It should be understood that the flowcharts included herein do not depict the syntax or operations of any particular circuitry or any particular programming language or type of programming language. Rather, the flowcharts illustrate functional information that may be used to fabricate circuitry or implement computer software algorithms for processing particular devices that perform techniques of the types described herein. Unless otherwise indicated herein, it should also be understood that the specific sequence of steps and / or acts set forth in each flowchart is merely illustrative of algorithms that may be implemented, and that implementations and embodiments of the principles described herein may vary.

[0079] Thus, in some embodiments, the techniques described herein may be embodied in computer-executable instructions implemented as software, including application software, system software, firmware, middleware, embedded code, or any other suitable type of computer code. Such computer-executable instructions may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may also be compiled as executable machine language code or intermediate code that runs on a framework or virtual machine.

[0080] When the techniques described herein are embodied as computer-executable instructions, these computer-executable instructions can be implemented in any suitable manner, including as a number of utility functions, each providing one or more operations to complete the execution of an algorithm operating according to these techniques. However, an instantiated "utility function" is a structural element of a computer system that, when integrated with and executed by one or more computers, causes the one or more computers to perform a particular operational role. A utility function can be part or all of a software element. For example, a utility function may be implemented as a function of a process, as a separate process, or as other suitable processing unit. When the techniques described herein are implemented as multiple utility functions, each utility function may be implemented in a unique manner and need not all be implemented in the same manner. Furthermore, these utility functions may be executed in parallel and / or serially as desired, and may pass information between each other using shared memory of the computers on which they are executing, a message-passing protocol, or any other suitable manner.

[0081] Generally, utility functions include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically, the functionality of utility functions can be combined and distributed as desired in the systems in which they operate. In some implementations, one or more utility functions that perform the techniques herein can together form a complete software package. These utility functions may, in alternative embodiments, be adapted to interact with other unrelated utility functions and / or processes to implement a software program application.

[0082] Several exemplary utility functions have been described herein to perform one or more tasks. However, it should be understood that the described utility functions and divisions of tasks are merely illustrative of the types of utility functions that may implement the exemplary techniques described herein, and that embodiments are not limited to any particular number, division, or type of utility functions. In some implementations, all functionality may be implemented in a single utility function. Also, in some implementations, some of the utility functions described herein may be implemented together with other utility functions or separately (i.e., as a single unit or separate units), or some of these utility functions may not be implemented.

[0083] Computer-executable instructions implementing the techniques described herein (whether implemented as one or more convenience features or otherwise) are, in some embodiments, encoded on one or more computer-readable media to provide the media with functionality. Computer-readable media include magnetic media such as hard disk drives, optical media such as compact discs (CDs) or digital versatile discs (DVDs), persistent or non-persistent solid-state memory (e.g., flash memory, magnetic RAM), or other suitable storage media. Such computer-readable media may be implemented in any suitable manner. As used herein, a "computer-readable medium" (also referred to as a "computer-readable storage medium") refers to a tangible storage medium. A tangible storage medium is non-transitory and has at least one physical structural element. As used herein, a "computer-readable medium" refers to at least one physical structural element that has at least one physical characteristic that can be changed in some way during the process of creating a medium with embedded information, recording information on the medium, or encoding the medium with information. For example, the magnetization state of a portion of the physical structure of the computer-readable medium can be changed during the recording process.

[0084] Additionally, some of the technologies described above involve storing information (e.g., data and / or instructions) in a particular manner for use in those technologies. In some implementations of these technologies (e.g., implementations in which the technologies are embodied as computer-executable instructions), the information is encoded on a computer-readable storage medium. Where particular structures are described herein as advantageous formats for storing this information, those structures can be used to provide a physical organization of the information when encoded on the storage medium. These advantageous structures can then impart functionality to the storage medium by affecting the operation of one or more processors that interact with the information, for example, by increasing the efficiency of computer operations performed by the processors.

[0085] In some implementations (but not all implementations) in which the techniques may be embodied as computer-executable instructions, these instructions may be executed by one or more suitable computing devices operating in any suitable computer system, or one or more computing devices (or one or more processors of one or more computing devices) may be programmed to execute the computer-executable instructions. A computing device or processor may be programmed to execute the instructions when the instructions are stored in a manner accessible to the computing device or processor, such as a data store (e.g., on-chip cache or instruction registers, computer-readable storage accessible via a bus, computer-readable storage media accessible via one or more networks, and media accessible by the device / processor, etc.). Utility functions containing these computer-executable instructions may be integrated with and direct the operation of a single general-purpose programmable digital computing device, a cooperative system of two or more general-purpose computing devices that share processing power and jointly perform the techniques described herein, a single computing device or a cooperative system of computing devices (co-located or geographically distributed) dedicated to performing the techniques described herein, one or more field-programmable gate arrays (FPGAs) for performing the techniques described herein, or any other suitable system.

[0086] A computing device may include at least one processor, a network adapter, and a computer-readable storage medium. The computing device may be, for example, a desktop or laptop personal computer, a personal digital assistant (PDA), a smartphone, a mobile phone, a server, or any other suitable computing device. The network adapter may be any suitable hardware and / or software that enables the computing device to communicate wired and / or wirelessly with any other suitable computing device over any suitable computing network. A computing network may include wireless access points, switches, routers, gateways, and / or other networking equipment, and any suitable wired and / or wireless communication medium for exchanging data between two or more computers, including the Internet. The computer-readable medium may be adapted to store data to be processed and / or instructions to be executed by the processor. The processor enables the processing of data and execution of instructions. The data and instructions may be stored in the computer-readable storage medium.

[0087] A computing device may further include one or more components and peripherals, including input / output devices. These devices can be used, among other things, to provide a user interface. Examples of output devices that can be used to provide a user interface include a printer or display screen for visually displaying output, and a speaker or other sound-generating device for audibly displaying output. Examples of input devices that can be used in a user interface are keyboards, pointing devices such as mice or touchpads, and digitizing tablets. As another example, a computing device can receive input information via voice recognition or other audible formats.

[0088] The embodiments described above illustrate techniques implemented with circuits and / or computer-executable instructions. It should be understood that some embodiments may be in the form of a method, of which at least one example is provided. Operations performed as part of a method may be ordered in any suitable manner. Thus, while illustrated embodiments show operations as sequential, embodiments may be configured to perform operations in an order different from that illustrated, including performing some operations simultaneously.

[0089] Various aspects of the above-described embodiments may be used alone, in combination, or in various configurations not specifically discussed in the above-described embodiments, and therefore are not limited in their application to the details and arrangements of components set forth in the foregoing description or illustrated in the drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0090] The use of ordinal numbers such as "first," "second," "third," etc. to modify claim elements in the claims does not, in itself, imply any priority, precedence, or ranking of one claim element over another, or any chronological order in which method actions are performed, but is merely used as a descriptive term to distinguish a claim element having a particular name from another element having the same name (except for the use of ordinal numbers) to distinguish the claim elements.

[0091] Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "having," "having," "including," "involving," and variations thereof herein is meant to encompass the items listed thereafter, and equivalents thereof, as well as additional items.

[0092] The word "exemplary" is used herein to mean serving as an example, instance, or illustration. Thus, any embodiments, implementations, processes, features, etc. described herein as exemplary are to be understood as illustrative examples and not as preferred or advantageous examples, unless expressly stated otherwise.

[0093] While several aspects of at least one embodiment have been described above, it should be understood that various alterations, modifications, and improvements will readily occur to those skilled in the art. Such alterations, modifications, and improvements are intended to be part of this disclosure, and are intended to be within the spirit and scope of the principles described herein. Accordingly, the foregoing description and drawings are by way of example only.

[0094] The claims are set forth below.

Claims

1. 1. A system for determining stereo correspondence between a first pixel fingerprint of a first set of images and a second pixel fingerprint of a second set of images, the system comprising: the processor is configured to perform the steps of receiving a first set of images of a scene and a second set of images of the scene from one or more image sensors, where the second set of images are acquired from a different viewpoint than the first set of images; The processor is further configured to perform the step of determining a first pixel fingerprint based on a first set of images, the first pixel fingerprint including a first set of pixel values ​​at a first pixel location in each image of the first time-lapse image set; The processor is further configured to perform the step of determining a second pixel fingerprint based on a second set of images, wherein the second pixel fingerprint includes a second set of pixel values ​​at second pixel locations in each image of the second time-lapse image set; The processor is further configured to perform the step of generating a first binary pixel fingerprint based on the first pixel fingerprint, where the first binary pixel fingerprint comprises a first set of binary values ​​generated by comparing each of the first set of pixel values ​​to a threshold; The processor is further configured to perform the step of generating a second binary pixel fingerprint based on the second pixel fingerprint, where the second binary pixel fingerprint comprises a second set of binary values ​​generated by comparing each of the second set of pixel values ​​to a threshold; and the processor is configured to perform the step of determining whether a stereo correspondence exists between a first pixel fingerprint of a first set of images and a second pixel fingerprint of a second set of images based at least in part on a comparison of the first binary pixel fingerprint and the second binary pixel fingerprint; The above system.

2. The system of claim 1 , wherein the first set of images and / or the second set of images of a scene are received from a reverse camera / projector.

3. 10. The system of claim 1, further comprising a projector configured to project a set of light patterns onto the scene, wherein each image in the first set of images is acquired using a different light pattern than the set of light patterns.

4. The system of claim 3 , wherein the set of light patterns comprises a fixed pattern that is translated, rotated and / or transformed to project different light patterns onto a scene.

5. the first pixel fingerprint includes the first set of pixel values ​​as an ordered set corresponding to a temporal order of the first image set; The system of claim 1 , wherein the second pixel fingerprint includes the second set of pixel values ​​as an ordered set corresponding to a temporal order of the second set of images.

6. 2. The system of claim 1, wherein generating the first set of binary values ​​comprises assigning to each pixel value in the first set of pixel values ​​a zero if the pixel value exceeds a threshold value and a one if the pixel value does not exceed a threshold value.

7. 2. The system of claim 1, wherein comparing the first binary pixel fingerprint with the second binary pixel fingerprint comprises comparing corresponding binary values ​​in the first binary pixel fingerprint and the second binary pixel fingerprint.

8. the processor is configured to perform the step of normalizing the first pixel fingerprint, where normalizing the first pixel fingerprint comprises normalizing a first set of pixel values ​​at a first pixel location in each image of the first set of images to create a first normalized pixel fingerprint that encompasses the first set of normalized pixel values; and the processor is configured to perform the step of normalizing the second pixel fingerprint, where normalizing the second pixel fingerprint comprises normalizing a second set of pixel values ​​at second pixel locations in each image of the second set of images to create a second normalized pixel fingerprint comprising the second set of normalized pixel values; where: generating the first binary pixel fingerprint includes generating the first binary pixel fingerprint based on the first normalized pixel fingerprint; generating the second binary pixel fingerprint includes generating the second binary pixel fingerprint based on the second normalized pixel fingerprint; The system of claim 1 .

9. the processor is configured to perform the step of generating a third binary pixel fingerprint based on the first normalized pixel fingerprint, where the third binary pixel fingerprint includes a third set of binary values ​​generated by comparing an absolute value of each of the first set of normalized pixel values ​​to a confidence threshold; and the processor is configured to perform the step of generating a fourth binary pixel fingerprint based on the second normalized pixel fingerprint, wherein the fourth binary pixel fingerprint includes a fourth set of binary values ​​generated by comparing an absolute value of each of the second set of normalized pixel values ​​to a confidence threshold; The system of claim 8.

10. Comparing the first binary pixel fingerprint with the second binary pixel fingerprint comprises: comparing an OR value of corresponding binary values ​​in the first, third and fourth binary pixel fingerprints with an OR value of corresponding binary values ​​in the second, third and fourth binary pixel fingerprints; The system of claim 9.

11. 2. The system of claim 1, wherein comparing the first binary pixel fingerprint and the second binary pixel fingerprint comprises determining whether corresponding binary values ​​in the first binary pixel fingerprint and the second binary pixel fingerprint are within a Hamming distance threshold.

12. 2. The system of claim 1, wherein the processor is configured to apply an image filter to each image in the first image set prior to determining the first pixel fingerprint in the first image set.

13. Determining whether a stereo correspondence exists between a first pixel fingerprint of a first set of images and a second pixel fingerprint of a second set of images includes: performing successive time-course correlations to generate correlation values ​​between the first pixel fingerprint and the second pixel fingerprint; determining that a potential correspondence exists between the first pixel fingerprint and the second pixel fingerprint based on the correlation value exceeding a correlation threshold and previous correlation values ​​for previous pairs of pixel fingerprints; The system of claim 1 , comprising:

14. Determining whether a stereo correspondence exists between the first pixel fingerprint of the first set of images and the second pixel fingerprint of the second set of images includes:

14. The system of claim 13, further comprising performing a hole-filling operation to interpolate initial correspondences for pixel fingerprints that have not yet been determined to have potential correspondences with other pixel fingerprints.

15. The system of claim 1 , wherein the first pixel location and / or the second pixel location are selected based on a skip parameter.

16. The first pixel position (i １ , j １ ) is calculated based on the skip parameter S. １ Modulo S=0 and j １ 16. The system of claim 15, wherein the modulo S is selected to be 0.

17. 1. A method for determining stereo correspondence between a first pixel fingerprint of a first set of images and a second pixel fingerprint of a second set of images, the method comprising: the processor is configured to receive a first set of images of a scene and a second set of images of the scene from one or more image sensors, where the second set of images is acquired from a different viewpoint than the first set of images; The processor is configured to determine a first pixel fingerprint based on a first set of images, where the first pixel fingerprint includes a first set of pixel values ​​at a first pixel location in each image of the first time-lapse image set; the processor is configured to determine a second pixel fingerprint based on a second set of images, wherein the second pixel fingerprint includes a second set of pixel values ​​at second pixel locations in each image of the second time-lapse image set; The processor is configured to generate a first binary pixel fingerprint based on the first pixel fingerprint, where the first binary pixel fingerprint includes a first set of binary values ​​generated by comparing each of the first set of pixel values ​​to a threshold; The processor is configured to generate a second binary pixel fingerprint based on the second pixel fingerprint, where the second binary pixel fingerprint comprises a second set of binary values ​​generated by comparing each of the second set of pixel values ​​to a threshold; and further configured to: the processor is configured to determine whether a stereo correspondence exists between a first pixel fingerprint of a first set of images and a second pixel fingerprint of a second set of images based at least in part on a comparison of the first binary pixel fingerprint and the second binary pixel fingerprint; The above method.

18. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, the at least one computer hardware processor receives a first set of images of a scene and a second set of images of the scene from one or more image sensors, where the second set of images is acquired from a different viewpoint than the first set of images; the at least one computer hardware processor determines a first pixel fingerprint based on the first set of images, where the first pixel fingerprint includes a first set of pixel values ​​at a first pixel location in each image of the first time-lapse image set; the at least one computer hardware processor determines a second pixel fingerprint based on a second set of images, wherein the first pixel fingerprint includes a second set of pixel values ​​at second pixel locations in each image of the second time-lapse image set; the at least one computer hardware processor generates a first binary pixel fingerprint based on the first pixel fingerprint, wherein the first binary pixel fingerprint includes a first set of binary values ​​generated by comparing each of the first set of pixel values ​​to a threshold; The at least one computer hardware processor generates a second binary pixel fingerprint based on the second pixel fingerprint, where the second binary pixel fingerprint includes a second set of binary values ​​generated by comparing each of the second set of pixel values ​​to a threshold; and further the at least one computer hardware processor determines whether a stereo correspondence exists between a first pixel fingerprint of a first set of images and a second pixel fingerprint of a second set of images based at least in part on comparing the first binary pixel fingerprint and the second binary pixel fingerprint; The computer-readable storage medium, wherein the instructions are configured to:

19. 1. A system for determining stereo correspondence between a first pixel fingerprint of a first set of images and a second pixel fingerprint of a second set of images, the system comprising: The processor is configured to perform the steps of receiving a first set of images of a scene and a second set of images of the scene from one or more image sensors, where the second set of images is acquired from a different viewpoint than the first set of images; the processor is configured to perform the step of determining a first pixel fingerprint based on a first set of images, wherein the first pixel fingerprint comprises a first set of pixel values ​​at different locations in space and / or time relative to a first pixel location in each image of the first set of images; the processor is configured to perform the step of determining a second pixel fingerprint based on a second set of images, wherein the second pixel fingerprint comprises a second set of pixel values ​​at different locations in space and / or time relative to a second pixel location in each image of the second set of images; The processor is configured to perform the step of generating a first binary pixel fingerprint based on the first pixel fingerprint, where the first binary pixel fingerprint includes a first set of binary values ​​generated by comparing each of the first set of pixel values ​​to a threshold; The processor is configured to perform the step of generating a second binary pixel fingerprint based on the second pixel fingerprint, where the second binary pixel fingerprint includes a second set of binary values ​​generated by comparing each of the second set of pixel values ​​to a threshold, and further comprising: the processor is configured to perform the step of determining whether a stereo correspondence exists between a first pixel fingerprint of a first set of images and a second pixel fingerprint of a second set of images based at least in part on a comparison of the first binary pixel fingerprint and the second binary pixel fingerprint; The above system.

Images