A learnable cost volume for determining pixel correspondence

Abstract

A method includes obtaining a first plurality of feature vectors associated with a first image and a second plurality of feature vectors associated with a second image. The method also includes generating a plurality of transformed feature vectors by transforming each corresponding feature vector of the first plurality of feature vectors by a kernel matrix trained to define an elliptical inner product space. The method also includes generating a cost volume by determining a plurality of inner products for each corresponding transformed feature vector in the plurality of transformed feature vectors, wherein each corresponding inner product of the plurality of inner products is between the corresponding transformed feature vector and a corresponding candidate feature vector of a corresponding subset of the second plurality of feature vectors. The method also includes determining pixel correspondences between the first image and the second image based on the cost volume.

Description

Technical Field

[0001] The present disclosure relates to a method for determining pixel correspondence, and more particularly, to a method for determining pixel correspondence using a learnable cost volume. Background Art

[0002] Pixel correspondence between two images may indicate how pixels in the first image relate to pixels in the second image. The first image and the second image may be captured at different times and / or from different perspectives and may therefore provide different representations of an environment. A pair of pixels that are mapped to each other may represent the same portion of an environment, indicating apparent displacement of the portion of the environment relative to the camera over time and / or due to the different perspectives of the images. Pixel correspondence may be determined for and used in a number of different applications. In one example, pixel correspondence between two images captured at different times may be determined as part of an optical flow calculation. In another example, pixel correspondence between two simultaneously captured images may be determined as part of a stereo disparity calculation. Summary of the invention

[0003] The pixel correspondence between the two images can be determined based on a comparison of a feature vector associated with a pixel of the first image and a feature vector associated with a pixel of the second image. This comparison of feature vectors can generate a cost volume that indicates the similarity between the pair of pixels for each pair of compared pixels. The similarity between the two pixels can be quantified by the inner product between the feature vectors associated with the two pixels. However, instead of using the Euclidean inner product, a learned / trained elliptical inner product can be used to generate the cost volume. Specifically, the first feature vector of the pair of pixels can be transformed by a learned / trained kernel matrix, and the transformed first feature vector can then be multiplied with the second feature vector of the pair of pixels. The kernel matrix can be learned / trained to define an elliptical inner product space. The cost volume generated based on the learned / trained elliptical inner product can then be used to determine the pixel correspondence between the two images.

[0004] In a first example, a computer-implemented method includes obtaining (i) a first plurality of feature vectors associated with a first image and (ii) a second plurality of feature vectors associated with a second image. The computer-implemented method also includes generating a plurality of transformed feature vectors by transforming each corresponding feature vector of the first plurality of feature vectors by a kernel matrix trained to define an elliptical inner product space. The computer-implemented method also includes generating a cost volume by determining a plurality of inner products for each transformed feature vector in the plurality of transformed feature vectors. Each inner product of the plurality of inner products may be between a corresponding transformed feature vector and a corresponding candidate feature vector of a corresponding subset of the second plurality of feature vectors. The computer-implemented method also includes determining pixel correspondences between the first image and the second image based on the cost volume.

[0005] In a second example, a system includes a processor and a non-transitory computer-readable storage medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations. The operations include obtaining (i) a first plurality of feature vectors associated with a first image and (ii) a second plurality of feature vectors associated with a second image. The operations also include generating a plurality of transformed feature vectors by transforming each corresponding feature vector of the first plurality of feature vectors by a kernel matrix trained to define an elliptical inner product space. The operations also include generating a cost volume by determining a plurality of inner products for each transformed feature vector of the plurality of transformed feature vectors. Each inner product of the plurality of inner products may be between corresponding candidate feature vectors of a corresponding subset of the corresponding transformed feature vector and the second plurality of feature vectors. The operations also include determining pixel correspondences between the first image and the second image based on the cost volume.

[0006] In a third example, a non-transitory computer-readable storage medium is provided on which instructions are stored that, when executed by a computing system, cause the computing system to perform operations. The operations include obtaining (i) a first plurality of feature vectors associated with a first image and (ii) a second plurality of feature vectors associated with a second image. The operations also include generating a plurality of transformed feature vectors by transforming each corresponding feature vector of the first plurality of feature vectors by a kernel matrix trained to define an elliptical inner product space. The operations also include generating a cost volume by determining a plurality of inner products for each transformed feature vector of the plurality of transformed feature vectors. Each inner product of the plurality of inner products may be between a corresponding transformed feature vector and a corresponding candidate feature vector of a corresponding subset of the second plurality of feature vectors. The operations also include determining pixel correspondences between the first image and the second image based on the cost volume.

[0007] In a fourth example, a system is provided that includes a device for obtaining (i) a first plurality of feature vectors associated with a first image and (ii) a second plurality of feature vectors associated with a second image. The system also includes a device for generating a plurality of transformed feature vectors by transforming each corresponding feature vector of the first plurality of feature vectors by a kernel matrix trained to define an elliptical inner product space. The system also includes a device for generating a cost volume by determining a plurality of inner products for each of the plurality of transformed feature vectors. Each inner product of the plurality of inner products may be between a corresponding transformed feature vector and a corresponding candidate feature vector of a corresponding subset of the second plurality of feature vectors. The system also includes a device for determining pixel correspondences between the first image and the second image based on the cost volume.

[0008] One or more of the following features may apply to one or each of the above examples.

[0009] The kernel matrix can be symmetric and positive definite.

[0010] The kernel matrix may be defined based on the product of (i) a diagonal matrix including a plurality of positive diagonal values ​​and (ii) an orthogonal matrix.

[0011] The kernel matrix may be defined via eigendecomposition by determining (i) a first product between the transpose of a diagonal matrix and an orthogonal matrix, and (ii) a second product between the first product and the diagonal matrix.

[0012] Each of the plurality of positive diagonal values ​​of the diagonal matrix may include a respective weight applied to a corresponding dimension of each respective eigenvector of the first plurality of eigenvectors.The orthogonal matrix may be configured to apply a rotation to each respective eigenvector of the first plurality of eigenvectors.

[0013] Each corresponding positive diagonal value in the plurality of positive diagonal values ​​may be represented as a function of a corresponding training parameter. The function may map (i) a real value assigned to the corresponding training parameter to (ii) a positive real value. The kernel matrix is ​​trained to define the elliptical inner product space by iteratively adjusting the real value assigned to the corresponding training parameter for each corresponding positive diagonal value.

[0014] The orthogonal matrix may be an element of a subset of a special set of orthogonal matrices. The subset may be selected so that the eigenvalues ​​of the matrices of the subset exclude negative eigenvalues.

[0015] The matrices of the subsets may be connected. An orthogonal matrix may be represented using Clay representation as: (i) the difference between the identity matrix and the training matrix, and (ii) the product of the identity matrix and the inverse of the sum of the training matrices. The training matrix may be skew-symmetric. The kernel matrix may be trained by initializing the training matrix to the identity matrix and iteratively adjusting the values ​​of the training matrix to define an elliptical inner product space.

[0016] The orthogonal matrix can be an element of a Stiefel matrix manifold comprising a plurality of block orthogonal matrices. The kernel matrix can be trained to define an elliptical inner product space by Riemannian gradient descent, comprising: determining a gradient projection from the Stiefel matrix manifold to a tangent space at the orthogonal matrix, adjusting a value of the orthogonal matrix based on the gradient projection, and determining a retraction of the orthogonal matrix as the adjustment into the Stiefel matrix.

[0017] Each respective feature vector of the first plurality of feature vectors may be associated with a corresponding pixel of the first image. Each respective feature vector of the second plurality of feature vectors may be associated with a corresponding pixel of the second image.

[0018] For respective transformed feature vectors associated with corresponding pixels at a particular pixel location within the first image, a corresponding subset of the second plurality of feature vectors may be associated with a plurality of pixels within a search window within the second image that includes the particular pixel location.

[0019] The first image and the second image may each represent an environment. Determining pixel correspondence between the first image and the second image may include determining, based on the cost volume and for each corresponding pixel in the plurality of pixels in the first image, a corresponding pixel in the second image that is determined to represent the same portion of the environment as the corresponding pixel.

[0020] The first image may represent an environment at a first time. The second image may represent an environment at a second time. Determining pixel correspondence between the first image and the second image may include determining, by one or more machine learning models, an optical flow between the first image and the second image based on a cost volume.

[0021] The first image and the second image may be stereo images of the environment. Determining pixel correspondence between the first image and the second image may include determining a depth image based on a cost volume by one or more machine learning models.

[0022] Obtaining the first plurality of feature vectors and the second plurality of feature vectors may include processing the first image by one or more machine learning models to generate the first plurality of feature vectors, and processing the second image by one or more machine learning models to generate the second plurality of feature vectors.

[0023] The first plurality of eigenvectors and the second plurality of eigenvectors may correspond to the first image and the second image of the first size ratio. A third plurality of eigenvectors associated with the first image and a fourth plurality of eigenvectors associated with the second image may be obtained. The third plurality of eigenvectors and the fourth plurality of eigenvectors may correspond to the first image and the second image of the second size ratio. The second size ratio may be greater than the first size ratio. The fourth plurality of eigenvectors may be distorted based on the pixel correspondence between the first image and the second image of the first size ratio. The second plurality of transformed eigenvectors may be generated by a kernel matrix by transforming each eigenvector in the third plurality of eigenvectors. The second cost body may be generated by determining a second plurality of inner products for each transformed eigenvector of the second plurality of transformed eigenvectors. Each inner product of the second plurality of inner products may be between a corresponding transformed eigenvector of the second plurality of transformed eigenvectors and a corresponding candidate eigenvector of a corresponding subset of the distorted fourth plurality of eigenvectors. Based on the second cost body, a second pixel correspondence between the first image and the second image of the second size ratio may be determined.

[0024] These and other examples, aspects, advantages and alternatives will become apparent to those of ordinary skill in the art by reading the following detailed description (with reference to the accompanying drawings where appropriate). In addition, the summary and other descriptions and drawings provided herein are intended only to illustrate implementations by way of example, and therefore, many variations are possible. For example, structural elements and process steps may be rearranged, combined, distributed, eliminated or otherwise changed while remaining within the scope of the examples as described in the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A computing system according to examples described herein is shown.

[0026] Figure 2 A computing device according to examples described herein is shown.

[0027] Figure 3A Feature vectors according to examples described herein are shown.

[0028] Figure 3B An image search window is shown according to examples described herein.

[0029] Figure 3C A cost volume according to examples described herein is shown.

[0030] Figure 4 A trainable cost volume model according to examples described herein is shown.

[0031] Figure 5A Aspects of Euclidean inner product space according to examples described herein are shown.

[0032] Figure 5B Aspects of an elliptical inner product space are shown according to examples described herein.

[0033] Figure 6 A system for determining optical flow according to examples described herein is shown.

[0034] Figure 7 A flow chart according to examples described herein is shown.

[0035] Fig. 8A and 8B Test results according to examples described herein are shown. DETAILED DESCRIPTION

[0036] Example methods, devices, and systems are described herein. It should be understood that the words "example" and "exemplary" herein mean "serving as an example, instance, or illustration." Any example or feature described herein as "example," "exemplary," "and / or" illustrative is not necessarily to be construed as preferred or advantageous over other examples or features unless so stated. Therefore, other examples may be utilized and other changes may be made without departing from the scope of the subject matter presented herein.

[0037] Therefore, the examples described herein are not meant to be limiting.It will be readily understood that aspects of the present disclosure, as generally described herein and illustrated in the accompanying drawings, may be arranged, substituted, combined, separated, and designed in a variety of different configurations.

[0038] Furthermore, unless the context suggests otherwise, the features shown in each of the figures in the drawings may be used in combination with each other. Therefore, the drawings should generally be viewed as constituent aspects of one or more overall examples, and it is understood that not every example requires all of the features shown.

[0039] Additionally, any enumeration of elements, blocks, or steps in this specification or claims is for clarity. Therefore, such enumeration should not be interpreted as requiring or implying that these elements, blocks, or steps follow a specific arrangement or are performed in a specific order. Unless otherwise specified, numbers are not drawn to scale.

[0040] I. Overview

[0041] Pixel correspondence between two images (sometimes referred to as pixel disparity) can indicate a pair of pixels, where each pixel in the pair of pixels provides a different representation of the same part of the environment. Pixel correspondence can be determined based on a cost volume. The cost volume can indicate the similarity between multiple potential matching pixel pairs in the first image and the second image. Specifically, the cost volume can indicate multiple similarity metrics for each corresponding pixel in the first image. Each corresponding similarity metric in the multiple similarity metrics can be associated with a corresponding candidate pixel in a plurality of candidate pixels within a search window in the second image. The search window (e.g., a square, rectangle, etc.) can be used to select multiple candidate pixels from the second image, and its position in the second image can be based on (e.g., centered on) the coordinates of the corresponding pixel in the first image. Therefore, the search window can indicate the candidate pixel in the second image that is most likely to correspond to the corresponding pixel in the first image.

[0042] A similarity measure between a first pixel and a second pixel can be based on an inner product between a first feature vector associated with the first pixel and a second feature vector associated with the second pixel. The feature vector associated with the pixel can be determined by processing the pixel and / or one or more adjacent pixels by a machine learning algorithm (e.g., a convolutional neural network), a rule-based image processing algorithm, and / or a combination thereof. Therefore, the feature vector can represent the information content of the pixel and / or one or more adjacent pixels. Therefore, the inner product between the first feature vector and the second feature vector can represent the degree of similarity between the two feature vectors.

[0043] However, in some cases, some dimensions of the feature vector may be correlated with each other and may therefore at least partially represent repeated information. In addition, some dimensions of the feature vector may contain information useful in identifying pixel correspondences, while other dimensions may contain noise or irrelevant information. These problems may stem from the fact that the combination of algorithms used to generate feature vectors sometimes generates noise and / or irrelevant vector values. Constructing the cost volume using the Euclidean inner product space does not provide a method for considering the correlation between different dimensions of the feature vector and / or a method for weighting the distinguishing dimensions of the feature vector differently from the dimensions that represent noise and / or less relevant information.

[0044] Therefore, instead of using the Euclidean inner product to define the cost volume, the cost volume can be defined based on the elliptical inner product. Specifically, the elliptical inner product can be determined by transforming the first eigenvector by a kernel matrix and then multiplying the transformed first eigenvector by the second eigenvector. The kernel matrix can be configured to define the elliptical inner product space via a training process. Therefore, the cost volume generated using the kernel matrix can be referred to as a learning cost volume and / or a training cost volume. When the image includes large pixel displacements, textureless areas, motion blur, occlusion, illumination changes, and non-Lambertian reflections, etc., the learned cost volume allows pixel correspondences to be determined more accurately.

[0045] In some cases, a particular trained kernel matrix may be specific to feature vectors generated using a particular set of algorithms (e.g., convolutional neural networks). Thus, when the algorithm used to generate the feature vectors is modified, the correlation and noise generated thereby may change. Thus, the kernel matrix may be retrained to appropriately adjust the elliptical inner product space to handle the correlation and noise generated by the modified algorithm.

[0046] In order to define an effective inner product space, the kernel matrix can be symmetric and positive definite. Explicitly imposing symmetry and positive definite constraints on the kernel matrix during training may complicate the training process because each update to the kernel matrix during training needs to both follow the gradient and result in a symmetric positive definite matrix. Therefore, potential updates that follow the gradient but do not result in symmetric and positive definite matrices can be discarded until it is determined that an update that does result in a symmetric and positive definite matrix is ​​obtained. This iterative process can be time consuming and computationally wasteful. Therefore, instead of directly imposing symmetry and positive definite constraints on the kernel matrix, the kernel matrix can be decomposed into a diagonal matrix and an orthogonal matrix via eigenvalue decomposition.

[0047] A diagonal matrix may include eigenvalues ​​of the kernel matrix along its diagonal. To satisfy the requirement of positive definiteness, the eigenvalues ​​may be positive. A diagonal matrix may be operable to reweight the different dimensions of the eigenvectors, which may be geometrically viewed as a stretching of the coordinate axes. For example, a diagonal matrix may amplify the distinguishing dimensions of the eigenvectors while suppressing dimensions representing noise and / or indiscriminate signals. The eigenvalues ​​represented by the diagonal matrix may be parameterized by a function that maps real values ​​of training parameters to positive real values ​​of the eigenvalues. Thus, the training process may be performed by modifying unconstrained training parameters rather than the positive eigenvalues ​​themselves.

[0048] As the name implies, an orthogonal matrix can be an orthogonal matrix such that the transpose of an orthogonal matrix is ​​equal to the inverse of the orthogonal matrix. An orthogonal matrix can be operated to linearly transform the eigenvectors into a new eigenspace that accounts for the correlations between its different dimensions. This transformation can be viewed geometrically as a rotation of the coordinate axes.

[0049] Orthogonal matrices can be trained by parameterizing them as skew-symmetric training matrices using the Clay representation / transformation, or by Riemannian gradient descent on the Stiefel matrix manifold. In either case, the orthogonality of the orthogonal matrices can be achieved as a result of the training process without imposing any explicit constraints on them, thereby allowing the training process to freely follow gradients determined with respect to the training parameters.

[0050] Specifically, using the Clay representation, an orthogonal matrix can be selected from a subset of a special orthogonal matrix group, where the matrices of the subset are connected and do not include matrices with negative eigenvalues ​​in their spectrum. The matrices of the subset can be represented using the Clay transform as the product of (i) the difference between the identity matrix and the skew-symmetric training matrix and (ii) the inverse of the sum of the identity matrix and the skew-symmetric training matrix. Because the matrices of the subset are connected and include the identity matrix, the skew-symmetric training matrix can be initialized to the identity matrix, and its value can be iteratively refined until the desired orthogonal matrix is ​​reached. The desired orthogonal matrix can be reached when the cost volume generated based on it allows the determination of pixel correspondences with at least a threshold accuracy.

[0051] On the other hand, using Riemannian gradient descent may involve selecting an initial orthogonal matrix from the Stiefel matrix manifold and determining the standard gradient at a point on the Stiefel matrix manifold corresponding to the initial orthogonal matrix. The standard gradient may then be projected into a tangent space that is tangent to the Stiefel matrix manifold at a point on it corresponding to the initial orthogonal matrix, resulting in a natural gradient. The natural gradient may be used to update the values ​​of the projected initial orthogonal matrix, resulting in an updated matrix that may no longer be orthogonal because it resides in the tangent space, not the Stiefel matrix manifold. Therefore, the updated matrix may be retracted back into the Stiefel matrix manifold, resulting in an orthogonal update matrix.

[0052] II. Example Computing Device

[0053] Figure 1 An example size form factor of computing system 100 is shown. Computing system 100 may be, for example, a mobile phone, a tablet computer, or a wearable computing device. However, other examples are possible. Computing system 100 may include various elements, such as body 102, display 106, and buttons 108 and 110. Computing system 100 may also include forward-pointing camera 104, rear-pointing cameras 112 and 113, forward-pointing infrared camera 114, and infrared mode projector 116.

[0054] Front-pointing camera 104 may be positioned on a side of body 102 that generally faces a user when in operation (e.g., on the same side as display 106). Rear-pointing cameras 112 and 113 may be positioned on a side of body 102 opposite front-pointing camera 104. Referring to cameras as front-facing and rear-facing is arbitrary, and computing system 100 may include multiple cameras positioned on various sides of body 102. Front-pointing camera 104 and rear-facing cameras 112 and 113 may each be configured to capture images in the visible spectrum.

[0055] Display 106 may represent a cathode ray tube (CRT) display, a light emitting diode (LED) display, a liquid crystal (LCD) display, a plasma display, an organic light emitting diode (OLED) display, or any other type of display known in the art. In some examples, display 106 may display a current image captured by front camera 104, rear camera 112, rear camera 113, and / or infrared camera 114, and / or a digital representation of an image that may be captured or most recently captured by one or more of these cameras. Thus, display 106 may be used as a viewfinder for the camera. Display 106 may also support touch screen functionality that may adjust settings and / or configurations of any aspect of computing system 100.

[0056] The forward camera 104 may include an image sensor and related optical elements such as a lens. The forward camera 104 may provide a zoom capability or may have a fixed focal length. In other examples, interchangeable lenses may be used with the forward camera 104. The forward camera 104 may have a variable mechanical aperture and a mechanical and / or electronic shutter. The forward camera 104 may also be configured to capture still images, video images, or both. In addition, the forward camera 104 may represent, for example, a single-view camera. The rearward camera 112, the rearward camera 113, and / or the infrared camera 114 may be arranged similarly or differently. Additionally, one or more of the forward camera 104, the rearward camera 112, the rearward camera 113, or the infrared camera 114 may be an array of one or more cameras. In one example, the rearward camera 112 and the rearward camera 113 may form part of a stereo camera.

[0057] One or more of the forward camera 104, the rearward camera 112, and / or the rearward camera 113 may include or be associated with an illumination component that provides a light field in the visible spectrum to illuminate the target object. For example, the illumination component may provide a flash or constant illumination of the target object. The illumination component may also be configured to provide a light field that includes one or more of structured light, polarized light, and light with specific spectral content. In the context of the examples herein, other types of light fields that are known and used to recover a three-dimensional (3D) model from an object may be used.

[0058] Infrared pattern projector 116 can be configured to project an infrared structured light pattern onto a target object. In one example, infrared projector 116 can be configured to project a dot pattern and / or a flood pattern. Thus, infrared projector 116 can be used in combination with infrared camera 114 to determine multiple depth values ​​corresponding to different physical features of a target object.

[0059] That is, infrared projector 116 can project a known and / or predetermined dot pattern onto a target object, and infrared camera 114 can capture an infrared image of the target object including the projected dot pattern. Computing system 100 can then determine the correspondence between an area in the captured infrared image and a specific portion of the projected dot pattern. Given the position of infrared projector 116, the position of infrared camera 114, and the position of an area corresponding to a specific portion of the projected dot pattern within the captured infrared image, computing system 100 can then use triangulation to estimate the depth to the surface of the target object. By repeating this operation for different areas corresponding to different portions of the projected dot pattern, computing system 100 can estimate the depth of various physical features or portions of the target object. In this way, computing system 100 can be used to generate a three-dimensional (3D) model of a target object.

[0060] Computing system 100 may also include an ambient light sensor that may continuously or from time to time determine the ambient brightness (e.g., in terms of visible light and / or infrared light) of a scene that may be captured by cameras 104, 112, 113, and / or 114. In some implementations, the ambient light sensor may be used to adjust the display brightness of display 106. Additionally, the ambient light sensor may be used to determine an exposure length for one or more of cameras 104, 112, 113, or 114, or to assist in such determination.

[0061] The computing system 100 may be configured to capture images of a target object using the display 106 and the forward camera 104, the rearward camera 112, the rearward camera 113, and / or the forward infrared camera 114. The captured images may be a plurality of still images or a video stream. Image capture may be triggered by activating a button 108, pressing a soft key on the display 106, or by some other mechanism. Depending on the implementation, images may be automatically captured at specific time intervals, for example, when the button 108 is pressed, under appropriate lighting conditions for the target object, when the digital camera device 100 is moved a predetermined distance, or according to a predetermined capture schedule.

[0062] As described above, the functionality of computing system 100 may be integrated into a computing device such as a wireless computing device, a cell phone, a tablet computer, a laptop computer, etc. For purposes of example, Figure 2 is a simplified block diagram illustrating some of the components of an example computing device 200 that may include a camera component 224 .

[0063] By way of example and not limitation, computing device 200 may be a cellular mobile phone (e.g., a smart phone), a still camera, a video camera, a computer (such as a desktop, notebook, tablet, or handheld computer), a personal digital assistant (PDA), a home automation component, a digital video recorder (DVR), a digital television, a remote control, a wearable computing device, a gaming console, a robotic device, or other type of device. Computing device 200 may be equipped with at least some image capture and / or image processing capabilities. It should be understood that computing device 200 may represent a physical image processing system, a specific physical hardware platform on which an image sensing and processing application operates in software, or other combinations of hardware and software configured to perform image capture and / or processing functions.

[0064] like Figure 2 As shown, computing device 200 may include a communication interface 202 , a user interface 204 , a processor 206 , data storage 208 , and a camera component 224 , all of which may be communicatively connected together via a system bus, network, or other connection mechanism 210 .

[0065] The communication interface 202 may allow the computing device 200 to communicate with other devices, access networks, and / or transport networks using analog or digital modulation. Thus, the communication interface 202 may facilitate circuit-switched and / or packet-switched communications, such as plain old telephone service (POTS) communications and / or Internet Protocol (IP) or other packetized communications. For example, the communication interface 202 may include a chipset and an antenna that are arranged to communicate wirelessly with a radio access network or access point. In addition, the communication interface 202 may take the form of or include a wired interface, such as an Ethernet, Universal Serial Bus (USB), or High-Definition Multimedia Interface (HDMI) port. The communication interface 202 may also take the form of or include a wireless interface, such as Wi-Fi, The communication interface 202 may include a global positioning system (GPS) or a wide area wireless interface such as WiMAX or 3GPP Long Term Evolution (LTE). However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used on the communication interface 202. In addition, the communication interface 202 may include multiple physical communication interfaces (e.g., Wi-Fi interfaces, interface and wide area wireless interface).

[0066] User interface 204 can be used to allow computing device 200 to interact with human or non-human users, such as receiving input from a user and providing output to the user.Therefore, user interface 204 can include input components such as keypad, keyboard, touch-sensitive panel, computer mouse, trackball, joystick, microphone, etc.User interface 204 can also include one or more output components, such as a display screen that can be combined with a touch screen.Display screen can be based on CRT, LCD and / or LED technology, or other technologies known now or developed later.User interface 204 can also be configured to generate (multiple) audio outputs via loudspeaker, speaker jack, audio output port, audio output device, earphone and / or other similar devices.User interface 204 can also be configured to receive and / or capture (multiple) audible speech, (multiple) noise and / or (multiple) signal by microphone and / or other similar devices.

[0067] In some examples, user interface 204 may include a display that serves as a viewfinder for still camera and / or video camera functions (e.g., in both the visible and infrared spectrums) supported by computing device 200. Additionally, user interface 204 may include one or more buttons, switches, knobs, and / or dials that facilitate configuration and focusing of camera functions and capture of images. Some or all of these buttons, switches, knobs, and / or dials may be implemented via a touch screen.

[0068] Processor 206 may include one or more general purpose processors (e.g., microprocessors) and / or one or more special purpose processors (e.g., digital signal processors (DSPs), graphics processing units (GPUs), floating point units (FPUs), network processors, or application specific integrated circuits (ASICs). In some cases, the special purpose processors may be capable of image processing, image alignment, and merging images, among other capabilities. Data storage 208 may include one or more volatile and / or non-volatile storage components, such as magnetic storage, optical storage, flash storage, or organic storage, and may be integrated in whole or in part with processor 206. Data storage 208 may include removable and / or non-removable components.

[0069] The processor 206 may be capable of executing program instructions 218 (e.g., compiled or uncompiled program logic and / or machine code) stored in the data store 208 to perform various functions described herein. Thus, the data store 208 may include a non-transitory computer-readable medium having program instructions stored thereon that, when executed by the computing device 200, cause the computing device 200 to perform any method, process, or operation disclosed in this specification and / or the accompanying drawings. The execution of the program instructions 218 by the processor 206 may cause the processor 206 to use the data 212.

[0070] For example, program instructions 218 may include an operating system 222 (e.g., an operating system kernel, (multiple) device drivers, and / or other modules) and one or more applications 220 (e.g., a camera function, an address book, email, web browsing, social networking, an audio-to-text function, a text translation function, and / or a gaming application) installed on computing device 200. Similarly, data 212 may include operating system data 216 and application data 214. Operating system data 216 may be primarily accessed by operating system 222, and application data 214 may be primarily accessed by one or more of application programs 220. Application data 214 may be arranged in a file system that is visible to or hidden from a user of computing device 200.

[0071] Applications 220 may communicate with operating system 222 via one or more application programming interfaces (APIs). For example, these APIs may facilitate applications 220 to read and / or write application data 214, send or receive information via communication interface 202, receive and / or display information on user interface 204, and the like.

[0072] In some dialects, the application programs 220 may be referred to simply as “apps.” Additionally, the application programs 220 may be downloaded to the computing device 200 through one or more online application stores or application markets. However, the application programs may also be installed on the computing device 200 in other ways, such as via a web browser or through a physical interface (e.g., a USB port) on the computing device 200.

[0073] The camera assembly 224 may include, but is not limited to, an aperture, a shutter, a recording surface (e.g., photographic film and / or an image sensor), a lens, a shutter button, an infrared projector, and / or a visible light projector. The camera assembly 224 may include components configured to capture images in the visible spectrum (e.g., electromagnetic radiation having a wavelength of 400-700 nanometers) and components configured to capture images in the infrared spectrum (e.g., electromagnetic radiation having a wavelength of 701 nanometers-1 millimeter). The camera assembly 224 may be controlled, at least in part, by software executed by the processor 206.

[0074] III. Example Cost Volume Model

[0075] Figure 3A An example feature vector associated with a pixel of an image is shown. Specifically, Figure 3A An image 300 is shown having a width W and a height H, each of which is measured in pixels. Thus, image 300 includes H×W pixels. Each pixel of image 300 may be associated with a corresponding feature vector. For example, pixel 302 may be associated with feature vector 304, which may include a plurality of values ​​F1-FC (i.e., F1, F2, F3, F4, F5, F6, F7, and F8 through F9). C ), each value is associated with a corresponding dimension of feature vector 304.

[0076] Feature vector 304 may be generated by processing pixel 302 and / or one or more other pixels adjacent to pixel 302 via one or more machine learning models. For example, feature vector 304 may represent convolution features of pixel 302 and / or its adjacent pixels determined via one or more convolutional neural networks. In another example, feature vector 304 may represent the output of processing pixel 302 and / or its adjacent pixels via one or more rule-based image processing algorithms. In yet other examples, the values ​​F1-F2 of feature vector 304 may be determined in other ways. C Regardless of the value F1-F of the feature vector 304, CRegardless of the specific manner in which the feature vector 304 represents the information content of the pixel 302 and / or one or more other pixels adjacent thereto (e.g., the presence of certain colors, geometric features, and / or other patterns therein). Therefore, the corresponding feature vectors of two different pixels can be used to determine a similarity measure between the two different pixels. When the two different pixels are part of two different images, a similarity measure can be used to find the correspondence between the pixels of the two images. In some cases, pixel correspondence can be referred to as pixel disparity.

[0077] therefore, Figure 3B FIG. 1 shows a portion of a process for finding pixel correspondences between two images. Specifically, Figure 3B Including image 300 and image 306, each of which can provide a different representation of a common environment. Images 300 and / or 306 can be generated using cameras 104, 112, 113 and / or 114 of computing system 100 and / or camera component 224 of computing device 200. In one example, images 300 and 306 can be captured substantially simultaneously using a stereo camera, and thus the environment can be represented from different perspectives. Therefore, finding pixel correspondences between images 300 and 306 can serve as a basis for determining a depth image based on images 300 and 306. In another example, image 300 can be captured at a first time, and image 306 can be captured at a second time after the first time, so each can represent an environment at a different point in time. Therefore, finding pixel correspondences between images 300 and 306 can serve as a basis for determining the optical flow associated with images 300 and 306.

[0078] The correspondence between a particular pixel in image 300 and another pixel in image 306 may be determined by comparing the particular pixel in image 300 with a plurality of candidate pixels of image 306 located within a search window. The location of the search window in image 306 may be based on the coordinates of the particular pixel in image 300. For example, a search window 310 shown in image 306 may be used to find a match for pixel 308A in image 300. Search window 310 may have a width U and a height V and may be centered on pixel 308B in image 306. Pixels 308A and 308B may share the same coordinates in images 300 and 306, respectively. That is, pixel 308A may be located at row 5, column 5 of image 300, and pixel 308B may be located at row 5, column 5 of image 306.

[0079] The corresponding feature vector for each pixel in image 300 may be compared to the U x V candidate feature vector for the corresponding pixel in image 306. Search window 310 may indicate the maximum expected displacement of the environmental feature represented by pixel 308A between images 300 and 306. In other embodiments, search window 310 may include, but may not be centered on pixel 308B. Each corresponding pixel in image 300 may be associated with a corresponding search window in image 306 that includes the location of the corresponding pixel (e.g., by being centered on that location).

[0080] like Figure 3C As shown, the cost volume can be generated by comparing the feature vector of each corresponding pixel in the image 300 with the candidate feature vectors of multiple candidate pixels in the image 306 located within the corresponding search window. Specifically, Figure 3C Cost volume 312 includes an H x W x U*V array of similarity measures resulting from a comparison of feature vectors of images 300 and 306. Specifically, image 300 includes H x W pixels, each pixel being associated with a corresponding feature vector. Each of these H x W feature vectors is compared with a U*V candidate feature vector in image 306 that represents a potential pixel correspondence (i.e., associated with a pixel within a corresponding search window). Thus, each of the H x W pixels of image 300 is associated with a matching U*V potential pixel, resulting in a H x W x U*V dimension for cost volume 312. In some embodiments, cost volume 312 may instead be represented as a 4-dimensional H x W x U*V tensor, rather than Figure 3C 3D H x W x U*V tensors shown, but may represent the same information content.

[0081] Cost body 312 may represent costs associated with a plurality of possible pixel correspondences between images 300 and 306. Cost body 312 may be processed by one or more machine learning models to select a corresponding pixel in image 306 for each of a plurality of pixels in image 300. A pixel in image 300 and a corresponding pixel in image 306 that matches it should represent the same and / or substantially the same portion of the environment, albeit from different angles and / or at different times. Rather than optimizing a local cost for a pairing, one or more machine learning models may be configured to select individual pixel pairings that improve or optimize the global cost of such pairings. That is, one or more machine learning models may select pixel correspondences that work well and / or best on images 300 and 306 as a whole, rather than just for some of the pixels therein.

[0082] In some embodiments, the similarity measure between two feature vectors of two corresponding pixels can be represented by the Euclidean inner product between the two feature vectors. However, the Euclidean inner product can limit the representation ability of the cost volume 312 for at least two reasons. First, the Euclidean inner product may not be able to explain the correlation between different dimensions of the feature vector. Second, each dimension of the feature vector contributes equally to the Euclidean inner product. Therefore, the dimension representing noise can be given the same weight by the Euclidean inner product as the dimension representing the distinguishing signal useful in determining the exact pixel correspondence. Therefore, instead of using a fixed calculation based on the Euclidean inner product to determine the cost volume 312, a trained cost volume model can be used to construct the cost volume 312.

[0083] Figure 4 A cost volume model 400 is shown that can be trained to account for correlations between different dimensions of a feature vector and reweight the contribution of each feature vector dimension to the cost volume. The cost volume model 400 can be configured to compare the feature vectors 404-406 of the image 402 with the feature vectors 410-412 of the image 408 to determine a cost volume 434. Figure 4 The images 402 and 408 may represent Figure 3B The images 300 and 306, and the cost volume 434 can represent Figure 3C The pixel correspondence matching model 436 may be configured to determine pixel correspondences between corresponding pixels of the images 402 and 408 based on the cost volume 434 .

[0084] Cost volume model 400 may include vector comparison selector 414, which may be configured to iterate candidate pixel pairs of images 402 and 408. Specifically, vector comparison selector 414 may be configured to compare each corresponding feature vector of feature vectors 404-406 with each corresponding candidate feature vector of a corresponding plurality of candidate feature vectors selected from feature vectors 410-412. The corresponding plurality of feature vectors may be selected based on a search window corresponding to the corresponding feature vector from image 402, such as relative to the search window. Figure 3B Thus, a corresponding plurality of candidate feature vectors may be associated with pixels that are within a threshold distance (eg, defined by a search window) of pixel coordinates associated with corresponding feature vectors from image 402 .

[0085] Thus, vector comparison selector 414 may implement functions k′=(i-(U-1) / 2+k) and l′=(j-(V-1) / 2+l), where (i, j) represent the coordinates of the pixel associated with the corresponding feature vector in image 402, U represents the width of the search window, V represents the height of the search window, and (k′, l′) represent the coordinates of the pixel that is within the search window and is associated with a particular candidate feature vector in image 408. The value of index i may be in the range of 0 to H, the value of index j may be in the range of 0 to W, the value of index k may be in the range of 0 to U, and the value of index l may be in the range of 0 to V. Thus, vector comparison selector 414 may iterate through (i, j) for each pixel in image 402, and for each corresponding pixel in image 402, iterate through (k′, l′) for each pixel in image 408 that is within the search window corresponding to the corresponding pixel of pixel 402.

[0086] The cost volume model 400 may be configured to apply a transposition operator 416 to the corresponding feature vectors of the feature vectors 404-406 selected for comparison by the vector comparison selector 414. The cost volume model 400 may also be configured to apply an inner product operator 430 to the corresponding feature vectors after transposition and the kernel matrix 428. The cost volume model 400 may also be additionally configured to apply an inner product operator 432 to the output of the inner product operator 430 and the candidate feature vector selected by the vector comparison selector 414 from the corresponding subset of the feature vectors 410-412. Thus, the cost volume 400 may implement the function where F 1 is a W x H x C tensor representing feature vectors 404-406 (where C represents the dimension of each feature vector), F 2 is a W x H x C tensor representing the eigenvectors 410-412, is a vector with C values ​​and represents the 1 The corresponding eigenvectors for comparison are selected in , W is a C x C matrix representing the kernel matrix 428, and is a vector with C values ​​and represents the candidate feature vectors selected from F2 for comparison.

[0087] The kernel matrix 428 may be learned and / or trained to define an elliptical inner product space. For comparison, the cost volume determined using the Euclidean inner product may be expressed as the function The matrix is ​​taken and The inner product of is not used for transformation before As a result of training, kernel matrix 428 may be configured to decorrelate and reweight the eigenvectors , thereby allowing the cost volume 434 to more accurately represent each candidate pixel correspondence. Since the kernel matrix 428 defines the inner product space, in order to comply with the mathematical definition and characteristics of the inner product, the kernel matrix 428 can be symmetric (ie, W = W T ) and positive definite matrices (i.e., having only positive eigenvalues). Using a non-symmetric and / or positive definite kernel matrix, and thus not defining an efficient inner product space, may result in a cost volume that does not have an efficient inner product space to produce a useful cost volume when finding pixel correspondences.

[0088] The elements of the kernel matrix 428 may represent learnable and / or trainable parameters of the cost volume model 400. However, imposing symmetry and positive definiteness constraints during the training process may hinder the training process of the cost volume model 400. Therefore, the kernel matrix 428 may be represented as the product of the orthogonal matrix 418 and the diagonal matrix 422. Specifically, based on eigendecomposition and / or spectral decomposition, the kernel matrix 428 may be represented as W=P T ΛP, where W represents the kernel matrix 428, Λ represents the diagonal matrix 422, P represents the orthogonal matrix 418, and P T = represents the orthogonal matrix 418 transformed by the transposition operator 420. Therefore, the orthogonal matrix 418 can be transformed by the transposition operator 420, the inner product operator 426 and / or the like. Figure 4 The inner product operator 424 shown operates on the orthogonal matrix 418 and / or the diagonal matrix 422 to calculate W=P T ΛP is used to determine the kernel matrix 428.

[0089] The diagonal matrix 422 may include eigenvalues ​​of the kernel matrix 428 along its diagonal. That is, Λ=diag(λ1, . . . , λ C ), where for all i from 1 to C, λ i > 0. The diagonal matrix 422 may have eigenvectors The effect of re-weighting different dimensions, each eigenvalue λ i Constructing feature vectors The orthogonal matrix 418 may be such that W=P T ΛP or equivalently P T =P -1 The orthogonal matrix 418 may have the effect of rotating the coordinate axes, thereby decorrelating the eigenvectors different dimensions.

[0090] Since the kernel matrix 428 remains unchanged for each inner product calculation between feature vectors during use of the cost volume model 400, the kernel matrix 428 can be calculated based on the orthogonal matrix 418 and the diagonal matrix 422, stored by the cost volume model 400, and reused multiple times. That is, in order to reduce computational overhead, for each calculation performed by the operators 416, 430, and 432 based on the feature vectors 404-406 and 410-412, the kernel matrix 428 may not be recalculated by the operators 420, 424, and 426. However, the kernel matrix 428 may be recalculated each time the orthogonal matrix 418 and / or the diagonal matrix 422 are updated due to training.

[0091] Figure 5A and 5B The effect of multiplying the feature vector by the kernel matrix 428 is intuitively shown. Specifically, Figure 5A The Euclidean inner product space defined relative to the Euclidean reference frame 500 is intuitively shown, and Figure 5B The elliptical inner product space defined relative to the elliptical reference frame 510 is intuitively shown. The eigenvector f1 drawn along the Euclidean reference frame 500 represents the inner product space of the ellipse from F 1 The example feature vectors selected from F 2 Example feature vectors selected in .

[0092] In the Euclidean reference system 500, the feature vector f1 may be correlated with the feature vector f2. Specifically, the feature vector f1 includes an x ​​component oriented in the same direction as the feature vector f1. The elliptical reference system 510 may be trained to adjust the degree of correlation between the feature vectors f1 and f2. Specifically, as Figure 5B As shown, the elliptical reference frame 510 may be transformed relative to the Euclidean reference frame 500 such that the eigenvectors f1 and f2 are orthogonal to each other, thereby illustrating that the kernel matrix 428 may be trained to decorrelate at least some dimensions of the eigenvectors.

[0093] The transformed feature vector f1W can be rotated and / or scaled relative to the feature vector f1 by transformation via the kernel matrix 428. Due to this rotation and / or scaling, the degree of correlation between the transformed feature vector f1W and the feature vector f2 in the elliptical reference system 510 is different from the degree of correlation between the feature vector f1 and the feature vector f2 in the Euclidean reference system 500. Therefore, the kernel matrix 428 can be trained to adjust the scaling value generated by taking the inner product between the two feature vectors, so that the scaling value associated with similar feature vectors is increased and the scaling value associated with different feature vectors is reduced. In some cases, this may cause the transformed feature vector to be orthogonal or approximately orthogonal to (i.e., uncorrelated with) some candidate feature vectors, thereby indicating a poor pixel correspondence match, and coincident or approximately coincident with (i.e., highly correlated with) one or more other candidate feature vectors, thereby indicating a good pixel correspondence match.

[0094] IV. Example Training of the Cost Volume Model

[0095] Directly training the kernel matrix 428 may be difficult because, in order to define an effective inner product space, the values ​​of the kernel matrix 428 may be selected so that it is symmetric and positive definite. Therefore, in each training iteration, the kernel matrix 428 may be updated based on a gradient descent calculation using values ​​that (i) follow the gradient and (ii) cause the kernel matrix 428 to be symmetric and positive definite. Directly finding values ​​of the kernel matrix 428 that satisfy these two criteria may be difficult and time consuming. Therefore, the kernel matrix 428 may instead be represented based on an orthogonal matrix 418 and a diagonal matrix 422, each of which may provide a corresponding set of adjustable training parameters that facilitate training / learning of the kernel matrix 428.

[0096] Due to the positive definiteness of kernel matrix 428, diagonal matrix 422 may include positive values ​​along its diagonal (i.e., eigenvalues ​​of kernel matrix 428). Rather than explicitly restricting the training process to selecting only positive values ​​for diagonal matrix 422 (i.e., when defining Λ = diag(λ1, ..., λ C ) when the constraint λ is imposed i > 0), the element λ can be parameterized using a function that maps real values ​​to positive real values. i For example, we can use λ i =(π+2arctan(t i )) / (π-2arctan(t i )) to define a diagonal matrix 422, where t i is a training parameter that can be modified during training. By replacing the elements λ of the diagonal matrix 422 i Expressed as t i The real value of is mapped to λ i A positive real-valued function that can modify the training parameter t based on the gradient descent process i , but not the training parameters ti The range of possible values ​​imposes any additional constraints.

[0097] The orthogonal matrix 418 can also be parameterized to simplify the training process. In one embodiment, the training of the orthogonal matrix 418 can be facilitated by using the Clay transform to represent the orthogonal matrix 418 with the training matrix S and the identity matrix I. Specifically, the orthogonal matrix 418 can be represented according to P = (IS) (I + S) -1 To represent, where P represents an orthogonal matrix, and is a special orthogonal group SO consisting of matrices whose eigenvalues ​​do not include negative eigenvalues * (n). That is, Where σ(A) represents the spectrum (i.e., eigenvalue) of A. The training matrix S can be skew-symmetric, so that S T =-S, which allows (I+S) to be reversible.

[0098] Define the special orthogonal group SO * The matrix set of (C) may be a connected set and may include a C x C identity matrix I as its element. Therefore, in training, the orthogonal matrix P may be initialized to the identity matrix I. Assume P * is the optimal orthogonal matrix, since the matrices of the special orthogonal group (including P * and I) are connected, so after initializing the orthogonal matrix P as the identity matrix I, P can be obtained through training * Therefore, there exists a continuous path connecting the identity matrix I and any orthogonal matrix P that includes the optimal orthogonal matrix P * The special orthogonal group SO * (C) element.

[0099] The elements of the training matrix S can be parameterized so that they can be freely adjusted during training without imposing any additional constraints on them. For example, in the case where the training matrix S is an element of a three-dimensional real number space, the training matrix can be parameterized with training parameters s1, s2, and s3 according to S = [row(0, -s1, -s2), row(s1, 0, -s3), row(s2, s3, 0)]. T =-S), additional parameters can be used to define a higher dimensional training matrix S.

[0100] Therefore, the training matrix parameters s1-s can be adjusted based on the gradients calculated through backpropagation of the cost volume model. C (Define orthogonal matrix 418) and / or diagonal matrix parameters t1-t C (Define diagonal matrix 422) to train kernel matrix 428. In particular, the matrix parameters s1-s Cand / or diagonal matrix parameters t1-t C can be adjusted freely without any explicit constraints imposed on it, because the symmetry and positive definiteness of kernel matrix 428 arise from how the values ​​of these parameters map to the values ​​of the elements of orthogonal matrix 418 and diagonal matrix 422.

[0101] In an alternative embodiment, the training of the orthogonal matrix 418 can be performed using Riemannian gradient descent on the Stiefel matrix manifold. The Stiefel matrix manifold can be based on V k (R n )={A∈R n x k |A T A=I k} to define, where R n x k represents the space of nxk orthogonal matrices. Specifically, we can get the space of nxk orthogonal matrices from the Stiefel matrix manifold V C (R C )={A∈R C x C |A T A=I C Choose an orthogonal matrix P in , the Stiefel matrix manifold can contain all C x C (square) orthogonal matrices in it so that the orthogonality condition of the orthogonal matrix P is satisfied.

[0102] At the beginning of training, an initial value can be selected for the orthogonal matrix P so that the initial orthogonal matrix P is located on the Stifel matrix manifold V C (R C ). For example, the initial values ​​of the orthogonal matrix P and the diagonal matrix Λ can be selected to initialize the kernel matrix W as the unit matrix, that is, W=I. The gradient matrix Z at the point on the Stifel matrix manifold corresponding to the initial orthogonal matrix P can be determined. The gradient matrix Z can represent the standard gradient associated with the initial orthogonal matrix P. According to Π P (Z) = (I-PP T )Z+P*skew(P T Z), skew(X): = (XX T ) / 2, the gradient matrix Z can be projected onto the tangent space that is tangent to the Stiefel matrix manifold at the point on which it corresponds to the initial orthogonal matrix P. The projection of the gradient matrix Z onto the initial orthogonal matrix P is Π P (Z) can represent the natural gradient on the Stiefel matrix manifold.

[0103] Based on the projection Π P (Z) Perform gradient descent to determine the updated matrix P′. For example, the updated matrix P′ can be calculated according to P′=P-η*Π P(Z) is determined, where η is a modifiable constant representing the training rate of the orthogonal matrix. Since the updated matrix P′ is an element of the tangent space and is not necessarily orthogonal, the updated matrix P′ can be contracted back into the Stiefel matrix manifold to generate an updated orthogonal matrix. Specifically, the updated matrix P′ can be determined according to θ P (P′)=(P+P′)(I+(P′) T P′) 1 / 2 Contract back to the Stiefel matrix manifold to generate the updated orthogonal matrix P.

[0104] The projection of the updated orthogonal matrix, the adjustment of the values ​​of the projected orthogonal matrix, and the retraction of the adjusted orthogonal matrix can be repeated one or more times until a final orthogonal matrix P is obtained. The final orthogonal matrix P can provide a target accuracy level of pixel matching relative to a test data set including multiple image pairs together with the diagonal matrix Λ.

[0105] Typically, the values ​​of the orthogonal matrix P and the diagonal matrix Λ can be determined by training these matrices based on a training data set including multiple pairs of images. Each of the two images in each pair can provide a different representation of a common environment so that there is a correspondence between at least some of the pixels in the two images. The orthogonal matrix P and the diagonal matrix Λ can be assigned initial values, based on which the initial values ​​of the kernel matrix W can be determined. Multiple image pairs of the training data set can be processed by the cost volume model 400 using the kernel matrix W assigned with the initial values ​​to determine the training cost volume for each pair in the multiple pairs, which can be represented by the cost volume 434 and can be designated as a "training" cost volume because it is determined as part of the training process. The pixel correspondence matching model 436 can use the training cost volume to determine the training pixel correspondence.

[0106] The training pixel correspondences may be compared to the true pixel correspondences and / or their approximations via a loss function. In supervised learning applications, the loss function may explicitly compare the training pixel correspondences to the known true pixel correspondences. In some supervised applications, the training data set may include synthetic (e.g., computer-generated) image pairs with known pixel correspondences. In unsupervised learning applications, the loss function, among other possible loss functions, may measure photometric loss (e.g., how visually similar two pairs of pixels actually are to each other), smoothness loss (e.g., how abrupt the apparent displacement change between adjacent pixels is), and / or local contrast normalization loss. Thus, the loss function in an unsupervised learning application may provide an approximation of the ground truth correspondences since the explicit ground truth correspondences may not be known.

[0107] The gradient of the loss function can be determined relative to the training parameters defining the orthogonal matrix P and the diagonal matrix Λ, and the gradient can be used to determine one or more updates to the training parameters. Once these training parameters are updated, the kernel matrix W can be updated based on the updated orthogonal matrix P and the updated diagonal matrix Λ, and another iteration of the training process can be performed relative to the training data set. Therefore, the kernel matrix W can be iteratively refined. When the cost volume model 400 generates a cost volume that allows the pixel correspondence matching model 436 to determine the pixel correspondence of the training data set (or another data set retained for testing) with at least a threshold accuracy (e.g., 75%, 90%, 99%, etc.) (as measured by one or more loss functions), the training can be considered complete.

[0108] V. Example Application of the Cost Volume Model

[0109] Figure 6 The cost volume model 400 implemented as part of an optical flow system 640 is shown, and the optical flow system 640 is configured to determine the optical flow between the image pair. Specifically, the optical flow system 640 can be configured to determine the optical flow between the image 600 and the image 608. The optical flow system 640 may include the cost volume model 400, the feature distortion model 622, the optical flow model 624, and the optical flow upsampling model 628. Each of these components of the optical flow system 640 can be implemented as a hardware component, a software component, or a combination of the two. In addition, each of these components can represent a machine learning model, a rule-based algorithm, or a combination of the two. Therefore, in some embodiments, the cost volume model 400, the feature distortion model 622, the optical flow model 624, and / or the optical flow upsampling model 628 can be jointly trained, thereby allowing the cost volume model 400 to learn to generate the cost volume 434 so that it is useful to other components when determining the optical flow.

[0110] Images 600 and 608 may each be associated with one or more sets of feature vectors corresponding to one or more size ratios. For example, images 600 and 608 may be associated with feature vectors 606 and 614, respectively, corresponding to size ratio 620, feature vectors 604 and 612, respectively, corresponding to size ratio 618, and feature vectors 602 and 610, respectively, corresponding to size ratio 616. Size ratio 620 may correspond to full resolution versions of images 600 and 608 (e.g., 1,879,200 pixels per image), size ratio 618 may correspond to half resolution versions of images 600 and 608 (e.g., 939,600 pixels per image), and size ratio 616 may correspond to quarter resolution versions of images 600 and 608 (e.g., 469,800 pixels per image). Thus, as Figure 6As shown by the relative sizes of the corresponding blocks in , feature vector 604 may include half the number of feature vectors 606, and feature vector 602 may include one quarter the number of feature vectors 606. Other size ratios are possible.

[0111] The optical flow system 640 may be configured to process the feature vectors of the images 600 and 608 in a pyramidal manner, starting from the lowest resolution size scale 616, all the way to the full resolution size scale 620. Specifically, the cost volume 400 may be configured to generate the cost volume 434 based on the feature vectors 602 and 610. In subsequent iterations corresponding to the size scales 618 and 620, the feature vectors of the image 608 may be warped by the feature warp model 622 based on the previously determined optical flow. During the first iteration corresponding to the size scale 616, the feature vector 610 may not be warped because the previously determined optical flow may not be available.

[0112] The optical flow model 624 may be configured to determine an optical flow 626 based on the cost volume 434. The optical flow 626 may indicate, for each corresponding pixel of a plurality of pixels of the image 600, a direction and magnitude in which the corresponding pixel is shifted relative to the image 608. Thus, the optical flow 626 may include a flow field indicating an apparent movement of pixels of the image 600 between the image 600 and the image 608. The optical flow model 624 may represent an example implementation of the pixel correspondence matching model 436, and / or may include the pixel correspondence matching model 436 as a component thereof. The optical flow upsampling model 628 may be configured to generate an upsampled optical flow 630 based on the optical flow 626. Specifically, the upsampled optical flow 630 may correspond to the next higher size ratio, i.e., the size ratio 618.

[0113] The feature warping model 622 may be configured to warp the feature vector 612 corresponding to the size scale 618 based on the upsampled optical flow 630 determined based on the feature vectors 602 and 610 at the size scale 616. Thus, the large-scale flow determined at the size scale 616 may be accounted for by the feature warping model 622 at the size scale 618, allowing the processing at the size scale 618 to focus on the smaller-scale flow. Thus, in a second iteration corresponding to the size scale 618, the cost volume model 400 may be configured to generate the cost volume 434 based on the feature vector 604 and the warped feature vector 612. The optical flow model 624 may be configured to determine the optical flow 626 based on the cost volume 434 and the upsampled optical flow 630 determined based on the size scale 616, thus taking into account the large-scale flow from the scale 616 and the small-scale flow from the scale 618. The optical flow upsampling model 628 may be configured to generate the upsampled optical flow 630 based on the optical flow 626 at the scale 618.

[0114] Feature warping model 622 may be configured to warp feature vector 614 corresponding to size scale 620 based on upsampled optical flow 630 corresponding to size scale 618. Warped feature vector 614 and feature vector 606 may be processed by cost volume model 400 to generate cost volume 434 for size scale 620, and optical flow model 624 may generate optical flow 626 based on cost volume 434 corresponding to size scale 620 and upsampled optical flow 630 from size scale 618. Since optical flow 626 at size scale 620 may correspond to the full resolution of images 600 and 608, optical flow 626 may no longer be upsampled.

[0115] By using a cost volume model 400 including a trainable kernel matrix 428, rather than a cost volume model that relies on a Euclidean inner product, the accuracy of the optical flow field generated by the optical flow system 640 can be improved. Therefore, by improving the accuracy of the optical flow field, the cost volume model 400 can also improve the quality of the output of other downstream tasks (such as video interpolation, video prediction, video segmentation and / or action recognition, etc.) that utilize the determined optical flow field. The cost volume model 400 can additionally or alternatively be used as part of other systems that determine pixel correspondences between two or more images, such as for the purpose of determining a depth image based on a stereo image. In some cases, these other systems can use a pyramid structure similar to the optical flow system 640.

[0116] In some embodiments, the cost volume model 400 may include a corresponding kernel matrix for each size ratio. Thus, the cost volume model 400 may include a first kernel matrix corresponding to the size ratio 616, a second kernel matrix corresponding to the size ratio 618, and a third kernel matrix corresponding to the size ratio 620. As described above, each of the first, second, and third kernel matrices may be trained using a corresponding orthogonal matrix and a diagonal matrix.

[0117] Including the cost volume model 400 in the optical flow system 640 may increase the size of the optical flow system 640 by an amount that depends on the number and size of the kernel matrices. For example, two 64×64 kernel matrices and three 128×128 kernel matrices may add 2(64 2 )+3(128 2 )=57,344 parameters, which may constitute a relatively small increase. An example implementation of the optical flow system 640 without the cost volume model 400 may include approximately 41 million parameters, which means that the addition of the cost volume model 400 increases the size of the optical flow system 640 by less than 2%.

[0118] VI. Example Operation

[0119] Figure 7A flowchart of operations related to determining pixel correspondence between two images based on a learnable / trainable cost volume is shown. These operations may be performed by one or more of the computing system 100, computing device 200, cost volume model 400, pixel correspondence matching model 436, and / or optical flow system 640 among other possible types of devices or device subsystems. Figure 7 The examples of the present invention can be simplified by removing any one or more of the features shown therein. In addition, these examples can be combined with the features, aspects and / or implementations of any previous drawings, or otherwise described in this article.

[0120] Block 700 may involve obtaining (i) a first plurality of feature vectors associated with a first image and (ii) a second plurality of feature vectors associated with a second image.

[0121] Block 702 may involve generating a plurality of transformed feature vectors by transforming each respective feature vector of a first plurality of feature vectors by a kernel matrix trained to define an elliptical inner product space.

[0122] Block 704 may involve generating a cost volume by determining a plurality of inner products for each respective transformed feature vector of the plurality of transformed feature vectors. Each respective inner product of the plurality of inner products may be between a corresponding transformed feature vector and a corresponding candidate feature vector of a corresponding subset of the second plurality of feature vectors.

[0123] Block 706 may involve determining pixel correspondences between the first image and the second image based on the cost volume.

[0124] In some examples, the kernel matrix can be symmetric and positive definite.

[0125] In some examples, a kernel matrix can be defined based on the product of (i) a diagonal matrix including a plurality of positive diagonal values ​​and (ii) an orthogonal matrix.

[0126] In some examples, the kernel matrix may be defined via eigendecomposition by determining (i) a first product between the transpose of a diagonal matrix and an orthogonal matrix, and (ii) a second product between the first product and the diagonal matrix.

[0127] In some examples, each of the plurality of positive diagonal values ​​of the diagonal matrix may include a respective weight applied to a corresponding dimension of each respective eigenvector of the first plurality of eigenvectors.The orthogonal matrix may be configured to apply a rotation to each respective eigenvector of the first plurality of eigenvectors.

[0128] In some examples, each corresponding positive diagonal value of the plurality of positive diagonal values ​​can be represented as a function of a corresponding training parameter. The function can map (i) a real value assigned to the corresponding training parameter to (ii) a positive real value. The kernel matrix can be trained to define an elliptical inner product space by iteratively adjusting the real value assigned to the corresponding training parameter for each corresponding positive diagonal value.

[0129] In some examples, the orthogonal matrix may be an element of a subset of a special set of orthogonal matrices. The subset may be selected such that the eigenvalues ​​of the matrices of the subset exclude negative eigenvalues.

[0130] In some examples, the matrices of the subsets may be connected. An orthogonal matrix may be represented using Clay representation as: (i) the difference between the identity matrix and the training matrix, and (ii) the product of the inverse of the sum of the identity matrix and the training matrix. The training matrix may be skew-symmetric. The kernel matrix may be trained to define an elliptical inner product space by initializing the training matrix to the identity matrix and iteratively adjusting the values ​​of the training matrix.

[0131] In some examples, the orthogonal matrix can be an element of a Stiefel matrix manifold comprising a plurality of block orthogonal matrices. The kernel matrix can be trained to define an elliptical inner product space by Riemannian gradient descent, comprising determining a projection of a gradient from the Stiefel matrix manifold to a tangent space at an orthogonal matrix, adjusting a value of the orthogonal matrix based on the gradient projection, and determining a retraction of the orthogonal matrix as the adjustment into the Stiefel matrix.

[0132] In some examples, each respective feature vector of the first plurality of feature vectors may be associated with a corresponding pixel of the first image. Each respective feature vector of the second plurality of feature vectors may be associated with a corresponding pixel of the second image.

[0133] In some examples, for respective transformed feature vectors associated with corresponding pixels at a particular pixel location within the first image, a corresponding subset of the second plurality of feature vectors may be associated with a plurality of pixels within a search window within the second image that includes the particular pixel location.

[0134] In some examples, the first image and the second image may each represent an environment. Determining pixel correspondence between the first image and the second image may include determining, based on the cost volume and for each corresponding pixel in a plurality of pixels in the first image, a corresponding pixel in the second image, the corresponding pixel in the second image being determined to represent the same portion of the environment as the corresponding pixel.

[0135] In some examples, the first image may represent an environment at a first time, and the second image may represent an environment at a second time. Determining pixel correspondence between the first image and the second image may include determining, by one or more machine learning models, an optical flow between the first image and the second image based on a cost volume.

[0136] In some examples, the first image and the second image may be stereo images of the environment.Determining pixel correspondence between the first image and the second image may include determining, by one or more machine learning models, a depth image based on a cost volume.

[0137] In some examples, obtaining the first plurality of feature vectors and the second plurality of feature vectors may include processing the first image by one or more machine learning models to generate the first plurality of feature vectors, and processing the second image by one or more machine learning models to generate the second plurality of feature vector models.

[0138] In some examples, the first plurality of feature vectors and the second plurality of feature vectors may correspond to the first image and the second image of the first size ratio. A third plurality of feature vectors associated with the first image and a fourth plurality of feature vectors associated with the second image may be obtained. The third plurality of feature vectors and the fourth plurality of feature vectors may correspond to the first image and the second image of the second size ratio. The second size ratio may be greater than the first size ratio. The fourth plurality of feature vectors may be distorted based on the pixel correspondence between the first image and the second image of the first size ratio. The second plurality of transformed feature vectors may be generated by transforming each feature vector in the third plurality of feature vectors by a kernel matrix. The second cost body may be generated by determining a second plurality of inner products for each transformed feature vector of the second plurality of transformed feature vectors. Each corresponding inner product of the second plurality of inner products may be between a corresponding transformed feature vector of the second plurality of transformed feature vectors and a corresponding candidate feature vector of a corresponding subset of the distorted fourth plurality of feature vectors. Based on the second cost body, a second pixel correspondence between the first image and the second image of the second size ratio may be determined.

[0139] VII. Example Operation

[0140] Fig. 8A and 8B The test results of various optical flow models / methods performed on commonly used benchmark datasets are shown. Specifically, Fig. 8AResults of various supervised models / methods are shown for the Sintel benchmark dataset (both clean and final passes) and the KITTI 2015 benchmark dataset. More details of the Sintel benchmark dataset are provided in the paper titled “ANaturalistic Open Source Movie for Optical Flow Evaluation” written by Daniel J. Butler, Jonas Woolf, Garrett B. Stanley, and Michael J. Black and published by the 2012 European Conference on Computer Vision. Additional details of the KITTI 2015 benchmark dataset are provided in the paper titled “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite” written by Andreas Geiger, Philipp Lanz, and Raquel Artusson and published at the 2012 Conference on Computer Vision and Pattern Recognition.

[0141] Figure 8B The test results of various unsupervised models performed on the KITTI 2015 benchmark dataset are shown. Both benchmark datasets include a training subset (which can be reserved for model training) and a test subset (which can be reserved for model testing). Fig. 8A and 8B In each of , the results of the best performing model with respect to each metric are represented by the dark pattern of the corresponding table cell.

[0142] Fig. 8A The supervised models shown include FlowNet2, DCFlow, MirrorFlow, SpyNet, LiteFlowNet, PWC Net, PWC Net+, IRR-PWC, HD3, and VCN, each of which is described in detail in the corresponding published papers. For example, the details of the VCN model are provided in the paper titled "Volumetric Correspondence Networks for Optical Flow", written by Gengshan Yang and Deva Ramanan and published by the 2019 Conference on Neural Information Processing Systems. Adding "-ft" after the model name indicates that the version of the model has been additionally "fine-tuned" trained on the Sintel and / or KITTI 2015 benchmark datasets. Unless otherwise noted, Fig. 8AThe numbers reported in represent the average endpoint error (AEPE), which measures the average of the per-pixel differences between the estimated optical flow and the ground-truth optical flow over all image pixels. "Fl-all" represents the percentage of outliers (e.g., >5% endpoint error) averaged over all ground-truth pixels (e.g., background and foreground pixels). "(-)" indicates that a particular test result is not available.

[0143] Fig. 8A Each of the models reported in uses a Euclidean cost volume model or another predetermined (e.g., non-learnable or non-trainable) cost volume model. Therefore, the cost volume model in each of these supervised optical flow models can be replaced by a trainable / learnable cost volume model 400 to improve the performance of the supervised optical flow model. Therefore, Fig. 8A The bottom row of corresponds to the VCN model paired with the cost volume model 400 (i.e., VCN+LCV). In four of the seven tests (i.e., the Sintel final test and all KITTI 2015 tests), VCN+LCV scores higher than all other reported supervised optical flow models. In addition, as can be seen from the comparison of the bottom row with the top row, adding the cost volume model 400 to the VCN optical flow model improves the performance of the VCN optical flow model on all tests except the Sintel clean test.

[0144] Figure 8B The unsupervised models shown in include DSTFlow, GeoNet, UnFLow, DF Net, OccAwareFlow, Back2FutureFlow, SelFlow, and DDFlow, each of which is described in detail in the corresponding published papers. All metrics for AEPE and Fl are the same as defined above. "Fl-bg" indicates the average percentage of outliers on background true pixels (e.g., >5% endpoint error). "Fl-fg" indicates the average percentage of outliers on foreground true pixels (e.g., >5% endpoint error).

[0145] Figure 8B Each of the models reported in uses a Euclidean cost volume model or another predetermined (e.g., non-learnable or non-trainable) cost volume model. Therefore, the cost volume model in each of these unsupervised optical flow models can be replaced by a trainable / learnable cost volume model 400 to improve the performance of the unsupervised optical flow model. Therefore, Figure 8BThe bottom row of corresponds to the DDFlow model paired with the cost volume model 400 (i.e., DDFlow+LCV). In two of the four metrics (i.e., KITTI 2015 test Fl-fg and Fl-all), DDFlow+LCV scores higher than all other reported unsupervised optical flow models, including SelFlow, which is an improved version of DDFlow. In addition, as can be seen from the comparison of the bottom row with the upper rows, adding the cost volume model 400 to the DDFlow optical flow model improves the performance of the DDFlow optical flow model relative to all metrics.

[0146] Adding the cost volume model 400 to any other reported supervised and / or unsupervised optical flow models can further improve the performance of these optical flow models. For example, the cost volume model 400 may allow the optical flow model to operate more robustly under varying lighting conditions, in the presence of noise (e.g., natural or artificially added noise), and in the presence of artificially added adversarial patches. In some embodiments, an untrained version of the cost volume model 400 (e.g., where the kernel matrix W is initialized to the identity matrix) may be added to a pre-trained implementation of the optical flow model. The cost volume model 400 and the pre-trained optical flow model may then be further trained to fine-tune the cost volume model 400 to collaborate with the optical flow model. This further training may result in modifications to the cost volume model 400 and / or the pre-trained optical flow model, resulting in a system that determines a more accurate optical flow field than an unmodified pre-trained optical flow model operating without the cost volume model 400.

[0147] VIII. Conclusion

[0148] The present disclosure is not limited to the specific examples described in this application, which are intended to be illustrations of various aspects. Many modifications and changes can be made without departing from its scope, which will be apparent to those skilled in the art. In addition to those methods and devices described herein, those skilled in the art will clearly see from the foregoing description that functionally equivalent methods and devices within the scope of the present disclosure. These modifications and changes are intended to fall within the scope of the appended claims.

[0149] The above detailed description describes various features and operations of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the accompanying drawings, similar symbols generally identify similar components unless the context dictates otherwise. The examples described herein and in the accompanying drawings are not meant to be limiting. Other examples may be utilized and other changes may be made without departing from the scope of the subject matter presented herein. It will be readily understood that aspects of the present disclosure, as generally described herein and shown in the accompanying drawings, may be arranged, replaced, combined, separated, and designed in a variety of different configurations.

[0150] With respect to any or all message flow diagrams, scenarios, and flow charts discussed in the figures and herein, each step, block, and / or communication may represent information processing and / or information transmission according to an example. Alternative examples are included within the scope of these examples. In these alternative examples, for example, depending on the functionality involved, the operations described as steps, blocks, transmissions, communications, requests, responses, and / or messages may be performed in a different order than shown or discussed, including substantially concurrently or in reverse order. In addition, more or fewer blocks and / or operations may be used with any message flow diagram, scenario, and flow chart discussed herein. Figure 1 The message flow diagrams, scenarios and flow diagrams may be used together, and these message flow diagrams, scenarios and flow diagrams may be combined with each other in part or in whole.

[0151] The steps or blocks representing information processing may correspond to circuits that can be configured to perform specific logical functions of the methods or techniques described herein. Alternatively or additionally, the blocks representing information processing may correspond to a module, a segment, or a portion of a program code (including associated data). The program code may include one or more instructions executable by a processor for implementing specific logical operations or actions in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium (such as a storage device including a random access memory (RAM), a disk drive, a solid-state drive, or another storage medium).

[0152] Computer readable media may also include non-transitory computer readable media, such as computer readable media that store data for a short period of time, such as register memory, processor cache, and RAM. Computer readable media may also include non-transitory computer readable media that store program code and / or data for a longer period of time. Therefore, computer readable media may include, for example, auxiliary or persistent long-term storage such as read-only memory (ROM), optical or magnetic disks, solid-state drives, compact disk read-only memory (CD-ROM). Computer readable media may also be any other volatile or non-volatile storage system. Computer readable media may be considered as computer readable storage media, for example, or tangible storage devices.

[0153] In addition, the steps or blocks representing one or more information transfers may correspond to information transfers between software and / or hardware modules in the same physical device. However, other information transfers may be performed between software modules and / or hardware modules in different physical devices.

[0154] The specific arrangements shown in the figures should not be considered limiting. It should be understood that other examples may include more or less of each element shown in a given figure. In addition, some of the elements shown may be combined or omitted. In addition, examples may include elements not shown in the figures.

[0155] Although various aspects and examples have been disclosed herein, other aspects and examples will be apparent to those skilled in the art. The various aspects and examples disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

1. A computer-implemented method comprising: obtaining (i) a first plurality of feature vectors associated with the first image and (ii) a second plurality of feature vectors associated with the second image; generating a plurality of transformed feature vectors by transforming each corresponding feature vector in the first plurality of feature vectors using a kernel matrix trained to define an elliptical inner product space; generating a cost volume by determining a plurality of inner products for each respective transformed feature vector in the plurality of transformed feature vectors, wherein each respective inner product in the plurality of inner products is between a respective transformed feature vector and a respective candidate feature vector of a respective subset of the second plurality of feature vectors; as well as Pixel correspondence between the first image and the second image is determined based on the cost volume. 2 . The computer-implemented method of claim 1 , wherein the kernel matrix is ​​symmetric and positive definite.

3. The computer-implemented method of any one of claims 1-2, wherein the kernel matrix is ​​defined based on the product of (i) a diagonal matrix including a plurality of positive diagonal values ​​and (ii) an orthogonal matrix.

4. The computer-implemented method of claim 3, wherein the kernel matrix is ​​defined via eigendecomposition by determining (i) a first product between the transpose of the diagonal matrix and the orthogonal matrix, and (ii) a second product between the first product and the diagonal matrix.

5. The computer-implemented method of claim 3 , wherein each respective positive diagonal value of the plurality of positive diagonal values ​​of the diagonal matrix comprises a respective weight applied to a corresponding dimension of each respective eigenvector of the first plurality of eigenvectors, and wherein the orthogonal matrix is ​​configured to apply a rotation to each respective eigenvector of the first plurality of eigenvectors.

6. A computer-implemented method according to claim 3, wherein each corresponding positive diagonal value of the plurality of positive diagonal values ​​is represented as a function of a corresponding training parameter, wherein the function maps (i) a real value assigned to the corresponding training parameter to (ii) a positive real value, and wherein the kernel matrix is ​​trained to define the elliptical inner product space by iteratively adjusting the real value assigned to the corresponding training parameter for each corresponding positive diagonal value.

7. The computer-implemented method of claim 3, wherein the orthogonal matrix is ​​an element of a subset of a special set of orthogonal matrices, wherein the matrices of the subset are connected.

8. The computer-implemented method of claim 7, wherein the orthogonal matrix is ​​represented using Clay representation as the product of: (i) the difference between an identity matrix and a training matrix, and (ii) the inverse of the sum of the identity matrix and the training matrix, wherein the training matrix is ​​skew-symmetric, wherein the kernel matrix is ​​trained to define the elliptical inner product space by initializing the training matrix to the identity matrix and iteratively adjusting values ​​of the training matrix.

9. The computer-implemented method of claim 3, wherein the orthogonal matrix is ​​an element of a Stiefel matrix manifold comprising a plurality of block orthogonal matrices, and wherein the kernel matrix is ​​trained by Riemannian gradient descent to define the elliptic inner product space, the Riemannian gradient descent comprising: determining a gradient projection from the Stiefel matrix manifold to the tangent space at the orthogonal matrix; adjusting a value of the orthogonal matrix based on the gradient projection; as well as The retraction of the orthogonal matrix is ​​determined as an adjustment into the Stiefel matrix manifold.

10. The computer-implemented method of any of claims 1-2, wherein each respective feature vector in the first plurality of feature vectors is associated with a corresponding pixel of the first image, wherein each respective feature vector in the second plurality of feature vectors is associated with a corresponding pixel of the second image.

11. The computer-implemented method of claim 10, wherein for a respective transformed feature vector associated with a corresponding pixel located at a particular pixel location within the first image, a corresponding subset of the second plurality of feature vectors is associated with a plurality of pixels located within a search window that includes the particular pixel location within the second image.

12. The computer-implemented method of any of claims 1-2, wherein the first image and the second image each represent an environment, and wherein determining pixel correspondence between the first image and the second image comprises: Based on the cost volume and for each corresponding pixel in a plurality of pixels in the first image, a corresponding pixel in the second image is determined that is determined to represent the same portion of the environment as the corresponding pixel.

13. The computer-implemented method of claim 12, wherein the first image represents an environment at a first time, wherein the second image represents an environment at a second time, and wherein determining pixel correspondence between the first image and the second image comprises: An optical flow between the first image and the second image is determined based on the cost volume by one or more machine learning models.

14. The computer-implemented method of claim 12, wherein the first image and the second image are stereo images of the environment, and wherein determining pixel correspondence between the first image and the second image comprises: A depth image is determined based on the cost volume by one or more machine learning models.

15. The computer-implemented method of any one of claims 1-2, wherein obtaining the first plurality of feature vectors and the second plurality of feature vectors comprises: generating the first plurality of feature vectors by processing the first image through one or more machine learning models; as well as The second plurality of feature vectors are generated by processing the second image by the one or more machine learning models.

16. The computer-implemented method of any of claims 1-2, wherein the first plurality of feature vectors and the second plurality of feature vectors correspond to first and second images of a first size ratio, and wherein the method further comprises: obtaining (i) a third plurality of feature vectors associated with the first image and (ii) a fourth plurality of feature vectors associated with the second image, wherein the third plurality of feature vectors and the fourth plurality of feature vectors correspond to a second size ratio of the first image and the second image, wherein the second size ratio is greater than the first size ratio; warping the fourth plurality of feature vectors based on pixel correspondence between the first image and the second image at the first size ratio; generating a second plurality of transformed eigenvectors by transforming each corresponding eigenvector in the third plurality of eigenvectors by the kernel matrix; generating a second cost volume by determining a second plurality of inner products for each respective transformed feature vector in the second plurality of transformed feature vectors, wherein each respective inner product in the second plurality of inner products is between a respective transformed feature vector in the second plurality of transformed feature vectors and a respective candidate feature vector of a respective subset of the warped fourth plurality of feature vectors; as well as Based on the second cost volume, a second pixel correspondence between the first image and the second image at the second size ratio is determined.

17. A system comprising: processor; and A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1-16.

18. A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed by a computing system, cause the computing system to perform the method according to any one of claims 1-16.

Images