Efficient cost volume processing within iterative process

Abstract

A processor-implemented method includes processing a single level cost volume across a plurality of processing levels by changing a receptive field across each of the processing levels. The method further includes performing a learning-based correspondence estimation task based on the processing. The change may include processing the cost volume at different resolutions at each processing stage while maintaining the same neighborhood sampling radius. The resolution may increase from a first processing stage to a later processing stage. The change may also include changing a neighborhood sampling radius at each of the processing stages while maintaining the same resolution. The task may be optical flow estimation or stereo estimation.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application No. 18 / 358,857, filed on July 25, 2023, and entitled “EFFICIENT COST VOLUME PROCESSING WITHIN ITERATIVE PROCESS,” which claims the benefit of U.S. Provisional Patent Application No. 63 / 401,793, filed on August 29, 2022, and entitled “EFFICIENT COST VOLUME PROCESSING WITHIN ITERATIVE PROCESS,” the disclosures of which are expressly incorporated by reference in their entirety. Technical Field

[0003] The present disclosure relates generally to data processing, and more particularly to computationally efficient and memory efficient cost volume processing for iterative processes such as optical flow estimation and stereo estimation. Background Art

[0004] Many devices and systems allow for capturing a scene by generating an image (or frame) and / or video data (including multiple frames) of the scene. For example, a camera or a device including a camera can capture a sequence of frames of a scene (e.g., a video of the scene). In some cases, the sequence of frames can be processed for performing one or more functions, can be output for display, and can be output for processing and / or consumption by other devices, among other uses.

[0005] One common type of processing performed on a sequence of frames is motion estimation, which involves tracking the motion of an object or point across multiple frames. For example, motion estimation can include determining an optical flow map that describes the displacement of pixels in a frame relative to corresponding pixels in previous frames. Motion estimation can be used in a variety of applications, including computer vision systems, extended reality systems, data compression, image segmentation, and autonomous vehicle operation, among other applications. Summary of the invention

[0006] In some aspects of the disclosure, a processor-implemented method performed by at least one processor includes: processing, by the at least one processor, a single level cost volume across a number of processing stages by varying a receptive field of the cost volume processing across each of the processing stages. The processor-implemented method also includes: performing, by the at least one processor, a learning-based correspondence estimation task based on the processing.

[0007] Various aspects of the present disclosure relate to an apparatus. The apparatus has a memory and one or more processors coupled to the memory. The processor is configured to process a single level cost volume across a number of processing stages by changing the receptive field of the cost volume processing across each of the processing stages. The processor is further configured to perform a learning-based correspondence estimation task based on the processing.

[0008] Some aspects of the present disclosure relate to an apparatus. The apparatus includes: means for processing a single level cost volume across multiple processing stages by varying a receptive field of the cost volume processing across each of the multiple processing stages. The apparatus also includes: means for performing a learning-based correspondence estimation task based on the processing.

[0009] In some aspects of the present disclosure, a non-transitory computer readable medium having program code recorded thereon is disclosed. The program code is executed by a processor and includes: program code for processing a single level cost volume across multiple processing stages by changing the receptive field of the cost volume processing across each of the multiple processing stages. The program code also includes program code for performing a learning-based correspondence estimation task based on the processing.

[0010] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0011] The foregoing and other features and examples will become more apparent after reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Illustrative examples of the present application are described in detail below with reference to the following drawings.

[0013] Figure 1 is a block diagram illustrating an example optical flow estimation system according to some examples.

[0014] Figure 2 is an illustration of optical flow estimation for pixels according to some examples.

[0015] Figure 3A and Figure 3B is an illustration of a flow target graph and a flow search area according to some examples.

[0016] Figure 4A and Figure 4B is an illustration of a correlation volume according to some examples.

[0017] Figure 5is a block diagram illustrating an example optical flow estimation system using volumetric sampling according to some examples.

[0018] Fig. 6A , Figure 6B , Figure 6C and Fig.6D is an illustration of the cumulative distribution function of an example optical flow dataset according to some examples.

[0019] Fig. 7A , Figure 7B , Figure 7C and Fig.7D are illustrations of example optical flow search region shapes and initial estimated pixel locations according to some examples.

[0020] Fig. 8A is a block diagram illustrating an example architecture for image processing using coarse-to-fine cost volumes in accordance with various aspects of the present disclosure.

[0021] Figure 8B is a flow chart illustrating an example of a process for performing image-based processing in accordance with various aspects of the present disclosure.

[0022] Fig. 9 is a diagram illustrating an example of a deep learning neural network according to some examples.

[0023] Fig.10 is a diagram illustrating an example of a convolutional neural network according to some examples.

[0024] Fig.11 is a diagram illustrating an example of a system for implementing certain aspects described. DETAILED DESCRIPTION

[0025] Optical flow and stereo estimation are important components for cameras, automotive, extended reality (XR), and other Internet of Things (IoT) applications. Recent developments in deep learning-based solutions for optical flow and stereo estimation demonstrate significant performance gains over traditional methods. However, as part of the architecture, such methods adapt multiple cost volumes and process the cost volume multiple times. Cost volume-based methods are computationally expensive and require significant memory. Therefore, such solutions are not feasible for edge computing or low compute / power deployments.

[0026] According to aspects of the present disclosure, memory efficient processing of cost volumes is suitable for reducing peak memory. Instead of multiple levels of cost volumes, only a single level of cost volume is processed at each state or iteration. The novel processing scheme significantly reduces computation. In order to capture coarse and fine information with a single level of cost volume, the effective receptive field of the cost volume processing can be changed in two ways. First, while keeping the neighborhood sampling radius fixed, the resolution of the shared cost volume can be changed with coarser and finer resolutions for different levels or iterations. Alternatively, when processing multiple times, the resolution of the cost volume can be fixed for each iteration or level, and the neighborhood sampling radius is varied. In addition, non-uniform sampling can occur. Although the present description is primarily about image processing, the present disclosure is not limited thereto. For example, a wider range of learning-based correspondence estimation tasks such as LiDAR and RADAR processing can also be improved by the techniques of the present disclosure. In these cases, different initial levels of the neural network can be used.

[0027] Some aspects and examples of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and examples can be applied independently, and some of them can be applied in combination. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the subject matter of the present application. However, it will be apparent that each example can be implemented without these specific details. Each accompanying drawing and description are not intended to be restrictive.

[0028] The following description provides only illustrative examples and is not intended to limit the scope, applicability or configuration of the present disclosure. Specifically, the following description will provide a feasible description for implementing the illustrative examples to those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope of the present application as set forth in the appended claims. For example, although this description is primarily about optical flow estimation, the technology of the present disclosure is not limited thereto.

[0029] Motion estimation is the task of tracking the movement of one or more regions (e.g., an object or a portion of an object, an instance or a portion of an instance, a background portion of a scene or a portion of a background portion, etc.) across a sequence of frames. In some cases, motion estimation can be performed using optical flow estimation techniques. Optical flow estimation is the task of estimating pixel-level correspondences between frames of a video. For example, an optical flow estimation system may identify pixels of a frame (also referred to as a source frame) that correspond to a portion of a real-world object.

[0030] The optical flow estimation system may determine corresponding pixels (e.g., pixels depicting the same portion of a real-world object) within a subsequent frame (sometimes referred to as a target frame). The optical flow estimation system may estimate the motion of an object between frames (e.g., between a source frame and a target frame) by determining an optical flow vector corresponding to a displacement and / or distance between a pixel in a source frame and a corresponding pixel in a target frame. For example, an optical flow vector may indicate a displacement (e.g., corresponding to a direction and distance of movement) between coordinates corresponding to an initial pixel (from a source frame) and coordinates corresponding to a subsequent pixel (from a target frame).

[0031] In some cases, the optical flow estimation system may determine one or more optical flow maps. An optical flow map may include one or more optical flow vectors corresponding to the motion between two frames. In some examples, an optical flow map may include an optical flow vector for each pixel (or approximately each pixel) within a frame. An optical flow map that includes an optical flow vector for each pixel (or approximately each pixel) within a frame may be referred to as a dense optical flow map.

[0032] In some cases, a large amount of time and / or computing power may be required to generate dense optical flow maps, which may be detrimental to many applications of motion estimation. Examples of applications that utilize motion estimation include various computer vision tasks and camera applications in which object (pixel) motion is involved, including video recognition, autonomous driving, video compression, object and / or scene tracking, visual inertial odometry (VIO), video object segmentation, extended reality (e.g., virtual reality, augmented reality, and / or mixed reality), etc. It is desirable to perform higher performance optical flow estimation in chips and / or devices, which can result in increased accuracy, reduced computational complexity, reduced latency, reduced power consumption, and reduced memory usage, among other benefits.

[0033] As noted above, optical flow may involve the problem of estimating dense correspondences between a pair of frames or images. Existing solutions typically compute dense optical flow for an entire source frame or image (e.g., all pixels in the source frame or image), and search for correspondences within an entire target frame or image (e.g., by searching all pixels in the target frame or image). Methods that search for corresponding pixels within all pixels of a target frame may result in significantly large search areas and / or search volumes (e.g., correlation volumes), which in turn may result in unnecessarily high computational complexity, higher latency, higher power consumption, and / or higher memory requirements.

[0034] Some existing optical flow estimation systems perform optical flow estimation by searching for correspondences between each of the N pixels of the source frame within each of the N pixels of the target frame. The correlation between the N pixels of the source frame and the N pixels of the target frame can be achieved by forming a correlation volume containing N×N values. An example of such optical flow estimation using a correlation volume is described in “RAFT: Recurrent All-Pairs Field Transforms for Optical Flow” by Zachary Teed et al., 2020, which is hereby incorporated by reference in its entirety and for all purposes. According to the big O notation O(N 2 ), the amount of memory and computational effort required to perform this general optical flow estimation is proportional to the size of the input source frame. Big O notation represents the way in which the order of magnitude of the computational and memory requirements grows as the input size grows.

[0035] In order to quickly perform optical flow estimation, such as in real-time optical flow estimation, the correlation volume can be stored in a low-latency memory of a computing system (e.g., computing system 1100 described below). The lowest latency memory in the computing system, sometimes referred to as a tightly coupled memory (TCM), can be limited in size. For example, the TCM can be included on the same silicon die or in the same package as a processor of the computing system, thereby limiting the available size that can be used for the TCM. In some cases, when the size of the correlation volume exceeds the size of the TCM, some or all of the correlation volume can be stored in a slower memory, such as a memory that can be accessed by the processor via direct memory access (DMA). Therefore, techniques that can reduce the size of the correlation volume can be desirable, which can achieve faster optical flow estimation in systems with limited available TCM.

[0036] Systems, apparatus, methods, and computer-readable media (collectively, "systems and techniques") are described for performing optical flow estimation for frames using volumetric sampling. Frames of a video may also be referred to as images. In some cases, the optical flow estimation system may determine a subset of pixels of a target frame (referred to as a flow search region) for performing optical flow estimation for each pixel of a source frame. In some cases, the optical flow estimation system may generate a dense optical flow map based on searching for correspondences of pixels of the source frame within the flow search region of the target frame (e.g., by including pixels of the flow search region in a correlation volume).

[0037] The optical flow estimation system can perform optical flow estimation in a flow search area of ​​a target frame, which can contain significantly fewer pixels than a conventional full-frame optical flow estimation. The optical flow estimation system can generate an optical flow map with reduced latency and / or fewer computing resources. For example, the optical flow estimation system can cause optical flow estimation to be performed in a flow search area that includes 20% of the pixels of the target frame, a flow search area that includes 10% of the pixels of the target frame, a flow search area that includes 5% of the pixels of the target frame, a flow search area that includes 2.5% of the pixels of the target frame, or a flow search area that includes any other number or percentage of pixels based on the described technology.

[0038] In some examples, the size of the flow search area in the target frame may be set to a constant size based on characteristics of the video data being analyzed, physics-based motion constraints, combinations thereof, and / or other factors. The optical flow estimation system may search for correspondences for each of the N pixels of the source frame within a constant C pixels of the flow search area in the target frame. The correlation between the N pixels of the source frame and the C pixels per source volume may be achieved by forming a correlation volume containing N×C values. In the following description, this technique may be referred to as volume sampling. According to the big O notation O(N), the amount of memory and computational effort used to perform optical flow estimation within the constant flow search area is proportional to the size of the input source frame. As the number of pixels N of the source frame grows, the flow search area size may remain constant, in which case the growth no longer scales quadratically, but is linear with respect to N. The described volume sampling technique may be used with any technique involving the use of a correlation volume (or cost volume) to perform correspondence estimation, such as semantic matching estimation, geometric matching estimation, optical flow, scene flow, or any other calculation employing similar techniques.

[0039] In some aspects, the optical flow estimation system may determine the size of the flow search area based on the characteristics of the video data being analyzed by the optical flow estimation system. In some examples, the characteristics of the video data may be determined based on the cumulative distribution function (CDF) of the optical flow vector values ​​generated by the optical flow estimation system. In some examples, the CDF may be generated based on a representative data set including data similar to the optical flow estimation task that the optical flow estimation system is designed to solve. For example, an optical flow estimation system that can be configured to analyze vehicle traffic data may set the size of its flow search area based on the CDF of a data set containing a video of vehicle traffic. In an illustrative example, the flow search area may be sized so that 98% of the optical flow vectors in each direction of motion fall within the search area. For example, if 98% of the optical flow vectors in a given data set indicate horizontal motion between three pixels to the left and two pixels to the right, the width of the flow search area may be set to five. In another example, if 98% of the optical flow vectors in a given data set indicate vertical motion between two pixels up and eight pixels down, the height of the flow search area may be set to ten. In some cases, the size of the flow search area may be determined by a machine learning system and / or algorithm based on a neural network (e.g., a deep neural network).

[0040] In some aspects, the optical flow estimation system may determine a flow target map. The flow target map may provide an initial estimated location of a pixel in a target frame. In some cases, the initial estimated location of the pixel may include within a flow search region. In some examples, the initial estimated pixel location may be centered within the corresponding flow search region. In some aspects, the initial estimated pixel location may be located eccentrically relative to one or more dimensions of the flow search region. In some aspects, the optical flow estimation system may determine the flow target map based on optical flow history information. In some specific implementations, the optical flow history information may include information about a previous optical flow map calculated by the optical flow estimation system. In some specific implementations, the optical flow history information may include information about an object detected in a frame corresponding to the optical flow map, so that the optical flow map vector may be related to the motion of a particular object.

[0041] By providing the optical flow estimation system with a good initial estimate of the location of the source pixel in the target frame, the accuracy of the optical flow estimation can be improved. In some cases, having an initial estimate for the direction of movement of the pixel means that the optical flow estimation system can utilize a smaller flow search area. As described in more detail below, a neural network-based machine learning system and / or algorithm (e.g., a deep neural network) can determine a flow search area within a flow target map and / or a target frame.

[0042] Additional details regarding a system for optical flow estimation using volume sampling are provided with respect to the various figures. Figure 11 is a diagram illustrating an example of an optical flow estimation system 100 that may perform an optical flow estimation process. The optical flow estimation system 100 may have various components, including a flow target engine 102, a search area engine 104, and an optical flow map engine 106. The components of the optical flow estimation system 100 may include electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), or other suitable electronic circuits), computer software, firmware, or any combination thereof for performing the various operations described. Although the optical flow estimation system 100 is shown as including certain components, one of ordinary skill will appreciate that the optical flow estimation system 100 may include more than one component. Figure 1 For example, in some cases, the optical flow estimation system 100 may also include Figure 1 One or more memories (e.g., RAM, ROM, cache, buffer, etc.) and / or processing devices not shown.

[0043] The optical flow estimation system 100 can be a computing device or a part of multiple computing devices. In some cases, one or more computing devices including the optical flow estimation system 100 may also include one or more wireless transceivers for wireless communication and / or a display for displaying one or more frames or images. In some examples, the computing device including the optical flow estimation system 100 can be an electronic device, such as a camera (e.g., a digital camera, an IP camera, a video camera, a camera phone, a video phone, or other suitable capture device), a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, an extended reality device (e.g., a virtual reality device, an augmented reality device, and / or a mixed reality device), a vehicle or a computing device or component of a vehicle, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device, or any other suitable electronic device.

[0044] The optical flow estimation system 100 may receive, access, or obtain frames 103 as input. In some examples, the optical flow estimation system 100 may perform an optical flow estimation process in response to one or more of the frames 103 being captured by a camera or a computing device (e.g., a mobile device, etc.) including a camera. Frames 103 may include a single frame or multiple frames. In some cases, frames 103 may include video frames of a video sequence or a static image in a set of continuously captured static images. In an illustrative example, a set of continuously captured static images may be captured and displayed to a user as a preview of a scene in the field of view of the camera, which may help the user decide when to provide input to cause an image to be captured for storage. In another illustrative example, a burst mode or other similar mode that captures multiple continuous images may be used to capture a set of continuously captured static images. The frames may be: red-green-blue (RGB) frames having red, green, and blue components per pixel; luma, chroma red, chroma blue (YCbCr) frames having a luma component and two chroma (color) components (chroma red and chroma blue) per pixel; or any other suitable type of color or monochrome picture.

[0045] In some examples, optical flow estimation system 100 may capture frame 103. In some examples, optical flow estimation system 100 may receive, access, or obtain frame 103 from a frame source. In some cases, the frame source may include one or more image capture devices and / or one or more video capture devices (e.g., a digital camera, a digital video camera, a phone with a camera, a tablet with a camera, or other suitable capture device), an image and / or video storage device (including any suitable storage device, such as described below), or a video storage device. Fig.11 1130), an image and / or video archive containing stored images, an image and / or video server or content provider providing image and / or video data, an image and / or video feed interface receiving images from a video server, streaming source or other content provider, a computer graphics system for generating computer graphics image and / or video data, a combination of such sources, or other sources of image frame content. In some cases, multiple frame sources may provide frames to the optical flow estimation system 100.

[0046] In some implementations, the optical flow estimation system 100 and the frame source may be part of the same computing device. For example, in some cases, a camera, phone, tablet, XR device, and / or other device with a frame source or image source (e.g., camera, storage device, etc.) may include an integrated optical flow estimation system. In some implementations, the optical flow estimation system 100 and the frame source may be part of separate computing devices. In an illustrative example, the frame source may include one or more cameras, and the computing device including the optical flow estimation system 100 may include a mobile or fixed telephone handset, a desktop computer, a laptop or notebook computer, a tablet computer, or other computing device.

[0047] In some examples, the optical flow estimation performed by the optical flow estimation system 100 may be performed using a single camera system of a computing device. In other examples, the optical flow estimation performed by the optical flow estimation system 100 may be performed using a dual camera system of a computing device. In some cases, more than two cameras may be used in a camera system to perform optical flow estimation.

[0048] The optical flow estimation system 100 may process the frames 103 to generate an optical flow map (e.g., optical flow map 108) by performing optical flow estimation on pixels within a pair of frames in the frames 103. The optical flow map 108 may include one or more optical flow vectors corresponding to the movement of features and / or pixels between the two frames. In some cases, the two frames may be directly adjacent frames or consecutive frames within a series of frames. In some cases, the two frames may be separated by one or more intermediate frames (which may be referred to as non-adjacent frames). The frame between the two frames that occurs earlier in the series of frames may be referred to as a source frame, and the frame that occurs later in the series of frames may be referred to as a target frame.

[0049] In some examples, the optical flow estimation system 100 may receive, access, or obtain an optical flow history 110 as input. In some cases, the optical flow history 110 may include and / or be based on an optical flow map 108 determined by the optical flow estimation system 100 in one or more previous iterations of optical flow estimation. The optical flow history 110 may also include and / or be based on features detected during optical flow estimation, such as features extracted during object detection, semantic segmentation, etc. Any data included in the optical flow history 110 other than the optical flow map 108 data from a previous frame will be referred to as supplemental history data 112. For example, the optical flow history 110 may include and / or be based on features detected during optical flow estimation, such as features extracted during object detection, semantic segmentation, etc. Fig.10 8 may indicate that the pixel at pixel position L1 has an optical flow value (e.g., an optical flow vector) V1. Supplemental history data 112 included in optical flow history 110 may include data that classifies pixel P1 as part of a particular foreground object, such as a ball. In some cases, supplemental history data 112 may be output by optical flow estimation system 100 in addition to optical flow map 108.

[0050] In some cases, optical flow history 110 may be uninitialized, meaning that the optical flow history may not contain historical optical flow information at times, such as when optical flow estimation system 100 has not yet processed any of frames 103. In some examples, optical flow history 110 may include the most recently determined optical flow map 108. In some cases, the most recently determined optical flow map 108 may correspond to a most recently processed pair of frames in frames 103. In this case, the source frame of the current optical flow estimation may be the same as the target frame from a previous optical flow estimation performed by optical flow estimation system 100. In some cases, there may be a gap in time between the most recently determined optical flow map 108 and the frame in frames 103 currently being processed by optical flow estimation system 100. In some cases, optical flow history 110 may include optical flow maps 108 (and in some cases, supplemental history data 112) from optical flow estimation performed on frames two or more frames prior to the current target frame being processed by optical flow estimation system 100.

[0051] In some examples, the optical flow history 110 may include information from multiple frames preceding the target frame being processed by the optical flow estimation system 100. For example, the optical flow history 110 may include information from at least two frames preceding the current frame being processed by the optical flow estimation system 100. In some examples, the optical flow history 110 may include optical flow maps 108 from multiple previous frames processed by the optical flow estimation system 100. In such cases, the optical flow history 110 may also include supplemental history data 112 corresponding to each of the previous frames. In some cases, the optical flow history 110 may include one or more optical flow maps 108 without the supplemental history data 112.

[0052] In such cases, the flow target engine 102 may apply the available information in the optical flow history 110 to determine the initial estimated pixel location in the target frame. For example, if the optical flow map 108 is available to the flow target engine 102, the flow target engine 102 may determine the initial estimated pixel location in the target frame by offsetting each pixel location in the source frame by the flow vector from the optical flow map 108. For example, the optical flow map 108 may indicate that a pixel at position L1 in a previous source frame is moved to position L2 in a previous target frame. In this case, the flow target engine 102 may use position L2 as the initial estimated pixel location in the target frame. When there is no available optical flow history 110 information, the flow target engine 102 may not be able to provide the initial estimated pixel location offset. In this case, the flow target engine 102 may use the original pixel location in the source frame as the initial estimated location in the target frame. For example, a pixel at position L1 in the source frame may have an initial estimated pixel location L1 in the target frame.

[0053] In some examples, the flow target engine 102 may combine the supplemental history data 112 with the optical flow map 108 to determine an initial estimate. For example, the optical flow map 108 may indicate that the pixel P3 moves from a position L3 in a previous source frame to a position L4 in a previous target frame. In some implementations, the displacement vector of the movement from L3 to L4 may be represented by an optical flow vector at position L3 of the optical flow map 108.

[0054] The supplemental history data 112 may indicate that a pixel at position L3 in a previous source frame is part of a ball that moved from point A to point B across multiple frames. In this case, the flow target engine 102 may perform object detection or semantic segmentation and determine that a pixel at position L4 in the current source frame is part of the same ball. In this case, the pixel belonging to the ball may have moved from position L3 to L4 across a previous pair of frames, so the target map engine knows the last position of the pixel belonging to the ball. The flow target engine 102 may provide an initial estimate that the pixel at position L4 in the current source frame will move to position L5 in the current target frame, where the offset between position L4 and position L5 is the same as the offset between position L3 and L4 in the previous iteration of the optical flow estimation system 100.

[0055] The flow target engine 102 may provide an initial estimate of where features and / or pixels in the source frame will appear in the target frame being processed by the optical flow estimation system 100. In some cases, the flow target engine 102 may access a plurality of previous optical flow maps 108 corresponding to a plurality of previous frames processed by the optical flow estimation system 100 (e.g., via the optical flow history 110). For example, the flow target engine 102 may classify and / or label objects in the frame (e.g., using any type or form of object recognition technology, such as using one or more classification neural networks). Based on the classification and / or labeling, the flow target engine 102 may determine pixels corresponding to an object (e.g., a person's face, a ball, a portion of a vehicle, etc.).

[0056] In some implementations, the flow target engine 102 can utilize the optical flow history 110 information from multiple frames to determine the motion of an object across multiple frames to determine the trajectory of the detected object. For example, the detected object can be a ball thrown across frames. Between the first frame and the second frame, the ball can have a trajectory captured in the optical flow. Fig.10 8 and combined with the speed and / or displacement in the optical flow history 110. In some cases, the ball may also be detected in the third frame, and the optical flow map 108 generated between the second frame and the third frame may indicate that the speed of the ball has slowed down.

[0057] In some cases, flow target engine 102 may utilize information from optical flow history 110 from multiple previous frames (e.g., two or more optical flow maps 108) to determine the estimated trajectory of the ball. In some examples, flow target engine 102 may include a Kalman filter that determines the estimated trajectory of pixels and / or objects. In some cases, the Kalman filter may be used in conjunction with object recognition to determine the estimated trajectory of pixels and / or objects. The calculation of object motion across multiple frames of optical flow history 110 is sometimes referred to as sequential flow consistency or sequential flow consistency calculation.

[0058] In some cases, the optical flow history 110 of multiple frames may be stored in one or more memories (e.g., a circular memory) that stores optical flow history information for the most recent Y frames, where Y is an integer, such as 3 frames, 5 frames, etc. In some cases, the flow target engine 102 may adaptively change the number of frames of the optical flow history 110 stored in the one or more memories (e.g., a circular memory). For example, if the flow target engine 102 is using 5 frames of the optical flow history 110 to calculate sequential flow consistency and determines that the motion in a particular sequence of frames 103 follows a linear consistent pattern, the flow target engine 102 may reduce the number of frames of the optical flow history 110 stored in the one or more memories (e.g., a circular memory). For example, the number of frames of the optical flow history 110 may be reduced to 3. In another example, the flow target engine 102 may only utilize a single frame of the optical flow history 110. For example, the flow target engine 102 may apply the optical flow vectors stored in the optical flow map 108 of the previous frame to the pixels of the current source frame. If the optical flow estimation system 100 and / or the flow target engine 102 determines that the accuracy of the optical flow estimation is falling below an accuracy threshold, the flow target engine 102 may increase the number of frames of the optical flow history 110 stored in one or more memories (e.g., a loop memory) and used for sequential flow consistency calculations.

[0059] In some cases, the streaming target engine 102 may use a machine learning system and / or algorithm to determine the initial estimated pixel and / or feature locations in the target frame. For example, the machine learning system and / or algorithm may be any type or form of deep neural network (DNN). In one illustrative example, the machine learning algorithm may include a visual geometry group (VGG) algorithm. In another illustrative example, the machine learning system and / or algorithm may include a residual neural network (ResNet). Any other machine learning system and / or algorithm may be used.

[0060] In some cases, the neural network may be trained to determine features of objects within a frame. Features may include object labels, object classifications, object boundaries, and other features. In some cases, the neural network may be trained by inputting many frames or images with known object features into the neural network. In some cases, the neural network may also be trained by inputting many frames or images with known motion characteristics into the neural network to determine previous motion associated with the determined features. After the neural network has been sufficiently trained, the neural network may determine features and / or motion associated with the determined features of a new frame (e.g., frame 103) that is input to the neural network during inference.

[0061] In some implementations, search area engine 104 may determine a flow search area for searching each pixel of the source frame in the target frame. In some examples, search area engine 104 may determine a search area size sufficient to obtain an accurate optical flow estimate. In some examples, search area engine 104 may combine each of the flow search areas in the target frame into a correlation volume.

[0062] The size of the flow search area applied by the search area engine 104 may be applied based on characteristics of the data being processed by the optical flow estimation system 100. For example, when the optical flow estimation system 100 is processing frames 103 from a cartoon animation, the motion of objects may not be constrained by physics and a relatively large search area may be required.

[0063] As described above, the size of the flow search area can be adapted according to the characteristics of the data set being analyzed by the optical flow estimation system. In some cases, the size of the flow search area can be adapted based on the following information about Table 1, Table 2, Fig. 6A , Figure 6B , Figure 6C and Fig.6D The size of the flow search region selected by the search region engine 104 may determine the computational and memory requirements for performing optical flow estimation.

[0064] Thus, the search area engine 104 may also determine the size of the flow search based at least in part on available computing and memory resources (e.g., the amount of TCM) and processing speed requirements for optical flow estimation (e.g., whether optical flow estimation needs to occur in real time). In some cases, a neural network-based machine learning system and / or algorithm (e.g., a deep neural network) may be used to determine the size of the flow search area.

[0065] In some examples, the confidence level associated with the initial estimated pixel position of the flow target engine 102 can be adjusted based on the age of the optical flow history 110. In some aspects, the age of the optical flow history 110 can correspond to the number of frames 103 between the current target frame being processed by the optical flow estimation system 100 and the previous target frame corresponding to the optical flow history 110. For example, a lower confidence level can be applied when the optical flow history 110 is based on a target frame that is thirty frames prior to the current target frame than when the optical flow history 110 is based on the immediately preceding frame.

[0066] In some cases, as the confidence level decreases, a lower weight may be applied to the optical flow history 110. In such cases, the size of the flow search area applied by the search area engine 104 may be dynamically increased to compensate for the lower confidence level. In some cases, the size of the flow search area applied by the search area engine 104 may be dynamically reduced when a higher confidence level exists for a frame (or entry) stored in the optical flow history 110. For example, as more optical flow history information becomes available and / or in the case where the optical flow history information is from one or more frames immediately preceding the current frame being processed, the optical flow estimation system 100 may utilize a smaller flow search area. In some cases, the initial estimated pixel locations determined by the flow target engine 102 may be stored in the optical flow map 108. In some cases, the optical flow estimation system 100 may determine weights or filters (utilizing coefficients) to weight among the available frames (or entries) in the optical flow history 110. For example, the weighting may include a finite impulse response filter, an infinite impulse response filter, a monotonically decreasing weight based on time distance, equal weights (boxcar), any combination thereof, and / or other weighting.

[0067] The optical flow map engine 106 of the optical flow estimation system 100 may determine an optical flow vector corresponding to a pixel between a source frame and a target frame of the frame 103. In some cases, the optical flow vector may indicate a direction and magnitude of movement of the pixel. For example, the optical flow vector may describe a displacement between a coordinate corresponding to a position of the pixel within an initial frame (e.g., a source frame) and a coordinate corresponding to a position of the pixel within a subsequent frame (e.g., a target frame). The optical flow map engine 106 may use any type or form of optical flow estimation technique to determine the position of the pixel within the target frame.

[0068] In some cases, the optical flow map engine 106 may determine an optical flow vector between each pixel in the source frame and a pixel of the target frame based on the flow search area determined by the search area engine 104. For example, the optical flow map engine 106 may perform all pair correlations within a correlation volume generated by the search area engine 104. Additionally or alternatively, the optical flow map engine 106 may use any suitable machine learning algorithm (e.g., a deep neural network) to determine the optical flow vector. In some cases, the machine learning algorithm used to determine the optical flow vector may be different from the machine learning algorithm used to select the initial estimated pixel location (e.g., the flow target engine 102).

[0069] In some cases, all pair correlations may determine which features in the target frame correspond to features in the source frame. The optical flow map engine 106 may generate values ​​of the optical flow map 108 for the target frame based on the optical flow vectors. In some examples, the optical flow map engine 106 may generate an incremental optical flow map corresponding to a motion estimate between two adjacent frames. In other examples, the optical flow map engine 106 may generate a cumulative optical flow map (in which case the optical flow map 108 is adjusted or updated at each frame) corresponding to a motion estimate between two frames having one or more intermediate frames therebetween. For example, the optical flow map engine 106 may determine an incremental optical flow map between all or a portion of directly adjacent frames within a series of frames.

[0070] The optical flow map engine 106 may use the incremental optical flow map to update the cumulative optical flow map between the first frame in the series of frames and the current frame in the series of frames. To update the cumulative optical flow map, the optical flow map engine 106 may add the incremental optical flow vector between the current frame and the previous frame to the corresponding optical flow vector of the cumulative optical flow map. In some cases, the optical flow map engine 106 may use the incremental optical flow map to update the cumulative optical flow map between multiple iterations of optical flow estimation between two adjacent frames.

[0071] The optical flow map 108 output by the optical flow map engine 106 can be used for various purposes and / or tasks. For example, as mentioned above, the optical flow map (e.g., 108) can be used in applications including: computer vision systems, extended reality systems, data compression, image segmentation, autonomous vehicle operation, and other applications.

[0072] Figure 2 This is an example that can be Figure 1 1 is a diagram of an example of an optical flow estimation process performed by the optical flow estimation system 100. Figure 2 An example of a first frame 201 (also referred to as a source frame) and a second frame 203 (also referred to as a target frame) in a frame sequence is illustrated. Frames 201 and 203 may correspond to Figure 1 Frame 201 is shown as having dimensions of W pixels wide by H pixels high (expressed as W×H). A person of ordinary skill will appreciate that first frame 201 may include Figure 2 The number of different positions than those illustrated (e.g., more pixel positions may be included). For example, the first frame 201 may include a 4K (or ultra high definition (UHD)) frame at a resolution of 3,840×2,160 pixels, an HD frame at a resolution of 1,920×1,080 pixels, or any other suitable frame having another resolution. Frame 201 includes pixels P1, P2, P3, P4, P5, P6, and P7. As shown, pixel P1 has a position 202A. Pixel position 202A may include a (W, H) pixel position (4, 1) relative to the upper leftmost pixel position (0, 0). In an illustrative example, pixels P1, P2, and P3 may correspond to the tip of a person's nose, and pixel P5 may correspond to a boundary between the person's face and the background of the frame.

[0073] The second frame 203 has corresponding pixel positions that are the same as the pixel positions of the first frame 201 (having dimensions W×H), such as the upper left pixel in the first frame 201 (at pixel location or position (0,0)) corresponds to the upper left pixel in the second frame 203 (at pixel location or position (0,0)). As shown, pixel P1 has moved from pixel location 202A in the first frame 201 to an updated pixel P1 location 202B in the second frame 203. The original position 202C of pixel P1 from the first frame 201 is shown in the second frame 203 to allow easy visualization of the change in position of pixel P1 between the first frame 201 and the second frame 203. It should be understood that the pixel position 202C in the second frame 203 can be a position in the second frame 203 occupied by a different pixel from the first frame 201 (e.g., P2, P3, P4, P5, P6, P7, or any other pixel from the first frame 201). In some cases, pixel position 202C may be occupied by data that is not present in any pixel in first frame 201 (eg, a new object entering the frame).

[0074] The updated pixel P1 position 202B may include a (W, H) pixel position (5, 2) relative to the top leftmost pixel position (0, 0). An optical flow vector may be calculated for pixel P1, indicating a velocity, displacement, or optical flow of pixel P1 from the first frame 201 to the second frame 203. In an illustrative example, the optical flow vector of pixel P1 between frames 201 and 203 (shown by vector 205) is (1, 1), indicating that pixel P1 has moved one pixel position to the right and one pixel position downward. In some cases, the optical flow estimation system 100 may determine a total of W×H optical flow vectors, which may represent one optical flow vector for each pixel position in the source frame, including determining optical flow vectors for the remaining pixels P2-P7 (not shown for simplicity). The W×H optical flow vectors may be stored in an optical flow map, such as Figure 1As described above and below, the computational effort and memory requirements for determining the W×H optical flow vectors of the optical flow map 108 may depend on the size of the search area (and in some cases, the resulting search volume) used to locate each pixel in the target frame.

[0075] Figure 3A is an example that can be used in optical flow estimation systems such as Figure 1 FIG. 1 is a diagram of an example of a flow target map 316 and flow search areas 326 and 328 utilized in the optical flow estimation system 100 . Figure 3A Two frames are illustrated: a source frame 310 and a target frame 322. The source frame 310 and the target frame 322 can have dimensions of width=W pixels and height=H pixels, with a total of W×H pixels in each frame. In some implementations, an optical flow estimation system (e.g., 100) can perform optical flow estimation between the source frame 310 and the target frame 322. In some implementations, the flow target map 316 can include initial optical flow estimates (e.g., offset vectors) that can represent the estimated speed or movement of each pixel in the source frame 310 into the target frame 322. In some examples, as described above, the initial optical flow estimates in the flow target map 316 can be based on an optical flow history, such as Figure 1 The optical flow history 110 of the source frame 310 is obtained by searching the target frame 322 for each pixel of the source frame 310 . The flow target map 316 may have the same dimensions as the source frame 310 and the target frame 322 , such as W×H values. During optical flow estimation, the optical flow estimation system 100 may search the target frame 322 for each pixel of the source frame 310 .

[0076] exist Figure 3A In the example of , the initial estimated pixel position 324 in the target frame 322 may correspond to the pixel 312 of the source frame 310, which is offset by the corresponding value 318 in the flow target map 316. In addition, the initial estimated pixel position 324 of the target frame 322 is included in the flow search area 326. The optical flow estimation system according to the present disclosure can use the flow search area 326 to determine the optical flow of the pixel 312 between the source frame 310 and the target frame 322. The flow search area 326 can be a rectangle with a height = h pixels and a width = w pixels, and can contain a smaller number of pixels than the full target frame 322. In some cases, without departing from the scope of the present disclosure, the described flow search areas (including the flow search areas 326 and 328) can be shapes other than rectangles.

[0077] Similarly, the initial estimated pixel position 330 of the target frame 322 may correspond to the pixel 314 of the source frame 310, which is offset by the corresponding value 320 in the flow target map 316. As illustrated, the pixel position 330 of the target frame 322 is included in the flow search area 328. Figure 3AIt should be understood that each pixel in the source frame 310 can be offset by a corresponding value in the flow target map 316 to determine an initial estimated location of each pixel in the target frame 322. Each pixel in the target frame 322 can also have a corresponding flow search area in the target frame 322 that includes its initial estimated location. In some specific implementations, the search areas in the target frame 322 corresponding to each pixel of the source frame 310 can be cascaded to form a correlation volume 332. In some examples, by having a flow search area that is less than the full frame for each pixel, the total size of the correlation volume can be less than a correlation volume that includes a full frame of pixels (e.g., corresponding to W×H pixels of the target frame 322) for each pixel in the source frame 310. This reduction in the correlation volume can reduce the required size of the TCM and reduce the total computational work required to perform optical flow estimation between the source frame 310 and the target frame 322 from O(N×H). 2 ) scales to O(N) scale, as described above.

[0078] Although the two-dimensional source frame 310 and the target frame 322 are discussed in terms of pixels, Figure 3A Although the above description and other examples provided by the invention provide a method for extracting features, the same techniques can be applied more generally to features. In one example, image pixels of the source frame 310 and the target frame 322 can be considered as features. In another case, features can be extracted from the source frame 310 and the target frame, as described above with respect to Figure 1 The flow target engine 102 described above. In addition, the described techniques are not limited to two-dimensional feature maps. The use of initial flow estimates in the flow target map 316 and the flow search area can be more generally applied to the source feature map F 0 and the target feature map F 1 In some respects, f 0 It can be F 0 The feature points in in is the coordinate of the feature point in the D-dimensional feature map, and the coordinate value belongs to the set of integers Z. Similarly, f 1 It can be F 1 The feature points in The feature map can have shape S 1 ×S 2 ×…×S D .

[0079] Figure 3A One illustrative example shape described for a two-dimensional feature map is a rectangle with dimensions H×W. For example, the source feature map F 0 It can represent a feature map of dimension H×W, where F 0 The features in the target feature map F may be original image pixel values, or may be features extracted from the source frame 310 using object detection, semantic segmentation, or any other suitable feature extraction technique.1 It can represent a feature map of dimension H×W, where F 1 The features in may be original image pixel values, or may be features extracted from the target frame 322 through object detection, semantic segmentation, or any other suitable feature extraction technique of the target frame 322.

[0080] The optical flow estimation system 100 may determine F 0 The feature f in 0 and F 1 The feature f in 1 The corresponding relationship between them. 0 and f 1 The estimated displacement between 1 , Δ 2 , ..., Δ D )∈R D Δ can be F 0 and F 1 Any value representable in such that For example, a pixel may be moved from the upper left corner of the source frame to the lower right corner of the target frame, resulting in Δ=(S 1 , S 2 ).

[0081] The flow target map F includes the initial estimated spatial displacement values ​​of the optical flow estimation f，T Can be S 1 ×S 2 ×…×S D In some cases, if there is no optical flow history available to the optical flow estimation system 100, the flow target map may not include initial displacement values, so that F f，T = 0. In some cases, when there is a single previous optical flow estimate Flow available for the optical flow estimation system 100 Prev In the case of , the initial estimated displacement value can be set equal to the previous optical flow estimate, so that F f，T =Flow Prev In some cases, Flow Prev It can also be S 1 ×S 2 ×…×S D In one illustrative example, Flow Prev It may be the most recent optical flow map of dimension H×W generated by the optical flow estimation system 100 .

[0082] In some cases, the optical flow history 110 may include optical flow estimates, semantic segmentation results, and / or object detection results (e.g., supplementary historical information) for multiple previous frames. In such cases, the optical flow estimation system 100 may derive an object motion model (also referred to as sequential flow consistency) based on multiple previous frames of the optical flow history. In some cases, a machine learning model system and / or algorithm based on a neural network (e.g., a deep neural network) may be used to determine the trajectory of one or more objects based on the history of observations. In some cases, a Kalman filter may be used to determine the trajectory of one or more objects based on the optical flow history of multiple previous frames. In one example, a sequence of frames in a video may include a ball moving from point A to point B. The ball may slow down or decrease in height, for example, due to gravity. The optical flow estimation system 100 (e.g., using the flow target engine 102) may generate an estimated trajectory of the ball and estimate the possible positioning of the ball in the target frame. In some cases, the optical flow estimation system 100 may use the estimated trajectory (also referred to as sequential flow consistency) to determine the initial estimated spatial displacement values ​​of the pixels and / or features of the ball.

[0083] In some cases, the sequential flow consistency term can be expressed as the value of the previous optical flow map Flow Prev In the case where multiple frames of optical flow history are available, the initial estimated displacement value can be set equal to the previous optical flow estimate offset sequence flow consistency term ΔFlow SFC , so that F f，T =Flow Prev +ΔFlow SFC As mentioned above, ΔFlow SFC The sequential flow consistency term may be derived based on the temporal / sequential consistency of the motion of the object across multiple frames that may be derived from the optical flow history 110. In some cases, the sequential flow consistency offset may be calculated for all features. In some cases, the sequential flow consistency term may be calculated only for features that exhibit strong sequential flow consistency characteristics. For example, the sequential flow consistency term may be calculated only for foreground objects (such as the ball described above). In such cases, the value of the sequential flow consistency vector may be set to zero for pixels and / or features that do not have a corresponding sequential flow consistency calculation.

[0084] Figure 3B Another illustration of the relationship between pixels 312 and 314 (respectively) of source frame 310 and corresponding initial estimated pixel locations 338 and 344 in target frame 322 and flow search areas 340 and 346 is provided. Figure 3B As illustrated, pixel 312 has an initial estimated pixel location 338 and a corresponding flow search area 340 in target frame 322. As shown, initial estimated pixel location 338 falls within flow search area 340 having height = h pixels and width = w pixels.

[0085] Similarly, pixel 314 has a corresponding initial estimated pixel location 344 and a corresponding flow search area 346 having a height = h pixels and a width = w pixels. The initial estimated pixel location of each pixel in source frame 310 may be stored in a flow target map (e.g., Figure 3A As noted above, the flow search areas 340 and 346 may be included in a correlation volume such as Figure 3A In some existing optical flow estimation system implementations, the flow search region for each pixel in the source frame 310 may include pixels of the entire target frame 322. In such implementations, the flow search region for each pixel may have a width=W pixels and a height=H pixels. In such implementations, the correlation volume may be formed by full frame pixels of the target frame 322.

[0086] Figure 4A An example of a correlation volume 432 including flow search areas 433A, 433B to 433N is illustrated. The correlation volume 432 may be similar to Figure 3A The correlation volume 332 is the same as the correlation volume 332 and performs a similar function as the correlation volume. Figure 4A As shown, each of the flow search areas 433A, 433B to 433N may have a width = w features and a height = h features, such that the total number of values ​​(e.g., feature values) in each flow search area 433A, 433B to 433N is equal to w×h features. Figure 3A As described above, the width w and height h of the flow search regions 433A, 433B to 433N may be smaller than the corresponding width W and height H of the full target frame (eg, the target frame 322 ).

[0087] In some implementations, a source feature map such as feature map F 0 Each feature of may have a corresponding flow search region. In such a specific implementation, if the source feature map has N features, the correlation volume 432 may include N flow search regions. For example, for Figure 3A For each pixel of source frame 310 of the source frame 310, there may be a flow search region included in correlation volume 432. As described above, source frame 310 may have a height = H pixels and a width = W pixels, such that N = H × W. In some cases, feature vectors extracted from source frame 310 may also have a height = H and a width = W, such that N = H × W. In some implementations, each of flow search regions 433A, 433B through 433N may include w × h pixels or features. In such cases, the total number of pixels or features included in correlation volume 432 may be equal to H × W × w × h.

[0088] Figure 4B443A, 443B to 443N, which may be used when an optical flow estimation system searches for each pixel of a source frame within each pixel of a target frame. In some implementations, a source feature map such as a source feature map F 0 Each feature of may have a corresponding flow search region. In such a specific implementation, if the source feature map has N features, the correlation volume 442 may include N flow search regions. For example, for Figure 3A For each pixel of source frame 310 of the image processing unit 310, there may be a flow search area included in correlation volume 442, such that N=H×W. In some cases, the feature vector extracted from source frame 310 may also have a height=H and a width=W, such that N=H×W. In some implementations, each of flow search areas 443A, 443B to 443N may include W×H pixels or features. In such cases, the total number of pixels or features in correlation volume 442 may be equal to H×W×H×W. Table 1 below includes example values ​​of H, W, h, and w for different examples of optical flow system implementations according to examples of the present disclosure.

[0089]

[0090] Table 1

[0091]

[0092] Table 2

[0093] Table 1 provides example sizes of correlation volumes for four different optical flow datasets: Flying Chairs, Flying Things, Sintel, and KITTI2015 ("KITTI"). The values ​​in the columns labeled H and W in Table 1 represent the height (H) and width (W) of the frames within each of the corresponding datasets. For example, the frames in the KITTI dataset have a height of 288 pixels and a width of 960 pixels. In some cases, optical flow estimation may be performed only on a sub-portion of the frame to reduce the memory and computational requirements for optical flow operations. The values ​​in the columns labeled H' and W' of Table 1 represent 1 / 8 of the height H and width W of the corresponding dataset.

[0094] In some examples, optical flow estimation may be performed on a center crop (e.g., the center portion of a video frame) with height = H′ and width = W. The values ​​in the column labeled Full Volume of Table 1 represent the size of the correlation volume formed using each pixel of the target frame as a flow search region, such as Figure 4B In some implementations, the center of the target frame may also be cropped to a dimension of H′×W. Figure 4BAs can be appreciated from the description of , the correlation volume size may be equal to H′×W×H′×W′. In one illustrative example, for a KITTI dataset with cropped source and target images of dimensions 36×120, the resulting correlation volume may include 18,662,400 values. The values ​​in the column labeled Case 1 of Table 1 represent the use of a flow search area that is smaller than the full target frame (such as the above Figure 3A The optical flow search area 326 and 328 of case 1) is the correlation volume size of the correlation volume created. The optical flow search area of ​​case 1 has a height h=32 and a width w=32. The resulting correlation volume size for case 1 may be equal to H′×W′×h×w. For the KITTI dataset, the resulting correlation volume may include 2,298,240 values ​​(e.g., pixel values). Therefore, the amount of memory required to store the correlation volume for case 1 for the KITTI dataset is 23.7% of the correlation volume from the full volume column.

[0095] Table 2 is a continuation of Table 1. The dataset and volume columns of Table 1 are repeated in Table 2 to facilitate cross-references between the tables. The values ​​in the columns labeled Case 2, Case 3, Case 4, and Case 5 of Table 2 represent the results of a search using a flow search area that is smaller than the full target frame (such as the above Figure 3A The correlation volume size of the correlation volume created from the flow search area 326 and 328 of the cropped source frame is shown in Figure 1. Each of the column headings for these four cases includes the corresponding optical flow search area dimensions: height = h and width = w. The optical flow search area dimensions of each case are smaller than the corresponding dimensions H' and W of the cropped source frame or target frame. For example, the values ​​in the case 1 column represent the size of the volume created from the flow search area with height = 14 and width = 38 for each data set.

[0096] As shown in the table, the correlation volumes for Case 2, Case 3, Case 4, and Case 5 of the KITTI dataset may require 12.31%, 6.37%, 2.50%, and 1.67% of the memory required for the full volume correlation volume. The reduced correlation volume size can also lead to a corresponding reduction in the computational work for pairwise correlations within the correlation volume. In some cases, these memory and computational savings of almost an order of magnitude can be achieved, and the accuracy cost will not be relatively low. For example, the experimental results of Case 2 and Case 3 show that when compared with the full volume case, the resulting optical flow map covers 97.25% of the feature correlations for the KITTI dataset and 97.41% of the feature correlations for the Sintel dataset. This means that less than 3% of the optical flow information is lost due to the reduction of the optical flow search area.

[0097] As shown in Table 1 above for the full volume case, for each of the W'×H'=N pixels in the source frame, there may be W'×H'=N possible locations within the flow search region of the target frame that are searched to find the pixel's location in the target frame. Therefore, the amount of memory and computational effort required to complete optical flow estimation is O(N 2 ). In some examples, as described above, optical flow estimation can be performed using a flow search region that is smaller than the entire target frame. The flow search region can be a fixed size h×w that is not proportional to N, and therefore, the memory and computational effort required to complete optical flow estimation is reduced to O(N).

[0098] In some cases, the size of the dimensions h and w of the flow search area can be determined based on a statistical model of the optical flow for a particular data set and / or optical flow task. In one example, the dimensions of the flow search area can be determined based on the cumulative distribution function (CDF) of the data set. In such cases, the dimensions of the flow search area can be derived based on the selected probability confidence value. For example, the flow search area dimensions can be selected to target a population coverage rate of >95%, a population coverage rate of >97%, a population coverage rate of >99%, etc. FIG. 6A to FIG. 6D Example CDFs are shown and discussed.

[0099] Figure 5 is a diagram illustrating an example of an optical flow estimation system 500. In some cases, all or a portion of the optical flow estimation system 500 may correspond to Figure 1 The optical flow estimation system 100 of and / or included in the optical flow estimation system. For example, the engines of the optical flow estimation system 500 (e.g., the feature extraction engine 502, the flow prediction engine 504, the feature sampling engine 506, the correlation volume engine 508, and the optical flow calculation engine 510) can be configured to perform all or part of the functions performed by the engines of the optical flow estimation system 100 and / or any additional functions. As will be explained in more detail below, the optical flow estimation system 500 can use volume sampling to perform functions optimized for optical flow estimation.

[0100] like Figure 5 As shown, the optical flow estimation system 500 can receive a source frame I S and target frame I T In one example, source frame I S Indicates that in the target frame I T The frame received previously. For example, source frame I S Can be in the frame sequence with the target frame I T Directly adjacent. Source frame I S and target frame I T It can be input to the feature extraction engine 502. Figure 5 As shown, source frame I S and target frame IT The features may be concatenated or otherwise combined before being passed to the feature extraction engine 502 .

[0101] In some examples, feature extraction engine 502 may determine the source frame I S and / or target frame I T In one example, the contextual features associated with the pixels may include feature vectors extracted from the frame using a machine learning system and / or algorithm. An example of a machine learning system and / or algorithm that can be used is a deep neural network trained for feature extraction. Fig. 9 and Fig.10 Illustrative examples of deep neural networks are described. The feature vector may indicate features, such as labels or classifications of pixels, visual attributes and / or properties of pixels, semantic features of pixels, and other features.

[0102] In some cases, the feature vector may include information related to the spatial characteristics of the pixel. The spatial characteristics may include the association of the pixel with an object boundary, the smoothness of the pixel, discontinuities associated with the pixel, and other characteristics. In some cases, the spatial characteristics may include a spatial confidence associated with the importance and / or relevance of the pixel to the overall optical flow estimate. For example, pixels with high spatial confidence may be highly important and / or relevant to the optical flow estimate (e.g., high motion).

[0103] In some cases, the feature vector may include information related to the temporal characteristics of the pixel. In some cases, the temporal characteristics of the pixel may include one or more characteristics associated with the motion of the pixel, including the speed of the motion, the acceleration of the motion, and other characteristics. In one example, the temporal characteristics may include a confidence associated with the importance and / or relevance of the motion of the pixel to the overall optical flow estimate. For example, pixels with high temporal confidence may be highly important and / or relevant to the optical flow estimate.

[0104] In some cases, the feature extraction engine 502 may determine multi-scale context features associated with the frame. The multi-scale context features may include features associated with the frame at various scales (e.g., resolutions). For example, the feature extraction engine 502 may determine context features associated with a high-scale (e.g., full resolution) version of the frame. Additionally or alternatively, the feature extraction engine 502 may determine context features associated with one or more lower-scale (e.g., reduced resolution) versions of the frame. In some cases, context features associated with different scales may be utilized at different steps of the optical flow estimation process. For example, utilizing low-scale feature vectors may improve the efficiency of some optical flow estimation steps, while utilizing high-scale feature vectors may improve the quality and / or accuracy of other optical flow estimation steps.

[0105] In some cases, the contextual features associated with a pixel may include contextual features associated with pixels surrounding the pixel and / or near the pixel, as noted above. For example, each pixel of a frame may represent a central pixel surrounded by one or more neighbor pixels. In some examples, a neighbor pixel may refer to any pixel directly adjacent to the central pixel (e.g., horizontally, vertically, and / or diagonally adjacent). In other examples, a neighbor pixel may refer to a pixel that is separated from the central pixel by no more than a threshold distance or a threshold number of pixels (e.g., 2 pixels, 3 pixels, etc.). In another example, a neighbor pixel may be a pixel that has a high spatial and / or temporal association with a pixel. These pixels may be adjacent to the central pixel, or may not be adjacent to the central pixel (e.g., non-local).

[0106] The feature extraction engine 502 may determine contextual features for any number of neighbor pixels associated with the center pixel. For example, the feature extraction engine 502 may extract and collect as many contextual features as needed for one or more steps of the optical flow estimation process (explained in more detail below). The feature sampling engine 506 may also associate contextual features of neighbor pixels with the center pixel. For example, the feature extraction engine 502 may cascade, group, and / or otherwise store the contextual features of neighbor pixels in combination with the contextual features of the center pixel within a data structure associated with the center pixel.

[0107] The data structure may include an index corresponding to the coordinates of the center pixel. In one example, the feature extraction engine 502 may fuse the contextual features associated with each of the related neighbor pixels by weighting, summing, concatenating, and / or other techniques. For example, the feature extraction engine 502 may be based on equation f p，i , i∈{0, 1, ...., C-1}, C∈R to determine the fused context features so that the feature f is derived for pixel p p，i , where C represents the number of channels.

[0108] Associating the context features of neighbor pixels with the context features of the center pixel can improve the accuracy of optical flow estimation. For example, determining and storing the context features of neighbor pixels in conjunction with the center pixel can help the optical flow estimation system 500 accurately identify the pixel corresponding to the center pixel in subsequent frames. The context information associated with the neighbor pixels can help the optical flow estimation system 500 accurately select the corresponding pixel from multiple candidate pixels.

[0109] The flow prediction engine 504 may provide information from the flow target graph (e.g., Figure 3AIn some cases, the flow prediction engine 504 may also include the dimensions of the flow search area that can be utilized by the correlation volume engine 508 below. In some cases, the flow prediction engine 504 may also supply supplemental historical data, such as Figure 1 The supplemental historical data 112 may be combined with features generated by the feature extraction engine 502 to perform sequential flow consistency calculations as described above. For example, if the flow prediction engine 504 includes supplemental historical data about the trajectory of a ball, then the features associated with the ball in the source frame extracted by the feature extraction engine 502 may be combined with the supplemental historical information to provide an initial estimated location of the ball in the target frame.

[0110] In some cases, feature sampling engine 506 may receive features (e.g., represented by one or more feature vectors) extracted by feature extraction engine 502. Feature sampling engine 506 may perform operations to sample and / or regroup sample points of the features. For example, feature sampling engine 506 may retrieve and / or group feature vectors (or sample points of features in feature vectors) to facilitate subsequent stages of processing.

[0111] The correlation volume engine 508 may receive the sampled feature vectors from the feature sampling engine 506. The correlation volume engine 508 may perform correlation volume calculations on the sampled feature vectors. For example, using the feature vectors from two input frames (source frame I S and target frame I T The correlation volume engine 508 can calculate pairwise correlations in several pairwise combinations (e.g., for all possible pairwise combinations) using the output of the sampled feature maps of the flow search area within the source frame as input. Each correlation parameter represents two features (one feature per frame (e.g., from source frame I S A feature and from the target frame I T A feature of a feature)))

[0112] The correlation volume determined by the correlation volume engine 508 may be used as input to a subsequent optical flow estimation (eg, performed by the optical flow computation engine 510). In one illustrative example, a set of pixels (eg, a tensor comprising data) may have H s W s C and h t w tThe dimensions or shape of C, where H represents the height of the source feature map, W represents the width of the source feature map, h represents the height of the optical flow search area within the target frame, w represents the width of the optical flow search area within the target frame, and C represents the number of channels (or in some cases, depth) in the neural network used for the optical flow estimation system 500. In some examples, the correlation volume engine 508 can calculate the correlation volume using the following equation: where f s 、f t ∈R C They are source frame I S and target frame I T The characteristics of individual pixels in a set of pixels.

[0113] The optical flow calculation engine 510 may receive the correlation volume calculation (e.g., the calculated correlation volume) from the correlation volume engine 508. The optical flow calculation engine 510 may use the features in the correlation volume calculation to perform point-by-point (e.g., pixel-by-pixel) optical flow estimation. In some cases, the optical flow calculation engine 510 may use one or more neural network operations (e.g., one or more convolutional layers, one or more residual convolution blocks, and / or other network operations) to refine and / or adjust the optical flow estimate. For example, the optical flow calculation engine 510 may determine an optical flow estimate for a particular feature vector. In one example, the optical flow calculation engine 510 may perform optical flow estimation to determine an optical flow vector for a pixel or a region of pixels. In some examples, the optical flow calculation engine 510 may perform an optical flow estimation to determine an optical flow vector for a pixel or a region of pixels corresponding to the source frame I. S and target frame I T The characteristics of the source frame I S and target frame I T Same resolution.

[0114] The optical flow F calculated by the optical flow calculation engine 510 f It can be calculated according to the following equation:

[0115]

[0116] where Ff,T is the initial flow target graph (e.g., as provided by flow prediction engine 504), and ΔF f，i is an iterative adjustment to the value of the flow target vector generated during an iteration of optical flow estimation. For example, iterative optical flow estimation can be performed where the optical flow estimation system includes a recurrent neural network (RNN) or utilizes any iterative estimation technique. In such an iterative case, the values ​​of the optical flow targets in the optical flow map may be updated in small increments per iteration until they converge to a final value. For example, in an optical flow estimation system with three iterations (e.g., N=3), F f The values ​​can be:

[0117] F f=F f，T +ΔF f，0 +ΔF f，1 +ΔF f，2

[0118] In some cases, the optical flow computation engine 510 may perform only a single iteration, in which case the initial flow target vector will be a single value ΔF f，0 renew.

[0119] As mentioned above, the feature extraction engine 502 can determine multi-scale context features associated with pixels of a frame. In some cases, various steps of the optical flow estimation process can utilize context features of different scales. For example, the optical flow computation engine 510 can utilize extracted features in the form of feature pyramids at multiple scales, concatenated and / or fused features at one or more scales, or other combinations of features.

[0120] In some examples, the optical flow computation engine 510 may take the most recent pixel-level or patch-level (e.g., for a pixel and associated neighboring pixels) flow (or displacement) estimate to look up the corresponding correlation metric in the correlation volume along with the appropriate neighborhood context (e.g., association with neighboring pixels). In such examples, the looked-up metric may be further fed into a feed-forward recurrent neural subnet or convolution / filter layer to estimate and update a new per-pixel or per-patch flow or displacement.

[0121] Fig. 6A and Figure 6B Illustrate a method that can be used to determine Figure 1 An example cumulative distribution function (CDF) of a first example video dataset of shapes and / or sizes of flow search regions used by the optical flow estimation system 100. Fig. 6A CDF 602 illustrates the CDF of the optical flow vector components of the first data set in the width dimension (eg, left-right movement). Fig. 6A The CDF 602 may be obtained by performing optical flow estimation using an optical flow estimation system such as optical flow estimation system 100 and plotting the resulting width dimension components of the optical flow estimation values. The y-axis values ​​of the cumulative distribution function represent the percentage of values ​​of the optical flow vector components having values ​​in the width dimension that are less than the corresponding x-axis values ​​of the plot.

[0122] like Fig. 6A As illustrated, approximately 98% of the optical flow values ​​of the first data set in the width dimension (e.g., between the 1st percentile and the 99th percentile) fall between -10 and 15. In some cases, knowledge of the characteristics of the CDF of the first data set can be used to set the width of the flow search region for optical flow estimation by the optical flow estimation system. In one illustrative example, the width w of the flow search region for optical flow estimation can be set to 25 to match Fig. 6AThe illustrated range corresponds to -10 to 15.

[0123] Figure 6B The CDF 604 of the optical flow vector components in the height dimension (eg, moving up and down) of the first data set is illustrated. Figure 6B The CDF 604 of may be obtained by performing an optical flow estimation (such as that performed by the optical flow estimation system 100) and plotting the resulting height dimension component of the resulting optical flow values. Figure 6B As illustrated, approximately 98% (e.g., between the 1st and 99th percentiles) of the optical flow values ​​of the first data set in the height dimension fall between -5 and 6. In an illustrative example, the width w of the flow search area for optical flow estimation may be set to 11 to correspond to Figure 6B The range shown corresponds to -5 to 6. Figure 6B As shown, the optical flow values ​​in the height dimension tend to have larger positive values ​​than negative values.

[0124] If targeted Fig. 6A and Figure 6B The illustrative example of a flow search region of 25 pixels in width and 11 pixels in height corresponds to Case 3 described above with respect to Table 2. The illustrative values ​​are derived based on an experimental CDF obtained from the Sintel dataset.

[0125] Figure 6C The CDF 606 of the second data set in the width dimension is illustrated. Figure 6C As illustrated, approximately 98% of the optical flow values ​​of the first data set in the width dimension (e.g., between the 1st and 99th percentiles) fall between -16 and 22. In one illustrative example, the width w of the flow search region for optical flow estimation may be set to 38 to correspond to Figure 6C The illustrated range corresponds to -16 to 22.

[0126] Fig.6D The CDF 608 of the second data set in the height dimension is illustrated. Fig.6D As illustrated, approximately 98% (e.g., between the 1st and 99th percentiles) of the optical flow values ​​of the first data set in the height dimension fall between -4 and 10. In one illustrative example, the width w of the flow search region for optical flow estimation may be set to 14 to correspond to Fig.6D The illustrated range corresponds to -4 to 10.

[0127] If targeted Figure 6C and Fig.6DThe illustrative example of a flow search region of 38 pixels in width and 14 pixels in height corresponds to Case 2 described above with respect to Table 2. The illustrative values ​​are derived based on an experimental CDF obtained from the KITTI dataset.

[0128] Fig. 7A , Figure 7B , Figure 7C and Fig.7D This is an example that can be Figure 1 1. An example diagram of an example optical flow search area used by the optical flow estimation system 100. Fig. 7A , Figure 7B , Figure 7C and Fig.7D As shown, a first pixel position 702A in a source frame 701 of a frame sequence may have a corresponding pixel position 702B in a target frame 703 of the frame sequence. In the illustrated example, the source frame 701 and the target frame 703 each have a height of H pixels and a width of W pixels. Once the corresponding pixel position 702B is found within the search region, an optical flow vector may be generated by determining the amount of movement of the pixel, as described above with respect to Figure 2 described.

[0129] Fig. 7A An example of a flow search area 720 including each pixel of the target frame to be searched is illustrated. As described above, the optical flow estimation system (eg, the optical flow estimation system 100) may search each pixel of the source frame 701 within the target frame 703 to generate an optical flow map. Fig. 7A In the illustrated example of , the flow search area 720 includes W×H total search positions equal to the number of pixels in the target frame 703. As described above, utilizing a full frame as the flow search area may incur significant costs in terms of both memory and computational effort. For example, performing point-to-point correlation within the correlation volume based on a full frame search of the target frame may be O(N×H) in terms of computational and memory costs. 2 ) to zoom.

[0130] Figure 7B An example of a flow search area 722 is illustrated, which is a square having a width and height of three pixels and centered at an initial flow target 732 in the target frame 703. In some cases, a square flow search area 722 may be used when the data set has a symmetric CDF in both the width and height dimensions. In some examples, a symmetric flow search area 722 may be used when the characteristics of the video data to be analyzed by the optical flow estimation system are not available and there is no basis for searching in one direction over another.

[0131] In some aspects, the width and height of the square flow search area 722 may vary depending on the characteristics of the video data being analyzed (e.g., characteristics of the CDF). Further, in some aspects, the width and height of the square flow search area 722 may vary depending on whether there is available optical flow history information for use by the optical flow estimation system. For example, when there is no available optical flow history information, a larger flow search area may be required.

[0132] Conversely, when one or more frames of optical flow history are available, a smaller flow search area can be used. In one illustrative example, when no optical flow history is available, the optical flow estimation system can use a square flow search area 722 of 32×32 pixels, and when one or more frames of optical flow history are available, a square flow search area of ​​16×16 pixels can be used.

[0133] Figure 7C An example optical flow search area 724 is illustrated as a rectangle having a width of three pixels and a height of five pixels and an initial flow estimate 734 centered in both the width and height dimensions. In some cases, a rectangular optical flow search area may be utilized when the video data set being processed by the optical flow estimation system has a greater variation in optical flow values ​​in one dimension than in another. For example, in the illustrated example, when the height and width of the CDF are close to symmetry (such as in the example above), the initial flow estimate 734 may be centered in both the width and height dimensions. Fig. 6A and Figure 6B When CDF 602 and CDF 604 are illustrated, an optical flow search area 724 may be used.

[0134] Fig.7D An example optical flow search area 726 is illustrated having the same shape as optical flow search area 724, but with an initial flow estimate 736 that is not centered within the flow search area. In some aspects, optical flow search area 726 may be utilized when the video data set being processed has a greater distribution in one direction than in another direction in the width and / or height dimensions. For example, flow search area 726 may correspond to Fig.6D As shown in CDF 608, the CDF has a distribution in the height dimension that is biased toward positive values, which may correspond to a downward shift. Fig.7D There is shown one row of pixels above the initial flow estimate 736 and three rows of pixels below the initial flow estimate 736. Thus, the optical flow search area 726 may also include more pixels below the initial flow estimate 736 than above the initial flow estimate 736.

[0135] The described systems and techniques provide volume sampling for generating a correlation volume using a subset of pixels and / or features in a flow search region from a target frame, in contrast to performing correlation using a correlation volume based on all pixels of the target frame. This solution can be beneficial for optical flow estimation performed on devices with limited computational and memory resources (specifically, low latency memory such as TCM). In some cases, the optical flow estimation can also utilize a flow target map to provide an initial estimated location of pixels and / or features in the target frame. In some cases, the flow target map can be based on a single previous optical flow estimation, and in some cases, the flow target map can be based on multiple previous optical flow estimations.

[0136] By determining a flow search area that is smaller than the size of the full object frame but does not significantly reduce the accuracy of optical flow estimation, an optical flow estimation system (e.g., optical flow estimation system 100 and optical flow estimation system 500) can calculate the optical flow based on O(N) instead of O(N). 2 ) is proportional to the size of the input frame. The size of the flow search area can be determined based on the characteristics of the data being analyzed by the optical flow estimation system in combination with the amount of available optical flow history information and the confidence that the optical flow history information applies to the current frame being processed by the optical flow estimation system. The disclosed optical flow estimation system can use volume sampling to optimize optical flow estimation for any combination of available optical flow map information (e.g., whether there is information from a single previous frame or multiple previous frames), supplemental history information (e.g., feature information extracted from previous frames), and / or information about the characteristics of the data (e.g., the CDF of a representative data set for the optical flow estimation problem to be solved).

[0137] The described systems and techniques may also employ a single level cost volume with a changeable resolution. The systems and techniques may also include dynamic coarse-to-fine lookups and sampling of local grids of regions.

[0138] Fig. 8A 8 is a block diagram illustrating an example architecture 800 for image processing using a coarse-to-fine cost volume in accordance with various aspects of the present disclosure. Fig. 8A As shown, the example architecture 800 may include a convolutional feature encoder g 0 802 and context encoder 804.

[0139] The example architecture 800 may receive an image I 1 and I 2 As input. Image I 1 and I 2 The convolutional feature encoder g may include a portion of an image sequence, such as, for example, consecutive frames of a video, etc. (e.g., as described in the present disclosure). 0 802 can be applied to image I 1 and I 2 To extract the corresponding image features gθ (I 1 ), g θ (I 2 ). By the convolutional feature encoder g 0 802 generated image feature g θ (I 1 ), g θ (I 2 ) can be where K is the downsampling scale (e.g., 16 or 8), D is the feature dimension (e.g., 64, 128, or 256), H is the height, and W is the width.

[0140] Given an image I 1 , I 2 and image feature g θ (I 1 ), g θ (I 2 ), then all relevant or cost volumes Cg can be performed θ (I 1 ), g θ (I 2 ) (eg, 806a-806c) may be calculated by applying a dot product, where the value of each element may be defined by: C ijkl =∑ h g θ (I 1 ) ijh ·g θ (I 2 ) klh , where C ijkl is represented as the image feature g θ (I 1 ) ijh The i-th row and j-th column of and the image features g θ (I 2 ) klh The channel-wise h dot product output between the kth row and the 1st column of . Fig. 8A Cost volumes 806a-806c of different resolutions that may be generated are shown. Although three cost volumes (e.g., 806a-806c) of different resolutions are shown, the present disclosure is not limited thereto. It will be appreciated that more or fewer cost volumes of different resolutions may be generated. Each of the cost volumes 806a-806c may be generated in different iterations. That is, a single level of cost volume may be generated and applied in each iteration. For example, cost volume 806a may be generated in an initial iteration. In the initial iteration, a lower, coarser resolution may be employed. Then, with each subsequent iteration, for example, the resolution of the cost volume may be increased.

[0141] The context encoder 804 can be used together with the convolutional feature encoder g 0 802 separately, or in some aspects, combined in a single encoder. The context encoder 804 can be applied to a base image (e.g., image I 1 ). The context encoder 804 may extract a base image (eg, image I 1 ) which may be output and supplied to an update block (e.g., 810a-810c).

[0142] Correlation Lookup L C The operator (eg, 808a-808c) may generate a correlation feature map, which may be based on the current optical flow estimate f representing the optical flow in the x and y directions, respectively. k =(f 1 , f 2 ) to obtain. Then, for I 1 For each pixel x=(u, v) in , the local neighborhood area around x (e.g., 812a-812c) can be represented by To define I 2 The corresponding relationship in the local neighborhood is a set of integer offsets within a radius r. The neighborhood regions 812a-812c (eg, regions of interest in an image) may be changed, however, the resolution within an iteration may remain the same.

[0143] Cost volumes 806a-806c C at different resolutions 1 , C 2 , C 3 It can be obtained, for example, by pooling over the feature map used for correlation lookup or pooling along, for example, the last two dimensions. For coarse-to-fine lookup, for each update step (or iteration), a single level cost volume C can be used i (e.g., 806a) and the corresponding search L C (e.g., 808a).

[0144] Each update block 810a-810c may take the current flow estimate f k , Correlation Lookup L C The update block (e.g., 810a) may include a function for processing the current flow estimate f k The preprocessing convolution layer can be used to estimate the current flow f k And the correlation features are output respectively. k and r k . Output feature z k 、r kTogether with the output of the context encoder 804 and the previous hidden state (e.g., 814a, 814b), it can be input to the update block (e.g., 810b). For the lookup cascade, z k-1 , z k-2 , and r k-1 , r k-2 It may also be supplied to an update block (e.g., 810b). The update blocks (e.g., 810a-810c) may then compute a flow estimate 816. An overall flow estimate 816 may be generated and updated with each successive iteration using the element-wise sum of the outputs of the update blocks (e.g., 810a-810c).

[0145] According to aspects of the present disclosure, a single level cost volume (e.g., 806a-806c) may be employed in each iteration. The resolution of the cost volume (806a-806c) may be selected, for example, to be 1 / 16. As described, a coarse-to-fine strategy for selecting the resolution of the cost volume (e.g., 806a-806c) to have an effective receptive field that changes across iterations can reduce overall latency when capturing large motions. A coarser resolution (e.g., 806a) may be used for the cost volume of the initial iteration to provide a good initialization for further refinement. Then, for later iterations, the resolution of the cost volume (e.g., 806b or 806c) may be increased so that refinement can focus on finer correspondences. In some aspects, one or two iterations of a coarser cost volume resolution may be used after a finer refinement to capture whether there are any areas with large displacements.

[0146] To dynamically change the cost volume, average pooling may be performed on the feature maps of the convolutional encoder to modify the resolution of the cost volume. Thus, the cost volume resolution may change across iterations. In some aspects, the correlation lookup radius may also change, and irregular sampling (e.g., using 812a-812c) may also be used with the same cost volume resolution across updates. However, such irregular sampling or larger neighborhood sampling (e.g., using 812a-812c) may be inefficient on hardware.

[0147] Because the receptive field may change relative to the cost volume (e.g., 806a-806c), the parameters (e.g., weights) of the update blocks (e.g., 810a-810c) may not be shared across all iterations. However, in some aspects, weight sharing may be used for update blocks (e.g., 810a-810c) that process cost volumes of the same resolution.

[0148] In order to preserve more information than the local neighborhood information captured in the lookup operation (e.g., using 812a-812c), the correlation lookup L CThe output of (eg, 808a-808c) may be concatenated with previous flow estimates (eg, which may also be equivalent to positioning encoding) over one or more previous iterations as additional input to the update blocks (eg, 810a-810c).

[0149] In some aspects, a Just-in-Time (JiT) approach may also be employed to further reduce peak memory utilization. The JiT approach may provide a computational alternative to sampling each pixel only by a grid of certain required features based on the corresponding neighborhood. The linear complexity construction and correlation lookup operations may be decomposed to achieve low peak memory for, for example, a tightly coupled memory (TCM) (e.g., 4MB) constraint on a typical smartphone. Even during such on-demand lookups, a three-dimensional cost volume may be constructed based on a fixed-size lookup radius (R), which may use an O(N×2R) memory complexity with N=(H / 16×W / 16), which may not be feasible to fit into a memory less than 2MB.

[0150] Specifically, in order to provide a system involving a balanced pipeline of memory read / write access and neural network processing on the target hardware, each pixel may include N slice (For example, image feature g θ (I 1 ) the number of tiles in the decomposition) and R (e.g., the radius of the search range in either direction).

[0151] Instead of processing the image feature g once θ (I 1 )’s entire feature map, image feature g θ (I 1 ) can be based on the local neighborhood area Process a few pixels at a time in a sliding window manner. From the image feature g θ (I 2 ) can be retrieved to construct the correlation volume of the current slice, and can be aggregated to generate the overall correlation volume of the current iteration. In this specific implementation, the real-valued part and the sub-pixel part of the current flow estimate can be decomposed into first using the real-valued neighborhood to construct the cost volume with a sliding window method, and later adjusted for the sub-pixel part using a bilinear shift (or bilinear sampling) method.

[0152] In some aspects, each iteration may employ bilinear sampling to perform warping based on the current estimate of optical flow. Bilinear sampling involves a series of point-by-point grid operations followed by a series of bilinear interpolation operations based on the current estimate of optical flow. Bilinear interpolation operations may be inefficient in hardware because it may not be possible to perform with vectorization and thus may add a bottleneck to latency. To address this issue, a mathematically equivalent bilinear shift may replace grid sampling with a vectorizable element-by-element operator, since the grid input may be uniform, which is suitable for dense optical flow estimation.

[0153] Based on the shift along the x dimension (Δx) and the shift along the y dimension (Δy), the grid sampling operation can be decomposed into bilinear operations associated with the two-dimensional shift. The bilinear operation can be mapped to vectorizable element-wise operators on the hardware, thereby achieving flexibility with respect to compile / run time by enabling further decomposition into smaller operators / tiles.

[0154] Given an input tensor T, an x-shift (Δx), and a y-shift (Δy), the first input tensor T can be split into two equal parts indexed by a one-pixel offset along the x-direction and then interpolated using the x-shift Δx and (1-Δx). Similar operations can then be repeated along the y-direction.

[0155] Figure 8B 8 is a flow chart illustrating an example of a process 850 for performing image-based processing according to various aspects of the present disclosure. Figure 8B As shown, in some aspects, process 850 may include processing a single level cost volume across a number of processing stages by varying the receptive field of the cost volume processing across each of the processing stages (block 852). For example, the varying may include processing a cost volume of different resolutions at each of the plurality of processing stages while maintaining the same neighborhood sampling radius. The resolution may increase from a first processing stage to a later processing stage. The varying may also include varying the neighborhood sampling radius at each of the processing stages while maintaining the same resolution. Additionally, in another illustrative aspect, the processing may be performed using only one level cost volume at each of the processing stages.

[0156] In some aspects, process 850 may also include performing a learning-based correspondence estimation task based on the processing (block 854). For example, the task may be optical flow estimation, stereo estimation, multi-view stereo estimation, or simultaneous localization and mapping (SLAM) estimation.

[0157] In some examples, the described processes (e.g., process 850 and / or other processes described) can be performed by one or more computing devices or apparatuses (such as a computer having Fig.111100) is performed by one or more computing devices of the computing device architecture of the computing system 1100 shown. In various aspects, for example, process 850 can be performed by one or more processors such as processor 1110.

[0158] In one example, process 850 may be implemented by having Figure 1 The computing device of the computing device architecture of the computing system 1100 of the optical flow estimation system 100 shown in the figure is executed. In some examples, the computing device may include a mobile device (e.g., a mobile phone, a tablet computing device, etc.), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a television, a vehicle (or a computing device of a vehicle), a robotic device, and / or any other computing device having resource capabilities for performing the described processes (including process 850).

[0159] In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more transmitters, receivers, or combined transmitter-receivers (e.g., referred to as transceivers), one or more cameras, one or more sensors, and / or other components configured to perform the steps of the described processes. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0160] Components of a computing device may be implemented in circuits. For example, a component may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), a neural processing unit (NPU), and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described.

[0161] Process 850 is illustrated as a logic flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0162] Additionally, the described processes (including process 850 and / or other processes described) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, implemented by hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes multiple instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0163] As described above, the described image processing techniques can be implemented using a neural network-based machine learning system. Illustrative examples of neural networks that can be used include one or more convolutional neural networks (CNNs), autoencoders, deep belief networks (DBNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), gated recurrent units (GRUs), any combination thereof, and / or any other suitable neural network.

[0164] Fig. 9 is an illustrative example of a deep learning neural network 900 that can be used by an image processing system. An input layer 920 includes input data. In an illustrative example, the input layer 920 may include data representing pixels of an input video frame. The neural network 900 includes a plurality of hidden layers 922a, 922b to 922n. The hidden layers 922a, 922b to 922n include "n" hidden layers, where "n" is an integer greater than or equal to one. The number of hidden layers may include as many layers as required for a given application. The neural network 900 also includes an output layer 924 that provides an output generated by the processing performed by the hidden layers 922a, 922b to 922n. In an illustrative example, the output layer 924 may provide a classification for an object in an input video frame. The classification may include a category that identifies a type of object (e.g., a person, a dog, a cat, or other object).

[0165] Neural network 900 is a multi-layer neural network composed of interconnected nodes. Each node can represent a piece of information. The information associated with these nodes is shared between different layers, and each layer retains the information as it processes it. In some cases, neural network 900 may include a feedforward network, in which case there is no feedback connection in which the output of the network is fed back into itself. In some cases, neural network 900 may include a recursive neural network, which may have loops that allow information to be carried across nodes when reading in input.

[0166] Information can be exchanged between nodes by node-to-node interconnection between each layer. The nodes of input layer 920 can activate a group of nodes in the first hidden layer 922a. For example, as shown in the figure, each input node in the input nodes of input layer 920 is connected to each node in the node of the first hidden layer 922a. The nodes of hidden layers 922a, 922b to 922n can transform this information by applying activation function to the information of each input node. Then, the information derived from this transformation can be passed to the nodes of next hidden layer 922b, and these nodes can be activated, and these nodes can perform their own specified functions. Example functions include convolution, up sampling, data conversion and / or any other suitable function.

[0167] The output of hidden layer 922b may then activate nodes of the next hidden layer, and so on. The output of the last hidden layer 922n may activate one or more nodes of output layer 924, providing an output at the one or more nodes. In some cases, although a node (e.g., node 926) in neural network 900 is shown as having multiple output lines, the node has a single output, and all lines shown as output from the node represent the same output value.

[0168] In some cases, each node or interconnection between nodes may have a weight, which is a set of parameters derived from the training of neural network 900. Once neural network 900 is trained, it may be referred to as a trained neural network, which may be used to classify one or more objects. For example, an interconnection between nodes may represent a piece of information learned about the interconnected nodes. The interconnection may have a tunable digital weight that may be tuned (e.g., based on a training data set), thereby allowing neural network 900 to adapt to inputs and be able to learn as more and more data is processed.

[0169] The neural network 900 is pre-trained to process features from the data in the input layer 920 using different hidden layers 922a, 922b to 922n to provide an output through the output layer 924. In an example where the neural network 900 is used to identify objects in an image, the neural network 900 may be trained using training data that includes both images and labels. For example, training images may be input into the network, where each training image has a label indicating the class of one or more objects in each image (essentially, indicating to the network what the objects are and what features they have). In one illustrative example, the training images may include an image of the number 2, in which case the label for the image may be [0 0 10 0 0 0 0 0 0].

[0170] In some cases, neural network 900 may use a training process called back propagation to adjust the weights of the nodes. Back propagation may include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update are performed for one training iteration. For each set of training images, this process may be repeated for a certain number of iterations until neural network 900 is trained well enough so that the weights of each layer are accurately tuned.

[0171] For the example of identifying an object in an image, a forward pass may include passing a training image through the neural network 900. Prior to training the neural network 900, the weights are initially randomized. The image may include, for example, an array of numbers representing pixels of the image. Each number in the array may include a value from 0 to 255 describing the intensity of the pixel at that location in the array. In one example, the array may include a 28×28×3 array of numbers with 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or a brightness and two chrominance components, etc.).

[0172] For the first training iteration of the neural network 900, the output will likely include values ​​that do not prioritize any particular class due to the random selection of weights at initialization. For example, if the output is a vector with probabilities that an object includes different classes, the probability values ​​for each of the different classes may be equal or at least very similar (e.g., for ten possible classes, each class may have a probability value of 0.1). Using the initial weights, the neural network 900 is unable to determine low-level features and therefore cannot make an accurate determination of what the classification of the object may be. A loss function may be used to analyze the errors in the output. Any suitable loss function definition may be used. An example of a loss function includes the mean squared error (MSE). The MSE is defined as It calculates the sum of the actual answer minus one-half the square of the predicted (output) answer. The loss can be set equal to the value E total .

[0173] For the first training images, the loss (or error) will be high because the actual values ​​will be very different from the predicted outputs. The goal of training is to minimize the amount of loss so that the predicted outputs are the same as the training labels. The neural network 900 can perform a backward pass by determining which inputs (weights) contribute most to the network's loss, and the weights can be adjusted so that the loss is reduced and ultimately minimized.

[0174] The derivative of the loss with respect to the weight (expressed as dL / dW, where W is the weight at a particular layer) can be calculated to determine the weight that contributes most to the loss of the network. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated so that they change in the opposite direction of the gradient. The weight update can be expressed as Where w represents the weight, w i Represents the initial weight, and η represents the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes a larger weight update, and a lower value indicates a smaller weight update.

[0175] Neural network 900 may include any suitable deep network. One example includes a convolutional neural network (CNN) that includes an input layer and an output layer with multiple hidden layers between the input layer and the output layer. Fig. 9 An example of a CNN is described. The hidden layers of the CNN include a series of convolutional layers, nonlinear layers, pooling layers (for downsampling), and fully connected layers. The neural network 900 may include any other deep networks other than CNN, such as an autoencoder, a deep belief network (DBN), a recursive neural network (RNN), etc.

[0176] Fig.10 is an illustrative example of a convolutional neural network 1000 (CNN 1000). An input layer 1020 of CNN 1000 includes data representing an image. For example, the data may include an array of numbers representing pixels of the image, where each number in the array includes a value from 0 to 255 describing the intensity of the pixel at that location in the array. Using the previous example from above, the array may include a 28×28×3 array of numbers having 28 rows and 28 columns of pixels and 3 color components (e.g., red, green, and blue, or luma and two chroma components, etc.).

[0177] The image may be passed through a convolutional hidden layer 1022a, an optional non-linear activation layer, a pooling hidden layer 1022b, and a fully connected hidden layer 1022c to be output at an output layer 1024. Fig.10Only one hidden layer in each hidden layer is shown in , but one of ordinary skill will understand that multiple convolutional hidden layers, non-linear layers, pooling hidden layers, and / or fully connected layers may be included in CNN 1000. As previously described, the output may indicate a single category of an object, or may include a probability of a category that best describes an object in the image.

[0178] The first layer of CNN 1000 is a convolutional hidden layer 1022a. The convolutional hidden layer 1022a analyzes the image data of the input layer 1020. Each node of the convolutional hidden layer 1022a is connected to an area of ​​nodes (pixels) of the input image called a receptive field. The convolutional hidden layer 1022a can be viewed as one or more filters (each filter corresponding to a different activation or feature map), where each convolution iteration of the filter is a node or neuron of the convolutional hidden layer 1022a. For example, the area of ​​the input image covered by the filter at each convolution iteration will be the receptive field of the filter. In an illustrative example, if the input image includes a 28×28 array and each filter (and corresponding receptive field) is a 5×5 array, there will be 24×24 nodes in the convolutional hidden layer 1022a. Each connection between a node and the receptive field of that node learns a weight, and in some cases learns an overall bias, so that each node learns to analyze its specific local receptive field in the input image. Each node of the convolutional hidden layer 1022a will have the same weights and biases (referred to as shared weights and shared biases). For example, the filter has an array of weights (numbers) and the same depth as the input. For the video frame example, the filter will have a depth of 3 (according to the three color components of the input image). An illustrative example size of the filter array is 5×5×3, corresponding to the size of the node's receptive field.

[0179] The convolutional nature of the convolutional hidden layer 1022a is due to the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 1022a may start at the upper left corner of the input image array and may be convolved around the input image.

[0180] As noted above, each convolution iteration of the filter can be considered a node or neuron of the convolution hidden layer 1022a. In each convolution iteration, the value of the filter is multiplied by the corresponding number of original pixel values ​​of the image (e.g., a 5×5 filter array is multiplied by a 5×5 array of input pixel values ​​at the top left corner of the input image array). The multiplications from each convolution iteration can be added together to obtain the sum for that iteration or node.

[0181] Next, the process continues at the next position in the input image, depending on the receptive field of the next node in the convolutional hidden layer 1022a. For example, the filter may be moved by a step amount to the next receptive field. The step amount may be set to 1 or other suitable amount. For example, if the step amount is set to 1, the filter will move 1 pixel to the right at each convolution iteration. Processing the filter at each unique position in the input volume produces a number representing the filter result at that position, thereby producing a sum value determined for each node of the convolutional hidden layer 1022a.

[0182] The mapping from the input layer to the convolutional hidden layer 1022a is called an activation map (or feature map). The activation map includes a value for each node that represents the result of the filter at each location of the input volume. The activation map may include an array that includes various summed values ​​produced by each iteration of the filter over the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (with a stride of 1), the activation map will include a 24×24 array. The convolutional hidden layer 1022a may include several activation maps in order to identify multiple features in the image. Fig.10 The example shown includes three activation maps. Using the three activation maps, the convolutional hidden layer 1022a can detect three different kinds of features, each of which is detectable across the entire image.

[0183] In some examples, a nonlinear hidden layer may be applied after the convolutional hidden layer 1022a. A nonlinear layer may be used to introduce nonlinearity into a system that has already computed linear operations. An illustrative example of a nonlinear layer is a rectified linear unit (ReLU) layer. The ReLU layer may apply the function f(x)=max(0, x) to all values ​​in the input volume, which changes all negative activations to 0. Thus, the ReLU may increase the nonlinear properties of the CNN 1000 without affecting the receptive field of the convolutional hidden layer 1022a.

[0184] The pooling hidden layer 1022b may be applied after the convolutional hidden layer 1022a (and after the nonlinear hidden layer when used). The pooling hidden layer 1022b may be used to simplify the information in the output from the convolutional hidden layer 1022a. For example, the pooling hidden layer 1022b may take each activation map output from the convolutional hidden layer 1022a and use a pooling function to generate a condensed activation map (or feature map). Max pooling is an example of a function performed by a pooling hidden layer. The pooling hidden layer 1022b uses other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions. A pooling function (e.g., a maximum pooling filter, an L2 norm filter, or other suitable pooling filter) is applied to each activation map included in the convolutional hidden layer 1022a. In Fig.10In the example shown, three pooling filters are used for the three activation maps in the convolutional hidden layer 1022a.

[0185] In some examples, maximum pooling can be used by applying a maximum pooling filter (e.g., having a size of 2×2) with a stride (e.g., equal to the dimension of the filter, such as a stride of 2) to the activation map output from the convolutional hidden layer 1022a. The output from the maximum pooling filter includes the maximum number in each sub-region of the filter convolution. Using a 2×2 filter as an example, each unit in the pooling layer can summarize the area of ​​2×2 nodes in the previous layer (where each node is a value in the activation map). For example, four values ​​(nodes) in the activation map will be analyzed by a 2×2 maximum pooling filter at each iteration of the filter, where the maximum of the four values ​​is output as the "maximum" value. If such a maximum pooling filter is applied to an activation filter with a dimension of 24×24 nodes from the convolutional hidden layer 1022a, the output from the pooling hidden layer 1022b will be an array of 12×12 nodes.

[0186] In some examples, an L2 norm pooling filter may also be used. The L2 norm pooling filter includes calculating the square root of the sum of the squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (rather than calculating the maximum value as done in max pooling), and using the calculated value as output.

[0187] Intuitively, a pooling function (e.g., max pooling, L2 norm pooling, or other pooling functions) can determine whether a given feature is found anywhere in a region of an image, and the exact location information can be discarded. This can be done without affecting the results of feature detection, because once a feature has been found, the exact location of the feature is not as important as its approximate location relative to other features. Max pooling (and other pooling methods) provides the following benefits: there are far fewer pooled features, thereby reducing the number of parameters required in the later layers of CNN 1000.

[0188] The final layer of connections in the network is a fully connected layer that connects each node from the pooling hidden layer 1022b to each of the output nodes in the output layer 1024. Using the above example, the input layer includes 28×28 nodes that encode the pixel intensities of the input image, the convolutional hidden layer 1022a includes 3×24×24 hidden feature nodes based on applying a 5×5 local receptive field (for the filter) to three activation maps, and the pooling layer 1022b includes a layer of 3×12×12 hidden feature nodes based on applying a maximum pooling filter to a 2×2 region across each of the three feature maps. Extending this example, the output layer 1024 may include ten output nodes. In this example, each node of the 3×12×12 pooling hidden layer 1022b is connected to each node of the output layer 1024.

[0189] The fully connected hidden layer 1022c can obtain the output of the previous pooled hidden layer 1022b (which should represent an activation map of high-level features) and determine the features most relevant to a particular category. For example, the fully connected hidden layer 1022c layer can determine the high-level features that are most strongly associated with a particular category and can include weights (nodes) for the high-level features. The product between the weights of the fully connected hidden layer 1022c and the pooled hidden layer 1022b can be calculated to obtain probabilities for different categories. For example, if the CNN 1000 is being used to predict that an object in a video frame is a person, there will be high values ​​in the activation map representing high-level features of a person (e.g., there are two legs, a face at the top of the object, two eyes at the upper left and upper right of the face, a nose in the middle of the face, a mouth at the bottom of the face, and / or other features common to people).

[0190] In some examples, the output from output layer 1024 may include an M-dimensional vector (in the previous example, M=10), where M may include the number of categories that the program must choose from when classifying an object in an image. Other example outputs may also be provided. Each number in the N-dimensional vector may represent the probability that an object belongs to a certain category. In an illustrative example, if the 10-dimensional output vector represents objects of ten different categories is [0 0 0.050.80 0.150 0 0 0], then the vector indicates that the image is a 5% probability of an object of the third category (e.g., a dog), the image is an 80% probability of an object of the fourth category (e.g., a person), and the image is a 15% probability of an object of the sixth category (e.g., a kangaroo). The probability for a category can be considered as the confidence level that the object is part of that category.

[0191] Fig.11 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Fig.11An example of a computing system 1100 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using a connection 1105. The connection 1105 may be a physical connection using a bus, or a direct connection into the processor 1110, such as in a chipset architecture. The connection 1105 may also be a virtual connection, a networked connection, or a logical connection.

[0192] In some examples, computing system 1100 is a distributed system in which the functionality described in the present disclosure may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some examples, one or more of the described system components represent a number of such components, each of which performs some or all of the functionality for which the component is described. In some cases, a component may be a physical device or a virtual device.

[0193] The example system 1100 includes at least one processing unit (CPU or processor) 1110 and connections 1105 that couple various system components including system memory 1115, such as read-only memory (ROM) 1120 and random access memory (RAM) 1125, to the processor 1110. The computing system 1100 may include a cache 1112 of high-speed memory directly connected to, in close proximity to, or integrated as part of the processor 1110.

[0194] Processor 1110 may include any general purpose processor and hardware or software services such as services 1132, 1134, and 1136 stored in storage device 1130 configured to control processor 1110, as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1110 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0195] To enable user interaction, the computing system 1100 includes an input device 1145 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1100 may also include an output device 1135 that can be one or more of several output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1100. The computing system 1100 may include a communication interface 1140, which may generally govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using a wired and / or wireless transceiver, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Wireless signal transmission, Low energy (BLE) wireless signal transmission, Wireless signaling, radio frequency identification (RFID) wireless signaling, near field communication (NFC) wireless signaling, dedicated short range communication (DSRC) wireless signaling, 802.11 Wi-Fi wireless signaling, wireless local area network (WLAN) signaling, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signaling, public switched telephone network (PSTN) signaling, integrated services digital network (ISDN) signaling, 3G / 4G / 5G / LTE cellular data network wireless signaling, ad hoc network signaling, radio wave signaling, microwave signaling, infrared signaling, visible light signaling, ultraviolet light signaling, wireless signaling along the electromagnetic spectrum, or some combination thereof. The communication interface 1140 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1100 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction on operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0196] The storage device 1130 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0197] Storage device 1130 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1110, causes the system to perform functions. In some examples, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to necessary hardware components such as processor 1110, connection 1105, output device 1135, etc.

[0198] In some aspects, system 1100 may include: means for processing and means for executing. In some examples, means for processing and means for executing may include processor 1110, system memory 1115, ROM 1120 and / or RAM 1125, any combination thereof, or other components of system 1100.

[0199] As used, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or over a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media may have stored thereon code and / or machine-executable instructions that may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent using any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0200] In some examples, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically exclude media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0201] Specific details are provided in the above description to provide a thorough understanding of the examples provided. However, it will be understood by those skilled in the art that the examples can be implemented without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks that include devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. In addition to those components shown and / or described in the accompanying drawings, additional components may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the examples in unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details in order to avoid obscuring the examples.

[0202] The above may describe various examples as processes or methods, which are depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe an operation as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or main function.

[0203] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from computer readable media. Such instructions may include, for example, instructions and data that configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0204] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The described functionality may also be embodied in peripheral devices or add-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0205] The instructions, the media for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0206] In the foregoing description, various aspects of the present application are described with reference to the specific examples of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative examples of the present application have been described in detail, it should be understood that the inventive concept can be embodied and adopted in various other ways, and the appended claims are intended to be interpreted as including such variations, unless limited by the prior art. The various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, without departing from the broader spirit and scope of this specification, each example can be utilized in any number of environments and applications other than those described. Therefore, the description and the accompanying drawings should be considered to be illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that in an alternative example, the method can be performed in an order different from the described order.

[0207] Those of ordinary skill in the art will understand that without departing from the scope of the present specification, the less than ("<") and greater than (">") symbols or terms used may be replaced with less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively.

[0208] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0209] The phrase “coupled to” means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (eg, connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0210] Claim language or other language that states "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that recites "at least one of A and B" means A, B, or A and B. In another example, claim language that recites "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language that recites "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0211] The various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosed examples may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0212] The described techniques may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices, mobile phones, and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions that, when executed, perform one or more of the above methods. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0213] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in the present disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the described technology. In addition, in some aspects, the described functionality may be provided within a dedicated software module or hardware module configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0214] Specific implementation examples are provided in the following numbered clauses:

[0215] 1. A processor-implemented method executed by at least one processor, the processor-implemented method comprising:

[0216] processing, by the at least one processor, a single level cost volume across a plurality of processing stages by varying a receptive field of cost volume processing across each of the plurality of processing stages; and

[0217] A learning-based correspondence estimation task is performed by the at least one processor based on the processing.

[0218] 2. A processor-implemented method according to clause 1, wherein the changing comprises processing the single level cost volume of a different resolution at each of the plurality of processing stages while maintaining the same neighborhood sampling radius.

[0219] 3. A processor-implemented method according to clause 1 or 2, wherein the resolution increases from a first processing stage to a later processing stage.

[0220] 4. The processor-implemented method of clause 1, wherein the changing comprises changing a neighborhood sampling radius at each of the plurality of processing stages while maintaining the same resolution.

[0221] 5. A processor-implemented method according to any one of clauses 1 to 4, wherein the learning-based correspondence estimation task comprises optical flow estimation, stereo estimation, simultaneous localization and mapping (SLAM), or multi-view stereo.

[0222] 6. A processor-implemented method according to any of clauses 1 to 5, wherein one level cost volume is processed in each processing stage of the plurality of processing stages.

[0223] 7. A processor-implemented method according to any one of clauses 1 to 6, further comprising: sampling a subset of pixels in the receptive field, the learning-based correspondence estimation task being calculated based on the subset of pixels.

[0224] 8. A device, comprising:

[0225] at least one memory; and

[0226] at least one processor coupled to the at least one memory, the at least one processor being configured to:

[0227] processing a single level cost volume across a plurality of processing stages by varying the receptive field of the cost volume processing across each of the plurality of processing stages; and

[0228] A learning-based correspondence estimation task is performed based on the processing.

[0229] 9. The apparatus of clause 8, wherein the at least one processor is further configured to process the single level cost volume at different resolutions at each of the plurality of processing stages while maintaining a same neighborhood sampling radius.

[0230] 10. Apparatus according to clause 8 or 9, wherein the resolution increases from a first processing stage to a later processing stage.

[0231] 11. The apparatus of clause 8, wherein the at least one processor is further configured to: change a neighborhood sampling radius at each processing stage of the plurality of processing stages while maintaining the same resolution.

[0232] 12. An apparatus according to any of clauses 8 to 11, wherein the learning-based correspondence estimation task comprises optical flow estimation, stereo estimation, simultaneous localization and mapping (SLAM) or multi-view stereo.

[0233] 13. An apparatus according to any of clauses 8 to 12, wherein the at least one processor is further configured to: process a level cost volume in each processing stage of the plurality of processing stages.

[0234] 14. An apparatus according to any of clauses 8 to 13, wherein the at least one processor is further configured to: sample a subset of pixels in the receptive field, and the learning-based correspondence estimation task is calculated based on the subset of pixels.

[0235] 15. A non-transitory computer readable medium having program code recorded thereon, the program code being executed by a processor and comprising:

[0236] program code for processing a single level cost volume across a plurality of processing stages by varying a receptive field of the cost volume processing across each of the plurality of processing stages; and

[0237] Program code for performing a learning-based correspondence estimation task based on the processing.

[0238] 16. The non-transitory computer readable medium of clause 15, wherein the program code further comprises: program code for processing the single level cost volume of different resolutions at each of the plurality of processing stages while maintaining a same neighborhood sampling radius.

[0239] 17. The non-transitory computer-readable medium of clause 15 or 16, wherein the resolution increases from a first processing stage to a later processing stage.

[0240] 18. The non-transitory computer-readable medium of clause 15, wherein the program code further comprises: program code for changing a neighborhood sampling radius at each of the plurality of processing stages while maintaining the same resolution.

[0241] 19. A non-transitory computer-readable medium according to any one of clauses 15 to 18, wherein the learning-based correspondence estimation task comprises optical flow estimation, stereo estimation, simultaneous localization and mapping (SLAM), or multi-view stereo.

[0242] 20. The non-transitory computer readable medium of any of clauses 15 to 19, wherein the program code further comprises: program code for processing a level cost volume in each processing stage of the plurality of processing stages.

[0243] 21. A non-transitory computer readable medium according to any one of clauses 15 to 20, wherein the program code further comprises: program code for sampling a subset of pixels in the receptive field, the learning-based correspondence estimation task being calculated based on the subset of pixels.

[0244] 22. A device, comprising:

[0245] means for processing a single level cost volume across a plurality of processing stages by varying the receptive field of the cost volume processing across each of the plurality of processing stages; and

[0246] Means for performing a learning-based correspondence estimation task based on the processing.

[0247] 23. The apparatus of clause 22, further comprising means for processing the single level cost volume of different resolutions at each of the plurality of processing stages while maintaining a same neighborhood sampling radius.

[0248] 24. Apparatus according to clause 22 or 23, wherein the resolution increases from a first processing stage to a later processing stage.

[0249] 25. The apparatus of clause 22, further comprising means for varying a neighborhood sampling radius at each of the plurality of processing stages while maintaining the same resolution.

[0250] 26. An apparatus according to any of clauses 22 to 25, wherein the learning-based correspondence estimation task comprises optical flow estimation, stereo estimation, simultaneous localization and mapping (SLAM) or multi-view stereo.

[0251] 27. An apparatus according to any of clauses 22 to 26, further comprising means for processing a level cost volume in each processing stage of the plurality of processing stages.

[0252] 28. An apparatus according to any of clauses 22 to 27, further comprising: means for sampling a subset of pixels in the receptive field, the learning-based correspondence estimation task being computed based on the subset of pixels.

[0253] It is to be understood that the claims are not limited to the precise configuration and components illustrated above. Various modifications, changes and variations may be made in the arrangement, operation and details of the methods and apparatus described above without departing from the scope of the claims.

Claims

1. A processor-implemented method executed by at least one processor, the processor-implemented method comprising: processing, by the at least one processor, a single level cost volume across a plurality of processing stages by varying a receptive field of processing of the cost volume across each of the plurality of processing stages; as well as A learning-based correspondence estimation task is performed by the at least one processor based on the processing.

2. The processor-implemented method of claim 1 , wherein the changing comprises processing the single level cost volume at a different resolution at each of the plurality of processing stages while maintaining a same neighborhood sampling radius.

3. The processor-implemented method of claim 2, wherein the resolution increases from a first processing stage to a later processing stage. 4 . The processor-implemented method of claim 1 , wherein the changing comprises changing a neighborhood sampling radius at each of the plurality of processing stages while maintaining the same resolution.

5. The processor-implemented method of claim 1, wherein the learning-based correspondence estimation task comprises optical flow estimation, stereo estimation, simultaneous localization and mapping (SLAM), or multi-view stereo.

6. The processor-implemented method of claim 1, wherein one level cost volume is processed in each processing stage of the plurality of processing stages.

7. The processor-implemented method of claim 1 , further comprising: A subset of pixels in the receptive field is sampled, and the learning-based correspondence estimation task is calculated based on the subset of pixels.

8. A device, comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: processing a single level cost volume across a plurality of processing stages by varying a receptive field of processing of the cost volume across each of the plurality of processing stages; as well as A learning-based correspondence estimation task is performed based on the processing.

9. The apparatus of claim 8, wherein the at least one processor is further configured to process the single level cost volume of different resolutions at each of the plurality of processing stages while maintaining a same neighborhood sampling radius.

10. The apparatus of claim 9, wherein the resolution increases from a first processing stage to a later processing stage.

11. The apparatus of claim 8, wherein the at least one processor is further configured to change a neighborhood sampling radius at each of the plurality of processing stages while maintaining the same resolution.

12. The apparatus of claim 8, wherein the learning-based correspondence estimation task comprises optical flow estimation, stereo estimation, simultaneous localization and mapping (SLAM), or multi-view stereo.

13. The apparatus of claim 8, wherein the at least one processor is further configured to process a level cost volume in each processing stage of the plurality of processing stages.

14. The apparatus of claim 8, wherein the at least one processor is further configured to: sample a subset of pixels in the receptive field, and the learning-based correspondence estimation task is calculated based on the subset of pixels.

15. A non-transitory computer readable medium having program code recorded thereon, the program code being executed by a processor and comprising: program code for processing a single level cost volume across a plurality of processing stages by varying a receptive field of cost volume processing across each of the plurality of processing stages; and Program code for performing a learning-based correspondence estimation task based on the processing.

16. The non-transitory computer readable medium of claim 15, wherein the program code further comprises: Program code for processing the single level cost volume of different resolutions at each of the plurality of processing stages while maintaining a same neighborhood sampling radius.

17. The non-transitory computer readable medium of claim 16, wherein the resolution increases from a first processing stage to a later processing stage.

18. The non-transitory computer readable medium of claim 15, wherein the program code further comprises: Program code for varying a neighborhood sampling radius at each of the plurality of processing stages while maintaining the same resolution.

19. The non-transitory computer-readable medium of claim 15, wherein the learning-based correspondence estimation task comprises optical flow estimation, stereo estimation, simultaneous localization and mapping (SLAM), or multi-view stereo.

20. The non-transitory computer readable medium of claim 15, wherein the program code further comprises: Program code for processing a level cost volume in each processing stage of the plurality of processing stages.

21. The non-transitory computer readable medium of claim 15, wherein the program code further comprises: Program code for sampling a subset of pixels in the receptive field, the learning-based correspondence estimation task being computed based on the subset of pixels.

22. A device, comprising: means for processing a single level cost volume across a plurality of processing stages by varying a receptive field of the cost volume processing across each of the plurality of processing stages; and Means for performing a learning-based correspondence estimation task based on the processing.

23. The apparatus according to claim 22, further comprising: Means for processing the single level cost volume of different resolutions at each of the plurality of processing stages while maintaining a same neighborhood sampling radius.

24. The apparatus of claim 23, wherein the resolution increases from a first processing stage to a later processing stage.

25. The apparatus according to claim 22, further comprising: Means for varying a neighborhood sampling radius at each of the plurality of processing stages while maintaining the same resolution.

26. The apparatus of claim 22, wherein the learning-based correspondence estimation task comprises optical flow estimation, stereo estimation, simultaneous localization and mapping (SLAM), or multi-view stereo.

27. The apparatus according to claim 22, further comprising: Means for processing a level cost volume in each processing stage of the plurality of processing stages.

28. The apparatus according to claim 22, further comprising: A component for sampling a subset of pixels in the receptive field, and the learning-based correspondence estimation task is calculated based on the subset of pixels.

Images