Hierarchical data organization for compact optical streams

Abstract

This application relates to hierarchical data organization for dense optical flow. A computer vision system includes an image producing device for capturing successive two-dimensional (2D) images of a scene, a first memory for storing the successive 2D images (1000), a second memory for storing a growing window of successive rows of a reference image (1002) and a growing window of successive rows of a current image (1004), and a third memory for storing a sliding window of pixels extracted from the growing window of the reference image (1006). The reference image and the current image are a pair of successive 2D images stored in the first memory. The pixels in the sliding window are stored in tile form. A dense optical flow engine (DOFE) is configured to determine a dense optical flow map (1008) for the pair of successive 2D images. The DOFE uses the sliding window as a search window for pixel correspondence searches.

Description

[0001] Information related to divisional application

[0002] This application is a divisional application. Its parent application is the invention patent application filed on June 29, 2018, with application number 201880031199.0 and invention title "Hierarchical Data Organization for Compact Optical Flow". Technical Field

[0003] This invention generally relates to computer vision systems, and more specifically, to dense optical flow processing in such systems. Background Technology

[0004] Advanced driver assistance systems, or ADAS, are becoming increasingly common in automobiles. These systems provide drivers with information about their surroundings and may automatically intervene to prevent accidents and road traffic incidents. Three-dimensional (3D) scene understanding using image sensors is a fundamental step in building this understanding. Scene motion understanding using optical flow is a common technique for providing accurate motion estimation. To determine optical flow, a time series of images (videos) of the same scene are captured by a camera. Successive image pairs are processed to determine pixel displacements between consecutive images. The object motion / volume corresponding to the pixels in the captured scene can be analyzed using pixel displacement or flow vector information.

[0005] One approach to optical flow that can be used in embedded security systems is sparse optical flow. Sparse optical flow is a feature-based approach where only selected pixel flows across a continuous image are identified, such as image edges, corners, etc. A more accurate approach to optical flow is dense optical flow, where all pixel flows across a continuous image are identified. However, using dense optical flow in such systems is challenging due to the real-time processing requirements of embedded systems, as current methods for dense optical flow processing are computationally intensive and require significant data bandwidth. Summary of the Invention

[0006] The embodiments relate to methods and apparatuses for hierarchical data organization for dense optical flow processing in computer vision systems. In one aspect, a computer vision system is provided, comprising: an image generation apparatus configured to capture consecutive two-dimensional (2D) images of a scene; a first memory configured to store the consecutive 2D images; a second memory configured to store growth windows of consecutive rows of a reference image extracted from the first memory and growth windows of consecutive rows of a current image extracted from the first memory, wherein the reference image and the current image are a pair of consecutive 2D images; a third memory configured to store sliding windows of pixels extracted from the growth windows of consecutive rows of the reference image, wherein the pixels in the sliding windows are stored in the form of tiles; and a dense optical flow engine (DOFE) configured to determine a dense optical flow graph of the consecutive 2D images, wherein the DOFE uses the sliding windows as search windows for pixel correspondence searching.

[0007] In one aspect, a method is provided for operating a computer vision system to determine a dense optical flow graph of two consecutive two-dimensional (2D) images of a scene, comprising storing the two consecutive 2D images in a first memory of the computer vision system; maintaining a growing window of consecutive rows of a reference image of the two consecutive 2D images and a growing window of consecutive rows of a current image of the two consecutive 2D images in a second memory of the computer vision system, wherein the consecutive rows of the reference image and the consecutive rows of the current image are extracted from the first memory; maintaining a sliding window of pixels of the reference image in a third memory, wherein the pixels are extracted from the growing window of the consecutive rows of the reference image in the second memory, and wherein the pixels are stored in the sliding window in the form of tiles; and determining the dense optical flow graph of the two consecutive 2D images by a dense optical flow engine (DOFE) of the computer vision system, wherein the DOFE uses the sliding window as a search window for pixel correspondence searching. Attached Figure Description

[0008] Figure 1 This is a high-level block diagram of a system-on-a-chip instance.

[0009] Figure 2 yes Figure 1 A high-level block diagram of an example optical flow accelerator of a SOC.

[0010] Figure 3 illustrate Figure 2 The processing steps of the Compact Optical Flow Engine (DOFE) of the optical flow accelerator to generate a compact optical flow vector map.

[0011] Figure 4The instance space predictors used for predictor search in DOFE are shown.

[0012] Figure 5 This is a high-level block diagram of the DOFE instance architecture.

[0013] Figure 6 This shows an instance data hierarchy that supports DOFE optical stream processing.

[0014] Figure 7 This illustrates an example of a sliding search window used in DOFE's optical flow processing.

[0015] Figure 8 This is an example illustrating the tile organization of a sliding search window.

[0016] Figure 9 This is an example illustrating the best and worst case memory accesses using a tiled organization.

[0017] Figure 10 This is a flowchart of a method for determining a dense optical flow map. Detailed Implementation

[0018] In the drawings, similar elements are indicated by similar reference numerals.

[0019] Moving objects in the real world exhibit three-dimensional (3D) motion. A camera capturing a moving object captures a two-dimensional (2D) projection of the actual 3D motion. The 2D motion of the moving object can be determined by observing a series of captured images. The 2D motion of image pixels from image to image is called the optical flow of a pixel and is defined as the displacement vector of a pixel between two consecutive images. For compact optical flow, the optical flow of each pixel is determined.

[0020] Determining the optical flow of pixels is essentially about finding the "best correspondence" for a problem. Therefore, given two consecutive images, the problem is to find the best matching location of a pixel in the earlier image in the newer image. Generally, the search for the best-matching pixel is performed within a predetermined region in the newer image, known as the search range. Predictor-based search strategies can be used to avoid searching for a correspondence at every pixel within the search range.

[0021] The embodiments provide dense optical flow processing that meets real-time performance requirements in embedded computer vision systems. In some embodiments, a hardware accelerator for dense optical flow graph computation is provided. The hardware accelerator includes novel features that improve the performance of dense optical flow computation, such as: pixel-group-based search for matching pixels, which reduces search time; hierarchical data organization by tiling to manage data bandwidth; and / or advanced predictor evaluation that avoids data re-retrieval.

[0022] Figure 1This is a high-level block diagram of an instance of a system-on-a-chip (SOC) 100 configured to support computer vision processing in camera-based ADAS. The SOC 100 includes dual general-purpose processors (GPPs) 102, dual digital signal processors (DSPs) 104, a vision processor 106, and an optical flow accelerator 112 coupled via high-speed interconnects 122. As explained in more detail herein, the optical flow accelerator 112 is configured to perform dense optical flow processing on incoming camera image pairs. The vision processor 106 is a vector processor tuned for computer vision processing such as gradient computation, binning, and histogram normalization. The GPP 102 hosts the operating system and provides overall control over the operation of the SOC 100, including the arrangement of the processing performed by the optical flow accelerator 112. The DSP 104 provides support for computer vision processing such as object detection and classification, which can utilize the output of the optical flow accelerator 112.

[0023] The SOC 100 also includes a direct memory access (DMA) component 108, a camera capture component 110 that can be coupled to an external camera 124, a display management component 114, on-chip random access memory (RAM) 116, such as computer-readable media, and various input / output (I / O) peripherals 120 coupled to the processor and optical streaming accelerator 112 via interconnects 122. The camera 124 can be coupled to the SOC 100 via analog connections, Ethernet cables, low-voltage differential signaling (LVDS) cables, or other suitable wired or wireless connections. Furthermore, the SOC 100 includes a safety component 118, which includes safety-related functionalities that implement automotive safety compliance requirements. Such functionalities may include support for cyclic redundancy checks (CRC) on data, a clock comparator for drift detection, error signaling, a windowed watchdog timer, and self-testing for damage and failure of the SOC.

[0024] Figure 2 This is a high-level block diagram of example optical flow accelerator 112. Optical flow accelerator 112 includes configuration interconnect 206, hardware thread scheduler (HTS) 204, dense optical flow engine (DOFE) 202, memory mapped register (MMR) 208, shared memory interconnect 210, and shared memory 212.

[0025] Configuration interconnect 206 provides an interface for configuring parameters of optical flow accelerator 112. Configuration interconnect 206 also provides an interface for observation states for the processing performed by optical flow accelerator 112.

[0026] The HTS204 implements thread management and control triggering of the processing threads in the DOFE 202. The HTS204 also manages message passing and control between the optical flow accelerator 112 and other components in the SOC 100, such as the DSP 104.

[0027] Shared memory 212 stores input and output data for the Compact Optical Flow Engine 202. Shared memory 212 comprises four sets of static random access memory. Shared memory interconnect 210 is a crossbar with pipelined command and response handling. DMA 108 is connected to shared memory interconnect 210 and is used to move data processed by DOFE 202 into shared memory and to remove optical flow data generated by DOFE 202 from optical flow accelerator 112 for use by other components on SOC 100.

[0028] MMR 208 is configured to store various control and configuration parameters for DOFE 202. Parameters may include buffer size and address in shared memory 212, horizontal and vertical search ranges, confidence score configuration values, etc.

[0029] DOFE 202 is configured to find an optical flow vector map between all pixels of a pair of input images referred to herein as the reference image and the current image. The current image is the image received at time t-1, and the reference image is the image received at time t. The compact optical flow method implemented by DOFE 202 is based on a motion / flow vector predictor and uses a coarse-to-fine correspondence search strategy that leverages the image pyramids of the reference image and the current image.

[0030] The coarse-to-fine correspondence search is based on the fact that large pixel displacements between two consecutive images can be reduced to smaller displacements at lower spatial resolutions, thereby reducing the complexity of the correspondence search problem. For this search method, an image structure is created from each input image, scaled down sequentially. For example, a Gaussian image pyramid can be created by filtering the input image using a 5×5 Gaussian low-pass filter and then downsampling the filtered image in both the horizontal and vertical directions. The resulting image is then subjected to the same processing, and this loop is repeated until the desired number of pyramid levels are produced, for example, six levels. Each loop of this processing produces a smaller image with increased smoothness and reduced spatial sampling density, i.e., reduced image resolution.

[0031] In the resulting pyramid, assuming there are five higher levels, the original image can be referred to as the base pyramid level or level 0, and the smaller resolution images can be referred to as higher pyramid levels, such as levels 1 to 5. It should be noted that in the example Gaussian image pyramid, each pyramid level is 1 / 4 the size of the next lower level. Therefore, a pixel at a particular pyramid level corresponds to a 2×2 block of four pixels at the next lower pyramid level. Similarly, the distance measured at a higher pyramid level is half the distance at the next lower pyramid level.

[0032] Figure 3 This diagram illustrates the processing steps of DOFE 202 in generating a dense optical flow vector map. This diagram assumes the image pyramid has five upper levels as described above. More or fewer levels can be used. Each pyramid level is then processed to generate an optical flow vector for each pixel in the image, starting from the highest pyramid level. At each upper pyramid level, a predictor-based search strategy 300 is applied to find an initial match, which is then further refined using a hierarchical step-size search strategy 302 to generate an integer-pixel accurate flow vector for each pixel. A median filter 304 is applied to the flow vector to remove impulse estimation noise. The filtered flow vector is scaled up by a factor of 306 and used as the initial predictor at the next lower pyramid level.

[0033] At the basal pyramid level, a predictor-based search strategy 308 is applied to the original image to find initial matches, which are then further refined using a hierarchical step-size search strategy 310 to produce estimated integer-pixel accurate flow vectors for each pixel. After determining the integer-pixel flow vectors, the pixel correspondences are further refined 312 to sub-pixel level accuracy. A median filter 314 is applied to the refined flow vectors to remove impulse estimation noise. Any suitable median filter can be used. A confidence map 316 is generated by calculating a quantitative estimate of the correctness, called a confidence score, for each final estimated flow vector. The final estimated flow vectors and the confidence map are then packaged together to form a dense optical flow map, which is stored in external memory. Any suitable technique for calculating the confidence score can be used.

[0034] Predictor searches 300, 308 and step-size searches 302, 310 at each pyramid level search point find the best-matching pixel in the reference image for each pixel in the current image. For a pixel in the current image, the search for the best-matching pixel in the reference frame is limited to the search window in the current frame. The best match is determined based on the matching cost calculated for each candidate matching pixel in the current and reference images. Before calculating the matching cost, a census transform is applied to the current pixel value and the candidate matching reference pixel value to reduce errors from illumination variations, and a census signature is calculated for each pixel based on the corresponding transformed value. The matching cost is then measured based on the Hamming distance between the census signatures of the pixels. Any suitable census transform and method for calculating the census signature and Hamming distance can be used.

[0035] Predictor searches 300 and 308 use predictor trimming of the entire search window to avoid searching for correspondences at every pixel within the search window. These searches fundamentally assume the spatial and temporal locality and smoothness of motion; that is, pixels in small neighborhoods are very likely to have similar motions, and the motion changes smoothly in both space and time. Furthermore, instead of searching every pixel in the current frame, the search is performed on non-overlapping pixel blocks, which may be referred to in this paper as pixel groups. For example, pixel groups are assumed to be 2×2 pixel blocks. Other suitable block sizes can be used.

[0036] In predictor searches 300 and 308, it is assumed that each 2×2 pixel block at any pyramid level can be predicted using a set of flow vector values ​​selected from the actual / estimated flow vectors of the block's spatial or temporal neighbors; these flow vector values ​​can be referred to as flow vector predictors. A predictor is defined with reference to the top-left pixel of the 2×2 pixel block, which may be referred to herein as the anchor pixel. Any suitable type and / or number of predictors can be used. In some embodiments, five spatial predictors and a single temporal predictor are used. Figure 4 The diagram describes five spatial predictors. These spatial predictors are the flow vectors of the pixel to the upper left (TL) of the anchor pixel, the pixel to the top (T) of the anchor pixel, the pixel to the upper right (TR) of the anchor pixel, the pixel immediately to the left of the anchor pixel estimated in the previous pyramid level, and the pixel in the same position as the anchor pixel estimated in the previous pyramid level. The temporal predictor is the flow vector of the pixel in the same position as the anchor pixel estimated in the previous image.

[0037] In some embodiments, an additional spatial predictor is used, which is the flow vector of the pixels n pixels to the left of the anchor pixel in the same row. The value of n depends on the pipeline depth of the DOFE 202. This predictor may be referred to as a delayed left-side predictor.

[0038] Spatial predictors are used at all pyramid levels, and temporal predictors are used at the base pyramid level. The predictor that generates matching candidates with the minimum matching cost is selected as the winning predictor for the pixel group. Step searches 302 and 310 are then applied around the winning predictor from predictor searches 300 and 308 to further refine the matching positions. Step searches 302 and 310 aim to refine the optimal matching position from the winning predictor within a finite two-dimensional (2D) region around the winning predictor's position. Step searches 302 and 310 are performed for each pixel in the pixel group to determine the flow vector for each pixel.

[0039] In some embodiments, a hierarchical step-size search technique known as 3-1 step-size search is used. Initially, eight pixel locations are searched around the winning predictor location. Eight pixel locations are selected such that these locations are three pixels away from the winning predictor location in each of eight directions: north, northeast, east, southeast, south, southwest, west, and northwest. The best matching location from this search is further refined by selecting eight similarly positioned pixels at a one-pixel distance from the best matching location. The best matching location from this one-pixel search is selected as the final integer-pixel accurate flow vector.

[0040] Figure 5 yes Figure 2 A high-level block diagram of the instance architecture of DOFE 202. Census Transformation Component 506, Predictor Search Engine (PSE) 504, and Step Search Engine (SSE) 502 are coupled in series and configured to perform the previously described census transformation, predictor search, and step search on the current and reference pixel data stored in the search range buffer 508. Subpixel Refinement Component 518 is configured to perform the previously described subpixel refinement on the best matching position produced by the step search performed at the base pyramid level. Median Filter Component 514 is configured to perform the previously described median filtering, and Confidence Score Component 516 is configured to generate a confidence score for the selected best matching position.

[0041] Output packing component 520 is configured to pack a series of flow vectors from median filter component 514 and a series of corresponding confidence scores from confidence score component 516 to generate a flow vector graph, which is stored in output buffer 522. Shared memory interface 512 is configured to read the flow vector graph from output buffer 522 and store the graph in shared memory 512 via shared memory interconnect 210. DMA 108 can then transfer the flow vector graph from shared memory 512 to external memory, for example... Figure 1 RAM 116 on the chip.

[0042] To optimize the access bandwidth required for the search of corresponding relationships, a hierarchical data organization is used. For example... Figure 6 As illustrated, in some embodiments, the data hierarchy comprises three memory levels, wherein the storage capacity gradually decreases from level 3 (L3) to level 1 (L1), while the access bandwidth gradually increases. The L3 memory may be external to the optical flow accelerator 112, such as on-chip RAM 116, while the L2 and L1 memories are contained within the optical flow accelerator 112, serving as shared memory 212 and search range buffer 508, respectively.

[0043] In this hierarchy, the L3 memory stores the reference image and the current image, the L2 memory stores a subset of concurrent pixel rows for each image, and the L1 memory stores a search window extracted from the reference image rows in the L2 memory. Furthermore, a subset of the reference image and the current image in the L2 memory can be maintained according to a growing window method, in which a circular buffer of consecutive image rows read from the L3 memory is maintained for each image in parallel with optical stream processing. The number of rows in each of the circular buffers can be any suitable number. In some embodiments, the minimum number of rows in the growing window can be based on the number of pixels required for the census transform and the number of pixels required to calculate the Hamming distance and the vertical search range. In some embodiments, image rows in pixel group rows are extracted from the L3 memory; for example, if the pixel group is 2×2, then two rows are extracted.

[0044] like Figure 7 As described, a search window in L1 memory, such as a search range buffer 508, can be maintained according to a sliding window method, wherein a cyclic buffer of consecutive columns of pixel patches read from the growth window of the reference image is maintained in parallel with the optical stream processing. Pixel group processing moves to the right in raster scan order, adding new columns of pixel patches while discarding the same number of columns of pixel patches. The size of the sliding window can be based on the search range used for correspondence matching.

[0045] Using tile organization in the L1 memory search window minimizes the number of memory accesses required to perform correspondence searches. Candidate pixel locations in the reference image require pixel regions of a specified size, determined by the census transform used and the matching cost calculation. Because motion is unrestricted, the flow vector and predictor can point to any pixel location within the search range. Therefore, the pixel region used for cost calculation can be centered on any pixel. If the pixels are in raster scan order, access to all pixels in the pixel region may require multiple accesses. Arranging the reference image pixels into pixel tiles reduces the number of required accesses.

[0046] Furthermore, tiles are "striped" across multiple memory banks in L1 memory. Any suitable number of memory banks can be used. Each memory bank is sized to store multiple complete tiles, and tiles are stored in the memory banks so that they can be accessed in a single loop. The specific arrangement of tiles on the memory banks may depend on, for example, the number of available memory banks and the size of the search window.

[0047] consider Figure 8 An example is provided. In this example, a 4×4 tile and four memory banks are assumed. Other tile sizes and / or other memory organization embodiments are possible. A small example image 802 is shown, where pixels arranged in raster scan order are logically divided into 4×4 tiles. An example L1 memory 804 is also shown. With this pixel organization, pixel areas can be decomposed according to 4×4 tiles and read through a finite number of accesses. For example, such as... Figure 9 As explained, there is a 14×142D area in sixteen (best case) or twenty-five (worst case) 4×4 pixel tiles based on the location of the search vector, therefore sixteen (best case) or twenty-five (worst case) accesses are required to extract the desired pixel area. In another example, there may be sixteen L1 memory banks, where vertically adjacent 4×4 tiles are placed in different groups.

[0048] Return to reference Figure 5 The shared memory interface 512 is further configured to read pixel data of a reference frame from a growth window in shared memory 212 via shared memory interconnect 210. The pixel data is provided to the line-to-block conversion component 510 for processing and storage in the search range buffer 508. The line-to-block conversion component 510 is configured to convert the pixel data read from shared memory 212 into pixel tiles for storage in the search range buffer 508. Any suitable pixel data size and pixel tile size can be used.

[0049] In some embodiments, in addition to performing a step-size search, SSE 502 also performs high-level predictor evaluation for some of the predictors used by PSE 504. Therefore, after determining the winning flow vector for each pixel in the current pixel group, SSE 502 evaluates the matching cost of any of these flow vectors that will be used as predictors for the next row of pixel groups. The flow vectors and their matching costs are stored, for example, in a first-in-first-out buffer for use by PSE 504. In some embodiments, a single matching cost, i.e., the minimum matching cost of a flow vector, is stored.

[0050] For example, assuming a 2×2 pixel group and the previously mentioned instance predictor, the flow vectors of the bottom two pixels of the current pixel group can be used as the top and upper right predictors for the pixel group immediately below the current pixel group. Therefore, SSE 502 evaluates the matching cost of these two flow vectors relative to the pixel that will be the anchor pixel of the pixel group immediately below. Furthermore, the bottom right pixel of the current pixel group can be used as the upper left predictor of the pixel group to the lower right of the current pixel group. Therefore, SSE 502 evaluates the matching cost of the flow vectors relative to the pixel that will be the anchor pixel of the pixel group to the lower right.

[0051] Figure 10 It is used to determine, for example Figure 1 , 2 The flowchart illustrates a method for determining a dense optical flow graph in a computer vision system, as described in section 5. To determine the dense optical flow graph, a reference image and a current image are stored in L3 memory (1000). Furthermore, during the process of determining the dense optical flow graph, a growth window (1002) of consecutive rows of the reference image is maintained in L2 memory, and a growth window (1004) of consecutive rows of the current image is also maintained in L2 memory. A sliding window of pixel patches formed from pixels read from the growth window of the reference image is also maintained in L1 memory. The sliding window is used as a search window for searching pixel correspondences to determine the dense optical flow graph (1008).

[0052] Other embodiments

[0053] For example, embodiments have been described herein with reference to ADAS. However, embodiments for computer vision applications other than ADAS are possible, such as industrial applications, robotics, and drones.

[0054] In another example, an embodiment has been described herein in which the anchor pixel of the pixel group is assumed to be the top-left pixel. However, it is possible for the anchor pixel to be another pixel in the pixel group.

[0055] In another example, an embodiment in which a current image and a reference image are captured by a camera has already been described herein. However, it is possible to use another image generating device to generate the current image and the reference image. For example, a graphics processor can generate sequentially composited images of a scene, for example, for use in games or virtual reality applications.

[0056] In another example, embodiments have been described herein in the context of processing sequential 2D pixel images. However, embodiments are possible where the input is a pair of current 2D images and a reference 2D image of data points and the output is the displacement of each data point between the current 2D image and the reference 2D image. Such 2D images of data points may include non-sequential camera images where pixels are data points, thermal maps where temperature is the data points, pressure maps where pressure measurements are the data points, occupancy area maps where the occupied area is the data points, etc.

[0057] Although method steps may be presented and described in a sequential manner herein, one or more of the steps shown in the figures and / or described herein may be performed simultaneously, combined, and / or performed in an order different from that shown in the figures and / or described herein. Therefore, embodiments are not limited to the specific order of the steps shown in the figures and / or described herein.

[0058] In this specification, the term "coupled" and its derivatives mean indirect, direct, optical, and / or radio connection. Thus, if a first device is coupled to a second device, the connection can be, for example, a direct electrical connection, an indirect electrical connection via other devices and connections, an optical electrical connection, and / or a radio connection.

[0059] Modifications to the described embodiments are possible within the scope of the claims, and other embodiments are also possible.

Claims

1. A method for dense optical flow, comprising: Receive a first image containing a first set of pixels and a second image containing a second set of pixels, wherein the first image and the second image are two consecutive images, and the first image follows the second image in the time dimension; Divide the first pixel set into a pixel subset; For each of the pixel subset: Select the anchor pixel from the corresponding subset of pixels; For each of a set of predictors: The corresponding predictor is applied to the anchor pixel to determine the predicted position in the second pixel set; and Determine the matching cost for the predicted location; and Based on the matching cost, the closest matching region is determined for the corresponding subset of pixels in the second pixel set.

2. The method of claim 1, wherein the set of predictors for the anchor pixel includes spatial predictors for pixels in the first pixel set, wherein the pixels in the first pixel set are adjacent to the anchor pixel, and the pixels in the first pixel set are not in the corresponding pixel subset of the anchor pixel.

3. The method of claim 1, wherein the set of predictors for the anchor pixel comprises: A first spatial predictor for a first pixel, wherein the first pixel is directly adjacent to the anchor pixel; A second spatial predictor for a second pixel, wherein the second pixel is directly obliquely adjacent to the anchor pixel; and A third spatial predictor for a third pixel, wherein the third pixel is directly obliquely adjacent to the anchor pixel.

4. The method of claim 1, wherein the set of predictors for the anchor pixel includes spatial predictors for pixels in a third image, wherein the third image is obtained by downsampling the first image.

5. The method of claim 1, wherein the set of predictors for the anchor pixel includes temporal predictors for pixels in the second pixel set, wherein the pixels in the second pixel set are at the same position as the anchor pixel.

6. The method of claim 1, wherein determining the nearest matching region for the corresponding subset of pixels comprises determining the nearest matching predictor among the set of predictors based on the matching cost of the set of predictors.

7. The method according to claim 6, wherein: Determining the closest matching region for the corresponding subset of pixels includes applying the closest matching predictor to the corresponding subset of pixels to determine the closest matching region; and The method further includes performing a step-size search around the nearest matching region to determine a flow vector for each pixel in the corresponding subset of pixels.

8. The method of claim 7, further comprising providing the flow vector for determining the optical flow of the third image, wherein the first image is obtained by downsampling from the third image.

9. The method of claim 7, further comprising: The flow vector is filtered to produce a filtered flow vector; The filtered stream vector is scaled to produce a scaled stream vector; and The scaled flow vector is provided to determine the optical flow of the third image, wherein the first image is obtained by downsampling from the third image.

10. The method of claim 1, wherein each of the pixel subsets is a 2×2 non-overlapping pixel set.

11. A circuit device comprising: Predictor search engines, configured to: Receive a first image and a second image, wherein the first image and the second image are two consecutive images, and the first image follows the second image in the time dimension; Divide the first image into a subset of pixels; For each of the pixel subset: Select the anchor pixel from the corresponding subset of pixels; For each of a set of predictors: The corresponding predictor is applied to the anchor pixel to determine the predicted position in the second image; and Determine the matching cost for the predicted location; and Based on the matching cost, the closest matching predictor is determined for the corresponding subset of pixels in the second image; and A step-size search engine coupled to the predictor search engine, the step-size search engine being configured such that, for each of the pixel subset: The nearest-match predictor is applied to the corresponding subset of pixels to determine the predictor position; and A step-size search is performed around the predictor position to determine the optical flow vector for each pixel in the corresponding subset of pixels.

12. The circuit arrangement of claim 11, further comprising: A median filter component coupled to the step-size search engine is configured to apply a median filter to the optical flow vector.

13. The circuit arrangement of claim 11, wherein the circuit arrangement is configured to: Scale the optical flow vector to produce a scaled optical flow vector; and The scaled optical stream vector is provided to the predictor search engine for use in a set of predictors for the third image.

14. The circuit device of claim 13, wherein the first image is obtained by downsampling the third image.

15. The circuit arrangement of claim 11, wherein the set of predictors for the anchor pixel includes spatial predictors for pixels in the first image, wherein the pixels in the first image are adjacent to the anchor pixel, and the pixels in the first image are not in the corresponding subset of pixels of the anchor pixel.

16. The circuit arrangement of claim 11, wherein the set of predictors for the anchor pixel comprises: A first spatial predictor for a first pixel, wherein the first pixel is directly adjacent to the anchor pixel; A second spatial predictor for a second pixel, wherein the second pixel is directly obliquely adjacent to the anchor pixel; and A third spatial predictor for a third pixel, wherein the third pixel is directly obliquely adjacent to the anchor pixel.

17. The circuit arrangement of claim 11, wherein the set of predictors for the anchor pixel includes spatial predictors for pixels in a third image, wherein the third image is obtained by downsampling the first image.

18. The circuit arrangement of claim 11, wherein the set of predictors for the anchor pixel includes temporal predictors for pixels in the second image, wherein the pixels in the second image are at the same position as the anchor pixel.

19. The circuit arrangement of claim 11, wherein each of the pixel subsets is a 2×2 non-overlapping pixel set.

Images