Image processing system and method for optical flow estimation

Abstract

The present application embodiment relates to an image processing system and method for optical flow estimation. An image processing system (100) includes a processor (102) and an optical flow OF determination logic (104), the optical flow OF determination logic is used to quantify the relative motion of features in a first frame of a video and a second frame of the video, and provide at least one of a temporal and spatial ordering image (110) relative to the two frames of the video. The OF determination logic (104) configures the processor (102) to perform OF estimation between the first frame and the second frame using a pyramid block matching (PBM) method to generate an initial optical flow OF estimation value with integer pixel resolution at a base pyramid level, and refines the initial OF estimation value using at least one pass of a modified Lucas-Kanad LK method to provide a modified OF estimation value with fractional pixel resolution.

Description

[0001] Information about divisional applications

[0002] This application is a divisional application of the invention patent application with the application date of December 5, 2016, application number "201680064214.2", and invention name "Quasi-parametric Optical Flow Estimation". Technical Field

[0003] The present invention relates to optical flow estimation involving the use of pyramidal block matching. Background Art

[0004] Optical flow (or optical flow) is the observed motion of objects in a sequence of images due to the relative motion between an optical sensor (e.g., a camera) and the objects present in the images. The term optical flow is often applied in the field of computer vision to incorporate related techniques from image processing and navigation control (e.g., motion detection, object segmentation, contact time information, extended focus calculation, illumination, motion compensated coding, and stereo disparity measurement). Such techniques are particularly useful for automotive driver assistance systems, robotics, and other applications that apply machine vision.

[0005] Searching for the best matching patch between two arrays of image data is a necessary step in image processing. For example, some stereoscopic imaging systems compute the disparity between a left image and a right image by finding a 2D patch in the right image that best matches a given 2D patch in the left image. In another example, alignment of two three-dimensional (3D) point clouds can be achieved by searching for the best 3D patch match between volumes. In another example, a video compression algorithm can determine motion between two consecutive images using an optical flow algorithm that matches patches between two images.

[0006] A coarse-to-fine resolution pyramid approach can be used for optical flow algorithm matching. Generally speaking, in a pyramid approach, an initial search is performed at a resolution lower than the original image, and then the initial search results are refined at one or more higher resolutions. The several resolution levels in the search pyramid are implementation dependent. Compared to an exhaustive search at high resolution, the use of a pyramid search approach is generally faster and has a higher tolerance to local minima.

[0007] Camera-based systems use a variety of computer vision (CV) techniques to implement advanced driver assistance systems (ADAS), which are designed to increase the driver's situational awareness and road safety by providing necessary information, warnings, and automatic intervention to reduce the likelihood or severity of accidents. Government safety regulations and independent rating systems drive the development and wider adoption of ADAS, with camera-based systems becoming a key differentiator between original equipment manufacturers (OEMs). Camera-based systems are widely used in ADAS because of their reliability robustness, ability to support multiple applications, and most importantly, the flexibility to support more and more ADAS applications in the future. CV techniques represent complex, high-performance, and low-power computing problems, especially low-level CV techniques that extract high-definition, high-density depth (stereo) and motion (optical flow) information from camera images. Summary of the invention

[0008] The described example includes an image processor having optical flow logic implementing a quasi-parametric optical flow measurement (QP-OFM) algorithm that combines a pyramidal block matching (PBM) method (which is a parametric method) with a modified Lucas-Kanade (LK) method (which is a parametric method) to obtain a high-precision estimate over a large range of optical flow (OF). The PBM method performs OF estimation with integer pixel resolution and then refines the OF estimate obtained by the PBM using at least one pass of the modified LK method to obtain a corrected optical flow estimate with fractional pixel resolution. One pass of the modified LK method generally provides a good cost-benefit balance because it does not require interpolation and data re-extraction.

[0009] The described example includes an image processing system including a processor and OF determination logic for quantifying relative motion of features present in a first frame of a video (e.g., a query image) and a second frame of the video (e.g., a reference image), providing at least one of a temporally and spatially ordered image relative to the two frames of the video. The OF determination logic configures the processor to perform OF estimation between the first frame and the second frame using a PBM method to produce an initial OF estimate with integer pixel resolution at a base (lowest) pyramid level, and to refine the initial OF estimate with a modified LK method to provide a revised OF estimate with fractional pixel resolution. The QP-OFM algorithm significantly simplifies and improves a PBM-based dense OF (DOF) estimation algorithm. The example described below exhibits improved performance with respect to known DOF estimation algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram of a system that implements the disclosed QP-OFM algorithm for determining and applying OF, according to an example embodiment.

[0011] Figure 2 is a flow chart illustrating steps in an example method of performing optical flow measurement using the disclosed QP-OFM according to an example embodiment.

[0012] Figure 3 is a simplified block diagram of a disclosed example QP-OFM arrangement.

[0013] Figure 4 A method for computing a 24-bit census descriptor for a central pixel x, taking into account its 24 neighbors (5x5 neighborhood) and their relationships, according to an example embodiment is described.

[0014] Figure 5 Illustrate the 5×5 neighborhood pixels used in the census signature calculation for a pixel.

[0015] Figure 6 Description Example support window.

[0016] Figure 7 An example predictor list for an optical flow estimate containing the 4 neighbors of the top left pixel in a pixel group is shown.

[0017] Figure 8 An illustration of predictor evaluation involving 5 candidate predictors for the current pixel group and a winner predictor therebetween is provided.

[0018] Fig. 9 A coarse-to-fine search pattern around a winner predictor is shown.

[0019] Fig. 10A Depicts an example SoC that provides high programmable computing power by offloading low-level compute-intensive and repetitive CV tasks to a hardware accelerator (HWA).

[0020] Fig. 10B Implementation details of an example scheme are described as follows: storing and handling reference and current picture data in local memory within a HWA or on a programmable processor, which involves using a combination of two levels of local memory, one memory being used to store picture data from a first frame of a video and from a second frame in a growing window manner, and another memory storing a sliding window of picture data from the second frame of the video. DETAILED DESCRIPTION

[0021] In the drawings, similar reference numerals are used to designate similar or equivalent elements. Some actions or events may occur in different orders and / or simultaneously with other actions or events. In addition, some of the actions or events described may not be required to implement the method according to this specification.

[0022] In this description, the terms "coupled to" or "coupled with..." (etc.), as used herein without further limitation, are intended to describe either an indirect or direct electrical connection. For example, if a first device is "coupled" to a second device, the connection may be through a direct electrical connection in which only parasitic effects exist in the path, or through an indirect electrical connection via intermediaries that include other devices and connections. For indirect coupling, the intermediaries generally do not modify the information of the signal, but may adjust its current level, voltage level, and / or power level.

[0023] Figure 1 A block diagram of a system 100 for determining and applying OF according to a number of disclosed embodiments is shown. The system 100 includes at least one image sensor 106, an image processor 102, and a control system 108. The image processor 102 may be a microprocessor, a digital signal processor (DSP), or a microcontroller unit (MCU). The image processor 102 includes one or more processors and a storage device. The image sensor 106 may include a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS) image sensor, or other photodetector for converting light into an electrical signal. The image sensor 106 may include a plurality of photodetectors arranged in a two-dimensional array. The image sensor 106 may periodically capture an image representing a field of view of the image sensor 106. For example, the image sensor 106 may capture 15, 30, 60, or any suitable number of images per second. The image sensor 106 may be incorporated into a digital camera. Some disclosed embodiments may include a plurality of image sensors 106, such as when a plurality of image sensors 106 are used.

[0024] The image captured by the image sensor 106 may be provided to the image processor 102 as an array of one or more binary values, where each binary value may represent the intensity or color of light detected at a particular photodetector (i.e., a picture element (pixel)) of the image sensor 106. Each image provided by the image sensor 106 to the image processor 102 may be referred to as a frame. The image processor 102 analyzes or manipulates the image 110 received from the image sensor 106 to extract information from the image 110. The image processor 102 includes Figure 1 OF determination logic (optical flow logic), shown as "Optical Flow" 104 in FIG. 1 , analyzes an image 110 received from an image sensor 106 to measure the optical flow of a plurality of elements or features present in the image 110. As described above, the described image processor 102 may be implemented by a HWA.

[0025] The optical flow logic 104 applies a disclosed QP-OFM algorithm, which (as described above) combines the PBM method with the modified LK method to obtain accurate and precise OF estimates over a large range (i.e., multiple different distance values). The PBM method performs OF estimates at integer pixel resolution and then uses at least one pass of the modified LK method to refine the obtained OF estimates to obtain modified OF estimates with fractional pixel resolution. The refined optical flow estimates may be filtered using a post-processing filter to filter out estimates that may be noisy. In the HWA implementation described below, typically only one modified LK pass is used. However, in other implementations, it is generally best practice to seek a balance between computational complexity and benefit, because more iterations may require interpolation and repeated fetching of pixel data from memory.

[0026] The PBM method as used herein refers to an OF estimation method that converts each of the frames of a video into a hierarchical image pyramid. The image pyramid includes a plurality of image levels. The image resolution is reduced with each higher one of the image levels. For each image level and for each pixel in the first frame, the processor is configured to establish an initial estimate of the position of the pixel in the second frame as a predefined value or a candidate position that minimizes a cost function, and to apply a plurality of sequential searches starting from the initial estimate that minimizes the cost function, and establish a refined estimate of the position of the pixel in the second frame.

[0027] As used herein, the LK method refers to a modified version of the differential OF estimation method developed by Bruce D. Lucas and Takeo Kanade. The LK method assumes that the optical flow is substantially constant in the local neighborhood of the pixel under consideration, and solves the underlying optical flow equation for all pixels in the neighborhood by a least squares criterion. By combining information from several nearby pixels, the LK method can resolve the inherent ambiguity of the optical flow equation.

[0028] The LK method can use the following relationship to solve the OF equation defined for a pixel in the image using gradient information from the neighborhood of the pixel and the least squares principle:

[0029]

[0030] where the horizontal and vertical flow components are U and V, respectively, Gx, Gy are the horizontal and vertical spatial gradients computed for the first frame, and Gt is the computed temporal gradient between the two images between which OF is estimated. The conventional LK method computes OF pixel by pixel in each iteration after performing warping (motion compensation) of the second image. Gx, Gy are defined for all pixels within an n×n (n=3, 5, 7, etc.) neighborhood of a pixel, and The operator defines an element-wise multiplication of 2D vectors. If the 2×2 matrix in the above equation is non-invertible, ie if its determinant is zero, then it can be regularized or U and V can be set to zero.

[0031] With respect to the modified LK method for use with the disclosed QP-OFM algorithm, the inputs are the same Gx, Gy, and Gt as described above but calculated during a step search process (described below) to obtain the horizontal and vertical flow components U and V, respectively. The U and V flow components are represented as real numbers in a fixed point representation, for example, with 4 bits assigned to store the fractional part. These flow values ​​represent the existing OF estimate F1 int The incremental refinement (obtained by the PBM method at integer pixel resolution) referred to as the "initial OF estimate" in the method 200 described below, undergoes processing to achieve the "revised OF estimate" with higher accuracy and fractional pixel resolution in the method 200. The values ​​of U and V may be clamped in the range [-1, 1] before they are added to the existing flow estimate to obtain the final flow output F1. This operation can be expressed in equation form as:

[0032] F1=F1 int +Clamp((U,V),-1,1)

[0033] In one embodiment, the U and V computations may be implemented using only simple integer arithmetic operations of multiplication, addition, and comparison as shown below, followed by clamping operations in fixed point (including division operations involved in inverse matrix computations using the transposed adjoint matrix).

[0034]

[0035]

[0036]

[0037]

[0038] in

[0039]

[0040]

[0041]

[0042] as well as

[0043]

[0044] All possible values ​​of U and V can be represented by a suitable fixed point representation with a 4-bit fractional part.

[0045] The disclosed modified LK method removes the image warping used in the known LK method described above. Assuming OF is substantially constant in a local neighborhood and equal to the pixel under consideration, the image warping step is removed by using one pass. Significantly, this allows the reuse of pixel data extracted during the step search process for Gt calculation, which is particularly useful in situations where random access to memory can be avoided. As described above, the modified LK method also splits the calculation into subtasks of spatial gradient calculation and remaining operations.

[0046] A post-processing filter (see below) can be used Figure 3 The corrected OF measurements are filtered by a post-processing filter 306 in the image processor 102 to obtain a final output OF estimate 112 generated by the image processor 102 that can be provided to the control system 108. The control system 108 can apply the corrected OF measurements 112 to control the motion of the system 100, present motion information to a user of the system 100, etc. For example, if the system 100 is an advanced driver assistance system (ADAS), the control system 108 can apply the corrected OF measurements 112 to determine whether the vehicle should change speed and / or direction based on the relative motion of the vehicle and objects detected by the image processor 102. In some ADAS implementations, the control system 108 can autonomously change the vehicle speed and direction based at least in part on the corrected OF measurements 112, while in other embodiments, the control system 108 can provide an alert to the operator of the vehicle based on the corrected OF measurements 112, indicating that a change in speed and / or direction may be desirable. Similarly, in robotics and other motion control applications, control system 108 may control the movement (speed and / or direction) of elements of system 100 based on corrected optical flow measurements 112 .

[0047] Figure 2 is a flow chart illustrating steps in an example method 200 for OF measurement using the disclosed QP-OFM according to an example embodiment. In this example, the modified LK method is applied only at the base level of the pyramid. Step 201 includes acquiring a first frame of a video (e.g., a reference image) and a second frame of the video (e.g., a query image), the first and second frames providing at least one of a temporally and spatially ordered image. Optionally, stored historical flow estimates or auxiliary estimates derived from a parametric model of image velocity may also be provided as a prior indication of image velocity used as a predictor during the PBM process. In normal operation, when the given input is two temporally ordered images, the QP-OFM algorithm estimates the motion as the instantaneous image velocity (pixel motion) and optionally together with the prior / auxiliary flow estimates that provide an indication of the image velocity.

[0048] Step 202 includes using a processor to implement the stored QP-OFM algorithm that performs steps 203 and 204. Step 203 includes performing optical flow estimation between a first frame and a second frame using a PBM method to produce initially obtained flow estimate values ​​with integer pixel resolution. The PBM method initially performs optical flow estimation at the highest pyramid level using a block matching (BM) process based on a stepwise search method (described below) that minimizes a cost function value to obtain accurate motion estimation. These optical flow estimate values ​​are then filtered (e.g., using a 2D 5×5 median filter), appropriately scaled up and then further refined in sequence at each lower pyramid level (again using a spatial predictor and a stepwise search method) until the base pyramid level. No filtering or scaling of the OF estimate values ​​is required after the BM process at the base level. In one embodiment, a suitable cost function is the Hamming distance across binary feature descriptors.

[0049] At the base pyramid level, the predictor configuration may be changed to include a temporal predictor or an auxiliary predictor, but the step search may remain the same. At the base pyramid level, after the PBM process is completed, a pass of the modified LK step (step 204 described below) is performed to refine the obtained flow estimate to obtain a modified optical flow estimate with fractional pixel resolution, thereby enabling the determination of accurate pixel motion. According to another embodiment, a pass of the modified LK method may also be applied while processing higher pyramid levels after the BM process.

[0050] To reduce computation and data bandwidth, instead of BM processing each pixel, at each pyramid level, the BM process can operate on a 2x2 pixel group (2x2 neighboring pixel block) granularity, where different computation blocks (e.g., predictor evaluation and step search) can operate on 4 pixels simultaneously. This approach can also be applied to all possible pixel group configurations, including 1x1 (=pixel), 3x3, 2x4, etc., although experiments have found that 2x2 pixel groups are optimal for the desired quality and HWA efficiency. In this scheme, the predictor is evaluated for only one representative pixel in the pixel group to determine the best estimate of OF (the winner predictor), which is then refined independently for all pixels in the group during the step search process.

[0051] Stream post-processing involves using a 2D median filter of size 5×5 (height×width) (see Figure 3306 in the flow post-processing filter 306). The post-processing block uses the OF estimate (Fi) as input and produces a filtered OF output (F). Alternatively, a pseudo 1D separable median filter of size 9×9, for example, may be used. The 9×9 filter may be separated into 3×9 and 9×3 filters that are applied sequentially (or reversibly) in the order described. The height and width of the filter are configured so that the number of flow samples (27) used in the median calculation is small but provides high quality post-processing. In the case of this filter, a boundary pixel extension of the flow field on all sides by 4 pixels is performed. The confidence assignment block (see description below) Figure 3 The confidence assignment block 307 in can use the cost function value calculated during the PBM step and the local variability of the flow estimation value obtained to assign the confidence level of each pixel in a learning based framework.

[0052] Step 204 includes refining the initial flow estimate obtained by the PBM using at least one pass of the modified LK method to provide a modified OF estimate with fractional pixel resolution. As described above, the modified LK method modifies the known LK algorithm to remove the conventional image warping step and splits the calculation steps into two sets of gradient calculation (spatial and temporal) operations and the remaining operations. This LK step therefore performs fractional pixel refinement again using the data extracted by the PBM method. The LK step can use the reduced precision gradient data for OF refinement. The LK step can also perform computational tasks involving matrix inversion using only integer arithmetic.

[0053] As described above, the method may optionally include post-processing filtering (see Figure 3 A flow post-processing filter 306 in , which filters the flow vector output by the QP-OFM algorithm block 304). For example, a pseudo 1d separable median filter may be used to remove impulse noise from the estimated flow vector.

[0054] Figure 3 A simplified block diagram of an example QP-OFM arrangement 300 is shown, which includes an input 310 to an OF computation block 302 , which includes a QP-OFM algorithm block 304 , where the OF computation block 302 provides an output 312 .

[0055] The inputs 310 are typically two grayscale images. Any of the chrominance channel information represented as one binary value per pixel may also be used. A query image is shown as image 1 and a reference image is shown as image 2, between which the measurement / estimate OF is arranged by a QP-OFM scheme. Previous / auxiliary optical flow estimates are shown provided as other inputs. Such previous estimates are temporally preceding the current optical flow estimate and are typically provided at the same resolution as the input images (image 1 and image 2) and have 100% expected density estimated for each pixel or group of pixels.

[0056] The image pyramid generation block (shown as PyramidGeneration) 304a receives an input 310 comprising a grayscale image (I) comprising image 2 and image 1, a programmed number of pyramid levels N, and a sigma value (Sg) to be used to derive a Gaussian filter kernel (e.g., from a processor). The image pyramid generation block 304a outputs an image pyramid of an N-level multi-resolution representation of the image such that each nth image in the pyramid is one-half the resolution in the horizontal and vertical dimensions of the image at the (n-1)th level. The image pyramid generation block 304a typically provides pre-processing. In some cases, if the input image has dimensions W×H and the image pyramid is to have N levels, the image may be padded on the bottom and right so that the resulting image dimensions W'×H' are 2 (N-1) or 2 (N) The padded pixel may have a value of 0. Assume that the padded image is regarded as I'.

[0057] An example method of the image pyramid generation block 304a is described below:

[0058] Assume (let) P i represents the image in the i-th pyramid level.

[0059] Initialization: P1 = I', derive a 2D Gaussian filter core G of size 5×5

[0060] For (FOR)(i=2; i<=N; i++)%, i represents the level in the image pyramid

[0061] Use Gaussian filter kernel G to filter P (i-1) Filter to obtain the filtered image P' (i-1)

[0062] By selecting an alternative pixel P' in both directions (i-1) Get the scaled-down image P i

[0063] END

[0064] In addition to the pyramid generation block 304a, the QP-OFM algorithm block 304 also includes a block matching optical flow estimation block 304b (implementing the block matching, filtering and scaling functionalities involved in the PBM process (step 203)) and a local method based on differential techniques 304c of the optical flow estimation block (for implementing the modified LK (step 204)). Blocks 304b and 304c process one pyramid level at a time starting from the top of the pyramid. Block 304c is enabled only for the base of the pyramid in one embodiment and at all levels in another embodiment. The block matching optical flow estimation block 304b provides OF estimates with integer pixel precision. The optical flow estimation block 304c provides a modified OF estimate with fractional pixel resolution. The filtering and scaling functionalities of block 304b may be disabled at the base of the pyramid for embodiments in which block 304c is enabled only for the base of the pyramid. In an embodiment when block 304c is enabled at all levels, the filtering and scaling functionalities of block 304b are disabled for the base of the pyramid, and for higher levels, these functionalities are used after correction (by block 304c) of the OF estimate (obtained by block 304b).

[0065] As described above, the input to the pyramid generation block 304a includes the two grayscale images (query image and reference image) for which the optical flow is estimated and the historical image velocity estimates or temporal predictors (all pixels of size W' x H'). Figure 3 As shown by the dashed path in FIG, the temporal predictor is an optional input, and the pyramid generation block 304a does not obtain its multi-scale representation.

[0066] The QP-OFM algorithm block 304 outputs include the following for the next pyramid level (F i-1 ) is a properly filtered and scaled flow vector or a refined flow estimate (F i ). The QP-OFM algorithm block 304 output for levels above the base level is consumed within the QP-OFM algorithm block 304 and is passed to the stream post-processing block 306 at the base level and to the confidence assignment block 307. As disclosed above, for levels other than the base level, the stream may be filtered with a simple 2D median filter, where the stream estimate resolution is increased to match that of the lower levels and the stream estimate is scaled with an appropriate scaling factor. This may be followed by a rounding operation when applying the modified LK pass at all pyramid levels. For the base level, the QP-OFM algorithm block 304 output is passed to the stream post-processing block 306 for post-processing by a pseudo 1D separable median filter and to 307 for confidence assignment.

[0067] A pre-calculation process may also be provided before the QP-OFM algorithm pads (towards the bottom and right) so that all levels have integer sizes and exactly the same (e.g., 0.5) scaling factors. Initialization may include the flow estimate (F N ) is initialized to 0.

[0068] The following describes an example method provided by QP-OFM algorithm blocks 304b and 304c for pyramid levels. The described process is for pyramid levels except for the binary census transform step, but algorithm block 304b can also provide functionality for preparing binary pixel descriptors through binary census transforms for query and reference images.

[0069] For (all 2×2 pixel groups in the query image)%, process in raster scan order

[0070] Prepare a list of predictors for the top left pixel in the pixel group

[0071] · Calculate the median of the horizontal and vertical components of the predictor. Assume that the resulting flow vector is considered as the median predictor.

[0072] Find the best predictor (winner predictor) that minimizes the cost function value in the reference image

[0073] Set the winner predictor to the current optical flow estimate for all 4 pixels in the current 2x2 pixel group for (all pixels in the current 2x2 pixel group in the query image)

[0074] o3-1 search: Perform a planar coarse-to-fine (skip pel 3 + skip pel 1) block matching search around the current optical flow estimate that minimizes the cost function value.

[0075] Skip 1 search period when LK step is enabled for the current level

[0076] The query image and the motion compensated reference image are used to calculate the horizontal (Gx) and vertical (Gy) image gradient data of the query image and the temporal (Gt) gradient. If these calculations are performed during the skip pixel 1 search, then the LK step does not need to fetch any data from memory for motion compensation.

[0077] Note: Gradients are calculated using grayscale pixel information rather than census transform data.

[0078] ο Modified LK step (: if LK step is enabled for the current level)

[0079] Solve for changes in the optical flow estimate using the image gradient data Gx, Gy, and Gt

[0080] Clamp changes in the range [-1,1] in horizontal and vertical directions

[0081] - Update the optical flow estimate obtained during the step search process by adding the flow value calculated in this step.

[0082] Finish

[0083] Finish

[0084] If (i≠1) it is not the base level

[0085] Get the flow estimate for the next pyramid level

[0086] Input: The estimated flow value of the current level (F i )

[0087] Output: Flow estimate for the next pyramid level (F i-1 )

[0088] deal with:

[0089] - Perform an estimate of the updated flow F i 2D median filtering

[0090] - Scale up the resolution of the flow estimate

[0091] - Scale the flow value using the appropriate scaling factor and round to the nearest integer value

[0092] ELSE

[0093] Input: Flow estimate with fractional pixel accuracy (F1)

[0094] Output: Post-processed optical flow (F)

[0095] deal with:

[0096] - Perform post-filtering on the obtained flow estimate F1 to obtain the output flow F

[0097] Crop F to original image size

[0098] Finish

[0099] As described above, the OF calculation block 302 is also shown as having a flow post-processing filter 306 that receives unfiltered flow vector data from the QP-OFM algorithm block 304 and outputs filtered flow vector data. The OF calculation block 302 is also shown as having a confidence assignment block 307. The confidence assignment block 307 receives the Hamming distance (HD) cost and the unfiltered flow vector generated during the optical flow estimation process from the QP-OFM algorithm block 304, and generates confidence data.

[0100] The flow post-processing filter 306 is shown as outputting filtered flow vector data having horizontal and vertical flow components, which provides a compact OF estimate output 312b. Confidence data is a confidence level assigned for each pixel by a confidence assignment block 307, shown as being provided to a confidence map 312a. The compact optical flow estimate 312b is typically provided at the same resolution as the input images (Image 1 and Image 2), with an expected density of 100%, and a 16-bit fixed point bit resolution per pixel per flow component (7 and 8 integer bits for the vertical and horizontal flow components, respectively, and 4 fractional bits for both).

[0101] The computational details and explanations for the various QP-OFM algorithm steps are now described. With respect to the Gaussian filtering process in the pyramid generation block 304a, the pyramid level image may be filtered using a Gaussian filter of size 5×5 so that the immediately above pyramid level image may be generated by discarding alternating pixels from the filtered image. The Gaussian kernel derivation is provided below. In 2-D (x, y), the isotropic Gaussian kernel of size 5×5 has the following form:

[0102]

[0103] Where Sg is the distance sigma value used to derive the Gaussian filter kernel, and (x,y) represents the row and column positions of the filter coefficients in the Gaussian kernel. In a 5×5 filter kernel, the possible row and column positions are in the range [0,4]. This scheme uses an Sg value of 1.61. The filter coefficients so obtained are real numbers represented in fixed points. With respect to the filtering process (e.g., implemented by a separate FIR filter), at any pyramid level "n", by making the image P n Convolve with the Gaussian mask G defined in the above section to obtain the Gaussian filtered image P' n .

[0104] P′ n =G*P n

[0105] During the convolution process, boundary pixels are filtered by performing a boundary extension of 2 pixels on each side.

[0106] The binary feature based coarse to fine block matching optical flow estimation block 304b may also perform a binary census transformation. Figure 4 A method for computing a 24-bit census descriptor for a central pixel, taking into account its 24 neighbors (5×5 neighborhood) and its relationship thereto, is described in . A bit corresponding to a neighboring pixel is set to 1 if it has a grayscale value greater than or equal to pixel x, otherwise it is set to 0. Figure 5Illustration of the 5×5 neighborhood pixels used in the census signature calculation for a pixel. During the calculation, the census transform using the boundary pixel boundary extension technique is used to replace the missed pixels. The square box in the middle of the shaded area is the central pixel for which the census transform is calculated.

[0107] Regarding the cost function, as described above, the PBM method uses the binary census transform and Hamming distance of pixel description as the cost function during predictor evaluation and step search. When evaluating the cost function value at the predicted / searched position, the pixels surrounding the current pixel in the query image (see the description of the support window below) are considered. Figure 6 ) and a neighborhood of the same size around the predicted / searched position in the reference image. These neighborhoods act as the query data and reference data that are compared bit by bit to find inconsistencies. To do this, the binary signatures of the 81 pixels in the query and reference data are concatenated into two binary vectors. If Qv and Rv represent these vectors, then the binary Hamming distance between them is calculated as:

[0108]

[0109] in Defines a bitwise XOR operation between two binary vectors. The bit count operation counts the number of bits that are set to one after the bitwise XOR operation, and the output HD is the Hamming distance between the two data. A list of predictors can also be used for preparation. The predictors are existing OF estimates of spatial and temporal neighbors that can be used as initial OF estimates for the block matching search-based OF estimation process for a pixel. The OF estimation algorithm can use 5 different predictors per pixel group. These include the OF estimates of the 4 neighbors of the top left pixel in the pixel group (in Figure 7 These are the pixels that precede the top-left pixel when traversing the image from left to right and top to bottom in raster scan order.

[0110] Considering the pipelined design of HWA, the left predictor is expected to come from the left neighbor of the top left pixel in the pixel group, where the OF estimate of the pixel derived in a higher pyramid level and appropriately scaled for the current pyramid level is the predictor value. This predictor is called the left predictor of the pyramid. In an alternative design, the left predictor value may come from a pixel whose OF estimate has been refined in the current pyramid level and is at a certain pixel distance to the left of the top left pixel. This predictor may be called the delayed left predictor. The fifth predictor is the existing OF estimate of the current pixel, which is derived in a higher pyramid level and appropriately scaled for the current pyramid level. At the base pyramid level, the left / delayed left predictor of the pyramid is disabled and replaced with the temporal predictor when enabled. The predictor usage is summarized in Table 1 below, and in Figure 7The relative pixel positions (relative to the top left position in the pixel group) of the top and left spatial predictors are provided as depicted in .

[0111] Table 1: Overview of instance predictor usage by pyramid level

[0112]

[0113] Temporal predictors may be at fractional pixel resolution, rounded to the nearest integer position. Other spatial predictor values ​​are obtained during a stepwise search of the refinement of the OF estimate for the corresponding pixel or from higher pyramid level estimates. They are typically at integer pixel resolution but in the case of using the LK step at each pyramid level, they may have fractional pixel precision and should be rounded to the nearest integer. The top predictor is typically not available for the top row of a picture, and the top left and left predictors are not available for left boundary pixels, and the top right predictor is not available on the right boundary. In this context, the unavailable predictors should typically be replaced with the pyramid co-location predictor or set to the (0,0) value. When the pyramid co-location predictor of the pyramid is used at the highest pyramid level for this purpose, it takes the value (0,0).

[0114] The median predictor may be calculated by independently finding the median of the horizontal and vertical components of the selected predictor to be used in the search process.

[0115] With respect to the selection of the best predictor, considering an example 9×9 support window around the top left pixel in the pixel group, the cost function values ​​are estimated at all prediction positions in the reference image and the predictor that is selected as the winner predictor produces the minimum cost function value. In the case where the cost function value is the same for more than one predicted position, the first predictor in the evaluation order that results in the minimum value is selected as the winner predictor. An example evaluation order is 1) current of the pyramid, 2) left side / delayed left side / time of the pyramid, 3) top right, 4) top, and 5) top left. The predictor evaluation for the top right, top, and top left predictors can be combined with the step of searching for the corresponding pixel positions in the top row to minimize data reading from memory.

[0116] Figure 8 An illustration is provided involving the predictor evaluation of 5 candidate predictors for the current pixel group and a winner predictor therein. The winner predictor is now set to the current OF estimate value for all pixels in the pixel group and is further refined through the stepwise search process explained below.

[0117] During the step-by-step search process, an in-plane coarse-to-fine search may be performed to refine the current OF estimate such that the cost function value is minimized. The cost function used during the step-by-step search is defined as HD+MVCost. Where MVcost is defined as the product of the motion smoothness factor (as sample values, λ=24) and the vector distance (city block or the sum of absolute differences of horizontal and vertical components) between the search point and the median predictor. The search is performed in a specific pattern such that computational complexity is minimized while providing a wider range of refinement.

[0118] In stage 1 (skip 3), a skip 3 search across 9 positions (3×3 grid) centered on the winner predictor is performed. The search point is at a distance of 3 pixels from the current optical flow estimate or the winner predictor position. Figure 8 In FIG. 1 , these nine positions are depicted by a dashed pattern pixel of the top left pixel in the pixel group. The nine candidate positions are evaluated starting from the top left position in the grid in raster scan order. In the event that the cost function value is the same for more than one search position, the first search position in the evaluation order that produces the minimum value is selected as the position and the OF is estimated accordingly.

[0119] In stage 2 (skip 1), the OF estimate obtained in the previous stage is refined by searching across 9 points in a 3×3 neighborhood marked with a random fill pattern (skip 1). The winner of stage 2 gives the best integer-pixel motion estimate around the winner predictor. In the event that the cost function value is the same for more than one search position, the first search position in the evaluation order (raster scan) that produces the minimum value is selected as the winner position and the OF estimate is updated accordingly.

[0120] With respect to search range restrictions, at any pyramid level when a candidate position at which a cost function value is to be evaluated (during predictor evaluation or step search) is at a distance greater than a threshold referred to as the search range in the horizontal (191 pixels) or vertical (63 pixels) direction, the corresponding search position may be ignored. If all candidate positions during the predictor search step are at a distance greater than the search range, the current optical flow estimate value may be set to (0,0). This may be achieved by setting the predictor value at the same position of the pyramid to (0,0).

[0121] exist Fig. 9 The coarse-to-fine search pattern around the winner predictor is illustrated in FIG. Fig. 9 Illustrating an example search path for a 3-1 search for the two top pixels in a pixel group, a refined OF for a top left pixel in the pixel group, and the refinement range provided by the step search block (different pattern pixels depict possible results from the step search refinement for the top left pixel). Fig. 9 It is also illustrated that the OF estimates for two pixels may be the same at the beginning of the step search process, but during the step search process, their OF estimates may diverge.

[0122] With regard to the pre-evaluation of the predictors for the lower row of pixel groups, at any pixel during the stepwise search process, for the pixel group to which one of the top predictor values ​​is provided, the predicted position (the OF of the pixel after the skip 1 search) is evaluated using the already extracted data and the cost is stored for subsequent use. Thus, the data read does not have to be repeated during the predictor selection step.

[0123] Regarding the calculation of spatial and temporal gradients used in the LK step when the LK step is enabled for a level, these calculations are performed for all pixels during the step search process to avoid repeated data reading by the LK step block. In this modified LK step, the spatial and temporal image gradients are calculated (using grayscale pixel values) for all pixel positions in the 3×3 neighborhood of the current pixel in the query image.

[0124] The horizontal spatial gradient (Gx) and the vertical spatial gradient (Gy) are calculated using the central difference operator (mask = [-1 0 1]), so the pixels within the 5×5 neighborhood of the current pixel are required. At the image boundaries, the spatial gradient is calculated using a boundary extension (2 or 1 pixel as needed).

[0125] Considering the neighborhood of the current pixel in the query image and a neighborhood of the same size of the position it is estimated to have moved to in the reference image (this is the winner position obtained during skip one search), the temporal image gradient (Gt) is calculated for all pixels in the 3×3 neighborhood. The pixel-by-pixel Gt calculation is performed by subtracting the reference image pixel value from the corresponding pixel value in the query image. At image boundaries, the gradient is calculated using boundary extension (1 pixel).

[0126] For a 12-bit input image, the spatial and temporal gradient values ​​have 13 bits of precision. This precision is reduced to 10 bits by performing a 2-bit right shift and clamping the post-shift values ​​between -512 and 511.

[0127] The PBM method described above estimates pixel-level OF at integer pixel resolution, and this estimate is generally close to actual pixel motion. However, this estimate is further refined to more accurately match the actual pixel motion. As described above, in order to refine the initial OF estimate that produces the PBM to provide fractional pixel accuracy ("corrected OF estimate"), the QP-OFM algorithm uses at least one pass of the modified LK method.

[0128] Regarding 2D median filtering of flow estimates, after updating all flow estimates at pyramid levels except the base pyramid level, the flow estimates may be filtered using a 2D median filter of size 5×5. At image boundaries, the median value is typically calculated using boundary extension (2 pixels). Regarding flow resampling, post-2D median filtering, for all pyramid levels except the base level, the flow estimate resolution is scaled up by using the nearest neighbor interpolation and the flow vector is scaled by the resolution upscaling factor (2) to obtain the flow estimate for the next lower pyramid level.

[0129] Regarding flow post-processing, if enabled by the user through configuration, flow post-processing in the QP-OFM algorithm involves the use of a 2D median filter of size, for example, 5×5. The post-processing block 306 takes the optical flow estimate (Fi) from the QP-OFM algorithm block 304 as input and produces a filtered optical flow output (F) shown as filtered flow vector data. For filtering boundary pixels, an operation of extending the flow field boundaries on all sides by 2 pixels is performed. Alternatively, a pseudo 1D separable median filter of size 9×9 (height×width) can also be used. The 9×9 filter is separated into 3×9 and 9×3 filters that are applied sequentially in the order described or in reverse. The height and width of the separated filters are configured so that the number of flow samples used in the median calculation can provide high quality post-processing. In this case of filtering boundary pixels, an operation of extending the flow field boundaries on all sides by 4 pixels is performed.

[0130] Due to the relatively low computational requirements compared to some conventional methods, such as LK and Horn Schunck (HnS), higher resolutions and frame rates can be achieved in real time than was previously possible.

[0131] Some other specific differences of the disclosed QP-OFM algorithm over other solutions (e.g., the hierarchical model-based motion estimation methods disclosed by Bergen, Anandan, Hanna, and Hingorani) include: (a) a form of coarse-to-fine search using predictors and motion smoothness factors; (b) use of (optional) historical image velocity estimates to improve optical flow estimation accuracy; (c) reduction of per-pixel predictor evaluation requirements by selecting the order of pixel groups (2×2 pixel blocks) processed and using the same set of predictors for the entire pixel group; (d) reduction of random memory accesses for predictor evaluation by pre-evaluation of top-right, top, and top-left predictors during step-wise search refinement for the above pixels; (e) complete removal of interpolation from the algorithm by using a block matching method combined with an LK step to estimate fractional pixel motion, and achieving large savings in computational requirements; (f) pre-computation of spatial and temporal gradients during the block matching process; (g) use of a pseudo 1d separable filter; and (h) the ability to balance computational and optical flow accuracy requirements by varying the number of predictors and search resolution.

[0132] Advantages of the disclosed QP-OFM algorithm include: (a) performing the search using only a computationally simple optimization function "Hamming distance" (between binary feature descriptors within a support window); (b) minimizing random accesses to memory by using a common set of predictors for pixel groups, using a pattern of non-exhaustive coarse-to-fine search strategy for optical flow estimation to minimize computation, thereby saving precious resources; (c) pre-evaluating top predictors, saving memory to computation block data bandwidth (extracting reference pixel data for cost function evaluation), and thus minimizing design complexity; (d) pre-computing spatial and temporal gradients used in the modified LK step, further saving memory block data bandwidth (e.g., extracting reference pixel data for temporal gradient computation); (e.g., extracting reference pixel data for temporal gradient computation); and (e.g., extracting reference pixel data for temporal gradient computation). ) and thus minimize design complexity; (e) as part of the design complexity reduction and DDR bandwidth reduction (described below), the two levels of local memory can be used with L2 to store current and reference picture data in a growing window manner, and with L1 to store a sliding window of reference picture data to store the pixels required to enable block matching search and LK steps within a desired range; (f) a new algorithm is defined by combining parametric (PBM) and non-parametric (LK) methods, removing multiple interpolation operations required for fractional pixel motion estimation, which is a significant advantage over existing algorithms; and (g) even the operations involved in the LK step involve integer arithmetic and therefore only a constant number of multiplication and addition / subtraction operations are required to achieve very accurate calculation of fractional pixel motion estimation.

[0133] The disclosed QP-OFM algorithm can be implemented as a system-on-chip (SoC) solution, which provides high programmable computing power by sharing low-level computationally intensive and repetitive CV tasks with HWA. OF technology, which forms the fundamental building block of camera-based ADAS, processes a large amount of pixel data for each frame to produce high-level information that is recognized as a good candidate for HWA design.

[0134] Fig. 10A An example SoC 1000 formed on a substrate 1005 (chip) having a semiconductor surface (e.g., a silicon substrate) is depicted, which provides high programmable computing power by sharing low-level computationally intensive and repetitive CV tasks with a HWA 1015. The SoC 1000 includes a general purpose processor (GPP) 1010 that performs general computer vision and signal processing and control tasks. The HWA 1015 performs low-level CV tasks such as OF and stereo disparity measurement. Another processor, shown as a DSP 1020, performs high-level vision and signal processing tasks such as object detection, scene understanding and reconstruction, and a system peripherals block 1030 includes program and data storage 1030a. The system peripherals 1030 interface with sensors, peripherals, and communication devices. Interconnects (e.g., one or more interconnect metal levels) shown as 1022 on the substrate 105 couple the GPP 1010, the HWA 1015, and the DSP 1020 to the system peripherals block 1030 on the substrate 1005.

[0135] Fig. 10B Depicting the storage of reference and current picture data as shown in HWA (eg Fig. 10A 1090 . The implementation details of an example scheme 1090 in local memory of level 1 local memory (L1) and level 2 local memory (L2) within HWA 1015 (shown in FIG. 1015 ) are shown. This implementation aims to minimize the movement of data from memory (e.g., random access memory (RAM)) to the HWA implementing the OF logic. In this scheme 1090, both query and reference picture data to allow correct functioning of the OF estimation process for all pixel groups in the query image 1050 are fetched and stored in such a way that they are only fetched from memory (e.g., RAM) once. If the query image 1050 and the reference image 1060 are pyramid level images between which OF is being estimated, and the HWA is performing the computational tasks involved in computing the OF for the pixel group 1070, then Fig. 10B Describes the extent to which image pixel data needs to be in local memory. Fig. 10B In , the query image is abbreviated as QI and the reference image is abbreviated as RI.

[0136] Considering an embodiment using a 2x2 pixel group size, a 9x9 block matching support window, and a 5x5 neighborhood size for calculating the census signature, the query image data of the size required for the optical flow calculation is the 14x14 neighborhood 1056 surrounding the pixel group location 1070. Similarly, in evaluating each of the predictors or step search locations, pixel data from the 13x13 neighborhood surrounding that location is required. In addition, considering a search range of ±191 pixels in the horizontal direction and ±63 pixels in the vertical direction, a rectangular region of 396x140 pixels in the pixel block 1066 surrounding the pixel group location 1070 is required, along with modified LK calculations as needed, to evaluate all possible "valid" predicted locations and step search locations. When the entire pixel block 1066 is not within the picture boundaries, appropriate logic for search range limitation is required (the evaluated locations must be within the search range and picture boundaries). When the pixel block 1056 and the 13x13 neighborhood surrounding the search location are not contained within the picture boundaries, appropriate boundary extension logic is applied.

[0137] Since the order in which pixel groups are processed is in a raster scan (i.e., from left to right and from top to bottom), in effect, pixel blocks 1056 and 1066 slide across the image in the same pattern. Given the requirements and randomness of data access in these regions, it is advantageous to keep this data in the internal local memory of the HWA. However, if only the sliding pixel blocks are stored in the local memory, then when processing the pixel group data at, for example, pixel group location 1075, the entire block similar to pixel block 1056 or 1066 surrounding that location needs to be fetched from memory (e.g., RAM), resulting in repeated reading of multiple pixels, greatly increasing bandwidth requirements, and computing resources may also have to remain idle while the local memory is filled with the required data.

[0138] It is possible to overcome these limitations and reduce bandwidth requirements by fetching pixel data only once and reusing it across adjacent pixel groups and pixel group rows. In this regard, the query image data for the entire 14 pixel rows 1053 required to process all pixel groups in pixel group location 1070 can be fetched into a partition of on-chip local memory, and the HWA can access the required group of 14×14 pixel data from there. When processing for pixel group location 1075 begins, pixel data 1052 can be discarded, the remainder of the pixel data from memory block 1053 can be stored (retained) in local memory, and additional pixel data 1054 can be fetched into local memory. Similarly, 140 reference image pixel rows on physical memory block 1063 can be stored in another partition of on-chip local memory when processing pixel groups in pixel group location 1070, and fetched from there to a sliding pixel block 1066 in another memory closer to the HWA allows faster access to the data.

[0139] In addition, similar to the memory management performed for the query image data, when processing for the pixel group position 1075 begins, the reference image data 1062 may be discarded, and the remainder of the data from the physical memory block 1063 may be retained in the local memory and the additional pixel data 1064 may be fetched into the local memory for processing of the remainder of the pixel group in the row. The manner of storing the pixel data for an entire row in the local memory in physical memory blocks such as 1053 and 1063 may be referred to as a growing window, and the manner of storing a sliding pixel block such as 1066 may be referred to as a sliding window. In one embodiment, the local memory used to store the growing window of pixel data such as 1053, 1063 and the sliding pixel block such as 1066 may be a logical partition of a physical memory block, and in another embodiment, it may be stored in more than one physical memory block, for example, the pixel data from the memory blocks 1053 and 1063 may be stored in L2 memory, and the data corresponding to the sliding pixel block 1066 may be stored in L1 memory. In another embodiment, when a processor is used to implement the OF logic, a similar multi-layer local memory architecture and picture data management scheme may be used.

[0140] Applications of the disclosed OF estimation include solving ADAS tasks such as moving object segmentation, object tracking, contact time (collision) estimation, depth estimation, and scene understanding. Compared to known DOF estimation algorithms, the disclosed OF estimation enables improved performance ADAS applications such as obstacle detection and collision avoidance automatic emergency braking.

[0141] Compared to a full PBM based DOF algorithm, the disclosed example QP-OFM algorithm evaluates only about 24 search positions per pixel, uses no interpolation, and computes only about 1.33 census transforms per pixel. This represents a significant computational savings when processing 60 million pixels per second, which can be achieved by adding a portion of the computation of the OF estimate using a modified LK block. The QP-OFM algorithm further simplifies the HWA design by splitting the computational operations of the functional blocks into logical subsets and combining subsets of the operations of different sub-blocks into a set so that they can reuse the pixel data extracted for the operations. An example of this type of design is to pre-evaluate the top predictor of a pixel group during a step search operation of the top pixel group. Another example is the pre-computation of spatial and temporal gradients for use by the LK step during the step search process. Such repackaging of computational tasks of different functional blocks has resulted in an almost 6x reduction in SL2 memory complexity compared to the algorithm disclosed in U.S. Patent Application Publication No. 2015 / 0365696 for the disclosed QP-OFM algorithm.

[0142] Modifications are possible in the described embodiments and other embodiments are possible within the scope of the claims.

Claims

1. An image processing system, comprising: A processor capable of implementing: performing optical flow OF estimation between a first frame and a second frame of the video using a pyramidal block matching (PBM) method to generate an initial optical flow OF estimation value with integer pixel resolution at a base pyramid level, wherein the PBM method is an OF estimation method that converts each of the frames of the video into a hierarchical image pyramid, and refining the initial optical flow OF estimate using at least one pass of a modified Lucas-Kanad method to provide a revised OF estimate having a fractional pixel resolution, The modified Lucas-Kanad method does not include an image warping step, and it splits the calculation of the modified OF estimate into subtasks of spatial and temporal gradient calculations and then other operations, uses reduced precision gradient data for OF refinement, and performs computational tasks including matrix inversion using only integer arithmetic.

2. The system of claim 1, wherein the PBM method uses a hierarchical coarse-to-fine search strategy using predictors and motion smoothness factors, a binary census transform for pixel descriptions, and a Hamming distance as a cost function. 3 . The system of claim 1 , wherein the PBM method uses pixel group order processing using the same group predictor for each entire pixel group and a pre-evaluation strategy for the group predictor.

4. The system of claim 1, wherein the PBM method uses pre-computation of spatial gradients and temporal gradients during a block matching step of the PBM method.

5. A system according to claim 1, wherein the OF determination logic is partially implemented by a hardware accelerator HWA, which uses a combination of two levels of local memory, one of which is used to store picture data from the first frame and the second frame of the video in a growing window manner, and the other memory stores a sliding window of the picture data from the second frame of the video.

6. The system of claim 2, wherein OF determination logic configures the processor to include a motion vector cost value in the cost function for each sequential search of the search strategy, the motion vector cost value combining a motion smoothness factor and a distance between a median predictor value and a candidate pixel.

7. The system of claim 1, wherein the at least one pass consists of a single pass of applying the modified Lucas-Kanad method at all pyramid levels of a search pyramid used by the PBM method, including the base pyramid level, after the PBM method ends.

8. The system of claim 1, further comprising a post-processing filter for post-processing filtering using a pseudo 1d-separable median filter to remove impulse noise from the modified OF estimate.

9. The system of claim 1, wherein the first frame of video is a reference image and the second frame of video is a query image.

10. A method for optical flow OF estimation, comprising: A stored quasi-parametric optical flow measurement QP-OFM algorithm is implemented using a processor, the QP-OFM algorithm combining a pyramidal block matching PBM method with a modified Lucas-Kanad method, wherein the PBM method is an OF estimation method that converts each of the frames of a video into a hierarchical image pyramid, the QP-OFM algorithm performing: Using the PBM method, perform optical flow OF estimation between a first frame and a second frame of a video to generate an initial OF estimation value with integer pixel resolution at a base pyramid level, wherein the first frame of the video is a reference image and the second frame of the video is a query image; as well as refining the initial OF estimate using at least one pass of the modified Lucas-Kanad method to provide a revised OF estimate having a fractional pixel resolution, The modified Lucas-Kanad method does not include an image warping step, and it splits the calculation of the modified OF estimate into subtasks of spatial and temporal gradient calculations and then other operations, uses reduced precision gradient data for OF refinement, and performs computational tasks including matrix inversion using only integer arithmetic.

11. The method of claim 10, wherein the PBM method uses a hierarchical coarse-to-fine search strategy using predictors and motion smoothness factors, a binary census transform for pixel descriptions, and Hamming distance as a cost function.

12. The method of claim 10, wherein the PBM method uses a pixel group order processing using the same group predictor for each entire pixel group and a pre-evaluation strategy for the group predictor.

13. The method of claim 10, wherein the PBM method uses pre-computation of spatial and temporal gradients during a block matching step of the PBM method.

14. The method of claim 10, wherein the QP-OFM algorithm is at least partially implemented as a system-on-chip (SOC) comprising a hardware accelerator (HWA), wherein the HWA uses a combination of two levels of local memory, wherein one memory is used to store picture data from the first frame and from the second frame of the video in a growing window manner, and another memory stores a sliding window of the picture data from the second frame of the video.

15. The method of claim 10, wherein the method is implemented by an image processing system including a processor configured to generate a scene analysis. 16 . The method according to claim 15 , wherein the scene analysis is used by an advanced driver assistance system (ADAS) for obstacle detection or collision avoidance automatic emergency braking.

17. The method of claim 11, wherein the method includes in the cost function for each sequential search of the search strategy a motion vector cost value that combines a motion smoothness factor and a distance between a median predictor value and a candidate pixel.

18. The method of claim 10, further comprising post-processing filtering using a pseudo 1d separable median filter to remove impulse noise from the modified OF estimate.

19. The method of claim 10, wherein the at least one pass consists of a single pass of applying the modified Lucas-Kanad method at all pyramid levels of a search pyramid used by the PBM method, including the base pyramid level, after the PBM method ends.

20. An image processing system, comprising: a first circuit for performing an optical flow OF estimation value between a first frame and a second frame of a video using a pyramid block matching (PBM) method to generate an initial optical flow OF estimation value with integer pixel resolution at a base pyramid level, wherein the PBM method is an OF estimation method that converts each of the frames of the video into a hierarchical image pyramid, and second circuitry for refining the initial optical flow OF estimate using at least one pass of a modified Lucas-Kanad method to provide a revised OF estimate having a fractional pixel resolution, The modified Lucas-Kanad method does not include an image warping step, and it splits the calculation of the modified OF estimate into subtasks of spatial and temporal gradient calculations and then other operations, uses reduced precision gradient data for OF refinement, and performs computational tasks including matrix inversion using only integer arithmetic.

21. An image processing system, comprising: a first device for performing an optical flow OF estimation value between a first frame and a second frame of a video using a pyramid block matching (PBM) method to generate an initial optical flow OF estimation value with integer pixel resolution at a base pyramid level, wherein the PBM method is an OF estimation method that converts each of the frames of the video into a hierarchical image pyramid, and second means for refining the initial optical flow OF estimate using at least one pass of a modified Lucas-Kanad method to provide a revised OF estimate having a fractional pixel resolution, The modified Lucas-Kanad method does not include an image warping step, and it splits the calculation of the modified OF estimate into subtasks of spatial and temporal gradient calculations and then other operations, uses reduced precision gradient data for OF refinement, and performs computational tasks including matrix inversion using only integer arithmetic.

Images