Region-based motion estimation and modeling for accurate region-based motion compensation for efficient video processing or coding

Abstract

A system (4000) for performing efficient motion-based video processing using region-based motion, comprising: a region-based motion analyzer (4100), the region-based motion analyzer (4100) including one or more substrates (4102) and logic (4104) coupled to the one or more substrates (4102), the logic (4104) being operable to: obtain multiple block motion vectors for multiple blocks of a current frame with respect to a reference frame; modify the plurality of block motion vectors, wherein modifying the plurality of block motion vectors includes one or more of the following operations: smoothing at least a portion of the plurality of block motion vectors, merging at least a portion of the plurality of block motion vectors, and discarding at least a portion of the plurality of block motion vectors; segment the current frame into multiple regions, the regions comprising a background region type containing a background motion region and a foreground region type containing, in some cases, a single foreground motion region and, in other cases, multiple foreground motion regions; and a power supply (4007) to provide power to the area-based motion analyzer (4100), the logic further serving to: before segmenting the current frame into several areas: to restrict the modified multiple block motion vectors by excluding a portion of the frame in some cases; after segmenting the current frame into several areas: compute a plurality of region-based candidate motion models based on the constrained modified plurality of block motion vectors for the current frame with respect to the reference frame for the background region type and the foreground region type individually, each region-based candidate motion model including a set of region-based candidate motion model parameters representing a region-based motion of each region type of the current frame; determine a best region-based motion model from the plurality of candidate region-based motion models on a frame-by-frame basis and on a region-type basis, each best region-based motion model including a set of best region-based motion model parameters representing the region-based motion of each region type of the current frame; modify an accuracy of the parameters of the best region-based motion model in response to one or more application parameters; map the parameters of the best modified-accuracy region-based motion model into a pixel-based coordinate system to determine a plurality of mapped region-based motion curvature vectors for a plurality of control grid points of the reference frame; predict and encode the plurality of mapped region-based motion curvature vectors for the current frame with respect to a plurality of previously mapped region-based motion curvature vectors; determining a best subpixel filter to use for interpolation at a 1 / 8 pixel location or a 1 / 16 pixel location from among two or more subpixel filter selections per region and per frame; and apply the plurality of mapped region-based motion curvature vectors at the sub-pixel locations per region to the reference frame and perform the interpolation of the pixels based on the determined best sub-pixel filter to generate a region-based motion-compensated curved reference frame.

Description

TECHNICAL FIELD

[0001] The embodiments generally relate to region-based motion estimation. More specifically, the embodiments relate to a technique that provides accurate region-based motion compensation to improve video processing efficiency. BACKGROUND

[0002] Numerous previous approaches have been attempted to improve global motion estimation using various approaches to achieve better global motion compensation and consequently enable higher coding efficiency. However, most previous solutions typically use a frame-based approach to improve global motion estimation.

[0003] For example, one group of techniques has attempted to improve robustness by filtering the often noisy motion field typically available from block motion estimation and used as a first step in global motion estimation. Another group of techniques has attempted to improve global motion-compensated prediction using pixel-based motion or model adaptivity (e.g., in a specialized case of panoramas) or higher-order motion models. Another group of techniques has attempted to improve the quality of global motion estimation by using better estimation accuracy, improved framing, or using variable-block-size motion. Another group of techniques has attempted to obtain better coding efficiency at low bit cost by improving model efficiency.Yet another group of techniques has attempted to address the problem of complexity or performance. In "A NEW TWO-STAGE GLOBAL / LOCAL MOTION ESTIMATION BASED ON A BACKGROUND / FOREGROUND SEGMENTATION," Fabrice Moscheni et al. describe that motion estimation and compensation are very effective in eliminating temporal redundancy. The motion present in a scene can be considered primarily as local motion superimposed on the camera motion. Furthermore, a two-stage approach for global / local motion estimation is presented. Global motion estimation relies only on the background information. It is based on a matching procedure, and the global motion model is chosen affinely. The simulation results show that the proposed method offers significant improvements over conventional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The various advantages of the embodiments will become apparent to one skilled in the art by reading the following description and the appended claims with reference to the following drawings, in which: Fig. 1 is an illustrative block diagram of an exemplary region-based motion analyzer system according to one embodiment; Fig. 2 is an illustrative graphical representation of an exemplary region-based parametric motion analysis process according to one embodiment; Fig. 3 is an illustrative block diagram of a more detailed exemplary region-based motion analyzer system according to one embodiment; Fig. 4 is an illustrative block diagram of an exemplary video encoder according to one embodiment; Fig.5 is an illustrative block diagram of an exemplary advanced video coding video encoder according to one embodiment; Fig. 6 is an illustrative block diagram of an exemplary high performance video coding video encoder; Fig. 7 is an illustrative graphical representation of an exemplary structure of a group of images according to one embodiment; Fig. 8 is an illustrative graphical representation of various exemplary models of global motion related to chirping according to one embodiment; Fig. 9A-9D are illustrative diagrams of an exemplary curve fitting model of the Levenberg-Marquardt algorithm (LMA) for approximating global motion according to one embodiment; Fig.10 is an illustrative block diagram of an exemplary local motion field noise reduction filter according to one embodiment; Fig. 11 is an illustrative block diagram of an exemplary region segmenter according to one embodiment; Fig. 12 is an illustrative diagram of an exemplary histogram distribution of locally computed parameters of the affine model of global motion according to one embodiment; Fig. 13 is an illustrative video sequence of an example of the difference between global and local block-based vectors according to one embodiment; Fig. 14 is an illustrative diagram of an exemplary histogram distribution of locally computed parameters of an affine model of global motion using a random sampling approach according to one embodiment; Fig.15 is an illustrative video sequence of an example of the difference between global and local block-based vectors according to one embodiment; Fig. 16 is an illustrative video sequence of an example of various computed candidate selection masks according to one embodiment; Fig. 17 is an illustrative video sequence of an example of various computed candidate selection masks according to one embodiment; Fig. 18 is an illustrative block diagram of an exemplary global motion model (GMM) calculator for segmentation, according to one embodiment; Fig. 19 is an illustrative block diagram of an exemplary motion vector selector for GMM segmentation estimation according to one embodiment; Fig.20 is an illustrative video sequence of an example segmentation method using color to assist motion, according to one embodiment; Fig. 21 is a block diagram of an exemplary background motion region segmenter according to one embodiment; Fig. 22 is a block diagram of an exemplary foreground motion region segmenter according to one embodiment; Fig. 23 is an illustrative video sequence of exemplary morphology-based post-processing according to one embodiment; Fig. 24 is an illustrative video sequence of an exemplary region segmentation method for low resolution content according to one embodiment; Fig. 25 is an illustrative video sequence of an exemplary region segmentation method for low resolution content according to one embodiment; Fig. 26 is an illustrative video sequence of an exemplary region segmentation method for low resolution content according to one embodiment; Fig. 27 is an illustrative video sequence of an exemplary region segmentation method for standard definition content according to one embodiment; Fig. 28 is an illustrative video sequence of an exemplary region segmentation method for high-resolution content according to one embodiment; Fig. 29 is a block diagram of an exemplary morphologically based region post-processor according to one embodiment; Fig. 30 is an illustrative video sequence of an example of compensation for detected non-content regions according to one embodiment; Fig.31 is an illustrative block diagram of an exemplary multi-region based motion estimator and modeler according to one embodiment; Fig. 32 is an illustrative block diagram of an exemplary motion vector selector for region-based motion modeling (RMM) estimation according to one embodiment; Fig. 33 is an illustrative block diagram of an exemplary adaptive subpixel interpolation filter selector for region-based motion modeling (RMM), according to one embodiment; Fig. 34 is an illustrative block diagram of an exemplary adaptive RMM calculator and selector according to one embodiment; Fig. 35 is an illustrative diagram of an example of a translational 4-parameter model of global motion according to one embodiment; Fig.36 is an illustrative block diagram of an exemplary adaptive range-based compensator according to one embodiment; Fig. 37 is an illustrative block diagram of an exemplary range motion parameter and head encoder according to one embodiment; Fig. 38 is an illustrative diagram of an exemplary probability distribution of the best prior codebook models according to one embodiment; Fig. 39A-39D are an illustrative flowchart of an example process for the region-based motion analyzer system according to one embodiment; Fig. 40 is an illustrative block diagram of an exemplary video coding system according to one embodiment; Fig. 41 is an illustrative block diagram of an example logic architecture according to an embodiment; Fig.42 is an illustrative block diagram of an exemplary system according to one embodiment; and Fig. 43 is an illustrative diagram of an example of a small form factor system according to one embodiment. DETAILED DESCRIPTION

[0005] As described above, numerous previous approaches have attempted to improve global motion estimation through various approaches to achieve better global motion compensation and consequently enable higher coding efficiency. However, most previous solutions typically use a frame-based approach to improve global motion estimation.

[0006] While some schemes have managed to advance the state of the art, the actual gains achieved have been limited or significantly missed their targets. What has been missing so far is a comprehensive approach to improving the estimation and compensation of global motion and the parameter coding problem.The implementations described here represent such a solution to the existing shortcomings of the existing state-of-the-art, which include: low robustness or reliability in consistently and accurately measuring global motion; insufficiently accurately measured global motion estimate; computed global motion estimate leading to a frame with poorly compensated global motion and consequently to an error in prediction with poorly compensated global motion; insufficient gain from using global motion, even in scenes with global motion; high bit cost of encoding global motion parameters; high computational complexity of the algorithms; and low adaptivity / high error rates for complex and noisy content.

[0007] As described in more detail below, the implementations described here can provide a solution to the technical problem of significantly improving the quality of global motion estimation, the accuracy of global motion compensation, and efficient encoding of global motion parameters—in a robust and complexity-bounded manner—in video scenes with global motion. For example, instead of a single frame-based global motion, multiple dominant motions can be compensated on a region-by-region basis, adding considerable flexibility over the frame-based approach (as is often typical in global motion operations, for example).

[0008] The term "region-based," as used herein, refers to "region-based motion modeling" and the like, to distinguish them from "global motion modeling" and the like. While the operation of "region-based motion modeling" may have many similarities to "global motion modeling," the term "region-based" as used herein means that the described region motion compensation operations act on a region whose size may be smaller than an entire frame, whereas global motion compensation operations in the art typically refer to operations across an entire frame unless explicitly described otherwise.To further explain, in global motion modeling, global motion estimation can be performed either for a full video frame or for a video frame excluding a specific region (such as excluding a region with local motion), but global motion compensation must be applied to a full frame. However, in region-based motion estimation, motion parameters are estimated for a region, and region-based motion compensation is also performed on a region-by-region basis. Consequently, in global motion modeling, only one set of global motion parameters is needed to represent a parametric motion of a frame, but in region-based motion modeling, the number of motion parameter sets required is the same as the number of regions in a frame.

[0009] In some implementations, a highly adaptive and accurate approach can be used to address the problem of estimating and compensating for parametric motion of each dominant region (such as the background and foreground regions) in video scenes. The solution can be content-adaptive because it uses adaptive modeling of the frame into regions and the motion of each region using the best of several models used to estimate global motion. Furthermore, the region-based motion estimation parameters themselves can be calculated using either of two optimization-based approaches, depending on the selected global motion model.Using the estimated region-based motion parameters, region-based motion compensation can be performed using interpolation filters that are adaptive to the content type. Furthermore, the region-based motion parameters can be encoded using a highly adaptive approach that utilizes either a codebook or a context-based differential coding approach for efficient bit representation. The aforementioned improvements in region-based motion estimation / compensation can be achieved under the constraint of keeping complexity as low as possible. Overall, the implementations presented here represent an adaptive and efficient approach for accurate region-based parametric motion representation for efficient video coding.

[0010] The solutions described here can, for example, estimate the motion of dominant regions within a video sequence using an improved motion filtering and selection technique for computing region-based motion models that computes multiple region-based motion models for a number of different parametric models per region (e.g., as opposed to only per frame). From the computed region-based motion models, a determination and selection of the best region-based motion model and the best subpixel interpolation filter per dominant region of a frame can be performed to perform motion compensation. The computed parameters of the region-based motion model can then be efficiently encoded using a combination of codebook and differential coding techniques.

[0011] Accordingly, some of the implementations described below represent a fast, robust, novel, accurate, and efficient method for performing both region-based and global motion estimation and compensation in video scenes with global motion. Some implementations described below represent a significant step forward in the state of the art and may be applicable to various applications, including improved long-range prediction, motion-compensated filtering, frame rate conversion, and improved compression efficiency of lossy / lossless video, scalable video, and multi-viewpoint / 360-degree video. This tool is expected to be a candidate for integration into future video standards, although it should also be possible to incorporate this tool into extensions to current and upcoming standards, such as H.264, H.265, AOM AV1 or H.266.

[0012] Fig.1 is an illustrative block diagram of an exemplary region-based motion analyzer system 100 arranged in accordance with at least some implementations of the present disclosure. In various implementations, the exemplary region-based motion analyzer system 100 may include a video scene analyzer 102 (e.g., a video scene analyzer and frame buffer), a local block-based motion field estimator 104 (e.g., a local (e.g., block) motion field estimator), a local motion field noise reduction filter 106 (e.g., a local motion field noise reduction filter), a region segmenter 107 (e.g., a region (object) segmenter), a multi-region-based motion estimator and modeler 108 (e.g., a multi-region-based motion estimator and modeler), an adaptive region-based motion compensator 110 (e.g.,an adaptive range motion compensator), a range-based motion model parameter and head entropy encoder 112 (e.g., the range motion parameters and heads of the entropy encoder), the like, and / or combinations thereof. For example, some of the individual components of range-based motion analyzer system 100 may not be used in all embodiments. In such an example, video scene analyzer 102 may be used in some implementations or eliminated in other implementations.

[0013] As described in more detail below, the exemplary region-based motion analyzer system 100 may be adaptive in nature and may combine the use of statistical techniques with segmentation techniques to increase the quality and accuracy of motion modeling in a given frame. For example, the region-based motion analyzer system 100 may divide a frame into several different motion regions, adapting between models of varying complexity on a region-by-region basis to support various motion patterns that may be dynamically changing. Additionally, the region-based motion analyzer system 100 may also adapt subpixel interpolation filtering operations according to the type of texture that is dominant in the given region.

[0014] As illustrated, Fig.1 is a high-level conceptual graphical representation of a region-based motion analyzer system 100 that includes aggregated building blocks to simplify discussion. Video frames are shown input to video scene analyzer 102, which performs scene analysis, such as scene change detection (and optionally scene transition detection), and also has buffered frames to allow for the use of reordered frames (e.g., according to a picture group organization scheme). Next, a pair of frames (e.g., a current frame and a reference frame) is input to local block-based motion field estimator 104, which calculates motion vectors for all blocks of the frame. Next, this motion field is filtered by a local motion field noise reduction filter 106 to remove noisy motion vector regions.

[0015] The filtered motion field is then input to the region segmenter 107, which segments the frame into multiple regions (e.g., two or three regions, excluding static regions such as black bars, black letterbox regions, static logo regions, and / or the like) using a region mask. The region mask is then provided by the region segmenter 107 to the multi-region motion estimator and modeler 108. The multi-region motion estimator and modeler 108 can calculate a region-based motion estimate by trying different motion models per frame and selecting the best one.Next, the selected motion field parameters are encoded by the region-based motion model parameter and head entropy encoder 112, with both the region mask and the motion field being provided to the adaptive region-based motion compensator 110, which generates the region-based motion compensated regions and the region-based motion compensated frame.

[0016] In operation, the region-based motion analyzer system 100 operates based on the fundamental principle that exploiting region-based parametric motion in video scenes is the key to further compression gains, both through integration into current-generation encoders and the development of new-generation encoders. Furthermore, the implementations described here offer improvements on all fronts compared to the current state of the art, e.g., region-based parametric motion estimation, region-based parametric motion compensation, and region-based parameter coding.

[0017] When it comes to region-based motion estimation, significant care is required not only in selecting a region-based motion model, but also in how that motion model is computed. Implementations for lower- or middle-order models (such as 4 or 12 parameters) can use a least-squares estimation method and / or a random sampling method. Implementations for higher-order models (such as 8 and 12 parameters) can use the Levenberg-Marquardt algorithm (LMA). For an 8-parameter region-based model, implementations can identify many available options, such as the bilinear model, the perspective model, and the pseudoperspective model. Through rigorous testing, the pseudoperspective model has been found to be often the most consistent and robust.In order to be able to determine a separate motion model for each region type (e.g., the background region and the foreground region(s), an initial correct segmentation of a frame into a background region and a foreground region(s) is required. While this is not a simple task, it is quite useful even if the region boundaries are somewhat imprecise. Furthermore, the task of finding any parameters of the region-based motion model is complicated by the noisiness of the motion field, therefore, a good algorithm for filtering a region-based motion field has been developed to separate outlier vectors that would otherwise contribute to corrupting the calculation of the motion field of each of the regions. While the same region-based motion model is expected to be suitable for the same (e.g.,corresponding) region can be used in a group of frames that have similar motion characteristics, content-based adaptivity and digital sampling may require more or less independent selection of one motion model per main region type (e.g., background region and one or more foreground regions) of each frame from a number of available motion models. Furthermore, due to the cost of the region-based motion parameter coding bits, rate distortion constraints may also be used in the selection of the region-based motion models. Finally, in some implementations, additional care may be taken during region-based motion estimation to avoid including inactive regions of a frame.

[0018] As described in more detail below, in operation, once the best motion model is selected for each region type (e.g., background region and one or more front regions) per frame, the model parameters require efficient coding. For this, implementations here may use a hybrid approach combining a small codebook per region and directly coding the remaining coefficients of that region using the past prediction if no closest match for the region-based parameters is found in the codebook. Some rate-distortion trade-offs may be used to keep the coding bit cost low.Furthermore, because a current region-based motion model and a previous region-based motion model used for prediction may differ in the number and type of coefficients, a coefficient mapping strategy can be used by the implementations here to enable successful prediction, which can reduce the residual coefficients that need to be encoded. The codebook index or the coefficient residuals per region can be entropy encoded and sent to a decoder.

[0019] After entropy coding the coefficient residuals, to which a prediction is added to generate the reconstructed coefficients, or alternatively using the coefficients indexed by the codebook per region, region-based motion compensation can be performed in the decoder, which may require sub-pixel interpolation. The headers in the encoded stream for each region type of a frame can be used to specify the interpolation accuracy and filtering from a selection of four available interpolation filter combinations to produce a correct region-based motion-compensated prediction; where various filtering options were evaluated at the encoder, and a best selection was made and signaled via a per-frame per region bitstream.

[0020] Fig.Figure 2 is an illustrative graphical representation of an exemplary region-based parametric motion analysis process 200 arranged in accordance with at least some implementations of the present disclosure. In various implementations, Fig. 2 illustrates the different steps in this process using a selected middle movement section of a “City” sequence. A reference frame F refThe sequence and the current frame F are shown, which are used in block motion estimation, which generates a motion field (MVF). The motion field is then filtered to a filtered MVF and is aided by other features (such as color and texture) to determine the segmented background / foreground regions per frame. Next, the best region-based parametric motion model is determined separately for each of the segmented regions. These models are then used to compute corresponding motion-compensated regions for each of the two regions. The generation of these models requires the use of the best corresponding interpolation filters to produce sub-pixel motion compensation with high accuracy.The two individual motion-compensated regions together form the motion-compensated interpolation frame, which is differentiated from the original frame to calculate the remaining frame, which is shown to have low energy almost everywhere for this sequence.

[0021] Accordingly, the implementations of the region-based motion analyzer system 100 can be implemented to provide the following improvements compared to the other solutions: they use moderate complexity only when absolutely necessary to reduce the motion-compensated residual; they provide a high degree of adaptivity for complex content; they provide a high degree of adaptivity for noisy content; they provide high robustness in consistently and accurately measuring global motion; the ability to handle static black bars / borders, thus not adversely affecting the calculated global motion; the ability to handle static logos and text overlays, thus not adversely affecting the calculated global motion.the improvements in the computed global motion estimate result in a frame with well-compensated global motion and consequently in a lower error of the global motion-compensated prediction; they typically provide good gain from global motion in scenes with small or slow-moving local motion regions; and / or they typically provide low bit cost of encoding the global motion parameters.

[0022] Fig.3 is an illustrative block diagram of a more detailed example region-based motion analyzer system 100 arranged in accordance with at least some implementations of the present disclosure. In various implementations, the example region-based motion analyzer system 100 may include a video scene analyzer 102 (e.g., an input video GOP processor 302 and a video preprocessor 304), a local block-based motion field estimator 104 (e.g., a block-based motion estimator), a local motion field noise reduction filter 106 (e.g., a motion vector noise reduction filter), a region segmenter 107 (e.g., a moving region segmenter), a multi-region-based motion estimator and modeler 108 (e.g., a region-based motion estimator and modeler), an adaptive region-based motion compensator 110 (e.g.,an adaptive region-based motion compensator), a region-based motion model parameter and head entropy encoder 112 (e.g., a region-based motion model and head entropy encoder), the like, and / or combinations thereof. For example, some of the individual components of the region-based motion analyzer system 100 may not be used in all embodiments. In such an example, the video scene analyzer 102 may be used in some implementations or eliminated in other implementations. Additionally, the region-based motion analyzer system 100 may include a reference frame storage buffer 306, a parameter initializer 308, and a parameter storage buffer 310.

[0023] As illustrated, the region-based motion analyzer system 100 may operate by first organizing the input video into the form of a group of pictures (GOP) via the input video GOP processor 302. Next, the current frame F and the reference frame F ref be analyzed in a preprocessing step to detect scene changes and to reset the memory buffers / codebook used for entropy coding via the video preprocessor 304. If the current frame is not the first frame in the scene, then the block-based motion estimation can be performed between the current frame F and the reference frame F ref via the local block-based motion field estimator 104, where F refvia the reference frame storage buffer 306. The resulting motion vector field (MVF) is susceptible to noise, so in an attempt to minimize the amount of noise-related outlier vectors, motion vector noise reduction filtering can be applied to the motion vector field (MVF) via the local motion field noise reduction filter 106. Next, the filtered vectors can be used to calculate the motion-based region segmentation mask via the region segmenter 107, referred to as regions in this block diagram. The core of the proposed algorithm is adaptive region-based parametric motion estimation and modeling, which can use the filtered motion vectors and the region mask as the input to the multi-region-based motion estimator and modeler 108.This step can utilize the adaptive selection of motion vectors for region-based parametric motion estimation. Additionally, multiple models (e.g., three models) of varying complexity can be evaluated, and the most appropriate model can be selected to model the region-based motion region of the current frame. The computed region-based motion models (RMMs) can then be passed to the compensation step, which uses subpixel interpolation filtering adaptively selected via the adaptive region-based motion compensator 110 (e.g., from four available filters) that best matches the texture type in the given region. The RMM parameters can be converted to the representation of the MVs of the reference points and reconstructed with quantized accuracy.The adaptive region-based motion compensator 110 can output the reconstructed frame and the final SAD / residuals. Finally, the RMM parameters (in the form of the MVs of the reference points) can be encoded with a codebook-based entropy encoder via the region-based motion model parameter and head entropy encoder 112. The parameters can be encoded either as an index of the existing RMM from the codebook or as the residuals from an existing codeword via the region-based motion model parameter and head entropy encoder 112. The residuals can be encoded with adaptive modified Exp-Golomb codes (e.g., three tables with codes of different peak qualities are used).

[0024] The video preprocessor 304 may perform scene change detection. The current and a reference (e.g., the previous) frames are analyzed to detect the beginning of a new scene and reinitialize the information related to the previous frames. For example, the video preprocessor 304 may signal the parameters to an initializer to initialize the parameter storage buffer 310 in response to a particular scene change. For example, the preprocessor 304 may perform spatial subsampling of the input video for scene change detection. Conversion of the YUV420 input frames to block-accurate YUV444 frames may be performed (where, for example, Y is in 4 × 4 block accuracy, while U and V are in 2 × 2 block accuracy).Additionally, advanced scene change detection (SCD) can be performed to detect the beginning of a new scene and reinitialize the information related to previous frames.

[0025] The local block-based motion field estimator 104 may use the block-based motion estimation to generate a block-level motion vector field between the current frame and the reference frame(s). In one implementation, graphics hardware-accelerated video motion estimation (VME) routines may be used to generate block-based motion vectors.

[0026] The local motion field denoising filter 106 may use motion vector filtering to generate a smoother motion vector field from the existing raw field generated by the local block-based motion field estimator 104. This process may serve to eliminate outlier vectors from the motion vector field.

[0027] The region segmenter 107 may perform the segmentation of a current frame into moving regions. In such operations, a current frame may be divided into a given number of motion regions. This process may include the following steps: (1) computing a global motion model for the segmentation operations; (2) performing background motion region segmentation; (3) performing segmentation of the remaining (e.g., the remaining foreground) motion regions; and / or (4) performing morphologically based post-processing of the segmented regions.

[0028] The region segmenter 107 may perform the computation of the global motion model for segmentation. For example, the region segmenter 107 may estimate a global motion model for the current frame from which a foreground / background mask may be computed. This step may include computing an initial affine global motion model via random sampling and then generating a candidate set of motion vector selection masks from which the final affine global motion model for segmentation may be obtained. The selection masks may be used to specify which vectors should be used when estimating the model parameters.

[0029] The region segmenter 107 can perform background motion region segmentation. For example, the region segmenter 107 can calculate the background motion region of the current frame using the motion assisted by color segmentation. In such a color segmentation-assisted motion process, two probabilistic maps can be obtained: a global motion probabilistic map, or GMP for short, and a dominant color probabilistic map, or DCP for short. In this process, a GMP map can be used to generate the initial foreground / background segments. If the background motion region (as defined, for example, by the background motion segments) contains very little texture, a shape of the region can be assisted by its DCP map.

[0030] The region segmenter 107 may perform the segmentation of the remaining (e.g., the remaining foreground) motion regions. For the remaining foreground regions, the region segmenter 107 may calculate the GMP map of each region (e.g., the number of remaining foreground regions may be one less than the total number of motion regions determined in the step of estimating the number of motion regions). The region segmenter 107 may use the GMP map of each region to generate the motion region segments in the current frame. If a region has very little texture, its DCP map may be calculated and used to correct the shape of that foreground region.

[0031] Region segmenter 107 can perform morphologically based post-processing of the segmented regions. For example, region segmenter 107 can use morphological opening and closing to clean the motion region segmentation mask of a potential type of shot noise, which is typically common to almost all segmentation methods. Additionally, small object removal can also be used to remove noise-related small segmented blobs. Finally, the region boundary of a mask can be smoothed by a smoothing filter to remove small spikes and similar artifacts.

[0032] The multi-region motion estimator and modeler 108 may utilize region-based motion model generation, which may include multiple steps. Such operations may include, for example, multiple steps: (1) selecting which motion vectors to include in the parametric model estimation for each region, (2) adapting the subpixel filtering method for each region (because, for example, different regions may have different texture properties), and / or (3) adaptively selecting one motion model per region. Such adaptive selection of motion models per region may be used to estimate near-optimal parametric region-based motion models.

[0033] The multi-region-based motion estimator and modeler 108 can perform the selection of motion vectors for region-based motion model estimation. For example, a random sampling-based global motion estimation approach can be used to estimate an initial affine model of global motion for each region. Blocks whose global motion vector is similar to the corresponding block-based motion vector can be marked as selected. Such operations can be performed hierarchically for each region separately by increasing the similarity threshold to multiple hierarchy levels (e.g., four hierarchy levels). An additional mask (e.g., the fifth hierarchy level) can be obtained by eroding the global / local mask from a first hierarchy level.For each mask within a region, an affine model can be computed, using its SAD-based region-level error estimate to select the best mask. If none of the hierarchical refinement models beats the initial affine model (e.g., with respect to the smallest error), the selected block inclusion / exclusion mask for the given region can be fixed to include all blocks from that region.

[0034] The multi-region motion estimator and modeler 108 may select an adaptive region-based subpixel filter. This operation may be performed adaptively depending on the sharpness of the video content within a region. For example, there may be four (or another suitable number) subpixel filtering methods selected for different types of video content, including, for example, the following filter types: (1) a 1 / 16-pixel accurate bilinear filter, used primarily for content with a fuzzy texture, (2) a 1 / 16-pixel accurate bicubic filter, used for content with somewhat blurry and normal texture levels, (3) a 1 / 8-pixel accurate AVC-based filter, typically used for normal and somewhat sharp content, and (4) a 1 / 8-pixel accurate HEVC-based filter, typically used for the sharpest types of content.The selection can be performed using an error measure estimate, where the filter with the smallest error estimate per range can be selected.

[0035] The multi-domain motion estimator and modeler 108 can perform adaptive domain-based motion model calculation and selection. In such operations, there may be multiple (e.g., two) defined modes that can adapt between different motion models for each domain: (1) a mode 0 (the default mode) that can adaptively switch between a domain-based translational 4-parameter, affine 6-parameter, and pseudo-perspective 8-parameter motion model on a per-frame basis; and (2) a mode 1 that can adaptively switch between a domain-based affine 6-parameter, pseudo-perspective 8-parameter, and biquadratic 12-parameter motion model on a per-domain basis.

[0036] The adaptive region-based motion compensator 110 can perform region-based motion model-based compensation. Such region-based motion model-based compensation can be performed for each block at a pixel level within the block using a corresponding region-based motion model with the selected sub-pixel filtering method. For each pixel within a block, a motion vector can be calculated using the corresponding region-based motion model, and the pixel can be moved to a sub-pixel position according to the predetermined sub-pixel filtering method. Consequently, a pixel on one side of the block can have a different motion vector than a pixel on the other side of the same block. The compensation can be performed using quantized / reconstructed region-based motion model parameters.Additionally, the parameter coefficients can be represented as a quotient with a scaled denominator (e.g., to a power of two) to achieve fast performance (e.g., using bitwise shift instead of division).

[0037] The region-based motion model parameter and head entropy encoder 112 may perform codebook-based encoding of the parameters of the region-based motion model. Such codebook-based encoding of the parameters of the region-based motion model may, for example, be used to encode the parameters of the region-based motion model. Such codebook-based encoding of the parameters of the region-based motion model may be based on the concept of reference points. The motion vectors corresponding to the reference points may be predicted, with the residuals being encoded with modified Exp-Golomb codes. The predictions may be generated from the codebook containing several (e.g., up to eight) most recently occurring parameters of the region-based motion model separately for each region.

[0038] Some implementations described here generally relate to improvements in motion estimation, representation, and compensation, which are key components of an interframe coding system that can directly improve the overall coding efficiency of interframe coding. Specifically, some of the implementations described here introduce systems and methods to enable significant improvements in global motion estimation, global motion compensation, and global motion parameter coding to improve interframe coding efficiency. The improvements include, but are not limited to, improved modeling of complex global motion and a compact representation of global motion parameters.In comparison, conventional interframe video coding typically uses block-based motion estimation, motion compensation, and motion vector coding, which can mainly compensate for local translator motion, and consequently is not only limited in many ways in its ability to deal with complex global motion, but also does not enable efficient motion representation.

[0039] For reference, block-based motion estimation is the core approach to motion compensation in both recent video coding standards, such as ITU-T H.264 / ISO MPEG AVC and ITU-T H.265 / ISO MPEG HEVC, as well as in upcoming standards under development, such as ITU-T H.266 and the AOM AV1 standard. Fig. 5-7 below provide a brief overview of interframe video coding at a high level.

[0040] With respect to the area-based motion analyzer system 100 according to the Fig. 1 and / or 3, there are a number of practical problems in the development of a system for the work of estimating and compensating for global motion. [1] High complexity - the calculation of the global motion estimate (which typically begins after calculating the block motion vectors) is a highly computationally intensive process in an open form, typically requiring a least-squares minimization type solution, which is iterative and whose fast convergence is not always guaranteed; [2] Motion range constraints - if the motion range of the initial block is insufficient with respect to a fast actual motion, the resulting global motion estimate is likely to be quite inaccurate, leading to a large prediction residual; [3] Insufficient robustness - noisy motion vectors contribute to misdirection of the global motion estimation process, which not only makes convergence difficult but can also lead to poor global motion estimates; [4] the interaction of local / global motion - the local motion of objects often interferes with the global motion calculation, causing global motion estimates to be either slightly inaccurate or even significantly erroneous; [5] the mismatch of the motion model to the actual motion in the scene - For example, if a fixed four-parameter model of global motion is used to represent changes in perspective in a video scene, the measured motion parameters may be erroneous; [6] the limitations of the extent of frame boundaries for large motion cases - while this issue affects both local block motion compensation and global motion compensation, the local block motion vectors do not have to follow the actual motion, sometimes providing reasonable results due to random coincidences;[7] limitations due to an inactive static region in the content - if such a region (including, for example, the presence of black bars, black borders, letterboxing, pillarboxing, etc.) is not discarded from the global motion estimation, the resulting estimates may be erroneous, leading to poor motion-compensated prediction; [8] limitations due to static logos or overlaid text - if a region of static logos and a globally moving background region are not separated for global motion estimation, then it is difficult to find an accurate global motion estimate, and consequently the quality of global motion compensation is likely to suffer;[9] the coding bit cost of the global motion parameters - because both local and global motion tend to be present in a video scene simultaneously, sending the global motion parameters alone is not sufficient, and they must be sent in addition to the local motion vectors; consequently, only a limited bit cost of global motion can be afforded; and

[10] the accuracy of global motion compensation - if limited-precision sub-pixel interpolation is performed with simpler filters to reduce the complexity of motion compensation (this may be important, for example, because this process must be performed at both the encoder and the decoder), the resulting prediction is often fuzzy, producing no small residual signal.

[0041] Fig.4 is an illustrative block diagram of an example video encoder 400 arranged in accordance with at least some implementations of the present disclosure. In various implementations, video encoder 400 may be configured to perform video encoding and / or implement video codecs according to one or more evolved video codec standards, such as, but not limited to, the evolved video coding video compression standard (e.g., AVC / H.264) or the high performance video coding video compression standard (e.g., HEVC / H.265). Further, in various embodiments, video encoder 400 may be implemented as part of an image processor, a video processor, and / or a media processor.

[0042] The term "encoder," as used herein, may refer to an encoder and / or a decoder. Similarly, the term "encoding," as used herein, may refer to encoding via an encoder and / or decoding via a decoder. For example, video encoder 400 may include a video encoder with an internal video decoder, as shown in Fig. 4, while a companion encoder may include a video encoder (not independently illustrated here), both of which are examples of an “encoder” that can encode.

[0043] In some examples, the video encoder 400 may include additional elements, which for clarity are shown in Fig.4. The video encoder 400 may include, for example, a processor, a radio frequency (RF) transmitter / receiver, a display, an antenna, and / or the like. Furthermore, the video encoder 400 may include additional elements, such as a speaker, a microphone, an accelerometer, a memory, a router, network interface logic, and / or the like, which are not shown for clarity. Fig. 4 have not been shown.

[0044] The video encoder 400 may operate on the general principle of interframe coding or, more specifically, motion compensated transform (DCT) coding on which modern standards are based (although some details may be different for each standard).

[0045] Motion estimation is performed using fixed- or variable-size blocks of one frame of video with respect to another frame, resulting in shifting motion vectors that are then encoded and sent to the decoder, which uses these motion vectors to generate motion-compensated prediction blocks. While interframe encoders support both intra- and interframe coding, it is interframe coding (which involves efficiently encoding the residual signal between the original blocks and the corresponding motion-compensated prediction blocks) that provides the significant coding gain.One thing to note is that encoding a large number of highly accurate block motion vectors (due to variable-block-size partitioning and motion-compensated motion with at least 1 / 4-pixel accuracy, as required to reduce the residual signal) poses a challenge for efficient video coding due to the required coding bits for the motion vectors, even though clever motion vector prediction and coding techniques have been developed. Another problem with block motion vectors is that they can, at best, represent a translational motion model and cannot faithfully represent complex motion.

[0046] The key idea in modern interframe coding is therefore to combine temporally predictive (motion-compensated) coding, which adapts to the movement of objects between frames of the video and is used to calculate a motion-compensated difference residual, and spatial transform coding, which converts the spatial blocks of pixels into blocks of frequency coefficients, typically by a DCT (with a block size such as 8 × 8), followed by a reduction in the accuracy of these DCT coefficients by quantization to adapt the video quality to the available bit rate.Because the resulting transform coefficients have energy redistributed toward lower frequencies, some of the small-valued coefficients become zero after quantization, just as some of the high-frequency coefficients with higher quantization errors may be encoded or even skipped altogether. These and other properties of the transform coefficients, such as frequency location and the fact that some quantized levels occur more frequently than others, enable the use of frequency-domain sampling of the coefficients and entropy coding (in its most basic form, variable-length coding) to achieve additional compression gains.

[0047] Interframe coding involves coding using up to three picture types (e.g., I-pictures, P-pictures, and B-pictures) arranged in a fixed or adaptive picture structure that is repeated multiple times and collectively referred to as a group of pictures (GOP). The I-pictures are typically used to provide a cleaning refresh for random access (or channel changing) at frequent intervals. The P-pictures are typically used for basic interframe coding using motion compensation and can be used sequentially or interlaced with an arrangement of B-pictures; the P-pictures can provide moderate compression. The B-pictures, which are bidirectionally motion-compensated and encoded interframe pictures, can provide the highest level of compression.

[0048] Because motion compensation is difficult to perform in the transform domain, the first step in an interframe encoder is to generate a motion-compensated prediction error in the pixel domain. For each block of the current frame, a prediction block is found and distinguished in the reference frame using the motion vector calculated during motion estimation to generate the prediction error signal. The resulting error signal is transformed using a 2D DCT, quantized by an adaptive quantizer (e.g., a "quantizer") 408, and encoded using an entropy encoder 409 (e.g., a variable-length coder (VLC) or an arithmetic entropy coder), and buffered for transmission over a channel.

[0049] As illustrated, the video content in operation 404 may be distinguished with the output from the internal decoding loop 405 to form the remaining video content.

[0050] The residual content can be subjected to 406 video transformation operations in a transformation module (e.g., the “block DCT”) and 408 video quantization processes in a quantizer (e.g., a “quantizer”).

[0051] The output of the transform module (e.g., the "block DCT") 406 and the quantizer (e.g., the "quantizer") 408 may be provided to an entropy encoder 409 and an inverse transform module (e.g., the "inverse quantizer") 412 and a dequantization module (e.g., an "inverse block DCT") 414. The entropy encoder 409 may output an entropy-encoded bitstream 410 for transmission to a corresponding decoder.

[0052] Within an internal decoding loop of video encoder 400, the inverse transform (e.g., "inverse quantization") module 412 and the dequantization (e.g., "inverse block DCT") module 414 may implement the inverse of the operations performed by the transform (e.g., "block DCT") module 406 and the quantizer (e.g., "quantizer") 408 to provide the recovered residual content. The recovered residual content may be added to the output from the internal decoding loop to form the reconstructed decoded video content. Those skilled in the art may appreciate that the transform and quantization modules, the dequantization modules, and the inverse transform modules described herein may employ scaling techniques.The decoded video content may be provided to a decoded image store 120, a motion estimator 422, a motion-compensated predictor 424, and an intra-predictor 426. A selector 428 (e.g., a "selector") may output mode information (e.g., intra-mode, intermediate-mode, etc.) based on the intra-prediction output of the intra-predictor 426 and the intermediate-prediction output of the motion-compensated predictor 424. It is recognized that the same and / or similar operations as described above may be performed in the decoder-only implementations of the video encoder 400.

[0053] Fig.5 is an illustrative block diagram of an exemplary advanced video coding (AVC) video encoder 500 arranged in accordance with at least some implementations of the present disclosure. In various implementations, video encoder 500 may be configured to perform video coding and / or implement video codecs in accordance with one or more advanced video codec standards, such as, but not limited to, the advanced video coding video compression standard (e.g., AVC / H.264) or the high performance video coding video compression standard (e.g., HEVC / H.265). Further, in various embodiments, video encoder 500 may be implemented as part of an image processor, a video processor, and / or a media processor.

[0054] As illustrated, Fig.5 is a block diagram of an AVC encoder that follows the principles of the generalized interframe video encoder 400 according to Fig. 4, discussed earlier, follows. Each frame is partitioned into macroblocks (MBs) corresponding to 16 × 16 luminance (and two corresponding 8 × 8 chrominance) signals. Each MB can potentially be used as is, or partitioned into either two 16 × 8, two 8 × 16, or four 8 × 8 for prediction. Each 8 × 8 can also be used as is, or partitioned into two 8 × 4, two 4 × 8, or four 4 × 4 for prediction. The exact partitioning decision depends on the available coding bit rate versus distortion optimization (full or partial).

[0055] For each MB, a coding block mode can be assigned from an intra-, inter-, or skip mode in unidirectionally predicted (P) pictures. B (bidirectional) predicted pictures are also supported and include an additional MB or block-based direct mode. Even P pictures can refer to multiple (4 to 5) previous references.

[0056] In the high profile, the allowed transform block size is 4 × 4 and 8 × 8, which encode the residual signal (generated by the intra-prediction or the motion-compensated intermediate prediction). The generated transform coefficients are quantized and entropy-coded using an arithmetic encoder with context-adaptive binary arithmetic coding (CABAC). A filter in the coding loop ensures that the interfering block noise is filtered out, benefiting both objective and subjective quality.

[0057] In some examples, during operation of video encoder 500, the current video information may be provided to a picture reorderer 542 in the form of a slice of the video data. Picture reorderer 542 may determine the picture type (e.g., I, P, or B slices) of each video slice and reorder the video slices as needed.

[0058] The current video frame can be partitioned so that each MB can potentially be used as is, or partitioned into either two 16×8, two 8×16, or four 8×8 for prediction, where each 8×8 can also be used as is, or partitioned into two 8×4, two 4×8, or four 4×4 for prediction in the prediction partitioner 544 (e.g., the "MB partitioner"). An encoding partitioner 546 (e.g., a "4×4 / 8×8 residual partitioner") can partition the remaining macroblocks.

[0059] The coding partitioner 546 may first be subjected to known video transformation and quantization processes by a transform 548 (e.g., a 4 × 4 DCT / 8 × 8 DCT), which may, for example, perform a discrete cosine transform (DCT) operation. Next, a quantizer 550 (e.g., a quantizer) may quantize the resulting transform coefficients.

[0060] The output of the transform and quantization operations may be provided to an entropy encoder 552, as well as an inverse quantizer 556 (e.g., an inverse quantizer) and an inverse transform 558 (e.g., an inverse 4 × 4 DCT / inverse 8 × 8 DCT). Encoder 552 (e.g., the "CAVLC / CABAC encoder") may output an entropy-encoded bitstream 554 for transmission to a corresponding decoder.

[0061] Within the internal decoding loop of the video encoder 500, an inverse quantizer 556 and an inverse transform 558 may implement the inverse of the operations performed by the transform 548 and the quantizer 550 to provide the output to a remainder assembler 560 (e.g., a 4 × 4 / 8 × 8 remainder assembler).

[0062] The output of the residual assembler 560 may be provided to a loop that includes a prediction assembler 562 (e.g., a block assembler), a deblocking filter 564, a decoded picture buffer 568, a motion estimator 570, a motion-compensated predictor 572, a decoded macroblock line plus one buffer 574 (e.g., a decoded MB line + 1 buffer), an intra-prediction direction estimator 576, and an intra-predictor 578. As in Fig.5B, the output of either the motion-compensated predictor 572 or the intra-predictor 578 is selected via a selector 580 (e.g., a selector), and may be combined with the output of the residual assembler 560 as input to the deblocking filter 564, and may be distinguished from the output of the prediction partitioner 544 to act as an input to the encoding partitioner 546. An encoding controller 582 (e.g., an encoding controller, RD optimizer, and rate controller) may operate to perform rate distortion optimization (RDO) operations and control the rate of the video encoder 500.

[0063] Fig.6 is an illustrative block diagram of an exemplary high-performance video coding (HEVC) video encoder 600 arranged in accordance with at least some implementations of the present disclosure. In various implementations, the video encoder 600 may be configured to perform video coding and / or implement video codecs according to one or more evolved video codec standards, such as, but not limited to, the evolved video coding video compression standard (e.g., AVC / H.264) or the high-performance video coding video compression standard (e.g., HEVC / H.265). Further, in various embodiments, the video encoder 600 may be implemented as part of an image processor, a video processor, and / or a media processor.

[0064] As in Fig.6, the high-level operation of the video encoder 600 follows the principles of the general interframe encoder described earlier Fig. 4. The video encoder 600 according to Fig. 6, for example, is also a motion-compensated interframe transform coder, typically using a combination of either only I- and P-pictures or I-, P-, and B-pictures (it is stated that in HEVC, a generalized B-picture (GBP) may be used instead of the P-picture) in a non-pyramid or pyramid GOP arrangement. Furthermore, like H.264 / AVC coding, not only B-pictures (which may use bidirectional references) but also a P-picture may use multiple references (where these references are unidirectional for P-pictures). As in previous standards, B-pictures include forward and backward references, thus necessitating picture scrambling.

[0065] In some examples, during operation of video encoder 600, the current video information may be provided to a picture reorderer 642 in the form of a frame of video data. Picture reorderer 642 may determine the picture type (e.g., I-, P-, or B-frame) of each video frame and reorder the video frames as needed.

[0066] The current video frame may be split from the largest coding units (LCUs) into coding units (CUs), where a coding unit (CU) may be recursively partitioned into smaller coding units (CUs). Additionally, the coding units (CUs) may be partitioned into prediction units (PUs) for prediction in the prediction partitioner 644 (e.g., the "LC_CU and PU partitioner"). An encoding partitioner 646 (e.g., a "REs_CU_TU partitioner") may partition the remaining coding units (CUs) into transformation units (TUs).

[0067] The coding partitioner 646 may be subjected to known video transformation and quantization processes, first by a transform 648 (e.g., a 4 × 4 DCT / VBS DCT), which may, for example, perform a discrete cosine transform (DCT) operation. Next, a quantizer 650 (e.g., a quantizer) may quantize the resulting transform coefficients.

[0068] The output of the transform and quantization operations may be provided to an entropy encoder 652, as well as an inverse quantizer 656 (e.g., an inverse quantizer) and an inverse transform 658 (e.g., an inverse 4 × 4 DCT / VBS DCT). The entropy encoder 652 may output an entropy-encoded bitstream 654 for transmission to a corresponding decoder.

[0069] Within the internal decoding loop of the video encoder 600, an inverse quantizer 656 and an inverse transform 658 may implement the inverse of the operations performed by the transform 648 and the quantizer 650 to provide the output to a residual assembler 660 (e.g., a residual TU_CU assembler).

[0070] The output of the residual assembler 660 may be provided to a loop that includes a prediction assembler 662 (e.g., a PU_CU and CU_LCU assembler), a deblocking filter 664, an adaptive sample offset filter 666 (e.g., an adaptive sample offset (SAO)), a decoded picture buffer 668, a motion estimator 670, a motion-compensated predictor 672, a buffer 674 for a decoded line of a largest coding unit plus one (e.g., a buffer for a decoded LCU line + 1), an intra-prediction direction estimator 676, and an intra-predictor 678. As in Fig.6B, the output of either the motion-compensated predictor 672 or the intra-predictor 678 is selected via a selector 680 (e.g., a selector), and may be combined with the output of the residual assembler 660 as input to the end-blocking filter 664, and may be distinguished from the output of the prediction partitioner 644 to act as an input to the encoding partitioner 646. An encoding controller 682 (e.g., an encoding controller, RD optimizer, and rate controller) may operate to perform rate-distortion optimization (RDO) operations and control the rate of the video encoder 600.

[0071] In operation, the partitioner of the largest coding unit (LCU) to the coding units (CU) partitions the LCUs to the CUs, where a CU can be recursively partitioned into smaller CUs. The partitioner of the CU to a prediction unit (PU) partitions the CUs for prediction into the PUs, where the TU partitioner partitions the remaining CUs into the transform units (TUs). The TUs correspond to the size of the transform blocks used in transform coding. The transform coefficients are quantized according to Qp in the bitstream. The different Qps can be specified for each CU depending on the maxCuDQpDepth, with LCU-based adaptation being the smallest granularity. The coding decisions, the quantized transformed difference, and the motion vectors and modes are encoded in the bitstream using a context-adaptive binary arithmetic coder (CABAC).

[0072] An encoder controller controls the degree of partitioning performed, which depends on the quantizer used in transform coding. The CU / PU assembler and the TU assembler perform the inverse function of the partitioner. The decoded (each DPCM encoder contains a decoder loop) intra- / motion-compensated difference partitions are assembled after the inverse DST / DCT, to which the PUs are added. The reconstructed signal is then deblocked and SAO-filtered, which accordingly reduce the occurrence of artifacts and restore the edges affected by the coding. HEVC uses intra- and inter-prediction modes to predict the sections of the frames and encodes the difference signal by transforming it. HEVC uses various transform sizes called transform units (TU).The transform coefficients are quantized according to the Qp in the bitstream. Different Qps can be specified for each CU depending on the maxCuDQpDepth.

[0073] AVC or HEVC coding classifies images or frames into one of three basic picture types: I-picture, P-picture, and B-picture. Both AVC and HEVC also allow for out-of-order coding of B-pictures, with a typical method being to encode a group of pictures (GOP) out of order in a pyramid configuration. The typical pyramid GOP configuration uses a group of pictures (GOP) size (gopsz) of 8 frames. The out-of-order delay of the B-pictures in the pyramid configuration is referred to as the picture level in the pyramid (piclvl).

[0074] Fig.7 shows an exemplary structure of a group of images 700. The group of images 400 shows a first 17 frames (frames 0 to 16) which include a first frame (a frame 0), an intraframe, followed by two GOPs, each with eight images. In the first GOP, frame 8 is a P-frame (or may also be a generalized B-frame (GPB frame)), being a level 0 frame in the pyramid. Whereas frame 1 is a first-level B-frame, frames 2 and 6 are second-level B-frames, and frames 1, 3, 5, and 7 are all third-level B-frames. For example, frame 1 is called a P-frame (or may also be a generalized B-frame (GPB frame)). B. is called the first-level B-frame because it only needs the I-frame (or the last P-frame of the previous GOP) as the previous reference and the actual P-frame of the current GOP as the next reference to generate the predictions necessary to encode frame 1.In fact, frame 1 may use more than 2 references, although 2 references may be used to illustrate the principle. Furthermore, frames 2 and 6 are referred to as the second-level B-frames because they use the first-level B-frame (frame 1) as a reference along with an adjacent I- and P-frame. Similarly, the level 3 B-frames use at least one level 2 B-frame as a reference. A second GOP (frames 9 to 16) of the same size is shown, which uses the encoded P-frames of the previous GOP (instead of the I-frame as in the case of the previous GOP), e.g., frame 8, as a reference; the rest of the second GOP operates exactly the same as the first GOP. Regarding the coding order, the coded bit stream encodes frame 0, followed by frame 8, frame 4, frame 2, frame 1, frame 3, frame 6, frame 5, frame 7, etc., as shown in the figure.

[0075] The term "encoder," as used herein, may refer to an encoder and / or a decoder. The term "encoding," as used herein, may similarly refer to encoding via an encoder and / or decoding via a decoder. For example, video encoder 400, 500, 600, and the like may include a video encoder with an internal video decoder, as described in the Fig. 4, Fig. 5 and Fig. 6, while a companion encoder may only include a video decoder (not independently illustrated here), both of which are examples of an “encoder” capable of encoding. Models of the global movement:

[0076] Fig.Figure 8 is an illustrative graphical representation of various example global motion models 800 related to chirping, arranged in accordance with at least some implementations of the present disclosure. In various implementations, the example global motion models 800 may include a translational non-chirping model, an affine non-chirping model, a bilinear non-chirping model, a perspective (e.g., projective) chirping model, a pseudo-perspective chirping model, a biquadratic chirping model, the like, and / or combinations thereof.

[0077] A number of global motion models have been proposed in the published literature. Generally, a particular motion model represents a compromise between the complexity of the model and its ability to handle various types of camera-related motion, depth / perspective projections of the scene, etc. Models are often classified into linear (i.e., simple) models and nonlinear (i.e., complex) models. Linear models can handle normal camera operations, such as translational motion, rotation, and even zoom. More complex models, which are typically nonlinear and contain at least one quadratic (or higher-order) term, are often used in cases where there is complex scene depth, strong perspective projection effects in the scene, or simply when more accuracy is needed for a given application.One disadvantage of nonlinear models is their higher computational complexity. On the other hand, translational and affine models are more error-prone when a noisy motion vector field is used for the GME. The most commonly used models for estimating global motion in video coding applications are simpler, linear models.

[0078] A given vector field (MX i , MY i ), i = 0, ..., N - 1, where N is the number of motion vectors in the frame. Then, each position (x i , y i ), which corresponds to the center of block i of the frame, according to the motion vector (MX i , MY i ) as follows (xi',yi') moves: xi'=xi+MXi, yi'=yi+MXi.

[0079] A simple 4-parameter motion model aims to approximate these movements of the global motion of the frame positions by a single linear equation with a total of 4 parameters {a0, a1, a2, a3}: xi'=a0xi+a1, yi'=a2yi+a3.

[0080] This equation defines a translational 4-parameter motion model. Another 4-parameter model is called a pseudo-affine 4-parameter motion model. A pseudo-affine motion model is defined as: xi'=a0xi+a1yi+a2, yi'=a0yi−a1xi+a3 defined.

[0081] The advantage of the pseudo-affine model is that it can often estimate additional types of global motion while having the same number of parameters as a simple translational model. One of the most commonly used motion models in practice is the 6-parameter affine global motion model. It can more accurately estimate most of the typical global motion caused by camera operations. The affine model is defined as follows: xi'=a0xi+a1yi+a2, yi'=a3xi+a4yi+a5.

[0082] Unfortunately, linear models cannot properly handle camera panning and tilting. Consequently, nonlinear models are required for video scenes with these effects. More complex models are typically represented with quadratic terms. They are widely used for video applications such as medical imaging, remote sensing, or computer graphics. The simplest nonlinear model is the bilinear 8-parameter model of global motion, known as: xi'=a0xiyi+a1xi+a2yi+a3, yi'=a4xiyi+a5xi+a6yi+a7 defined.

[0083] Another popular 8-parameter model is a perspective (or projective) 8-parameter model of global motion. This model is designed to handle video scenes with strong perspective, generating a global motion field that follows a more complex nonlinear distribution. The projective model is defined as follows: xi'=a0xi+a1yi+a2a6xi+a7yi+1, yi'=a3xi+a4yi+a5a6xi+a7xi+1.

[0084] A variant of the perspective model, called the pseudoperspective model, has been shown to have good overall performance because it can handle perspective projections and related effects, such as chirping (the effect of increasing or decreasing the spatial frequency with respect to spatial location).

[0085] A pseudoperspective 8-parameter model is defined as follows: xi'=a0xi2+a1xiyi+a2xi+a3yi+a4, yi'=a1yi2+a0xiyi+a5xi+a6yi+a7.

[0086] The pseudoprojective model has an advantage over the perspective model because it typically has lower computational complexity during the estimation process, while simultaneously handling all perspective-related effects in a global 2D motion field. It has been known that the perspective model is notoriously difficult to estimate and often requires many more iterations in the estimation process.

[0087] Fig. Figure 8 shows the image effects of the different modeling functions. As illustrated, the pseudoperspective model can produce both chirping and convergence effects, being the best approximation of a perspective image using low-order polynomials. It also shows that the bilinear function, although it also has eight parameters like the perspective and pseudoperspective models, cannot capture the chirping effect.

[0088] Finally, for video applications requiring very high modeling accuracy, a biquadratic model can be used, which can handle all degrees of freedom of camera operations and all effects of perspective mapping. It is a 12-parameter model and therefore the most expensive in terms of coding costs. The 12-parameter biquadratic model is defined as follows: xi'=a0xi2+a1yi2+a2xiyi+a3xi+a4yi+a5, yi'=a6xi2+a7yi2+a8xiyi+a9xi+a10yi+a11.

[0089] Table 1 summarizes the global motion models mentioned above. It is possible to define other models, even with higher-order polynomials (e.g., a 20-parameter bicubic model), but they are rarely used in practice due to the extremely high coding costs. Table 1 - Summary of the models of global movement and their properties: Movement model Model equation Number of parameters translational <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> (x'y')=(a0x+a1a2x+a3)<h2 style=";text-align:left;direction:ltr"> 4 pseudoaffin <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> (x'y')=(a0x+a1y+a2a0y−a1x+a3)<h2 style=";text-align:left;direction:ltr"> 4 affin <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> (x'y')=(a0x+a1y+a2a3x−a4y+a5)<h2 style=";text-align:left;direction:ltr"> 6 bilinear <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> (x'y')=(a0xy+a1x+a2y+a3a4xy+a5x+a6y+a7)<h2 style=";text-align:left;direction:ltr"> 8 perspective <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> (x'y')=((a0x+a1y+a2) / (a6x+a7y+1)(a3x+a4y+a5) / (a6x+a7y+1))<h2 style=";text-align:left;direction:ltr"> 8 pseudo-perspective (x'y')=(a0xi2+a1xiyi+a2xi+a3yi+a4a0yi2+a0xiyi+a5xi+a6yi+a7) 8 biquadratic (x'y')=(a0x2+a1y2+a2xy+a3x+a4y+a5a6x2+a7y2+a8xy+a9x+a10y+a11) 12

[0090] The approaches to estimating a model of global movement: The most common techniques used to estimate the parameters of the desired model are based on least squares fitting. Next, several least-squares-based methods used to calculate the parameters of a global motion model are described.

[0091] The model of global movement - the least squares estimation: The least-squares fitting method is often used to estimate the optimal parameter values ​​of the motion model. It is a standard approach for finding solutions to overdetermined systems (e.g., sets of equations with more equations than unknowns).

[0092] When estimating the global motion, a motion vector field is given, (MX i , MYi ), i = 0, ..., N - 1, where N is the number of motion vectors in the frame. According to the motion field, each position (x i , y i ), which corresponds to the center of block i of the frame, according to the motion vector (mx i , my i ) as follows (xi',yi') moves: xi'=xi+MXi, yi'=yi+MYi.

[0093] In a translational 4-parameter motion model, the goal is to approximate 4 parameters {a0, a1, a2, a3} such that the difference between the observed data (xi',yi') and the modeled data (a0x i + a1, a2y i + a3) is minimized. The least squares approach minimizes the following two square errors with respect to the parameters {a0, a1} and {a2, a3}: SEa0a1=∑i=0N−1(xi'−(a0xi+a1))2, SEa2a3=∑i=0N−1(yi'−(a2yi+a3))2.

[0094] Typically, the number of parameters (4 in this example) is much smaller than the total number of vectors used for estimation, making it an overdetermined system.

[0095] For linear models of global motion (such as translational, pseudo-affine, and affine), the minimum sum of squares is found by determining the partial derivatives with respect to each parameter and setting them to zero. This leads to a set of linear equations whose solution represents the global minimum in the sense of squared errors, e.g., the least-squares error. The above equation for a 4-parameter affine motion model with respect to {a0, a1} is expanded as follows: SEa0a1=∑i=0N−1(xi'2+a02xi2+a12−2a0xixi'−2a1xi'+2a0a1xi)2==∑i=0N−1xi '2+a02∑i=0N−1xi2+a12N−2a0∑i=0N−1xixi'−2a1∑i=0N−1xi'+2a0a1∑i=0N−1xi.

[0096] Determining the partial derivatives of the above equations yields the following system: ∂SEa0a1∂a0=2a0∑i=0N−1xi2+2a1∑i=0N−1xi−2∑i=0N−1xixi'=0, ∂SEa0a1∂a0=2a0∑i=0N−1xi+2a1N−2∑i=0N−1xi'=0.

[0097] The above system can be expressed as the following matrix equation, the solution of which determines the two unknown parameters {a0, a1}: (a0a1)=(∑i=0N−1xi2∑i=0N−1xi∑i=0N−1xiN)−1(∑i=0N−1xixi'∑i=0N−1x1').

[0098] Similarly, the second set of parameters {a2, a3} can be expressed as the solution of the following matrix equation: (a2a3)=(∑i=0N−1yi2∑i=0N−1yi∑i=0N−1yiN)−1(∑i=0N−1yiyi'∑i=0N−1y1').

[0099] If a determinant-based solution for the matrix inverse is used, the two matrix equations above can be further written as: (a0a1)=1N∑i=0N−1xi2−(∑i=0N−1xi)2(N−∑i=0N−1xi−∑i=0N−1xi∑i=0N−1xi2)(∑i=0N−1xixi'∑i=0N−1xi'); and (a2a3)=1N∑i=0N−1yi2−(∑i=0N−1yi)2(N−∑i=0N−1yi−∑i=0N−1yi∑i=0N−1yi2)(∑i=0N−1yiyi'∑i=0N−1yi') be expressed.

[0100] Finally, the matrix equations provide the following least squares expressions for directly solving the unknown parameters of a 4-parameter affine model of global motion: a0=N∑i=0N−1xixi'−∑i=0N−1xi∑i=0N−1xi'N∑i=0N−1xi2−(∑i=0N−1xi)2, a1=∑i=0N−1xi2∑i=0N−1xi'−∑i=0N−1xi∑i=0N−1xixi'N∑i=0N−1xi2−(∑i=0N−1xi)2, a2=N∑i=0N−1yiyi'−∑i=0N−1yi∑i=0N−1yi'N∑i=0N−1yi2−(∑i=0N−1yi)2, a3=∑i=0N−1yi2∑i=0N−1yi'−∑i=0N−1yi∑i=0N−1yiyi'N∑i=0N−1yi2−(∑i=0N−1yi)2

[0101] Using the same procedure, the least-squares fitting equations can be determined for pseudo-affine 4-parameter global motion models and affine 6-parameter global motion models. For nonlinear global motion models, a nonlinear least-squares fitting method, such as the Levenberg-Marquardt algorithm (LMA), can be used. An overview of the LMA is presented next.

[0102] The model of global motion - the Levenberg-Marquardt least squares solution: The Levenberg-Marquardt algorithm is a well-founded method for solving nonlinear least squares problems. It was first published by Levenberg in 1944 and rediscovered by Marquardt in 1963. The LMA is an iterative procedure. To begin a minimization, the user must provide an initial estimate for the parameters. Like many fitting algorithms, the LMA finds only a local minimum, which is not necessarily the global minimum. In the case of multiple minima, the algorithm only converges to the global minimum if the initial estimate is already fairly close to the final solution. In the context of estimating the parameters of the global motion model, setting the parameters to prior values ​​(e.g., the parameters of the previous frame(s)) generally improves performance.

[0103] The LMA interpolates between two different nonlinear least squares solution methods: (1) the Gauss-Newton algorithm (GNA) and (2) the gradient descent method. The LMA is more robust than the GNA in the sense that it often finds a solution, even when starting very far from the final minimum. Analysis has shown that the LNA is, in fact, the GNA with a confidence region, where the algorithm constrains the convergence step size to the size of the confidence region in each iteration to avoid straying too far from the optimum.

[0104] Once again (xi',yi') i = 0, ..., N - 1 the observed data, e.g., the new positions of the center (x i , y i ) of the i-th block of a frame moved according to the block-based motion vector field. A model is said to be separable if the xi'− and yi' model functions -Model functions have exactly the same structure of the independent variable and the parameter a k only when calculating either xi' or yi', but not used by both. Otherwise, the model is said to be inseparable. Therefore, the affine, bilinear, and biquadratic models are separable, while the translational, pseudo-affine, perspective, and pseudo-perspective models are inseparable.

[0105] Let β = (a0, a1, ..., a n-1 ) is the vector of parameters of an n-parameter model to be used to model the global motion. For a separable model of global motion, first the parameters β x' = (a0, ... , a (n / 2) -1 ), where the remaining parameters β y' = (a n / 2 , ..., a n - 1). On the other hand, for non-separable models, 2N data points are generated, where if i < N, the equation of the x'-model is used, while if N ≤ i < 2N, the equation of the y'-model is used. For simplicity of argument, the LMA algorithm for modeling global motion is described by the first part of calculating the separable parameters, e.g., calculating the parameter associated with the equation of the x'-model.

[0106] In each LMA iteration step, the parameter β is replaced by a new estimate β + δ. To determine the step vector δ, the functions f(x i , β + δ) are approximated by their linearizations as follows: f(xi,β+δ)≈f(xi,β)+Jiδ.

[0107] Whereby Ji=∂f(xi,β)∂β

[0108] Then the sum of the squared errors S(β + δ) is given as S(β+δ)≈∑i=0N−1(xi'−f(xi,β)−Jiδ)2 approximated.

[0109] The sum of the squared error function S has a gradient of zero with respect to β at its minimum. Determining the derivative of S(β + δ) with respect to δ and setting the result to zero yields the following equation: (JTJ)δ=JT(x'−f(β)).

[0110] Where J is the Jacobian matrix whose i-th row J i and f and x' are vectors whose i-th component f(x i , β) respectively. xi' This defines a set of linear equations whose solution is the unknown vector δ.

[0111] The Fig.9A-9D are illustrative diagrams 900, 902, 904, and 906 of an exemplary Levenberg-Marquardt Algorithm (LMA) curve-fitting model for approximating global motion, arranged in accordance with at least some implementations of the present disclosure. In various implementations of the LMA curve-fitting model for approximating global motion: diagrams 900 and 902 show graphical representations of the motion vector field (x-dimension) of the global motion range of the "Stefan" sequence (the points) and the affine 6-parameter model fit computed using the LMA (the lines). On the other hand, diagrams 904 and 906 show the biquadratic 12-parameter model for "Stefan" computed via the LMA. As can be observed, the perspective and zoom effects in this scene require a higher-order model than the linear affine 6-parameter model.

[0112] Levenberg contributed to replacing this equation with a “damped” variant that uses a non-negative parameter λ to control the rate of reduction of the error function S: (JTJ+λI)δ=x'−f(β).

[0113] A smaller λ value brings the LMA closer to the GNA, while a larger λ value brings it closer to the gradient descent method. If either the length of the computed step δ or the reduction of S from the final parameter vector β + δ falls below specified limits, the LMA iteration stops, with the final β being output as the solution.

[0114] Marquardt improved the final LMA equation to avoid slow convergence in the direction of a small gradient. He replaced the identity matrix I with the diagonal matrix, which consists of the diagonal elements of the matrix J. T J, which leads to the final equation of the Levenberg-Marquardt algorithm: (JTJ+λdiag(TJ))δ=x'−f(β).

[0115] Marquardt recommended an initial value of λ in a general case. However, for LMAs modeling global motion, some implementations may instead use a method where the initial parameter λ is set to the square root of the sum of the squared errors of the initial model parameters.

[0116] The LMA can also be used to calculate linear parameters. However, empirical data show that the direct estimation of the least squares fit yields practically the same SAD error compared to the LMA, but with several key advantages: (1) the calculation of the linear 4- and 6-parameter models can be performed in a single pass, and (2) the direct calculation of least squares for the linear models provides a higher correlation of the parameters from frame to frame, thus making coding less expensive, while the LMA yields more tuned coefficients. However, the calculation of the nonlinear models is often best performed using the LMA method. Fig. Figures 9A-9D show an example of the LMA estimation of the 6- and 12-parameter models of global motion.

[0117] An overview of the coding of the parameters of the global movement model

[0118] The global motion parameters are typically calculated as floating-point numbers, and as such, they are not easily transferable to the decoder. The MPEG-4 standard proposes encoding the global motion parameters using so-called "reference points" or "control grid points." The motion vectors of the reference points are transmitted as the global motion parameters. The motion vectors of the reference points are easier to encode, with the parameters being reconstructed from the decoded vectors in the decoder. Because the vectors are quantized (e.g., to half-pixel accuracy in MPEG-4), the method is lossy. However, the reconstructed coefficients typically produce a very similar global motion field, with the loss of quality being tolerable.

[0119] MPEG-4 uses up to four reference points, which can support translational, affine, and perspective models. The number of reference points that must be sent to the decoder depends on the complexity of the motion model. If an eight-parameter model (such as a perspective model) is used in MPEG-4, four points are required to determine the unknown parameters by solving the linear system. For a four-parameter model and the six-parameter models, the number of required reference points is reduced to two and three, respectively.

[0120] The reference points are located at the corners of the bounding box. The bounding box can be the entire frame or a smaller rectangle within the frame. The locations of these parameters are defined as follows: z0=(x0,y0), z1=(x1,y1)=(x0+W,y0), z2=(x2,y2)=(x0,y0+H), z3=(x3,y3)=(x0+W,y0+H). Where (x0, y0) is the coordinate of the top left corner, W is the width and H is the height of the frame or bounding box.

[0121] The estimated global motion model can be applied to the reference points, resulting in the following motion vectors: MXi=xi'−xi, MYi=yi'−yi.

[0122] Where i = 0, ... , 3 and (xi',yi') can be calculated using the equation of the global motion model. When the decoder receives the vectors (MX i , MY i), it can reconstruct the global motion parameters. If a 4-parameter model is used, the decoder receives two vectors (MX0, MY0) and (MX3, MY3), corresponding to the reference points z0 and z3, respectively. For the case where the global motion is defined over the entire frame, the reference points are z0 = (0, 0) and z3 = (W, H), where W and H are the width and height of the frame. To reconstruct the parameters a0, ..., a3 of a translational model of global motion, the following two systems are solved: (a0a1)=(x01x31)−1(x0+MX0x3+MX3)=(01W1)−1(x0'x3')=(−1W1W10)(x0'x3'), (a2a3)=(y01y31)−1(y0+MY0y3+MY3)=(01H1)−1(y0'y3')=(−1H1H10)(y0'y3').

[0123] When using a 6-parameter model, the decoder can receive three vectors (MX0, MY0), (MX1, MY1), and (MX2, MY2), corresponding to the reference points z0, z1, and z2, respectively. The reference points are z0 = (0, 0), z1 = (W, 0), and z2 = (0, H). Reconstructing the parameters a0, ..., a5 of an affine model of global motion can be accomplished by solving the following two systems: (a0a1a2)=(001W010H1)−1(x0+MX0x1+MX1x2+MX2)=(−1W1W0−1H01H100)(x0'x1'x2'), (a3a4a5)=(001W010H1)−1(y0+MY0Y1+MY1Y2+MY2)=(−1W1W0−1H01H100)(y0'y1'y2').

[0124] Similarly, the other parameter models can be reconstructed by solving the linear system determined by the motion vectors of the reference points.

[0125] For efficient representation, MPEG-4 allows the motion vectors to be transmitted differentially. A 4-parameter model is assumed. Then, the motion vector (MX0, MY0) for grid point z0 is encoded as is, while the motion vector for grid point z3 is differentially encoded using (MX3 - MX0, MY3 - MY0)

[21] . The differentials are encoded using the exponential Golomb code.

[0126] An exponential Golomb code, or Exp-Golomb code for short, is a type of universal code used to encode any non-negative integer. The following rule can be used to encode a non-negative integer n with the Exp-Golomb code: 1) represent n + 1 and write this number of zero bits preceding the preceding bit sequence; 2) because the motion vector differences are not strictly non-negative integers, they are converted to non-negative binary digits in the MPEG-4 standard; and 3) count the number of digits in the binary representation of n + 1, subtracting one representation. The value of the motion vector difference m is denoted as v. m as follows: Table 2 below illustrates the first 11 Exp-Golomb codes for the integers (m) and the nonnegative integers (vm). Table 2 m v m Exp-Golomb code Bit length 0 0 1 1 1 1 010 3 -1 2 011 3 2 3 00100 5 -2 4 00101 5 3 5 00110 5 -3 6 00111 5 4 7 0001000 7 -4 8 0001001 7 5 9 0001010 7 -5 10 0001011 7 ... ... ... ...

[0127] Table 2 above shows the first few Exp-Golomb codes. If a motion vector (a difference to be coded) is -1 in MPEG-4, the encoder can represent it with a 3-bit codeword "011." This representation can be efficient because the probability of the differences is similar to the probability distribution represented by Exp-Golomb coding. The preprocessing

[0128] The preprocessing component of the proposed algorithm may include: a) downsampling the input frame from pixel resolution to block resolution and b) scene change detection, which decides whether the current frame is part of the new scene or not.

[0129] For example, downsampling can be performed on the input frames to improve the processing speed of segmentation and also to reduce the noise level in the region segmentation process. To obtain a higher-quality downsampled frame, downsampling can be performed by averaging the pixel values ​​at a block level. Downsampling converts an input YUV420 frame into a block-accurate YUV444 frame, where the luminance signal can be downsampled by 4 (i.e., 4 × 4 block accuracy), while the chrominance signal can be downsampled by 2 (i.e., 2 × 2 block accuracy). For a high-resolution 1080p sequence, the luminance can be subsampled from a 1920 × 1080 resolution to a 480 × 270 resolution, while the chrominance can be subsampled from a 960 × 540 resolution to a 480 × 270 resolution.

[0130] Detecting a scene change may be necessary in some implementations to properly reset the algorithm parameters. Generally, any generic scene change detector can be used in this step; however, a more advanced scene change detector (SCD) is used, which reliably and efficiently performs scene change detection as the preprocessing stage. During scene change detection, each input frame at the original pixel resolution is fed into the SCD algorithm, which calculates a scene change flag (scf). If the flag is set, the current frame is part of the new scene, and the buffer of previous GMM parameters is initialized. The details of the SCD procedure are omitted here. Motion estimation

[0131] The proposed region-based motion modeling approach uses a block-based motion vector field as a basis from which each of the models can be computed. Although, in general, any block-based motion estimator could be used to compute a motion vector field, such an estimator can be based on hardware-accelerated GPU graphics VME routines. The hardware-accelerated motion estimation approach enables significantly faster processing.

[0132] VME is a motion estimation routine that estimates the motion vector field to the accuracies of one or more blocks, relying on a graphics GPU. In some implementations described here, VME can be used to obtain block-blurred motion vector fields at 16 × 16 and 8 × 8 levels. The VME routine can use a full-search motion estimation method with a given search range. Unfortunately, the maximum VME search range is often limited and cannot handle very fast-moving regions and / or larger distances between the current and reference frames. Therefore, a multi-stage VME-based method for block-based motion estimation can be used to support larger search ranges. In particular, a 3-stage VME method can be used, which is described next.

[0133] A multi-stage VME can use subsampled frames in the previous stage to estimate the initial motion vectors for the current frame, e.g., the starting position of the VME motion search for each block. The subsampling factor can depend on both the number of stages and the frame resolution. In the first stage, the current and reference frames of the low-resolution sequences are subsampled by 8 in each direction. If a frame width is less than 600, a frame height is less than 300, and the product of the frame width and frame height is less than 180,000, then the sequence is classified as a low-resolution sequence. On the other hand, for other (larger) resolutions, the first stage can use a subsampling factor of 16 in each direction. The subsampling of the 2nd stage is by 4 in each direction for all resolutions. Finally, a 3rd stage VME canThe (final) stage uses the full-resolution frames, generating motion vector fields with 16 × 16 and 8 × 8 block accuracies. Such an example of a 3-stage VME algorithm might include the following steps: 1. If H < 300 and W < 600 and W × H < 180,000, then set ld = 1; otherwise set ld = 0. 2. Given the current frame F and the reference frame F ref the subsampled luminance frames SF' and SFref' as the input to the VME of stage 1. Subsampling is performed by 8 in each direction if ld = 1 and by 16 if ld = 0. 3. The 1st stage VME routine using SF' and SFref' within the search area set to 64 × 32. 4. Filter the output of the 1st stage motion vector field and resize it to generate the input to the 2nd stage as follows: a. Remove the isolated noise-like motion vectors from the 16 × 16 and 8 × 8 output motion vector fields. For a given vector (mx (j,i) , my (j,i) ) at the subsampled position (j, i), where w and h are the width and (respectively) the height of the subsampled motion vector field, perform: i. If j > 0, d L = abs(mx (j,i) - mx (j-i) ) + abs(my (j,i) - my (j-1,i) ), otherwise d L = ∞. ii. If i > 0, d T = abs(mx (j,i) - mx (j,i-1) ) + abs(my (j,i) - my (j,i-1) ), otherwise d T = ∞. iii. If j < w - 1, d R = abs(mx (j,i) - mx (j+1,i) ) + abs(my (j,i) - (my (j+1,i) ), otherwise d R = ∞. iv. If i < h - 1, d B = abs(mx (j,i) -mx (j,i+1) ) + abs(my (j,i) - my (j,i+1)), otherwise d B = ∞. v. d = min (d L , d T , d R , d B ) set. vi. If d > T (in a software implementation T = 16), then do the following: 1. If d = d L is, then (mx (j,i) , my (j,i) ) by (mx (j-1,i) , my (j-1,i) ) substitute. 2. Otherwise, if d = d T is, (mx (j,i) , my (j,i) ) by (mx (j,i-1 ), my (j,i-1) ) substitute. 3. Otherwise, if d = d R is, (mx (j,i) , my (j,i) ) by (mx (j+1,i) , my (j+1,i) ) substitute. 4. Otherwise, if d = d B is, (mx (j,i) , my (j,i) ) by (mx (j,i+1) , my (j,i+1) ) substitute. b. Merge the 16 × 16 and 8 × 8 output motion vectors into a merged 8 × 8 motion vector field: if the SAD of a 16 × 16 block is up to 2% higher than the sum of four co-located 8 × 8 blocks, then use a repeat motion vector of a 16 × 16 block in the merged field; otherwise, copy four co-located motion vectors from an 8 × 8 motion vector field. It is noted here that for low-resolution sequences, the resulting 8 × 8 block size at the subsampled resolution corresponds to a 64 × 64 block size at the original full resolution, while at the other (higher) resolutions it corresponds to a 128 × 128 block at the original resolution. c. Upsample (resize) the merged motion vector field in each dimension by 2 for the low resolution and by 4 for the other resolutions. For the other resolutions, also rescale the motion vectors in the merged motion vector field by 2 (i.e., multiply each coordinate by 2). 5. Use the resulting fused motion vector field as the input motion vectors for the 2nd stage VME. 6. Given the current frame F and the reference frame F ref the subsampled luminance frames SF and SF ref as the input to the VME of the second stage. Subsampling is performed by 4 in each direction. 7. The 2nd stage VME routine using SF and SF ref with the search area set to 64 × 32. 8. Filter and resize the output of the 2nd stage motion vector field to generate the input to the 3rd stage as follows: a. Remove the isolated noise-like motion vectors from the 16 × 16 and 8 × 8 output motion vector fields. For a given vector (mx (j,i) , my (j,i) ) at the subsampled position (j, i), where w and h are the width and (respectively) the height of the subsampled motion vector field, using the same algorithm as in 4.a. b. Merge the 16 × 16 and 8 × 8 output motion vectors into a merged 8 × 8 motion vector field as in 4.b. c. Calculate the fused median motion vector field by applying a 5 × 5 median filter to the fused motion vector field. d. The block-based SAD for both the merged MVF and the merged median MMF using the current luminance frame SF and the reference luminance frame SF ref calculate. e. Generate the final fused motion vector field by selecting either a vector from the fused MFV or from the fused median MMF depending on which of the two has a smaller block SAD. f. Upsample (resize) the final merged motion vector field by 4 in each dimension and rescale the motion vectors in the merged motion vector field by 2 (i.e., multiply each coordinate by 2). 9. Use the resulting fused motion vector field as the input motion vectors for the 3rd stage VME. 10. The 3rd level VME routine using SF and SF ref with the search area set to 64 × 32.

[0134] The output of the third-stage VME algorithm contains 16 × 16 and 8 × 8 block-based motion vector fields (where the block size is, for example, relative to the full frame resolution). Next, we describe how these vectors are filtered so that the noisy correspondences are removed during the motion estimation stage and replaced with more accurate motion vectors relative to the actual motion of the underlying visual objects in the scene. Motion vector filtering

[0135] The motion estimation search often produces false motion vector matches, called outlier motion vectors. The outlier motion vectors are generated as a result of random matches during the motion estimation phase and do not correspond to the actual motion. The outliers occur either in flat areas or in blocks containing edges / texture patterns that are susceptible to the aperture problem. The aperture problem refers to the fact that the motion of a visual object resembling a repeated 1-dimensional pattern (e.g., a bar or an edge) cannot be uniquely determined when viewed through a small aperture (e.g., a block-sized window in block-based motion estimation). This is exactly what happens during the block-based motion estimation phase.

[0136] Even if they have a small prediction error, incorrect motion vectors can have a significant negative impact on the global motion estimation phase. If multiple incorrect vectors are calculated to calculate the global motion, the equation would be incorrect, and consequently, the global motion error would be large.

[0137] To address this problem, some implementations described here are designed and implemented with a motion filtering technique that reduces motion vector outliers and improves the motion vector field used for global motion estimation, as described in more detail below.

[0138] Fig.10 is an illustrative block diagram of an exemplary local motion field noise reduction filter arranged in accordance with at least some implementations of the present disclosure. In various implementations of the local motion field noise reduction filter 106, an ld signal may be used to switch between filtering the block-based 8×8 motion vectors at an isolated motion vector refiner 1002 and filtering the block-based 16×16 motion vectors at an isolated motion vector refiner 1004. The signal value may be preset to ld = 1 if the sequence is a low-resolution sequence, or ld = 0 otherwise. For the low-resolution sequences, the input may be a block-based 8×8 motion vector field, which is then filtered in two steps: (1) by removing isolated motion vectors (e.g.,Motion vectors that are very different from their four direct neighbors) in an isolated motion vector refiner 1002, and (2) by merging some four 8 × 8 vectors into a single co-located 16 × 16 vector at the 16 × 16 and 8 × 8 motion vector merger 1006. For other resolutions, filtering is performed in one step simply by removing the isolated motion vectors. Removing the isolated motion vectors can be performed by comparing, coordinate-wise, the sum of absolute differences (SAD) between a motion vector and its upper, left, right, and lower direct neighbors. If all four differences are greater than a similarity threshold (which may be set to 16, for example, in some implementations), then the vector is replaced by the smallest sum of absolute differences (SAD) of any corresponding direct neighbor.

[0139] In the case of low-resolution sequences, the merging step can be performed by calculating the sum of the SADs of the four 8 × 8 vectors in the 8 × 8 field and comparing it with the SAD of the co-located 16 × 16 motion vector. If the SAD of the 16 × 16 vector is within a small percentage (e.g., 1%) of the error of the sum of the four co-located 8 × 8 vectors, then the four 8 × 8 vectors can be merged and replaced by the single co-located 16 × 6 vector. An example of such an algorithm might include the following steps: 1. If H < 300 and W < 600 and W × H < 180,000, then set ld = 1; otherwise set ld = 0. 2. If ld = 1, then do the following: a. Remove the isolated noise-like motion vectors from the 16 × 16 and 8 × 8 output motion vector fields. For a given vector (mx (j,i) , my (j,i)) at the subsampled position (j, i), where w and h are the width and (respectively) the height of the subsampled motion vector field, perform: i. If j > 0, d L = abs(mx (j,i) - mx (j-i) ) + abs(my (j,i) - my (j-1,i) ), otherwise d L = ∞. ii. If i > 0, d T = abs(mx (j,i) - mx (j,i-1) ) + abs(my (j,i) - my (j,i-1) ), otherwise d T = ∞. iii. If j < w - 1, d R = abs(mx (j,i) - mx (j+1,i) ) + abs(my (j,i) - (my (j+1,i) ), otherwise d R = ∞. iv. If i < h - 1, d B = abs(mx (j,i) -mx (j,i+1) ) + abs(my (j,i) - my (j,i+1) ), otherwise d B = ∞. v. d = min (d L , d T , d R , d B ) set. vi. If d > T (in this implementation T = 16), then do the following: 1. If d = d L is, then (mx (j,i) , my (j,i) ) by (mx (j-1,i) , my (j-1,i) ) substitute. 2. Otherwise, if d = d T is, (mx (j,i) , my (j,i) ) by (mx (j,i-1) , my (j,i-1) ) substitute. 3. Otherwise, if d = d R is, (mx (j,i) , my (j,i) ) by (mx (j+1,i) , my (j+1,i) ) substitute. 4. Otherwise, if d = d B is, (mx (j,i) , my (j,i) ) by (mx (j,i+1) , my( j,i+1) ) substitute. b. Merge the 16 × 16 and 8 × 8 output motion vectors into a merged 8 × 8 motion vector field: if the SAD of a 16 × 16 block is up to 2% higher than the sum of four co-located 8 × 8 blocks, then use a repeat motion vector of a 16 × 16 block in the merged field; otherwise, copy four co-located motion vectors from an 8 × 8 motion vector field. It is noted here that for low-resolution sequences, the resulting 8 × 8 block size at the subsampled resolution corresponds to a 64 × 64 block size at the original full resolution, while at the other (higher) resolutions it corresponds to a 128 × 128 block at the original resolution. c. Output the fused 8 × 8 motion vector field to be used to calculate the parameters of the global motion model. 3. Otherwise, if ld = 0, then do the following: a. Remove the isolated noise-like motion vectors from the 16 × 16 motion vector field. For a given vector (mx (j,i) , my (j,i) ) at the subsampled position (j, i), where w and h are the width and (respectively) the height of the subsampled motion vector field, perform the following: i. If j > 0, d L = abs(mx (j,i) - mx (j-i) ) + abs(my (j,i) - my (j-1,i) ), otherwise d L = ∞. ii. If i > 0, d T = abs(mx (j, i) - mx (j,i-1) ) + abs(my (j,i) - my (j,i-1) ), otherwise d T = ∞. iii. If j < w - 1, d R = abs(mx (j,i) - mx (j+1,i) ) + abs(my (j,i) - (my (j+1,i) ), otherwise d R = ∞. iv. If i < h - 1, d B = abs(mx (j,i) -mx (j,i+1) ) + abs(my (j,i) - my (j,i+1) ), otherwise d B= ∞. v. d = min (d L , d T , d R , d B ) set. vi. If d > T (in this implementation T = 16), then do the following: 1. If d = d L is, then (mx (j,i) , my (j,i) ) by (mx (j-1,i) , my (j-1,i) ) substitute. 2. Otherwise, if d = d T is, (mx (j,i) , my (j,i) ) by (mx (j,i-1) , my (j,i-1) ) substitute. 3. Otherwise, if d = d R is, (mx (j,i) , my (j,i) ) by (mx (j+1,i) , my (j+1,i) ) substitute. 4. Otherwise, if d = d B is, (mx (j,i) , my (j,i) ) by (mx (j,i+1) , my (j,i+1) ) substitute. b. Output the filtered 16 × 16 motion vector field to be used to calculate the parameters of the global motion model. Segmentation into movement areas

[0140] Fig.11 is an illustrative block diagram of region segmenter 107 arranged in accordance with at least some implementations of the present disclosure. In various implementations, the exemplary region segmenter 107 may include a global motion model calculator 1102 for segmentation, a parameter buffer 1104, a background motion region segmenter 1106, a foreground motion region segmenter 1108, and / or a morphologically based region post-processor 1110.

[0141] In operation, the region segmenter 107 may operate to divide a frame into a number of motion regions. The number of motion regions may typically be limited, for example, to one to three regions per frame. An additional region may be enabled (thus allowing a maximum of four regions in the frame) indicating regions of stationary, inactive content, such as black bars and regions generated by letterboxing, pillarboxing, circular or clipped circular fisheye cameras, the like, and / or combinations thereof.

[0142] As illustrated, the region segmenter 107 may operate such that the region segmentation may include the following stages: 1) calculating the global motion model for segmentation via the global motion model for segmentation calculator 1102, 2) segmenting the frame into foreground / background motion regions (background motion region segmentation) via the background motion region segmenter 1106, 3) segmenting the remaining (foreground) motion regions (if applicable) via the foreground motion region segmenter 1108, and / or 4) post-processing the segmented motion regions using morphological operations via the morphologically based region post-processor 1110.

[0143] In the illustrated example, region segmenter 107 may calculate multiple (e.g., 1-3) motion regions in the current frame. The first step may be to calculate an affine global motion model for segmentation (referred to as a GMM) via global motion model for segmentation calculator 1102. Either the currently calculated model or one of the previous models (e.g., the two previous models) may be selected as the GMM via global motion model for segmentation calculator 1102 via parameter buffer 1104.

[0144] Then, using this GMM model, the background motion region segmenter 1106 can segment the current frame into the background motion region and the other regions. Either purely motion-based segmentation is used, or, if a strong dominant color is present, color-assisted motion segmentation is used. The binary segmentation mask (BGMP) can be generated based on this segmentation using the background motion region segmenter 1106.

[0145] In the next step, potential additional (e.g., foreground motion) regions can be detected and segmented via the foreground motion region segmenter 1108. The foreground motion region segmentation process can, for example, use a dominant motion and peak analyzer to determine whether 0, 1, or 2 additional motion-based (e.g., foreground) regions are present in the frame, generating a mask of the unprocessed regions.

[0146] After all regions have been segmented, the mask of the raw regions can be post-processed via the morphologically based region post-processor 1110 to reduce the segmentation noise and make the mask of the raw regions more stable. Calculating the global motion model for segmentation

[0147] In some implementations, the first step in segmenting the frame into motion regions is to determine the global motion model used for background motion region segmentation. This model, referred to as the global motion model for segmentation, is derived using an initial 6-parameter affine global motion model through random sampling. From this initial model, a 6-parameter affine global motion model for segmentation is finally computed, which is used to derive the foreground / background segmentation mask. Initially, random sampling is used to filter out outlier motion vectors, such as motion vectors inconsistent with the global motion, from the motion vector field. Random sampling provides statistics from which a robust global motion model can be derived.

[0148] An affine model of global motion has 6 unknown parameters that need to be estimated, therefore any 3 selected motion vectors (MX0, MY0), (MX1, MY1) and (MX2, MY2) at the positions (x0, y0), (x1, y1) and (x2, y2) from the motion vector field can be used to solve the system of equations for the parameters (assuming they form the independent system) as follows: a0=x0'(y1−y2)−x1'(y0−y2)+x2'(y0−y1)x0y1−x0y2−x1y0+x1y2+x2y0−x2y1, a1=−x0'(x1−x2)+x1'(x0−x2)+x2'(x0−x1)x0y1−x0y2−x1y0+x1y2+x2y0−x2y1, a2=x0'(x1y1−x2y2)−x1'(x0y2−x2y0)+x2'(x0y1−x1y0)x0y1−x0y2−x1y0+x1y2+x2y0−x2y1, a3=−y0'(y1−y2)−y1'(y0−y2)+y2'(y0−y1)x0y1−x0y2−x1y0+x1y2+x2y0−x2y1, a4=−y0'(x1−x2)−y1'(x0−x2)−y2'(x0−x1)x0y1−x0y2−x1y0+x1y2+x2y0−x2y1, a5=y0'(x1y2−x2y1)−y1'(x0y2−x2y0)+y2'(x0y1−x1y0)x0y1−x0y2−x1y0+x1y2+x2y0−x2y1. Where xi'=xi+MXi and yi'=yi+MYi for i = {0, 1, 2}.

[0149] The number of motion vectors from which random samples are taken depends on the video resolution. For standard and high resolution, the block size is set to 8. For low-resolution sequences, the block size is set to 16. If a frame width is less than 600, a frame height is less than 300, and the product of the frame width and height is less than 180,000, then the sequence can be classified as a low-resolution sequence, with the motion vector field from which the samples are taken being an 8 × 8 motion vector field. Otherwise, a 16 × 16 motion vector field is used as a pool from which the motion vector random samples are drawn.

[0150] In some implementations, the random sampling approach uses the above equations to solve for the parameters a0, ..., a5 by randomly selecting three motion vectors. The parameters calculated from the selected vectors, which form an independent system, are called the local parameters. After a large sample of the local parameters has been collected, the statistical properties of the collected data can be used to estimate a stable set of global motion parameters. The final random sample based on the estimated parameters is referred to herein as the initial affine model of global motion. The algorithm for calculating the initial affine model of global motion is described next. 1. Set N to a total number of motion vectors in the input motion vector field. 2. Initialize 6 histograms H0, ..., H5 of a selected size to 0. The histogram size, which is given here by S H determines how many classes are supported by each of the 6 histograms. More classes mean more accuracy in estimating a parameter within the parameter range. However, too many classes would produce a flat-looking histogram, making it more difficult to determine the correct peak. In one implementation, the following value was S H = 128. 3. Select a range of values ​​for each parameter. The following ranges are used: a 0,4 ∈ [0.95, 1.05), a 1,3 ∈ [-0, 1, 0,1), a2 ∈ [-64, 64), a5 ∈ [-48, 48). 4. For each parameter (e.g., 128 in one implementation), assign equidistant subranges within the selected range to a class in the histogram. For example, for the parameter a0, a range [0.95, 1.05) is divided into 128 classes (subranges): [0.95, 0.95078125), [0.95078125, 0.9515625), ..., [1.04921875, 1.05]. 5. For i = 0 to N, do the following: a. Select 3 random positions and the corresponding vectors in the motion vector field. b. Calculate the local affine 6-parameter model from 3 points / vectors. c. For each parameter, determine the histogram class into whose subrange the parameter value falls. d. If a parameter value falls within a valid subrange, increment the histogram count at the index of that subrange. 6. Detect the 6 highest peaks in each of the 6 histograms and select the corresponding sub-ranges. 7. Set the candidate initial affine model of global motion to parameters corresponding to the mean of its peak subrange. For example, if a peak is located at the 2nd position in H0, then the peak subrange for the parameter a0 is [0.95078125, 0.9515625], and the parameter a0 = (0.95078125 + 0.9515625) / 2 = 0.951171875. 8. Add the previous two initial affine motion models to the candidate set (obviously, nothing is added at the beginning of the scene, for example, with a model being added after the first frame). 9. If there is only one candidate, select it as the initial affine motion model for the current frame. On the other hand, if the number of candidates is greater than one, then: a. Calculate the SAD measure of all candidates as follows. For each candidate: i. Generate a reconstructed frame to pixel accuracy using the parameters of the candidate global motion model. ii. Calculate the SAD between the pixels of the reconstructed frame and the pixels of the current frame. b. Select the candidate with the smallest SAD as the initial affine model of the motion for the current frame.

[0151] Fig.12 is an illustrative diagram of an example histogram distribution 1200 of the locally computed parameters of the affine model of global motion, arranged in accordance with at least some implementations of the present disclosure. In various implementations, the histogram distribution 1200 illustrates an example of the "City" sequence where the current frame is frame 1 and the reference frame is frame 0. For example, diagram (a) shows B. the histogram H0, where the peak is located at position 64, which corresponds to the subrange [1.0, 1.00078125), so that the parameter a0 is set to the center point 1.000390625, the diagram (b) shows the histogram H1, where the peak is located at position 64, which corresponds to the subrange [0.0 0.0015625), so that the parameter a1 is set to the center point 0.00078125, the diagram (c) shows the histogram H2, where the peak is located at position 66,which corresponds to the subrange [2.0 3.0) so that the parameter a2 is set to the midpoint 2.5, the graph (d) shows the histogram H3 where the peak is located at position 64, which corresponds to the subrange [0.0 0.0015625) so that the parameter a3 is set to the midpoint 0.00078125, the graph (e) shows the histogram H4 where the peak is located at position 64, which corresponds to the subrange [1.0 1.00078125) so that the parameter a4 is set to the midpoint 1.000390625, and the graph (f) shows the histogram H5 where the peak is located at position 64, which corresponds to the subrange [0.0 0.75) so that the parameter a5 is set to the midpoint 0.375.

[0152] Fig.13 is an illustrative video sequence 1300 of an example of the difference between global and local block-based vectors, arranged in accordance with at least some implementations of the present disclosure. In various implementations, the video sequence 1500 shows an example of the "City" sequence, which uses an initial affine model of global motion from the previous figure, the Fig.12, where: frame (a) illustrates the current frame with an 8 × 8 block-based motion vector field shown with arrows, frame (b) illustrates the current frame with an 8 × 8 global motion vector field (also shown with arrows) derived from the computed initial affine model of the global motion, and frame (c) illustrates the difference heatmap (e.g., the darker shading indicates smaller differences between the global and local block-based vectors, while the lighter shading indicates larger differences).

[0153] Fig.14 is an illustrative diagram of an example histogram distribution 1400 of locally computed parameters of an affine model of global motion using a random sampling approach, arranged in accordance with at least some implementations of the present disclosure. In various implementations, the histogram distribution 1400 illustrates an example of the histograms of the locally computed parameters of an affine model of global motion using the random sampling approach for the "Stefan" sequence, where the current and reference frames are one frame apart. In the illustrated example, diagram (a) shows a histogram H0 where the peak is located at position 84, corresponding to the subrange [1.015625, 1.01640625), such that the parameter a0 is set to the midpoint 1.016015625. Diagram (b) shows a histogram H1 where the peak is located at position 64.which corresponds to the subrange [0.0, 0.0015625), so that the parameter a1 is set to the center point 0.00078125, the diagram (c) shows a histogram H2 where the peak is located at position 35, which corresponds to the subrange [-29.0, -28.0), so that the parameter a2 is set to the center point -28.5, the diagram (d) shows a histogram H3 where the peak is located at position 64, which corresponds to the subrange [0.0, 0.0015625), so that the parameter a3 is set to the center point 0.00078125, the diagram (e) shows a histogram H4 where the peak is located at position 79, which corresponds to the subrange [1.01171875, 1.0125), so that the parameter a4 is set to the center point 1.012109375, and the diagram (f) shows a histogram H5 where the peak is located at position 58, corresponding to the subrange [-4.5, -3.75), so that the parameter a5 is set to the midpoint -4.125. ,

[0154] Fig.15 is an illustrative video sequence 1500 of an example of an initial affine model of global motion arranged in accordance with at least some implementations of the present disclosure. In various implementations, the video sequence 1500 illustrates an example of the "Stefan" sequence, which includes an initial affine model of global motion from the preceding figure, Fig.14 shows. In the illustrated example, frame (a) illustrates the current frame with an 8 × 8 block-based motion vector field (shown with arrows), frame (b) illustrates the current frame with an 8 × 8 global motion vector field (also shown with arrows) derived from the computed initial affine model of the global motion, and frame (c) illustrates the difference heatmap (e.g., where the darker shading indicates smaller differences between the global and local block-based vectors, while the lighter shading indicates larger differences).

[0155] In some examples, the initial affine global motion model can be used to estimate the affine global motion model for segmentation. The selection process can involve generating both a number of candidate selection masks and the corresponding affine motion models, and then selecting the model for segmentation that yields the smallest estimated error. The proposed selection procedure is described next.

[0156] To correctly estimate global motion from the given motion vector field, it can be crucial to both first select which motion vectors should be included and first select which should be excluded. This task is not straightforward due to imperfect motion vector fields and the difficulty of perfectly separating the blocks into moving visual objects. To address this problem, some implementations described here may use a candidate set (e.g., a set of 7, although a different number could be used) of possible block selection masks from which the affine model of global motion can be selected for segmentation.

[0157] The initial affine model of global motion can first be used to generate several candidate selection masks (e.g., 5, although a different number could be used). The selection masks obtained from the initial affine model are essentially binary masks that classify all frame blocks into two classes: 1) globally moving blocks and 2) locally moving blocks. Blocks whose global motion vector computed from the initial model is not similar to the corresponding block-based motion vector are marked as local, while the other blocks are marked as global. A binary mask can be used to indicate the motion vectors belonging to a global motion. An additional mask can be obtained by eroding the mask from the first level of the hierarchy.For each of the 5 masks, an affine model of the global motion can be computed using the described least squares fitting algorithm, so that only the motion vectors specified by the mask are used in the least squares computation process.

[0158] Then, a SAD-based error measure can be calculated for both these five models and the initial affine model of global motion. The affine model for segmentation can be set to the one with the smallest error measure. Furthermore, the current selection mask can be set to the mask associated with the one of the five hierarchical models with the smallest error measure.

[0159] Next, in an attempt to generate a more accurate affine model of global motion for segmentation, additional refinement steps (e.g., 2, although a different number could be used) can be performed on the currently selected mask. First, all blocks lying on a frame boundary and all blocks with very low texture activity can be removed. The error of this alternative selection mask is compared to the error of the currently selected mask, with the better mask set as the current mask. If the new refined mask is better, the affine model for segmentation can be set to the model computed from the refined mask.

[0160] Finally, a second refinement can be performed, where only the high-texture blocks (e.g., those containing multiple edges, corners, and complex patterns) are selected from the current selection mask, forming a final candidate mask. Again, the error of the second refined mask is compared with the error of the currently selected mask, and the better mask is set as the current selection mask. Finally, if the final candidate selection mask yields a smaller error, the affine model of global motion for segmentation can be set to the model computed from the final candidate mask.

[0161] In some implementations, the algorithm for computing the affine model of global motion for segmentation may include the following steps: 1. For i = 1 to 4, do the following: a. set t = 0. b. Set the minimum local object size m to a value that estimates the minimum number of global blocks that should be present in the mask. In this implementation, m = 0.1 × N (10% of the total number of blocks). c. The global motion vector field {GMX j , GMY j}, j = 0, ..., N - 1 using the initial affine model of the global motion. i.e. set t = t + i. e. For each position j in the motion vector field e j = abs (GMX j - MX j ) + abs (GMY j - MY j ) calculate. If e j ≤ t, then the mask M i Set [j] = 1, otherwise M i Set [j] = 0. f. If the sum of all values ​​of M i is less than m, then repeat and go back to step 1c. 2. Set the mask M0 to erode the mask M1. 3. For all 5 masks M0, ..., M4, compute affine 6-parameter models using the described least squares fitting algorithm such that only the motion vectors specified by the mask are used in the least squares computation process. 4. Calculate the SAD measure for all 5 least-squares-fitted affine models and set l ∈ {0, ..., 4} to the index of the model with the smallest value of the SAD measure. Set the current selection mask to mask M l set. 5. Compare the SAD measure of the selected l-th affine parameter model with the SAD measure of the initial affine model of the global motion vector and set the current best initial affine model of the global motion to the model that provides the smaller SAD measure. 6. If H < 300 and W < 600 and W × H < 180,000, then set T = 4, otherwise set T = 6. 7. Create an additional candidate mask M5 by refining the current selection mask as follows: a. Remove all blocks that lie on the frame boundary. b. Remove all blocks whose minimum Rs and Cs texture dimensions are smaller than the threshold T. 8. For mask M5, compute a 6-parameter affine model using the least squares fitting algorithm so that only the motion vectors specified by the mask are used in the least squares computation process. 9. Calculate the SAD measure of the computed affine model for M5. 10. Compare the SAD measures of the current model and the calculated model for mask M5 and set the current model and the current mask to those with the smallest SAD measure. 11. The threshold values ​​T RS to 1.5 × the average Rs value in the Rs / Cs 2-D arrangement and T CSto 1.5 × the average Cs value in the R S / C S -Set 2-D arrangement. 12. Create the final candidate selection mask M6 by refining the current selection mask as follows: a. All blocks whose Rs texture dimension is smaller than the threshold T RS and whose Cs texture measure is smaller than the threshold T CS is, remove. 13. For the mask M6, compute a 6-parameter affine model using the least squares fitting algorithm such that only the motion vectors specified by the mask are used in the least squares computation process. 14. Calculate the SAD measure of the computed affine model for M6. 15. Compare the SAD measures of the current model and the calculated model for mask M6 and set the current model and mask to those with the smallest SAD measure. Output the current model and mask as the final selection mask to be used in calculating the global motion model.

[0162] Fig.16 is an illustrative video sequence of an example of various computed candidate selection masks 1600 arranged in accordance with at least some implementations of the present disclosure. In various implementations, this example of the computed candidate selection masks 1600 for the “flower” sequence may include: the frame (a), the original YUV frame, the candidate selection mask (b), the eroded estimated mask M0 at the 1st level of the hierarchy (M0-based error measure = 987505), the candidate selection mask (c), the estimated mask M1 at the 1st level of the hierarchy (M1-based error measure = 970341), the candidate selection mask (d), the estimated mask M2 at the 2nd level of the hierarchy (M2-based error measure = 1002673), the candidate selection mask (e), the estimated mask M3 at the 3rd level of the hierarchy (M3-based error measure = 1373757), the candidate selection mask (f), the estimated mask M4 at the 4th level of the hierarchy (M5-based error measure = 1373757), the candidate selection mask (g), the estimated mask M5 at the 6th level of the hierarchy (M6-based error measure = 1373757), the candidate selection mask (h), the estimated mask M6 at the 7th level of the hierarchy (M7-based error measure = 1373757), the candidate selection mask (f), the estimated mask M7 at the 8th level of the hierarchy (M8-based error measure = 1373757), the candidate selection mask (g), the estimated mask M8 at the 9th level of the hierarchy (M9-based error measure = 1373757), the candidate selection mask (h), the estimated mask M9 at the 1st level of the hierarchy (M1-based error measure = 1373757), the candidate selection mask (i), the estimated mask M10 at the 1st level of the hierarchy (M1-based error measure = 1373757), the candidate selection mask (i), the estimated mask M11 at the 1st level of the hierarchy (M1-based error measure = 1373757), the candidate selection mask (i), the estimated mask M1Level of the hierarchy (M4-based error measure = 1417258), the candidate selection mask (g), the mask M5 refined using the best mask from the 5 hierarchical candidates (in this case, the best candidate is M1, e.g., the candidate with the smallest error measure), with flat blocks and frame boundary blocks removed from the mask (M5-based error measure = 972156), and the candidate selection mask (h), the mask M6 refined using the best mask from the previous 6 candidates (in this case, it is still M1, e.g.,), with only high-texture blocks (such as blocks containing multiple edges, corners, and complex patterns) selected (M6-based error measure = 981807). In this example, the final candidate selection mask is set to M1.

[0163] Fig.17 is an illustrative video sequence of an example of various computed selection masks 1700 arranged in accordance with at least some implementations of the present disclosure. In various implementations, this example of the computed selection masks 1700 for the “Stefan” sequence may include: the frame (a), the original YUV frame, the candidate selection mask (b), the encoded estimated mask M0 at the 1st level of the hierarchy (M0-based error measure = 1365848), the candidate selection mask (c), the estimated mask M1 at the 1st level of the hierarchy (M1-based error measure = 1363467), the candidate selection mask (d), the estimated mask M2 at the 2nd level of the hierarchy (M2-based error measure = 1318886), the candidate selection mask (e), the estimated mask M3 at the 3rd level of the hierarchy (M3-based error measure = 1327907), the candidate selection mask (f), the estimated mask M4 at the 4th level of the hierarchy (M5-based error measure = 1327907).Level of the hierarchy (M4-based error measure = 1349339), the candidate selection mask (g), the mask M5 refined using the best mask from the 5 hierarchical candidates (in this case, the best candidate is M2, e.g., the candidate with the smallest error measure), with flat blocks and frame boundary blocks removed from the mask (M5-based error measure = 1313352), and the candidate selection mask (h), the mask M6 refined using the best mask from the 6 hierarchical candidates (in this case, it is mask M5), with only high-texture blocks (such as blocks containing multiple edges, corners, and complex patterns) selected (M6-based error measure = 1348624). In this example, the final candidate selection mask is set to M5.

[0164] Fig.18 is an illustrative block diagram of an exemplary global motion model (GMM) calculator 1102 for segmentation, arranged in accordance with at least some implementations of the present disclosure. In various implementations, the global motion model calculator 1102 for segmentation may include a range histogram initializer 1802, a randomly sampled affine parameter histogram generator 1804, a histogram peak selector 1806, a subsampled SAD-based affine model parameter selector 1808, a BP parameter storage buffer 1810, a GMM MV selector 1812 for segmentation estimation, and an affine GMM least squares parameter calculator 1814.

[0165] As illustrated, Fig. 18 a detailed block diagram of the first block according to Fig.11, the global motion model calculator 1102 for segmentation. The global motion model calculator 1102 can calculate the affine global motion model for segmentation, which is later used to generate the background motion region segmentation mask. First, the region histograms can be initialized via the region histogram initializer 1802 and set to all 0 counts. The regions of the histogram are determined empirically, as previously described.

[0166] Next, for a given block-based motion vector field, three MVs can be randomly selected frm_sz times via the randomly sampled affine parameter histogram generator 1804. For each triplet of randomly selected MVs, a 6-parameter motion model can be computed using the least squares approach. Then, each of the 6 parameters can be mapped to a region in a corresponding histogram, and a histogram count can be incremented within that region.

[0167] After the histograms have been collected, the next step may be to analyze them and select the highest histogram peaks using the histogram peak selector 1806. For each selected peak, a parameter value can be calculated as the center of the given range. This results in an estimated 6-parameter affine model of the global motion, denoted in the block diagram as Parameter_Peaks.

[0168] Then, up to 2 previous models (previous_parameters) from the BP parameter storage buffer 1810 may be tested together with a computed model to select the model having the smallest subsampled SAD (SSAD), referred to herein as affine_parameter, via the subsampled SAD-based parameter selector 1808 of an affine model.

[0169] Next, the affine model affine_parameter can be used to generate the selection mask M, which selects, via the selector 1812 of the MVs for a GMM for segmentation estimation, which motion vectors from the mvs of the block-based motion vector field are to be used when estimating the final affine model of the global motion for segmentation (this block is called Fig. 19 described in more detail below).

[0170] Finally, a least squares fit together with the motion vectors mvs, the mask M and the current and reference frames F and F ref be used to estimate the affine GMM parameters for segmentation via the affine GMM least squares parameter calculator 1814.

[0171] Fig.19 is an illustrative block diagram of an exemplary motion vector selector 1812 for a GMM for segmentation estimation, arranged in accordance with at least some implementations of the present disclosure.In various implementations, the motion vector selector 1812 for a GMM for segmentation estimation may include a generator 1902 of initial selection masks of MVs for GMM estimation, an affine least squares GMM parameter calculator 1904, a binary 2 × 2 kernel erosion operator 1906, a downsampled SAD residuals calculator 1908, a minimum SAD residual-based candidate selector 1910, a medium to strong texture-based selection mask refiner 1912, an affine GMM parameter calculator 1914, a downsampled SAD residuals calculator 1916, a minimum SAD residual-based candidate selector 1918, a strong corner block-based selection mask refiner 1922, a calculator 1924 the parameter of an affine GMM, a calculator 1926 of the downsampled SAD residuals, and a candidate selector 1928 based on the minimum SAD residual. .

[0172] As illustrated, a detailed block diagram of the sixth block according to Fig. 18 the selector 1812 of the motion vectors for a GMM for segmentation estimation. Fig.Figure 19 illustrates an example of the steps that can be used to calculate the selection mask, which can be used to identify which motion vectors in the global motion model are to be used for the segmentation estimation phase. In the first step of this process, an initial affine model is used to generate, via the initial MV selection mask generator 1902 for GMM estimation, an estimated global motion field at the center of the block with the same size as the blocks of the block-based motion vector field. Such a global motion vector field is then discriminated (e.g., by calculating the sum of the absolute differences for each vector coordinate) with the block-based field. The differences are classified with four different adaptively chosen thresholds to generate four binary candidate selection masks M 1, ..., 4 to obtain.

[0173] By eroding the mask M1 with a 2 × 2 kernel via a binary 2 × 2 kernel erosion operator 1906, an additional mask M0 can be computed.

[0174] Next, via the affine least squares GMM parameter calculator 1904, five affine models are calculated using the least squares fitting method according to the five binary selection masks (e.g., a vector is used in the fitting process if the mask value is 1; otherwise, it is skipped). This generates the parameters defined by params 0, ..., 4 designated initial 5 candidate models.

[0175] For each of them, a downsampled SAD error can be calculated via the downsampled SAD residual calculator 1908 using the current and reference frames as input, with the mask M' corresponding to the minimum error being selected via the minimum SAD residual-based candidate selector 1910.

[0176] Two more candidate masks can then be generated. The first candidate mask, denoted M5, can be obtained via the medium-to-strong texture-based selection mask refiner 1912 by refining M' so that only blocks with medium and strong texture are retained, while the blocks with flat texture are removed. Additionally, the frame boundaries can also be removed because most of the uncovered area appears there, yielding unreliable vectors. Similarly, the corresponding affine model for M5 and the corresponding subsampled SAD error can be calculated via the affine GMM parameter calculator 1914. Then, either M5 or M' is selected (denoted by M'') via the downsampled SAD residual calculator 1916 and the minimum SAD residual-based candidate selector 1918.

[0177] The selected M'' can be input to the 2nd refinement step, which can generate the candidate selection mask M6 by selecting only the high-texture blocks (e.g., the blocks with high values ​​of both Rs and Cs) from the input mask M'' via the strong-corner-based block selection mask refiner 1922. Using the same steps as before, the corresponding affine model for M6 can be calculated via the affine GMM parameter calculator 1924 and the corresponding subsampled SAD error, where, according to the smallest error, either M6 or M'' can be selected as the final selection mask via the downsampled SAD residual calculator 1926 and the minimum SAD residual-based candidate selector 1928. Background motion area segmentation

[0178] To determine which motion regions are present in the given frame, the first step may be to segment the frame into two main motion-based regions: (1) the global motion region (also referred to as the background motion region) and (2) the local motion region (also referred to as the foreground motion regions). The global motion region may itself be a region, typically the largest region in the frame. It is worth noting that the background motion region may not be moving at all; for example, the background motion region could be stationary. In this case, the region "moves" with a global vector of (0, 0). On the other hand, the local motion region may consist of 0 or more foreground motion regions, depending on the content of the scene.Consequently, the first step may be to determine the background motion region, a process referred to as background motion region segmentation.

[0179] In some implementations, purely motion-based segmentation can be extended to assist with color for content that has a significantly low-textured dominant color within the motion region. This extension of the algorithm can improve the quality, temporal stability, and therefore also the codability of the motion region mask within the RMM. A given motion region can be analyzed for the presence of a single dominant color in the low-texture portion of the region. If the dominant color of the region is present in a significant percentage (e.g., above 85%), it can be used to improve the region boundary.The blocks of the given area that contain little to no of the particular dominant color can be removed, and the blocks that consist most of the colors that are similar to the dominant color can be added to that area. An example is shown in . Fig. 20 shown below.

[0180] The proposed operation of background motion region segmentation may include the following steps: 1. W × H shall denote a full-frame resolution. If H < 300 and W < 600 and W × H < 180,000, then set N = 8; otherwise, set N = 16. 2. Calculate the global motion probability map (GMP map) as follows: a. Generate the global motion vector field (GMVF) (W / N) × (H / N) by calculating the global motion vector for the center pixel of each N × N block in the current frame using the affine model of global motion for segmentation previously calculated. b. Be (gmx i , gmy i ) is the i-th motion vector in the GMVF and let (mx i , my i ) is the i-th motion vector in the block-based motion vector field (generated by the motion estimation step). Then, the i-th value of the GMP map (W / N) × (H / N) is set to: GMP(i) = abs (gmx i - mx i ) + abs (gmy i - my i ) . 3. A binarization threshold T m for the probability map of the global movement. 4. The threshold value T m to obtain a BGMP designated (binary) 2-level mask of GMP: a. For all blocks i in the GMP: i. If GMP(i) < T m , then BGMP(i) = 1. ii. Otherwise BGMP(i) = 0. 5. Calculate the probability map of the dominant color as follows: a. The color histogram H c initialize to 0. b. For all blocks i in the BGMP, if BGMP(i) = 1, then collect N / 4 YUV colors from the co-located blocks in the subsampled (W / 4) × (H / 4) YUV444 frame SF and add the counts to H c add. c. The dominant color (d Y , d U , d V ) to the highest peak in H C set. d. Subsample SF to the (W / N) × (H / N)-YUV444 frame SFF. e. For all i in the (W / N) × (H / N)-DCP map, DCP(i) = 8 × abs (d Y - SSF y(i) ) + abs (d Y - SSF y(i) ) + abs(d_y - SSF_y(i)). 6. If N = 8, the (flat) threshold T fof the low Rs / Cs value to 4; otherwise T f set to 6. 7. Set the counter c = 0 and the color similarity threshold T C = 16. 8. For all i in the (W / N) × (H / N)-BGMP map: a. If BGMP (i) = 1 and DCP(i) < T C and max(Rs(i), Cs(i) < T f , then set c = c + 1. 9. If c > 0.85 × (W / N) × (H / N), then reset the BGMP as follows: a. For all i in (W / N) × (H / N): i. If DCP(i) < T C and max(Rs(i), Cs(i) < T f , then set BGMP(i) = 1. ii. Otherwise, set BGMP(i) = 0. 10. Output the BGMP as the background motion area segmentation mask.

[0181] Fig.20 is an illustrative video sequence of an exemplary segmentation method 2000 using color to assist motion, arranged in accordance with at least some implementations of the present disclosure. In various implementations, the segmentation method 2000 shows an example of a high-resolution (1080p) "touchdown sequence" that uses color assistance in conjunction with motion. For example, frame (a) shows the original frame, frame (b) shows the final motion regions without using motion-only segmentation (SAD = 8442667, 2 regions), and frame (c) shows the final motion regions generated using color-assisted motion segmentation (SAD = 8133606, 2 regions), full frame SAD = 10914919.

[0182] Fig.21 is a block diagram of an exemplary background motion region segmenter 1106 arranged in accordance with at least some implementations of the present disclosure. In various implementations, the background motion region segmenter 1106 may include a global motion vector field calculator 2102, a motion vector difference and global motion probability map calculator 2104, a binarization threshold estimator 2106, a 2-level global motion probability classifier 2108, a masked color histogram calculator 2110, a dominant color histogram peak selector 2112, a color difference and scaler 2114, a frame subsampler 2116, and a masked low texture and dominant color analyzer 2118.

[0183] As illustrated, Fig.21 a detailed block diagram of the third block according to Fig. 11, the background motion region segmenter 1106. The background motion region segmenter 1106 may generate a binary segmentation mask indicating a (e.g., global) background motion region against the rest of the frame, which may include either static or locally moving blocks.

[0184] In operation, the previously computed affine GMM for segmentation can first be used to compute a global motion vector field, denoted by GMVF, via the global motion vector field calculator 2102. The field can be computed by applying the GMM's affine parameter equation to the center of the block position (e.g., using the same block size as in the block-based mvs field).

[0185] Then, the differences between the GMVF and the mvs can be calculated via the global motion probability map calculator 2104 and scaled in the range 0-255 to generate the so-called global motion probability map (GMP map). The GMP map can then be scaled using the calculated threshold T m (generated via the binarization threshold estimator 2106) is binarized into the binary mask denoted by BGMP' via the 2-level global motion probability classifier 2108.

[0186] Next, a masked color hologram can be calculated using the masked color histogram calculator 2110 using the BGMP to mask out the blocks that only move globally. Using the dominant color histogram peak selector 2112, the peaks can be determined from the histogram (col_hist) and a dominant color (dom_col) can be generated. Using the dominant color and the subsampled YUV444 frame SSF with the adjusted resolution from the frame subsampler 2116, the color differences can be calculated and scaled to the range 0-255 (the DCP map) using the color difference and scaler 2114.The DCP map can be used together with the RsCs(F) and BGMP' masks to calculate, via the masked low texture and dominant color analyzer 2118, the percentage of low texture blocks with dominant color in the background motion region, represented as a binary mask of the color-assisted BGMP. An analysis can be performed to determine if the percentage of these blocks is high enough (in some implementations, this percentage threshold may be set to 85% or more of the background motion blocks of the BGMP', for example), in which case the use_col control signal can be set to 1. Otherwise, the use_col control signal can be set to 0. If the use_col control signal is 1, then the color-assisted BGMP can be output as the final binary mask of the background motion region (BGMP); otherwise, the BGMP' mask can be output as the BGMP. The segmentation of the remaining (foreground) movement areas

[0187] Once the background motion region is segmented, the remaining region (e.g., the foreground motion region minus the detected stationary region with inactive content) can potentially be further segmented. An analysis can be performed to determine whether the foreground region should be split into two separate regions. If the foreground region should be further split, the following motion-based segmentation can be performed within the current foreground region: 1. Calculate the dominant motion vector (from the block-based MVF) in the non-background motion region of the segmentation mask BGMP (i.e., where the mask values ​​are 0): a. The motion vector histogram H m initialize to 0. b. For all blocks i in the BGMP, if BGMP(i) = 0, then collect the i-th motion vector from the MVF, the counter values ​​to the H m add. 2. Set the dominant motion vector (dmx, dmy) to the highest peak in the H m set. 3. Calculate the motion vector differences according to the probability map of the masked dominant motion (DMP map) as follows: a. For all blocks i in the BGMP, if BGMP(i) = 0, then DMP(i) = abs(dmx - mx i ) + abs(dmy - my i ). Here, (mx i , my i ) the i-th motion vector in the block-based MVF. 4. The binarization threshold T m1 Calculate the differences and apply it to all foreground blocks, thus splitting the foreground region into two foreground regions. The resulting binary mask is denoted by BDMP. 5. Analyze the solidity and size of the dominant motion area in the BDMP mask: if the largest 4-connected segment in the BDMP is at least 10% of the frame, add the foreground area defined by the BDMP to the area mask. Otherwise, skip to step 6. 6. Create the final area mask by adding the area mask of the area with inactive content, if any.

[0188] Fig.22 is a block diagram of an exemplary foreground motion region segmenter 1108 arranged in accordance with at least some implementations of the present disclosure. In various implementations, the foreground motion region segmenter 1108 may include a binary mask inverter 2202, a masked MV histogram peak selector 2204, a dominant MV histogram peak selector 2206, a masked MV difference and scaler 2208, a binarization threshold estimator 2210, a 2-level global motion probability classifier 2212, a 2-level segment solidity and size analyzer 2214, and a motion region mask generator 2216.

[0189] As illustrated, Fig. 22 a detailed block diagram of the fourth block according to Fig.11, the foreground motion region segmenter 1108. The foreground motion region segmenter 1108 can generate up to two remaining foreground regions (if necessary) and can generate the final region mask. In this example, the foreground motion region segmenter 1108 can operate as a two-stage cascade segmentation system. First, the BGMP mask defining the background motion region can be inverted so that the remaining non-background region is enabled via the binary mask inverter 2202 (e.g., the bit mask has a value of 1).

[0190] Then, using the inverted mask iBGMP, a masked histogram of motion vectors for the frame can be calculated using the block-based motion vectors mvs via the masked MV histogram peak selector 2204. The histogram, denoted as mv_hist, can be analyzed via the dominant MV histogram peak selector 2206, where the peaks can be selected to obtain the dominant motion vector within the foreground motion region. Next, the mvs of the motion vector field can be differentiated from the dominant motion vector via the masked MV difference and scaler 2208, where the results can be scaled in the ranges 0-255 into the dominant color probability map (DMP map).

[0191] A binarization threshold can be estimated for the resulting DMP map via the binarization threshold estimator 2210, which can then be binarized into the binary 2-level mask BDMP via the 2-level global motion probability classifier 2212. Next, the segment solidity and size analysis can be performed via the 2-level global motion probability classifier 2212 to determine whether the new foreground region defined by the BDMP is significant or not. If it is significant, the control signal add_reg is set to 1 (otherwise, it is set to 0). If add_reg is 0, then there are no foreground regions, and the resulting region mask is generated via the motion region mask generator 2216 with only 1-2 regions (as defined by the BGMP).Otherwise, the area mask is generated via the motion area mask generator 2216 with only 2-3 areas (as defined by the BGMP and BDMP masks). The morphologically based post-processing of the segmented areas

[0192] The generated region mask can be post-processed to produce more stable and less noisy region segments. Morphological opening and closing can be used to clean the motion region segmentation mask of the type of shot noise that is typically common to almost all segmentation methods. Additionally, a two-level small object removal process can also be used to remove noise related to small segmented blobs. Finally, the region boundary of a mask can be smoothed by a smoothing filter to remove all small spikes and similar noise artifacts at the region boundaries. An example of region mask post-processing is shown in Fig. 23 shown below.

[0193] The region mask post-processing algorithm consists of the following steps: 1. Apply morphological opening with a 2 × 2 kernel to the region mask. Morphological opening is defined as image erosion followed by image dilation. 2. Apply morphological reasoning with the 2 × 2 kernel to the region mask. Morphological reasoning is defined as image dilation followed by image erosion. 3. Set the resolution scaling factor rsf = max(1, (W / 350) * (H / 300)). 4. For all segments S in the area mask, perform the following: a. If the size of S is smaller than rsf × 4, then perform the removal of the 1st level of the small segments of the area mask as follows: i. Calculate the bounding box of S. ii. Extend the bounding box of S by 1 on each side. iii. Collect the histogram of the range indices within the extended bounding box. iv. Setting the counter value of the current range index of S in the histogram to 0. v. Replacing the range index of segment S by the range index whose count is the highest in the histogram. b. Otherwise, if the size of S is greater than or equal to rsf × 4 and less than rsf × 8, then perform the removal of the 2nd level of small segments in the area mask as follows: i. Calculate the bounding box of S. ii. Extend the bounding box of S by 1 on each side. iii. Collect the histogram of the range indices within the extended bounding box. iv. Setting the counter value of the current range index of S in the histogram to 0. v. Calculate the SAD of S using the motion models corresponding to all regions whose region index in the histogram is not zero. vi. Select the range index whose corresponding SAD is the smallest. vii. If the SAD corresponding to the selected new range index is within 5% of the existing SAD of S, then replacing the range index of S with the selected new range index. 5. Smooth the mask vertically and horizontally with an image (aba→mapaaa): if the previous and next values ​​are the same, then replace the current value with the previous / next value.

[0194] Fig.23 is an illustrative video sequence of an exemplary morphologically based post-processing method 2300 arranged in accordance with at least some implementations of the present disclosure. In various implementations in the morphologically based post-processing method 2300, frame (a) shows the original frame, frame (b) shows the region mask before post-processing, and frame (c) shows the region mask after post-processing, showing a cleaner-looking mask.

[0195] The examples of the final segmented region masks for several sequences with different resolutions are shown below in Fig. 24-Fig. 2328 illustrates.

[0196] Fig.24 is an illustrative video sequence of an exemplary region segmentation method 2400 for low-resolution content, arranged in accordance with at least some implementations of the present disclosure. In various implementations, the region segmentation method 2400 illustrates the segmentation of a "Stefan" low-resolution (CIF) sequence. For example, frame (a) shows the original frame, while frame (b) shows the final motion region mask (containing three motion regions and one region of non-active content).

[0197] Fig.25 is an illustrative video sequence of an exemplary region segmentation method 2500 for low-resolution content, arranged in accordance with at least some implementations of the present disclosure. In various implementations, the region segmentation method 2500 illustrates the segmentation of a low-resolution (CIF) "flower" sequence. For example, frame (a) shows the original frame, while frame (b) shows the final motion region mask (e.g., including three motion regions and a region of non-active content).

[0198] Fig.26 is an illustrative video sequence of an exemplary region segmentation method 2600 for low-resolution content, arranged in accordance with at least some implementations of the present disclosure. In various implementations, the region segmentation method 2600 illustrates the segmentation of a low-resolution (CIF) "bus" sequence. For example, frame (a) shows the original frame, while frame (b) shows the final motion region mask (e.g., containing three motion regions).

[0199] Fig.27 is an illustrative video sequence of an exemplary region segmentation method 2700 for standard-definition content, arranged in accordance with at least some implementations of the present disclosure. In various implementations, the region segmentation method 2700 illustrates the segmentation of a standard-definition (704×576) "city" sequence. For example, frame (a) shows the original frame, while frame (b) shows the final motion region mask (e.g., containing two motion regions).

[0200] Fig.28 is an illustrative video sequence of an exemplary region segmentation method 2800 for high-resolution content, arranged in accordance with at least some implementations of the present disclosure. In various implementations, the region segmentation method 2800 illustrates the segmentation of a high-resolution (1080p) "parking scene" sequence. For example, frame (a) shows the original frame, while frame (b) shows the final motion region mask (e.g., containing three motion regions).

[0201] Fig.Figure 29 is a block diagram of an exemplary morphologically based region post-processor 1110 arranged in accordance with at least some implementations of the present disclosure. In various implementations, the morphologically based region post-processor 1110 may include a morphological open / close (2x2 kernel) operator 2902, a small-size segment remover 2904, a SAD-based medium-size segment remover 2906, and a segment mask smoothing processor 2908.

[0202] As illustrated, Fig. 29 a detailed block diagram of the fifth block according to Fig.11, of the morphologically based region post-processor 1110. The morphologically based region post-processor 1110 can clean the region mask of typical noise related to the segmentation process. First, the 2 × 2 kernel-based morphological opening and closing operators can be applied to the region mask via the 2 × 2 kernel morphological opening / closing operator 2902. Next, small blobs can be removed from the resulting mask (e.g., where the minimum allowed blob size is resolution-dependent) via the small-size segment remover 2904. Thereafter, the medium-sized blobs may be removed via the SAD-based remover 2906 for medium-sized segments if their blob SAD, which is generated by removing the blob and reassigning another region to it, is higher than the SAD of the blob with the old region up to a tolerable threshold (e.g.,before removal). Finally, the resulting mask is smoothed via the segment mask smoothing processor 2908 by a peak removal filter described earlier in this section. The detection of areas with inactive content

[0203] A video can often contain an area of ​​inactive content, which can cause problems when calculating or applying global motion. Such non-content areas can include black bars and areas due to letterboxing, pillarboxing, circular or truncated circular fisheye cameras, etc. Detecting and excluding such an area can greatly improve GMM results.

[0204] Fig.30 is an illustrative video sequence 3000 of an example of compensation for detected non-content regions arranged in accordance with at least some implementations of the present disclosure. In various implementations, Fig. Figure 30 shows an example video sequence 3000, where black bars are detected and removed from the GMM, resulting in an impact on quality. The algorithm for detecting letterboxing and pillarboxing regions and removing them from the GMM is described next: 1. For all pixels in F that are at the left edge of the frame, perform the following: a. Scanning the current luminance frame F y from left to right and interrupt at an interrupt position if RsCs(F y ) greater than a threshold T bar (which is set to 240 in this implementation) or if the pixel value of F ythe threshold T blk for black bars (here T blk = 20 used); 2. Determine the dominant break position of the left frame edge, which is called L brp is defined as the multiple of 4 pixels closest to the majority of the left edge break positions; 3. If L brp is larger than 4 pixels, smaller than 1 / 3 of W (the frame width) and 90% or more of the break positions of the left edge are within a 4-pixel distance of the L brp then explain the non-content area on the left margin, which extends to L brp pixels wide spanned; and 4. Repeat steps 1-3 for the right edge, top edge, and bottom edge to detect the non-content area on the remaining sides of the frame.

[0205] In the Fig.In the example illustrated in Figure 30, the video sequence 3000 based on the "Stefan" sequence shows that there are 4 pixel thick bars detected at the top and right edge of the frame. The non-content area is then excluded from the GMM compensation, applying zero motion. The remaining area is modeled with the normal GMM, as shown in Fig. 30 represents.

[0206] The “Stefan” video sequence 3000, for example, shows the compensation of the detected non-content region (e.g., 2 bars, top and right, both 4 pixels thick, are detected and encoded, a [0, 0] movement is used into the bar region), where: frame (a) is the current original luminance frame, frame (b) is the reference luminance frame (1 frame removed), frame (c) is the reconstructed frame without bar detection, frame (d) is the residual frame with bar detection (SAD = 1087071), frame (e) is the reconstructed frame with bar detection, and frame (f) is the residual frame without bar detection (SAD = 933913). The generation of area-based motion models

[0207] The operations of generating region-based motion models may include several steps: (1) selecting which motion vectors to include in the estimation of the parametric model for each region, (2) adapting the subpixel filtering methods for each region (because, for example, different regions may have different texture properties), and (3) adaptively selecting a motion model per region. This part of the proposed exemplary algorithm can be used to estimate the motion models to be used for the detected motion regions in the frame.

[0208] Fig.31 is an illustrative block diagram of a multi-region motion estimator and modeler 108 arranged in accordance with at least some implementations of the present disclosure. In various implementations, the multi-region motion estimator and modeler 108 may include a motion vector selector 3106 for RMM estimation, an adaptive subpixel interpolation filter selector 3108, and an adaptive region motion model calculator and selector 3110.

[0209] In the illustrated example, Fig.31 illustrates a portion of the exemplary multi-region motion estimator and modeler 108 that computes a parametric motion model for each region. In the first step, the motion vector selector 3106 for RMM estimation may compute two potential motion vector selection masks for each region. Based on the smallest subsampled SAD of the region, the motion vector selector 3106 for RMM estimation may select one selection mask per region.

[0210] Next, one of the four possible subpixel interpolation filters can be selected based on the minimum subsampled SAD via the adaptive subpixel interpolation filter selector 3108. There may be multiple (e.g., four) predefined filters in the illustrated example. The filters may include, for example, the following filter types: (1) a (bilinear) 1 / 16-pixel smooth texture filter, (2) a (bicubic) 1 / 16-pixel medium texture filter, (3) a 1 / 8-pixel medium sharp texture filter (a modified AVC filter), and (4) a 1 / 8-pixel sharp texture filter (a modified HEVC filter), the like, and / or combinations thereof.

[0211] Finally, given the selected filters and selection masks, a region-based motion model can be selected for each region using the adaptive region motion model calculator and selector 3110. For each region, one of the three possible models is selected depending on the operating mode. For the given value of the operating mode control signal, either standard models or high-complexity models are used as candidates. If the value of the operating mode signal is 0, the system can adaptively select one of the following models per region: (1) a translational 4-parameter model, (2) an affine 6-parameter model, and (3) a pseudo-perspective 8-parameter model.On the other hand, if the value of the signal Mode = 1, the system can adaptively select on a range basis between: (1) a 6-parameter affine model, (2) a pseudoperspective 8-parameter model, and (3) a 12-parameter biquadratic model. The selection of motion vectors for the estimation of the region-based motion model

[0212] To estimate a more accurate motion model for each region, it is often important to select the motion vectors (within the given region) used in the model estimation process. For each region, there may be several (e.g., three) candidate selection masks computed, with one of them being selected to be used in the motion model estimation process.

[0213] Fig.32 is an illustrative block diagram of an exemplary motion vector selector 3106 for RMM estimation arranged in accordance with at least some implementations of the present disclosure. In various implementations, the motion vector selector 3106 for RMM estimation may include an affine RMM parameter calculator 3204, a downsampled SAD residual calculator 3208, a medium to strong texture-based selection mask refiner 3212, an affine RMM parameter calculator 3214, a downsampled SAD residual calculator 3216, a minimum SAD residual-based candidate selector 3218, a strong corner block-based selection mask refiner 3222, an affine RMM parameter calculator 3224, a downsampled SAD residual calculator 3226, and a minimum SAD residual-based candidate selector 3228.

[0214] As illustrated, Fig. 32 a detailed block diagram of the first block according to Fig. 31, of the selector 3106 of the motion vectors for the RMM estimation. Fig. Figure 32 shows the steps that may be required to calculate the selection mask used to identify which motion vectors to use in the region-based motion model estimation phase. In the first step of this process, the first candidate mask for each region is the entire region mask, denoted here by M0. For this, the RMM parameters can be calculated via the affine RMM parameter calculator 3204, and the downsampled SAD, denoted SAD0, can then be obtained via the downsampled SAD residual calculator 3208.

[0215] Next, a second candidate mask, denoted M1, can be obtained by refining M0 such that only medium and strong texture blocks are retained, while the flat texture blocks are removed via the medium-to-strong texture-based selection mask refiner 3212. In addition, the frame boundaries are also removed because most of the uncovered area appears there, yielding unreliable vectors. Similarly, the corresponding affine model for M5 (denoted Params1) can then be competed via the affine RMM parameter calculator 3214, and the corresponding downsampled SAD error, denoted SAD1, can be determined via the downsampled SAD residual calculator 3216. Then, either M0 or M1 (denoted M') is selected via the minimum SAD residual-based candidate selector 3218, placing them in the 2ndA refinement step is input, which generates the candidate selection mask M2 by selecting only the high-texture blocks (e.g., the blocks with high values ​​of both Rs and Cs) from the input mask M' via the strong-corner-based selection mask refiner 3222. Using the same steps as before, the corresponding affine model M2, denoted Params2, can be calculated via the affine RMM parameter calculator 3224, the corresponding subsampled SAD error, denoted SAD2, can be determined via the downsampled SAD residual calculator 3226, and according to the smallest error, either M2 or M' is selected as the final selection mask via the downsampled SAD residual calculator 3208.This is repeated for all areas so that the final mask M contains the binary information about which MVs are to be included in the calculation of the RMMs and which MVs are to be excluded from the calculation of the RMMs.

[0216] The selection of the motion vectors to be used for the motion model estimation for the region R can be performed as follows: 1. Apply the first binary selection mask M0 to all blocks in the frame that are part of the area R. 2. If H < 300 and W < 600 and W × H < 180,000, then set T = 4, otherwise set T = 6. 3. Create an additional candidate mask M1 by refining the current selection mask as follows: a. Remove all blocks that lie on the frame boundary. b. Remove all blocks whose minimum Rs and Cs texture dimensions are smaller than the threshold T. 4. For masks M0 and M1, compute a 6-parameter affine model using the least squares fitting algorithm such that only the motion vectors specified by the mask are used in the least squares computation process. 5. Calculate the SAD measures for the computed affine models corresponding to M0 and M1, compare them, and set the current model and mask to the one with the smallest SAD measure. 6. The threshold values ​​T RS to 1.5 × the average Rs value in the Rs / Cs 2-D arrangement and T CS to 1.5 × the average Cs value in the Rs / Cs 2-D arrangement. 7. Create the final candidate selection mask M2 by refining the current selection masks as follows: a. All blocks whose Rs texture dimension is smaller than the threshold T RS and whose Cs texture measure is smaller than the threshold T CS is, remove. 8. For the mask M2, compute a 6-parameter affine model using the least squares fitting algorithm such that only the motion vectors specified by the mask are used in the least squares computation process. 9. Calculate the SAD measure for the computed affine model corresponding to M2 and compare it with the SAD measure of the current model and set the current model and the current mask to the one with the smallest SAD measure. 10. Output the current mask as the mask for selecting the motion vectors for region R, which will be used to calculate the motion model of this region.

[0217] The adaptive selection of the area-based subpixel filter

[0218] To maximize the gains for region-based motion modeling, an optimal subpixel filtering for motion compensation can be adaptively selected for each region. Here, one of four different subpixel filtering methods is selected for a region. Table 3 lists the four filters used in such an implementation. Table 3 The subpixel filters used in the area-based motion compensation and filter Number of taps accuracy Typically suitable for bilinear filter for every 1 / 16 pixel positions 2 1 / 16 pixels blurry texture bicubic filter for all 1 / 16 pixel positions 16 1 / 16 pixels blurry and normal texture AVC-based filter[1 -5 20 20 -5 1] for 1 / 2 pixel and 1 / 4 pixel positions bilinear for 1 / 8 pixel positions 62 1 / 8 pixel normal and sharp texture HEVC-based filter [-1 4 -11 40 40 -11 4 -1] for 1 / 2 pixel positions [-1 4 -10 58 17 -5 1 0] and [0 1 -5 17 58 -10 4 -1] for 1 / 4 pixel positions [0 -1 9 9 -1 0] for 1 / 8 pixel positions 8886 1 / 8 pixel sharp texture

[0219] The optimal filter for a given region may be content-dependent. Typically, sharper luminance content can be better filtered using HEVC-based and AVC-based filters. For example, a HEVC-based filter usually performs better on content with a very sharp texture. On the other hand, fuzzier textured luminance regions can be better filtered using bicubic and bilinear filters, with bicubic filters likely performing better when interpolating medium-textured regions. Obviously, the most suitable region filters yield the smallest SAD of the reconstructed frame compared to the current frame. To select the most optimal filter for a given region, a simplified subsampled SAD (SSAD) measure can be calculated for speed. The procedure for automatically selecting an optimal filter for a region is described next.

[0220] Fig.33 is an illustrative block diagram of an exemplary adaptive subpixel interpolation filter selector 3108 arranged in accordance with at least some implementations of the present disclosure.In various implementations, the adaptive subpixel interpolation filter selector 3108 may include a frame subsampler 3302, an RMM-based subsampled prediction frame generator 3304, a (bilinear) soft filter coefficient 3306, an RMM-based subsampled prediction frame generator 3308, a mean (bicubic) filter coefficient 3310, an RMM-based subsampled prediction frame generator 3312, a mean (AVC-based) filter coefficient 3314, an RMM-based subsampled prediction frame generator 3316, a sharp (HEVC-based) filter coefficient 3318, a frame subsampler 3320, a SAD residual calculator 3322, a SAD residual calculator 3324, a SAD residual calculator 3326, a SAD residual calculator 3328 and a minimum SAD based selector 3330.

[0221] The illustrated example shows a detailed view of the second block after Fig.31, of the adaptive subpixel interpolation filter selector 3108. In Fig. 33, the adaptive subpixel interpolation filter selector 3108 is used to adaptively select the appropriate subpixel filter according to the video content for each region. Fig. In the exemplary adaptive subpixel interpolation filter selector 3108 illustrated in Figure 33, there are four default filters (although a different number could be used): (1) a (bilinear) 1 / 16-pixel smooth texture filter, (2) a (bicubic) 1 / 16-pixel medium texture filter, (3) a 1 / 8-pixel medium sharp texture filter (a modified AVC filter), and (4) a 1 / 8-pixel sharp texture filter (a modified HEVC filter). The previously computed initial affine model of global motion is used to move all pixels in the reference frame according to the model and all four filter candidates.

[0222] The reference frame F refcan be subsampled via the frame subsampler 3302 to produce a subsampled reference frame SF ref Similarly, the current frame F may be subsampled via the frame subsampler 3320 to generate a subsampled current frame SFs.

[0223] For each pixel in the subsampled reference frame SF ref The resulting motion vectors (depending on the filter candidate) are rounded to an accuracy of either 1 / 16 pixel or 1 / 8 pixel via the RMM-based subsampled prediction frame generators 3304, 3308, 3312 and 3316. This results in four prediction frames, which are subsampled by PSF S (which is calculated using the smooth subpixel filter candidate via the (bilinear) soft filter coefficient 3306), PSF M (which is the result of using the mean subpixel filter candidate over the mean (bicubic) filter coefficient 3310), PSFMSh (which is the result of using the medium-sharp subpixel filter candidate over the mean (AVC-based) filter coefficient 3314) and PSF Sh (which is the result of using the sharp subpixel filter candidate via the sharp (HEVC-based) filter coefficient 3318). Next, the subsampled SAD error is calculated via the SAD residual calculators 3322, 3324, 3326, and 3328 for all 4 candidates for each region, where the minimum SAD criterion can be used to select the final subpixel filter filts of the region via the minimum SAD-based selector 3130.

[0224] An example algorithm for automatically selecting the optimal subpixel filtering for a region R may include the following steps: 1. SSAD i = 0, i = 0, ..., 3, set. 2. For each filter flt i, i = 0, ..., 3, perform the following: a. For each N × N block in the reference frame that is part of the region R: i. Take the pixel in the center of the frame, calculate the motion vector according to the affine model of the global motion for R calculated in the previous part (e.g., the model for R calculated in the step of selecting the motion vector mask). ii. The calculated vector depending on the filter flt i round to either 1 / 8 or 1 / 16 pixel accuracy (the corresponding filter accuracy is shown in Table 3). iii. The interpolated subpixel value corresponding to the calculated motion vector using the filter flt i calculate. iv. Calculate the absolute difference between the center pixel of the current block in the current frame and the calculated interpolated subpixel value and the SSAD i increase by this amount. 3. The filter flt i , for which the SSAD i Select the smallest one. The calculation and selection of the adaptive area-based movement model

[0225] Depending on the operating mode, the algorithm can select the region-based motion models with optimal complexity in this step. There may be multiple operating modes (e.g., two operating modes, although the number could vary), as defined in some implementations described here. In such an example, the operating modes may include: 1. Mode 0 (the default mode) is a mode that can be designed for sequences with normal motion complexity. Mode 0 can, for example, adaptively switch on a per-frame basis between region-based translational 4-parameter, affine 6-parameter, and pseudo-perspective 8-parameter motion models. 2. Mode 1 - is a mode that can be designed for sequences with complex motion (such as sequences with high perspective depth, fast motion, etc.). Mode 1 can, for example, adaptively switch on a per-frame basis between region-based affine 6-parameter, pseudo-perspective 8-parameter, and biquadratic 12-parameter motion models.

[0226] For typical applications, the adaptive translational 4-parameter, affine 6-parameter, and pseudoperspective 8-parameter modes (e.g., mode 0) can be used. Therefore, mode 0 may be specified as a default mode in some implementations described here.

[0227] Fig.34 is an illustrative block diagram of an example adaptive RMM calculator and selector 3110 arranged in accordance with at least some implementations of the present disclosure.In various implementations, the adaptive RMM calculator and selector 3110 may include a 4-parameter translational least squares RMM calculator 3402, an RMM-based prediction frame generator 3404, a SAD residual calculator 3406, a 6-parameter affine least squares RMM calculator 3412, an RMM-based prediction frame generator 3414, a SAD residual calculator 3416, an LMA-based (Levenberg-Marquardt algorithm-based) pseudoperspective 8-parameter RMM calculator 3422, an RMM-based prediction frame generator 3424, a SAD residual calculator 3426, an LMA-based (Levenberg-Marquardt algorithm-based) biquadratic 12-parameter RMM calculator 3432, an RMM-based Prediction frame generator 3434, a SAD residual calculator 3436, a minimum SAD parameter index (parindx) calculator, and an RMM selector based on the parameter index (parindx).

[0228] In the illustrated example, Fig. 34 a detailed view of the third block after Fig. 31, the adaptive RMM calculator and selector 3110. In Fig. 34, the final global motion model is generated via the adaptive RMM calculator and selector 3110. A control signal "Mode" can be used to select between the standard models and the high-complexity models. While the illustrated example shows four overall models, with three models used in a first mode and three models used in a second mode, it is recognized that a different number of modes may be used, a different number of overall models may be used, and / or a different number of models per mode may be used.

[0229] In the illustrated example, the control signal "Mode" can be used to select between the standard models and the high-complexity models as follows: if "Mode" = 0, the adaptive RMM calculator and selector 3110 can adaptively select one of the following models: (1) the translational 4-parameter model, (2) the affine 6-parameter model, and (3) the pseudo-perspective 8-parameter model. Otherwise, if "Mode" = 1, the adaptive RMM calculator and selector 3110 can select between: (1) the affine 6-parameter model, (2) the pseudo-perspective 8-parameter model, and (3) the biquadratic 12-parameter model.

[0230] In any case, three of the four models can be calculated using the motion vector selection mask M and the corresponding model calculation method (e.g., least squares fitting for the four-parameter models and the six-parameter models, the Levenberg-Marquardt algorithm (LMA) for the eight-parameter models and the 12-parameter models). B. 3 of the 4 models are calculated using the motion vector selection mask M and the corresponding model calculation method via the corresponding calculator 3402 of a translational 4-parameter least squares RMM, the calculator 3412 of an affine 6-parameter least squares RMM, the calculator 3422 of an LMA-based (based on the Levenberg-Marquardt algorithm) pseudo-perspective 8-parameter RMM and the calculator 3432 of an LMA-based (based on the Levenberg-Marquardt algorithm) biquadratic 12-parameter RMM.

[0231] For the 3 calculated models, using the previously selected subpixel filtering method filt and the reference frame F ref the corresponding prediction frames are generated via the corresponding RMM-based prediction frame generators 3404, 3414, 3424 and / or 3434.

[0232] Furthermore, the frame-based SAD error for the 3 prediction frames with respect to the current frame F can be calculated via the SAD residual calculators 3406, 3416, 3426 and / or 3436.

[0233] Finally, the SAD errors are weighted and compared so that the smallest weighted SAD is used to select the appropriate model via the minimum SAD parameter index (parindx) calculator and the parameter index (parindx)-based RMM selector.

[0234] In mode 0, the 6-parameter affine model can be set to the affine model of global motion calculated in the motion vector mask selection step. The temporary 4-parameter motion model can be calculated using the direct least-squares fitting approach described above. It is important to note that the selected motion vector mask calculated in the refinement step can be used to filter only the motion vectors relevant to region-based motion. Least-squares fitting can be performed on the motion vectors from the motion vector field whose corresponding value in the selected motion vector mask is 1. Next, the pseudoperspective 8-parameter model can be calculated using the Levenberg-Marquardt (LMA) algorithm for nonlinear least-squares fitting.Similarly, a new parameter set (an 8-parameter model) can be computed using only the motion vectors from the motion vector field whose corresponding value in the global / local binary mask from the previous step is 1. Once the parameters for all models are available, the SAD measure for the 4-, 6-, and 8-parameter models, known as SAD, can be calculated. 4p , SAD 6p or SAD 8p , can be calculated for each region. The SAD measure can be the sum of the absolute differences between the current luminance frame and the reconstructed luminance frame. The reconstructed frame can be obtained by applying the equations of the region-based motion model to all pixels in the reference frame on a frame-by-frame basis. In this process, either 1 / 8 or 1 / 16 pixel accuracy can be used, depending on the selected subpixel filter.

[0235] The quality control parameter in mode 0, denoted by δ0, can be calculated as follows: δ0 = 0.01 × min (SAD 4p , SAD 6p , SAD 8p ).

[0236] The selection of the final parameter model can be performed as follows: If SAD 6p < SAD 8p + δ0 and SAD 4p < SAD 6p + δ0, then select the translational 4-parameter model to model the global motion in the current domain; If SAD 6p < SAD 8p + δ0 and SAD 4p ≥ SAD 6p + δ0, then select the affine 6-parameter model to model the global motion in the current domain; If SAD 6p ≥ SAD 8p + δ0 and SAD 4p < SAD 6p + δ0, then select the translational 4-parameter model to model the global motion in the current area; and If SAD6p ≥ SAD 8p + δ0 and SAD 4p ≥ SAD 6p + δ0, then select the pseudo-perspective 8-parameter model to model the global motion in the current region.

[0237] In mode 1, the affine 6-parameter model can also be set to the previously calculated affine model. The pseudoperspective 8-parameter model and the biquadratic 12-parameter model can be calculated using the Levenberg-Marquardt algorithm (LMA algorithm) for nonlinear least squares fitting. These parameter sets can only be calculated using the motion vectors from the motion vector field whose corresponding value is in the binary motion vector selection mask. Once the parameters for all models are available, the SAD measure for the 6-, 8-, and 12-parameter models, known as SAD 6p , SAD 8p or SAD 12pThe SAD measure can be calculated as the sum of the absolute differences between the current luminance frame and the reconstructed luminance frame. The reconstructed frame can be obtained by applying the equations of the region-based motion model to all pixels in the reference frame. In this process, either 1 / 8 or 1 / 16 pixel accuracy can be used, depending on the selected subpixel filter.

[0238] The quality control parameter in mode 1, denoted by δ1, can be calculated as follows: δ1 = 0.01 × min (SAD 6p , SAD 8p , SAD 12p ).

[0239] The selection of the final parameter model can be performed as follows: If SAD 8p < SAD 12p + δ1 and SAD 6p < SAD 8p+ δ1, then select the translational 4-parameter model to model the global motion in the current domain; If SAD 8p < SAD 12p + δ1 and SAD 6p ≥ SAD 8p + δ1, then select the affine 6-parameter model to model the global motion in the current domain; If SAD 8p ≥ SAD 12p + δ1 and SAD 6p < SAD 8p + δ1, then select the translational 4-parameter model to model the global motion in the current area; and If SAD 8p ≥ SAD 12p + δ1 and SAD 6p ≥ SAD 8p + δ1, then select the pseudo-perspective 8-parameter model to model the global motion in the current region. The precise motion compensation based on the area-based motion model

[0240] At the beginning of the region-based motion compensation phase, both a model (e.g., with either 4, 6, 8, or 12 parameters depending on the operating mode) and a subpixel filtering method were selected. Although region-based motion compensation processes the blocks of a region at once, the RMM can be applied at a pixel level within the given block. In other words, for each pixel within a block, a region-based motion vector can be calculated, and the pixel can be moved to a subpixel position according to the previously determined subpixel filtering method. Consequently, a pixel on one side of the block can have a different motion vector than a pixel on the other side of the same block, as illustrated by an example in Fig. 35 is illustrated below.

[0241] Fig.35 is an illustrative diagram of an example translational 4-parameter global motion model 3500 arranged in accordance with at least some implementations of the present disclosure. In various implementations, an example translational 4-parameter model 3500 may be applied to different pixel positions within an 8x8 block. It is noted that different global motion vectors may appear within a block.

[0242] In the illustrated example, the selected block size depends on the resolution. In one example, for standard and high resolution, the block size might be set to 8; while for low-resolution sequences, the block size might be set to 16. If a frame width is less than 600, a frame height is less than 300, and the product of the frame width and frame height is less than 180,000, then the sequence can be classified as a low-resolution sequence, although other numbers can be used.

[0243] Fig.36 is an illustrative block diagram of an exemplary adaptive global motion compensator 110 arranged in accordance with at least some implementations of the present disclosure. In various implementations, the adaptive global motion compensator 110 may include a GMM parameter to reference point MV converter 3602, a reference point MV to GMM parameter reconstructor 3604, a global motion model-based prediction frame generator 3606, and an SAD residual calculator 3608.

[0244] In the illustrated example, Fig. 36 the details of the (originally in Fig.1) that can generate the final RMM-based SAD. The input to the first block of this method can be the final parameters of the region-based motion models, which are applied to the frame-based reference points via the GMM parameter to reference point MVs converter 3602. As previously described in detail, the number of reference points depends on the model. An n-parameter model uses n / 2 reference points. Therefore, in the first step, n / 2 motion vectors corresponding to the motion at the reference points can be calculated. The calculated motion vectors can be quantized to 1 / 4 pixel accuracy.

[0245] Next, the reconstructed parameters can be generated from the reference points using the reconstructor 3604 of the MVs of the reference points to the GMM parameters. The reconstructed parameters can be obtained by solving the system of equations for the motion vectors at the reference points separately for each region. Furthermore, the reconstructed parameters can be represented as a quotient with the denominator scaled to a power of 2. This means that the parameters can only be applied with multiplication and binary shift operations for the sake of speed.

[0246] Then, the prediction frame P can be calculated by applying the reconstructed parameters of the motion model to the pixels of the reference frame F refare generated, wherein the subpixel positions are interpolated separately for each region using the global motion model-based prediction frame generator 3606 with the previously selected region-based filters filts. Finally, the corresponding frame-based SAD can be calculated from the predicted frame P and the current frame F using the SAD residual calculator 3608.

[0247] Because it is not possible to encode the actual floating-point representation of the global motion model parameters, an approximation of the parameters is performed. The method of representing the RMM parameters is based on the concept of reference points (also called control grid points) described above. According to this representation, an n-parameter RMM model requires n / 2 reference points. A motion vector may need to be sent at each reference point to reconstruct the parameters on the decoder side. The accuracy of the encoded motion vectors at the reference points determines the accuracy of the approximation of the RMM parameters. In some implementations, this accuracy can be set to 1 / 4 pixel accuracy.

[0248] The locations of the reference points are defined as follows: z0=(x0,y0), z1=(x1,y1)=(x0+W,y0), z2=(x2,y2)=(x0,y0+H), z3=(x3,y3)=(x0+W,y0+H), z4=(x4,y4)=(x0−W,y0), z5−(x5,y5)=(x0,y0−H).

[0249] For a 4-parameter model, the points z0 and z3 are used. Applying the translational model g4 of the global motion to z0 and z3 yields the globally moving points g4(z0) = (g4(x0), g4(y0)) = (a0x0 + a1,a2y0 + a3) and g4(z3) = (a0x3 + a1,a2y3 + a3). On the other hand, for a 6-parameter model, the points z0, z1, and z2 are used. Applying the affine model g6 of the global motion to z0, z1, and z2 yields the globally moving points g6(z i ) = (g6(x i ),g6(y i ) = (a0x i + a1y i + a2,a3x i + a4y i + a5), i = 0, 1, 2. For an 8-parameter model, the points z0, z1, z2, and z3 are used. Applying the pseudo-perspective model g8 of global motion to z0, z1, z2, and z3 yields the globally moving points g8(zi)=(g8(xi),g8(yi))=(a0xi2+a1xiyi+a2xi+a3yi+a4,a1yi2+a0xiyi+a5xi+a6yi+a7), i = 0, 1, 2, 3. Finally, for a 12-parameter model, all 6 points (z0, z1, z2, z3, z4, and z5) are used. Applying the biquadratic 12-parameter model g 12 the global movement on z0, z1, z2, z3, z4 and z5 provides the globally moved points g12(zi)=(g12(xi),g12(yi))=(a0xi2+a1yi2+a2xiyi+a3xi+a4yi+a5,a6xi2+a7yi2+a8xiyi+a9xi+a10yi+a11), i= 0, 1, 2, 3, 4, 5.

[0250] As discussed earlier, the motion vectors at the reference points define a system of equations whose solution determines the reconstructed parameters of the global motion model. To enable fast processing, the reconstructed parameters can be approximated with a ratio of two integers, where the denominator is a power of 2. In this way, applying the RMM to any pixel location in the frame can be performed using multiplication and binary shift operations.

[0251] To reconstruct the 4-parameter model {a0, a 1, a2, a3} from the given model g4 (applied to the 1 / s pixel accuracy), the following equation can be used: a¯0=g4(x3)−g4(x0)sW,a¯1=g4(x0)s,a¯2=g4(y3)−g4(y0)sH,a¯3=g4(y0)s.

[0252] To enable rapid modeling of global motion, this equation can be modified as follows: a¯0=d02k,a¯1=d12k,a¯2=d22l,a¯3=d32l.

[0253] Where d0=(2k / (SW))×(g4(x3)−g4(x0)),d1=(2k / s)×g4(x0),k=⌈log2sW⌉,d2=(2l / (sH))×(g4(y3)−g4(y0)),d3=(2l / s)×g4(y0),l=⌈log2sH⌉ is.

[0254] To apply the reconstructed model g4 of global motion to a pixel location (x, y), the following equation can be used without division: g¯4(x)=a¯0x+a¯1=d02kx+d12k=(d0x+d1)>>k and g¯4(y)=a¯2y+a¯3=d22ly+d32l=(d2x+d3)>>l.

[0255] Where >> denotes the bitwise shift to the right.

[0256] To obtain the reconstructed 6-parameter model {a0,...,a5} from the given model g6 (applied to 1 / s pixel accuracy), the following equation can be used: a¯0=g6(x1)−g6(x0)sW,a¯1=g6(x2)−g6(x0)sH,a¯2=g6(x0)s, a¯3=g6(y1)−g6(y0)sW,a¯4=g6(y2)−g6(y0)sH,a¯5=g6(y0)s.

[0257] This equation can be modified to allow rapid modeling of global motion as follows: a¯0=d02k,a¯1=d12k,a¯2=d22k,a¯3=d32k,a¯4=d42k,a¯5=d52k.

[0258] Where d0=(2k / (sW))×(g6(x1)−g6(x0)),d1=(2k*(sH))×(g6(x2)−g6(x0)),d2=(2k / s)×g6(x0),d3=(2k / (sW))×(g6(y1)−g6(y0)),d4=(2k / (sH))×(g6(y2)−g6(y0)),d5=(2k / s)×g6(y0),k=⌈log2(s2WH)⌉ is.

[0259] To apply the reconstructed model g6 of the global motion to a pixel location (x, y), the following equation can therefore be used without division: g¯6(x)=a¯0x+a¯1y+a¯2=d02kx+d12ky+d22k=(d0x+d1y+d2)>>k and g¯6(y)=a¯3x+a¯4y+a¯5=d32kx+d42ky+d52k=(d3x+d4y+d5)>>k.

[0260] In the case of the pseudo-perspective model, to obtain the reconstructed 8-parameter model {a0, ...,a7} from the given model g8 (applied to the 1 / s pixel accuracy), the following equation can be used: a¯0=g8(y0)−g8(y1)−g8(y2)+g8(y3)s2WH, a¯1=g8(x0)−g8(x1)−g8(x2)+g8(x3)s2WH, a¯2=−sHg8(x0)−sWg8(y0)−sHg8(x1)+sWg8(y1)+sWg8(y2)−sWg8(y3)s2WH, a¯6=−sHg8(x0)−sWg8(y0)+sHg8(x1)+sWg8(x2)+sWg8(y2)−sWg8(x3)s2WH.

[0261] As in the previous cases of the simpler models, this equation can be expressed as follows: a¯i=di2k,i=0,…,7.

[0262] To apply the reconstructed model g8 of global motion to a pixel location (x, y), the following equation can be used without division: g¯8(x)=(d0x2+d1xy+d2x+d3y+d4)>>k and g¯8(y)=(d1y2+d0xy+d5x+d6y+d7)>>k.

[0263] Where k=⌈log2(s2WH)⌉ is.

[0264] Finally, in the case of the biquadratic model, to obtain the reconstructed 12-parameter model {a0,...,a 11} from the given model g (applied to the 1 / s pixel accuracy) 12 To obtain the following equation can be used: a¯0=−2g12(x0)+g12(x1)+g12(x4)2s2W2,a¯1=−2g12(x0)+g12(x2)+g12(x5)2s2H2, a¯2=g12(x0)+g12(x1)−g12(x2)+g12(x3)2s2WH2,a¯3=g12(x1)−g12(x4)sW, a¯3=g12(x2)−g12(x5)sH,a¯5=g12(x0)s, a¯6=−2g12(y0)+g12(y1)+g12(y4)2s2W2, a¯7=−2g12(y0)+g12(y2)+g12(x5)2s2H2, a¯8=g12(y0)−g12(y1)−g12(y2)+g12(y3)2s2WH2,a¯9=g12(y1)−g12(y4)sW, a¯10=g12(y2)*−g12(y5)sH,a¯11=g12(y0)s.

[0265] As in the previous cases of the simpler models, this equation can be expressed as follows: a¯i=di2k,i=0,…,11.

[0266] Where k=⌈log2(s2W2H2)⌉ is.

[0267] To reconstruct the model g 12 To apply the global motion to a pixel location (x, y), the following equation can therefore be used without division: g¯12(x)=(d0x2+d1y2+d2xy+d3x+d4y+d5)>>k and g¯12(y)=(d6x2+d7y2+d8xy+d9x+d10y+d11)>>k.

[0268] Based on the calculated SAD, either the calculated parameters of the global motion model are encoded or the model is approximated from a set of previous models. Typically, the approximated model produces a larger SAD than the calculated model, but it is usually encoded using a significantly smaller number of bits. The details of the encoding process are described next. The efficient coding of the parameters of the area-based movement model

[0269] In typical video content, consecutive frames within the same scene, and even frames separated by several frames (but still within the same scene), maintain the same or very similar motion characteristics. In other words, sudden changes in global motion, such as direction or magnitude, are rare occurrences within a video scene. Therefore, global motion models for consecutive or nearby frames are unlikely to change much. Furthermore, models from the most recent previous frames typically perform very well as global motion models for the current frame. In this sense, the method for encoding global motion parameters in the MPEG-4 standard is suboptimal because it does not fully utilize previous models from the recent past.Accordingly, some implementations here may use an encoding algorithm that fully exploits the redundancy of the previous global motion models to represent and encode the RMM parameters.

[0270] The proposed method for encoding RMM parameters, like the global motion encoding method of the MPEG-4 standard, can rely on reference points to represent a model. The MPEG-4 global motion encoding method has been described above.

[0271] A codebook is a collection of prior parameters represented as the motion vectors of the reference points based on the global motion. Initially, the codebook is empty because no prior models are known. As frames are processed, the codebook is updated to include the newly coded models. Only unique models can be added. When the codebook becomes full, e.g., when the number of models in the codebook is the same as the maximum capacity of the codebook, the oldest model is replaced by the newest model. The codebook is therefore content-adaptive because it changes during the encoding / decoding process. In the experiments based on the implementations described here, the best performance / complexity tradeoff was achieved with a codebook of size 8. Consequently, in some implementations, the codebook size is set to 8, although a different size could be used.Each area is assigned a separate codebook.

[0272] As discussed previously, the number of motion vectors required to represent a model depends on the number of parameters in the model itself. Assume that each frame uses a 6-parameter affine model (e.g., mode 0). Then, the parameters of each model can be represented with 3 motion vectors associated with 3 reference points. A full codebook in such an example would therefore contain a total of 24 motion vectors associated with the reference points of the previous models.

[0273] Fig.37 is an illustrative block diagram of an exemplary region-based motion model parameter and head entropy encoder 112 arranged in accordance with at least some implementations of the present disclosure.In various implementations, the region-based motion model parameter and head entropy encoder 112 may include a converter 3702 of the RMM parameters to the mvs of the reference points, a codebook 3704 of the mvs of the previous RMM reference points, a frame-distance-based scaler 3706, an adaptor 3708 of the codewords to the mvs of the RMM reference points, a codeword VLC selector (a codeword selector of a variable length code) 3710, the codeword VLCs 3712, a model converter and frame-distance-based scaler 3714, a residue calculator 3716 of the mvs of the reference points, a residual entropy encoder 3718, the modified Golomb codes 3720, a least bit cost selector 3722, a residue calculator 3726 of the mvs of the reference points and a residue entropy encoder 3728.

[0274] Fig.Figure 37 shows an exemplary region-based motion model parameter and head entropy encoder 112 that can be used to encode the parameters of the global motion model. In this example, the region-based motion model parameter and head entropy encoder 112 performs the illustrated operations separately for each region's RMM parameter set (e.g., on a region-by-region basis). In this example, the encoding of the RMM parameters is based on the codebook principle. A codebook of up to 8 most recently encountered parameters is maintained via the codebook 3704 of the mvs of the previous RMM reference points and updated with each new frame. There is a separate codebook maintained for each model, therefore resulting in a total of 3 codebooks in the system per region. The entries in the codebook (e.g., the codewords) are used as the predictors for the current parameters.Each codeword contains the motion vectors of the reference points (corresponding to the previous RMM parameters) along with the information about the number of parameters as well as the frame distance fd (the distance between the current frame and the reference frame) and the direction dir used in estimating the model.

[0275] The final calculated RMM parameters are first converted to frame-level reference points via the RMM parameter-to-mvs converter 3702. As previously described in detail, the number of reference points depends on the model. An n-parameter model uses n / 2 reference points. Therefore, in the first step, n / 2 motion vectors corresponding to the motion at the reference points are calculated. The calculated motion vectors can be quantized to 1 / 4 pixel accuracy.

[0276] In the illustrated example, two coded bits can be computed in parallel: (1) the coded residues with the most recent codeword via the residual entropy encoder 3718 and (2) the coded residues with the most closely matched codeword via the residual entropy encoder 3728 and / or with the codebook index code via the codeword VLC selector (the codeword selector of a variable length code) 3710.

[0277] In the first approach, the most recent model of all three codebooks is selected from the codebook 3704 of the MVs of the previous RMM reference points, denoted by "most recent" in the graphical representation. It is then scaled via the model converter and frame-distance-based scaler 3714 according to the fd and dir values ​​to match the distance and direction of the ref_pts_mvs. In addition to the scaling, the model is translated to match the number of points in the current model. In the case where the current model has more points than the most recent model, the model is reconstructed, and the missing MVs of the additional points are calculated and added to the MVs of the points of the most recent model. The resulting predicted points are denoted in the graphical representation as predicted_most_recent_ref_pts_mvs.

[0278] The resulting predicted points predicted_latest_ref_pts_mvs can be differentiated via the residual calculator 3716 of the mvs of the reference points with ref_pts_mvs to generate the residuals.

[0279] Such residues can then be encoded via the residue entropy encoder 3718 with the modified Golomb code from the modified Golomb codes 3720. The modified Golomb codes can be adaptive, choosing a sharp, medium, or flat table based on the previous residue size.

[0280] The first encoded bits may be redirected to the least-bit-cost selector 3722, which serves to select the method with the lowest bit cost. The least-bit-cost selector 3722 also has as input the second encoded bits obtained in the second path, as previously discussed.

[0281] In the second path, the calculated points ref_pts_mvs are compared with the points from the corresponding codebook using the RMM reference point mvs adjuster 3708. Before the comparison, the points from the previous RMM reference point mvs codebook 3704 can be scaled according to the fd and dir values ​​via the frame-distance-based scaler 3706. If the ref_pts_mvs match an entry in the codebook, the exact_match control signal is set to 1 via the RMM reference point mvs adjuster 3708, and the process outputs the codebook index bits as the second set of coded bits.

[0282] Otherwise, the exact match is set to 0 via the RMM reference point mvs adjuster 3708, where the ref_pts_mvs are encoded differently as follows. The most accurate model calculated by the RMM reference point mvs adjuster 3708, denoted in the graphical representation as scaled_adjusted_ref_pts_mvs, is used to calculate the residuals via the reference point mvs residual calculator 3726. The residuals are calculated and encoded via the residual entropy encoder 3728 using the modified adaptive Golomb codes from the modified Golomb codes 3720. The codebook index bits and the remaining bits are combined into the second set of encoded bits. The final step is to select the encoding method and output the final encoded bits preceded by a one-bit select bit.This is repeated for each region, with the final output bits consisting of the final coded bits of the individual region, all appended to form the frame-based final coded bits.

[0283] Each entry in the codebook is also associated with a codeword selected from the codeword VLCs 3712 by the codeword VLC selector (the codeword selector of a variable-length code) 3710, which is used to encode its index. The probability distribution of the most optimal codebook model with respect to the current frame is slightly offset from the most recent model, as shown in Fig. 38. Based on these observations, Table 4 defines the variable-length code (VLC) tables used to encode the codebook indices.

[0284] Fig.38 is an illustrative graph 3800 of an example probability distribution of the best prior codebook models arranged in accordance with at least some implementations of the present disclosure. In various implementations, graph 3800 illustrates a probability distribution of the best prior codebook models (with codebook indices 0-7) for the current frame. It can be observed that the most likely optimal model is the newest model (index 0), while the least likely model is the oldest model (index 7). However, the distribution is not too peaked. Table 4 below illustrates the variable length codes used to encode the codeword index in the codebook in the RMM: Codeword index VLCs for all codewords in the codebook depending on the size of the codebook Size= 0 / 1 Size= 2 Size= 3 Size= 4 Size= 5 Size= 6 Size= 7 Size= 8 0 - 0 0 00 00 00 00 00 1 - 1 10 01 01 01 01 01 2 - - 11 10 10 100 100 100 3 - - - 11 110 101 101 101 4 - - - - 111 110 110 1100 5 - - - - - 111 1110 1101 6 - - - - - - 1111 1110 7 - - - - - - - 1111

[0285] In the proposed approach, each model can have its own codebook. In both mode 0 and mode 1, there can be multiple (e.g., three) codebooks, which are maintained because each of the models allows for multiple (e.g., up to three) models.

[0286] The codebook-based methods described here can alternate between encoding an exact model with the codebook index and encoding the index and error residuals. To determine which encoding method is appropriate for the given framework, the SADs of all prior parameters from the codebook of the corresponding model can be computed. The parameter set corresponding to the smallest SAD can be selected, and the SAD of the computed model can be compared to it. If the SAD of the selected codebook model is up to a threshold (e.g., 1% larger than the SAD of the computed model), the codebook model can be selected and coded according to Table 4. Otherwise, the computed model can be selected. Next, a method for encoding the computed model using a predictive approach is described.

[0287] The encoding of the computed global motion model can be performed by encoding the residuals of the predicted global motion vectors of the reference points (i.e., the control grid points). As discussed earlier, the number of reference points depends on the number of model parameters. The prediction of the motion vectors at the reference points can be performed using the global motion model from the previous frame, even if the model of the current frame and that of the previous frame might be different. In the case where the models of the current and previous frames are the same, or if the model of the current frame uses fewer reference points, the motion vectors at the grid points can be copied from the previous frame. However, if the model of the current frame is more complex, e.g.If more points than the model of the previous frame are used, then the motion vectors of the reference points of the previous frame are all copied, and the additional missing reference points can be calculated using the model from the previous frame. Once the predicted reference points have been obtained, the difference (the residual) between them and the motion vectors at the reference points corresponding to the calculated model of the global motion of the current frame can be obtained and encoded with "modified" generalizing Golomb codes.

[0288] Instead of relying on the Exp-Golomb code as in MPEG-4 encoding of the global motion parameters, some implementations can use an adaptive VLC method that can select one of three contexts based on the previously observed differences / residuals. If a prior difference is small (the size is ≤ 4), the sharp VLC table can be used. The sharp VLC table can be a modified generalized Exp-Golomb code with k = 0, where the first 15 entries are modified to sizes {1, 3, 3, 4, 4, 6, 6, 7, 7, 7, 7, 7, 7, 8, 8}. An example VLC table is shown in Table 5. In the case where the prior difference has a medium size (> 4 and ≤ 64), the mean VLC table can then be used.The mean VLC table can be a modified generalized Exp-Golomb code with k = 2, where the first 30 entries are modified to sizes {3, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 7, 7, 7, 7, 7, 8, 8, 8, 8, 8}. An example VLC table is shown in Table 6. Finally, if the prior differences are large (> 64), the shallow VLC table can be used. The flat VLC table can be a modified generalized Exp-Golomb code with k = 5, where the first 40 entries are modified to sizes {5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7, 7, 8, 8, 8, 8, 8}. An example VLC table is shown in Table 7.

[0289] The following tables show the details of the “modified” generalized Golomb codes used in the RMM. The value m of the difference of the motion vectors is calculated using the following rule as a non-negative integer v m shown. vm={2m−1if m>0,−1mif m≤0. Table 5 below illustrates the sharp VLC table using a modified generalized Exp-Golomb code with k = 0, where the first 15 entries are modified to better fit the experimentally observed statistics: m v m Exp-Golomb code Bit length 0 0 1 1 1 1 011 3 -1 2 010 3 2 3 0011 4 -2 4 0010 4 3 5 000111 6 -3 6 000110 6 4 7 0001011 7 -4 8 0001010 7 5 9 0001001 7 -5 10 0001000 7 6 11 0000111 7 -6 12 0000110 7 7 13 00001011 8 -7 14 00001010 8 ... ... Exp-Golomb (k = 0) ... Table 6 below illustrates the mean VLC table using a modified generalized Exp-Golomb code with k = 2, where the first 30 entries are modified to better fit the experimentally observed statistics: m v m VLC code Bit length m v m VLC code Bit length 0 0 111 3 8 15 001111 6 1 1 1101 4 -8 16 001110 6 -1 2 1100 4 9 17 001101 6 2 3 1011 4 -9 18 001100 6 -2 4 1010 4 10 19 0001111 7 3 5 1001 4 -10 20 0001110 7 -3 6 1000 4 11 21 0001101 7 4 7 01111 5 -11 22 0001100 7 -4 8 01110 5 12 23 0001011 7 5 9 01101 5 -12 24 0001010 7 -5 10 01100 5 13 25 00010011 8 6 11 01011 5 -13 26 00010010 8 -6 12 01010 5 14 27 00010001 8 7 13 01001 5 -14 28 00010000 8 -7 14 01000 5 15 29 00001111 8 ... ... Exp-Golomb(k = 2) ... Table 7 below illustrates the flat VLC table using a modified generalized Exp-Golomb code with k = 5, where the first 40 entries are modified to better fit the experimentally observed statistics: m v m VLC code Bit length m v m VLC code Bit length 0 0 11111 5 -10 20 101010 6 1 1 111101 6 11 21 101001 6 -1 2 111100 6 -11 22 101000 6 2 3 111011 6 12 23 100111 6 -2 4 111010 6 -12 24 100110 6 3 5 111001 6 13 25 100101 6 -3 6 111000 6 -13 26 100100 6 4 7 110111 6 14 27 100011 6 -4 8 110110 6 -14 28 100010 6 5 9 110101 6 15 29 1000011 7 -5 10 110100 6 -15 30 1000010 7 6 11 110011 6 16 31 1000001 7 -6 12 110010 6 -16 32 1000000 7 7 13 110001 6 17 33 0111111 7 -7 14 110000 6 -17 34 0111110 7 8 15 101111 6 18 35 01111011 8 -8 16 101110 6 -18 36 01111010 8 9 17 101101 6 19 37 01111001 8 -9 18 101100 6 -19 38 01111000 8 10 19 101011 6 20 39 01110111 8 ... ... Exp-Golomb(k = 5) ...

[0290] Fig. 39A-39D illustrate a region-based motion estimation and compensation process 3900 arranged in accordance with at least some implementations of the present disclosure. In various implementations, the process 3900 may generally be performed across one or more components of the region-based motion analyzer system 100 (e.g., the region-based motion analyzer system 100 of Fig. 1 and / or Fig. 3), which has already been discussed, should be implemented.

[0291] If in operation 3902 "ld = (H < 300) and (W < 600) and (WH < 1800)" a frame width is less than 600, a frame height is less than 300, and the product of the frame width and height is less than 180,000, the sequence is classified as low-resolution via the low-resolution flag (ld). If H < 300 and W < 600 and W × H < 180,000, then ld = 1; otherwise, ld = 0.

[0292] In operation 3904 “i = 0”.

[0293] In operation 3906 "scf = advanced scene change detection (SCD) of frame f", scene change detection may be performed to set the scene change flag (scf).

[0294] In operation 3908 "SF = subsampled frame F from YUV420 to YUV444 by 4 in each dir. For Y and by 2 in each dir. For U and V", subsampling converts the input YUV420 frame into a block-accurate YUV444 frame, where the luminance signal is subsampled by 4 (e.g., 4 × 4 block accuracy) while the chrominance signal is subsampled by 2 (e.g., 2 × 2 block accuracy).

[0295] In operation 3910 “scf = 1”, the scene change flag (scf) = 1 indicates that a scene change has been detected, while the scene change flag (scf) = 0 indicates that no scene change has been detected.

[0296] If operation 3910 is satisfied (e.g., a scene change has been detected), in operation 3912 "Reset the initial motion vectors for motion estimation to 0; clear the previous entries of the memory buffers BF, BP, and the codebook CB," the initial motion vectors for motion estimation may be reset to zero and the memory buffers BF, BP, and the codebook CB may be cleared of the previous entries.

[0297] If operation 3910 is not satisfied (e.g., no scene change has been detected), block motion estimation between the current frame F and the reference frame Fref may be performed in operation 3914 "Perform motion estimation (ME) using the current frame F and the reference frame Fref, which depends on the GOP used; output both estimated 8 × 8 and estimated 16 × 16 motion vector fields (MVFs)".

[0298] In operation 3916 “ld = 1”, a determination may be made as to whether the current frame has low resolution, where the low resolution flag (ld) = 1 indicates low resolution.

[0299] If operation 3916 is satisfied (e.g., the current frame F has low resolution), in operation 3918 "Remove the isolated MVs from the 8 × 8 and 16 × 16 MVFs and merge 4 8 × 8 MVs from the 8 × 8 MVF into a single 16 × 16 MV from the 16 × 16 MVF if the SAD is up to 1% higher" the primary isolated MVs from 8 × 8 may be removed.

[0300] In operation 3920 “MVs = filtered and merged 8 × 8 MVF; W B = W / 8, H B = H / 8, B = 8" the remaining motion vectors can be filtered and merged.

[0301] If operation 3916 is not satisfied (e.g., the current frame F is not low resolution), the primary isolated MVs may be removed from 16×16 in operation 3922 “Remove the isolated MVs from the 16×16 MVF”.

[0302] In operation 3924 “MVs = filtered 16 × 16 MVF; W B = W / 16, H B = H / 16, B = 16" the remaining motion vectors can be filtered and merged.

[0303] In operation 3926 “The random sampling of 3 MVs (W B H B-times) and collect a histogram of the corresponding affine model parameters. Detect the peaks and set the initial affine model iaff' to the midpoint of the peak regions. Repeated random sampling of three motion vectors can be performed at once to calculate the affine model parameters. For each parameter, a histogram can be used to detect a peak to set the initial affine model iaff' to a midpoint of the peak region.

[0304] In operation 3928 “set iaff to either iaff' or to one of up to 2 prior affine parameters from the memory buffer BP according to the minimum subsampled SAD (SSAD)”, both two prior affine models of the motion of two prior frames and the initial affine model iaff' are used to select the best initial affine model iaff.

[0305] In operation 3930, "Generate 7 binary candidate motion vector selection masks using iaaf, morphological operators, and RsCs texture measures to select the blocks whose MVs are to be included in the final GMM estimation; select one with minimal SAD," multiple binary candidate motion vector selection masks can be generated based on the best initial affine model iaff. A best candidate motion vector selection mask with minimal error can be selected from the binary candidate motion vector selection mask.

[0306] The term “RsCs” as used here is defined as the square root of the square of the average row differences and the square of the average column differences over a given block of pixels.

[0307] In operation 3932 "Recompute the iaff model using least squares fitting by selecting the MVs that match the selection mask", the best initial affine model iaff can be recomputed based on the best selection mask.

[0308] In Operation 3934 “A global W B × H BCalculate the motion vector field GMVF (by applying the iaff model to the block centers) and then calculate the differences (the SAD of the MV coordinates) between the GMVF and the MVs; calculate the binarization threshold and apply it to the differences to calculate the two-level classification mask BGMP' of the global motion region in F. The previously calculated affine GMM for segmentation can be used to calculate the global motion vector field, denoted by GMVF. This field can be calculated by applying the GMM's affine parameter equation to the block position center (e.g., using the same block size as in the block-based mvs field). Then, the differences between the GMVF and the mvs can be calculated and scaled in the range of 0-255, generating the so-called global motion probability map (GMP map).The GMP map can then be binarized into a binary mask denoted by BGMP' via the 2-level global motion probability classifier 2108 using the calculated threshold T_m (generated via the binarization threshold estimator 2106).

[0309] In operation 3936, "Calculate the dominant color using the BFMP range" with low RsCs texture of the global motion range of the BGMP'; if the percentage of overlapping blocks between the BGMP' and the BFMP' is high, set use_col = 1, otherwise set use_col = 0. A masked color histogram can be calculated using the BGMP' to mask only the globally moving blocks. The histogram peaks (col_hist) can be determined, and a corresponding dominant color (dom_col) can be generated. Using the dominant color and the subsampled YUV444 frame SSF with the set resolution, the color differences can be calculated and scaled in the range 0-255 (the DCP map).The DCP map can be used together with the RsCs(F) and BGMP' masks to calculate the percentage of low-textured blocks in dominant color in the background motion area, represented as a color-aided binary mask BGMP. An analysis can be performed to determine if the percentage of these blocks is high enough (in some implementations, this percentage threshold may be 85% or more of the background motion blocks from the BGMP', for example), in which case the use_col control signal can be set to 1. Otherwise, the use_col control signal can be set to 0.

[0310] In operation 3938 “use col = 1” a determination may be made as to whether the use_col signal is 1 or 0.

[0311] In operation 3940 "BGMP = BGMP'", if the use_col signal is 1, then the color-assisted BGMP can be output as the final binary mask of the background motion area (BGMP).

[0312] In operation 3942 "BGMP = BGMP", if the use_col signal is 0, then the BGMP mask can be output as the BGMP.

[0313] In operation 3944, "Add the background region to the regions. Within the foreground motion region of the BGMP, calculate the dominant MV, generate the differences with the MVs, and binarize them into the mask BDMP using the calculated threshold. If the connected region is significant, add a new foreground region to the regions. Repeat the same process for the 0-value region of the BDMPs in a cascade, where, if the resulting connected region is significant, add the 2nd foreground region to the regions.", the BGMP mask defining the background motion region can be inverted so that the remaining non-background region is turned on (e.g., the bitmask has a value of 1). Then, using the inverted mask iBGMP, a masked histogram of the motion vectors for the frame can be calculated using the block-based motion vectors mvs.The histogram denoted by mv_hist can be analyzed, with peaks being selected to obtain the dominant motion vector within the foreground motion region. The mvs of the motion vector field can then be discriminated against the dominant motion vector, and the results can be scaled to the range 0-255 in the dominant color probability map (DMP map). A binarization threshold can be estimated for the resulting DMP map, and the map can be binarized into the binary 2-level mask BDMP. Next, segment solidity and size analysis can be performed to determine whether the new foreground region defined by the BDMP is significant or not. If it is significant, the control signal add_reg is set to 1 (otherwise, it is set to 0).If add_reg is 0, then there are no foreground regions, and the resulting region mask is generated via the motion region mask generator 2216 with only 1-2 regions (as defined by the BGMP). Otherwise, the region mask is generated with only 2-3 regions (as defined by the BGMP and BDMP masks).

[0314] In operation 3946 "Apply the morphological operators (open + close), the removal of small segments and the smoothing filter to the regions", all regions are segmented, and the mask of the unprocessed regions can be post-processed to reduce the segmentation noise and make the mask of the unprocessed regions more solid.

[0315] In operation 3948 "Operating Mode = 0," a determination regarding an operating mode can be made. Operating Mode 0 (the default operating mode) is an operating mode designed for sequences with normal motion complexity. Operating Mode 1 is an operating mode designed for sequences with complex motion (e.g., sequences with high perspective depth, fast motion, etc.).

[0316] If operation 3948 is satisfied, in operation 3950 "For each region, compute the translational 4-parameter, affine 6-parameter, and pseudo-perspective 8-parameter models using the MVs of the given region," when operating in mode 0 (the default mode), process 3900 may adaptively switch on a per-region basis between the translational 4-parameter, affine 6-parameter, and pseudo-perspective 8-parameter models of global motion.

[0317] If operation 3948 is not satisfied, in operation 3952 "For each region, compute the affine 6-parameter, the pseudo-perspective 8-parameter, and the biquadratic 12-parameter models using the MVs of the given region," when operating in mode 1, process 3900 may adaptively switch on a region-by-region basis between the affine 6-parameter, the pseudo-perspective 8-parameter, and the biquadratic 12-parameter models of global motion.

[0318] In operation 3954 "For each region, choose the model with the smallest SSAD (allowing a higher-order model with up to 1% higher SSAD tolerance)", the final parameters of the global motion model can be selected based on the smallest subsampled error.

[0319] In operation 3956 "In each region, apply its rmm model to the subsampled reference frame SFref with 4 different subpixel interpolation filters: (1) a (bilinear) 1 / 16-pixel soft filter, (2) a (bicubic) medium 1 / 16-pixel filter, (3) a medium-sharp 1 / 8-pixel filter, and (4) a sharp 1 / 8-pixel filter; within each region, calculate four corresponding SSADs with respect to the subsampled current frame SF; set flt to the filters of the regions that have the smallest SSAD", within each region, the recalculated best initial affine model iaff can be applied to a subsampled reference frame SFref with several different subpixel interpolation filters to select the filter for each region that has the smallest error.

[0320] In operation 3958, "For each region, set ref_pts_mvs to the motion vectors at the frame reference points obtained with the rmms and reconstruct the global motion model from the ref_prs_mvs, resulting in a quantized model rmm_rec," the final parameters of the region-based motion model can be applied to the frame-based reference points for each region to form the motion vectors ref_pts_mvs of the reference points. The calculated motion vectors ref_pts_mvs can be quantized, for example, to 1 / 4 pixel accuracy. Next, the reconstructed parameters rmm_rec can be generated from the motion vectors ref_pts_mvs of the reference points. The reconstructed parameters rmm_rec can be obtained by solving the system of equations for the motion vectors at the reference points.

[0321] In operation 3960, "Apply rmm_rec to Fref according to the region mask to generate the prediction frame PF, and calculate and output the final SAD from the PF and F using the subpixel interpolation filter flt," the reconstructed rmm_rec parameters may be applied to the reference frame Fref according to a region-by-region basis (e.g., via the region mask) to generate the prediction frame PF. The prediction frame PF may be generated by applying the reconstructed rmm_rec parameters to the pixels of the reference frame Fref, where the subpixel positions may be interpolated using the previously selected filter filt.

[0322] In operation 3962 “set fd and dir to the frame distance and the direction of prediction between the frames F and Fref”, a frame distance fd (the distance between the current frame F and the reference frame Fref) and a direction dir can be set by the frame distance and the direction of prediction used in estimating the model between the current frame F and the reference frame Fref.

[0323] In operation 3964 "Set r = 0 and Nr = the number of ranges in the range mask", an incremental range counter is set to zero and a number-of-ranges flag is set based on the calculated range mask.

[0324] In operation 3966, "set latest [r] to the latest model from CB[r], scale it according to fd and dir, and convert it to a number rmm[r] of parameters; calculate the residuals between latest [r] and ref_pts_mvs[r], and encode the residuals into coded bits bits0 [r] (amounting to b0 [r] bits) using adaptive modified Exp-Golomb encoders," for each region, the latest model can be selected from the at least one codebook (CB) of the region and then scaled according to the fd and dir values ​​to correspond to the distance ref_pts_mvs [r] and the direction of that region. In addition to scaling, the model of that region can be converted to correspond to the number of points in the current model.In the case where the current model has more points than the latest[r] model of the domain, the model can be reconstructed, and the missing MVs of the additional points can be calculated and added to the MVs of the points of the latest model. The resulting predicted points are differentiated using the domain-specific ref_pts_mvs[r] to generate the domain's residuals, which are then encoded with the modified Golomb code.

[0325] In operation 3968 "Set the scaled_adjusted_ref_pts_mvs[r] to the most exact ref_pts_mvs[r] match between the scaled (with respect to fd and dir) codewords of the CB[r] and set the exact_match to 1 if scaled_adjusted_ref_pts_mvs[r] = ref_pts_mvs[r] and to 0 otherwise", for each domain, the calculated points ref_pts_mvs[r] of that domain can be compared with the points from the corresponding codebook of that domain using a matcher to find the corresponding points from the domain-specific codebook (CB[r]). Before comparison, the points from the domain-specific codebook can be scaled according to the fd and dir values ​​to obtain the scaled fitted reference points scaled_fitted_ref_pts_mvs [r] of that domain.If the calculated points ref_pts_mvs [r] of this domain match an entry in the domain-specific codebook (CB[r]), the control signal exact_match is set to 1. Otherwise, exact_match is set to 0.

[0326] In operation 3970 "exact_match = 1", a determination may be made as to whether the exact_match control signal is set to 1 for an exact match or to 0 for no exact match.

[0327] If operation 3970 is not satisfied (e.g., no exact match), in operation 3972, "Compute the residuals between the scaled_matched_ref_pts_mvs[r] and the ref_pts_mvs[r] and encode the residuals into coded bits1 using adaptive modified Exp-Golomb codes," the most accurate model computed by the adjuster, denoted by scaled_matched_ref_pts_mvs[r], can be used for each region to compute the residuals with the ref_pts_mvs[r]. The residuals are computed and encoded using the modified adaptive Golomb codes. The codebook index bits and the remaining bits are combined into a second set of coded bits.

[0328] If operation 3970 is satisfied (e.g., an exact match), in operation 3974 "Encode the index of the scaled_adjusted_ref_pts_mvs[r] in the CB and prepend it to bits1 (amounting to b1 bits)" for each range, if the ref_pts_mvs[r] match an entry in the codebook CB[r], the control signal exact_match is set to 1, and process 3900 outputs the bits for the codebook index as the 2nd set of encoded bits.

[0329] In operation 3976 "Encode the index of the scaled_adjusted_ref_pts_mvs [r] in the CB[r] and prepend it to bits1 (which amounts to b1 bits)", for each range an index of the scaled_adjusted_ref_pts_mvs [r] is encoded in the CB[r] and prepend it to bits1.

[0330] In operation 3978 “b0 < b2” the bits b0 from operation 3966 are compared with the bits b1 from operation 3974 or 3976.

[0331] If operation 3978 is satisfied (e.g., bits b0 from operation 3966 are less than bits b1 from operation 3974 or 3976), in operation 3980, "Append the bits0 to the bits," the bits bits are appended to the running count of bits to be output for each range. The final output bits contain all of the final encoded bits of the individual range, all appended to form the frame-based final encoded bits.

[0332] If operation 3978 is not satisfied (e.g., bits b0 from operation 3966 are not less than bits b1 from operation 3974 or 3976), operation 3982, "Append bits1 to bits," appends bits1 to the running count of bits to be output for each range. The final output bits contain all of the final encoded bits of the individual range, all appended to form the frame-based final encoded bits.

[0333] In operation 3984 “r < Nr - 1” a determination may be made as to whether the counter r has finished counting all areas of the current frame.

[0334] If operation 3984 is satisfied, process 3900 iterates and increments counter r by one in operation 3986 "r = r + 1", reading the next range.

[0335] If operation 3984 is not satisfied, the final running count of bits to be output by the iterations of operation 3980 and / or 3982 is output in operation 3988. As stated above, the final output bits include all of the final encoded bits of the single range appended to form the frame-based final encoded bits.

[0336] In operation 3990 “i < N - 1” a determination may be made as to whether counter i is complete.

[0337] If operation 3990 is satisfied, in operation 3992 "i = i + 1; read the next frame F" the process 3900 iterates and increments the counter i by one, reading the next frame.

[0338] If operation 3990 is not satisfied, then process 3900 is terminated.

[0339] Fig.39 shows a high-level process 3900 of the domain-based motion analyzer system 100 (e.g., the domain-based motion analyzer system 100 of Fig. 1 and / or Fig. 3).

[0340] The embodiments of method 2900 (and the other methods herein) may be implemented in a system, apparatus, processor, reconfigurable device, etc., such as those described herein. More specifically, the hardware implementations of method 2900 may include configurable logic, such as PLAs, FPGAs, CPLDs, or fixed-functionality logic hardware, using circuit technology, such as ASIC, CMOS, or TTL technology, or any combination thereof. Alternatively or additionally, method 2900 may be implemented in one or more modules as a set of logic instructions stored in a machine- or computer-readable storage medium, such as RAM, ROM, PROM, firmware, flash memory, etc., for execution by a processor or computing device.The computer program code to perform the operations of the components may, for example, be written in any combination of one or more applicable / suitable OS programming languages, including an object-oriented programming language such as PYTHON, PERL, JAVA, SMALLTALK, C++, C#, or the like, and conventional procedural programming languages ​​such as the "C" programming language, or similar programming languages.

[0341] The embodiments or portions of method 2900 (and other methods herein) may be implemented in applications (e.g., through an application programming interface / API) or driver software running in an OS. Additionally, the logic instructions may include assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, state setting data, integrated circuit configuration data, state information that personalizes electronic circuitry, and / or other structural components native to the hardware (e.g., a host processor, central processing unit / CPU, microcontroller, etc.). The pre-segmentation

[0342] Pre-segmentation of a video sequence can be viewed as a coarse segmentation of a scene into regions based on some common feature, which may be scene-dependent. The common feature can be, for example, a color (or, in practice, a narrow band of colors) and / or a motion (or, in practice, a narrow band of motion parameters). The goal of pre-segmentation is often to partition a video sequence into global and local regions on a frame-by-frame basis. The consistency and coherence

[0343] The segmentation of each frame of the video sequence can be performed into three or more regions that are not only spatially and temporally consistent, but also semantically coherent. For example, to generate three regions, starting with a two-region segmentation, the foreground region can be segmented into two regions, resulting in the background region, a foreground region #1, and a foreground region #2. Furthermore, in an exemplary case of four regions, an additional foreground region #3 can be used. Similarly, if necessary, instead of the initial segmentation (e.g., pre-segmentation) of the foreground region, the background region can be divided as needed. The spatial consistency

[0344] For a general class of video sequence subjected to frame-wise segmentation into regions, spatial consistency can be defined as the ability to roughly segment the same spatial object, such as having almost the same shape and almost the same size. The temporal consistency

[0345] For a general class of video sequences subjected to frame-wise segmentation into regions, temporal consistency can be defined as the ability to roughly segment the same temporal object, such as having almost the same location (except for motion) and almost the same motion trajectory. The semantic coherence

[0346] For a restricted class of video sequences subjected to frame-wise segmentation into regions, semantic coherence can be defined as the segmented region having approximately the same shape, size, location, and / or trajectory, where it can be considered to represent the background, while the other region(s) can be considered to be the foreground region(s), such as foreground region #1, foreground region #2, etc. The coding of the explicit domain boundary form

[0347] In region-based video coding, which operates on frame-segmented regions, there is often a need to efficiently identify a region(s) across the encoded bitstream to the decoder. One way to accomplish this identification can be, for example, by the video encoder explicitly encoding the region boundary information (typically requiring a one-fold smaller number of boundary region shapes compared to the total number of regions). To reduce the encoding cost of the region boundary information, some implementations may use reduced precision, such as 4-pixel, 8-pixel, or even 16-pixel precision.Furthermore, the MPEG-4 Part 2 standard can provide an efficient method for region boundary coding that uses context information from prior neighbors as well as temporal prediction and arithmetic coding. Besides MPEG-4, other region shape coding techniques exist that may be simpler but also less efficient. The implicit range representation

[0348] The region boundary information can be expensive in bits depending on the accuracy with which it is transmitted. If region-based motion compensation is used in video coding, an alternative way of achieving the same goal (e.g., being able to identify which block belongs to which region) can be achieved by extending the coding of a mode table of the standard (e.g., the AVC standard, the HEVC standard, or the like), which might, for example, typically contain modes such as a skip mode, an intermediate mode, and / or an intra mode, to also be able to indicate to which region (e.g., the 'skip region 1' block or the 'intermediate region 1') a given coding block is assigned.The coding modes can typically be encoded very efficiently using arithmetic coding, therefore the shape information can be represented efficiently.

[0349] The embodiments of method 3900 (and the other methods herein) may be implemented in a system, apparatus, processor, reconfigurable device, etc., e.g., such as those described herein. More specifically, the hardware implementations of method 3900 may include configurable logic, such as PLAs, FPGAs, CPLDs, or fixed-functionality logic hardware, using circuit technology, such as ASIC, CMOS, or TTL technology, or any combination thereof. Alternatively or additionally, method 3900 may be implemented in one or more modules as a set of logic instructions stored in a machine- or computer-readable storage medium, such as RAM, ROM, PROM, firmware, flash memory, etc., for execution by a processor or computing device. The computer program code to perform the operations of the components may, for example,B. be written in any combination of one or more applicable / suitable OS programming languages, including an object-oriented programming language such as PYTHON, PERL, JAVA, SMALLTALK, C++, C# or the like, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0350] The embodiments or portions of method 3900 (and other methods herein) may be implemented in applications (e.g., through an application programming interface / API) or driver software running in an OS. Additionally, the logic instructions may include assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, state setting data, integrated circuit configuration data, state information that personalizes electronic circuitry, and / or other structural components native to the hardware (e.g., a host processor, central processing unit / CPU, microcontroller, etc.). The results

[0351] The results of SAD reduction and entropy coding of the GMM model parameters

[0352] An implementation was evaluated on test sets with different resolutions. The tabulated results use the following column headings: • F = the index of the current frame • R = the reference frame index • Ref-SAD = the 8 × 8 block-based SAD (the reference SAD) • RMM-SAD = is the SAD of a full frame of area-based movement • NBB = refers to the number of 16×16 blocks in the frame whose GMM SAD is better than or equal to the co-located Ref SAD • Bits = the total number of bits per frame spent encoding the GMM parameters and encoding the heads that signal the selected model and sub-pixel filter. SP Filter = denotes the selected sub-pixel filter (the values ​​are “1 / 16 BIL” = 1 / 16-pixel bilinear filter, “1 / 16 BIC” = 1 / 16-pixel bicubic filter, “1 / 8 AVC” = 1 / 8-pixel AVC-based filter, and “1 / 8 HEVC” = 1 / 8-pixel HEVC-based filter) • Mod = is the selected model of global motion (the values ​​are “4par” = translational 4-parameter model of global motion, “6-par” = affine 6-parameter model of global motion and “8par” = pseudo-perspective 8-parameter model of global motion) • RMM parameters = the final coefficients of the parameters of the area-based motion model

[0353] The reduction of the average SAD for low-delay IPP images Table 8 below illustrates the average SAD results of the RMM for CIF sequences (33 frames) with low-delay IPP images: sequence Ref-SAD (33-frame average) RMM-SAD (33-frame average) NBB (33-frame average) Bits (33-frame average) bus 389601 561486 146 78 City 210014 212870 227 70 flower 444764 636215 61 127 Stefan 614102 947909 60 99 mobile 503788 652820 114 43 football 349557 751048 53 90 foreman 213772 391640 108 69 Harbor 481541 514964 163 16 football 287422 672652 112 68 tennis 286460 483821 172 78 Tennis 2 352610 526263 194 54 coast 431386 534679 146 66 Table 9. The RMM SAD results for the CIF “bus” sequence (33 frames) with low-delay IPP images: F R Ref-SAD RMMs-SAD NBB Bits SP filter Mod RMMs parameters 1 0 450196 682912 166 120 a0 = 1,005859, a1 = 0.0, a2 = -4.5, a3 = 0,0, 1 / 16 a4 = 1,005371, 305098 311388 163 50 BIC 6par a5 = -0,75 a0 = 1,003418, a1 = 0,107422, a2 = -16,0, 1 / 16 <h2 style=";text-align:left;direction:ltr">a3 = 0,0, a4 = 1,0, 134422 357408 3 46 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 6par a5 = -0,25 a0 = 0,992188, a1 = 2,25, 1 / 16 a2 = 1,000977, 10676 14116 0 18 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 4par a3 = -0,25 2 1 456828 725190 88 75 a0 = 1,004883, a1 = -4,5, 1 / 8 a2 = 1,005859, 304524 323713 84 26 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 4par a3 = -1,0 a0 = 1,003418, a1 = 0,110352, a2 = -16,25, a3 = 0,000488, 1 / 16 a4 = 1,010254, 141763 386342 4 29 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 6par a5 = -2,0 a0 = 0,992676, a1 = 2,0, 1 / 16 a2 = 1,001953, 10541 15135 0 18 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 4par a3 = -0,5 3 2 449217 747536 142 92 a0 = 1,004883, <h2 style=";text-align:left;direction:ltr"> a1 = 0,0, a2 = -4,5, a3 = 0,0, 1 / 8 a4 = 1,005371, 310926 316729 141 27 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par a5 = -0,75 1 / 16 a0 = 0,998535, 126635 415500 1 19 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a1 = 0,0, a2 = 1,005859, a3 = -1,0 a0 = 0,992676, a1 = -0,009766, a2 = 4,5, a3 = 0,001465, 1 / 16 a4 = 1,002441, 11656 15307 0 44 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 6par a5 = -1,0 4 3 439079 575033 121 69 a0 = 1,004883, a1 = -0,000977, <h2 style=";text-align:left;direction:ltr"> a2 = -4.5, a3 = 0.0, 1 / 8 a4 = 1,004395, 310122 326171 113 26 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par a5 = -0,5 a0 = 1,004395, a1 = 0,25, 1 / 16 a2 = 1,002441, 116698 232015 8 24 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = -0,5 a0 = 0,991699, a1 = 2,5, 1 / 16 a2 = 1,003418, 12259 16847 0 17 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 4par a3 = -0,75 5 4 407518 480787 134 71 a0 = 1,004395, <h2 style=";text-align:left;direction:ltr"> a1 = 0,0, a2 = -4,5, a3 = 0,0, 1 / 8 a4 = 1,003418, 304139 327736 127 14 <h2 style=";text-align:left;direction:ltr"> AVC <h2 style=";text-align:left;direction:ltr"> 6par a5 = -0,5 a0 = 1,004395, a1 = 0,015625, a2 = -1,75, a3 = 0,0, 1 / 16 a4 = 0,997559, 90701 137328 7 39 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 6par a5 = 0,25 a0 = 0,992188, a1 = 2,25, 1 / 16 a2 = 1,001953, 12678 15723 0 16 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 4par a3 = -0,5 6 5 385958 504500 150 75 a0 = 1,00293, a1 = -4,25, 1 / 8 a2 = 1,001953, 259418 266689 146 25 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 4par a3 = -0,25 a0 = 0,998047, a1 = -0,026855, <h2 style=";text-align:left;direction:ltr"> a2 = 5,5, a3 = 0,0, 1 / 16 a4 = 1,000977, 113090 220025 4 41 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 6par a5 = -0,25 a0 = 0,992676, a1 = 2,0, 1 / 16 a2 = 1,001953, 13450 17786 0 7 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 4par a3 = -0,5 7 6 368091 468537 125 83 a0 = 1,001465, a1 = -0,000977, a2 = -4,0, a3 = -0,000488, 1 / 8 a4 = 1,000977, 245682 260004 122 24 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par a5 = 0,0 a0 = 1,001465, a1 = -0,002441, <h2 style=";text-align:left;direction:ltr"> a2 = 1.25, a3 = 0.0, 1 / 16 a4 = 0,999023, 109219 188226 3 29 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 6par a5 = 0,0 a0 = 1,004395, a1 = -0,006836, a2 = 0,25, 1 / 16 a3 = -0,000488, 13190 20307 0 28 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 6par <h2 style=";text-align:left;direction:ltr"> a4 = 1.0, a5 = 0.25 8 7 338548 412994 159 71 <h2 style=";text-align:left;direction:ltr"> a0 = 1,0, a1 = 0,0, 1 / 16 <h2 style=";text-align:left;direction:ltr"> a2 = -4,0, a3 = 0,0, 210334 236030 155 24 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 6par <h2 style=";text-align:left;direction:ltr"> a4 = 1,0, a5 = 0,0 a0 = 0,999512, 1 / 8 <h2 style=";text-align:left;direction:ltr"> a1 = 1.25, a2 = 1.0, 114898 150958 4 21 <h2 style=";text-align:left;direction:ltr"> AVC <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0, 0 a0 = 1,000488, a1 = -0,013184, a2 = 2,75, 1 / 16 a3 = -0,000488, 13316 26006 0 24 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 6par <h2 style=";text-align:left;direction:ltr"> a4 = 1.0, a5 = 0.25 9 8 336075 381676 209 51 <h2 style=";text-align:left;direction:ltr"> a0 = 1,0, a1 = 0,0, 1 / 16 <h2 style=";text-align:left;direction:ltr"> a2 = -4,0, a3 = 0,0, 215560 230083 196 6 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 6par <h2 style=";text-align:left;direction:ltr"> a4 = 1,0, a5 = 0,0 a0 = 0,998535, a1 = 1,5, 1 / 8 a2 = 0,995605, 105785 129085 13 21 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0, 5 a0 = 1,0, a1 = -0,25, 1 / 16 a2 = 0,999023, 14730 22508 0 22 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0,25 10 9 329192 424591 154 63 a0 = 1,000488, <h2 style=";text-align:left;direction:ltr"> a1 = 0,0, a2 = -4,0, a3 = 0,0, 1 / 16 a4 = 0,999023, 215718 237877 147 22 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 6par a5 = 0,25 a0 = 1,001953, 1 / 8 <h2 style=";text-align:left;direction:ltr"> a1 = 0.75, a2 = 1.0, 98474 163988 7 19 <h2 style=";text-align:left;direction:ltr"> AVC <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0, 0 a0 = 1,007813, 1 / 16 <h2 style=";text-align:left;direction:ltr"> a1 = -2.5, a2 = 1.0, 15000 22726 0 20 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0, 0 11 10 319843 397847 233 59 1 / 8 <h2 style=";text-align:left;direction:ltr"> a0 = 1,0, a1 = -4,0, 204049 220991 231 16 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 4par <h2 style=";text-align:left;direction:ltr"> a2 = 1,0, a3 = 0,0 a0 = 1,0, a1 = 0,004395, <h2 style=";text-align:left;direction:ltr"> a2 = 0.5, a3 = 0.0, 1 / 8 a4 = 0,994629, 100106 153241 2 34 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par a5 = 0,5 a0 = 1,0, a1 = -0,25, 1 / 16 a2 = 0,999023, 15688 23615 0 7 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0,25 12 11 339549 434032 200 58 <h2 style=";text-align:left;direction:ltr"> a0 = 1,0, a1 = 0,0, 1 / 8 <h2 style=";text-align:left;direction:ltr"> a2 = -4,0, a3 = 0,0, 201835 220878 195 7 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par <h2 style=";text-align:left;direction:ltr"> a4 = 1,0, a5 = 0,0 a0 = 1,0, a1 = -0,002441, a2 = 1,25, a3 = 0,000488, 1 / 16 a4 = 0,996582, 120399 192705 5 27 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 6par a5 = 0,25 a0 = 0,995605, a1 = 1,25, 1 / 16 a2 = 1,007813, 17315 20449 0 22 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 4par a3 = -1,75 13 12 363667 469814 168 67 a0 = 1,0, a1 = -0,000977, 1 / 16 <h2 style=";text-align:left;direction:ltr"> a2 = -4,0, a3 = 0,0, 225470 228733 165 20 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 6par <h2 style=";text-align:left;direction:ltr"> a4 = 1,0, a5 = 0,0 a0 = 0,999512, a1 = 0,75, 1 / 16 a2 = 0,997559, 121721 220674 3 22 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0,25 a0 = 0,98877, 1 / 16 <h2 style=";text-align:left;direction:ltr"> a1 = 3.25, a2 = 1.0, 16476 20407 0 23 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0, 0 14 13 367329 531075 166 43 <h2 style=";text-align:left;direction:ltr"> a0 = 1,0, a1 = 0,0, a2 = -4,25, 1 / 8 <h2 style=";text-align:left;direction:ltr"> a3 = 0,0, a4 = 1,0, 221842 235372 166 14 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par a5 = 0,0 1 / 16 a0 = 0,99707, 127359 270365 0 20 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a1 = 1,0, a2 = 0,999023, a3 = 0, 0 a0 = 1,0, a1 = -0,25, 1 / 16 a2 = 0,999023, 18128 25338 0 7 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0,25 15 14 366339 535804 196 54 <h2 style=";text-align:left;direction:ltr"> a0 = 1,0, a1 = 0,0, 1 / 8 <h2 style=";text-align:left;direction:ltr"> a2 = -4.5, a3 = 0.0, 228006 227902 185 17 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par <h2 style=";text-align:left;direction:ltr"> a4 = 1,0, a5 = 0,0 a0 = 0,995605, a1 = 1,0, 1 / 16 a2 = 0,997559, 119444 289024 2 19 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0,25 1 / 8 <h2 style=";text-align:left;direction:ltr"> a0 = 1,0, a1 = 0,0, 18889 18878 9 16 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 4par <h2 style=";text-align:left;direction:ltr"> a2 = 1,0, a3 = 0,0 16 15 376432 518436 164 62 <h2 style=";text-align:left;direction:ltr"> a0 = 1,0, a1 = 0,0, 1 / 8 <h2 style=";text-align:left;direction:ltr">a2 = -4.5, a3 = 0.0, 232008 248966 163 6 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par <h2 style=";text-align:left;direction:ltr"> a4 = 1,0, a5 = 0,0 a0 = 0,998047, a1 = 0,001953, a2 = 0,5, a3 = 0,000488, 1 / 8 a4 = 1,001953, 126729 244411 1 34 <h2 style=";text-align:left;direction:ltr"> AVC <h2 style=";text-align:left;direction:ltr"> 6par a5 = -0,5 a0 = 0,986328, a1 = 3,75, 1 / 16 a2 = 1,002441, 17695 25059 0 20 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 4par a3 = -0,75 17 16 375927 568442 166 43 <h2 style=";text-align:left;direction:ltr"> a0 = 1,0, a1 = 0,0, 1 / 8 <h2 style=";text-align:left;direction:ltr"> a2 = -4.5, a3 = 0.0, 242929 252844 165 6 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par <h2 style=";text-align:left;direction:ltr"> a4 = 1,0, a5 = 0,0 a0 = 0,998047, a1 = 0,5, 1 / 8 a2 = 0,998047, 114395 292476 0 16 <h2 style=";text-align:left;direction:ltr"> AVC <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0, 0 a0 = 0,990723, a1 = 2,5, 1 / 16 a2 = 1,000977, 18603 23122 1 19 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 4par a3 = -0,25 18 17 369877 583006 156 65 a0 = 1,0, a1 = -0,000977, 1 / 8 <h2 style=";text-align:left;direction:ltr"> a2 = -4.5, a3 = 0.0, 235293 230571 154 18 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par <h2 style=";text-align:left;direction:ltr"> a4 = 1,0, a5 = 0,0 a0 = 0,998047, a1 = 0,25, 1 / 16 a2 = 0,999023, 114169 313699 2 17 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0,0 a0 = 1,015137, a1 = 0,014648, a2 = -8,5, a3 = 0,000488, 1 / 8 a4 = 0,999023, 20415 38736 0 28 <h2 style=";text-align:left;direction:ltr"> AVC <h2 style=";text-align:left;direction:ltr"> 6par a5 = 0,0 19 18 349564 449149 165 135 <h2 style=";text-align:left;direction:ltr"> a0 = 1,0, a1 = 0,0, a2 = -4,75, 1 / 16 <h2 style=";text-align:left;direction:ltr"> a3 = 0,0, a4 = 1,0, 220761 237038 162 17 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 6par a5 = 0,0 a0 = 0,000026, a1 = -0,000171, a2 = 0,002604, a3 = 0,035645, a4 = -1,0, a5 = -0,003418, 1 / 8 a6 = 0,026594, 108350 176480 3 94 <h2 style=";text-align:left;direction:ltr"> AVC <h2 style=";text-align:left;direction:ltr"> 8par a7 = -1,0 a0 = 1,007324, a1 = -2,5, 1 / 16 a2 = 1,000977, 20453 35631 0 22 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0,0 20 19 363053 514003 181 60 a0 = 1,0, a1 = -0,000977, a2 = -5,0, 1 / 16 a3 = 0,000488, 227443 233953 180 20 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 6par <h2 style=";text-align:left;direction:ltr"> a4 = 1,0, a5 = 0,0 a0 = 0,998535, a1 = 0,0, 1 / 16 a2 = 0,999023, 113876 234885 1 13 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0,0 a0 = 1,035645, a1 = -11,0, 1 / 16 a2 = 1,006836, 21734 45165 0 25 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = -1,75 21 20 406103 571966 112 96 a0 = 1,0, a1 = -0,000977, 1 / 8 <h2 style=";text-align:left;direction:ltr"> a2 = -5.5, a3 = 0.0, 236805 247536 106 22 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par <h2 style=";text-align:left;direction:ltr"> a4 = 1,0, a5 = 0,0 a0 = 0,995117, a1 = 0,003418, a2 = 0,0, a3 = -0,001465, 1 / 16 a4 = 1,004395, 143808 270136 6 28 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 6par a5 = -0,5 a0 = 1,05127, a1 = 0,041504, a2 = -26,0, a3 = -0,004883, 1 / 16 a4 = 1,000977, 25490 54294 0 44 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 6par a5 = 1,25 22 21 388160 538760 113 74 a0 = 0,999512, a1 = -0,000977, <h2 style=";text-align:left;direction:ltr"> a2 = -5.5, a3 = 0.0, 1 / 16 a4 = 0,999023, 239462 263599 108 25 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 6par a5 = 0,25 1 / 8 a0 = 1,001465, 123459 237084 4 24 <h2 style=";text-align:left;direction:ltr"> AVC <h2 style=";text-align:left;direction:ltr"> 4par a1 = -0,75, a2 = 1,005859, a3 = -1,0 a0 = 1,018555, 1 / 16 a1 = -5,75, 25239 38077 1 23 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par <h2 style=";text-align:left;direction:ltr"> a2 = 1,0, a3 = 0,0 23 22 370195 564436 80 83 a0 = 1,000488, a1 = 0,000977, a2 = -6,25, a3 = 0,0, 1 / 16 a4 = 0,999023, 217635 253285 78 18 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 6par a5 = 0,25 a0 = 0,998047, a1 = -0,5, 1 / 16 a2 = 1,009766, 128855 265114 2 23 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = -1,5 a0 = 1,039063, a1 = 0,048828, a2 = -23,75, a3 = -0,000488, 1 / 16 a4 = 0,999023, 23705 46037 0 40 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 6par a5 = 0,5 24 23 381250 602303 155 75 <h2 style=";text-align:left;direction:ltr"> a0 = 1,0, a1 = 0,0, a2 = -6,25, 1 / 8 <h2 style=";text-align:left;direction:ltr"> a3 = 0,0, a4 = 1,0, 184341 194029 153 25 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par a5 = 0,0 a0 = 1,001465, a1 = -1,25, 1 / 16 a2 = 1,006836, 171709 365567 2 22 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = -1,25 a0 = 1,041016, a1 = -12,0, 1 / 16 a2 = 1,010254 25200 42707 0 26 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = -2,5 25 24 427065 704422 137 151 1 / 8 a0 = 0,999512, 236415 247716 136 19 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par a1 = -0,000977, a2 = -6,25, a3 = 0,000488, a4 = 1,000977, a5 = 0,0 a0 = -0,000194, a1 = -0,000783, a2 = 0,174006, a3 = 0,100586, a4 = -19,0, a5 = 0,029297, 1 / 16 a6 = 0,268308, 165703 410306 1 103 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 8par a7 = -22,0 a0 = 1,049805, a1 = -15,0, 1 / 16 a2 = 0,998047, 24947 46400 0 27 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0,5 26 25 396717 535947 130 69 <h2 style=";text-align:left;direction:ltr"> a0 = 1,0, a1 = 0,0, a2 = -6,75, 1 / 16 a3 = 0,000488, 212510 219955 123 20 <h2 style=";text-align:left;direction:ltr"> BIC <h2 style=";text-align:left;direction:ltr"> 6par <h2 style=";text-align:left;direction:ltr"> a4 = 1,0, a5 = 0,0 a0 = 0,998047, a1 = -0,5, 1 / 16 a2 = 1,001953, 160328 270020 7 23 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = -0,25 a0 = 1,03125, 1 / 16 a1 = -9,75, 23879 45972 0 24 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par <h2 style=";text-align:left;direction:ltr"> a2 = 1,0, a3 = 0,0 27 26 410356 569587 115 121 a0 = -0,000019, a1 = 0,000006, a2 = 0,006313, a3 = -0,005371, a4 = -28,0, a5 = 0,00293, 1 / 8 a6 = 0,001578, 202047 207334 104 57 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 8par a7 = 0,0 a0 = 0,995605, a1 = -0,004395, a2 = 0,0, a3 = 0,000488 1 / 8 a4 = 1,002441, 184338 309363 11 35 <h2 style=";text-align:left;direction:ltr"> AVC <h2 style=";text-align:left;direction:ltr"> 6par a5 = -0,5 a0 = 0,98291, a1 = 4,0, 1 / 16 a2 = 0,998047, 23971 52890 0 27 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0,5 28 27 419391 701873 104 105 a0 = 1,0, a1 = -0,000977, a2 = -7,25, 1 / 8 a3 = 0,000488, 212664 221263 101 21 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par <h2 style=";text-align:left;direction:ltr"> a4 = 1,0, a5 = 0,0 a0 = 0,99707, a1 = -0,019043, a2 = 2,25, a3 = 0,000488 1 / 16 a4 = 1,008789, 183066 432744 3 38 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 6par a5 = -1,5 a0 = 1,006348, a1 = 0,058105, a2 = -16,75, a3 = -0,004395, 1 / 16 a4 = 0,996582, 23661 47866 0 44 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 6par a5 = 2,0 29 28 446769 774599 131 109 a0 = 1,0, a1 = -0,000977, a2 = -7,25, 1 / 8 <h2 style=";text-align:left;direction:ltr"> a3 = 0,0, a4 = 1,0, 233241 235823 122 27 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par a5 = 0,25 a0 = 1,00293, a1 = 0,001953, 1 / 16 a2 = -2,0, 187942 512219 5 34 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 6par a3 = 0,001465, a4 = 1,010254, a5 = -1,75 <h2 style=";text-align:left;direction:ltr"> a0 = 0,0, a1 = 0,0, <h2 style=";text-align:left;direction:ltr"> a2 = 0,0, a3 = 0,0, 1 / 8 <h2 style=";text-align:left;direction:ltr"> a4 = 0,0, a5 = 0,0, 25586 26557 4 46 <h2 style=";text-align:left;direction:ltr"> AVC <h2 style=";text-align:left;direction:ltr"> 8par <h2 style=";text-align:left;direction:ltr"> a6 = 0,0, a7 = 0,0 30 29 430705 673063 125 84 a0 = 1,0, a1 = -0,000977, a2 = -7,25, 1 / 8 a3 = -0,000488, 231657 227703 125 24 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par <h2 style=";text-align:left;direction:ltr"> a4 = 1.0, a5 = 0.25 a0 = 1,000488, a1 = -1,5, 1 / 16 a2 = 0,995605, 174786 393591 0 24 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0,5 a0 = 1,077637, a1 = -22,5, 1 / 16 a2 = 1,014648, 24262 51769 0 34 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = -3,5 31 30 414954 681224 114 52 a0 = 1,0, a1 = -0,000977, a2 = -7,25, 1 / 8 <h2 style=";text-align:left;direction:ltr"> a3 = 0,0, a4 = 1,0, 220333 237129 112 6 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par a5 = 0,25 a0 = 1,003418, a1 = -2,25, 1 / 16 a2 = 1,001953, 171238 391712 2 19 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = -0,25 a0 = 1,053223, a1 = -16,25, 1 / 16 a2 = 0,994141, 23383 52383 0 25 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = 1,5 32 31 423283 644011 103 56 a0 = 1,0, 1 / 8 a1 = -0,000977, 239873 253978 99 6 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par a2 = -7,25, <h2 style=";text-align:left;direction:ltr"> a3 = 0,0, a4 = 1,0, a5 = 0,25 a0 = 0,998535, 1 / 8 <h2 style=";text-align:left;direction:ltr"> a1 = -1,0, a2 = 1,0, 156816 332641 4 23 <h2 style=";text-align:left;direction:ltr"> AVC <h2 style=";text-align:left;direction:ltr"> 4par a3 = 0,0 a0 = 1,029785, a1 = -9,5, 1 / 16 a2 = 0,989746, 26594 57392 0 25 <h2 style=";text-align:left;direction:ltr"> BIL <h2 style=";text-align:left;direction:ltr"> 4par a3 = 2,75 Table 10. The RMM SAD results for the CIF “City” sequence (33 frames) with low-delay IPP images: F R <h2 style=";text-align:left;direction:ltr"> Ref-SAD <h2 style=";text-align:left;direction:ltr"> RMMs-SAD <h2 style=";text-align:left;direction:ltr"> NBB <h2 style=";text-align:left;direction:ltr"> Bits <h2 style=";text-align:left;direction:ltr"> SP-Filter <h2 style=";text-align:left;direction:ltr"> Mod <h2 style=";text-align:left;direction:ltr"> RMMs-Parameter 1 0 212423 192437 319 145 a0 = 0,000021, a1 = -0,000007, a2 = -0,006787, a3 = -0,005371, a4 = 9,0, a5 = -0,00293, 1 / 8 a6 = -0,001736, 158412 140102 259 69 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 8par a7 = 0,0 a0 = 0,000119, a1 = 0,000015, a2 = -0,042535, a3 = -0,002441, a4 = 5,0, a5 = -0,022949, 1 / 8 a6 = -0,02036, 54011 52335 60 70 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 8par a7 = 3,0 2 1 229550 234466 223 79 a0 = 1,0, a1 = -0,001953, a2 = 1,75, a3 = 0,000488, 1 / 8 a4 = 1,0, 169572 154724 214 42 <h2 style=";text-align:left;direction:ltr"> HEVC <h2 style=";text-align:left;direction:ltr"> 6par a5 = -0,75 a0 = 1,0, 1 / 8 a1 = 0,0, 59978 79742 9 35 <h2 style=";text-align:left;direction:ltr"> AVC <h2 style=";text-align:left;direction:ltr"> 6par = 0,0, a3 = 0,005859, a4 = 1,000977, a5 = -1,75 3 2 205169 211963 224 50 a0 = 1,0, a1 = -0,000977, a2 = 1,25, a3 = 0,0, 1 / 8 a4 = 1,0, 151965 147291 207 26 <h2 style=";text-align:left;direction:ltr"> HEVC 6par a5 = -0,75 a0 = 1,0, a1 = -0,5, 1 / 8 a2 = 0,999023, 53204 64672 17 22 HEVC 4par a3 = -0,75 4 3 209697 206665 238 89 a0 = 0,999512, a1 = -0,001953, a2 = 1,25, a3 = -0,000488, 1 / 8 a4 = 1,0, 153360 152438 178 16 HEVC 6par a5 = -0,75 a0 = -0,00004, a1 = -0,000052, a2 = 0,018782, a3 = 0,012207, a4 = -5,0, a5 = 0,01123, 1 / 8 a6 = 0,022175, 56337 54227 60 71 HEVC 8par a7 = -6,0 5 4 215696 207325 282 92 a0 = 1,0, a1 = -0,001953, a2 = 1,25, a3 = 0,0, 1 / 8 a4 = 1,0, 158210 151872 233 19 HEVC 6par a5 = -0,5 a0 = 0,000089, a1 = -0,00005, a2 = -0,014205, a3 = 0,010254, a4 = -3,0, a5 = -0,023926, 1 / 8 a6 = -0,000631, 57486 55453 49 71 HEVC 8par a7 = 1,0 6 5 219246 229817 205 45 a0 = 1,0, a1 = -0,002441, a2 = 1,0, a3 = 0,0, 1 / 8 a4 = 1,0, 165371 169497 174 18 HEVC 6par a5 = -0,5 a0 = 1,001465, a1 = 0,0, a2 = -1,25, a3 = -0,000488, 1 / 8 a4 = 1,000977, 53875 60320 31 25 HEVC 6par a5 = -0,5 7 6 208662 238344 159 102 a0 = 1,000488, a1 = -0,002441, a2 = 0,75, a3 = 0,0, 1 / 8 a4 = 1,0, 159533 186796 114 29 HEVC 6par a5 = 0,0 a0 = 0,000212, a1 = -0,00003, a2 = -0,057134, a3 = 0,0, a4 = 0,0, a5 = -0,051758, 1 / 16 a6 = -0,020597, 49129 51548 45 71 BIC 8par a7 = 7,0 8 7 200582 188583 269 75 a0 = 1,0, a1 = -0,000977, a2 = 0,25, a3 = 0,000488, 1 / 8 a4 = 1,0, 146402 135954 213 19 HEVC 6par a5 = 0,0 a0 = -0,000008, a1 = -0,000013, a2 = 0,004182, a3 = 0,000977, a4 = -6,0, a5 = 0,004395, 1 / 8 a6 = 0,002762, 54180 52629 56 54 HEVC 8par a7 = 0,0 9 8 213050 192488 300 46 a0 = 1,0, a1 = -0,002441, 1 / 8 a2 = 0,25, 157005 139785 234 16 HEVC 6par a3 = 0,001465, a4 = 1,0, a5 = 0,0 a0 = 1,000488, a1 = -0,000977, a2 = -1,75, a3 = 0,000488, 1 / 8 a4 = 1,0, 56045 52703 66 28 HEVC 6par a5 = 0,25 10 9 209863 212980 232 64 a0 = -0,0, a1 = -0,0, a2 = 0,000868, a3 = -0,006836, a4 = 1,0, a5 = 0,004395, 1 / 8 a6 = 0,00071, 155920 153900 193 43 HEVC 8par a7 = 0,0 a0 = 1,0, a1 = -1,75, 1 / 8 a2 = 1,000977, 53943 59080 39 19 HEVC 4par a3 = 0, 0 11 10 187940 199108 200 37 a0 = 0,999512, a1 = -0,001953, a2 = 0,5, a3 = 0,0, 1 / 8 a4 = 1,0, 143245 141634 184 17 HEVC 6par a5 = 0,0 a0 = 1,0, a1 = -1,5, 1 / 8 a2 = 0,999023, 44695 57474 16 18 HEVC 4par a3 = 0, 0 12 11 210591 202473 235 49 a0 = 1,0, a1 = -0,003418, a2 = 1,0, a3 = 0,001465, 1 / 8 a4 = 1,0, 159901 148231 195 20 HEVC 6par a5 = 0,0 a0 = 1,000488, a1 = -0,000977, a2 = -1,0, a3 = 0,001465, 1 / 8 a4 = 1,000977, 50690 54242 40 27 HEVC 6par a5 = -0,25 13 12 206503 211452 212 41 a0 = 1,0, a1 = -0,003418, a2 = 1,5, a3 = 0,001465, 1 / 8 a4 = 1,0, 150555 146944 189 20 HEVC 6par a5 = 0,0 a0 = 1,001465, a1 = -1,0, 1 / 16 a2 = 1,000977, 55948 64508 23 19 BIC 4par a3 = 0, 0 14 13 220126 230887 178 49 a0 = 1,0, a1 = -0,003418, a2 = 1,75, a3 = 0,001953, 1 / 8 a4 = 0,999023, 158717 162731 143 21 HEVC 6par a5 = 0,25 a0 = 1,000488, a1 = -0,000977, a2 = -0,25, a3 = 0,0, 1 / 8 a4 = 0,999023, 61409 68156 35 26 HEVC 6par a5 = 0,5 15 14 216806 214247 236 94 a0 = 1,0, a1 = -0,002441, a2 = 2,25, a3 = 0,000488, 1 / 8 a4 = 1,0, 156268 157991 174 30 HEVC 6par a5 = -0,5 a0 = -0,000028, a1 = 0,000009, a2 = 0,002131, a3 = -0,005371, a4 = 2,0, a5 = 0,017578, 1 / 8 a6 = -0,000237, 60538 56256 62 62 HEVC 8par a7 = -4, 0 16 15 204315 184048 308 99 a0 = 0,000019, a1 = -0,000007, a2 = -0,005445, a3 = -0,008789, a4 = 9,0, a5 = 0,000488, 1 / 8 a6 = 0,0, 149776 136328 230 47 HEVC 8par a7 = -4, 0 a0 = 0,000042, a1 = 0,000002, a2 = -0,016572, a3 = -0,002441, a4 = 3,0, a5 = -0,001465, 1 / 16 a6 = -0,008207, 54539 47720 78 50 BIC 8par a7 = -3, 0 17 16 220697 235234 184 58 a0 = 1,0, a1 = -0,004395, a2 = 2,0, a3 = 0,001465, 1 / 8 a4 = 1,000977, 163720 180242 138 29 HEVC 6par a5 = -0,75 a0 = 0,999512, a1 = -0,000977, a2 = 0,0, a3 = 0,000488, 1 / 8 a4 = 1,0, 56977 54992 46 27 HEVC 6par a5 = -0,5 18 17 212522 208490 241 49 a0 = 1,0, a1 = -0,003418, a2 = 1,5, a3 = 0,001953, 1 / 8 a4 = 1,0, 157953 155194 186 22 HEVC 6par a5 = -0,75 a0 = 1,0, a1 = -0,000977, a2 = -0,5, a3 = 0,001953, 1 / 8 a4 = 1,0, 54569 53296 55 25 HEVC 6par a5 = -0,75 19 18 202083 210725 179 83 a0 = 1,0, a1 = -0,002441, a2 = 1,25, a3 = 0,001465, 1 / 8 a4 = 1,0, 150404 157936 127 23 HEVC 6par a5 = -0,5 a0 = 0,000063, a1 = 0,000018, a2 = -0,026752, a3 = -0,005371, 1 / 8 a4 = 0,0, 51679 52789 52 58 HEVC 8par a5 = -0,004395, a6 = -0,015388, a7 = 0,0 20 19 228566 222360 263 78 a0 = 1,0, a1 = -0,001953, a2 = 1,5, a3 = 0,0, 1 / 8 a4 = 1,000977, 164540 155123 209 23 HEVC 6par a5 = -0,75 a0 = 0,000062, a1 = 0,000025, a2 = -0,016651, a3 = -0,000977, a4 = -1,0, a5 = -0,01709, 1 / 8 a6 = -0,022333, 64026 67237 54 53 HEVC 8par a7 = 2,0 21 20 214900 231231 183 94 a0 = 1,0, a1 = -0,003418, a2 = 2,0, a3 = 0,001465, 1 / 8 a4 = 1,0, 160630 167022 147 22 HEVC 6par a5 = -0,5 a0 = 0,000284, a1 = -0,000052, a2 = -0,078993, a3 = 0,001953, a4 = 6,0, a5 = -0,063965, 1 / 8 a6 = -0,027304, 54270 64209 36 70 HEVC 8par a7 = 8,0 22 21 210792 212656 249 48 a0 = 1,0, a1 = -0,001953, a2 = 1,5, a3 = 0,000488, 1 / 8 a4 = 1,0, 157735 154479 204 25 HEVC 6par a5 = 0,0 a0 = 1,001953, a1 = -0,75, 1 / 8 a2 = 0,998047, 53057 58177 45 21 HEVC 4par a3 = 0, 5 23 22 209067 203311 273 40 1 / 8 a0 = 1,0, 157811 152582 206 23 HEVC 6par a1 = -0,001953, a2 = 1,0, a3 = 0,000488, a4 = 0,999023, a5 = 0,25 a0 = 1,000488, a1 = -1,0, 1 / 8 a2 = 1,0, 51256 50729 67 15 HEVC 4par a3 = 0,25 24 23 192579 188654 239 87 a0 = 1,0, a1 = -0,002441, a2 = 1,0, a3 = 0,0, 1 / 8 a4 = 1,0, 144162 140398 191 16 HEVC 6par a5 = 0,0 a0 = 0,00017, a1 = 0,000012, a2 = -0,063289, a3 = -0,003418, a4 = 2,0, a5 = -0,039551, 1 / 16 a6 = -0,031566, 48417 48256 48 69 BIC 8par a7 = 7,0 25 24 210923 202512 272 43 a0 = 1,0, a1 = -0,000977, a2 = 1,0, a3 = 0,0, 1 / 8 a4 = 1,0, 153889 146668 208 20 HEVC 6par a5 = -0,25 a0 = 1,0, a1 = -0,75, 1 / 8 a2 = 1,0, 57034 55844 64 21 HEVC 4par a3 = -0,25 26 25 220190 237977 184 96 a0 = 0,999512, a1 = -0,002441, a2 = 1,75, a3 = 0,001465, 1 / 8 a4 = 1,0, 159167 158211 158 22 HEVC 6par a5 = 0,25 a0 = 0,000044, a1 = -0,00007, a2 = 0,000158, a3 = 0,010254, 1 / 8 a4 = -3,0, 61023 79766 26 72 HEVC 8par a5 = -0,022949, a6 = 0,009075, a7 = 5,0 27 26 217409 242732 182 41 a0 = 1,0, a1 = -0,003418, a2 = 2,0, a3 = 0,001465, 1 / 8 a4 = 1,0, 167075 1 122 16 HEVC 6par a5 = 0,25 a0 = 1,0, a1 = -0,000977, a2 = 0,0, a3 = 0,001465, 1 / 8 a4 = 1,0, 50334 45808 60 23 HEVC 6par a5 = 0,25 28 27 201569 201045 217 110 a0 = 0,000018, a1 = -0,000012, a2 = -0,003867, a3 = -0,013672, a4 = 9,0, a5 = 0,004883, 1 / 8 a6 = 0,001341, 147695 148409 160 48 HEVC 8par a7 = 0,0 a0 = 0,000091, a1 = 0,000009, a2 = -0,032591, a3 = -0,008789, a4 = 4,0, a5 = -0,010742, 1 / 8 a6 = -0,02036, 53874 52636 57 60 HEVC 8par a7 = 3, 0 29 28 200677 195462 239 79 a0 = 0,000027, a1 = -0,000001, a2 = -0,009075, a3 = -0,017578, a4 = 10,0, a5 = 0,006348, 1 / 8 a6 = -0,004656, 149224 146386 178 49 HEVC 8par a7 = -2,0 a0 = 0,999512, a1 = -0,002441, a2 = 0,5, a3 = 0,001953, 1 / 8 a4 = 1,0, 51453 49076 61 28 HEVC 6par a5 = -0,5 30 29 200552 202578 206 51 a0 = 1,0, a1 = -0,004395, a2 = 2,25, a3 = 0,001953, 1 / 8 a4 = 1,0, 149921 150478 158 26 HEVC 6par a5 = -0,5 a0 = 1,0, a1 = -0,001953, a2 = 0,25, a3 = 0,001465, 1 / 8 a4 = 1,000977, 50631 52100 48 23 HEVC 6par a5 = -0,5 31 30 203812 236197 155 70 a0 = 0,000015, a1 = 0,0, a2 = -0,004498, a3 = -0,016602, a4 = 8,0, a5 = 0,007813, 1 / 8 a6 = -0,001736, 147832 147935 153 49 HEVC 8par a7 = -1,0 a0 = 1,0, a1 = -0,25, 1 / 8 a2 = 0,998047, 55980 88262 2 19 AVC 4par a3 = 0,25 32 31 203885 223401 180 55 a0 = 1,0, a1 = -0,004395, a2 = 2,0, a3 = 0,001953, 1 / 8 a4 = 1,0, 148303 151917 149 27 HEVC 6par a5 = -0,25 a0 = 1,000488, a1 = -0,001953, a2 = 0,0, a3 = -0,001465, 1 / 8 a4 = 0,998047, 55582 71484 31 26 HEVC 6par a5 = 0,5 Table 11. The SAD results of the RMM for the CIF “flower” sequence (33 frames) with low-delay IPP images: F R Ref-SAD RMMs-SAD NBB Bits SP-Filter Mod RMMs-Parameter 1 0 527540 708926 63 182 a0 = -0,000199, a1 = -0,000272, a2 = 0,094302, a3 = -0,048828, a4 = 5,0, a5 = 0,003418, 1 / 16 a6 = 0,056108, 45730 117600 15 112 Bill 8par a7 = -3,0 a0 = 1,001953, a1 = 0,010254, a2 = -1,25, a3 = 0,0, 1 / 8 a4 = 1,0, 406573 462448 46 31 HEVC 6par a5 = -0,25 a0 = 0,993652, a1 = 7,25, 1 / 16 a2 = 1,005371, 75237 128878 2 33 BIC 4par a3 = -1,75 2 1 463009 633694 63 152 a0 = -0,000038, a1 = -0,000005, a2 = 0,016888, a3 = -0,054688, a4 = 9,0, a5 = -0,003418, 1 / 16 a6 = -0,001263, 44411 111541 8 79 Bill 8par a7 = 0,0 a0 = 1,001465, a1 = 0,009766, a2 = -1,0, a3 = -0,000488, 1 / 8 a4 = 1,0, 346868 413480 48 28 HEVC 6par a5 = 0,0 a0 = 1,007813, a1 = 0,003418, a2 = 3,25, a3 = 0,001465, 1 / 16 a4 = 1,005371, 71730 108673 7 43 BIC 6par a5 = -2,0 3 2 355197 513141 75 122 a0 = -0,000126, a1 = -0,000215, a2 = 0,048532, a3 = -0,050293, 1 / 16 a4 = 8,0, 44273 111547 10 68 Bill 8par a5 = -0,00293, a6 = 0,038826, a7 = 0,0 a0 = 1,001953, a1 = 0,01123, a2 = -1,25, a3 = 0,0, 1 / 8 a4 = 1,0, 240094 309241 44 20 HEVC 6par a5 = 0,0 a0 = 1,012207, a1 = 0,003418, a2 = 2,5, a3 = -0,00293, 1 / 16 a4 = 1,005371, 70830 92353 21 32 BIC 6par a5 = -0,75 4 3 327923 486773 65 110 a0 = -0,00021, a1 = -0,000311, a2 = 0,082071, a3 = -0,048828, a4 = 7,0, a5 = -0,003418, 1 / 16 a6 = 0,066288, 42272 106832 12 62 Bill 8par a7 = -1,0 a0 = 1,001953, a1 = 0,010254, a2 = -1,25, a3 = 0,0, 1 / 8 a4 = 1,0, 210086 277605 51 19 HEVC 6par a5 = 0,0 a0 = 1,012207, a1 = 0,002441, a2 = 2,25, a3 = -0,00293, 1 / 16 a4 = 1,005371, 75565 102336 2 27 BIC 6par a5 = -0,75 5 4 424389 559797 74 103 a0 = -0,000111, a1 = -0,00017, a2 = 0,043324, a3 = -0,035645, a4 = 6,0, a5 = -0,001465, 1 / 16 a6 = 0,038905, 43770 100412 12 54 BIC 8par a7 = -1,0 a0 = 1,001953, 1 / 8 a1 = 0,010254, 298487 352855 55 17 HEVC 6par a2 = -1,25, a3 = -0,000488, a4 = 1,0, a5 = 0,0 a0 = 1,008301, a1 = 0,002441, a2 = 3,0, a3 = -0,004883, 1 / 16 a4 = 1,004395, 82132 106530 7 30 BIC 6par a5 = -0,25 6 5 426906 571831 73 120 a0 = -0,000073, a1 = -0,000061, a2 = 0,041509, a3 = -0,055664, a4 = 7,0, a5 = -0,003418, 1 / 16 a6 = 0,02257, 48326 119744 15 62 Bill 8par a7 = -2,0 a0 = 1,00293, a1 = 0,01123, a2 = -1,5, a3 = -0,000488, 1 / 8 a4 = 1,0, 297729 339208 52 23 HEVC 6par a5 = 0,0 a0 = 1,01123, a1 = 0,004395, a2 = 2,0, a3 = 0,0, 1 / 16 a4 = 1,004395, 80851 112879 6 33 Bill 6par a5 = -1,25 7 6 381103 561953 73 170 a0 = -0,000166, a1 = -0,000096, a2 = 0,058949, a3 = -0,064453, a4 = 8,0, a5 = -0,007813, 1 / 16 a6 = 0,015309, 42289 119515 9 63 Bill 8par a7 = 1,0 a0 = 1,00293, a1 = 0,010254, a2 = -1,25, a3 = -0,000488, 1 / 8 a4 = 1,0, 267467 331981 56 24 HEVC 6par a5 = 0,0 1 / 16 a0 = -0,000012, 71347 110457 8 81 BIC 8par a1 = 0,000058, a2 = 0,075442, a3 = 0,000977, a4 = 4,0, a5 = 0,009277, a6 = 0,001815, a7 = -6,0 8 7 361055 546999 54 108 a0 = -0,000185, a1 = -0,000304, a2 = 0,0756, a3 = -0,049316, a4 = 6,0, a5 = 0,001465, 1 / 16 a6 = 0,074968, 39855 117321 10 51 Bill 8par a7 = -3,0 a0 = 1,001953, a1 = 0,01123, a2 = -1,25, a3 = 0,0, 1 / 16 a4 = 1,0, 238230 321802 39 26 BIC 6par a5 = 0,0 a0 = 1,012207, a1 = 0,004395, a2 = 2,25, a3 = -0,005859, 1 / 16 a4 = 1,005371, 82970 107876 5 29 BIC 6par a5 = -0,25 9 8 332791 521954 51 103 a0 = -0,000136, a1 = -0,000089, a2 = 0,043403, a3 = -0,063477, a4 = 9,0, a5 = 0,009277, 1 / 16 a6 = 0,023122, 47542 120929 12 64 Bill 8par a7 = -2,0 a0 = 1,001953, a1 = 0,01123, a2 = -1,25, a3 = 0,0, 1 / 8 a4 = 1,0, 206465 291211 39 7 HEVC 6par a5 = 0,0 a0 = 1,009277, a1 = 0,003418, 1 / 16 a2 = 3,0, 78784 109814 0 30 BIC 6par a3 = -0,003418, a4 = 1,004395, a5 = -0,5 10 9 327951 513992 38 104 a0 = -0,000248, a1 = -0,000314, a2 = 0,079782, a3 = -0,052734, a4 = 9,0, a5 = 0,026367, 1 / 16 a6 = 0,099195, 38317 98346 9 62 Bill 8par a7 = -7,0 a0 = 1,001953, a1 = 0,010254, a2 = -1,0, a3 = 0,0, 1 / 16 a4 = 1,0, 211145 311970 26 19 BIC 6par a5 = 0,0 a0 = 1,01123, a1 = 0,004395, a2 = 2,5, a3 = -0,004883, 1 / 16 a4 = 1,004395, 78489 103676 3 21 BIC 6par a5 = -0,25 11 10 427390 570389 88 109 a0 = -0,00021, a1 = -0,000012, a2 = 0,059501, a3 = -0,078125, a4 = 9,0, a5 = 0,016113, 1 / 16 a6 = 0,016098, 37991 96276 11 61 Bill 8par a7 = -3,0 a0 = 1,001953, a1 = 0,01123, a2 = -1,25, a3 = -0,000488, 1 / 8 a4 = 1,0, 309698 374240 67 18 HEVC 6par a5 = 0,0 a0 = 1,018555, a1 = 0,003418, a2 = 1,25, a3 = -0,005859, 1 / 16 a4 = 1,004395, 79701 99873 10 28 BIC 6par a5 = -0,25 12 11 501845 607990 96 177 1 / 16 a0 = -0,000185, 40898 82668 7 64 BIC 8par a1 = -0,000143, a2 = 0,047191, a3 = -0,055664, a4 = 9,0, a5 = 0,015625, a6 = 0,048611, a7 = -4,0 a0 = -0,000003, a1 = -0,000003, a2 = 0,013573, a3 = 0,050293, a4 = -7,0, a5 = 0,0, 1 / 8 a6 = 0,003551, 358061 388183 87 79 HEVC 8par a7 = -1,0 a0 = 1,012207, a1 = 0,002441, a2 = 2,5, a3 = -0,001465, 1 / 16 a4 = 1,004395, 102886 137139 2 32 Bill 6par a5 = -1,0 13 12 570213 721544 57 114 a0 = -0,000198, a1 = -0,000189...

Claims

[1] A system (4000) for performing efficient motion-based video processing using region-based motion, comprising: a region-based motion analyzer (4100), the region-based motion analyzer (4100) including one or more substrates (4102) and logic (4104) coupled to the one or more substrates (4102), the logic (4104) being operable to: obtain multiple block motion vectors for multiple blocks of a current frame with respect to a reference frame; modify the plurality of block motion vectors, wherein modifying the plurality of block motion vectors includes one or more of the following operations: smoothing at least a portion of the plurality of block motion vectors, merging at least a portion of the plurality of block motion vectors, and discarding at least a portion of the plurality of block motion vectors; segment the current frame into multiple regions, the regions comprising a background region type containing a background motion region and a foreground region type containing, in some cases, a single foreground motion region and, in other cases, multiple foreground motion regions; and a power supply (4007) to provide power to the area-based motion analyzer (4100), the logic further serving to: before segmenting the current frame into several areas: to restrict the modified multiple block motion vectors by excluding a portion of the frame in some cases; after segmenting the current frame into several areas: compute a plurality of region-based candidate motion models based on the constrained modified plurality of block motion vectors for the current frame with respect to the reference frame for the background region type and the foreground region type individually, each region-based candidate motion model including a set of region-based candidate motion model parameters representing a region-based motion of each region type of the current frame; determine a best region-based motion model from the plurality of candidate region-based motion models on a frame-by-frame basis and on a region-type basis, each best region-based motion model including a set of best region-based motion model parameters representing the region-based motion of each region type of the current frame; modify an accuracy of the parameters of the best region-based motion model in response to one or more application parameters; map the parameters of the best modified-accuracy region-based motion model into a pixel-based coordinate system to determine a plurality of mapped region-based motion curvature vectors for a plurality of control grid points of the reference frame; predict and encode the plurality of mapped region-based motion curvature vectors for the current frame with respect to a plurality of previously mapped region-based motion curvature vectors; determining a best subpixel filter to use for interpolation at a 1 / 8 pixel location or a 1 / 16 pixel location from among two or more subpixel filter selections per region and per frame; and apply the plurality of mapped region-based motion curvature vectors at the sub-pixel locations per region to the reference frame and perform the interpolation of the pixels based on the determined best sub-pixel filter to generate a region-based motion-compensated curved reference frame. [2] The system (4000) of claim 1, wherein segmenting the current frame into multiple regions further comprises operations to: background-segment the current frame into the background motion region and a non-background motion region, wherein the initial segmentation of the frame into the background motion region and the non-background motion region is based on a purely motion-based segmentation when no dominant color is present and is based on a color-assisted motion-based segmentation when a dominant color is present; foreground-segment the non-background motion region from the single foreground motion region into the multiple foreground motion regions if an analysis of a dominant motion and the peaks indicates that more than one foreground motion region is present in the current frame; and wherein the plurality of regions further includes a static region if one or more types of inactive static regions are present in the current frame, the static region being subtracted from the non-background motion region prior to foreground segmentation, wherein the one or more inactive static regions include one or more of the following types of inactive static regions: black bar type inactive static regions, black border type inactive static regions, letterbox type inactive static regions, logo overlay type inactive static regions, and text overlay type inactive static regions. [3] The system (4000) of claim 1 or 2, wherein segmenting the current frame into the plurality of regions further comprises the following operations to: compute a set of parameters of an initial global motion model for an initial global motion model for the current frame; to use random sampling through multiple iterations to select a set of three linearly independent motion vectors at a time per iteration, each set of three linearly independent motion vectors consisting of linearly independent motion vectors used to compute a sampled six-parameter model of the global motion; and to generate a histogram for each of the sampled six-parameter global motion models to find a best model parameter from a peak value of each parameter, where a set of the best model parameters describes an initial global motion equation. [4] The system (4000) of claim 2, wherein the background segmentation is performed in at least some cases using multiple thresholds to generate multiple alternative binary masks. [5] The system of claim 2 or claims 2 and 3 or claim 4, wherein the segmentation of the current frame into the plurality of regions is performed in at least some cases by a morphological operation of erosion and dilation to form one or more revised segmentations of the plurality of regions. [6] The system (4000) of any of claims 1-5, wherein calculating the plurality of region-based candidate motion models further comprises operations to: selecting a set of global motion models per region in both a first mode selected from four-parameter models, six-parameter models, and eight-parameter models, and a second mode selected from six-parameter models, eight-parameter models, and twelve-parameter models, the first mode being selected for low-resolution scene sequences and the second mode being selected for high-resolution scene sequences; selecting a method for calculating each individual global motion model of the set of global motion models selected from the least squares and the Levenberg-Marquardt (LMA); and to select one or more convergence parameters for the selected least squares method and the Levenberg-Marquardt method. [7] The system (4000) of claim 6, further comprising operations to: to select a method for calculating each individual model of global motion depending on the order of the model, including for a four- and a six-parameter model using the least squares method and for an eight- and a twelve-parameter model using the Levenberg-Marquardt method; to carry out the calculation of each model of global motion using the related selected method; and to select a best model based on the least modified bias. [8] The system (4000) of claim 6, further comprising operations to: to select a method for calculating each individual model of global motion depending on the order of the model, including for a four- and a six-parameter model using the least squares method and for an eight- and a twelve-parameter model using the Levenberg-Marquardt method; to carry out the calculation of each model of global motion using the related selected method; and to select a best model based on a compromise of the best rate distortion optimization that considers both distortion and rate. [9] The system (4000) of any of claims 1-8, wherein modifying the accuracy of the parameters of the best region-based motion model further comprises operations to: determine the significance of each model parameter of the best area-based motion model to define an active area; to determine the application parameters, which include one or more of the following types of application parameters: the coding bit rate, the resolution and the required quality; and to assign each model parameter of the parameters of the best range-based motion model a different accuracy based on the determined significance in some cases, based on the determined application parameter in other cases, and based on the determined significance and the determined application parameter in further cases. [10] The system (4000) of any of claims 1-9, wherein mapping the parameters of the best modified accuracy region-based motion model to the pixel-based coordinate system to determine the plurality of mapped region-based motion curvature vectors for the plurality of control grid points of the reference frame further comprises operations to: to map the parameters of the modified-precision region-based motion model to pixel-domain-based mapped region-based motion curvature vectors as applied to the control grid points, where the control grid points comprise two vertices of a four-parameter frame, three vertices of a six-parameter frame, all four vertices of an eight-parameter frame, and four vertices plus two negative mirror vertices of a twelve-parameter frame. [11] The system (4000) of any of claims 1-10, wherein predicting and encoding the plurality of mapped region-based motion curvature vectors further comprises operations to: predict the curvature vectors of the current frame based on one or more previously stored curvature vectors to generate first predicted curvature vectors, wherein the previously stored curvature vectors are scaled to adjust for frame spacing; predict the curvature vectors of the current frame based on a plurality of codebook curvature vectors to generate second predicted curvature vectors, wherein the codebook curvature vectors are scaled to adjust for the frame spacing; calculate a difference of the curvature vectors of the current frame with the first and second predicted curvature vectors to generate residual curvature vectors; selecting a best one of the residual curvature vectors based on the minimum residual curvature vectors of the first prediction and the second prediction, resulting in the prediction of the selected curvature vectors; entropy-encode a codebook index associated with the predicted codebook curvature vectors when selecting the best residual curvature vectors based on the plurality of codebook curvature vectors, and entropy-encode the identification information associated with the one or more previously stored curvature vectors when selecting the best residual curvature vectors based on the one or more previously stored curvature vectors; and entropy encode the best residual curvature vectors. [12] The system (4000) of any of claims 1-10, wherein predicting and encoding the curvature vectors further comprises operations to: predict the curvature vectors of the current frame based on the most recently stored curvature vectors to generate first predicted curvature vectors, wherein the most recently stored curvature vectors are scaled to adjust for frame spacing, and wherein the most recently stored curvature vectors are mapped upon initialization to one-half of a number of the range-based motion parameters of the current frame; predict the curvature vectors of the current frame based on a plurality of codebook curvature vectors to generate second predicted curvature vectors, wherein the codebook curvature vectors are scaled to adjust for the frame spacing; calculate a difference of the curvature vectors of the current frame with the first and second predicted curvature vectors to generate residual curvature vectors; selecting a best one of the residual curvature vectors based on the minimum residual curvature vectors of the first prediction and the second prediction, resulting in the prediction of the selected curvature vectors; entropy-encode a codebook index associated with the predicted codebook curvature vectors when selecting the best residual curvature vectors based on the plurality of codebook curvature vectors, and entropy-encode the identification information associated with the most recently stored curvature vectors when selecting the best residual curvature vectors based on the most recently stored curvature vectors; and entropy encode the best residual curvature vectors. [13] The system (4000) of any of claims 1-12, wherein determining the best subpixel filter to use for interpolation at the 1 / 8 pixel location among the two or more subpixel filter selections per frame further comprises operations to: determine the application parameters, including one or more of the following types of application parameters: the encoding bit rate, the resolution and the required quality; determine the filter overhead bit cost that can be generated based on the application parameters to determine whether the best sub-pixel filter can be sent on one of the following basis: a per frame basis, a per slice basis, and a per large block basis; for each of the two or more sub-pixel filter selections, to determine: an AVC 1 / 4 pixel filter enhanced to 1 / 8 pixel accuracy and an HEVC 1 / 4 pixel filter enhanced to 1 / 8 pixel accuracy, and wherein the determination of the best sub-pixel filter is determined by calculating a residual of at least a portion of the current frame with respect to a corresponding portion of the region-based motion-compensated curved reference frame and selecting the best of the two or more per-frame sub-pixel filter selections that produces the smallest residual, wherein the selected portion of the current frame is based on the best sub-pixel filter from the per-frame basis, the per-slice basis, and the per-large block basis. [14] The system of any of claims 1-13, wherein determining the best subpixel filter further comprises operations to: determine the application parameters, including one or more of the following types of application parameters: the encoding bit rate, the resolution and the required quality; determine the filter overhead bit cost that can be generated based on the application parameters to determine whether the best sub-pixel filter can be sent on one of the following basis: a per frame basis, a per slice basis, and a per large block basis; for each of the four filter selections of the two or more sub-pixel filter selections, to determine: an AVC 1 / 4 pixel filter extended to 1 / 8 pixel accuracy, an HEVC 1 / 4 pixel filter extended to 1 / 8 pixel accuracy, a bilinear 1 / 16 pixel filter, and a bicubic 1 / 16 pixel filter, and wherein the determination of the best filter is determined by calculating a residual of at least a portion of the current frame with respect to a corresponding portion of the region-based motion-compensated curved reference frame and selecting the best of the four per-frame filters that produces the smallest residual, wherein the selected portion of the current frame is based on the best sub-pixel filter from the per-frame basis, the per-slice basis, and the per-large block basis. [15] A method for performing efficient motion-based video processing using region-based motion, comprising: Obtaining and modifying a plurality of block motion vectors of a current frame with respect to a reference frame of a video sequence, wherein the modification of the plurality of block motion vectors includes one or more of the following operations: smoothing at least a portion of the plurality of block motion vectors, merging at least a portion of the plurality of block motion vectors, and discarding at least a portion of the plurality of block motion vectors; performing pre-segmentation based on global features of motion in some cases and based on a combination of global features of color and motion in other cases, wherein the pre-segmentation comprises segmenting a background region type that includes a background motion region; Performing the segmentation of each frame of the video sequence into a plurality of regions based on the pre-segmentation and based on local features, wherein the local features include one or more of the following: local color features, local motion features, local texture features, and any combination thereof; wherein the plurality of regions are each spatially and temporally consistent, and wherein the segmentation comprises segmenting a foreground region type, which in certain cases includes a single foreground motion region and in other cases includes multiple foreground motion regions; Calculating a best region-based parametric motion model based on a plurality of modified region-based parametric motion models, including calculating the plurality of modified region-based parametric motion models using modified block motion vectors for at least one of the plurality of regions of the video sequence using least squares fitting in specific cases and iterative Levenberg-Marquardt optimization (LMA) in further cases, wherein the best region-based parametric motion model is one of the following: a 4-parameter motion model, a 6-parameter motion model, an 8-parameter motion model, and a 12-parameter motion model, and wherein the modified region-based parametric motion models are modified by adaptively reducing the accuracy of the model parameters for efficient coding; and Generating a prediction region for one of the plurality of regions of the region of the current frame using the parameters of the best region-based parametric motion model in the reference frame and in one of the plurality of regions of the video sequence for which the parameters of the best region-based parametric motion model have been calculated. [16] The method of claim 15, wherein performing the segmentation further comprises segmenting each frame of the video sequence into at least two regions that are not only spatially and temporally consistent, but also semantically coherent. [17] The method of claim 15 or 16, wherein calculating the best region-based parametric motion model further comprises: Calculating two modified region-based parametric motion models simultaneously for a selected region of the plurality of regions, wherein the two models include two of the following models: such as a 4-parameter model, another 4-parameter model, a 6-parameter model, another 6-parameter model, an 8-parameter model, another 8-parameter model, a 12-parameter 4-parameter model, and another 12-parameter model; and Selecting the best parametric model for this domain. [18] The method of claim 15 or 16, wherein calculating the best region-based parametric motion model further includes: Calculating two modified region-based parametric motion models simultaneously for the foreground region type and the background region type, wherein the two modified region-based parametric motion models include two of the following models: such as a 4-parameter model, another 4-parameter model, a 6-parameter model, another 6-parameter model, an 8-parameter model, another 8-parameter model, a 12-parameter 4-parameter model, and another 12-parameter model; and Selecting the best parametric model for both the foreground region type and the background region type. [19] The method of any of claims 15-18, wherein performing the segmentation further comprises: segmenting each frame of the video sequence into three or more regions that are not only spatially and temporally consistent, but also semantically coherent. [20] The method of any of claims 15-19, wherein generating the prediction range further comprises: Adaptively determining a first best subpixel filter to be used for the interpolation with a 1 / 8 pixel location accuracy based on a residual error among two choices: a first is a 1 / 4 pixel interpolation based on the AVC standard extended to 1 / 8 pixel, and a second is a 1 / 4 pixel interpolation based on the HEVC standard extended to 1 / 8 pixel; Adaptively determining a second best subpixel filter for use in the interpolation with a 1 / 16 pixel location accuracy based on a residual error among two choices: a first is a 1 / 16 pixel interpolation based on bilinear filtering and a second is a 1 / 16 pixel interpolation based on bicubic filtering; and Selecting a final best subpixel filter to use for interpolation from among the selections of the first best subpixel filter and the second best subpixel filter based on the residual error. [21] The method of any of claims 15-20, further comprising: encoding the parameters of the region-based motion model via predictive and entropy coding and encoding the region boundary information via explicit coding with a small block precision using one or more of the following accuracies: a 4-pixel small block precision, an 8-pixel small block precision, and a 16-pixel small block precision. [22] A method according to any one of claims 15-21, further comprising: encoding the parameters of the region-based motion model via predictive and entropy coding and encoding the region boundary information via implicit coding using an extension of standard coding mode tables to associate a coded block with the corresponding region to which the coded block belongs. [23] Apparatus (4300) comprising: means for carrying out the methods of any of claims 15-22. [24] Machine-readable memory (4008, 4214) comprising machine-readable instructions that, when executed, implement the methods of any of claims 15-22.

Images