Selective motion compensated frame interpolation

Abstract

A device includes one or more processors configured to execute instructions to obtain motion data indicating estimated motion between a first frame and a second frame of an input sequence of image frames, and to identify any frame regions of the first frame indicating motion greater than a motion threshold based on the motion data. The one or more processors are further configured to determine a motion metric associated with the identified frame region based on the motion data, and to perform a determination whether to generate an intermediate frame using motion compensated frame interpolation based on the motion metric and a size metric associated with the identified frame region. The one or more processors are further configured to generate the intermediate frame based on the determination, and to generate an output sequence of image frames including the intermediate frame between the first frame and the second frame.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to commonly owned U.S. non-provisional patent application No. 17 / 219,080, filed on March 31, 2021, the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to selective motion compensated frame interpolation. Background Art

[0004] Advances in technology have led to smaller and more powerful computing devices. For example, there are currently a variety of portable personal computing devices, including wireless phones such as mobile phones and smart phones, tablets and laptop computers that are small, lightweight, and easy to carry by users. These devices can communicate sound and data packets over wireless networks. In addition, many such devices incorporate additional functions, such as digital still cameras, digital video cameras, digital audio recorders, and audio file players. Likewise, such devices can process executable instructions, including software applications that can be used to access the Internet, such as web browser applications. As a result, these devices can include significant computing capabilities.

[0005] Such computing devices often incorporate the functionality of playing out a video stream. For example, a video stream may represent video content received (e.g., downloaded) from another device. Reducing the frame rate of a video to meet transmission bandwidth limitations may result in poor playback quality, such as increased jitter. Motion compensated frame interpolation is used at the playback device to increase the frame rate of a video clip for smoother playback. However, frame interpolation is computationally intensive and power consumption may be high. On smaller screens, such as mobile devices, the increase in playback smoothness brought about by frame interpolation may not be felt for scenes with relatively little motion. Summary of the invention

[0006] According to one embodiment of the present disclosure, a device includes a memory and one or more processors. The memory is configured to store instructions. The one or more processors are configured to execute the instructions to obtain motion data indicating estimated motion between a first frame and a second frame of an input sequence of image frames. The one or more processors are also configured to execute the instructions to identify any frame region of the first frame indicating motion greater than a motion threshold based on the motion data. The one or more processors are also configured to execute the instructions to determine a motion metric associated with the identified frame region based on the motion metric and size metric associated with the identified frame region to determine whether to use motion compensated frame interpolation to generate an intermediate frame. The one or more processors are also configured to execute the instructions to generate an intermediate frame based on the determination, and to generate an output sequence of image frames including the intermediate frame between the first frame and the second frame.

[0007] According to another embodiment of the present disclosure, a method includes obtaining motion data indicating estimated motion between a first frame and a second frame of an input sequence of image frames at a device. The method also includes identifying any frame regions of the first frame indicating motion greater than a motion threshold based on the motion data. The method also includes determining a motion metric associated with the identified frame region based on the motion data. The method also includes performing a determination of whether to generate an intermediate frame using motion compensated frame interpolation based on the motion metric and size metric associated with the identified frame region. The method also includes generating an intermediate frame at the device based on the determination. The method also includes generating an output sequence of image frames including an intermediate frame between the first frame and the second frame at the device.

[0008] According to another embodiment of the present disclosure, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to obtain motion data indicating estimated motion between a first frame and a second frame of an input sequence of image frames. When the instructions are executed by the one or more processors, the one or more processors are also caused to identify any frame region of the first frame indicating motion greater than a motion threshold based on the motion data. When the instructions are executed by the one or more processors, the one or more processors are also caused to determine a motion metric associated with the identified frame region based on the motion data. When the instructions are executed by the one or more processors, the one or more processors are also caused to perform a determination of whether to generate an intermediate frame using motion compensated frame interpolation based on the motion metric and size metric associated with the identified frame region. When the instructions are executed by the one or more processors, the one or more processors are also caused to generate an intermediate frame based on the determination. When the instructions are executed by the one or more processors, the one or more processors are also caused to generate an output sequence of image frames including an intermediate frame between the first frame and the second frame.

[0009] According to another embodiment of the present disclosure, an apparatus includes a component for obtaining motion data indicating estimated motion between a first frame and a second frame of an input sequence of image frames. The apparatus also includes a component for identifying any frame region of the first frame indicating motion greater than a motion threshold based on the motion data. The apparatus also includes a component for determining a motion metric associated with the identified frame region based on the motion metric and a size metric associated with the identified frame region. The apparatus also includes a component for performing a determination of whether to generate an intermediate frame using motion compensated frame interpolation based on the motion metric and the size metric associated with the identified frame region. The apparatus also includes a component for generating the intermediate frame based on the determination. The apparatus also includes a component for generating an output sequence of image frames including the intermediate frame between the first frame and the second frame.

[0010] Other aspects, advantages, and features of the present disclosure will become apparent after reviewing the entire application, including the following sections: Brief Description of the Drawings, Detailed Description, and Claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram of certain illustrative aspects of a system operable to perform selective motion compensated frame interpolation according to some examples of the present disclosure.

[0012] Figure 2 According to some examples of the present disclosure Figure 1 A diagram of illustrative aspects of a frame rate regulator of a system.

[0013] Figure 3 According to some examples of the present disclosure Figure 2 A diagram of an illustrative example of interpolation factor determination data used by a frame rate adjuster.

[0014] Figure 4 According to some examples of the present disclosure Figure 1 FIG. 1 is a diagram of an illustrative example of a frame generated by a system.

[0015] Figure 5 According to some examples of the present disclosure Figure 2 A diagram of illustrative aspects of a motion compensated frame interpolator of a frame rate adjuster.

[0016] Figure 6 An example of an integrated circuit operable to perform selective motion compensated frame interpolation according to some examples of the present disclosure is shown.

[0017] Figure 7 is a diagram of a mobile device operable to perform selective motion compensated frame interpolation in accordance with some examples of the present disclosure.

[0018] Figure 8is a diagram of a wearable electronic device operable to perform selective motion compensated frame interpolation according to some examples of the present disclosure.

[0019] Fig. 9 is a diagram of a head-mounted view (headset), such as a virtual reality or augmented reality headset, operable to perform selective motion compensated frame interpolation according to some examples of the present disclosure.

[0020] Fig.10 is a diagram of a first example of a vehicle operable to perform selective motion compensated frame interpolation in accordance with some examples of the present disclosure.

[0021] Fig.11 is a diagram of a second example of a vehicle operable to perform selective motion compensated frame interpolation according to some examples of the present disclosure.

[0022] Fig.12 According to some examples of the present invention, Figure 1 FIG. 1 is a diagram of a specific embodiment of a method of performing selective motion compensated frame interpolation by an apparatus.

[0023] Fig.13 is a block diagram of a specific illustrative example of an apparatus operable to perform selective motion compensated frame interpolation according to some examples of the present invention. DETAILED DESCRIPTION

[0024] Motion compensated frame interpolation is used to increase the frame rate of video clips for smoother playback. For example, by inserting motion compensated interpolated frames between each pair of original frames, the frame rate of the video stream is increased from 30 frames per second (fps) to 60fps. In full interpolation, the motion compensated interpolated frame represents half of the motion depicted between the original frame pairs. For example, if an object is shifted 50 pixels to the right between the first original frame and the second original frame, the object is shifted 25 pixels to the right between the first original frame and the interpolated frame, and 25 pixels to the right between the interpolated frame and the second original frame. This results in smoother playback of scenes containing motion. However, frame interpolation is computationally intensive, and power consumption may be high. On smaller screens, such as on mobile devices, for scenes with relatively less motion, the increase in playback fluency brought by frame interpolation may not be felt.

[0025] Disclosed are systems and methods for performing selective motion compensated frame interpolation. For example, a frame rate adjuster receives an input frame sequence and generates an output frame sequence based on the input frame sequence. The output frame sequence has a higher frame rate than the input frame sequence. For example, the frame rate adjuster adds one or more intermediate frames to the input frame sequence to generate the output frame sequence. For illustration, the input frame sequence includes a first frame followed by a second frame. The output frame sequence includes an intermediate frame between the first frame and the second frame.

[0026] The frame rate adjuster performs motion compensated interpolation or frame duplication to generate an intermediate frame. For example, the frame rate adjuster generates a motion vector indicating a motion detected between a first frame and a second frame. The frame rate adjuster determines a motion metric (e.g., average motion) and a size metric (e.g., frame percentage) corresponding to an area of ​​the first frame that is greater than a threshold motion. The frame rate adjuster determines whether to perform motion compensated frame interpolation based on the motion metric and the size metric. For example, when a larger percentage of the first frame corresponds to a higher motion, the frame adjuster performs motion compensated frame interpolation to generate an intermediate frame. Alternatively, when the first frame corresponds to a lower motion, or when a smaller percentage of the first frame corresponds to a higher motion, the frame adjuster performs frame duplication to generate an intermediate frame.

[0027] Specific aspects of the present disclosure are described below with reference to the accompanying drawings. In the specification, common features are designated by common reference numerals. As used herein, various terms are used only for the purpose of describing specific embodiments and are not intended to limit the embodiments. For example, the singular forms "a", "an", and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. In addition, some features described herein are singular in some embodiments and plural in other embodiments. For illustration, Figure 1 Depicts a system including one or more processors ( Figure 1 190 ), indicating that in some embodiments, the device 102 includes a single processor 190 , while in other embodiments, the device 102 includes multiple processors 190 .

[0028] As used herein, the terms "comprise," "comprises," and "comprising" may be used interchangeably with "include," "includes," or "including." Additionally, the term "wherein" may be used interchangeably with "it." As used herein, "exemplary" indicates an example, implementation, and / or aspect, and should not be interpreted as limiting or indicating a preferred or preferred implementation. As used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify elements (such as structures, components, operations, etc.) do not themselves indicate any priority or order of an element relative to another element, but merely distinguish an element from another element with the same name (but an ordinal term is used). As used herein, the term "a set" refers to one or more specific elements, and the term "plurality" refers to a plurality (e.g., two or more) of specific elements.

[0029] As used herein, "coupling" may include "communicative coupling", "electrical coupling" or "physical coupling", and may also (or alternatively) include any combination thereof. Two devices (or components) may be directly or indirectly coupled (e.g., communicative coupling, electrical coupling or physical coupling) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks or combinations thereof), etc. As illustrative, non-limiting examples, two electrically coupled devices (or components) may be included in the same device or different devices, and may be connected via electronic devices, one or more connectors or inductive coupling. In some embodiments, two communicatively coupled devices (or components), such as in electrical communication, may directly or indirectly send and receive signals (e.g., digital signals or analog signals) via one or more wires, buses, networks, etc. As used herein, "direct coupling" may include two devices that are coupled (e.g., communicative coupling, electrical coupling or physical coupling) without an intervening component.

[0030] In the present disclosure, terms such as "determine," "calculate," "estimate," "shift," "adjust," and the like may be used to describe how to perform one or more operations. It should be noted that these terms should not be construed as limiting, and other techniques may be utilized to perform similar operations. Additionally, as referred to herein, "generate," "calculate," "estimate," "use," "select," "access," and "determine" may be used interchangeably. For example, "generating," "calculating," "estimating," or "determining" a parameter (or signal) may refer to actively generating, estimating, calculating, or determining a parameter (or signal), or may refer to using, selecting, or accessing an already generated parameter (or signal), such as a parameter (or signal) generated by another component or device.

[0031] See also Figure 1 , discloses certain illustrative aspects of a system configured to perform selective motion compensated frame interpolation, and generally refers to the system as 100. System 100 includes a device 102 coupled to a display device 106. Device 102 is configured to perform selective motion compensated frame interpolation using a frame rate adjuster 140.

[0032] Device 102 includes one or more processors 190 coupled to memory 132 and modem 170. One or more processors 190 include frame rate regulator 140. Memory 132 is configured to store instructions 196. One or more processors 190 are configured to execute instructions 196 to perform one or more operations described herein. Modem 170 is configured to enable communication with one or more second devices, such as receiving a frame sequence 180 of one or more frames 101 (e.g., video frames, photo bursts, or a combination thereof). In particular aspects, device 102 is coupled to display device 106. As an illustrative example, display device 106 is described as being external to device 102. In some examples, display device 106 is integrated into device 102.

[0033] The frame rate adjuster 140 is configured to receive the frame sequence 180 and output a frame sequence 192. The frame sequence 192 has a higher frame rate (e.g., more frames per second) than the frame sequence 180. For example, the frame sequence 192 includes one or more frames 101 of the frame sequence 180, and also includes one or more intermediate frames 191 interspersed between the one or more frames 101. For illustration, the one or more frames 101 include a frame 101A, followed by a frame 101B, followed by a frame 101C. The frame rate adjuster 140 is configured to generate an intermediate frame 191A based on the frame 101A and the frame 101B, and output the intermediate frame 191A between the frame 101A and the frame 101B as in the frame sequence 192. In a particular embodiment, the frame sequence 192 includes an intermediate frame 191 between each pair of consecutive frames of the one or more frames 101. For example, frame sequence 192 includes intermediate frame 191A between frame 101A and frame 101B, and intermediate frame 191B between frame 101B and frame 101C. In an alternative embodiment, frame sequence 192 includes one or more intermediate frames 191 between at least one pair of consecutive frames of one or more frames 101.

[0034] In some implementations, device 102 corresponds to or is included in one or more types of devices. In the illustrative example, one or more processors 190 are integrated into a processor such as the one shown in FIG. Figure 7 The mobile phone or tablet computer device described, as referenced Figure 8 The wearable electronic device described, or as referenced Fig. 9 In another illustrative example, one or more processors 190 are integrated into a vehicle, such as a vehicle described in the accompanying drawings. Fig.10 and Fig.11 Further described.

[0035] During operation, the frame rate adjuster 140 receives a frame sequence 180 of one or more frames 101 (e.g., video frames). For example, the frame rate adjuster 140 receives frame 101A and frame 101B of the frame sequence 180. In certain embodiments, the frame rate adjuster 140 receives the frame sequence 180 from the modem 170, the memory 132, a second device (e.g., a storage device), or a combination thereof.

[0036] In a particular aspect, frame rate adjuster 140 obtains motion data (e.g., motion vectors) indicating estimated motion between frame 101A and frame 101B. In example 150, the motion data indicates that each of regions A, B, and C of frame 101A corresponds to a first horizontal motion (e.g., 1 pixel block to the right) and a first vertical motion (e.g., 2 pixel blocks down), region D corresponds to no horizontal motion and a second vertical motion (e.g., 1 pixel block down), and region X of frame 101A corresponds to no motion (e.g., the same location in each of frame 101A and frame 101B).

[0037] In a particular aspect, each region of the frame 101A has the same size. In an alternative aspect, at least one region of one or more frames 101 has a different size than another region of the frame 101A. In a particular aspect, the dimensions of one or more regions of the frame 101A are based on default data, configuration settings, user input, or a combination thereof. In a particular aspect, one or more regions of the frame 101A are square, rectangular, elliptical, irregular, or a combination thereof. In a particular aspect, the regions of the frame 101A do not overlap. In an alternative aspect, the regions of the frame 101A at least partially overlap. In a particular aspect, the pixel blocks have the same size, the same shape, or both as the regions of the frame 101A. In a particular aspect, the pixel blocks have different sizes, different shapes, or both as the regions of the frame 101A. In a particular aspect, the size, shape, or both of the pixel blocks are based on default data, configuration settings, user input, or a combination thereof. In a particular aspect, the pixel blocks include one or more pixels of the frame 101A.

[0038] The frame rate adjuster 140 determines one or more regional motion metrics (RMMs) 117 for a region (eg, a frame region) of the frame 101A. For example, the frame rate adjuster 140 determines one or more regional motion metrics (RMMs) 117 for a region (eg, a frame region) of the frame 101A. ) to determine the regional motion metric 117 for each region of frame 101A.

[0039] In example 150, the frame rate adjuster 140 determines a region motion metric 117A (e.g., 0) for region X that indicates no motion. The frame rate adjuster 140 determines a region motion metric 117B (e.g., ). The frame rate adjuster 140 determines a region motion metric 117C for region D (e.g., ).

[0040] The frame rate adjuster 140 identifies any frame region of the frame 101A that indicates motion greater than a motion threshold 111 (e.g., 2) based on motion data (e.g., one or more regional motion metrics 117). In particular aspects, the motion threshold 111 corresponds to default data, a configuration setting, a user input, or a combination thereof. In example 150, the frame rate adjuster 140 identifies regions A, B, and C as corresponding to motion greater than the motion threshold 111 in response to determining that each of regions A, B, and C has a regional motion metric 117B (e.g., 2.2) greater than the motion threshold 111 (e.g., 2). The frame rate adjuster 140 identifies regions X and D as corresponding to motion less than or equal to the motion threshold 111 in response to determining that region X has a regional motion metric 117A (e.g., 0) less than or equal to the motion threshold 111 (e.g., 2) and region D has a regional motion metric 117C (e.g., 1) less than or equal to the motion threshold 111.

[0041] In a particular aspect, frame rate adjuster 140 sets motion metric 115 to a first motion metric value (e.g., 0) and size metric 113 to a first size metric value (e.g., 0%) in response to determining that no region of frame 101A corresponds to motion greater than motion threshold 111. Alternatively, frame rate adjuster 140 determines motion metric 115 and size metric 113 based on region motion metrics 117 of the identified regions in response to identifying one or more regions of frame 101A indicating motion greater than motion threshold 111.

[0042] The frame rate adjuster 140 determines the motion metric 115 based on the regional motion metric 117 of each region (e.g., A, B, and C) identified as indicating motion greater than the motion threshold 111, and independently of the regional motion metrics (e.g., regional motion metric 117A and regional motion metric 117C) of the regions (e.g., regions X and D) identified as indicating motion less than or equal to the motion threshold 111. In particular embodiments, the motion metric 115 is indicated by the regional motion metrics 117 of the regions identified as indicating motion greater than the motion threshold 111 based on average motion (e.g., mean, median, or mode), maximum motion, range of motion, or a combination thereof. In the example 150, the motion metric 115 corresponds to an average (e.g., (2.2+2.2+2.2) / 3=2.2) of the regional motion metrics 117 of the regions (e.g., A, B, and C) identified as indicating motion greater than the motion threshold 111 (e.g., 2).

[0043] The frame rate adjuster 140 determines a size metric 113 associated with the frame regions (e.g., A, B, and C) identified as indicating motion greater than a motion threshold 111 (e.g., 2). In particular aspects, the size metric 113 is based on a combined size of the identified frame regions (e.g., A, B, and C), a percentage of the frame 101A that includes the identified frame regions (e.g., A, B, and C), or a combination thereof. In a particular example, the frame rate adjuster 140 determines the size metric 113 based on a first count (e.g., 3) of the regions (e.g., A, B, C) identified as indicating motion greater than the motion threshold 111 and a second count (e.g., 16) of all regions of the frame 101A (e.g., size metric 113 = first count / second count) (e.g., 3 / 16 = 18.75%).

[0044] The frame rate adjuster 140 performs a determination based on the size metric 113 and the motion metric 115 whether to use motion compensated frame interpolation to generate the intermediate frame 191A, as shown in FIG. Figure 2-Figure 3 Further described. For example, the intermediate frame generation option 120 includes performing motion compensated frame interpolation 124 and one or more alternatives to motion compensated frame interpolation 124 (e.g., performing frame duplication 122). In a particular aspect, in the case where the frame area indicating motion greater than the motion threshold 111 corresponds to a relatively larger portion of the frame 101A (as indicated by the size metric 113) or corresponds to a relatively larger motion (as indicated by the motion metric 115), the transition between the frame 101A and the frame 101B is predicted to be more noticeable during playback. In this case, the frame rate adjuster 140 uses the motion compensated frame interpolation 124 to increase playback smoothness. Alternatively, in the case where the frame area indicating motion greater than the motion threshold 111 (if any) corresponds to a relatively smaller portion of the frame 101A (as indicated by the size metric 113) or corresponds to a relatively small motion (as indicated by the motion metric 115), the transition between the frame 101A and the frame 101B is predicted to be less noticeable during playback. In this case, frame rate regulator 140 uses frame duplication 122 to save resources.

[0045] In response to determining that motion compensated frame interpolation is to be performed, the frame rate adjuster 140 generates an interpolated frame 123 as the intermediate frame 191A using the motion compensated frame interpolation 124. For example, the frame rate adjuster 140 generates the interpolated frame 123 such that the second motion between the frame 101A and the interpolated frame 123 is based on the interpolation weights 119 applied to the first motion between the frame 101A and the frame 101B. In a particular embodiment, the interpolation weights 119 are based on the size metric 113 and the motion metric 115, as shown in FIG. Figure 2-Figure 3Further described. In a particular embodiment, the interpolation weights 119 are based on predetermined weights. In a particular aspect, the interpolation weights 119 are based on default data, configuration settings, user input, or a combination thereof. In a particular aspect, a first copy value (e.g., 0) of the interpolation weights 119 corresponds to a copy of the frame 101A, and a second copy value (e.g., 1) of the interpolation weights 119 corresponds to a copy of the frame 101B. In a particular aspect, values ​​of the interpolation weights 119 between the first copy value and the second copy value correspond to interpolation. For example, a full interpolation value (e.g., 0.5) of the interpolation weights 119 corresponds to full interpolation. As another example, a half interpolation value (e.g., 0.25) corresponds to half interpolation.

[0046] In example 150, the motion compensated frame interpolation 124 corresponds to "full interpolation" to generate the interpolated frame 123. "Full interpolation" refers to generating the interpolated frame 123 so that the second motion between the interpolated frame 123 and each of the frames 101A and 101B is the full interpolation value (e.g., 0.5) of the interpolation weight 119 applied to the first motion between the frames 101A and 101B. For example, the first motion between the frames 101A and 101B indicates a first horizontal motion (e.g., 1 pixel block to the right) and a first vertical motion (e.g., 2 pixel blocks downward) for each of the regions A, B, and C, no horizontal motion and a second vertical motion (e.g., 1 pixel block downward) for the region D, and no motion for the region X. The frame rate adjuster 140 generates the interpolated frame 123 so that the second motion between the frame 101A and the interpolated frame 123 indicates a first specific horizontal motion (e.g., 0.5*1=0.5 pixel blocks to the right) and a first specific vertical motion (e.g., 0.5*2=1 pixel block downward) for each of the regions A, B, and C, no horizontal motion (e.g., 0.5*0=0) and a second specific vertical motion (e.g., 0.5*1=0.5 pixel blocks downward) for the region D, and no motion (e.g., 0.5*0 and 0.5*0) for the region X. The interpolated frame 123 is played back as the intermediate frame 191A between the frame 101A and the frame 101B, which will make the transition between the frame 101A and the frame 101B smooth (e.g., reduce jitter).

[0047] Alternatively, the frame rate adjuster 140 generates the intermediate frame 191A using a substitute of the motion compensated frame interpolation 124 in response to the determination not to perform the motion compensated frame interpolation. For example, the frame rate adjuster 140 performs the frame duplication 122 to generate the duplicate frame 121 as the intermediate frame 191A. In the example 150, the duplicate frame 121 corresponds to a copy of the frame 101A. In other examples, the duplicate frame 121 may correspond to a copy of the frame 101B. In a particular embodiment, generating the duplicate frame 121 corresponds to generating an additional frame based on the frame 101A (or the frame 101B). In another embodiment, generating the duplicate frame 121 corresponds to increasing (e.g., doubling) the playback time of the frame 101A (or the frame 101B) independently of generating the additional frame. In a particular embodiment, generating the duplicate frame 121 corresponds to including a reference to the frame 101A (or the frame 101B) twice in the playlist. Generating the duplicate frame 121 uses fewer resources (eg, power, processing cycles, and time) than generating the interpolated frame 123 .

[0048] The frame rate adjuster 140 generates a frame sequence 192 including an intermediate frame 191A between the frame 101A and the frame 101B. In certain aspects, the frame rate adjuster 140 provides the frame sequence 192 to the display device 106 for playback, stores the frame sequence 192 in the memory 132 or a storage device, provides (e.g., streams) the frame sequence 192 to another device, or a combination thereof.

[0049] The system 100 is thus capable of selective motion compensated frame interpolation based on the size of the higher motion region and the degree of motion indicated by the higher motion region. For example, the frame rate adjuster 140 transitions between performing motion compensated frame interpolation to increase playback smoothness and performing frame duplication to save resources.

[0050] See also Figure 2 , shows illustrative aspects of the frame rate adjuster 140. The frame rate adjuster 140 includes a motion estimator 204, which is coupled to an interpolation selector 208 via an interpolation factor generator 206. The interpolation selector 208 is coupled to each of the frame copier 210 and the motion compensated frame interpolator 214.

[0051] The motion estimator 204 receives the frame sequence 180. For example, the motion estimator 204 receives the frame 101A and the frame 101B. The motion estimator 204 generates motion data (MD) 205 indicating estimated motion between consecutive pairs of frames of the frame sequence 180. For example, the motion estimator 204 generates the motion data 205 (e.g., a set of motion vectors) indicating estimated motion between the frame 101A and the frame 101B. The motion estimator 204 provides the motion data 205 to each of the interpolation factor generator 206 and the motion compensated frame interpolator 214.

[0052] Interpolation factor (IF) generator 206 determines interpolation factor 207 based on motion data 205. For example, interpolation factor generator 206 determines size metric 113 and motion metric 115, as shown in FIG. Figure 1 As described. In a particular aspect, the interpolation factor generator 206 determines that the motion data 205 includes a set of motion vectors (e.g., 16 motion vectors) indicating estimated motion between the frame 101A and the frame 101B. The interpolation factor generator 206 determines that a subset of the motion vectors (e.g., 3 motion vectors) indicates motion greater than a motion threshold 111 (e.g., 2 pixel blocks). The interpolation factor generator 206 determines a size metric 113 (e.g., 3 / 16=18.75%) as a percentage of the motion vectors of the motion data 205 that indicate motion above the motion threshold 111. The interpolation factor generator 206 determines a motion metric 115 based on the motion (e.g., average motion or range of motion) indicated by the subset of motion vectors (e.g., 3 motion vectors). The interpolation factor generator 206 generates an interpolation factor 207 based on a comparison of the size metric 113 and the motion metric 115 with the interpolation factor determination data 270.

[0053] In certain aspects, the interpolation factor determination data 270 indicates a plurality of interpolation factor regions defined by a range of size metric values, a range of motion metric values, or a combination thereof. Figure 3 , an example 300 of interpolation factor determination data 270 is shown. Interpolation factor determination data 270 includes multiple interpolation factor regions, including a fully interpolated region 362, a transition region 364, and a duplicate region 366. For example, duplicate region 366 corresponds to small motion or no motion (e.g., low motion metric) indicated by a small portion of frame 101A (e.g., low size metric), where frame interpolation may result in an insignificant increase in playback smoothness. Fully interpolated region 362 corresponds to high motion (e.g., high motion metric) indicated by a large portion of frame 101A (e.g., high size metric), where frame interpolation may result in a significant increase in playback smoothness. Transition region 364 corresponds to high motion in a small portion of frame 101A or small motion in a large portion of frame 101A, where frame interpolation may result in a slightly significant increase in playback smoothness.

[0054] In some embodiments, the interpolation factor determination data 270 includes a full interpolation area 362 and a copy area 366, and does not include any transition area. In some embodiments, the interpolation factor determination data 270 includes multiple transition areas. In a particular aspect, the interpolation factor determination data 270 is based on a configuration setting, default data, user input, a detected context, an operating mode, a screen size, or a combination thereof. For illustration, an example 300 of the interpolation factor determination data 270 corresponds to a first configuration setting (e.g., a resource conservation setting), a first user input (e.g., a resource conservation input), a first detected context (e.g., a lecture video), a first operating mode (e.g., a low power mode or a battery conservation mode), a first screen size (e.g., less than a threshold screen size), or a combination thereof, and an example 350 of the interpolation factor determination data 270 corresponds to a second configuration setting (e.g., a playback smoothness setting), a second user input (e.g., a playback smoothness input), a second detected context (e.g., a movie), a second operating mode (e.g., a charging mode, a charged mode, or a full power mode), a second screen size (e.g., greater than or equal to a threshold screen size), or a combination thereof.

[0055] In certain aspects, the detected context includes the type of video content of the frame sequence 180. For example, playback smoothness is more relevant for a second type of video content (e.g., a movie) than for a first type of video content (e.g., a speech). In certain aspects, the detected context includes a calendar event associated with the frame sequence 180, with the playback time of the frame sequence 180, or both. For example, playback smoothness is more relevant for a second type of calendar event (e.g., an advertising campaign display) than for a first type of calendar event (e.g., fitness). In certain aspects, the detected context includes the amount of movement of the display device 106. For example, playback smoothness is less relevant when the frame sequence 192 is played back while the display device 106 is moving (e.g., the display device 106 is integrated into a virtual reality head-mounted view device, and the virtual reality head-mounted view device plays back the frame sequence 192 while the user of the virtual reality head-mounted view device is running).

[0056] In certain aspects, the frame rate adjuster 140 adjusts the boundaries of the region of interpolation factor determination data 270 based on the detected condition. For example, the detected condition is based on a configuration setting, a user input, a detected context, an operating mode, a screen size, or a combination thereof. To illustrate, the frame rate adjuster 140 shifts (e.g., to the right, to the top, or both) the boundaries of the region of interpolation factor determination data 270 (e.g., from example 300 to example 350) to increase the copy region 366 and decrease the full interpolation region 362 in response to detecting a lower power operating mode (e.g., low battery), a smaller screen size, a context indicating a first type of video (e.g., speech), or a combination thereof.

[0057] Increasing the copy area 366 and reducing the full interpolation area 362 increases the motion threshold that the motion metric 115 must meet, and increases the size threshold that the size metric 113 must meet, to trigger the use of motion compensated interpolation to generate an intermediate frame. Resource consumption is reduced by increasing the likelihood of using frame duplication, while also enabling motion compensated interpolation to increase playback smoothness for high motion and large frame partial motion. Alternatively, the frame rate regulator 140 shifts (e.g., to the left, bottom, or both) the boundary of the region of interpolation factor determination data 270 (e.g., from example 350 to example 300) in response to detecting a higher power operating mode (e.g., plugged into a power source), a larger screen size, a context indicating a second type of video (e.g., a movie), or a combination thereof to reduce the copy area 366 and increase the full interpolation area 362. Reducing the copy area 366 and increasing the full interpolation area 362 reduces the motion threshold that the motion metric 115 must meet, and reduces the size threshold that the size metric 113 must meet, to trigger the use of motion compensated interpolation to generate an intermediate frame. Increases playback smoothness by adding the possibility to use motion compensated interpolation, while also enabling frame duplication to save resources for low motion or small frame partial motion.

[0058] Each of the plurality of regions of the interpolation factor determination data 270 corresponds to a particular interpolation factor value. For example, the duplicate region 366 corresponds to a first interpolation factor value (e.g., 0), and the full interpolation region 362 corresponds to a second interpolation factor value (e.g., 1). In a particular aspect, the transition region 364 corresponds to a third interpolation factor value (e.g., 0.5).

[0059] The interpolation factor generator 206 generates the interpolation factor 207 indicating a specific interpolation factor value corresponding to the specific region in response to determining that the size metric 113 and the motion metric 115 correspond to (e.g., are in) the specific region of the interpolation factor determination data 270. For example, the interpolation factor generator 206 generates the interpolation factor 207 indicating a first interpolation factor value (e.g., 0) corresponding to the duplicate region 366 in response to determining that the size metric 113 and the motion metric 115 correspond to (e.g., are in) the duplicate region 366. Alternatively, the interpolation factor generator 206 generates the interpolation factor 207 indicating a second interpolation factor value (e.g., 1) in response to determining that the size metric 113 and the motion metric 115 correspond to (e.g., are in) the full interpolation region 362. In a particular embodiment, the interpolation factor generator 206 generates the interpolation factor 207 indicating a third interpolation factor value (e.g., 0.5) in response to determining that the size metric 113 and the motion metric 115 are in the transition region 364. A first interpolation factor value (eg, 0) indicates that no motion compensated frame interpolation is to be performed. A second interpolation factor value (eg, 1) indicates that full interpolation is to be performed. A third interpolation factor value (eg, 0.5) indicates that partial interpolation is to be performed.

[0060] Interpolation factor determination data 270 indicating a plurality of interpolation factor regions defined by a range of size metric values ​​and a range of motion metric values ​​is provided as an illustrative example. In some examples, the interpolation factor determination data 270 indicates a plurality of interpolation factor regions defined by a range of size metric values ​​and independent of a range of motion metric values. In these examples, the interpolation factor generator 206 generates an interpolation factor 207 indicating a particular interpolation factor value corresponding to the particular region in response to determining that the size metric 113 corresponds to (e.g., at) a particular region of the interpolation factor determination data 270. In other examples, the interpolation factor determination data 270 indicates a plurality of interpolation factor regions defined by a range of motion metric values ​​and independent of a range of size metric values. In these examples, the interpolation factor generator 206 generates an interpolation factor 207 indicating a particular interpolation factor value corresponding to the particular region in response to determining that the motion metric 115 corresponds to (e.g., at) a particular region of the interpolation factor determination data 270.

[0061] Back to Figure 2 , the interpolation factor generator 206 provides the interpolation factor 207 to each of the interpolation selector 208 and the motion compensated frame interpolator 214. The interpolation selector 208 determines whether to use motion compensated frame interpolation to generate the intermediate frame 191A based on the interpolation factor 207. For example, the determination of whether to use motion compensated frame interpolation is based on whether the interpolation factor 207 satisfies the interpolation criterion (e.g., whether the interpolation factor 207 is equal to 0). For illustration, the interpolation selector 208 determines that motion compensated frame interpolation is to be used in response to determining that the interpolation factor 207 satisfies the interpolation criterion (e.g., the interpolation factor 207 is not equal to 0), and sends an activate interpolation command 213 to the motion compensated frame interpolator 214. Alternatively, the interpolation selector 208 determines that motion compensated frame interpolation will not be used (e.g., an alternative to motion compensated frame interpolation will be used) in response to determining that the interpolation factor 207 fails to meet the interpolation criteria (e.g., the interpolation factor 207 is equal to 0), and sends an activation copy command 209 to the frame copier 210.

[0062] In response to receiving the activate interpolation command 213, the motion compensated frame interpolator 214 performs motion compensated frame interpolation on the frame 101A and the frame 101B to generate the interpolated frame 123 as the intermediate frame 191A, as shown in FIG. Figure 5 As further described, for example, the motion compensated frame interpolator 214 performs motion compensated frame interpolation based on the motion data 205, the interpolation factors 207, the frame 101A, the frame 101B, or a combination thereof.

[0063] In response to receiving the activation copy command 209, the frame copier 210 generates a copy frame 121 by copying one of the frame 101A or the frame 101B, as shown in FIG. Figure 1The frame rate adjuster 140 outputs the duplicate frame 121 as the intermediate frame 191A. For example, the frame rate adjuster 140 outputs the frame 191 between the frame 101A and the frame 101B in the frame sequence 192.

[0064] In a first embodiment, the interpolation factor determination data 270 does not include any transition regions. In this embodiment, the frame rate adjuster 140 switches between full interpolation and duplication to generate intermediate frames. In a second embodiment, the interpolation factor determination data 270 includes at least one transition region. In this embodiment, when the movement in the frame sequence 180 transitions via a transition region 364 between the full interpolation region 362 and the duplication region 366, the frame rate adjuster 140 transitions via partial interpolation between full interpolation and duplication to generate intermediate frames.

[0065] Frame rate adjuster 140 is thus capable of selective motion compensated frame interpolation based on size metric 113 and motion metric 115. In particular aspects, the criteria for selecting motion compensated frame interpolation can be dynamically changed by adjusting interpolation factor determination data 270 based on detected conditions (e.g., configuration settings, user input, detected context, operating mode, screen size, or a combination thereof).

[0066] In certain embodiments, the device 102 includes an always-on power domain and a second power domain, such as an on-demand power domain. In some embodiments, the first stage of the frame rate adjuster 140 is configured to operate in an always-on mode, and the second stage of the frame rate adjuster 140 is configured to operate in an on-demand mode. In certain aspects, the motion estimator 204, the interpolation factor generator 206, the interpolation selector 208, the frame copier 210, or a combination thereof, is included in the first stage of the frame rate adjuster 140, and the motion compensated frame interpolator 214 is included in the second stage of the frame rate adjuster 140.

[0067] The first stage is configured to generate an activate interpolation command 213 to start one or more operations at the second stage. In one example, the activate interpolation command 213 is configured to transition the second power domain from a low power mode to an active mode to activate one or more components of the second stage. For example, the interpolation selector 208 may include or be coupled to a power management circuit, a clock circuit, a head switch or foot switch circuit, a buffer control circuit, or any combination thereof. The interpolation selector 208 may be configured to start powering on the second stage, such as by selectively applying or raising the voltage of the power supply of the second stage, the second power domain, or both. As another example, the interpolation selector 208 may be configured to selectively gate or not gate to the clock signal of the second stage, such as preventing or enabling circuit operation without removing power.

[0068] The interpolated frame 123 generated by the second stage is provided to the frame rate adjuster 140. The frame rate adjuster 140 is configured to output the interpolated frame 123 as an intermediate frame 191A. By selectively activating the second stage based on the results of processing the frame at the first stage of the frame rate adjuster 140, the overall power consumption associated with performing selective motion compensated frame interpolation can be reduced.

[0069] See also Figure 4 , showing that Figure 1 400 of an illustrative example of frames generated by the system 100. Example 402 indicates a constant playback speed of frames of a sequence of frames 192 (eg, one or more frames 101 interspersed with one or more intermediate frames 191).

[0070] The full interpolation example 404 indicates that each of the one or more intermediate frames 191 has half the similarity between the previous frame 101 and the next frame 101. For example, the frame rate adjuster 140 generates the intermediate frame 191A so that the second motion between the intermediate frame 191A and each of the frame 101A and the frame 101B corresponds to half of the first motion between the frame 101A and the frame 101B, as shown in FIG. Figure 1 The full interpolation example 404 represents applying the full interpolation value (eg, 0.5) of the interpolation weight (IW) 119 to the first motion to generate the intermediate frame 191A.

[0071] The semi-interpolation example 406 indicates that each of the one or more intermediate frames 191 is closer in similarity to the previous frame 101 than to the next frame 101. For example, the frame rate adjuster 140 generates the intermediate frame 191A so that the second motion between the intermediate frame 191A and the frame 101A corresponds to one quarter of the first motion, and the third motion between the intermediate frame 191A and the frame 101B corresponds to three quarters of the first motion. The semi-interpolation example 406 indicates that the semi-interpolation value (e.g., 0.25) of the interpolation weight (IW) 119 is applied to the first motion to generate the intermediate frame 191A.

[0072] No interpolation example 408 indicates that each of the one or more intermediate frames 191 is a copy of the previous frame 101. For example, frame rate adjuster 140 generates intermediate frame 191A as a copy of frame 101A. No interpolation example 408 indicates that duplicate frame 121 is generated as intermediate frame 191A.

[0073] The transition example 410 indicates that the earlier intermediate frame 191 has half the similarity between the previous frame 101 and the next frame 101, the middle intermediate frame 191 is closer to the previous frame 101 than the next frame 101, and the later intermediate frame 191 is a copy of the previous frame 101. For example, the frame rate adjuster 140 transitions via partial interpolation between full interpolation and copying to generate one or more intermediate frames 191. The transition example 410 indicates that the interpolation weight 119 is updated from a full interpolation value (e.g., 0.5) to a no interpolation value (e.g., 0) as the one or more intermediate frames 191 are generated. Compared to switching between full interpolation and copying, the transition between full interpolation to copying via partial interpolation during playback is less noticeable.

[0074] See also Figure 5 , shows illustrative aspects of the motion compensated frame interpolator 214. The motion compensated frame interpolator 214 includes a motion vector processor 502 coupled to a frame renderer 510 via an occlusion detector 504, a motion vector projector 506, a fallback analyzer 508, or a combination thereof.

[0075] In a first embodiment, the motion compensated frame interpolator 214 determines the interpolation weight 119 based on a predetermined weight (e.g., 0.5). For example, the motion compensated frame interpolator 214 performs interpolation (e.g., full interpolation) corresponding to the predetermined weight (e.g., 0.5) independently of the interpolation factor 207. In a second embodiment, the motion compensated frame interpolator 214 determines the interpolation weight 119 based at least in part on the interpolation factor 207. For example, the motion compensated frame interpolator 214 applies the predetermined factor (e.g., 0.5) to the interpolation factor 207 to determine the interpolation weight 119. For illustration, a full interpolation value (e.g., 1) of the interpolation factor 207 corresponds to a full interpolation value (e.g., 0.5) of the interpolation weight 119. A half interpolation value (e.g., 0.5) of the interpolation factor 207 corresponds to a half interpolation value (e.g., 0.25) of the interpolation weight 119.

[0076] In a particular aspect, when the interpolation factor 207 has a full interpolation value, the interpolation weight 119 has the same value (e.g., 0.5) in both the first embodiment and the second embodiment. When the interpolation factor 207 has a value other than the full interpolation value (e.g., a half interpolation value), the interpolation weight 119 has a different value in the second embodiment than in the first embodiment.

[0077] The motion vector processor 502 generates a motion vector 503 based on the motion data 205, the frame 101A, and the frame 101B. For example, the motion vector processor 502 performs motion vector post-processing and refinement to generate the motion vector 503. The motion vector processor 502 provides the motion vector 503 to the occlusion detector 504, the motion vector projector 506, the fallback analyzer 508, or a combination thereof.

[0078] The motion vector projector 506 generates motion vector data 507 based on the interpolation weights 119. For example, in response to determining that the first motion vector of the motion vector 503 indicates a first pixel shift (e.g., 2 pixel blocks) in a first direction (e.g., downward) between the frame 101A and the frame 101B for the region A, the motion vector projector 506 generates motion vector data 507 including a second motion vector indicating a second pixel shift (e.g., 1 pixel block) in the first direction between the frame 101A and the interpolated frame 123. The second pixel shift is based on applying the interpolation weights 119 (e.g., 0.5) to the first pixel shift (e.g., second pixel shift=interpolation weight 119*first pixel shift). The motion vector projector 506 provides the motion vector data 507 to the frame renderer 510.

[0079] In example 550, frame 101A includes region E. Movement of region E between frame 101A and frame 101B crosses paths with movement of region A between frame 101A and frame 101B. Motion vector data 507 indicates that region E overlaps region A, and both region E and region A are visible in interpolated frame 123. For example, one of region E or region A corresponds to glass of a window.

[0080] The occlusion detector 504 detects possible occlusion in response to determining that the motion vector 503, the motion vector data 507, or both indicate that the movement of region E between frame 101A and frame 101B crosses a path with the movement of region A between frame 101A and frame 101B. The occlusion detector 504 generates occlusion data 505 in response to detecting the possible occlusion. For example, the occlusion data 505 includes occlusion data 505A (e.g., motion vector) corresponding to region E, the occlusion data 505A partially blocking region A in the interpolated frame 123. As another example, the occlusion data 505 includes occlusion data 505B (e.g., motion vector) corresponding to region A, the occlusion data 505B at least partially blocking region A in the interpolated frame 123. The occlusion detector 504 provides the occlusion data 505 to the frame renderer 510.

[0081] The fallback analyzer 508 generates fallback data 509 corresponding to the fallback of the interpolated frame 123 to the frame 101A in response to determining that the motion vector 503, the motion vector data 507, or both indicate a count greater than a threshold value of crossing paths. For example, the fallback data 509 indicates that the interpolated frame 123 is a copy of the frame 101A. The fallback analyzer 508 provides the fallback data 509 to the frame renderer 510.

[0082] The frame renderer 510 generates the interpolated frame 123 based on the motion vector data 507, the occlusion data 505, the backoff data 509, the frame 101A, the frame 101B, or a combination thereof. For example, the frame renderer 510 applies the motion vector data 507 (e.g., a motion vector) to the frame 101A to generate the interpolated frame 123 in response to determining that the occlusion data 505 indicates that no possible occlusion is detected. Alternatively, the frame renderer 510 applies the occlusion data 505 to the frame 101A to generate the interpolated frame 123 in response to determining that the occlusion data 505 indicates that less than a threshold count of occlusions is detected. For example, the frame renderer 510 selects the occlusion data 505A or the occlusion data 505B using various occlusion resolution techniques, and applies the selected one of the occlusion data 505A or the occlusion data 505B to the frame 101A to generate the interpolated frame 123. In particular aspects, the frame renderer 510 copies the frame 101A as the interpolated frame 123 in response to determining that the backoff data 509 indicates that greater than or equal to the threshold count of occlusions was detected.

[0083] The motion compensated frame interpolator 214 can therefore generate the interpolated frame 123 based on the interpolation weights 119. The motion compensated frame interpolator 214 can adjust the interpolated frame 123 based on the detected occlusion, including falling back to the frame 101A.

[0084] Figure 6 An embodiment 600 of the device 102 is depicted as an integrated circuit 602 that includes one or more processors 190. The integrated circuit 602 also includes a signal input 604, such as one or more bus interfaces, to enable the frame sequence 180 to be received for processing. The integrated circuit 602 also includes a signal output 606, such as a bus interface, to enable output signals, such as the frame sequence 192 to be transmitted. The integrated circuit 602 enables selective motion compensated frame interpolation to be implemented as a component in a system, such as Figure 7 The depicted mobile phone or tablet, Figure 8 The wearable electronic devices depicted, Fig. 9 A virtual reality headset or an augmented reality headset as depicted, or Fig.10 or Fig.11 The vehicle depicted.

[0085] Figure 7An implementation 700 is depicted in which, as an illustrative, non-limiting example, the device 102 includes a mobile device 702, such as a phone or tablet. The mobile device 702 includes a display device 106 (e.g., a display screen). Components of one or more processors 190, including a frame rate adjuster 140, are integrated into the mobile device 702 and are shown using dashed lines to indicate internal components that are generally not visible to a user of the mobile device 702. In a specific example, the frame rate adjuster 140 operates to generate a frame sequence 192, which is then processed to perform one or more operations at the mobile device 702, such as initiating a graphical user interface or otherwise displaying the frame sequence 192.

[0086] Figure 8 An embodiment 800 is depicted in which the device 102 includes a wearable electronic device 802, which is shown as a "smart watch." The frame rate regulator 140 is integrated into the wearable electronic device 802. In a specific example, the frame rate regulator 140 operates to generate a frame sequence 192, which is then processed to perform one or more operations at the wearable electronic device 802, such as initiating a graphical user interface or otherwise displaying the frame sequence 192 at a display device 106 (e.g., a display screen) of the wearable electronic device 802. In a specific example, the wearable electronic device 802 includes a haptic device that provides a haptic notification (e.g., a vibration) in response to detecting that the frame sequence 192 is ready for display. For example, the haptic notification can enable a user to look at the wearable electronic device 802 to view the frame sequence 192. The wearable electronic device 802 can therefore alert a user with a hearing impairment or a user wearing a head-mounted viewer that video data has been detected.

[0087] Fig. 9 An embodiment 900 is depicted in which the device 102 includes a portable electronic device corresponding to a virtual reality, augmented reality, or mixed reality head-mounted view device 902. The frame rate regulator 140 is integrated into the head-mounted view device 902. A visual interface device (e.g., a display device 106) is placed in front of the user's eyes to enable the augmented reality or virtual reality image or scene to be displayed to the user while the head-mounted view device 902 is worn. In a specific example, the visual interface device is configured to display a frame sequence 192.

[0088] Fig.10An embodiment 1000 is depicted in which device 102 corresponds to or is integrated into a vehicle 1002, which is shown as a manned or unmanned aerial device (e.g., a package delivery drone). Frame rate regulator 140, display device 106 (e.g., a display screen), or both are integrated into vehicle 1002. Frame sequence 192 can be displayed on display device 106 for a recipient, such as for delivering a communication, an advertisement, installation instructions, or a combination thereof.

[0089] Fig.11 Another embodiment 1100 is depicted in which the device 102 corresponds to or is integrated into a vehicle 1102, which is shown as a car. The vehicle 1102 includes one or more processors 190, and the one or more processors 190 include a frame rate adjuster 140. In a particular embodiment, in response to generating a frame sequence 192 via operation of the frame rate adjuster 140, the frame sequence 192 is displayed via a display device 106 (e.g., a display screen).

[0090] See also Fig.12 , showing a specific implementation of the method 1200 for selective motion compensated frame interpolation. In certain aspects, one or more operations of the method 1200 are performed by Figure 1 The frame rate regulator 140, one or more processors 190, the device 102, the system 100, Figure 2 motion estimator 204, interpolation factor generator 206, interpolation selector 208, frame copier 210, motion compensated frame interpolator 214, Figure 5 The method may be performed by at least one of a motion vector processor 502, an occlusion detector 504, a motion vector projector 506, a fallback analyzer 508, a frame renderer 510, or a combination thereof.

[0091] The method 1200 includes obtaining, at 1202, motion data indicating estimated motion between a first frame and a second frame of an input sequence of image frames. For example, Figure 2 The motion estimator 204 obtains motion data 205 (e.g., motion vectors) indicating estimated motion between the frame 101A and the frame 101B of the frame sequence 180 (e.g., image frames, video frames, photo bursts, or a combination thereof), as shown in reference Figure 2 as described.

[0092] The method 1200 also includes identifying any frame regions of the first frame indicating motion greater than a motion threshold based on the motion data at 1204. For example, Figure 2 The interpolation factor generator 206 identifies any frame regions of the frame 101A that indicate motion greater than the motion threshold 111 based on the motion data 205, as shown in FIG. Figure 2 as described.

[0093] The method 1200 also includes determining a motion metric associated with the identified frame region based on the motion data at 1206. For example, Figure 2 The interpolation factor generator 206 determines the motion metric 115 associated with the identified frame region based on the motion data 205, as shown in FIG. Figure 2 as described.

[0094] The method 1200 also includes performing a determination at 1208 whether to generate an intermediate frame using motion compensated frame interpolation based on the motion metric and the size metric associated with the identified frame region. Figure 2 The interpolation factor generator 206 generates the interpolation factor 207 based on the motion threshold 111 and the size metric 113, as shown in FIG. Figure 2 The interpolation selector 208 performs a determination based on the interpolation factor 207 whether to use motion compensated frame interpolation to generate the intermediate frame 191A, as described in reference to FIG. Figure 2 as described.

[0095] The method 1200 also includes generating an intermediate frame based on the determination at 1210. For example, Figure 2 The interpolation selector 208 generates one of the activation copy command 209 or the activation interpolation command 213 based on the interpolation factor 207, as shown in FIG. Figure 2 In certain aspects, the frame duplicator 210 generates the duplicate frame 121 as the intermediate frame 191A in response to receiving the activate duplication command 209, as shown in FIG. Figure 2 In an alternative aspect, the motion compensated frame interpolator 214 generates the interpolated frame 123 as the intermediate frame 191A in response to receiving the activate interpolation command 213, as shown in FIG. Figure 2 as described.

[0096] The method 1200 also includes generating an output sequence of image frames including an intermediate frame between the first frame and the second frame at 1212. For example, Figure 1 The frame rate regulator 140 generates a frame sequence 192 including an intermediate frame 191A between the frame 101A and the frame 101B, as shown in FIG. Figure 1 as described.

[0097] The method 1200 is thus capable of performing selective motion compensated frame interpolation based on the size metric 113 (e.g., corresponding to the size of the higher motion region) and the motion metric 115 (e.g., corresponding to the degree of motion indicated by the higher motion region). For example, the frame rate adjuster 140 transitions between performing motion compensated frame interpolation to increase playback smoothness and performing frame duplication to save resources.

[0098] Fig.12The method 1200 may be implemented by a field programmable gate array (FPGA) device, an application specific integrated circuit (ASIC), a processing unit such as a central processing unit (CPU), a DSP, a controller, another hardware device, a firmware device, or any combination thereof. As an example, Fig.12 The method 1200 may be performed by a processor executing instructions, such as reference Fig.13 as described.

[0099] See also Fig.13 , depicts a block diagram of a particular illustrative implementation of a device and is generally referred to as 1300. In various implementations, the device 1300 may have more than Fig.13 In an illustrative embodiment, device 1300 may correspond to, include, or be included within device 102. In an illustrative embodiment, device 1300 may execute reference Figure 1-12 Describes one or more operations.

[0100] In certain embodiments, device 1300 includes a processor 1306 (e.g., a central processing unit (CPU)). Device 1300 may include one or more additional processors 1310 (e.g., one or more DSPs). In certain aspects, Figure 1 The one or more processors 190 correspond to processor 1306, processor 1310, or a combination thereof. Processor 1310 may include a speech and music encoder-decoder (CODEC) 1308, which includes a voice encoder ("vocoder") encoder 1336, a vocoder decoder 1338, or both. Processor 1310 includes a frame rate adjuster 140.

[0101] The device 1300 may include a memory 132 and a CODEC 1334. The memory 132 may include instructions 196 executable by one or more additional processors 1310 (or processor 1306) to implement the functionality described with reference to the frame rate adjuster 140. The device 1300 may include a modem 170 coupled to an antenna 1352 via a transceiver 1350.

[0102] The device 1300 may include a display device 106 coupled to a display controller 1326. One or more speakers 1392 and one or more microphones 1390 may be coupled to a CODEC 1334. The CODEC 1334 may include a digital-to-analog converter (DAC) 1302, an analog-to-digital converter (ADC) 1304, or both. In a particular embodiment, the CODEC 1334 may receive an analog signal from the one or more microphones 1390, convert the analog signal to a digital signal using the analog-to-digital converter 1304, and provide the digital signal to a voice and music codec 1308. The voice and music codec 1308 may process the digital signal. In a particular embodiment, the voice and music codec 1308 may provide the digital signal to the CODEC 1334. The CODEC 1334 may convert the digital signal to an analog signal using the digital-to-analog converter 1302, and may provide the analog signal to the one or more speakers 1392.

[0103] In a particular embodiment, device 1300 may be included in a system-in-package or system-on-chip device 1322. In a particular embodiment, memory 132, processor 1306, processor 1310, display controller 1326, CODEC 1334, and modem 170 are included in a system-in-package or system-on-chip device 1322. In a particular embodiment, input device 1330 and power supply 1344 are coupled to system-in-package or system-on-chip device 1322. Also, in a particular embodiment, as shown in FIG. Fig.13 As shown, the display device 106, input device 1330, one or more speakers 1392, one or more microphones 1390, antenna 1352, and power supply 1344 are external to the system-on-chip device 1322. In particular embodiments, each of the display device 106, input device 1330, one or more speakers 1392, one or more microphones 1390, antenna 1352, and power supply 1344 can be coupled to a component of the system-on-chip device 1322, such as an interface or controller.

[0104] Device 1300 may include a smart speaker, a speaker bar, a mobile communication device, a smart phone, a cellular phone, a laptop, a computer, a tablet, a personal digital assistant, a display device, a television, a game console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, a vehicle, a head-mounted view, an augmented reality head-mounted view, a virtual reality head-mounted view, an aerial vehicle, a home automation system, a voice-controlled device, a wireless speaker and voice-activated device, a portable electronic device, an automobile, a computing device, a communication device, an Internet of Things (IoT) device, a virtual reality (VR) device, a base station, a mobile device, or any combination thereof.

[0105] In conjunction with the described embodiments, an apparatus includes means for obtaining motion data indicating estimated motion between a first frame and a second frame of an input sequence of image frames. For example, the means for obtaining the motion data may correspond to Figure 1 The frame rate regulator 140, one or more processors 190, the device 102, the system 100, Figure 2 The motion estimator 204, the processor 1306, the processor 1310, one or more other circuits or components configured to obtain motion data, or any combination thereof.

[0106] The apparatus also includes means for identifying any frame regions of the first frame indicating motion greater than a motion threshold based on the motion data. For example, the means for identifying may correspond to Figure 1 The frame rate regulator 140, one or more processors 190, the device 102, the system 100, Figure 2 The interpolation factor generator 206, the processor 1306, the processor 1310, one or more other circuits or components, or any combination thereof, are configured to identify any frame regions indicating motion greater than a motion threshold.

[0107] The apparatus further includes means for determining a motion metric associated with the identified frame region based on the motion data. For example, the means for determining the motion metric may correspond to Figure 1 The frame rate regulator 140, one or more processors 190, the device 102, the system 100, Figure 2 The interpolation factor generator 206, the processor 1306, the processor 1310, one or more other circuits or components configured to determine a motion metric, or any combination thereof.

[0108] The apparatus also includes means for performing a determination of whether to use motion compensated frame interpolation to generate an intermediate frame based on a motion metric and a size metric associated with the identified frame region. For example, the means for performing a determination of whether to use motion compensated frame interpolation may correspond to Figure 1 The frame rate regulator 140, one or more processors 190, the device 102, the system 100, Figure 2 The interpolation factor generator 206, the interpolation selector 208, the processor 1306, the processor 1310, one or more other circuits or components configured to perform a determination of whether to use motion compensated frame interpolation, or any combination thereof.

[0109] The apparatus further includes a component for generating an intermediate frame based on the determination. For example, the component for generating the intermediate frame may correspond to Figure 1 Frame rate regulator 140, one or more processors 190, device 102, system 100, Figure 2The frame copier 210, the motion compensated frame interpolator 214, the processor 1306, the processor 1310, one or more other circuits or components configured to generate intermediate frames, or any combination thereof.

[0110] The apparatus further comprises means for generating an output sequence of image frames including an intermediate frame between the first frame and the second frame. For example, the means for generating the output sequence may correspond to Figure 1 The frame rate regulator 140, one or more processors 190, the device 102, the system 100, Figure 2 The frame copier 210, the motion compensated frame interpolator 214, the processor 1306, the processor 1310, one or more other circuits or components configured to generate an output sequence, or any combination thereof.

[0111] In certain aspects, the apparatus further comprises means for generating an interpolation factor based on the size metric and the motion metric, wherein the determination of whether to use motion compensated frame interpolation to generate the intermediate frame is based on whether the interpolation factor satisfies an interpolation criterion. For example, the means for generating the interpolation factor may correspond to Figure 1 The frame rate regulator 140, one or more processors 190, the device 102, the system 100, Figure 2 The interpolation factor generator 206, the processor 1306, the processor 1310, one or more other circuits or components configured to generate interpolation factors, or any combination thereof.

[0112] In certain aspects, the apparatus further comprises means for receiving an input sequence of image frames. For example, the means for receiving may correspond to Figure 1 modem 170, frame rate regulator 140, one or more processors 190, device 102, system 100, Figure 2 motion estimator 204, frame copier 210, motion compensated frame interpolator 214, processor 1306, processor 1310, transceiver 1350, antenna 1352, one or more other circuits or components configured to receive an input sequence of image frames, or any combination thereof.

[0113] In some embodiments, a non-transitory computer-readable medium (e.g., a computer-readable storage device such as memory 132) includes instructions (e.g., instructions 196) that, when executed by one or more processors (e.g., one or more processors 1310 or processor 1306), cause the one or more processors to obtain motion data (e.g., motion data 205) indicating estimated motion between a first frame (e.g., frame 101A) and a second frame (e.g., frame 101B) of an input sequence of image frames (e.g., frame sequence 180). When the instructions are executed by the one or more processors, the one or more processors are further caused to identify any frame regions of the first frame that indicate motion greater than a motion threshold (e.g., motion threshold 111) based on the motion data. When the instructions are executed by the one or more processors, the one or more processors are further caused to determine a motion metric (e.g., motion metric 115) associated with the identified frame region based on the motion data. When the instructions are executed by the one or more processors, the one or more processors are further caused to determine whether to use motion compensated frame interpolation to generate an intermediate frame (e.g., intermediate frame 191A) based on the motion metric and size metric (e.g., size metric 113) associated with the identified frame region. When the instructions are executed by one or more processors, the one or more processors are further caused to generate an intermediate frame based on the determination. When the instructions are executed by one or more processors, the one or more processors are further caused to generate an output sequence of image frames (e.g., frame sequence 192), the output sequence including the intermediate frame between the first frame and the second frame.

[0114] Particular aspects of the disclosure are described in the following groups of various related clauses:

[0115] According to clause 1, a device includes: a memory configured to store instructions; and one or more processors configured to execute the instructions to: obtain motion data indicating estimated motion between a first frame and a second frame of an input sequence of image frames; based on the motion data, identify any frame regions of the first frame indicating motion greater than a motion threshold; based on the motion data, determine a motion metric associated with the identified frame region; based on the motion metric and a size metric associated with the identified frame region, perform a determination whether to use motion compensated frame interpolation to generate an intermediate frame; generate the intermediate frame based on the determination; and generate an output sequence of image frames including the intermediate frame between the first frame and the second frame.

[0116] Clause 2 includes the apparatus of clause 1, wherein the motion metric is based on an average motion, a maximum motion, a range of motion, or a combination thereof associated with the identified frame region.

[0117] Clause 3 includes the apparatus of clause 1 or clause 2, wherein the motion metric is based on an average motion associated with the identified frame region.

[0118] Clause 4 includes the apparatus of any of clauses 1 to 3, wherein the motion metric is based on a maximum motion associated with the identified frame region.

[0119] Clause 5 includes the apparatus of any of clauses 1 to 4, wherein the motion metric is based on a range of motion associated with the identified frame region.

[0120] Clause 6 includes the apparatus of any of clauses 1 to 5, wherein the size metric is based on a combined size of the identified frame regions, a count of the identified frame regions, a percentage of the first frame that includes the identified frame regions, or a combination thereof.

[0121] Clause 7 includes the apparatus of any of clauses 1 to 6, wherein the size metric is based on a combined size of the identified frame regions.

[0122] Clause 8 includes the apparatus of any of clauses 1 to 7, wherein the size metric is based on a count of identified frame regions.

[0123] Clause 9 includes the apparatus of any of clauses 1 to 8, wherein the size metric is based on a percentage of the first frame that includes the identified frame region.

[0124] Clause 10 includes the apparatus of any of clauses 1 to 9, wherein the one or more processors are configured to generate interpolation factors based on a size metric and a motion metric, and wherein the determination of whether to use motion compensated frame interpolation to generate the intermediate frame is based on whether the interpolation factors satisfy an interpolation criterion.

[0125] Clause 11 includes the apparatus of clause 10, wherein the one or more processors are further configured to generate the interpolation factor based on a comparison of the size metric and the motion metric to the interpolation factor determination data.

[0126] Clause 12 includes the device of clause 11, wherein the interpolation factor determination data is based on configuration settings, default data, user input, detected context, operating mode, screen size, or a combination thereof.

[0127] Clause 13 includes the apparatus of clause 11 or clause 12, wherein the interpolation factor determination data is based on a configuration setting.

[0128] Clause 14 includes the apparatus of any one of clauses 11 to 13, wherein the interpolation factor determination data is based on default data.

[0129] Clause 15 includes the apparatus of any of clauses 11 to 14, wherein the interpolation factor determination data is based on user input.

[0130] Clause 16 includes the apparatus of any of clauses 11 to 15, wherein the interpolation factor determination data is based on a detected context.

[0131] Clause 17 includes the apparatus of any of clauses 11 to 16, wherein the interpolation factor determination data is based on an operating mode.

[0132] Clause 18 includes the apparatus of any one of clauses 11 to 17, wherein the interpolation factor determination data is based on a screen size.

[0133] Clause 19 includes the apparatus of any of clauses 11 to 18, wherein the interpolation factor determination data indicates a plurality of interpolation factor regions defined by a range of size metric values ​​and a range of motion metric values, wherein each of the plurality of interpolation factor regions corresponds to a specific interpolation factor value, and wherein the one or more processors are further configured to generate an interpolation factor having an interpolation factor value corresponding to a specific interpolation factor region based on determining that the motion metric and the size metric are included in a specific interpolation factor region of the plurality of interpolation factor regions.

[0134] Clause 20 includes the apparatus of any of clauses 1 to 9, wherein the one or more processors are configured to generate interpolation factors based on the size metric, and wherein the determination of whether to use motion compensated frame interpolation to generate the intermediate frame is based on whether the interpolation factors satisfy an interpolation criterion.

[0135] Clause 21 includes the apparatus of clause 20, wherein the one or more processors are further configured to generate the interpolation factor based on a comparison of the size metric and the interpolation factor determination data.

[0136] Clause 22 includes the device of clause 21, wherein the interpolation factor determination data is based on configuration settings, default data, user input, detected context, operating mode, screen size, or a combination thereof.

[0137] Clause 23 includes the apparatus of clause 21 or clause 22, wherein the interpolation factor determination data is based on a configuration setting.

[0138] Clause 24 includes the apparatus of any one of clauses 21 to 23, wherein the interpolation factor determination data is based on default data.

[0139] Clause 25 includes the apparatus of any of clauses 21 to 24, wherein the interpolation factor determination data is based on user input.

[0140] Clause 26 includes the apparatus of any of clauses 21 to 25, wherein the interpolation factor determination data is based on a detected context.

[0141] Clause 27 includes the apparatus of any of clauses 21 to 26, wherein the interpolation factor determination data is based on an operating mode.

[0142] Clause 28 includes the apparatus of any one of clauses 21 to 27, wherein the interpolation factor determination data is based on a screen size.

[0143] Clause 29 includes the apparatus of any of clauses 21 to 28, wherein the interpolation factor determination data indicates a plurality of interpolation factor regions defined at least by a range of size metric values, wherein each of the plurality of interpolation factor regions corresponds to a specific interpolation factor value, and wherein the one or more processors are further configured to generate an interpolation factor having an interpolation factor value corresponding to a specific interpolation factor region based on determining that the size metric is included in the specific interpolation factor region of the plurality of interpolation factor regions.

[0144] Clause 30 includes the apparatus of any of clauses 1 to 9, wherein the one or more processors are configured to generate interpolation factors based on a motion metric, and wherein the determination of whether to use motion compensated frame interpolation to generate the intermediate frame is based on whether the interpolation factors satisfy an interpolation criterion.

[0145] Clause 31 includes the apparatus of clause 30, wherein the one or more processors are further configured to generate the interpolation factor based on a comparison of the motion metric and the interpolation factor determination data.

[0146] Clause 32 includes the device of clause 31, wherein the interpolation factor determination data is based on configuration settings, default data, user input, detected context, operating mode, screen size, or a combination thereof.

[0147] Clause 33 includes the apparatus of clause 31 or clause 32, wherein the interpolation factor determination data is based on a configuration setting.

[0148] Clause 34 includes the apparatus of any one of clauses 31 to 33, wherein the interpolation factor determination data is based on default data.

[0149] Clause 35 includes the apparatus of any of clauses 31 to 34, wherein the interpolation factor determination data is based on user input.

[0150] Clause 36 includes the apparatus of any of clauses 31 to 35, wherein the interpolation factor determination data is based on a detected context.

[0151] Clause 37 includes the apparatus of any of clauses 31 to 36, wherein the interpolation factor determination data is based on an operating mode.

[0152] Clause 38 includes the apparatus of any one of clauses 31 to 37, wherein the interpolation factor determination data is based on a screen size.

[0153] Clause 39 includes the apparatus of any of clauses 31 to 38, wherein the interpolation factor determination data indicates a plurality of interpolation factor regions defined at least by a range of motion metric values, wherein each of the plurality of interpolation factor regions corresponds to a specific interpolation factor value, and wherein the one or more processors are further configured to generate an interpolation factor having an interpolation factor value corresponding to a specific interpolation factor region based on determining that the motion metric is included in the specific interpolation factor region of the plurality of interpolation factor regions.

[0154] Clause 40 includes the apparatus of any of clauses 1 to 39, wherein the one or more processors are configured to generate the intermediate frame using motion compensated frame interpolation in response to determining that the interpolation factor satisfies the interpolation criterion.

[0155] Clause 41 includes the apparatus of any of clauses 1 to 40, wherein the one or more processors are configured to generate the intermediate frame using an alternative to motion compensated frame interpolation in response to determining that the interpolation factor fails to satisfy the interpolation criterion.

[0156] Clause 42 includes a device of any of clauses 1 to 41, wherein one or more processors are configured to generate the intermediate frame in response to a determination that motion compensated frame interpolation will be used to generate an intermediate frame, and based on a determination that motion data indicates a first motion between a first frame and a second frame, such that the second motion between the first frame and the intermediate frame is based on the first motion and an interpolation factor.

[0157] Clause 43 includes a device of any of clauses 1 to 41, wherein one or more processors are configured to generate the intermediate frame in response to a determination that motion compensated frame interpolation will be used to generate an intermediate frame, and based on a determination that motion data indicates a first motion between a first frame and a second frame, generate the intermediate frame, such that the second motion between the first frame and the intermediate frame is based on a predetermined weight applied to the first motion.

[0158] Clause 44 includes the apparatus of any of clauses 1 to 43, wherein the one or more processors are configured to generate the intermediate frame as a copy of one of the first frame or the second frame in response to a determination that motion compensated frame interpolation will not be used to generate the intermediate frame.

[0159] Clause 45 includes the apparatus of any of clauses 1 to 44, further comprising a modem configured to receive an input sequence of image frames.

[0160] According to clause 46, a method includes: obtaining motion data at a device indicating estimated motion between a first frame and a second frame of an input sequence of image frames; based on the motion data, identifying any frame regions of the first frame indicating motion greater than a motion threshold; based on the motion data, determining a motion metric associated with the identified frame region; based on the motion metric and a size metric associated with the identified frame region, performing a determination whether to use motion compensated frame interpolation to generate an intermediate frame; generating an intermediate frame at the device based on the determination; and generating at the device an output sequence of image frames including the intermediate frames between the first frame and the second frame.

[0161] Clause 47 includes the method of clause 46, wherein the motion metric is based on an average motion, a maximum motion, a range of motion, or a combination thereof associated with the identified frame region.

[0162] Clause 48 includes the method of clause 46 or clause 47, wherein the motion metric is based on an average motion associated with the identified frame region.

[0163] Clause 49 includes the method of any of clauses 46 to 48, wherein the motion metric is based on a maximum motion associated with the identified frame region.

[0164] Clause 50 includes the method of any of clauses 46 to 49, wherein the motion metric is based on a range of motion associated with the identified frame region.

[0165] Clause 51 includes the method of any of clauses 46 to 50, wherein the size metric is based on a combined size of the identified frame regions, a count of the identified frame regions, a percentage of the first frame that includes the identified frame regions, or a combination thereof.

[0166] Clause 52 includes the method of any of clauses 46 to 51, wherein the size metric is based on a combined size of the identified frame regions.

[0167] Clause 53 includes the method of any of clauses 46 to 52, wherein the size metric is based on a count of identified frame regions.

[0168] Clause 54 includes the method of any of clauses 46 to 53, wherein the size metric is based on a percentage of the first frame that includes the identified frame region.

[0169] Clause 55 includes the method of any of clauses 46 to 54, further comprising generating, at the device, interpolation factors based on the size metric and the motion metric, wherein the determination of whether to use motion compensated frame interpolation to generate the intermediate frame is based on whether the interpolation factors satisfy an interpolation criterion.

[0170] Clause 56 includes the method of clause 55, wherein the interpolation factor is based on a comparison of the size metric and the motion metric to the interpolation factor determination data.

[0171] Clause 57 includes the method of clause 56, wherein the interpolation factor determination data is based on configuration settings, default data, user input, detected context, operating mode, screen size, or a combination thereof.

[0172] Clause 58 includes the method of clause 56 or clause 57, wherein the interpolation factor determination data is based on a configuration setting.

[0173] Clause 59 includes the method of any one of clauses 56 to 58, wherein the interpolation factor determination data is based on default data.

[0174] Clause 60 includes the method of any one of clauses 56 to 59, wherein the interpolation factor determination data is based on user input.

[0175] Clause 61 includes the method of any of clauses 56 to 60, wherein the interpolation factor determination data is based on a detected context.

[0176] Clause 62 includes the method of any one of clauses 56 to 61, wherein the interpolation factor determination data is based on an operating mode.

[0177] Clause 63 includes the method of any one of clauses 56 to 62, wherein the interpolation factor determination data is based on a screen size.

[0178] Clause 64 includes the method of any one of clauses 56 to 63, wherein the interpolation factor determination data indicates a plurality of interpolation factor regions defined by a range of size metric values ​​and a range of motion metric values, wherein each of the plurality of interpolation factor regions corresponds to a specific interpolation factor value, wherein the motion metric and the size metric are included in a specific interpolation factor region of the plurality of interpolation factor regions, and wherein the interpolation factor has an interpolation factor value corresponding to the specific interpolation factor region.

[0179] Clause 65 includes the method of any of clauses 55 to 64, further comprising, in response to determining that the interpolation factor satisfies the interpolation criterion, generating the intermediate frame using motion compensated frame interpolation.

[0180] Clause 66 includes the method of any of clauses 55 to 65, further comprising, in response to determining that the interpolation factor fails to satisfy the interpolation criterion, generating the intermediate frame using an alternative to motion compensated frame interpolation.

[0181] Clause 67 includes the method of any of clauses 55 to 66, further comprising generating the intermediate frame in response to a determination that motion compensated frame interpolation will be used to generate an intermediate frame, and generating the intermediate frame based on a determination that motion data indicates a first motion between a first frame and a second frame, such that the second motion between the first frame and the intermediate frame is based on the first motion and an interpolation factor.

[0182] Clause 68 includes the method of any of clauses 46 to 54, further comprising generating, at the device, interpolation factors based on the size metric, wherein the determination of whether to use motion compensated frame interpolation to generate the intermediate frame is based on whether the interpolation factors satisfy an interpolation criterion.

[0183] Clause 69 includes the method of clause 68, wherein the interpolation factor is based on a comparison of the size metric to the interpolation factor determination data.

[0184] Clause 70 includes the method of clause 69, wherein the interpolation factor determination data is based on configuration settings, default data, user input, detected context, operating mode, screen size, or a combination thereof.

[0185] Clause 71 includes the method of clause 69 or clause 70, wherein the interpolation factor determination data is based on a configuration setting.

[0186] Clause 72 includes the method of any one of clauses 69 to 71, wherein the interpolation factor determination data is based on default data.

[0187] Clause 73 includes the method of any one of clauses 69 to 72, wherein the interpolation factor determination data is based on user input.

[0188] Clause 74 includes the method of any of clauses 69 to 73, wherein the interpolation factor determination data is based on a detected context.

[0189] Clause 75 includes the method of any of clauses 69 to 74, wherein the interpolation factor determination data is based on an operating mode.

[0190] Clause 76 includes the method of any one of clauses 69 to 75, wherein the interpolation factor determination data is based on a screen size.

[0191] Clause 77 includes the method of any of clauses 69 to clause 76, wherein the interpolation factor determination data indicates a plurality of interpolation factor regions defined at least by a range of size metric values, wherein each of the plurality of interpolation factor regions corresponds to a specific interpolation factor value, wherein the size metric is included in a specific interpolation factor region of the plurality of interpolation factor regions, and wherein the interpolation factor has an interpolation factor value corresponding to the specific interpolation factor region.

[0192] Clause 78 includes the method of any of clauses 46 to 54, further comprising generating, at the device, interpolation factors based on the motion metric, wherein the determination of whether to use motion compensated frame interpolation to generate the intermediate frame is based on whether the interpolation factors satisfy an interpolation criterion.

[0193] Clause 79 includes the method of clause 78, wherein the interpolation factor is based on a comparison of the motion metric and the interpolation factor determination data.

[0194] Clause 80 includes the method of clause 79, wherein the interpolation factor determination data is based on configuration settings, default data, user input, detected context, operating mode, screen size, or a combination thereof.

[0195] Clause 81 includes the method of clause 79 or clause 80, wherein the interpolation factor determination data is based on a configuration setting.

[0196] Clause 82 includes the method of any one of clauses 79 to 81, wherein the interpolation factor determination data is based on default data.

[0197] Clause 83 includes the method of any one of clauses 79 to 82, wherein the interpolation factor determination data is based on user input.

[0198] Clause 84 includes the method of any of clauses 79 to 83, wherein the interpolation factor determination data is based on a detected context.

[0199] Clause 85 includes the method of any one of clauses 79 to 84, wherein the interpolation factor determination data is based on an operating mode.

[0200] Clause 86 includes the method of any one of clauses 79 to 85, wherein the interpolation factor determination data is based on a screen size.

[0201] Clause 87 includes the method of any of clauses 79 to 86, wherein the interpolation factor determination data indicates a plurality of interpolation factor regions defined at least by a range of motion metric values, wherein each of the plurality of interpolation factor regions corresponds to a specific interpolation factor value, wherein the motion metric is included in a specific interpolation factor region of the plurality of interpolation factor regions, and wherein the interpolation factor has an interpolation factor value corresponding to the specific interpolation factor region.

[0202] Clause 88 includes the method of any of clauses 46 to 87, further comprising, in response to determining that the interpolation factor satisfies the interpolation criterion, generating the intermediate frame using motion compensated frame interpolation.

[0203] Clause 89 includes the method of any of clauses 46 to 88, further comprising, in response to determining that the interpolation factor fails to satisfy the interpolation criterion, generating the intermediate frame using an alternative to motion compensated frame interpolation.

[0204] Clause 90 includes the method of any one of clauses 46 to clause 89, further comprising generating the intermediate frame in response to a determination that motion compensated frame interpolation will be used to generate an intermediate frame, and generating the intermediate frame based on a determination that motion data indicates a first motion between a first frame and a second frame, such that the second motion between the first frame and the intermediate frame is based on the first motion and an interpolation factor.

[0205] Clause 91 includes the method of any one of clauses 46 to clause 89, further comprising generating the intermediate frame in response to a determination that motion compensated frame interpolation will be used to generate an intermediate frame, and generating the intermediate frame based on a determination that motion data indicates a first motion between a first frame and a second frame, such that the second motion between the first frame and the intermediate frame is based on a predetermined weight applied to the first motion.

[0206] Clause 92 includes the method of any of clauses 46 to 91, further comprising generating the intermediate frame as a copy of one of the first frame or the second frame in response to a determination that motion compensated frame interpolation is not to be used to generate the intermediate frame.

[0207] According to clause 93, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to: obtain motion data indicating estimated motion between a first frame and a second frame of an input sequence of image frames; based on the motion data, identify any frame regions of the first frame that indicate motion greater than a motion threshold; based on the motion data, determine a motion metric associated with the identified frame region; based on the motion metric and a size metric associated with the identified frame region, perform a determination whether to use motion compensated frame interpolation to generate an intermediate frame; generate the intermediate frame based on the determination; and generate an output sequence of image frames including the intermediate frame between the first frame and the second frame.

[0208] Clause 94 includes the non-transitory computer-readable medium of clause 93, wherein the motion metric is based on an average motion, a maximum motion, a range of motion, or a combination thereof associated with the identified frame region.

[0209] Clause 95 includes the non-transitory computer-readable medium of clause 93 or clause 94, wherein the motion metric is based on an average motion associated with the identified frame region.

[0210] Clause 96 includes the non-transitory computer-readable medium of any of clauses 93 to 95, wherein the motion metric is based on a maximum motion associated with the identified frame region.

[0211] Clause 97 includes the non-transitory computer-readable medium of any of clauses 93 to 96, wherein the motion metric is based on a range of motion associated with the identified frame region.

[0212] Clause 98 includes the non-transitory computer-readable medium of any of clauses 93 to 97, wherein the size metric is based on a combined size of the identified frame regions, a count of the identified frame regions, a percentage of the first frame that includes the identified frame regions, or a combination thereof.

[0213] Clause 99 includes the non-transitory computer-readable medium of any of clauses 93 to 98, wherein the size metric is based on a combined size of the identified frame regions.

[0214] Clause 100 includes the non-transitory computer-readable medium of any of clauses 93 to 99, wherein the size metric is based on a count of identified frame regions.

[0215] Clause 101 includes the non-transitory computer-readable medium of any of clauses 93 to 100, wherein the size metric is based on a percentage of the first frame that includes the identified frame region.

[0216] Item 102 includes a non-transitory computer-readable medium of any one of items 93 to 101, wherein when the instructions are executed by one or more processors, the one or more processors are further caused to generate an interpolation factor based on a size metric and a motion metric, wherein a determination of whether to use motion compensated frame interpolation to generate an intermediate frame is based on whether the interpolation factor satisfies an interpolation criterion.

[0217] Clause 103 includes the non-transitory computer-readable medium of clause 102, wherein the interpolation factor is based on a comparison of the size metric and the motion metric to the interpolation factor determination data.

[0218] Clause 104 includes the non-transitory computer-readable medium of clause 103, wherein the interpolation factor determination data is based on configuration settings, default data, user input, detected context, operating mode, screen size, or a combination thereof.

[0219] Clause 105 includes the non-transitory computer-readable medium of clause 103 or clause 104, wherein the interpolation factor determination data is based on a configuration setting.

[0220] Clause 106 includes the non-transitory computer-readable medium of any of clauses 103 to 105, wherein the interpolation factor determination data is based on default data.

[0221] Clause 107 includes the non-transitory computer-readable medium of any of clauses 103 to 106, wherein the interpolation factor determination data is based on user input.

[0222] Clause 108 includes the non-transitory computer-readable medium of any of clauses 103 to 107, wherein the interpolation factor determination data is based on a detected context.

[0223] Clause 109 includes the non-transitory computer-readable medium of any of clauses 103 to 108, wherein the interpolation factor determination data is based on an operating mode.

[0224] Clause 110 includes the non-transitory computer-readable medium of any of clauses 103 to 109, wherein the interpolation factor determination data is based on a screen size.

[0225] Clause 111 includes a non-transitory computer-readable medium of any one of clauses 103 to 110, wherein the interpolation factor determination data indicates a plurality of interpolation factor regions defined by a range of size metric values ​​and a range of motion metric values, wherein each of the plurality of interpolation factor regions corresponds to a particular interpolation factor value, wherein the motion metric and the size metric are included in a particular interpolation factor region of the plurality of interpolation factor regions, and wherein the interpolation factor has an interpolation factor value corresponding to the particular interpolation factor region.

[0226] Clause 112 includes the non-transitory computer-readable medium of any of clauses 102 to 111, wherein the instructions, when executed by one or more processors, further cause the one or more processors to generate an intermediate frame using motion compensated frame interpolation in response to determining that the interpolation factor satisfies the interpolation criterion.

[0227] Clause 113 includes the non-transitory computer-readable medium of any of clauses 102 to 112, wherein the instructions, when executed by one or more processors, further cause the one or more processors to generate an intermediate frame using an alternative to motion compensated frame interpolation in response to determining that the interpolation factor fails to satisfy the interpolation criterion.

[0228] Clause 114 includes a non-transitory computer-readable medium of any of clauses 102 to 113, wherein when the instructions are executed by one or more processors, further cause the one or more processors to generate the intermediate frame in response to a determination that motion compensated frame interpolation will be used to generate an intermediate frame, and based on a determination that motion data indicates a first motion between a first frame and a second frame, generate the intermediate frame, such that the second motion between the first frame and the intermediate frame is based on the first motion and an interpolation factor.

[0229] Clause 115 includes a non-transitory computer-readable medium of any of clauses 93 to 101, wherein the instructions, when executed by one or more processors, also cause the one or more processors to generate an interpolation factor based on the size metric, wherein a determination of whether to use motion compensated frame interpolation to generate an intermediate frame is based on whether the interpolation factor satisfies an interpolation criterion.

[0230] Clause 116 includes the non-transitory computer-readable medium of clause 115, wherein the interpolation factor is based on a comparison of the size metric to the interpolation factor determination data.

[0231] Clause 117 includes the non-transitory computer-readable medium of clause 116, wherein the interpolation factor determination data is based on configuration settings, default data, user input, detected context, operating mode, screen size, or a combination thereof.

[0232] Clause 118 includes the non-transitory computer-readable medium of clause 116 or clause 117, wherein the interpolation factor determination data is based on a configuration setting.

[0233] Clause 119 includes the non-transitory computer-readable medium of any of clauses 116 to 118, wherein the interpolation factor determination data is based on default data.

[0234] Clause 120 includes the non-transitory computer-readable medium of any of clauses 116 to 119, wherein the interpolation factor determination data is based on user input.

[0235] Clause 121 includes the non-transitory computer-readable medium of any of clauses 116 to 120, wherein the interpolation factor determination data is based on a detected context.

[0236] Clause 122 includes the non-transitory computer-readable medium of any of clauses 116 to 121 , wherein the interpolation factor determination data is based on an operating mode.

[0237] Clause 123 includes the non-transitory computer-readable medium of any of clauses 116 to 122, wherein the interpolation factor determination data is based on a screen size.

[0238] Clause 124 includes a non-transitory computer-readable medium of any of clauses 116 to 123, wherein the interpolation factor determination data indicates a plurality of interpolation factor regions defined at least by a range of size metric values, wherein each of the plurality of interpolation factor regions corresponds to a particular interpolation factor value, wherein the size metric is included in a particular interpolation factor region of the plurality of interpolation factor regions, and wherein the interpolation factor has an interpolation factor value corresponding to the particular interpolation factor region.

[0239] Item 125 includes a non-transitory computer-readable medium of any one of items 93 to 124, wherein the instructions, when executed by one or more processors, also cause the one or more processors to generate an interpolation factor based on a motion metric, wherein a determination of whether to use motion compensated frame interpolation to generate an intermediate frame is based on whether the interpolation factor satisfies an interpolation criterion.

[0240] Clause 126 includes the non-transitory computer-readable medium of clause 125, wherein the interpolation factor is based on a comparison of the motion metric to the interpolation factor determination data.

[0241] Clause 127 includes the non-transitory computer-readable medium of clause 126, wherein the interpolation factor determination data is based on configuration settings, default data, user input, detected context, operating mode, screen size, or a combination thereof.

[0242] Clause 128 includes the non-transitory computer-readable medium of clause 126 or clause 127, wherein the interpolation factor determination data is based on a configuration setting.

[0243] Clause 129 includes the non-transitory computer-readable medium of any of clauses 126 to 128, wherein the interpolation factor determination data is based on default data.

[0244] Clause 130 includes the non-transitory computer-readable medium of any of clauses 126 to 129, wherein the interpolation factor determination data is based on user input.

[0245] Clause 131 includes the non-transitory computer-readable medium of any of clauses 126 to 130, wherein the interpolation factor determination data is based on a detected context.

[0246] Clause 132 includes the non-transitory computer-readable medium of any of clauses 126 to 131 , wherein the interpolation factor determination data is based on an operating mode.

[0247] Clause 133 includes the non-transitory computer-readable medium of any of clauses 126 to 132, wherein the interpolation factor determination data is based on a screen size.

[0248] Clause 134 includes a non-transitory computer-readable medium of any of clauses 126 to 133, wherein the interpolation factor determination data indicates a plurality of interpolation factor regions defined at least by a range of motion metric values, wherein each of the plurality of interpolation factor regions corresponds to a particular interpolation factor value, wherein the motion metric is included in a particular interpolation factor region of the plurality of interpolation factor regions, and wherein the interpolation factor has an interpolation factor value corresponding to the particular interpolation factor region.

[0249] Clause 135 includes the non-transitory computer-readable medium of any of clauses 93 to 134, further comprising, in response to determining that the interpolation factor satisfies the interpolation criterion, generating the intermediate frame using motion compensated frame interpolation.

[0250] Clause 136 includes the non-transitory computer-readable medium of any of clauses 93 to 135, further comprising, in response to determining that the interpolation factor fails to satisfy the interpolation criterion, generating the intermediate frame using an alternative to motion compensated frame interpolation.

[0251] Clause 137 includes a non-transitory computer-readable medium of any of clauses 93 to 136, further comprising generating the intermediate frame in response to a determination that motion compensated frame interpolation will be used to generate an intermediate frame, and generating the intermediate frame based on a determination that motion data indicates a first motion between a first frame and a second frame, such that the second motion between the first frame and the intermediate frame is based on the first motion and an interpolation factor.

[0252] Clause 138 includes a non-transitory computer-readable medium of any of clauses 93 to 136, wherein the instructions, when executed by one or more processors, further cause the one or more processors to generate the intermediate frame in response to a determination that motion compensated frame interpolation will be used to generate an intermediate frame, and based on a determination that motion data indicates a first motion between a first frame and a second frame, generate the intermediate frame such that the second motion between the first frame and the intermediate frame is based on a predetermined weight applied to the first motion.

[0253] Clause 139 includes a non-transitory computer-readable medium of any of clauses 93 to 138, wherein the instructions, when executed by one or more processors, also cause the one or more processors to generate an intermediate frame as a copy of one of the first frame or the second frame in response to a determination that motion compensated frame interpolation will not be used to generate the intermediate frame.

[0254] According to clause 140, an apparatus comprises: a component for obtaining motion data indicating estimated motion between a first frame and a second frame of an input sequence of image frames; a component for identifying any frame regions of the first frame indicating motion greater than a motion threshold based on the motion data; a component for determining a motion metric associated with the identified frame region based on the motion metric and a size metric associated with the identified frame region; a component for performing a determination of whether to use motion compensated frame interpolation to generate an intermediate frame based on the determination; a component for generating the intermediate frame based on the determination; and a component for generating an output sequence of image frames including the intermediate frames between the first frame and the second frame.

[0255] Clause 141 includes the apparatus of clause 140, wherein the motion metric is based on an average motion, a maximum motion, a range of motion, or a combination thereof associated with the identified frame region.

[0256] Clause 142 includes the apparatus of clause 140 or clause 141, wherein the motion metric is based on an average motion associated with the identified frame region.

[0257] Clause 143 includes the apparatus of any of clauses 140 to 142, wherein the motion metric is based on a maximum motion associated with the identified frame region.

[0258] Clause 144 includes the apparatus of any of clauses 140 to 143, wherein the motion metric is based on a range of motion associated with the identified frame region.

[0259] Clause 145 includes the apparatus of any of clauses 140 to 144, wherein the size metric is based on a combined size of the identified frame regions, a count of the identified frame regions, a percentage of the first frame that includes the identified frame regions, or a combination thereof.

[0260] Clause 146 includes the apparatus of any of clauses 140 to 145, wherein the size metric is based on a combined size of the identified frame regions.

[0261] Clause 147 includes the apparatus of any of clauses 140 to 146, wherein the size metric is based on a count of identified frame regions.

[0262] Clause 148 includes the apparatus of any of clauses 140 to 147, wherein the size metric is based on a percentage of the first frame that includes the identified frame region.

[0263] Clause 149 includes the apparatus of any of clauses 140 to 148, further comprising means for generating interpolation factors based on the size metric and the motion metric, wherein the determination of whether to use motion compensated frame interpolation to generate the intermediate frame is based on whether the interpolation factors satisfy an interpolation criterion.

[0264] Clause 150 includes the apparatus of clause 149, wherein the interpolation factor is based on a comparison of the size metric and the motion metric to the interpolation factor determination data.

[0265] Clause 151 includes the apparatus of clause 150, wherein the interpolation factor determination data is based on configuration settings, default data, user input, detected context, operating mode, screen size, or a combination thereof.

[0266] Clause 152 includes the apparatus of clause 150 or clause 151 , wherein the interpolation factor determination data is based on a configuration setting.

[0267] Clause 153 includes the apparatus of any of clauses 150 to 152, wherein the interpolation factor determination data is based on default data.

[0268] Clause 154 includes the apparatus of any of clauses 150 to 153, wherein the interpolation factor determination data is based on user input.

[0269] Clause 155 includes the apparatus of any of clauses 150 to 154, wherein the interpolation factor determination data is based on a detected context.

[0270] Clause 156 includes the apparatus of any of clauses 150 to 155, wherein the interpolation factor determination data is based on an operating mode.

[0271] Clause 157 includes the apparatus of any of clauses 150 to 156, wherein the interpolation factor determination data is based on a screen size.

[0272] Clause 158 includes the apparatus of any one of clauses 150 to 157, wherein the interpolation factor determination data indicates a plurality of interpolation factor regions defined by a range of size metric values ​​and a range of motion metric values, wherein each of the plurality of interpolation factor regions corresponds to a specific interpolation factor value, and wherein the means for generating the interpolation factor generates an interpolation factor having an interpolation factor value corresponding to a specific interpolation factor region based on determining that the motion metric and the size metric are included in a specific interpolation factor region of the plurality of interpolation factor regions.

[0273] Clause 159 includes the apparatus of any of clauses 149 to 158, wherein the means for generating the intermediate frame generates the intermediate frame using motion compensated frame interpolation in response to determining that the interpolation factor satisfies the interpolation criterion.

[0274] Clause 160 includes the apparatus of any of clauses 149 to 159, wherein the means for generating the intermediate frame generates the intermediate frame using an alternative motion compensated frame interpolation in response to determining that the interpolation factor fails to satisfy the interpolation criterion.

[0275] Clause 161 includes an apparatus of any of clauses 149 to 160, wherein the component for generating an intermediate frame is responsive to a determination that motion compensated frame interpolation will be used to generate the intermediate frame, and generates the intermediate frame based on a determination that motion data indicates a first motion between a first frame and a second frame, such that the second motion between the first frame and the intermediate frame is based on the first motion and an interpolation factor.

[0276] Clause 162 includes the apparatus of any of clauses 140 to 148, further comprising means for generating interpolation factors based on the size metric, wherein the determination of whether to use motion compensated frame interpolation to generate the intermediate frame is based on whether the interpolation factors satisfy an interpolation criterion.

[0277] Clause 163 includes the apparatus of clause 162, wherein the interpolation factor is based on a comparison of the size metric to the interpolation factor determination data.

[0278] Clause 164 includes the apparatus of clause 163, wherein the interpolation factor determination data is based on configuration settings, default data, user input, detected context, operating mode, screen size, or a combination thereof.

[0279] Clause 165 includes the apparatus of clause 163 or clause 164, wherein the interpolation factor determination data is based on a configuration setting.

[0280] Clause 166 includes the apparatus of any of clauses 163 to 165, wherein the interpolation factor determination data is based on default data.

[0281] Clause 167 includes the apparatus of any of clauses 163 to 166, wherein the interpolation factor determination data is based on user input.

[0282] Clause 168 includes the apparatus of any of clauses 163 to 167, wherein the interpolation factor determination data is based on a detected context.

[0283] Clause 169 includes the apparatus of any of clauses 163 to 168, wherein the interpolation factor determination data is based on an operating mode.

[0284] Clause 170 includes the apparatus of any of clauses 163 to 169, wherein the interpolation factor determination data is based on a screen size.

[0285] Clause 171 includes an apparatus of any of clauses 163 to 170, wherein the interpolation factor determination data indicates a plurality of interpolation factor regions defined at least by a range of size metric values, wherein each of the plurality of interpolation factor regions corresponds to a particular interpolation factor value, wherein the size metric is included in a particular interpolation factor region of the plurality of interpolation factor regions, and wherein the interpolation factor has an interpolation factor value corresponding to the particular interpolation factor region.

[0286] Clause 172 includes the apparatus of any of clauses 140 to 148, further comprising generating, at the device, interpolation factors based on the motion metric, wherein the determination of whether to use motion compensated frame interpolation to generate the intermediate frame is based on whether the interpolation factors satisfy an interpolation criterion.

[0287] Clause 173 includes the apparatus of clause 172, wherein the interpolation factor is based on a comparison of the motion metric and the interpolation factor determination data.

[0288] Clause 174 includes the apparatus of clause 173, wherein the interpolation factor determination data is based on configuration settings, default data, user input, detected context, operating mode, screen size, or a combination thereof.

[0289] Clause 175 includes the apparatus of clause 173 or clause 174, wherein the interpolation factor determination data is based on a configuration setting.

[0290] Clause 176 includes the apparatus of any of clauses 173 to 175, wherein the interpolation factor determination data is based on default data.

[0291] Clause 177 includes the apparatus of any of clauses 173 to 176, wherein the interpolation factor determination data is based on user input.

[0292] Clause 178 includes the apparatus of any of clauses 173 to 177, wherein the interpolation factor determination data is based on a detected context.

[0293] Clause 179 includes the apparatus of any of clauses 173 to 178, wherein the interpolation factor determination data is based on an operating mode.

[0294] Clause 180 includes the apparatus of any of clauses 173 to 179, wherein the interpolation factor determination data is based on a screen size.

[0295] Clause 181 includes an apparatus of any of clauses 173 to 180, wherein the interpolation factor determination data indicates a plurality of interpolation factor regions defined at least by a range of motion metric values, wherein each of the plurality of interpolation factor regions corresponds to a specific interpolation factor value, wherein the motion metric is included in a specific interpolation factor region of the plurality of interpolation factor regions, and wherein the interpolation factor has an interpolation factor value corresponding to the specific interpolation factor region.

[0296] Clause 182 includes the apparatus of any of clauses 140 to 181, further comprising generating the intermediate frame using motion compensated frame interpolation in response to determining that the interpolation factor satisfies the interpolation criterion.

[0297] Clause 183 includes the apparatus of any of clauses 140 to 182, further comprising, in response to determining that the interpolation factor fails to satisfy the interpolation criterion, generating the intermediate frame using an alternative to motion compensated frame interpolation.

[0298] Clause 184 includes the apparatus of any of clauses 140 to 183, further comprising generating the intermediate frame in response to a determination that motion compensated frame interpolation will be used to generate an intermediate frame, and generating the intermediate frame based on a determination that motion data indicates a first motion between a first frame and a second frame, such that the second motion between the first frame and the intermediate frame is based on the first motion and an interpolation factor.

[0299] Clause 185 includes an apparatus of any of clauses 140 to 183, wherein the component for generating an intermediate frame is responsive to a determination that motion compensated frame interpolation will be used to generate the intermediate frame, and generates the intermediate frame based on a determination that motion data indicates a first motion between a first frame and a second frame, such that the second motion between the first frame and the intermediate frame is based on a predetermined weight applied to the first motion.

[0300] Clause 186 includes the apparatus of any of clauses 140 to 185, wherein the means for generating the intermediate frame generates the intermediate frame as a copy of one of the first frame or the second frame in response to a determination that motion compensated frame interpolation will not be used to generate the intermediate frame.

[0301] Clause 187 includes the apparatus of any of clauses 140 to 186, further comprising means for receiving an input sequence of image frames.

[0302] Clause 188 includes the apparatus of any of clauses 140 to 187, wherein the components for obtaining motion data, the components for identifying any frame region, the components for determining motion metrics, the components for performing the determination, the components for generating intermediate frames, and the components for generating an output sequence are integrated into at least one of a communication device, a computer, a display device, a television, a game console, a digital video player, a camera, a navigation device, a vehicle, a head-mounted view device, an augmented reality head-mounted view device, a virtual reality head-mounted view device, an aerial vehicle, a home automation system, a voice-controlled device, an Internet of Things (IoT) device, a virtual reality (VR) device, a base station, or a mobile device.

[0303] The skilled person will further appreciate that the various illustrative logic blocks, configurations, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or a combination of the two. Various illustrative components, blocks, configurations, modules, circuits, and steps have been generally described above with respect to their functions. Whether such functions are implemented as hardware or processor executable instructions depends on the specific application and the design constraints imposed on the overall system. For each specific application, a skilled person may implement the described functions in a variety of ways, and such implementation decisions should not be understood as resulting in a departure from the scope of the present disclosure.

[0304] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein can be directly embodied in hardware, a software module executed by a processor, or a combination of the two. The software module can reside in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a compact disk read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium can be a component of the processor. The processor and the storage medium can reside in an application specific integrated circuit (ASIC). The ASIC can reside in a computing device or a user terminal. In an alternative, the processor and the storage medium can reside in a computing device or a user terminal as discrete components.

[0305] The previous description of the disclosed aspects is provided to enable those skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the invention. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but to conform to the broadest scope that may be consistent with the principles and novel features defined by the appended claims.

Claims

1. A device comprising: a memory configured to store instructions; and One or more processors configured to execute the instructions to: obtaining motion data indicative of estimated motion between a first frame and a second frame of an input sequence of image frames; identifying, based on the motion data, any frame regions of the first frame indicating motion greater than a motion threshold; determining a motion metric associated with the identified frame region based on the motion data; performing a determination whether to generate an intermediate frame using motion compensated frame interpolation based on the motion metric and the size metric associated with the identified frame region; generating the intermediate frame based on the determining; as well as generating an output sequence of image frames including the intermediate frame between the first frame and the second frame, wherein the motion metric is based on an average motion, a maximum motion, a range of motion, or a combination thereof associated with the identified frame region, and Wherein the size metric is based on a combined size of the identified frame regions, a count of the identified frame regions, a percentage of the first frame that includes the identified frame regions, or a combination thereof.

2. The apparatus of claim 1 , wherein the one or more processors are configured to generate an interpolation factor based on the size metric, and wherein the determination of whether to use the motion compensated frame interpolation to generate the intermediate frame is based on whether the interpolation factor satisfies an interpolation criterion. 3 . The device of claim 2 , wherein the one or more processors are further configured to generate the interpolation factor based on a comparison of the size metric with the interpolation factor determination data. 4 . The device of claim 3 , wherein the interpolation factor determination data is based on configuration settings, default data, user input, detected context, operating mode, screen size, or a combination thereof.

5. The apparatus of claim 3 , wherein the interpolation factor determination data indicates a plurality of interpolation factor regions defined by a range of at least size metric values, wherein each of the plurality of interpolation factor regions corresponds to a specific interpolation factor value, and wherein the one or more processors are further configured to generate the interpolation factor having an interpolation factor value corresponding to a specific interpolation factor region of the plurality of interpolation factor regions based on determining that the size metric is included in the specific interpolation factor region of the plurality of interpolation factor regions.

6. The apparatus of claim 1 , wherein the one or more processors are configured to generate an interpolation factor based on the motion metric, and wherein the determination of whether to use the motion compensated frame interpolation to generate the intermediate frame is based on whether the interpolation factor satisfies an interpolation criterion.

7. The apparatus of claim 6, wherein the one or more processors are further configured to generate the interpolation factor based on a comparison of the motion metric with interpolation factor determination data.

8. The apparatus of claim 7, wherein the interpolation factor determination data is based on configuration settings, default data, user input, detected context, operating mode, screen size, or a combination thereof.

9. The apparatus of claim 7 , wherein the interpolation factor determination data indicates a plurality of interpolation factor regions defined at least by a range of motion metric values, wherein each of the plurality of interpolation factor regions corresponds to a specific interpolation factor value, and wherein the one or more processors are further configured to generate the interpolation factor having an interpolation factor value corresponding to a specific interpolation factor region of the plurality of interpolation factor regions based on determining that the motion metric is included in the specific interpolation factor region of the plurality of interpolation factor regions.

10. The apparatus of claim 1, wherein the one or more processors are configured to generate the intermediate frame using the motion compensated frame interpolation in response to determining that an interpolation factor satisfies an interpolation criterion.

11. The apparatus of claim 1, wherein the one or more processors are configured to generate the intermediate frame using an alternative to motion compensated frame interpolation in response to determining that an interpolation factor fails to satisfy an interpolation criterion.

12. The apparatus of claim 1 , wherein the one or more processors are configured to generate the intermediate frame in response to the determination that the motion compensated frame interpolation will be used to generate the intermediate frame, and based on a determination that the motion data indicates a first motion between the first frame and the second frame, generate the intermediate frame such that a second motion between the first frame and the intermediate frame is based on the first motion and an interpolation factor.

13. An apparatus according to claim 1, wherein the one or more processors are configured to generate the intermediate frame in response to the determination that the motion compensated frame interpolation will be used to generate the intermediate frame, and based on the determination that the motion data indicates a first motion between the first frame and the second frame, generate the intermediate frame, so that the second motion between the first frame and the intermediate frame is based on a predetermined weight applied to the first motion.

14. The apparatus of claim 1, wherein the one or more processors are configured to generate the intermediate frame as a copy of one of the first frame or the second frame in response to the determination that the motion compensated frame interpolation will not be used to generate the intermediate frame.

15. The apparatus of claim 1, further comprising a modem configured to receive the input sequence of image frames.

16. A method comprising: obtaining, at a device, motion data indicating estimated motion between a first frame and a second frame of an input sequence of image frames; based on the motion data, identifying any frame regions of the first frame indicating motion greater than a motion threshold; determining a motion metric associated with the identified frame region based on the motion data; performing a determination whether to generate an intermediate frame using motion compensated frame interpolation based on the motion metric and the size metric associated with the identified frame region; generating, at the device, the intermediate frame based on the determination; as well as generating at the device an output sequence of image frames including the intermediate frame between the first frame and the second frame, wherein the motion metric is based on an average motion, a maximum motion, a range of motion, or a combination thereof associated with the identified frame region, and Wherein the size metric is based on a combined size of the identified frame regions, a count of the identified frame regions, a percentage of the first frame that includes the identified frame regions, or a combination thereof.

17. The method of claim 16, further comprising generating, at the device, interpolation factors based on the size metric and the motion metric, wherein the determination of whether to use the motion compensated frame interpolation to generate the intermediate frame is based on whether the interpolation factors meet an interpolation criterion.

18. The method of claim 17, wherein the interpolation factor is based on a comparison of the size metric and the motion metric to interpolation factor determination data.

19. The method of claim 18, wherein the interpolation factor determination data is based on configuration settings, default data, user input, detected context, operating mode, screen size, or a combination thereof.

20. The method of claim 18, wherein the interpolation factor determination data indicates a plurality of interpolation factor regions defined by a range of size metric values ​​and a range of motion metric values, wherein each of the plurality of interpolation factor regions corresponds to a specific interpolation factor value, wherein the motion metric and the size metric are included in a specific interpolation factor region of the plurality of interpolation factor regions, and wherein the interpolation factor has an interpolation factor value corresponding to the specific interpolation factor region.

21. The method of claim 17, further comprising, in response to determining that the interpolation factor satisfies the interpolation criterion, generating the intermediate frame using the motion compensated frame interpolation.

22. The method of claim 17, further comprising, in response to determining that the interpolation factors fail to satisfy the interpolation criteria, generating the intermediate frame using a substitute for the motion compensated frame interpolation.

23. The method of claim 17 further comprises, in response to the determination that the motion compensated frame interpolation will be used to generate the intermediate frame, and based on a determination that the motion data indicates a first motion between the first frame and the second frame, generating the intermediate frame such that a second motion between the first frame and the intermediate frame is based on the first motion and the interpolation factor.

24. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: obtaining motion data indicative of estimated motion between a first frame and a second frame of an input sequence of image frames; based on the motion data, identifying any frame regions of the first frame indicating motion greater than a motion threshold; determining a motion metric associated with the identified frame region based on the motion data; performing a determination whether to generate an intermediate frame using motion compensated frame interpolation based on the motion metric and the size metric associated with the identified frame region; generating the intermediate frame based on the determining; as well as generating an output sequence of image frames including the intermediate frame between the first frame and the second frame, wherein the motion metric is based on an average motion, a maximum motion, a range of motion, or a combination thereof associated with the identified frame region, and Wherein the size metric is based on a combined size of the identified frame regions, a count of the identified frame regions, a percentage of the first frame that includes the identified frame regions, or a combination thereof.

25. An apparatus comprising: means for obtaining motion data indicative of estimated motion between a first frame and a second frame of an input sequence of image frames; means for identifying, based on the motion data, any frame regions of the first frame indicating motion greater than a motion threshold; means for determining a motion metric associated with said identified frame region based on said motion data; means for performing a determination whether to generate an intermediate frame using motion compensated frame interpolation based on the motion metric and the size metric associated with the identified frame region; means for generating said intermediate frame based on said determining; and means for generating an output sequence of image frames including said intermediate frame between said first frame and said second frame, wherein the motion metric is based on an average motion, a maximum motion, a range of motion, or a combination thereof associated with the identified frame region, and Wherein the size metric is based on a combined size of the identified frame regions, a count of the identified frame regions, a percentage of the first frame that includes the identified frame regions, or a combination thereof.

26. An apparatus according to claim 25, wherein the components for obtaining the motion data, the components for identifying any frame area, the components for determining the motion metric, the components for performing the determination, the components for generating the intermediate frames, and the components for generating the output sequence are integrated into at least one of a communication device, a computer, a display device, a television, a game console, a digital video player, a camera, a navigation device, a vehicle, a head-mounted view device, an augmented reality head-mounted view device, a virtual reality head-mounted view device, an aerial vehicle, a home automation system, a voice-controlled device, an Internet of Things (IoT) device, a virtual reality (VR) device, a base station or a mobile device.

Images