Image processing method, electronic device and storage medium

Abstract

An image processing method, electronic device and storage medium are provided. The method includes generating luminance gradient data from input frames including a first input frame and a second input frame; obtaining an optical flow vector field by performing multiscale optical flow calculation based on the luminance gradient data; scaling optical flow vectors in the optical flow vector field; and performing warping based on the second input frame and the scaled optical flow vectors to obtain an interpolated or extrapolated frame. This method can improve video contents or stream contents.

Description

IMAGE PROCESSING METHOD, ELECTRONIC DEVICE AND STORAGE MEDIUMCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 648,318, entitled “LIGHTWEIGHT FRAME INTERPOLATION SOLUTION FOR MOBILE GAMES USING FRAGMENT SHADER BASED OPTICAL FLOW ESTIMATION AND UI DETECTION”, filed on May 16, 2024, which is hereby incorporated in its entirety by this reference.TECHNICAL FIELD

[0002] The embodiments of the present application relate to electronic technology, and relate to, but are not limited to, an image processing method, electronic device and storage medium.BACKGROUND

[0003] Prior work on frame interpolation and temporal supersampling has been performed. Existing literature and applications are oriented towards PC applications and impose heavy computation workloads on the hardware.

[0004] Existing solutions focus on single frame qualities and often have complex designs. A common theme is to use additional inputs beyond the original RGB data to increase the amount of available information. Existing RGB based solutions need to buffer multiple consecutive frames. Other techniques use a wide array of internal game resources, such as G-buffers, albedo maps, normal maps, metallic maps, depth maps, etc. Neural network based solutions can generate the missing details but usually use a large model to attain acceptable accuracy. Other miscellaneous computer vision (C V) based approaches detect feature from input frames and use the feature points to perform perspective transformations.

[0005] Existing techniques are not well suited for mobile gaming applications. Attempting to replicate these solutions in mobile can either cause serious device overheating, which adversely affects battery lifespan and performance, or introduce severe visual artifacts such as flickering or distortion.

[0006] The main reason is that prior works are not optimized for power consumption and processing time. These existing solutions tend to use additional inputs and generate high- resolution intermediate data, increasing the used I / O bandwidth and compute resources. Deep learning-based approaches use multiple convolutional layers and a large number of I / O channels, making it impractical to run in real-time on the current mobile devices. Additionally, some approaches use a-priori knowledge of internal game states or resources, making them difficult to integrate or maintain in commercial applications. Other CV based approaches are not yet robust enough for deployment across a large variety of mobile games.SUMMARY

[0007] An object of the present application is to propose an image processing method, electronic device and storage medium for improving video contents or stream contents.

[0008] In a first aspect of the present application, provided is an image processing method, including generating, by a processing unit, luminance gradient data from input frames including a first input frame and a second input frame; obtaining an optical flow vector field by performing multiscale optical flow calculation based on the luminance gradient data; scaling optical flow vectors in the optical flow vector field; and performing warping based on the second input frame and the scaled optical flow vectors to obtain an interpolated or extrapolated frame.

[0009] In a second aspect of the present application, provided is an electronic device, including one or more processors; and memory configured to store instructions which, when executed by the one or more processors, cause the one or more processors to generate luminance gradient data from input frames including a first input frame and a second input frame; obtain an optical flow vector field by performing multiscale optical flow calculation based on the luminance gradient data; scale optical flow vectors in the optical flow vector field; and perform warping based on the second input frame and the scaled optical flow vectors to obtain an interpolated or extrapolated frame.

[0010] In a third aspect of the present application, a non-transitory computer-readable storage medium that is executed by a processor of an electronic device, causing the processor of the electronic device to perform the above method.

[0011] The embodiment of the present application provides the image processing method and the electronic device. The method includes generating luminance gradient data from input frames including a first input frame and a second input frame; obtaining an optical flow vector field by performing multiscale optical flow calculation based on the luminance gradient data; scaling optical flow vectors in the optical flow vector field; and performing warping based on the secondinput frame and the scaled optical flow vectors to obtain an interpolated or extrapolated frame. This method can enhance mobile gaming by interpolating frames, resulting in smoother gameplay (Interpolation is used for higher frame quality and extrapolation for lower latency). This application also facilities improving video contents or stream contents.DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or related art, the following figures that will be described in the embodiments are briefly introduced. It is obvious that the drawings are merely some embodiments of the present application, a person having ordinary skill in this field can obtain other figures according to these figures without paying the premise.

[0013] FIG. 1 is a schematic diagram illustrating RGB based frame interpolation architecture according to an embodiment of the present application.

[0014] FIG. 2 is a schematic diagram illustrating multiple interpolation / extrapolation frames generated between each pair of adjacent native frames according to an embodiment of the present application.

[0015] FIG. 3 is a schematic diagram illustrating RGB based frame interpolation algorithm pipeline according to an embodiment of the present application.

[0016] FIG. 4 is a flowchart of an image processing method according to an embodiment of the present application.

[0017] FIG. 5 is a flowchart of generating luminance gradient data according to an embodiment of the present application.

[0018] FIG. 6 is a schematic diagram illustrating the transformation of texture data in a preprocess pass according to an embodiment of the present application.

[0019] FIG. 7 is a schematic diagram illustrating preprocessed luminance gradient texture according to an embodiment of the present application.

[0020] FIG. 8 is a schematic diagram illustrating a first input frame inside a mobile game according to an embodiment of the present application.

[0021] FIG. 9 is a schematic diagram illustrating a second input frame inside a mobile game according to an embodiment of the present application.

[0022] FIG. 10 is a schematic diagram illustrating an optical flow vector field texture according to an embodiment of the present application.

[0023] FIG. 11 is a schematic diagram illustrating the same optical flow texture visualized as a vector field according to an embodiment of the present application.

[0024] FIG. 12 is a flowchart of performing warping based on a second input frame and scaled optical flow vectors according to an embodiment of the present application.

[0025] FIG. 13 is a schematic diagram illustrating overall conditional warping logic according to an embodiment of the present application.

[0026] FIG. 14 is a flowchart of UI check according to an embodiment of the present application.

[0027] FIG. 15 is a schematic diagram illustrating the result of a simple warping strategy according to an embodiment of the present application.

[0028] FIG. 16 is a schematic diagram illustrating how the UI check is performed according to an embodiment of the present application.

[0029] FIG. 17 is a schematic diagram illustrating the final result of conditional warping pass according to an embodiment of the present application.

[0030] FIG. 18 is a schematic diagram illustrating a hardware entity of an electronic device provided according to an embodiment of the present application.DETAILED DESCRIPTION OF EMBODIMENTS

[0031] Embodiments of the disclosure are described in detail with the technical matters, structural features, achieved objects, and effects with reference to the accompanying drawings as follows. Specifically, the terminologies in the embodiments of the present application are merely for describing the purpose of the certain embodiment, but not to limit the disclosure.

[0032] In this document, the term “ / ” should be interpreted to indicate “and / or.” A combination such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” or “A, B, and / or C” may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, where any combination may contain one or more members of A, B, or C.

[0033] The following table includes some abbreviations and interpretations that may be applied in some embodiments of the present application:

[0034] The present application may include a mobile GPU-based frame interpolation / extrapolation algorithm that are implemented as fragment shaders. The algorithm can be deployed at the system level to achieve low-cost frame interpolation / extrapolation for most real-time mobile games. The present application is capable of significantly improving games’ smoothness while only paying a small, fixed cost in GPU compute and memory, making it lightweight and mobile-friendly compared to existing state of the art. For games that do not natively support a high FPS, the present application can lift the game FPS to the mobile devices refresh rate. The power consumption of the solution is also much lower than rendering the game at the same framerate, which significantly increases battery life for resource-intensive mobile games. The fragment shader implementation of the present application may rely on the same mobile GPU and the same graphics context used for the game’s native rendering, eliminating additional hardware costs and minimizes additional VO, and memory overhead.

[0035] The present application may achieve these results by combining lightweight, efficient multiscale optical flow calculation, specialized UI detection logic, and conditional warping logic into the frame interpolation solution. A set of internal parameters may be used by the algorithm to support fine turning for different games, optimizing the performance for different game graphic styles, animation patterns, and rendering load. The algorithm may only use the RGB color information of two adjacent frames to 1) interpolate between the two input frames; or 2) extrapolate the frames after the second input frame before the next native frame arrives. The RGB inputs may be provided via OpenGL ES or Vulkan color textures. The algorithm, implemented using OpenGL ES and Vulkan shaders, renders the interpolated or extrapolated frames to output textures, which are then sent to the physical display.

[0036] The present techniques may be deployed as a system level game assistant service in mobile devices, the service automatically detects the launch of supported game packages, and then enables proposed temporal supersampling algorithm for the game instance.

[0037] As shown in FIG. 1, the service may obtain the native RGB game render results, consisting of two RGB textures from two adjacent frames, and forwards these textures to the algorithm. The algorithm computes the interpolated or extrapolated frames in real time. This may be done by using OpenGL ES or Vulkan shaders (the algorithm shares the game’s graphics backend). The output frames are then sent to the screen buffer rendering queue.

[0038] As shown in FIG. 2, multiple interpolation / extrapolation frames can be generated between each pair of adjacent native frames to achieve different interpolation ratios. For 1 : 1 interpolation, a single T=0.5 output frame is produced; for 1 :2 interpolation, two T=0.33, T=0.66 output frames are produced. Similarly for extrapolation, a single T=1.5 frame is produced, and for 1 :2 extrapolation two T=1.33, T=1.66 frames are produced. Both interpolation and extrapolation strategies are available in the algorithm and can be selected based on individual games. Interpolation is used for higher frame quality and extrapolation for lower latency.

[0039] The algorithm pipeline can be seen in FIG. 3. The overall pipeline logic is to perform per- pixel warping based on a motion vector field calculated by a compact multi-scale optical flow shading pass. The optical flow fragment shaders have been designed to balance performance and accuracy on mobile devices. The optical flow calculation may be adapted from the Lucas-Kanade method, which assumes that each neighborhood in the image satisfies the constant luminance constraint over an adjacent frame. A number of tunable parameters are used to optimize the optical flow accuracy for different game scenarios.

[0040] Warping is performed by sampling the second input frame, which contains the most up to date information, at an offset given by the optical flow field. The algorithm increases the game FPS by a fixed multiplier, which may be typically set to 1 :2 or 1 :3. For extrapolation, the algorithm warps the predicted frames under the assumption that the pixels will continue its current motion speed and direction, and for interpolation the algorithm warps the in-between frames under the assumption that pixels moved at a uniform speed between the two input frames. In both cases, it is calculated the sampling offsets by scaling the optical flow vector based on the temporal location of each output frame. The shader implementation can warp multiple frames simultaneously by writing to multiple render targets. The warping result may be supplemented by custom UI detection logic and filtering logic to stabilize the optical flow motion vector field and eliminate obvious distortions; blending is used to fill the pixels that cannot be warped correctly.

[0041] The present architecture is designed to take advantage of existing texture resources and rendering context in mobile GPU memory and minimizing IO cost. The overall interpolation / extrapolation pipeline includes 3 separate shading passes, that is, the preprocess pass, the multi-scale optical flow pass and the warping pass.

[0042] Referring to FIG. 4, an embodiment of the present application provides an image processing method 10 applied to an electronic device. The method 10 includes the following steps. These steps may be performed by a processing unit, which is preferred to be a graphic processingunit; however, a central processing unit may be applied as well. In other cases, the graphic processing unit and the central processing unit may cooperate with each other to implement these steps.

[0043] Step 12: generating luminance gradient data from input frames including a first input frame and a second input frame;

[0044] In this step, a preprocess pass is performed to obtain luminance gradient data, which are generated from input frames (e.g., native RGB input frames). For example, the input frames include a first input frame and a second input frame that are adjacent to each other. More specifically, as shown in FIG. 5, Step 12 may include the following steps:

[0045] Step 121 : downsizing the input frames by a predetermined factor; and

[0046] Step 122: generating the luminance gradient data from the downsized input frames.

[0047] The input frames including the first frame and the second frame may be downsized by a predetermined factor (e.g., 4). That is, if an input frame has a resolution of 1024x768, its downsized resolution would be 256x192. In this case, the downsized input frames are used in order to generate the luminance gradient data.

[0048] In an illustrated example, FIG. 6 shows the transformation of the texture data in the preprocess pass, the entire preprocess step can be done efficiently in a single fragment shader pass. The preprocess pass performs the following two objectives: a) downsizing the input RGB textures by a factor of 4, and b) packing the RGB color data in luminance gradients. The algorithm dynamically allocates an RGBA texture downsized from the input frames upon initialization and keeps reusing the same intermediate texture resource. Reallocation happens only when the game resolution is changed.

[0049] The downsizing factor of 4 is empirically determined to both preserve the luminance details of the original input RGB frames and significantly improve the spatial locality during the subsequent optical flow pass. The packing of RGB data into the intermediate luminance gradienttexture allows for efficient sampling during the optical flow computation, which would otherwise need to sample a large number of texture coordinates from the RGBA input frames, convert each sample into luminance value, and compute the luminance differential.

[0050] The preprocess pass is designed minimizing IO cost and improving cache efficiency of the subsequent optical flow computation. Compared to directly sampling the RGB input frames, sampling preprocessed inputs can reduce the power consumption of the optical flow computation by approximately 60%.

[0051] Step 14: obtaining an optical flow vector field by performing multiscale optical flow calculation based on the luminance gradient data;

[0052] In this step, based on the luminance gradient data from Step 12, multiscale optical flow calculation is performed so as to obtain an optical flow vector field. The multiscale optical flow calculation may be performed using Lucas-Kanade method. In some scenarios, the multiscale optical flow calculation may be performed by only accumulating optical flow data from the most interesting scales. For example, the optical flow data may be accumulated only from Leve of Details (LOD) of 2 and 3.

[0053] The following illustration provides the optical flow calculation process for the position marked by “o” in FIG. 7:

[0054] In FIG. 7, the preprocessed luminance gradient texture is shown. This texture has already been downsized by a factor of 4 in the preprocess shading pass, which corresponds to the LOD (Level of Details)=2 texture. To calculate the optical flow data at LOD=2 using the Lucas-Kanade method, it can simply sample the adjacent neighborhood pixels that store the packed luminance gradient data (marked by the dotted and left-hand twill cells) and apply the optical flow formula. Similarly, for LOD=3 it is sampled the same pack luminance texture at every other pixel inside a proportionally larger neighborhood (marked by left-hand twill and right-hand twill cells) and apply the same optical flow formula. The flow results from the two levels are then accumulated using:vx(o)-Vy(0 _

[0055] Here the values of Ix(q0, Iy(qi) denote unpacked values from the luminance gradient textures sampled at qi. The index z is used to iterate over the neighborhood around o.

[0056] It is noted that the Lucas-Kanade method can be replaced by a block matching method or other motion vector estimation methods. These algorithms may achieve similar level of image quality and perform better than the proposed solution in some scenarios but will also introduce different types of artifacts not in the proposed solution. They would also use a high level of optimization to match the performance of the proposed the solution.

[0057] In some embodiments, Step 14 may include filling optical flow values that are larger than a predetermined threshold with a zero optical flow vector. The optical flow vector is obtained via solving a linear system and uses a matrix inversion. When the matrix is ill-conditioned, the flow values can become extremely large or simply become NAN values. These extreme values become holes in the optical flow texture; it can be checked for these values and fill these holes with the (0, 0) vector.

[0058] In some embodiments, Step 14 may include downscaling the results from the multiscale optical flow calculation to obtain the optical flow vector field. Using a low-resolution texture to store the optical flow vector field further optimizes the process.

[0059] The novelty in the custom optical flow comes from any of the following techniques: a) remove mipmap generation and simulate LOD sampling by sampling at different strides, b) sampling packed luminance gradient data and directly use the samples in the Lucas-Kanade formula, c) use a very low-resolution texture to store the optical flow field, d) only accumulate optical flow data from the most interesting scales, it has empirically determined that LOD level 2 and 3 contains the most salient motion data, e) invalid value check and hole filling, f) use configurable parameters to optimize the result for different games. These techniques can produce a mobile optical flow implementation that produces reliable results using only a fraction of power consumed by traditional CV implementations.

[0060] The optical flow calculation passes in the present techniques reduce power consumption in a traditionally compute-intensive process while maintaining an acceptable accuracy for mobile games.

[0061] Similar to the preprocess pass, the multiscale optical flow calculation is also executed inside a fragment shader pass, which renders the optical flow output to another two-channel floating point texture. Below is an illustration of how the multi-scale optical flow works:

[0062] FIGs. 8 and 9 are example input frames inside a mobile game. The scene is rather complex, with objects of different depths moving at different speeds, and static UI overlay.

[0063] FIG. 10 shows the 2-channel optical flow vector field texture. Note that the optical flow texture is further downscaled from the preprocessing pass. The exact downscale ratio is a tunable parameter; for the best results, the optical flow texture is on average downscaled by a factor of 20 from the original RGB frames.

[0064] FIG. 11 shows the same optical flow texture visualized as a vector field. The result is a consistent field that serves as the basis for per pixel warping in the subsequent warping pass.

[0065] Step 16: scaling optical flow vectors in the optical flow vector field;

[0066] In this step, optical flow vectors in the optical flow vector field are adjusted or scaled. The scaled optical flow vectors are then used in subsequent warping process for obtaining an interpolated or extrapolated frame. In some embodiments, the optical flow vectors in the optical flow vector field may be scaled using a fixed multiplier (e.g., 1:2 or 1 :3). For interpolation, it is assumed that pixels moved at a uniform speed between the first input frame and the second input frame. For extrapolation, it is assumed that the pixels will continue its current motion speed and direction.

[0067] Step 18: performing warping based on the second input frame and the scaled optical flow vectors to obtain an interpolated or extrapolated frame.

[0068] In this step, an interpolated or extrapolated frame for higher frame quality (interpolation) or lower latency (extrapolation) is obtained by performing warping based on the second input frame in Step 12 and the scaled optical flow vectors in Step 16.

[0069] In some embodiments, additionally, the method may include using a mean filter to reduce noises in optical flow data of the optical flow vector field. The filtered data obtained after applying the mean filter to the optical flow vector field is then used for the warping. It is noted that the scaled optical flow vectors may also be obtained from the optical flow vector field with the mean filter applied.

[0070] More specifically, as shown in FIG. 12, Step 18 may include the following steps:

[0071] Step 181 : for a pixel in the first input frame, using the optical flow vector field to compute a location of a corresponding pixel in the second input frame;

[0072] Step 182: sampling a color of the corresponding pixel at the location in second input frame; and

[0073] Step 183 : using the sampled color and an offset given by the scaled optical flow vector of the pixel to obtain an resulted pixel on the interpolated or extrapolated frame.

[0074] Warping is a strategy for filling pixels in the interpolated / extrapolated frame. The warping is performed based on the second input frame. The color of a pixel on the second input frame and its scaled optical flow vector are used to determine an resulted pixel on the interpolated or extrapolated frame. More specifically, for each pixel in the output frame, the algorithm computes the corresponding location in the second input frame using the optical flow vector field, samples that corresponding location in second input frame, and uses the sampled color to fill the output frame pixel. This strategy produces accurate results in regions where the optical flow data is also accurate.

[0075] The warping pass may be a conditional warping pass which is the last fragment shading pass. FIG. 13 shows the overall conditional warping logic. A mean filter is first applied to the optical flow vector field. The filtered optical flow and the input frames are then used to determine regions that are likely to be user interface (UI). Regions that pass the UI check use the warping strategy, and regions that fail use a blending strategy.

[0076] More specifically, as shown in FIG. 14, the method may include the following steps:

[0077] Step 191 : performing an user interface (UI) check for a region on the interpolated or extrapolated frame;

[0078] Step 192: if the region passes the UI check, using the region obtained by the warping as an output region; and

[0079] Step 193 : if the region does not pass the UI check, using the region obtained by blending the first input frame and the second input frame as the output region.

[0080] The compact optical flow data can have holes, go outside the window, and lose accuracy over UI regions. FIG. 15 shows the result of a simple warping strategy. Compared to FIGs. 8 and 9, the bottom right corner is filled with incorrect data, and the bottom UI bar became distorted. To alleviate these issues, the conditional warping pass has a fallback strategy that simply blends thetwo input frames. Pure blending creates ghosting effects but is situationally preferrable over obvious distortions, therefore a UI check rule is used to only perform blending when it’s necessary.

[0081] The UI check rule avoids distortions by 1) detect cases where warping overwrites UI regions with neighboring pixels, or 2) warping would move UI pixels outside the UI region. FIG. 16 illustrates how the rule operates. For a region, if a frame i pixel is nearly identical a frame i+1 pixel, the bending strategy is used; otherwise, the warping is performed. If warp source history coming from the frame i pixel and optical flow shows the pixel has no significant changes, use the blending; otherwise, use the warping strategy. If the warped pixel has no significant changes, use the blending; otherwise, use the warping strategy.

[0082] It is noted that the UI check implementation can be omitted if the game can provide a UI mask, however this constitutes an additional input, which makes integration and maintenance.

[0083] FIG. 17 shows the final result of the conditional warping pass. The most obvious distortions have been fixed and the output frame becomes more coherent.

[0084] Parameter tuning is important for accurately capturing motion patterns for different types of mobile games. For example, it is needed to put more emphasis on global motions for first person camera zooming / panning and on local motions for complex fight scenes.

[0085] For this disclosure, optical flow downsize ratio, optical flow neighborhood kernel sizes, stride sizes, and LOD levels are adjustable for each supported game. It fine-tunes the selected parameter values by conducting image quality assessments or a large propriety game dataset and adjusting the parameter search space based on insights gained from internal metrics and user feedback.

[0086] The embodiment of the present application provides the image processing method and the electronic device. The method includes generating luminance gradient data from input frames including a first input frame and a second input frame; obtaining an optical flow vector field by performing multiscale optical flow calculation based on the luminance gradient data; scalingoptical flow vectors in the optical flow vector field; and performing warping based on the second input frame and the scaled optical flow vectors to obtain an interpolated or extrapolated frame. This method can enhance mobile gaming by interpolating frames, resulting in smoother gameplay (Interpolation is used for higher frame quality and extrapolation for lower latency). This application also facilities improving video contents or stream contents.

[0087] The present techniques enable smoother gameplay and can increase the FPS of any mobile game up to the screen’s refresh rate. This is useful for games that have high GPU render loads and cannot match the phone refresh rate natively.

[0088] In mobile gaming experience, higher FPS helps reduce motion sickness and eye strain, e.g., for players sensitive to visual motion artifacts such as motion blur and screen tearing. Consistent motion at higher framerates is less likely to cause discomfort or user fatigue during extended gaming sessions, resulting in more user engagement and better gameplay experience.

[0089] The present techniques also have power saving benefits. Games that already natively support 120 FPS can instead be rendered at 60 FPS and then interpolated to 120 FPS using the proposed solution. The user now has access to the same level of smooth gameplay, while also enjoying extended battery life. By delivering the same FPS with less power, the device is also less likely to overheat.

[0090] The present techniques have benefits outside the mobile gaming domain. Video contents and stream contents are also popular on mobile platforms, and the algorithm optimized for mobile gaming experience is powerful enough to be deployed in products that target other types of video contents.

[0091] Additionally, the algorithm can be used to supplement motion-vector based solutions. Motion vector based frame interpolation algorithms typically have great overall image quality that can match native frames. However, native motion vector data is absent for disocclusion regions and can be ill-defined over regions with alpha-blending. In such regions, a motion vector basedsolution can have visual artifacts such as jittering and tearing. The optical flow algorithm can be used to eliminate these visual artifacts.

[0092] It is noted that there are multiple ways to implement the same optical flow algorithm. The proposed solution can have an equivalent vertex shader or compute shader implementation, which would use different optimization techniques and may not be as efficient. It is possible to use other GPU backends such as OpenCL but the alternative may be more difficult to integrate.

[0093] An embodiment of the present application further provides an electronic device. FIG. 18 is a schematic diagram of a hardware entity of an electronic device provided in embodiments of this application. As illustrated in FIG. 18, the electronic device 1800 includes a memory 1801 and a processor 1802. The memory 1801 stores a computer program executable on the processor 1802, and the processor 1802 executes the program to implement steps in the image processing method provided in the above embodiments. In an embodiment, the processor can be a graphic processing unit (GPU), a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), a field programmable gate array (FPGA), and the like.

[0094] It should be noted that, the memory 1801 is configured to store instructions and applications executable by the processor 1802, and further to cache data to be processed or processed (for example, image data, audio data, voice communication data, and video communication data) by the processor 1802 and various modules in the electronic device 1800, which can be implemented through FLASH or random access memory (RAM).

[0095] Accordingly, embodiments of this application provide a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon. When executed by a processor, the computer program implements steps in the image processing method provided in the above embodiments.

[0096] Herein, it should be noted that: the descriptions of the above storage medium and device embodiments are similar to the descriptions of the above method embodiments, and the abovestorage medium and device embodiments have similar advantageous effects to the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for the purpose of understanding.

[0097] It is to be understood that “one embodiment” or “an embodiment” throughout the specification means that a particular feature, structure, or characteristic associated with the embodiment is included in at least one embodiment of this application. Thus, “in one embodiment” or “in an embodiment” appearing in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in various embodiments of this application, the magnitude of the sequence numbers of the above processes does not mean the order of execution. The order of execution of each process should be determined by its function and internal logic and should not constitute any limitation to the implementation process of the embodiments of this application. The serial numbers of the foregoing embodiments of this application are only for description, and do not represent the superiority of the embodiments of this application.

[0098] It is noted that in this application, the terms “including”, “containing” or any other variations thereof are intended to cover non-exclusive inclusion. As a result, a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or further includes elements inherent to the process, method, article, or device. If there are no more restrictions, the element defined by the sentence “including one...” does not exclude the existence of other same elements in the process, method, article, or device that includes the element.

[0099] In some embodiments of this application, it should be understood that the device / apparatus and method disclosed in embodiments provided herein may be implemented inother manners. For example, the device embodiments described above are merely illustrative; for instance, the division of the unit is only a logical function division and there can be other manners of division during actual implementations, for example, multiple units or components may be combined or may be integrated into another system, or some features may be ignored, omitted, or not performed. In addition, coupling or communication connection between each illustrated or discussed component may be direct coupling or communication connection, or may be indirect coupling or communication among devices or units via some interfaces, and may be electrical connection, mechanical connection, or other forms of connection.

[0100] The units described as separate components may or may not be physically separated, the components illustrated as units may or may not be physical units, that is, they may be in the same place or may be distributed to multiple network elements. All or part of the units may be selected according to actual needs to achieve the purpose of the technical solutions of the embodiments.

[0101] In addition, the functional units in various embodiments of this application may be integrated into one processing unit, or each unit may be physically present, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or a software function unit, or combination thereof.

[0102] It will be understood by those of ordinary skill in the art that, all or a part of the various methods of the embodiments described above may be accomplished by means of a program to instruct associated hardware. The program may be stored in a computer readable storage medium. When executed, the program executes steps including the above method embodiments. The aforementioned storage medium includes: a mobile storage device, an ROM, a magnetic disk or an optical disk and other mediums that can store program codes.

[0103] Alternatively, the integrated unit may be stored in a computer readable storage when it is implemented in the form of a software functional unit and is sold or used as a separate product. Based on such understanding, the technical solutions of this application essentially, or the part ofthe technical solutions that contributes to the related art, or all or part of the technical solutions, may be embodied in the form of a software product. The computer software product is stored in a memory and includes instructions for causing a computer device (which may be a cell phone, tablet, desktop, server, television, audio player, etc.) to perform all or part of the steps described in the various embodiments of this application. The aforementioned storage medium includes: a mobile storage device, an ROM, a magnetic disk or an optical disk and other mediums that can store program codes. The methods disclosed in the several method embodiments of this application can be combined without conflict to obtain new method embodiments. The features disclosed in the several product embodiments of this application can be combined without conflict to obtain new product embodiments. The features disclosed in the several method embodiments or device embodiments of this application can be combined without conflict to obtain new method embodiments or device embodiments. The above are some embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed by this application should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0104] While the present application has been described in connection with what is considered the most practical and preferred embodiments, it is understood that the present application is not limited to the disclosed embodiments but is intended to cover various arrangements made without departing from the scope of the broadest interpretation of the appended claims.

Claims

WHAT IS CLAIMED IS:

1. An image processing method, comprising: generating, by a processing unit, luminance gradient data from input frames including a first input frame and a second input frame; obtaining an optical flow vector field by performing multiscale optical flow calculation based on the luminance gradient data; scaling optical flow vectors in the optical flow vector field; and performing warping based on the second input frame and the scaled optical flow vectors to obtain an interpolated or extrapolated frame.

2. The method of claim 1, wherein the step of generating, by the processing unit, the luminance gradient data from the input frames including the first input frame and the second input frame comprises: downsizing the input frames by a predetermined factor; and generating the luminance gradient data from the downsized input frames.

3. The method of claim 1, wherein the multiscale optical flow calculation is performed using Lucas-Kanade method.

4. The method of claim 1, wherein the multiscale optical flow calculation is performed by only accumulating optical flow data from the most interesting scales.

5. The method of claim 4, wherein the optical flow data are accumulated only from Leve of Details (LOD) of 2 and 3.

6. The method of claim 1, wherein the step of obtaining the optical flow vector field by performing the multiscale optical flow calculation based on the luminance gradient data comprises: filling optical flow values that are larger than a predetermined threshold with a zero optical flow vector.

7. The method of claim 1, wherein the step of obtaining the optical flow vector field by performing the multiscale optical flow calculation based on the luminance gradient data comprises:downscaling the results from the multiscale optical flow calculation to obtain the optical flow vector field.

8. The method of claim 1, wherein in the step of scaling the optical flow vectors in the optical flow vector field, the optical flow vectors in the optical flow vector field are scaled using a fixed multiplier.

9. The method of claim 1, wherein the step of performing the warping based on the second input frame and the scaled optical flow vectors to obtain the interpolated or extrapolated frame comprises: for a pixel in the first input frame, using the optical flow vector field to compute a location of a corresponding pixel in the second input frame; sampling a color of the corresponding pixel at the location in second input frame; and using the sampled color and an offset given by the scaled optical flow vector of the pixel to obtain an resulted pixel on the interpolated or extrapolated frame.

10. The method of claim 1, further comprising: performing an user interface (UI) check for a region on the interpolated or extrapolated frame; if the region passes the UI check, using the region obtained by the warping as an output region; and if the region does not pass the UI check, using the region obtained by blending the first input frame and the second input frame as the output region.

11. The method of claim 10, wherein the UI check is performed either by detecting cases where the warping overwrites an UI region with neighboring pixels, or by checking whether the warping moves UI pixels outside the UI region.

12. The method of claim 1, further comprising: using a mean filter to reduce noises in optical flow data of the optical flow vector field.

13. An electronic device, comprising: one or more processors; andmemory configured to store instructions which, when executed by the one or more processors, cause the one or more processors to: generate luminance gradient data from input frames including a first input frame and a second input frame; obtain an optical flow vector field by performing multiscale optical flow calculation based on the luminance gradient data; scale optical flow vectors in the optical flow vector field; and perform warping based on the second input frame and the scaled optical flow vectors to obtain an interpolated or extrapolated frame.

14. The electronic device of claim 13, wherein the one or more processor is configured to: downsize the input frames by a predetermined factor; and generate the luminance gradient data from the downsized input frames.

15. The electronic device of claim 13, wherein the multiscale optical flow calculation is performed using Lucas-Kanade method.

16. The electronic device of claim 13, wherein the multiscale optical flow calculation is performed by only accumulating optical flow data from the most interesting scales.

17. The electronic device of claim 16, wherein the optical flow data are accumulated only from Leve of Details (LOD) of 2 and 3.

18. The electronic device of claim 13, wherein the one or more processor is configured to: fill optical flow values that are larger than a predetermined threshold with a zero optical flow vector.

19. The electronic device of claim 13, wherein the one or more processor is configured to: downscale the results from the multiscale optical flow calculation to obtain the optical flow vector field.

20. The electronic device of claim 13, wherein the optical flow vectors in the optical flow vector field are scaled using a fixed multiplier.

21. The electronic device of claim 13, wherein the one or more processor is configured to: for a pixel in the first input frame, use the optical flow vector field to compute a location of a corresponding pixel in the second input frame; sample a color of the corresponding pixel at the location in second input frame; and use the sampled color and an offset given by the scaled optical flow vector of the pixel to obtain an resulted pixel on the interpolated or extrapolated frame.

22. The electronic device of claim 13, wherein the one or more processor is configured to: perform an user interface (UI) check for a region on the interpolated or extrapolated frame; if the region passes the UI check, use the region obtained by the warping as an output region; and if the region does not pass the UI check, use the region obtained by blending the first input frame and the second input frame as the output region.

23. The electronic device of claim 22, wherein the UI check is performed either by detecting cases where the warping overwrites an UI region with neighboring pixels, or by checking whether the warping moves UI pixels outside the UI region.

24. The electronic device of claim 13, wherein the one or more processor is configured to: use a mean filter to reduce noises in optical flow data of the optical flow vector field.

25. A non-transitory computer-readable storage medium having stored instructions that is executed by a processor of an electronic device, causing the processor of the electronic device to perform the method of any of claims 1 to 12.

Images