Device and method for optimizing power consumption during frame rendering

Abstract

The present application relates to energy saving in graphics processing. To this end, the present application proposes an entity. The entity is configured to: obtain image data comprising a series of original frames from a graphics application; determine whether to enable frame prediction to render the image data according to information for enabling prediction obtained from the graphics application; if frame prediction is enabled, enable the graphics application to render a subset of the series of original frames to obtain rendered original frames; insert one or more predicted frames into the rendered original frames, wherein each predicted frame is calculated according to one or more original frames in the image data, and each original frame that is not rendered by the graphics application is replaced by one predicted frame in the one or more predicted frames; and output rendered image data, wherein the rendered image data comprises the one or more predicted frames and the rendered original frames obtained from the graphics application.

Description

Technical Field

[0001] The present invention relates generally to computer graphics processing, and more particularly to optimizing power consumption during frame rendering. In particular, the present invention relates to energy conservation in high-frame-rate graphics processing. To this end, the present invention provides an apparatus and method for reducing power consumption in real-time graphics processing. Background Art

[0002] High-frame-rate displays are becoming increasingly popular in modern smartphones. A few years ago, most mobile devices were typically equipped with 60Hz screens (i.e., screens with a refresh rate of 60Hz). However, today, 90Hz (and even higher) displays are rapidly increasing in popularity. Displays with higher refresh rates, or higher frame rates, reduce visual latency, make game loading smoother, and improve the user experience.

[0003] However, one problem is that devices with high frame rate displays consume more power, thus significantly shortening battery life. In addition, since applications / software that need to process graphics must generate more frames, the power consumption of the entire system will also increase. For example, compared with a device with a 60Hz display, the power consumption of the central processing unit (CPU) and graphics processing unit (GPU) in a device with a 120Hz display during graphics rendering may increase by 1.5 times to 2 times. Of course, the physical capacity of the battery is limited.

[0004] Therefore, a technical solution is needed to reduce the power consumption caused by increasing the frame rate during the frame rendering process. Summary of the Invention

[0005] In view of the above-mentioned shortcomings, embodiments of the present invention aim to optimize (in particular, reduce) the power consumption generated during frame rendering (for example, when processing graphics). Specifically, one goal is to propose a method for reducing or optimizing the power consumption generated during high frame rate real-time graphics rendering. Another goal is to provide a technical solution that is compatible with any application that needs to perform frame rendering. One of the purposes of the present invention is to ensure that the quality of the rendered frame is not affected.

[0006] These and other objects are achieved by the embodiments presented in the accompanying independent claims. Advantageous implementations of these embodiments are further defined in the dependent claims.

[0007] A first aspect of the present invention provides an entity for processing image data. The entity is configured to: obtain image data comprising a series of raw frames from a graphics application (e.g., by a CPU and / or a GPU); determine (e.g., by a CPU) whether to enable frame prediction to render the image data based on information for enabling prediction obtained from the graphics application;

[0008] If frame prediction is enabled, the graphics application is enabled (e.g., by a CPU and / or GPU) to render a subset of the series of original frames in the image data to obtain rendered original frames; one or more predicted frames are inserted into the rendered original frames obtained from the graphics application (e.g., by a CPU and / or GPU), wherein each of the one or more predicted frames is calculated based on one or more original frames in the image data, and each original frame not rendered by the graphics application is replaced by a predicted frame from the one or more predicted frames; and rendered image data is output (e.g., by a CPU and / or GPU), wherein the rendered image data includes the one or more predicted frames and the rendered original frames obtained from the graphics application. For example, the rendered image data is displayed on a display screen.

[0009] Specifically, the entity provided by the first aspect may include the necessary processing circuitry for performing the functions described herein, such as a CPU / GPU and memory. To implement embodiments of the present invention, the CPU and / or GPU reads instructions and / or data stored in the memory, and transmits requests / messages and data between the CPU and GPU. The graphics application may be an application that requires graphics processing such as frame rendering. For example, the graphics application may be a 3D video game. Therefore, the entity provided by the first aspect can optimize (and in particular, reduce) the power consumption generated during frame rendering when running the graphics application. This advantage is achieved by enabling frame prediction for frame rendering. Specifically, a portion of the frames generated by the graphics application (i.e., original frames) can be replaced with predicted frames, and the predicted frames can be calculated by the entity provided by the first aspect. Because at least a portion of the original frames are predicted frames, i.e., not rendered by the graphics application, the power consumption used to render these frames in the image data can be reduced. Notably, the original frame rate of the graphics processing can still be maintained because the discarded original frames that were not rendered are replaced with predicted frames, rather than simply discarded without replacement.

[0010] It should also be noted that the image data processing pipeline can be modified to obtain image data. This data may be different from the original frame. For example, the 3D scene layer and the user interface (UI) scene layer can be separated and only the 3D layer is used in the prediction algorithm.

[0011] In one implementation of the first aspect, the information for enabling prediction includes a first set of information, and the entity is further configured to obtain the first set of information by analyzing (e.g., by a CPU and / or GPU) a graphics application programming interface command from the graphics application, wherein the first set of information includes color information associated with the image data and at least one of depth information associated with the image data, a first category object mask, a second category object mask, and one or more camera matrices. For example, the GPU obtains the first set of information and sends the first set of information to the CPU. The CPU can then determine whether to enable frame prediction.

[0012] Typically, the first set of information may be semantic scene information that may be used to draw a frame on a display screen. The first set of information may also be obtained from the image data, or from the rendering process of the graphics application. Such information typically includes at least the color value of each pixel to be displayed on the display screen. In addition, it may also include a depth value, texture, template and / or camera matrix for each pixel on the display screen. It is noteworthy that the first set of information may represent a scene to be drawn on the display screen when rendering the corresponding frame.

[0013] In an implementation of the first aspect, the information for enabling prediction includes a second set of information, and the entity is further used to obtain the second set of information in the following manner: (for example, by a CPU and / or GPU) analyzing a graphics application programming interface command from the graphics application to obtain a first set of information, wherein the first set of information includes color information related to the image data and depth information related to the image data, a first category object mask, a second category object mask, and at least one of one or more camera matrices; extracting the second set of information based on the first set of information, wherein the second set of information includes at least one of a current graphics application scene, object classification, camera position, camera orientation, and camera movement behavior; and determining whether to enable frame prediction based on the second set of information.

[0014] Specifically, the first set of information can be used to obtain the second set of information. Based on the second set of information, the quality of the predicted frame can be determined in advance. Furthermore, based on the second set of information, different prediction algorithms can be selected to improve prediction quality. For example, the first set of information obtained by the GPU is sent to the CPU. The CPU can then extract the second set of information and determine whether to enable frame prediction.

[0015] In one implementation of the first aspect, the entity is further configured to classify objects in the image data into at least a first category and a second category (e.g., by a CPU and / or a GPU) based on the first set of information, wherein objects in the first category have a constant position within a coordinate system and objects in the second category have a dynamic position within the coordinate system.

[0016] It is worth noting that an object in an image or scene can represent an area in the image that includes a group of pixels with the same or similar attributes. Optionally, the object classification can be determined based on the first set of information (i.e., semantic scene information). The objects in the first category can also be static objects, and the objects in the second category can also be dynamic objects. Specifically, the object classification can be included in the second set of information.

[0017] In an implementation of the first aspect, the entity is further configured to: determine to enable frame prediction if one or more of the following conditions are met:

[0018] The current graphics application scenario matches one or more predetermined graphics application scenarios;

[0019] The camera movement behavior does not include excessive rotation and / or excessive linear movement;

[0020] The ratio of objects of the first category to objects of the second category does not exceed a first threshold.

[0021] To ensure prediction quality, frame prediction can optionally only be enabled in certain graphics application scenarios. For example, if the graphics application is a video game, information about the current game scene can be used to determine whether the current game mode is expected to have visual artifacts in the predicted frames. If the predicted frames are expected to have visual artifacts, the current game scene may not be suitable for frame prediction. Alternatively, if the expected scene includes excessively dynamic objects, the entity can determine not to enable frame prediction.

[0022] It should be noted that in some cases (e.g., when prediction quality is high), it may be designed to meet all conditions to enable frame prediction. In other cases, meeting one or two conditions is sufficient to enable frame prediction. This can be configured according to the specific implementation.

[0023] In an implementation of the first aspect, the entity is further configured to calculate the one or more predicted frames using a frame warping process (eg, by a GPU).

[0024] Specifically, the frame warping process is specially designed to reduce the power consumption of the entire system.

[0025] In one implementation of the first aspect, the entity is further configured to calculate a predicted frame by: calculating (e.g., by a CPU and / or a GPU) a correspondence between at least two original frames in the image data; and calculating (e.g., by a GPU) the predicted frame based on the correspondence and at least one original frame in the image data. For example, the CPU calculates the correspondence between the at least two original frames and sends the correspondence to the GPU. The GPU can then calculate the predicted frame.

[0026] Specifically, during the frame warping process, the calculated correspondence between the original frames can be obtained, and the predicted frame can be reconstructed (ie, restricted or generated) using the correspondence.

[0027] In an implementation of the first aspect, an original frame includes multiple image segments, and calculating the correspondence between at least two original frames in the image data includes: if at least two image segments in the at least two original frames are marked as one or more objects of the first category, using a first algorithm (for example, by a GPU) to calculate the image segment correspondence between the at least two image segments in the at least two original frames; or, if at least two image segments in the at least two original frames are marked as one or more objects of the second category, using a second algorithm (for example, by a GPU) to calculate the image segment correspondence between the image segments in the at least two original frames.

[0028] As described above, all objects in the image data (i.e., all objects in the intended scene) can be divided into two groups: static objects and dynamic objects. Different algorithms should be used to calculate image segment correspondences for different groups. Specifically, an image segment can be one pixel in size. Different motion algorithms can be used for pixels marked as dynamic objects and pixels marked as static objects.

[0029] In one implementation of the first aspect, the calculating the one or more predicted frames using a frame warping process (for example, by a GPU) includes calculating one or more predicted image segments for the predicted frame using the frame warping process based on the calculated image segment correspondence and at least one image segment.

[0030] Specifically, based on the image segment correspondences calculated using the first algorithm, corresponding predicted image segments are calculated for image segments labeled as static objects (i.e., objects of the first category). Similarly, based on the image segment correspondences calculated using the second algorithm, corresponding predicted image segments are calculated for image segments labeled as dynamic objects (i.e., objects of the second category). Notably, this efficient separation of objects by category (for use with different image segment correspondence algorithms) improves visual quality and reduces power consumption.

[0031] In one implementation of the first aspect, the calculating the one or more predicted frames using the frame warping process further includes: after performing the frame warping process, filling the missing areas in the predicted frame using a shading method (for example, by a GPU), wherein the predicted frame includes multiple image fragments, and the missing areas are parts of the predicted frame other than the one or more predicted image fragments.

[0032] As described above, the one or more predicted image segments may be calculated for the predicted frame. That is, during the frame warping process, the values ​​of some pixels in the predicted frame may not be calculated. A colorization method (e.g., a blurring algorithm) may be used to fill in missing pixels in the predicted frame. Specifically, the method may fill in missing areas using a weighted average of nearby pixels belonging to static objects.

[0033] In one implementation of the first aspect, the image segment correspondence includes a motion vector (MV), the first algorithm includes a reprojection technique, and the second algorithm includes a motion estimation algorithm. For example, the GPU calculates the MV and also calculates the image segment correspondence. In another example, the GPU calculates the MV and sends the MV to the CPU, so that the CPU can calculate the image segment correspondence.

[0034] In a specific implementation, a simple version of the image segment correspondence based on a two-component offset can be calculated. This two-component offset can be called an MV. It is worth mentioning that, for example, to improve image segment matching, other high-order correspondence models can also be used.

[0035] In an implementation of the first aspect, the frame warping process includes: constructing one or more grids for the at least two original frames (for example, by a CPU) in a coordinate system; and performing the frame warping process according to the one or more grids.

[0036] At least one master mesh can be defined for each original frame within the coordinate system. At least one additional mesh can also be defined, wherein the vertex coordinates of the new mesh may not align with the vertex coordinates of the master mesh. A mesh covering all frames can be used. To construct a predicted frame, a transformation according to a specific prediction algorithm can be applied to each vertex of the mesh, and a distorted image can be created accordingly. Specifically, since objects are classified into two different types, two different meshes can be used for each class of objects, and the warping process can also be applied independently.

[0037] In one implementation of the first aspect, the entity is further configured to: (e.g., by a PU and / or GPU) evaluate the quality of each of the one or more predicted frames; and (e.g., by a CPU) discard the predicted frame if the quality of the predicted frame does not meet one or more requirements. For example, the CPU determines to discard the predicted frame based on the quality evaluated by the CPU or the quality sent by the GPU.

[0038] In order to ensure the prediction quality and prevent artifacts from appearing on the predicted frames, the present invention proposes several quality control methods to estimate the quality of each predicted frame. Specifically, the evaluation can be performed in real time on a frame-by-frame basis.

[0039] In one implementation of the first aspect, the entity is configured to evaluate the quality of each of the one or more predicted frames by: evaluating a quality metric by the predicted frame on the GPU (e.g., by the GPU); and determining (e.g., by the GPU) that the quality of the predicted frame does not meet the one or more requirements if one or more artifacts are detected based on the quality metric.

[0040] One quality control method can evaluate a low-complexity quality metric over one or more predicted frames on the GPU, for example, the number of missing pixels for detecting large shaded areas and / or excessive motion for controlling temporal stability. This method can be used to trigger a fallback mechanism, i.e., if artifacts are detected based on the value of this quality metric, the predicted frame should not be used, i.e., it is discarded.

[0041] In an implementation of the first aspect, the entity is used to evaluate the quality of each of the one or more predicted frames in the following manner: (for example, by a CPU) analyzing data of a next original frame obtained from the CPU, wherein the next original frame is a frame after the original frame replaced by the predicted frame; if the difference between the data of the next original frame and the predicted frame exceeds a second threshold, (for example, by a CPU and / or GPU) determining that the quality of the predicted frame does not meet the one or more requirements.

[0042] Another quality control method may rely on information about the camera motion in the "future" frame. Specifically, if the "future" motion differs too much from the predicted frame, a fallback mechanism may also be triggered. It is worth noting that the data analyzed here can come from the next raw frame used in the rendering process (for example, this can be the camera matrix from the next raw frame).

[0043] In an implementation of the first aspect, the graphics application includes one of a game application, an augmented reality (AR) application, and / or a virtual reality (VR) application.

[0044] It is worth noting that the embodiment of the present invention proposes a technical solution that can be compatible with any graphics application or any AR / VR system.

[0045] A second aspect of the present invention provides a terminal for processing image data. The terminal includes an entity according to any one of claims 1 to 16. The terminal can be any one of a mobile phone (e.g., a smartphone), a desktop computer, a laptop computer, a PAD, and the like.

[0046] A third aspect of the present invention provides a method performed by an entity. The method comprises: obtaining image data comprising a series of original frames from a graphics application; determining whether to enable frame prediction to render the image data based on information for enabling prediction obtained from the graphics application; if frame prediction is enabled, enabling the graphics application to render a subset of the series of original frames in the image data to obtain rendered original frames; inserting one or more predicted frames into the rendered original frames obtained from the graphics application, wherein each of the one or more predicted frames is calculated based on one or more original frames in the image data, and each original frame not rendered by the graphics application is replaced by a predicted frame from the one or more predicted frames; and outputting rendered image data, wherein the rendered image data comprises the one or more predicted frames and the rendered original frames obtained from the graphics application.

[0047] The implementation of the method provided in the second aspect may correspond to the implementation of the entity provided in the first aspect. The method provided in the second aspect and its implementation achieves the same advantages and effects as the entity provided in the first aspect and its implementation.

[0048] A fourth aspect of the present invention provides a computer program product comprising program code, wherein when the program code is implemented on a processor, the program product is used to execute the method provided in the second aspect and any one of the implementations of the second aspect.

[0049] It should be noted that all devices, elements, units and modules described in this application can be implemented in software or hardware elements or any type of combination thereof. All steps performed by the various entities described in this application and the functions to be performed by the various entities described are intended to refer to the corresponding entities for performing the corresponding steps and functions. Although in the description of the following specific embodiments, the specific functions or steps performed by external entities are not reflected in the description of the specific detailed elements of the entities performing the specific steps or functions, it should be clear to the technician that these methods and functions can be implemented by corresponding hardware or software elements or any combination thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The following description of specific embodiments in conjunction with the accompanying drawings explains the above-mentioned various aspects and implementation methods of the present invention.

[0051] Figure 1A The structure of an entity provided by an embodiment of the present invention is shown.

[0052] Figure 1B It is a brief schematic diagram of processing image data provided by one embodiment of the present invention.

[0053] Figure 2 Shows the rendering timeline in the traditional method ( Figure 2 (a)) and a rendering timeline provided by an embodiment of the present invention ( Figure 2 (b)).

[0054] FIG3 shows an example of a rendering pipeline for two frames in a conventional method ( FIG3( a )) and an example of a rendering pipeline provided by an embodiment of the present invention ( FIG3( b )).

[0055] Figure 4 The diagram shows the correspondence between two original frames provided by an embodiment of the present invention.

[0056] Figure 5 An example of a frame prediction pipeline provided by an embodiment of the present invention is shown.

[0057] Figure 6 The figure shows an overall diagram of the technical solution provided by one embodiment of the present invention.

[0058] Figure 7 An embodiment of the present invention provides Figure 6 Details of the steps in .

[0059] Figure 8 The quality control steps provided by one embodiment of the present invention are shown.

[0060] Figure 9 The quality control steps provided by one embodiment of the present invention are shown.

[0061] Figure 10 A quality control step provided by one embodiment of the present application is shown.

[0062] Figure 11 Power consumption results for a real game scenario using a conventional method and using the method proposed by the present application are shown.

[0063] Figure 12 A method provided by one embodiment of the present application is shown. DETAILED DESCRIPTION

[0064] Illustrative embodiments of the present application are described with reference to the drawings. Although specific examples and implementations are provided herein, it should be understood that these are included for illustrative purposes only and are not meant to limit the scope of the present application.

[0065] Furthermore, one embodiment / exemplification can refer to other multiple embodiments / exemplifications. For example, any description given on one embodiment / exemplification, including but not limited to terms, elements, processes, explanations and / or technical advantages, apply to other multiple embodiments / exemplifications.

[0066] The present application proposes a method to solve the power consumption problem that occurs in the frame rendering process in high frame rate graphics applications (e.g., games, etc.).

[0067] Embodiments of the present application represent an energy saving system in which GPU / CPU-based rendering supplements any graphics application. The system can be hardware-independent, suitable for devices of different configurations, and does not have a substantial impact on the quality of the rendered frames.

[0068] The field of the present application is real-time graphics applications that support high frame rates.

[0069] Figure 1A and Figure 1BA brief schematic diagram of the structure of the entity 100 and the processing of image data by the entity 100 is shown. The entity 100 can comprise processing circuitry (not shown) for performing, conducting or initiating various operations of the entity 100 described herein. The processing circuitry can comprise hardware and software. The hardware can comprise analog circuitry or digital circuitry, or both analog circuitry and digital circuitry. The digital circuitry can comprise components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors, etc. In particular, one or more processors in the entity 100 can comprise, for example, one or more instances of a CPU, a GPU, a NPU, a processing unit, a processing circuit, a processor, an ASIC, a microprocessor, or other processing logic that can interpret and execute instructions. The entity 100 can also comprise storage circuitry storing one or more instructions that can be executed by the processor or processing circuitry, in particular under control of software. For example, the storage circuitry can comprise a non-transitory storage medium storing executable software code that, when executed by the processor or processing circuitry, causes various operations of the entity 100 to be performed. In one embodiment, the processing circuitry comprises one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory can comprise executable program code that, when executed by the one or more processors, causes the entity 100 to perform, conduct or initiate the operations or methods described herein.

[0070] The entity 100 can be comprised in a terminal, which can be any one of a mobile phone (e.g., a smartphone), a desktop computer, a notebook computer, a PAD, etc. In addition to the entity, the terminal can also comprise a display screen. Thus, when running a graphic application, the terminal can display images of the graphic application according to the image data processing provided by the embodiments of the present application.

[0071] In particular, the entity 100 is configured to obtain image data 101 comprising a series of original frames 1011 from a graphics application 110. In particular, the graphics application can be an application that requires graphics processing such as rendering, for example, the graphics application can be a 3D video game. The entity 100 is also configured to determine whether to enable frame prediction to render the image data 101 according to information 102 for enabling prediction obtained from the graphics application 110. Thus, if frame prediction is enabled, the entity 100 is configured to enable the graphics application 110 to render a subset of the series of original frames 1011 in the image data 101 to obtain rendered original frames.

[0072] Thus, the entity 100 is configured to insert one or more predicted frames 1012 into the rendered original frames obtained from the graphics application 110. In particular, each of the one or more predicted frames 1012 is computed according to one or more original frames 1011 in the image data 101. Notably, each original frame 1011 that is not rendered by the graphics application 110 is replaced by one of the one or more predicted frames 1012.

[0073] The entity 100 is also configured to output rendered image data 103, wherein the rendered image data 103 comprises the one or more predicted frames 1012 and the rendered original frames obtained from the graphics application 110. Notably, by outputting the rendered image data 103, the entity 100 can render these rendered image data on a display screen. The display screen can be comprised in the entity 100 or can be an external display screen.

[0074] In particular, the entity 100 can modify the image data processing pipeline to obtain the image data 101. For example, the 3D scene layer and the UI scene layer can be split and only the 3D layer is used in the prediction algorithm.

[0075] The entity 100 also modifies the behavior of the application to cause the graphics application 110 to reduce the frames to render. For example, the graphics application 110 can render 90 frames per second. When the entity 100 determines to enable frame prediction, the graphics application 110 can only render 45 frames per second, and the entity 100 can “render” the remaining missing 45 frames for the graphics application 110 by computing one or more predicted frames 1012. Since the original frames are replaced instead of simply discarded without replacement, the final frame rate is still 90 frames per second.

[0076] Embodiments of the present invention provide an entity 100 that can optimize (and in particular, reduce) the power consumption generated during frame rendering (e.g., for running a graphics application). This advantage is achieved by having the graphics application 110 not render a portion of a frame (i.e., the original frame 1011), but instead inserting a predicted frame calculated by the entity 100. Because at least a portion of the frame can be predicted, the power consumption of rendering the frame can be reduced. Notably, the original frame rate of the graphics processing can still be maintained because this portion of the original frame 1011 is replaced by the predicted frame, rather than simply deleted.

[0077] Figure 2 Shows the rendering timeline in the traditional method ( Figure 2 (a)) and a rendering timeline provided by an embodiment of the present invention ( Figure 2 FIG3 shows an example of a rendering pipeline for two frames in a conventional method ( FIG3( a )) and an example of a rendering pipeline provided by an embodiment of the present invention ( FIG3( b )).

[0078] Can be obtained from Figure 2 As shown in Figure 3(a), in traditional rendering, frames are processed sequentially. Figure 3(a) further illustrates that to render each of two consecutive frames, in one specific example, the CPU first performs several steps and provides animations to the GPU, which then converts these animations into displays (scenes) on the UI.

[0079] According to an embodiment of the present invention, a prediction frame is calculated. Figure 2 In the example shown in (b), predicted frame 4 can be calculated based on frame 1 and frame 3, where frame 1 and frame 3 are both original frames generated by the graphics application. As shown in Figure 3(b), in order to render the predicted frame, the CPU can directly obtain data (for example, from the previous frame that has been processed) and can provide the data to the GPU. Therefore, the GPU may need to reconstruct the predicted frame and merge the predicted frame with the UI. Specifically, the "UI" here can refer to the UI in the game frame. The graphics application can first render the 3D scene and then draw some UI elements in the main scene. For example, UI elements can be buttons, text messages, game maps, etc. for users to interact with the game.

[0080] It is worth noting that the power consumption generated by the GPU can be slightly increased, but the power consumption generated by the CPU can be significantly reduced. In a specific example, Figure 2 (a) and Figure 3(a), the CPU can use 444mA, while the GPU can use 164mA. This makes the total power consumption about 966mA (this may also include the power consumption of other components such as Wi-Fi, GSM, etc.). However, in Figure 2(b) and (c) in Fig. 3(b), the CPU can use 242 mA and the GPU can use 191 mA. This makes the sum of the power consumption about 758 mA. It can be clearly seen that the power consumption advantage can be achieved by performing the method proposed in the present application.

[0081] The input data for the entity 100 to determine whether the power saving mode can be enabled (i.e. whether frame prediction is enabled) can be semantic scene information, which can be obtained from the graphics application 110 by analyzing the graphics API commands, or from the rendered frames from the graphics application 110. As shown in Fig. 1(a), the frames can be rendered in the CPU and the GPU one by one. That is, when a frame is rendered, different information can be calculated, and thus can be obtained in different steps of the rendering. After a previous frame is rendered, the rendering result can be obtained from the graphics application 110. The semantic information can include at least color, and further include depth, dynamic object mask (second category object mask), texture, stencil and one or more camera matrices. It is worth noting that such information can be used to draw the frame on the display screen, i.e. draw the scene. Figure 2

[0082] According to an embodiment of the present application, the graphics application 110 can be a video game. The current game situation can be determined according to the above information. In particular, such information can be used to:

[0083] 1. Detect the current game situation (e.g. aiming, driving and running).

[0084] 2. Classify the objects in the scene into at least two categories, referred to as static objects and dynamic objects, wherein the static objects have constant world space positions and the dynamic objects change their world space positions due to movement.

[0085] 3. Extract the camera position, orientation and movement in the scene.

[0086] According to such information, i.e. the semantic scene information and the information obtained / extracted therefrom, the quality of the predicted frame 1012 can be determined in advance. If it is expected that the predicted frame 1012 does not include visible artifacts, the reconstruction process of the predicted frame 1012 can start.

[0087] The reconstruction of the predicted frame 1012 is based on the idea that the predicted frame 1012 can be constructed by any method. Preferably, the method is best very simple so that the power consumption produced is less than that of rendering the original frame 1011. In particular, such a method should be applicable to the main part of the application frame. The method can cause some obvious artifacts to be displayed on other areas.

[0088] The input data for the reconstruction process can be related to several original frames 1011, and in one specific example, two original frames 1011 can be used.​

[0089] The reconstruction process provided by the embodiment of the present invention may perform the following steps:

[0090] Step 1: For each original frame 1011 , define at least one master grid within the coordinate system.

[0091] In addition to the main mesh, at least one other mesh may be defined, wherein the vertex coordinates of the other mesh may not be aligned with the vertex coordinates of the main mesh.

[0092] • Another option provided by the present invention is to use a grid with adaptive block size in cases where the expected quality result of the predicted frame 1012 may be poor.

[0093] Step 2: Calculate the image segment correspondence between the two original frames 1011.

[0094] In certain cases, each image segment may have a size of one pixel.

[0095] As mentioned above, all objects in the scene are divided into two groups: static objects and dynamic objects. Different motion estimation algorithms can be used for pixels marked as dynamic objects and pixels marked as static objects.

[0096] It should be noted that each frame includes multiple image segments on the frame. Figure 4 FIG4 shows the image segment correspondence relationship between two original frames provided by an embodiment of the present invention. It can be seen that for an image segment on frame N, after searching in the search area, the best match of the image segment on frame N-1 is found.

[0097] The center of each segment can be considered a node in the grid. Taking the correspondence between at least two original frames as an example, image segment correspondence means finding the correspondence between a node in the first frame and the same node in the second frame. As described above, the present invention proposes classifying objects into two types. Therefore, two different grids can be used for each object category. However, it is also possible to classify objects into more than two types, thereby defining more grids.

[0098] Specifically, according to one embodiment of the present invention, the entity 100 may be configured to: if at least two image segments in the at least two original frames 1011 are labeled as one or more objects of a first category, calculate image segment correspondences between the at least two image segments in the at least two original frames 1011 using a first algorithm. Accordingly, the entity 100 may be configured to: if at least two image segments in the at least two original frames 1011 are labeled as one or more objects of a second category, calculate image segment correspondences between the image segments in the at least two original frames 1011 using a second algorithm. Specifically, the first algorithm should be different from the second algorithm.

[0099] Step 3: After computing the image segment correspondences, the predicted frame 1012 is reconstructed through a frame warping process that is specifically designed to reduce the power consumption of the entire system.

[0100] The frame warping process refers to reconstructing the predicted frame 1012 based on the calculated correspondence between the image segments in the original frame 1011 and the image segments in the predicted frame.

[0101] A frame warping process is performed according to the constructed grid, wherein new pixel values ​​(eg, color values) are calculated according to a predefined image warping process of at least one original frame 1011 .

[0102] In this technical solution, frame warping can be implemented as an extrapolation of the calculated image segment correspondences.

[0103] Step 4: During the frame warping process, the values ​​of some pixels in the predicted frame 1012 may not be calculated. These pixels may be referred to as missing pixels. Optionally, the last step may be to fill in the missing pixels using a coloring method.

[0104] The new result of the frame reconstruction is achieved by improving the visual quality because objects and motion fields are efficiently separated in the original frame 1011 according to the semantic information extracted from the graphics application 110 .

[0105] As mentioned above, the reconstruction of the predicted frame 1012 should be very simple to reduce power consumption. However, a few game frames may cause noticeable artifacts. Embodiments of the present invention also provide methods for estimating quality and preventing artifacts from appearing on the predicted frame 1012.

[0106] According to the embodiment of the present invention, three methods for quality estimation can be proposed:

[0107] The first approach is to analyze semantic scene information, i.e., the first set of information 102, in real time to adaptively control the ratio between the original frame and the predicted frame. Specifically, this may include:

[0108] 1. Use information about the current game scene to determine whether the current game mode is expected to have visual artifacts in the predicted frames; if visual artifacts are expected, disable frame prediction.

[0109] 2. Analyze the camera movement behavior; if the camera movement behavior includes excessive rotation and / or excessive linear movement, disable frame prediction.

[0110] 3. Analyze the ratio of static objects to dynamic objects; if the ratio is greater than a certain threshold, disable frame prediction.

[0111] According to steps 1 to 3, an immediate fallback mechanism for disabling frame prediction is provided.

[0112] It should be noted that the present invention also introduces an intelligent method for determining whether to enable energy saving (i.e., frame prediction). According to one embodiment of the present invention, the entity 100 can be used to determine whether to enable frame prediction if one or more of the following conditions are met:

[0113] The current graphics application scenario matches one or more predetermined graphics application scenarios;

[0114] Camera movement behavior does not include excessive rotation and / or excessive linear movement;

[0115] The ratio of objects of the first category to objects of the second category does not exceed a first threshold.

[0116] That is, to ensure prediction quality, frame prediction can only be enabled in certain graphics application scenarios. It should be noted that in some cases (e.g., when prediction quality is high), all conditions can be met to enable frame prediction. In other cases, meeting one or two conditions is sufficient to enable frame prediction. This can be configured based on the specific implementation.

[0117] The second approach can be done in parallel with rendering the frame on the GPU. Entity 100 can analyze data from the CPU for the next frame (if available), which is the "future" frame from the GPU's perspective. Specifically, this can include:

[0118] Perform quality detection by computing the differences between camera and dynamic object parameters in different frames.

[0119] • A fallback mechanism is used if the “future” camera and object motion behavior is significantly different from the data used to construct the predicted frame 1012. The fallback mechanism is discussed in detail in the next section.

[0120] A third quality control approach can evaluate a low-complexity quality metric over the predicted frames on the GPU, e.g., to detect the number of missing pixels in large shaded areas and / or to control excessive motion for temporal stability; if artifacts are detected based on the value of this quality metric, a fallback mechanism is used.

[0121] Notably, embodiments of the present invention also provide dynamic quality estimation for predicted frames 1012, as well as a fallback mechanism that provides the opportunity to control the ratio between original and predicted frames. The fallback mechanism can include disabling frame prediction, discarding the predicted frame during frame calculation / generation, and discarding the predicted frame after it is generated. Notably, when predicted frames 1012 are discarded, the entity stops replacing original frames 1011 and returns to the original frame rate.

[0122] The concept of using the fallback mechanism implemented in the system is based on the characteristics of the human visual system related to the perception of dynamic and smooth scenes:

[0123] If the game is very dynamic, the human visual system has difficulty detecting rapid changes, so regressions may occur frequently.

[0124] • However, if the game changes are smooth, then fallback is not desirable, but vertex correspondences become very efficient in terms of visual quality.

[0125] The system can use an adaptive fallback mechanism with at least three different modes:

[0126] Due to the signal from the first quality control method (implemented according to the first method for quality estimation), fallback occurs before generating the predicted frame 1012 and causing the graphics application 110 to render the original frame 1011 .

[0127] Fallback occurs after the predicted frame 1012 is generated but before it is displayed. If the shaded area of ​​the predicted frame is of the appropriate size, but the second quality control method has indicated the presence of artifacts, this results in discarding the predicted frame 1012 and then inserting the last original frame 1011 or causing the graphics application 110 to display the next original frame.

[0128] If the predicted frame does not meet the metric threshold, due to the signal from the third quality control method, fallback occurs during the frame generation process, which results in discarding the predicted frame and inserting the previous original frame.

[0129] One possible embodiment of the technical solution provided by the present invention is a system represented by an intermediate software layer between applications and a graphics API. This system can be integrated with the phone firmware or tablet firmware, or can serve as a proxy library and operate during phone right-click or tablet games.

[0130] The phone / tablet should include the basic components of a graphics system: CPU / GPU, memory, input and output devices. The phone / tablet should also include an operating system with graphics API support.

[0131] The application scenario may include: a user starts a mobile phone game, and the entity 100 that enables frame prediction starts working. The entity 100 may be integrated into the mobile phone firmware.

[0132] In one embodiment, the simplest case of image segment correspondence is used. Specifically, the image segment correspondence is based on two-component offsets that are applied to portions of one of the original frames 1011 to match each other; in this case, these two-component offsets are referred to as MVs. It is noted that other higher-order correspondence models may be used to improve frame matching in some cases, etc.

[0133] Figure 5 This shows how to implement a game rendering pipeline when frame prediction is enabled. The details of the "Motion Vector Calculation" step may include:

[0134] 1. This step calculates the MV between two original frames 1011.

[0135] 2. The UI is separated from the main scene and does not affect MV calculation.

[0136] 3. For some pixels marked as dynamic objects, use a motion estimation algorithm to determine the MV to make it more specific but not restrict generalization. Diamond search motion estimation can be used. For the remaining pixels marked as static objects, reprojection techniques can be used to calculate the MV.

[0137] 4. The reprojection technique is based on using an extrapolated MV between static pixel positions in the current and previous frames. One possible modification is to predict the future camera position, which is calculated as an extrapolation of the two previous camera positions. Another modification assumes that the future camera transformation will be similar to the transformation between the two previous frames.

[0138] The details of the "prediction frame generation" step may include:

[0139] 1. The coloring method in the current technical solution is implemented by a fuzzy algorithm, which uses a weighted average of nearby pixels belonging to static objects.

[0140] 2. The UI is rendered on top of the predicted frame 1012 and is equal to the UI in the original frame 1011 .

[0141] Figure 6 The overall solution of analyzing graphics API calls, extracting semantic information, reconstructing prediction frames, and enabling / disabling the current technical solution according to the quality standards evaluated in the quality control step is shown.

[0142] As can be seen, the semantic information analysis results determine whether frame prediction should be enabled. Furthermore, a dynamic quality estimation method is implemented to determine whether a fallback mechanism should be triggered, thereby determining whether frame prediction should be disabled. Details of the quality estimation method and fallback mechanism have been described in previous embodiments.

[0143] Figure 7 Shown is named Figure 6 Detailed information is provided for the "step of generating a predicted frame" in

[15] . Specifically, it can be seen that the two steps for quality control are performed in parallel with the step for generating the predicted frame. This means that fallback can occur during the frame generation process. It is also worth noting that other quality estimation methods are performed after the predicted frame is generated. This means that fallback can also occur after the frame generation process but before the predicted frame is displayed.

[0144] Figure 8 、 Figure 9 and Figure 10 The operations of the three quality control steps provided by the embodiment of the present invention are respectively shown. Each step is implemented according to one of the three methods for quality estimation (of the predicted frame) discussed above.

[0145] Figure 8 The operation of the first quality control step is shown, i.e., for enabling / disabling technical solutions according to the current parameters of camera motion, game scene and number of dynamic objects. Figure 9 The operation of the second quality control step is shown, i.e., for enabling / disabling technical solutions based on information about camera motion in "future" frames. This step can be performed when the "future" frame has not yet been rendered, but the camera motion information for the "future" frame is already available. Figure 10 The operation of the third quality control step is shown, ie, for enabling / disabling technical solutions based on the analysis of the predicted frame on the GPU. The details of these three quality estimation methods have been described in the previous embodiments.

[0146] Figure 11 The technical effect of this embodiment is shown. Specifically, Figure 11 (a) shows the power consumption results of a real-world gaming scenario, i.e., the original game (e.g., a third-person shooter) and the power consumption results of the proposed technology. The measurements were performed at the same initial temperature of the system-on-a-chip (SoC). Figure 11 (b) shows an example of a single measurement, i.e., the CPU power consumption of a 5-minute measurement session. In this embodiment, it can be seen that the power consumption in a real game scenario is about 208.43 mA or an average of about 21.50% ( Figure 11(a)). That is, the power consumption can be reduced by about 208.43 mA or about 21.50% on average.

[0147] Figure 12 Further illustrated is a method 1200 for image data processing provided by one embodiment of the present invention. In a specific embodiment of the present invention, method 1200 is performed by entity 100 shown in FIG1 . Specifically, method 1200 includes: step 1201: obtaining image data 101 including a series of original frames 1011 from graphics application 110; step 1202: determining whether to enable frame prediction to render image data 101 based on prediction-enabling information 102 obtained from graphics application 110. Method 1200 also includes step 1203: if frame prediction is enabled, enabling graphics application 110 to render a subset of the series of original frames 1011 in image data 101 to obtain rendered original frames. Then, method 1200 further includes: step 1204: inserting one or more predicted frames 1012 into the rendered original frame obtained from the graphics application 110, wherein each of the one or more predicted frames 1012 is calculated based on one or more original frames 1011 in the image data 101, and each original frame 1011 not rendered by the graphics application 110 is replaced by one of the predicted frames 1012; step 1205: outputting the rendered image data 103, wherein the rendered image data 103 includes the one or more predicted frames 101 and the rendered original frame obtained from the graphics application 110.

[0148] In summary, the main idea of ​​the present invention is to replace a portion of a frame generated by a graphics application with a predicted frame, which is calculated through a reconstruction process. Embodiments of the present invention include reconstruction of the predicted frame, dynamic quality estimation of the predicted frame, and a fallback mechanism that provides the opportunity to control the ratio between the original frame and the predicted frame.

[0149] The advantages of the present invention over traditional technical solutions may include:

[0150] Reduced power consumption

[0151] Provides a hardware-independent technical solution

[0152] Implemented an adaptive fallback mechanism to maintain visual quality (avoiding situations where there are noticeable visual artifacts)

[0153] Use semantic scene information to achieve high-quality predicted frames while using a simple reconstruction algorithm

[0154] Works with third-party applications without modifying them

[0155] It is worth noting that the technical solution proposed in the embodiments of the present invention can be applied to real-time graphics applications with high frame rates, including games, VR applications, and smart watches on different platforms. This technical solution can be integrated into the rendering engine to improve the effectiveness of using semantic information to achieve high-quality predicted frames. Reconstruction is applied to part of the frame while rendering the rest as usual. This helps to avoid artifacts on objects that are particularly difficult to predict. High-order motion models can be used to explain nonlinear object motion. The technical solution can also be combined with hardware video codecs to reduce power during motion estimation.

[0156] The present invention has been described with reference to various embodiments as examples and implementations. However, those skilled in the art will be able to understand and implement other variations in practicing the claimed invention based on a study of the drawings, the present invention, and the appended claims. In the claims and the specification, the word "comprising" does not exclude other elements or steps, and the quantifier "a" does not exclude a plurality. A single element or other unit may fulfill the functions of several entities or items described in the claims. The enumeration of certain measures in different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation.

[0157] In addition, any method provided by the embodiments of the present invention can be implemented in a computer program having a code module, which, when executed by a processing module, causes the processing module to perform the method steps. The computer program is included in a computer-readable medium of a computer program product. The computer-readable medium can basically include any memory, such as a read-only memory (ROM), a programmable read-only memory (PROM), an erasable PROM (EPROM), a flash memory, an electrically erasable PROM (EEPROM), or a hard disk drive.

[0158] In addition, those skilled in the art will appreciate that embodiments of the entity 100 include necessary communication capabilities in the form of functions, components, units, elements, etc. for implementing the technical solution. Examples of other such components, units, elements, and functions include: a processor, a memory, a buffer, a control logic, an encoder, a decoder, a rate matcher, a derate matcher, a mapping unit, a multiplier, a decision unit, a selection unit, a switch, an interleaver, a deinterleaver, a modulator, a demodulator, an input, an output, an antenna, an amplifier, a receiving unit, a transmitting unit, a DSP, a trellis-coded modulation (TCM) encoder, a TCM decoder, a power supply unit, a power supply line, a communication interface, a communication protocol, etc., which are appropriately configured together to implement the technical solution.

[0159] In particular, the one or more processors in entity 100 may include, for example, one or more instances of a CPU, a GPU, an NPU, a processing unit, a processing circuit, a processor, an ASIC, a microprocessor, or other processing logic that can interpret and execute instructions. Thus, the term "processor" may refer to a processing circuit system that includes a plurality of processing circuits, such as any, some, or all of the aforementioned processing circuits. The processing circuit system may also perform data processing functions for inputting, outputting, and processing data, including data buffering and device control functions, such as call processing control, user interface control, and the like.

Claims

1. An entity (100) for processing image data, characterized in that The entity (100) is used to: Obtaining image data (101) comprising a series of raw frames (1011) from a graphics application (110); determining whether to enable frame prediction to render the image data (101) based on the prediction enabling information (102) acquired from the graphics application (110); If frame prediction is enabled, enabling the graphics application (110) to render a subset of the series of original frames (1011) in the image data (101) to obtain rendered original frames; inserting one or more predicted frames (1012) into the rendered original frame obtained from the graphics application (110), wherein each of the one or more predicted frames (1012) is calculated based on one or more original frames (1011) in the image data (101), and each original frame (1011) not rendered by the graphics application (110) is replaced by a predicted frame from the one or more predicted frames (1012); Outputting rendered image data (103), wherein the rendered image data (103) includes the one or more predicted frames (1012) and the rendered original frame obtained from the graphics application (110); Wherein, the entity (100) is further used for: Calculating the one or more predicted frames using a frame warping process (1012); The entity (100) is configured to calculate a predicted frame (1012) in the following manner: Calculating a correspondence between at least two original frames (1011) in the image data (101); The predicted frame (1012) is calculated based on the correspondence and at least one original frame (1011) in the image data (101).

2. The entity (100) according to claim 1, characterized in that The information (102) for enabling prediction includes a first set of information, the entity for: The first set of information is obtained by analyzing a graphics application programming interface command from the graphics application (110), wherein the first set of information includes color information related to the image data (101), and also includes depth information related to the image data (101), a first category object mask, a second category object mask, and at least one of one or more camera matrices.

3. The entity (100) according to claim 1, characterized in that The information (102) for enabling prediction includes a second set of information, and the entity (100) is configured to obtain the second set of information by: Analyzing a graphics application programming interface command from the graphics application (110) to obtain a first set of information, wherein the first set of information includes color information associated with the image data (101), and at least one of depth information associated with the image data (101), a first category object mask, a second category object mask, and one or more camera matrices; The second set of information is extracted according to the first set of information, wherein the second set of information includes at least one of a current graphics application scene, an object classification, a camera position, a camera orientation, and a camera movement behavior.

4. The entity (100) according to claim 3, characterized in that The entity (100) is also used to: Based on the first set of information, objects in the image data (101) are classified into at least a first category and a second category, wherein objects of the first category have a constant position in a coordinate system and objects of the second category have a dynamic position in the coordinate system.

5. The entity (100) according to claim 3 or 4, characterized in that The entity (100) is used to: Frame prediction is enabled if one or more of the following conditions are met: The current graphics application scene matches one or more predetermined graphics application scenes; The camera movement behavior does not include excessive rotation and / or excessive linear movement; A ratio of objects in the first category to objects in the second category does not exceed a first threshold.

6. The entity (100) according to claim 4, characterized in that An original frame (1011) includes a plurality of image segments, and the calculating of a correspondence between at least two original frames (1011) in the image data (101) includes: If at least two image segments in the at least two original frames (1011) are labeled as one or more objects of the first category, calculating an image segment correspondence relationship between the at least two image segments in the at least two original frames (1011) using a first algorithm; or If at least two image segments in the at least two original frames (1011) are marked as one or more objects of the second category, an image segment correspondence between the at least two image segments in the at least two original frames (1011) is calculated using a second algorithm.

7. The entity (100) according to claim 6, characterized in that The calculating the one or more predicted frames using a frame warping process (1012) includes: One or more predicted image segments are calculated for the predicted frame (1012) using the frame warping process based on the calculated image segment correspondences and at least one image segment.

8. The entity (100) according to claim 7, characterized in that The calculating the one or more predicted frames using a frame warping process (1012) further comprises: After performing the frame warping process, a missing area in the predicted frame (1012) is filled using a coloring method, wherein the predicted frame (1012) includes a plurality of image segments and the missing area is a portion of the predicted frame (1012) other than the one or more predicted image segments.

9. The entity (100) according to claim 6, characterized in that The image segment correspondence includes motion vectors, the first algorithm includes a reprojection technique, and the second algorithm includes a motion estimation algorithm.

10. The entity (100) according to claim 1, characterized in that The frame warping process includes: constructing one or more grids for the at least two original frames (1011) within a coordinate system; The frame warping process is performed according to the one or more grids.

11. The entity (100) according to claim 1, characterized in that The entity (100) is also used to: evaluating a quality of each of the one or more predicted frames (1012); If the quality of a predicted frame (1012) does not meet one or more requirements, the predicted frame (1012) is discarded.

12. The entity (100) according to claim 11, characterized in that The entity (100) is configured to evaluate the quality of each of the one or more predicted frames (1012) by: evaluating a quality metric of the predicted frame (1012) on a graphics processing unit; If one or more artifacts are detected based on the quality metric, it is determined that the quality of the predicted frame (1012) does not meet the one or more requirements.

13. The entity (100) according to claim 11, characterized in that The entity (100) is configured to evaluate the quality of each of the one or more predicted frames (1012) by: Analyzing data of a next original frame obtained from a central processing unit, wherein the next original frame is a frame subsequent to the original frame (1011) replaced by the predicted frame (1012); If the difference between the data of the next original frame and the predicted frame (1012) exceeds a second threshold, it is determined that the quality of the predicted frame (1012) does not meet the one or more requirements.

14. The entity (100) according to claim 1, characterized in that The graphics application (110) includes one of a game application, an augmented reality application and / or a virtual reality application.

15. A terminal for processing image data, characterized in that: The terminal comprises an entity according to any one of claims 1 to 14.

16. The terminal for processing image data according to claim 15, characterized in that: The terminal further includes a display screen, and the entity is configured to output the rendered image data (103) in the following manner: The rendered image data (103) is output to the display screen to display the image of the graphics application.

17. A method (1300) performed by an entity (100), characterized in that The method (1300) comprises: Obtaining (1301) image data (101) comprising a series of raw frames (1011) from a graphics application (110); determining ( 1302 ) whether to enable frame prediction for rendering the image data ( 101 ) based on information for enabling prediction ( 102 ) obtained from the graphics application ( 110 ); If frame prediction is enabled, enabling the graphics application (110) to render a subset of the series of original frames in the image data (101) to obtain rendered original frames; inserting one or more predicted frames (1012) into the rendered original frame obtained from the graphics application (110), wherein each of the one or more predicted frames (1012) is calculated based on one or more original frames (1011) in the image data (101), and each original frame not rendered by the graphics application (110) is replaced by a predicted frame from the one or more predicted frames (1012); Outputting rendered image data (103), wherein the rendered image data (103) includes the one or more predicted frames (1012) and the rendered original frame obtained from the graphics application (110); The method (1300) further includes: Calculating the one or more predicted frames using a frame warping process (1012); The method (1300) comprises calculating a predicted frame (1012) by: Calculating a correspondence between at least two original frames (1011) in the image data (101); The predicted frame (1012) is calculated based on the correspondence and at least one original frame (1011) in the image data (101).

18. A computer program product comprising program code, characterized in that When the program code is implemented on a processor, it is used to perform the method (1300) according to claim 17.

Images