Image processing method and electronic equipment

Abstract

The invention discloses an image processing method and electronic equipment, and relates to the technical field of image processing, and the method comprises the steps: obtaining motion vector information between a first image frame and a second image frame; acquiring a coordinate position of a target point of the target image frame; correspondingly shifting the coordinate position of the target point of the target image frame according to the motion vector information and preset movement coefficients corresponding to the plurality of prediction image frames, and obtaining target point position maps corresponding to the plurality of prediction image frames; obtaining a plurality of prediction image frames based on the target point maps corresponding to the plurality of prediction image frames; and displaying the plurality of predicted image frames. According to the invention, power consumption of the electronic equipment can be reduced while multiple superframes are realized.

Description

[0001] This application is a divisional application of the invention patent application with application number 202310959609.0, application date August 1, 2023, and invention name “An image processing method and electronic device”. Technical Field

[0002] The embodiments of the present application relate to the field of image processing technology, and in particular, to an image processing method and electronic device. Background Art

[0003] As end-side games are constantly updated and iterated, the rendering process of the game engine of the end-side device becomes more complex and diverse, and the resource load is also increasing. Due to the limited battery life of the end-side device, in order to reduce unnecessary rendering overhead without significantly affecting the game quality, predictive frame technology has become a more mainstream research technology. Predictive frame technology refers to learning from historical real image frames to generate future predicted image frames.

[0004] With the continuous development of prediction frame technology, double prediction frame technology and multiple prediction frame technology have emerged. Multiple prediction frame technology refers to inserting multiple prediction image frames between two real image frames. Generating each prediction image frame requires the graphics processing unit (GPU) of the end-side device to perform a lot of calculations. The calculation of multiple prediction image frames by the GPU in a short period of time will increase the frequency and load of the GPU, resulting in a significant temperature rise of the end-side device, which significantly reduces the user experience. Summary of the invention

[0005] An embodiment of the present application provides an image processing method and an electronic device. In the process of implementing multiple interpolation prediction frames, the electronic device can offset the coordinate position of the target point of a non-predicted image frame based on the motion vector information between two adjacent non-predicted image frames calculated by the GPU and the movement coefficients corresponding to the multiple predicted image frames, and obtain the target point bitmaps of the multiple predicted image frames at one time, thereby obtaining the predicted image frames corresponding to the target point bitmaps of the multiple predicted image frames, reducing the number of operations of motion estimation and motion compensation performed by the electronic device, reducing the amount of calculation of the GPU in multiple super-frame processing, and while realizing multiple super-frames to improve the smoothness of the picture, it will not cause excessive load on the GPU.

[0006] In order to achieve the above-mentioned purpose, the embodiments of the present application adopt the following technical solutions.

[0007] In a first aspect, an image processing method is provided, the method comprising:

[0008] The electronic device obtains motion vector information between a first image frame and a second image frame, wherein the first image frame is a non-predicted image frame that has not been displayed, and the second image frame is a non-predicted image frame that has been displayed; and the second image frame is an image frame that is the previous image frame of the first image frame in display timing.

[0009] The electronic device obtains the coordinate position of a target point of a target image frame; the target image frame is a first image frame or a second image frame; the target point includes a geometric vertex corresponding to an object contained in the image frame.

[0010] The electronic device shifts the coordinate position of the target point of the target image frame accordingly according to the motion vector and the preset movement coefficients corresponding to the multiple predicted image frames, and obtains the target point bitmap corresponding to the multiple predicted image frames, wherein the target point bitmap is used to represent the target position of the target point in the predicted image frame.

[0011] The electronic device obtains a plurality of predicted image frames according to the target point bitmaps corresponding to the plurality of predicted image frames.

[0012] The electronic device displays the multiple predicted image frames respectively.

[0013] The electronic device displays the first image frame.

[0014] Among them, the first image frame can be a non-predicted image frame (real image frame) that has been rendered but not displayed and is currently captured; the second image frame can be a non-predicted image frame (real image frame) that has been displayed before the first image frame in the display timing.

[0015] The target point may be a geometric vertex obtained by geometrizing each object in the image frame by any means.

[0016] In this embodiment, the motion vector between the first image frame and the second image frame is the displacement of the dynamic object obtained by performing motion estimation on the first image frame and the second image frame.

[0017] In the present application, the electronic device offsets the coordinate position of the target point of the non-predicted image frame according to the motion vector information between two adjacent non-predicted image frames and the movement coefficients corresponding to multiple predicted image frames, and obtains the target point bitmaps of multiple predicted image frames at one time, thereby obtaining the predicted image frames corresponding to the target point bitmaps of multiple predicted image frames, reducing the number of motion estimation and motion compensation operations performed by the GPU in the electronic device, reducing the amount of calculation of the electronic device in multiple super frame processing, and optimizing the overall performance of the GPU in executing the multiple super frame technology while realizing multiple super frames to improve the smoothness of the picture.

[0018] In a possible implementation manner of the first aspect, the electronic device acquires a coordinate position of a target point of a target image frame, including:

[0019] The electronic device obtains the coordinate position of the target point of the target image frame through the vertex shader; the target point is the geometric vertex of the regular figure obtained by the vertex shader performing regular figure geometrization on the object contained in the target image frame.

[0020] The target image frame may be the first image frame or the second image frame. For example, the motion vector between the second image frame and the first image frame is calculated based on the first image frame, and the target image frame is the first image frame; if the motion vector is calculated based on the second image frame, for example, the motion vector between the first image frame and the second image frame is calculated based on the second image frame, the target image frame is the second image frame.

[0021] The vertex shader can geometrically shape the target image frame, for example, triangulate all objects contained in the target image frame, and obtain each vertex of the triangle as the target point. The objects contained in the image frame include various objects in the picture such as people, trees, and roads.

[0022] In the present application, the electronic device can obtain the coordinate position of the target point in the target image frame through the vertex shader, thereby obtaining the target coordinate position of each target point in the predicted image frame based on the coordinate position and motion vector of the target point in the target image frame. Since the coordinate position of the target point in the target image frame can effectively represent the dynamic elements and static elements in the target image frame, the target coordinate position of each target point in the predicted image frame is calculated based on the coordinate position of the target point in the target image frame, which can more accurately represent the relative positions of the dynamic elements and static elements in the predicted image frame, that is, the obtained predicted image frame is more accurate.

[0023] In another possible implementation manner of the first aspect, the electronic device correspondingly shifts the coordinate position of the target point of the target image frame according to the motion vector and preset movement coefficients corresponding to the multiple predicted image frames, and obtains the target point bitmaps corresponding to each of the multiple predicted image frames, including:

[0024] The electronic device inputs the motion vector information, the movement coefficients corresponding to each of the multiple predicted image frames, and the coordinate position of the target point of the target image frame into the geometry shader to perform corresponding offset operations, and obtains the coordinate position of the target point corresponding to each of the multiple predicted image frames;

[0025] Based on the coordinate positions of the target points of the multiple predicted image frames, the electronic device outputs the target point bitmaps corresponding to the multiple predicted image frames respectively through the geometry shader.

[0026] The preset motion coefficients of the multiple predicted image frames refer to the weight coefficients of the motion vectors corresponding to the respective predicted image frames. Different predicted image frames have different motion coefficients, and the sum of the motion coefficients of the multiple predicted image frames is 1. The motion coefficients of the predicted image frames are determined based on the number of predicted image frames, and the motion coefficients are preset.

[0027] In the present application, the electronic device can perform motion compensation operations on the predicted image frames through a geometric point shader, that is, through the geometric point shader, according to the coordinate position of the target point of the target image frame, the motion vector and the preset movement coefficients of multiple predicted image frames, the target point bitmap corresponding to multiple predicted image frames is obtained at one time, thereby reducing the number of times the electronic device performs motion compensation on multiple predicted image frames and reducing the amount of calculation of the electronic device.

[0028] In another possible implementation manner of the first aspect, the electronic device acquires multiple predicted image frames based on target point bitmaps corresponding to each of the multiple predicted image frames, including:

[0029] The electronic device uses a pixel shader to fill pixels on target point bitmaps of multiple predicted image frames to obtain multiple predicted image frames.

[0030] In the present application, the electronic device uses a pixel shader to fill pixels on the target point bitmap corresponding to each predicted image frame to obtain multiple predicted image frames. It can effectively obtain multiple pixel-filled predicted image frames at one time, improve the efficiency of the motion compensation operation, and reduce the power consumption of the electronic device performing motion compensation.

[0031] In another possible implementation manner of the first aspect, the electronic device performs pixel filling on target point bitmaps of multiple predicted image frames through a pixel shader to obtain multiple predicted image frames, including:

[0032] The electronic device fills pixels of the target point bitmap corresponding to each predicted image frame through a pixel shader to obtain an intermediate result image corresponding to each predicted image frame;

[0033] The electronic device performs image post-processing on the intermediate result images corresponding to each predicted image frame to obtain multiple predicted image frames; the image post-processing includes image completion and image filtering.

[0034] In the present application, there may be a problem of displacement and blanking parts in the multiple predicted image frames obtained by the electronic device through the vertex shader, geometry shader and pixel shader. In order to make the predicted image frames more accurate and more aesthetically pleasing, the electronic device can further perform image processing on the intermediate result image of the predicted image frame output by the pixel shader. This includes image completion processing for the displacement and blanking parts, and filtering processing for the completed edges after the image completion processing, etc., so that the obtained predicted image frames are more accurate.

[0035] In another possible implementation manner of the first aspect, the electronic device is preset with a multi-target rendering memory buffer; the multi-target rendering memory buffer includes a plurality of buffer areas, one buffer area is used to store an intermediate result image of a predicted image frame correspondingly; the method further includes:

[0036] The electronic device outputs the intermediate result images corresponding to the respective predicted image frames to the corresponding buffer areas in the preset multi-target rendering memory buffer.

[0037] The electronic device performs image post-processing on the intermediate result images corresponding to each predicted image frame to obtain multiple predicted image frames; the image post-processing includes image completion and image filtering, including:

[0038] The electronic device sequentially obtains intermediate result images corresponding to the predicted image frames from the preset multi-target rendering memory buffer, performs image post-processing, and obtains the predicted image frames.

[0039] In the present application, the electronic device pre-creates a multi-target rendering memory buffer for receiving intermediate result images of multiple predicted image frames output by a pixel shader. Each buffer area in the multi-target rendering memory buffer can store intermediate result images of multiple predicted image frames correspondingly, thereby realizing batch management of intermediate result images of multiple predicted image frames, so as to perform image completion and filtering processing operations on the intermediate result image of each predicted image frame later.

[0040] In another possible implementation manner of the first aspect, the method further includes:

[0041] The electronic device outputs the intermediate result images corresponding to each predicted image frame to a preset cache queue.

[0042] The electronic device performs image post-processing on the intermediate result images corresponding to each predicted image frame to obtain multiple predicted image frames; the image post-processing includes image completion and image filtering, including:

[0043] The electronic device sequentially obtains intermediate result images corresponding to the predicted image frame from a preset cache queue, performs image post-processing, and obtains the predicted image frame.

[0044] In the present application, the electronic device pre-creates a cache queue for receiving intermediate result images of multiple predicted image frames output by a pixel shader. The electronic device can perform image completion and filtering operations on the intermediate result image of each predicted image frame in turn according to the reading rules set such as first-in-first-out.

[0045] In another possible implementation manner of the first aspect, the electronic device acquires a motion vector between a first image frame and a second image frame, including:

[0046] The electronic device performs motion estimation on the first image frame and the second image frame to obtain a motion vector between the first image frame and the second image frame.

[0047] In the present application, the electronic device may use an optical flow method, an optical structure method, etc. to perform motion estimation on the first image frame and the second image frame to obtain a motion vector between the first image frame and the second image frame. In the present application, the electronic device calculates the motion vector once, and the electronic device may reuse the currently calculated motion vector in subsequent operations, thereby avoiding the operation of obtaining the first image frame and the second image frame multiple times, and also avoiding the operation of calculating the motion vector multiple times, thereby reducing the amount of calculation of the electronic device and reducing the power consumption of the electronic device.

[0048] In another possible implementation manner of the first aspect, the electronic device sequentially displays multiple predicted image frames based on a display order of the multiple predicted image frames, including:

[0049] The electronic device displays the plurality of predicted image frames in sequence based on the display order and display frequency of the plurality of predicted image frames.

[0050] In the present application, multiple predicted image frames correspond to different movement coefficients, and multiple predicted image frames correspond to different display orders. The electronic device displays the predicted image frames according to the display order and display frequency of each predicted image frame, which can improve the display smoothness of the image frame and optimize the user viewing experience.

[0051] In another possible implementation manner of the first aspect, the method further includes:

[0052] The electronic device obtains the display frequency according to the number of multiple predicted image frames and the default refresh frequency of the electronic device.

[0053] In the present application, the electronic device can calculate the display frequency of the predicted image frames according to the number of predicted image frames and the default refresh frequency of the electronic device. For example, the default refresh frequency is 40fps, and an image frame (including real image frames) is displayed every 25ms. The number of predicted image frames is 2, that is, two predicted image frames are inserted between every two real image frames, then the display frequency of the predicted image frames becomes 120fps, and an image frame (including real image frames and predicted image frames) is displayed every 8.3ms. Displaying the predicted image frames at the updated display frequency can indirectly increase the refresh frequency of the image frames, improve the display smoothness of the image frames, and optimize the user viewing experience.

[0054] In another possible implementation manner of the first aspect, a change difference between the first image frame and the second image frame is within a preset difference range.

[0055] In the present application, the image processing method provided in the present application is executed when the difference in change between the first image frame and the second image frame is within a preset difference range. If the difference in change between the first image frame and the second image frame is too large, in order to avoid prediction failure or confusion of image elements in the predicted image frame, multiple superframe processing may not be performed; or, if the difference in change between the first image frame and the second image frame is too small, the first image frame or the second image frame may be directly reused without additional calculation and processing of multiple superframes, thereby reducing the power consumption of the electronic device.

[0056] In a second aspect, an electronic device is provided, which includes a memory, a display screen and one or more processors; the memory and the display screen are coupled to the processor; the memory stores computer program code, and the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device executes a method as described in any one of the above-mentioned first aspects.

[0057] According to a third aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is executed on an electronic device, the electronic device can execute any one of the methods according to the first aspect.

[0058] According to a fourth aspect, a computer program product comprising instructions is provided, which, when executed on an electronic device, enables the electronic device to execute any one of the methods according to the first aspect.

[0059] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor, the processor being used to call a computer program in a memory to execute the method of the first aspect.

[0060] It can be understood that the beneficial effects that can be achieved by the electronic device described in the second aspect, the computer-readable storage medium described in the third aspect, the computer program product described in the fourth aspect, and the chip described in the fifth aspect provided above can refer to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A schematic diagram of a frame interval of a single superframe and a multiple superframe provided in an embodiment of the present application;

[0062] Figure 2 A schematic diagram of a single superframe technology process provided in an embodiment of the present application;

[0063] Figure 3 A schematic diagram of a flow chart of an existing image processing method provided in an embodiment of the present application;

[0064] Figure 4A flowchart of an image processing method provided in an embodiment of the present application;

[0065] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0066] Figure 6 A schematic diagram of the architecture of an electronic device provided in an embodiment of the present application;

[0067] Figure 7 A module interaction diagram of an image processing method provided in an embodiment of the present application;

[0068] Figure 8 A schematic diagram of the results of an image processing method provided in an embodiment of the present application;

[0069] Fig. 9 A schematic diagram of an image formed by a target point provided in an embodiment of the present application;

[0070] Fig.10 A schematic diagram of a dynamic element in an image formed by a target point provided in an embodiment of the present application;

[0071] Fig.11 A schematic diagram of the execution steps of various parts of a geometry shader provided in an embodiment of the present application;

[0072] Fig.12 A schematic diagram of a memory cache and a color attachment provided in an embodiment of the present application;

[0073] Fig.13 A schematic diagram of a processing process of generating a predicted image frame by a geometry shader in an image processing method provided in an embodiment of the present application;

[0074] Fig.14 A schematic diagram of a displacement blanking portion in an intermediate result of a predicted image frame provided in an embodiment of the present application;

[0075] Fig.15 A schematic diagram of the structure of another electronic device provided in an embodiment of the present application;

[0076] Fig.16 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and the appended claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include such expressions as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are a kind of "or" relationship.

[0078] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The term "connection" includes direct connection and indirect connection, unless otherwise specified. "First" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0079] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0080] With the continuous update and iteration of electronic devices, the rendering process of the game engine of electronic devices has become more complex and diverse, and the resource load has also increased. Due to the limited battery life of electronic devices, in order to reduce unnecessary rendering overhead without significantly affecting the game quality, predictive frame technology has become a more mainstream research technology.

[0081] Predictive frame technology refers to learning real image frames and inserting a predicted image frame between two real image frames. Therefore, predictive frame technology can also be called interpolation technology. Among them, real image frames refer to image frames that are normally drawn, rendered, synthesized and displayed, which are different from predicted image frames obtained by predictive frame technology. Exemplarily, predictive frame technology includes frame rate conversion (FRC) technology, motion estimation and motion compensation (MEMC) technology, etc.

[0082] A predicted image frame is a predicted image generated between two real image frames. Inserting a predicted image frame between two real image frames can improve the frame rate of the displayed image, thereby reducing the sense of screen freeze and reducing the power consumption of game rendering. The generation process of the predicted image frame includes: calculating the motion vector between the two real image frames before and after, and then shifting the pixels or pixel blocks of the real image frame based on the motion vector. Under normal circumstances, pixels or pixel blocks that are moved out of the image range are directly discarded, while pixels that are moved from outside the image into the image range need to be specially processed, for example, pixel filling (image completion), filtering and other image post-processing.

[0083] With the continuous development of prediction frame technology, double prediction frame technology and multiple prediction frame technology have emerged one after another. Multiple prediction frame technology refers to inserting multiple prediction image frames between two real image frames. However, the calculation of generating prediction image frames is mostly completed in the graphics processing unit (GPU) of the terminal device, and the GPU calculates multiple prediction image frames in a short time. Specifically, each time the GPU generates a prediction image frame, it needs to obtain the two image frames before and after the prediction image frame for calculation. To generate multiple prediction image frames, it is necessary to obtain the required two image frames before and after multiple times, and perform multiple motion estimation and motion compensation calculations based on the two image frames before and after. In this way, the more prediction image frames are inserted, the greater the amount of calculation of the GPU, resulting in an increase in the frequency and load of the GPU, resulting in a significant temperature rise of the terminal device. That is, the existing technology cannot achieve a balance between the multiple super frames to improve the smoothness of the picture and the power consumption (temperature) of the electronic device, which significantly reduces the user experience of the electronic device.

[0084] For example, compared with the single superframe technology, the multiple superframe technology generates multiple predicted image frames between two real image frames. Taking the double superframe as an example, Figure 1A schematic diagram of a real image frame, a single superframe, and a double superframe is given. Exemplarily, when superframe processing is not performed, the electronic device generates a real image frame at a transmission frame rate of 40 frames per second (fps), and the time interval between each two real image frames is 25ms. The electronic device uses the single superframe technology to generate a predicted image frame, and a predicted image frame is evenly inserted between each two real image frames. The time interval between each two image frames (including real image frames and predicted image frames) is 12.5ms, and the electronic device refreshes the image frame at 80fps. The electronic device uses the double superframe technology to generate a predicted image frame, and two predicted image frames are evenly inserted between each two real image frames. The time interval between each two image frames (including real image frames and predicted image frames) is 8.3ms, and the electronic device refreshes the image frame at 120fps.

[0085] Figure 2 A module interaction diagram of a single superframe technology is given. When an application generates a rendering requirement for a first real image frame, the central processing unit (CPU) of the electronic device performs a logical operation. The logical operation includes responding to inputs from peripheral devices, processing message data of the application, and precalculating data required for rendering. The CPU sends the image resource to the GPU for rendering, and sends the rendered first real image frame to the display module for synthesis and display.

[0086] When the application generates a rendering requirement for the second real image frame, and a single superframe operation is required, the CPU sends the image resource to the GPU for rendering after performing a logical operation. While rendering the current real image frame, the GPU obtains the first real image frame and calls the prediction frame processing module to process the prediction image frame according to the image resource (first real image frame, second real image frame) and rendering data. The obtained prediction image frame is sent to the display module for synthesis and display.

[0087] After sending the predicted image frame for display, the GPU sends the rendered second real image frame to the display module for synthesis and display.

[0088] In the prior art, to implement the multiple superframe technology, it is necessary to perform motion estimation, motion compensation, image completion, and image post-processing based on two adjacent real image frames. Figure 3 , Figure 3A flowchart for realizing a double superframe is given. To realize a double superframe, that is, insert two predicted image frames into two adjacent real image frames, it is necessary to perform motion estimation, motion compensation, image completion and image post-processing based on the two adjacent image frames to obtain predicted image frame A; then perform motion estimation, motion compensation, image completion and image post-processing based on the two adjacent image frames to obtain predicted image frame B. Thus, according to the display order of predicted image frame A and predicted image frame B, and the display frequency after multiple superframes, predicted image frame A and predicted image frame B are displayed between two adjacent real image frames.

[0089] Obviously, in the prior art, the GPU of the electronic device needs to perform motion estimation, motion compensation, image compensation and image post-processing operations based on the previous real image frame and the current real image frame in the process of generating the predicted image frame A; in the process of generating the predicted image frame B, it also needs to perform motion estimation, motion compensation, image compensation and image post-processing operations based on the previous real image frame and the current real image frame. That is, if the electronic device wants to generate N predicted image frames, it needs to perform N corresponding motion estimation, motion compensation, image compensation and image post-processing operations based on the previous real image frame and the current real image frame. Motion estimation and motion compensation are the main calculation steps in the prediction frame technology, and the time consumed by them accounts for 80% of the entire prediction image frame operation. In the process of multiple super frames, the GPU will inevitably increase its frequency and load by performing multiple motion estimation and motion compensation operations, resulting in a significant temperature rise of the end-side equipment, increasing the power consumption of the GPU, etc., which significantly reduces the user experience.

[0090] An embodiment of the present application provides an image processing method, which obtains motion vectors obtained by motion estimation of two previous and next real image frames based on the coordinate positions of each element of the two previous and next real image frames. The electronic device can offset the coordinate positions of target points of the non-predicted image frames according to the motion vector information between two adjacent non-predicted image frames and the movement coefficients corresponding to multiple predicted image frames, thereby obtaining the coordinate positions of target points of multiple predicted image frames, thereby obtaining multiple predicted image frames based on the coordinate positions of the target points of the multiple predicted image frames, and achieving the effect of obtaining multiple predicted image frames at one time. In the process of obtaining multiple predicted image frames, there is no need to calculate the motion vectors corresponding to each predicted image frame again and again to perform motion compensation to obtain multiple predicted image frames, which effectively reduces the GPU computing amount and overall power consumption of the electronic device, and while achieving the multiple super frame effect, balances the power consumption and smoothness of the GPU of the electronic device, and improves the device performance of the electronic device in executing multiple super frames.

[0091] The image processing method provided in the embodiment of the present application can be referred to Figure 4 shown. Figure 4A flowchart of the image processing method using double superframe as an example is given. Figure 3 In the process of generating predicted image frame A and predicted image frame B in the prior art, Figure 4 In an embodiment of the present application, the electronic device can perform motion estimation based on the previous real image frame and the current real image frame, and perform motion compensation based on the motion vector obtained by the motion estimation. In the process of generating predicted image frame A and predicted image frame B, the calculated motion vector can be reused for motion compensation, which reduces the number of motion estimation and motion compensation based on the previous real image frame and the current real image frame, and greatly reduces the amount of calculation of the GPU of the electronic device. For the later processes such as image completion and image post-processing, the computing resources consumed are relatively small, and each predicted image frame can be processed independently to obtain the final predicted image frame A and predicted image frame B. Then, the predicted image frame A and the predicted image frame B are displayed between the previous real image frame and the current real image frame according to the display frequency.

[0092] The image processing method provided in the embodiment of the present application can be applied to an electronic device. Figure 5 The electronic device 100 shown. The electronic device 100 may be a portable computer (such as a mobile phone), a tablet computer, a laptop computer, a personal computer (PC), a wearable electronic device (such as a smart watch), an augmented reality (AR) device, a virtual reality (VR) device, a car computer, or other device with a display function. The following embodiments do not impose any special restrictions on the specific form of the electronic device.

[0093] Please refer to Figure 5 , which shows a structural block diagram of an electronic device (such as electronic device 100) provided in an embodiment of the present application. The electronic device 100 may include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, a communication module 360, an audio module 370, a sensor module 380, a camera 393, a display screen 394, etc.

[0094] The structure shown in the embodiment of the present invention does not limit the electronic device 100. It may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0095] The processor 310 may include one or more processing units. For example, the processor 310 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0096] The controller can be a decision maker that directs the various components of the electronic device 100 to work in coordination according to the instructions. It is the nerve center and command center of the electronic device 100. The controller generates an operation control signal according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0097] The processor 310 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 310 is a high-speed cache memory, which can store instructions or data that the processor 310 has just used or cyclically used. If the processor 310 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 310, and thus improves the efficiency of the system.

[0098] In some embodiments, the processor 310 may include an interface. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0099] The interface connection relationship between the modules shown in the embodiment of the present invention is only for illustrative purposes and does not constitute a structural limitation on the electronic device 100. The electronic device 100 may adopt different interface connection methods in the embodiment of the present invention, or a combination of multiple interface connection methods.

[0100] The charging management module 340 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 340 may receive charging input from a wired charger through the USB interface 330. In some wireless charging embodiments, the charging management module 340 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 340 is charging the battery 342, it may also power the electronic device 100 through the power management module 341.

[0101] The power management module 341 is used to connect the battery 342, the charging management module 340 and the processor 310. The power management module 341 receives input from the battery 342 and / or the charging management module 340, and supplies power to the processor 310, the internal memory 321, the external memory interface 320, the display screen 394, the camera 393, and the communication module 360. The power management module 341 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some embodiments, the power management module 341 can also be set in the processor 310. In some embodiments, the power management module 341 and the charging management module 340 can also be set in the same device.

[0102] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the communication module 360, the modem and the baseband processor.

[0103] Antenna 1 is used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, a cellular network antenna can be reused as a wireless local area network diversity antenna. In some embodiments, the antenna can be used in combination with a tuning switch.

[0104] The modem may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 370A, a receiver 370B, etc.), or displays an image or video through a display screen 394. In some embodiments, the modem may be an independent device. In some embodiments, the modem may be independent of the processor 310 and be arranged in the same device as the RF module 350 or other functional modules.

[0105] The communication module 360 ​​can provide a communication processing module for wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (blue tooth, BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR), etc., which are applied to the electronic device 100. The communication module 360 ​​can be one or more devices integrating at least one communication processing module. The communication module 360 ​​receives electromagnetic waves via the antenna 1, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 310. The communication module 360 ​​can also receive the signal to be sent from the processor 310, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 1.

[0106] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the communication module 360, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global satellite positioning system (satellite based augmentation systems, SBAS), a global navigation satellite system (global navigation satellite system, GLONASS), a BeiDou navigation satellite system (BeiDou navigation satellite system, BDS), a Quasi-Zenith satellite system (Quasi-Zenith satellite system, QZSS) and / or a satellite based augmentation system (satellite based augmentation systems, SBAS).

[0107] The electronic device 100 implements the display function through a GPU, a display screen 394, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 394 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 310 may include one or more GPUs, which execute program instructions to generate or change display information.

[0108] The display screen 394 is used to display images, videos, etc. The display screen 394 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 394, where N is a positive integer greater than 1.

[0109] In the embodiment of the present application, the display screen 394 is used to display image frames, including real image frames and predicted image frames. Figure 4 In the example given, the display screen 394 displays the image frames in the order of the last real image frame, the predicted image frame A, the predicted image frame B, and the current real image frame.

[0110] The electronic device 100 can realize the shooting function through ISP, camera 393, video codec, GPU, display screen and application processor.

[0111] The ISP is used to process the data fed back by the camera 393. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise and brightness of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 393.

[0112] The camera 393 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 393, where N is a positive integer greater than 1.

[0113] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0114] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0115] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0116] The external memory interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 310 through the external memory interface 320 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0117] The internal memory 321 can be used to store computer executable program codes, which include instructions. The processor 310 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 321. The memory 321 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the memory 321 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, other volatile solid-state storage devices, a universal flash storage (UFS), etc.

[0118] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 370, speakers, receivers, microphones, headphone interfaces, and application processors.

[0119] The audio module 370 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 370 can also be used to encode and decode audio signals. In some embodiments, the audio module 370 can be arranged in the processor 310, or some functional modules of the audio module 370 can be arranged in the processor 310.

[0120] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. Taking the system as an example, the software structure of the electronic device 100 is exemplified.

[0121] Figure 6 1 is a software structure diagram of the electronic device 100 of an embodiment of the present invention. The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, The system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system library, and kernel layer.

[0122] The application layer can include a series of application packages.

[0123] like Figure 6 As shown, the application package may include applications such as games, camera, gallery, calendar, phone, map, WLAN, Bluetooth, music, short message, etc.

[0124] Among them, games, videos, or live broadcast applications require a certain degree of smoothness when playing pictures. Based on this requirement, the image processing method provided in this embodiment can balance the power consumption of electronic devices while achieving multiple super frame effects, thereby improving the overall experience of multiple super frame technology in electronic devices.

[0125] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0126] like Figure 6 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, a memory manager, and the like.

[0127] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0128] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0129] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0130] The phone manager is used to provide communication functions of the electronic device 100. For example, the management of call status (including answering, hanging up, etc.).

[0131] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0132] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0133] Android runtime includes core library and virtual machine. Android runtime is responsible for scheduling and management of Android system. Core library consists of two parts: one is the function that Java language needs to call, and the other is Android core library.

[0134] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0135] The system library may include multiple functional modules, such as image processing library, image engine, media library, image resource processing module, prediction frame processing module, etc.

[0136] The predicted frame processing module may include a motion estimation unit, a motion compensation unit, an image completion unit, etc. The predicted frame processing module is used to generate a predicted image frame after acquiring an image resource (real image frame).

[0137] The image resource processing module is used to obtain image resources (real image frames) and transmit the image resources to the prediction frame processing module.

[0138] The image processing library is used to draw, render, synthesize and display real images. The image processing library includes a 2D graphics processing library and a 3D graphics processing library. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0139] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of multiple common audio and video formats, as well as static image files. The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0140] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, Wi-Fi driver, communication driver, audio driver, sensor driver, Bluetooth driver, etc.

[0141] Figure 7 A flowchart of the interaction between different layers and multiple modules to implement the image processing method is given, refer to Figure 7 , taking the application as a game as an example.

[0142] The game application generates an image rendering event and sends the image resource data to the image processing library. The 3D graphics library in the image processing library performs image drawing, rendering and other operations based on the received image resource data.

[0143] The image resource processing module intercepts the image resource data (current real image frame) sent by the game application to the image processing library, and transmits the intercepted image resource data to the prediction frame processing module.

[0144] The motion estimation unit of the prediction frame processing module obtains the previous real image frame of the current real image frame according to the image resource data (current real image frame), performs motion estimation according to the current real image frame and the previous real image frame, and obtains a motion vector.

[0145] The motion compensation unit of the prediction frame processing module performs motion compensation on a plurality of prediction image frames according to the motion vectors calculated by the motion estimation unit to obtain a plurality of prediction image frames.

[0146] The motion compensation unit obtains the refresh frequency of the predicted image frame, and transmits each predicted image frame to the image completion unit of the predicted image frame processing module in sequence according to the refresh frequency. The refresh frequency of the predicted image frame can be calculated based on the default frequency of the electronic device and the number of predicted image frames.

[0147] The image completion unit of the prediction frame processing module performs image completion and other image post-processing on each received prediction image frame to obtain the prediction image frame to be displayed. The image completion unit sends the prediction image frame to be displayed to the display driver for display.

[0148] After the image completion unit completes the display of the predicted image frame, the 3D image library sends the rendered current real image frame to the display driver for synthesis and display.

[0149] The image processing method provided in the embodiment of the present application is referred to as Figure 8 As shown, image resources are intercepted in the rendering of real image frames, and based on the previous real image frame and the current real image frame, the calculation results (motion vectors) of motion estimation are reused in the process of generating predicted image frames. In the motion compensation (also called image warping) stage, intermediate result images of multiple predicted image frames are generated at one time, and image post-processing such as image completion and filtering are performed on the intermediate result images of the multiple predicted image frames. This can reduce the number of motion estimation and motion compensation operations while completing multiple predicted image frames, thereby reducing the power consumption caused by the predicted frame operation.

[0150] Specifically, the image processing method provided in the embodiment of the present application includes a super frame strategy stage in which the electronic device matches the rendering data of the image resource and a super frame strategy execution stage.

[0151] Among them, the electronic device matches the superframe strategy based on the rendering data of the image resource to match the most effective superframe strategy for the current image resource, and can also reduce the superframe operation when superframe processing is not required, thereby avoiding additional power consumption caused by redundant calculations of the electronic device.

[0152] Optionally include:

[0153] S201: A CPU of an electronic device receives a rendering event generated by an application and obtains image resources to be rendered.

[0154] The application may be a game application, a video application, a live broadcast application, etc. When the application displays a frame of an image, the electronic device needs to draw, render, synthesize and display the frame before the image can be displayed on the display screen. When the application generates a rendering event, the CPU of the electronic device can obtain the image resource to be rendered to perform the drawing, rendering and synthesis and display operations of the current real image frame.

[0155] In this embodiment, the image resource to be rendered is the image resource corresponding to the current real image frame, and the current real image frame may also be referred to as the first image frame.

[0156] S202: The CPU determines a superframe strategy that matches the rendering data of the image resource.

[0157] Among them, the superframe strategy includes multiple preset strategies such as exiting superframe processing (quiet), multiplexing images (reuse), and multiple superframe processing (mesh). Among them, exiting superframe processing refers to not performing single or multiple superframe processing operations, and directly rendering, synthesizing, and sending real image frames for display. Multiplexing images refers to inserting any real image frame between two real image frames as a false prediction image frame for display. Multiple superframe processing refers to the operation of processing multiple prediction image frames based on the method of multiplexing running vectors provided in this embodiment.

[0158] The image resource acquired by the CPU represents the current real image frame. The CPU can acquire the previous real image frame, and based on two adjacent real image frames, determine the change difference between the two adjacent real image frames to determine the super frame strategy matching them.

[0159] In some embodiments, if the difference in changes between two adjacent real image frames is large, it can be determined that the superframe strategy to be matched is to exit the superframe processing (quiet). For example, the previous real image frame includes a setting interface, and the current real image frame does not include the setting interface. In the case where at least one of the two adjacent real image frames includes a functional static interface such as a menu interface and a setting interface, there is no need to perform superframe processing to avoid the occurrence of positions such as picture dislocation caused by the predicted setting interface. At this time, the CPU can match its superframe strategy to exit the superframe processing (quiet). Alternatively, the game scene of the previous real image frame is A, and the game scene of the current real image frame is B. Scene switching is performed between the two real image frames, and the difference in the switching changes of this scene is large. If single or multiple superframe processing is used, it may be that the scene of the predicted image frame does not belong to scene A or scene B. At this time, the CPU can match its superframe strategy to exit the superframe processing (quiet).

[0160] In some embodiments, if there is almost no change difference between two adjacent real image frames, the matching super frame strategy reuse image (reuse) can be determined. For example, the two adjacent real image frames contain static elements, and static elements, as the name implies, will not change with the flow of time or operation. In this case, the change difference between the current real image frame and the image of the previous real image frame is less than the minimum value of the preset difference range, and the CPU can match its super frame strategy to reuse the image (reuse), that is, reuse the current real image frame or the previous real image frame as a pseudo-predicted image frame, and display the pseudo-predicted image frame between the current real image frame or the previous real image frame.

[0161] In some embodiments, if the change difference between two adjacent real image frames is within a preset difference range, the CPU may match its super frame strategy to multiple super frame processing (mesh).

[0162] Optionally, the multiple superframe processing (mesh) may include a superframe processing method based on network vertices, a superframe processing method based on pixels, etc. In this embodiment, the multiple superframe processing is performed using a superframe processing method based on mesh vertices.

[0163] S203: The CPU sends a superframe strategy matching the current image resources to the GPU.

[0164] In this embodiment, the GPU of the electronic device performs multiple superframe processing. After determining the superframe strategy that matches the current image resource, the CPU sends the superframe strategy to the GPU, and the GPU performs corresponding operations according to the received superframe strategy.

[0165] The embodiment of the present application takes the superframe strategy of multiple superframe processing (mesh) as an example for explanation.

[0166] S204: GPU obtains the last real image frame.

[0167] In this embodiment, the previous real image frame may also be referred to as the second image frame. The GPU may obtain the second image frame from the display queue.

[0168] S205 . The GPU performs motion estimation according to the first image frame and the second image frame to obtain a motion vector.

[0169] Among them, motion estimation is to extract the key feature points or contours of the object by processing the continuous image frames of the physical object, and infer the motion state of the object based on the position changes of these feature points. Commonly used motion estimation algorithms include optical flow method, structured light method, parallax method, etc.

[0170] In this embodiment, the GPU may use the optical flow method to perform motion estimation (motion vector) on the first image frame and the second image frame to obtain the motion vector MV between the two frames. The optical flow method is to infer the moving direction and speed of the object by analyzing the brightness change of the pixel points in the image. The optical flow is a two-dimensional vector field, which represents the displacement of a point from the first frame to the second frame.

[0171] S206 . The GPU performs motion compensation for the multiple predicted image frames according to the motion vectors to obtain intermediate result images of the multiple predicted image frames.

[0172] In this embodiment, the GPU can obtain the coordinate position of each target point in the first image frame and the second image frame. The target point can be a pixel point in the image, or a vertex that can represent each object in the image and is generated by geometric mapping based on the image frame. Fig. 9 , Fig. 9 A geometric mapping based on image frames is given, using triangles to represent the target point map of all objects in the image frame. Among them, each vertex of the triangle is the target point. If there are dynamic elements in the image frame, the vertices and side lengths of the triangles corresponding to the dynamic elements will be distorted. Fig.10 . Fig.10 The target point map of a dynamic object in the shape of a person is included in the figure. In order to better illustrate the shape and position of the person, Fig.10 The distorted parts of the target point bitmap are replaced by the character outline.

[0173] In this embodiment, the GPU can calculate the target coordinate position of the target point corresponding to each predicted image frame based on the coordinate position of each target point in the first image frame and the second image frame, and the preset target point movement coefficient of the predicted image frame. Based on the target coordinate position of the target point corresponding to each predicted image frame, the target point bitmap of each predicted image frame is obtained, and the pixel texture processing is performed on the target point bitmap to obtain the intermediate result image of each predicted image frame.

[0174] Optionally, the GPU can complete the above operation through a comprehensive shader, wherein the comprehensive shader includes a vertex shader (VS), a geometry shader (GS) and a pixel shader (FS). The process of the GPU performing motion compensation through the vertex shader (VS), the geometry shader (GS) and the pixel shader (FS) can be referred to Fig.11 shown.

[0175] The GPU inputs a real image frame and a motion vector into the vertex shader (VS) to obtain the coordinate positions of each target point in the real image frame. Among them, a real image frame can be a first image frame or a second image frame. If the motion vector is calculated based on the first image frame, for example, the element displacement (motion vector) between the second image frame and the first image frame is calculated based on the first image frame, then a real image frame is a first image frame; if the motion vector is calculated based on the second image frame, for example, the element displacement (motion vector) between the first image frame and the second image frame is calculated based on the second image frame, then a real image frame is a second image frame.

[0176] After obtaining the coordinate position and motion vector of the target point in the real image frame, based on the number of predicted image frames processed by multiple super frames and the movement coefficients corresponding to each predicted image frame, a target point bitmap formed by the target coordinate position of the target point after motion compensation corresponding to each predicted image frame is obtained.

[0177] Exemplarily, double superframe processing and the default refresh rate of an electronic device of 40fps are taken as an example for explanation.

[0178] The GPU may create a frame buffer (FB) for storing the predicted image frames and initialize the FB as a multiple render target (MRT) buffer. For example, the GPU may create a color attachment in the FB for placing each predicted image frame. The color attachment may also be understood as a buffer area for storing the intermediate result image of the predicted image frame output by the FS. Fig.12 shown.

[0179] For example, the specific implementation method of GPU creating and initializing FB can be referred to as follows:

[0180] glBindFramebuffer(GL_FRAMEBUFFER,fbo);

[0181] glFramebufferTexture2D(GL_FRAMEBUFFER,GL_COLOR_ATTACHMENT0,GL_TEXTURE_2D,textureA,0);

[0182] glFramebufferTexture2D(GL_FRAMEBUFFER,GL_COLOR_ATTACHMENT1,GL_TEXTURE_2D,textureB,0);

[0183] Among them, textureA represents the area where color attachment 1 is located, and textureB represents the area where color attachment 2 is located.

[0184] The memory buffer is used to receive the intermediate result image of the predicted image frame obtained by processing by the vertex shader (VS), the geometry shader (GS) and the pixel shader (FS).

[0185] Color attachment 1 (color0) is used to store the first predicted image frame; color attachment 2 (color1) is used to store the second predicted image frame.

[0186] The GPU obtains the motion coefficients corresponding to each predicted image frame in the double superframe process. For example, the motion coefficient corresponding to the first predicted image frame can be preset to 0.33; the motion coefficient corresponding to the second predicted image frame can be preset to 0.67.

[0187] Through the geometry shader (GS), the GPU obtains the target coordinate position of the target point after motion compensation corresponding to each predicted image frame based on the coordinate position of the target point in the real image frame, the movement coefficient corresponding to the first predicted image frame, and the movement coefficient corresponding to the second predicted image frame.

[0188] Exemplarily, the target coordinate position of the target point in the first predicted image frame can be expressed as:

[0189] GS0: position+0.33*MV.

[0190] Among them, position is the coordinate position of the target point in the real image frame; MV is the motion vector.

[0191] Exemplarily, the target coordinate position of the target point in the second predicted image frame can be expressed as:

[0192] GS0: position+0.67*MV.

[0193] Among them, position is the coordinate position of the target point in the real image frame; MV is the motion vector.

[0194] The GPU can obtain the target coordinate positions of target points in multiple predicted image frames at one time through the geometry shader (GS).

[0195] After obtaining the coordinate position of the target point in the first predicted image frame and the coordinate position of the target point in the second predicted image frame, the GPU performs pixel texture processing on the image formed by the target point in the first predicted image frame through the pixel shader (FS) to obtain an intermediate result image of the first predicted image frame. The GPU performs pixel texture processing on the image formed by the target point in the second predicted image frame to obtain an intermediate result image of the second predicted image frame.

[0196] Among them, the GPU performs pixel texture processing through the pixel shader (FS) refers to color rendering of the image formed by the target point, and the target point image is rendered into a color image. Since the GPU completes the calculation of the target coordinate position of the target point of multiple predicted image frames through the geometry shader (GS), the GPU performs texture sampling through the pixel shader (FS) to complete the color output. Exemplarily, the specific implementation method of the GPU performing texture sampling through the pixel shader (FS) to complete the color output can be referred to as follows:

[0197] color0=texture(cur,texcorrd);

[0198] color1=texture(cur,texcorrd);

[0199] The GPU can specify that the output of the pixel shader (FS) points to color attachment 1 and color attachment 2.

[0200] For example, the GPU may specify that the output of the pixel shader (FS) points to the color attachment 1 (color0) and the color attachment 2 (color1). The specific implementation method may refer to the following:

[0201] layout(location=1)out lowp vec4 color0;

[0202] layout(location=2)out lowp vec4 color1.

[0203] The flow chart of GPU motion compensation through vertex shader (VS), geometry shader (GS) and pixel shader (FS) can be referred to Fig.13shown.

[0204] After the electronic device inputs the motion vector and the real image frame into the vertex shader (VS), the coordinate position of the target point of the real image frame can be obtained; the coordinate position of the target point of the real image frame, the motion vector and the movement coefficient of each predicted image frame are input into the geometry shader (GS), and the geometry shader (GS) can output the image formed by the coordinate positions of the target points corresponding to multiple predicted image frames at one time ( Fig.13 In the given example, an image formed by the coordinate positions of the target points corresponding to two predicted image frames is output); the image formed by the coordinate positions of the target points corresponding to multiple predicted image frames is input to a pixel shader (FS), and the pixel shader (FS) fills pixels in the image formed by the coordinate positions of the target points corresponding to each predicted image frame to obtain a predicted image frame.

[0205] It should be noted that the intermediate result image of the predicted image frame output by the GPU to each color attachment in the FB through the vertex shader (VS), geometry shader (GS) and pixel shader (FS) contains the displacement blank part of the dynamic elements. The pixel shader (FS) can mark the displacement blank part. For example, a transparent channel is set to represent the displacement blank part, and its corresponding marking value is 0.5. The transparent channel can also represent other elements in the image. For example, static elements can be marked as 0 and dynamic elements can be marked as 1. The marking value is just an example and is not limited in this embodiment.

[0206] Optionally, the electronic device may also directly store the intermediate result image of the predicted image frame output by the pixel shader (FS) in a preset cache queue.

[0207] S207 , the GPU performs image completion and image post-processing on the intermediate result image of each predicted image frame to obtain a corresponding predicted image frame.

[0208] In this embodiment, the electronic device may obtain the intermediate result image of each predicted image frame from each color attachment of the FB; or may obtain the intermediate result image of each predicted image frame in sequence from a preset cache queue.

[0209] The GPU performs image completion and image post-processing on the intermediate result image of the predicted image frame in each color attachment in the FB to obtain the corresponding predicted image frame.

[0210] The GPU outputs the predicted image frame in each color attachment in the FB through the vertex shader (VS), geometry shader (GS) and pixel shader (FS). There are blank parts of dynamic elements in the intermediate result image, which need to be completed. Fig.14 The offset margin indicated.

[0211] Exemplarily, the GPU can obtain the pixel points corresponding to the displacement blank part according to the mark value of the preset transparent channel, and perform pixel completion of the pixel points according to the pixel values ​​of the pixel points in the non-blank part of the preset area of ​​each pixel point. For example, according to the pixel points B, C, D, and E in the non-blank part included in the preset area of ​​pixel point A of the displacement blank part, the pixel average value of pixel point B, pixel point C, pixel point D, and pixel point E is calculated, and the pixel value of pixel point A is used for pixel completion. Alternatively, the pixel value of pixel point F that is closest to pixel point A in the non-blank part included in the preset area is directly obtained, and the pixel value of pixel point A is used for pixel completion.

[0212] It should be noted that the non-blank portion included in the preset area of ​​pixel point A does not include dynamic elements, that is, the non-blank portion included in the preset area of ​​pixel point A includes static elements such as background, but does not include dynamic elements such as predicted characters.

[0213] In this embodiment, after image completion, there may be a problem that the edges of the completed part are obvious. Optionally, the GPU may perform filtering processing on the intermediate result image of the predicted image frame after image completion. Exemplarily, the filtering processing includes Gaussian filtering, mean filtering, adaptive smoothing filtering, and the like.

[0214] S208. The GPU sends the predicted image frames for display in sequence according to the refresh frequency of the multiple super-frame processing.

[0215] The default refresh rate of electronic devices is 40fps. After double superframe processing, the refresh rate is updated to 120fps. After the GPU sends the second image frame (previous real image frame) for display, it sends the first predicted image frame for display at an interval of 8.3mm; after the first predicted image frame is sent for display, the second predicted image frame is sent for display at an interval of 8.3mm; after the second predicted image frame is sent for display, the first image frame (current real image frame) is sent for display at an interval of 8.3mm, thereby realizing the purpose of displaying the first predicted image frame and the second predicted image frame between the first image frame and the second image frame, displaying the image frames at a higher refresh rate, and making the display screen of the electronic device smoother.

[0216] Because in the embodiment of the present application, the GPU calculates the target coordinate positions of the target points of multiple predicted image frames by reusing motion vectors, thereby avoiding multiple data acquisition operations of the first image frame and the second image frame, and reducing the number and amount of calculations for motion estimation based on the first image frame and the second image frame; in addition, the calculation of the target coordinate positions of the target points of multiple predicted image frames at one time is performed through a geometric shader, which also reduces the number and amount of calculations for motion compensation performed by the GPU, greatly reducing the power consumption and amount of calculation generated by the GPU in the process of performing multiple super frames, and optimizing the overall performance of the GPU in executing multiple super frame technology.

[0217] Fig.15 A possible structural diagram of the electronic device involved in the above embodiments is shown. Fig.15 The electronic device 1000 shown includes a processor 1001 , a display screen 1002 , and a storage module 1003 .

[0218] Among them, the processor 1001 can be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The processor may include an application processor and a baseband processor. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The storage module 1005 may be a memory.

[0219] For example, the processor 1001 may be Figure 5 The processor 310 shown; the display screen 1002 can be as shown Figure 5 The camera 393 shown, the storage module 1003 can be as follows Figure 5 The internal memory 321 shown. The electronic device provided in the embodiment of the present application can be Figure 5 An electronic device 100 is shown.

[0220] The present application also provides a chip system (eg, a system on a chip (SoC)), such as Fig.16As shown, the chip system includes at least one processor 701 and at least one interface circuit 702. The processor 701 and the interface circuit 702 can be interconnected by lines. For example, the interface circuit 702 can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit 702 can be used to send signals to other devices (such as a processor 701 or a camera of an electronic device). Exemplarily, the interface circuit 702 can read instructions stored in the memory and send the instructions to the processor 701. When the instruction is executed by the processor 701, the electronic device can perform the various steps in the above embodiments. Of course, the chip system may also include other discrete devices, which are not specifically limited in the embodiments of the present application.

[0221] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes each function or step executed by the electronic device 100 in the above-mentioned method embodiment.

[0222] The embodiment of the present application further provides a computer program product, which, when executed on a computer, enables the computer to execute the functions or steps executed by the electronic device 100 in the above method embodiment. For example, the computer may be the above electronic device 100.

[0223] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0224] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0225] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0226] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0227] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0228] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method is applied to an electronic device, and the method comprises: Acquire motion vector information between a first image frame and a second image frame, wherein the first image frame is a non-predicted image frame that has not been displayed, the second image frame is a non-predicted image frame that has been displayed, and the second image frame is a previous non-predicted image frame of the first image frame in display timing; Acquire the coordinate position of a target point of a target image frame, wherein the target image frame is the first image frame or the second image frame, and the target point includes a geometric vertex corresponding to an object contained in the image frame; According to the motion vector information and the preset movement coefficients corresponding to the plurality of predicted image frames, the coordinate position of the target point of the target image frame is shifted accordingly to obtain a target point bitmap corresponding to the plurality of predicted image frames, wherein the target point bitmap is used to represent the target position of the target point in the predicted image frame; Acquire the plurality of predicted image frames based on the target point bitmaps respectively corresponding to the plurality of predicted image frames; The plurality of predicted image frames are displayed.

2. The method according to claim 1, characterized in that The step of obtaining the coordinate position of the target point of the target image frame includes: The coordinate position of the target point of the target image frame is obtained by a vertex shader, wherein the target point is a geometric vertex of a regular figure obtained by the vertex shader performing regular figure geometrization on an object contained in the target image frame.

3. The method according to claim 1, characterized in that The step of shifting the coordinate position of the target point of the target image frame accordingly according to the motion vector information and the movement coefficients corresponding to the plurality of predicted image frames, and obtaining the target point bitmaps corresponding to the plurality of predicted image frames respectively, comprises: Input the motion vector information, the movement coefficients corresponding to each of the multiple predicted image frames, and the coordinate position of the target point of the target image frame into a geometry shader for performing corresponding offset operations to obtain the coordinate position of the target point corresponding to each of the multiple predicted image frames; Based on the coordinate positions of the target points of the multiple predicted image frames, the target point bitmaps corresponding to the multiple predicted image frames are outputted through the geometry shader.

4. The method according to claim 1, characterized in that The acquiring the multiple predicted image frames based on the target point bitmaps respectively corresponding to the multiple predicted image frames comprises: The target point bitmaps corresponding to the multiple predicted image frames are filled with pixels through a pixel shader to obtain the multiple predicted image frames.

5. The method according to claim 4, characterized in that The step of performing pixel filling on target point bitmaps corresponding to the plurality of predicted image frames respectively through a pixel shader to obtain the plurality of predicted image frames comprises: Filling pixels of the target point bitmap corresponding to each predicted image frame with a pixel shader to obtain an intermediate result image corresponding to each predicted image frame; Perform image post-processing on the intermediate result images corresponding to each of the predicted image frames to obtain a plurality of the predicted image frames, wherein the image post-processing includes image completion and image filtering.

6. The method according to claim 5, characterized in that The electronic device is preset with a multi-target rendering memory buffer; The multi-target rendering memory buffer includes a plurality of buffer areas, one buffer area is used to store an intermediate result image of a prediction image frame, and the method further includes: Outputting the intermediate result images corresponding to the respective predicted image frames to corresponding buffer areas in a preset multi-target rendering memory buffer; Performing image post-processing on the intermediate result images corresponding to each of the predicted image frames to obtain a plurality of the predicted image frames; The image post-processing includes image completion and image filtering, including: sequentially acquiring intermediate result images corresponding to the predicted image frame from the preset multi-target rendering memory buffer for image post-processing to acquire the predicted image frame.

7. The method according to claim 5, characterized in that The method further comprises: Outputting the intermediate result images corresponding to each of the predicted image frames to a preset cache queue; Performing image post-processing on the intermediate result images corresponding to each of the predicted image frames to obtain a plurality of the predicted image frames; The image post-processing includes image completion and image filtering, including: sequentially acquiring intermediate result images corresponding to the predicted image frame from the preset cache queue for image post-processing to acquire the predicted image frame.

8. The method according to any one of claims 1 to 7, characterized in that The change difference between the first image frame and the second image frame is within a preset difference range.

9. The method according to any one of claims 1 to 7, characterized in that: The obtaining of the motion vector between the first image frame and the second image frame comprises: Perform motion estimation on the first image frame and the second image frame to obtain a motion vector between the first image frame and the second image frame.

10. The method according to any one of claims 1 to 7, characterized in that The displaying of the plurality of predicted image frames comprises: According to the number of the plurality of predicted image frames and the default refresh frequency, acquiring a display transmission frequency; Based on the display order of the multiple predicted image frames and the display frequency, the multiple predicted image frames are displayed.

11. An electronic device, characterized in that: The electronic device includes a memory, a display screen and one or more processors; the memory and the display screen are coupled to the processor; the memory stores computer program code, and the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1-10.

12. A computer-readable storage medium, characterized in that: The method comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 10.

Images