Method, device, electronic equipment, storage medium and program product for image processing

Abstract

The present disclosure relates to the technical field of image processing, and specifically provides a method and apparatus for image processing, an electronic device, a storage medium and a program product. The method comprises: in a process of rendering a to-be-processed video, if a frame loss event is detected, obtaining a first video frame and a second video frame which are currently rendered and to be displayed; the first video frame and the second video frame are two adjacent video frames of the to-be-processed video; the frame loss event indicates a rendering timeout; generating an intermediate frame between the first video frame and the second video frame according to motion information of pixels between the first video frame and the second video frame; and displaying the first video frame, the intermediate frame and the second video frame. In this way, when a frame loss occurs, an intermediate frame between the first video frame and the second video frame can be generated in time based on motion information of pixels between the first video frame and the second video frame to perform frame interpolation, thereby improving the fluency of the video.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to an image processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] When playing a video, it is usually necessary to render, composite, and display the original video frames in sequence.

[0003] If there is a stutter during rendering, that is, the rendering of the current video frame times out, it will cause frame dropping, which will result in one less video frame being displayed.

[0004] Traditional techniques typically repeat the previous video frame when a frame drop event is detected. However, this method results in poor video smoothness. Summary of the Invention

[0005] To address the aforementioned technical problems, this disclosure provides an image processing method, apparatus, electronic device, storage medium, and program product.

[0006] In one aspect, this disclosure provides an image processing method, including:

[0007] During the rendering of the video to be processed, if a frame drop event is detected, the first and second video frames that have been rendered and are ready to be displayed are obtained; the first and second video frames are two adjacent video frames of the video to be processed; the frame drop event indicates a rendering timeout.

[0008] Based on the motion information of pixels between the first video frame and the second video frame, an intermediate frame between the first video frame and the second video frame is generated.

[0009] Display the first video frame, intermediate frames, and the second video frame.

[0010] In one embodiment, generating an intermediate frame between the first video frame and the second video frame based on the motion information of pixels between the first video frame and the second video frame includes:

[0011] Multiple cascaded encoders are used to extract intermediate features between the first and second video frames;

[0012] Multiple decoders are cascaded to obtain intermediate optical flow estimates for the first and second video frames based on intermediate features, and intermediate frames are generated based on the intermediate optical flow estimates.

[0013] In this model, intermediate optical flow estimation represents the motion information of pixels; multiple cascaded encoders and multiple cascaded decoders constitute the frame prediction model.

[0014] In one implementation, the method includes:

[0015] According to the set duration, the rendering status of the currently rendered third video frame is detected periodically.

[0016] If, based on the detection results, it is determined that the third video frame failed to render successfully within the set time period, then a frame drop event is detected.

[0017] In one embodiment, displaying a first video frame, an intermediate frame, and a second video frame includes:

[0018] If the third video frame is successfully rendered within the set time after the frame drop event, the first video frame, the intermediate frame, the second video frame, and the third video frame will be displayed in sequence.

[0019] The first video frame is rendered earlier than the second video frame.

[0020] In one embodiment, displaying a first video frame, an intermediate frame, and a second video frame includes:

[0021] If consecutive frame dropping events are detected, a new intermediate frame is obtained based on the intermediate frame and the second video frame; consecutive frame dropping events indicate that the third video frame fails to render again within the next set time period.

[0022] The first video frame, the intermediate frame, the new intermediate frame, and the second video frame are displayed sequentially.

[0023] The first video frame is rendered earlier than the second video frame.

[0024] In one aspect, this disclosure provides an image processing apparatus, comprising:

[0025] The detection unit is used to obtain the first and second video frames that are currently rendered and ready to be displayed if a frame drop event is detected during the rendering process of the video to be processed; the first and second video frames are two adjacent video frames of the video to be processed; the frame drop event indicates a rendering timeout.

[0026] The prediction unit is used to generate an intermediate frame between the first video frame and the second video frame based on the motion information of the pixels between the first video frame and the second video frame.

[0027] The display unit is used to display the first video frame, the intermediate frame, and the second video frame.

[0028] In one implementation, the prediction unit is used to:

[0029] Multiple cascaded encoders are used to extract intermediate features between the first and second video frames;

[0030] Multiple decoders are cascaded to obtain intermediate optical flow estimates for the first and second video frames based on intermediate features, and intermediate frames are generated based on the intermediate optical flow estimates.

[0031] In this model, intermediate optical flow estimation represents the motion information of pixels; multiple cascaded encoders and multiple cascaded decoders constitute the frame prediction model.

[0032] In one embodiment, the detection unit is used for:

[0033] According to the set duration, the rendering status of the currently rendered third video frame is detected periodically.

[0034] If, based on the detection results, it is determined that the third video frame failed to render successfully within the set time period, then a frame drop event is detected.

[0035] In one embodiment, the display unit is used for:

[0036] If the third video frame is successfully rendered within the set time after the frame drop event, the first video frame, the intermediate frame, the second video frame, and the third video frame will be displayed in sequence.

[0037] The first video frame is rendered earlier than the second video frame.

[0038] In one embodiment, the display unit is used for:

[0039] If consecutive frame dropping events are detected, a new intermediate frame is obtained based on the intermediate frame and the second video frame; consecutive frame dropping events indicate that the third video frame fails to render again within the next set time period.

[0040] The first video frame, the intermediate frame, the new intermediate frame, and the second video frame are displayed sequentially.

[0041] The first video frame is rendered earlier than the second video frame.

[0042] In one aspect, this disclosure provides an electronic device, including:

[0043] Processor; and

[0044] The memory stores computer instructions that cause the processor to perform the steps of the methods provided in the various alternative implementations of any of the image processing described above.

[0045] In one aspect, this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to perform the steps of the methods provided in various alternative implementations of any of the above-described image processing methods.

[0046] On one hand, this disclosure provides a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the steps of the method provided in various alternative implementations of any of the above-described image processing methods.

[0047] The image processing method in this embodiment includes, during the rendering of a video to be processed, if a frame drop event is detected, acquiring a first video frame and a second video frame that have been rendered and are to be displayed; the first video frame and the second video frame are two adjacent video frames of the video to be processed; the frame drop event indicates a rendering timeout; generating an intermediate frame between the first video frame and the second video frame based on the motion information of the pixels between the first video frame and the second video frame; and displaying the first video frame, the intermediate frame, and the second video frame. In this way, when frame drop occurs, an intermediate frame between the first video frame and the second video frame can be generated in a timely manner based on the motion information of the pixels between the first video frame and the second video frame for frame interpolation, thereby improving the smoothness of the video. Attached Figure Description

[0048] Figure 1 This is a timing diagram of a rendering anomaly in an embodiment of this disclosure.

[0049] Figure 2 This is a flowchart of an image processing method according to an embodiment of the present disclosure.

[0050] Figure 3 This is a structural example diagram of a frame prediction model in an embodiment of this disclosure.

[0051] Figure 4 This is an example diagram of a device architecture in an embodiment of this disclosure.

[0052] Figure 5 This is an interactive diagram of a frame prediction method according to an embodiment of this disclosure.

[0053] Figure 6 This is a detailed flowchart of an image display method according to an embodiment of the present disclosure.

[0054] Figure 7 This is a timing diagram of a frame interpolation method in an embodiment of this disclosure.

[0055] Figure 8 This is a timing diagram of a continuous frame interpolation method in an embodiment of this disclosure.

[0056] Figure 9 This is a structural block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0057] Figure 10This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0058] The technical solutions of this disclosure will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure. Furthermore, the technical features involved in the different embodiments of this disclosure described below can be combined with each other as long as they do not conflict with each other.

[0059] In real-world applications, when an application (APP) plays a video (e.g., a movie), the APP typically renders the original video frames, then uses a SurfaceFlinger to composite the rendered video frames, and finally displays the composited video frames on a monitor.

[0060] If a stutter occurs during rendering, it will cause frame drops, and SurfaceFlinger will detect a frame pending (isFramePending) event, i.e., a dropped frame event. Since the video frame has not been fully rendered, SurfaceFlinger will not perform compositing processing in this cycle, resulting in one less video frame being sent to the display panel.

[0061] The following is combined with Figure 1 The above rendering anomaly is explained below. See [link / reference]. Figure 1 The diagram shown is a timing diagram of a rendering anomaly. Figure 1 In this code, the app is used for rendering, SurfaceFlinger is used for graphics compositing, and the panel is used for display. `Vsync i` represents the i-th cycle, where i is a positive integer. `frame j` represents the j-th video frame, where j is a positive integer. For example, frame 1 completes the app's rendering in Vsync 1, graphics compositing in Vsync 2, and is displayed on the monitor / screen in Vsync 3.

[0062] Because frame 3 takes two cycles to render, it times out, and SurfaceFlinger will detect `isFramePending` in Vsync4. Since frame 3 is not rendered in time, subsequent graphics compositing and display cannot proceed. Traditionally, frame 2 is typically displayed again in Vsync5, and the processed frame 3 is displayed in Vsync6.

[0063] However, by using this method, the user actually watched frame 2 twice, resulting in poor video smoothness.

[0064] In view of the deficiencies of the aforementioned related technologies, this disclosure provides an image processing method, apparatus, electronic device, storage medium, and program product, which aim to improve the smoothness of video.

[0065] This disclosure provides an image processing method that can be applied to electronic devices. This disclosure does not limit the type of electronic device, which can be any suitable type of device, such as terminal devices and servers, etc. This disclosure will not elaborate further.

[0066] See Figure 2 The diagram shown is a flowchart of an image processing method according to an embodiment of this disclosure. The following is a description of the method in conjunction with... Figure 2 The method is described below, and the specific implementation process is as follows:

[0067] Step 201: During the rendering of the video to be processed, if a frame dropping event is detected, the first and second video frames that have been rendered and are yet to be displayed are obtained.

[0068] The first and second video frames are two adjacent video frames of the video to be processed; a frame drop event indicates a rendering timeout. Optionally, the video to be processed can be an animation, or any video such as High-Dynamic Range (HDR) video.

[0069] In one implementation, the rendering status of the currently rendered third video frame is periodically checked according to a set duration; if the detection result determines that the third video frame has not been successfully rendered within the set duration, a frame dropping event is detected.

[0070] In other words, a frame dropping event is triggered when the rendering time of the currently rendered video frame reaches any integer multiple of the set time.

[0071] For example, the set duration can be represented as T. When the rendering duration of a video frame reaches T, a frame drop event is triggered, and when the rendering duration of a video frame reaches 2T, a frame drop event is triggered again.

[0072] In practical applications, any method can be used to detect whether a frame loss event exists; no restrictions are imposed here.

[0073] Step 202: Generate an intermediate frame between the first video frame and the second video frame based on the motion information of the pixels between the first video frame and the second video frame.

[0074] In one implementation, the first video frame and the second video frame are input into a frame prediction model to obtain an intermediate frame between the first video frame and the second video frame.

[0075] The frame prediction model is used to extract intermediate features of the first video frame and the second video frame using multiple cascaded encoders, and to obtain intermediate optical flow estimates of the first video frame and the second video frame based on the intermediate features using multiple cascaded decoders, and to generate intermediate frames based on the intermediate optical flow estimates; the intermediate optical flow estimates represent the motion information of pixels.

[0076] The electronic device includes SurfaceFlinger, and its hardware abstraction layer also includes an intermediate feature optimization plugin, as well as an image enhancement chip.

[0077] SurfaceFlinger is used to call the intermediate feature optimization plugin in the hardware abstraction layer when a frame loss event is detected, and send a frame interpolation command to the image enhancement chip. The feature optimization plugin is used to provide an interface for calling the image enhancement chip. The image enhancement chip is used to obtain the first video frame and the second video frame from multiple buffers in shared memory corresponding to the video to be processed based on the frame interpolation command, and input the first video frame and the second video frame into the frame prediction model to obtain the intermediate frame.

[0078] The frame prediction model consists of multiple cascaded encoders and multiple cascaded decoders. In one implementation, step 202 can be performed using the following steps:

[0079] Multiple cascaded encoders are used to extract intermediate features from the first and second video frames; multiple cascaded decoders are used to obtain intermediate optical flow estimates of the first and second video frames based on the intermediate features, and intermediate frames are generated based on the intermediate optical flow estimates.

[0080] The cascaded encoders function as follows: In video frame interpolation, multiple cascaded encoders are used to progressively extract features from the video frames. The main role of the encoders is to extract features and learn representations from the input data. For the first and second video frames, the cascaded use of multiple encoders allows for a more comprehensive and in-depth capture of various feature information within the video frames. For example, the initial encoder might extract basic low-level features such as edges and textures, while subsequent encoders will further extract more abstract and semantically meaningful high-level features based on the previous encoders. This progressively deeper feature extraction approach helps to better understand the content of the video frames, providing a richer information foundation for subsequent optical flow estimation. The principle is as follows: Cascaded encoders are typically implemented using neural networks, with each encoder considered a small neural network module. These encoders are connected in a specific order, with the output of the previous encoder serving as the input to the next. During forward propagation, the video frames are processed sequentially by each encoder, with each encoder performing specific transformations and feature extraction operations on the input data. In this way, an intermediate feature representation containing rich feature information can ultimately be obtained.

[0081] The function of cascaded decoders is as follows: obtaining intermediate optical flow estimation based on intermediate features is a crucial step in video frame interpolation. Multiple cascaded decoders are used to estimate the intermediate optical flow between the first and second video frames based on the intermediate features extracted by the encoder. Optical flow estimation refers to calculating the motion trend of pixels in an image, i.e., the direction and distance of movement of each pixel per unit time. By decoding and analyzing intermediate features, combined with prior knowledge and algorithm models, the decoders can infer the motion of pixels between two video frames, thus obtaining the intermediate optical flow estimate. Using multiple decoders in cascade improves the accuracy and robustness of optical flow estimation because each decoder can further analyze and process the intermediate features, progressively optimizing the optical flow estimation result. The principle is as follows: similar to the encoder, the decoder is also constructed using a neural network. During training, the decoder learns how to predict optical flow information based on intermediate features. During inference, the intermediate features extracted by the encoder are input into the cascaded decoders. Each decoder processes the input features according to its own parameters and algorithms, producing the corresponding optical flow estimation result. The next decoder will further refine and improve the optical flow estimation based on the previous decoder to obtain more accurate results.

[0082] The function of generating intermediate frames is as follows: Generating intermediate frames based on intermediate optical flow estimation is the ultimate goal of video frame interpolation. After obtaining the intermediate optical flow estimates for the first and second video frames, this optical flow information can be used to generate intermediate frames. Intermediate optical flow estimation provides motion information for pixels. Based on this information, pixels in the first video frame can be moved and interpolated according to their corresponding motion trends, thereby generating intermediate frames. This makes the generated intermediate frames visually more natural and smooth, as if they were real frames from the original video sequence. The principle is as follows: The process of generating intermediate frames involves coordinate transformation and interpolation operations on pixels. Based on the pixel motion vectors obtained from the intermediate optical flow estimation, each pixel in the first video frame can be moved to a new position. For some pixels that require interpolation, their pixel values ​​can be calculated using a weighted average or other interpolation algorithms based on information from surrounding pixels. Finally, by combining all pixels according to their new positions and pixel values, the generated intermediate frame can be obtained.

[0083] Step 203: Display the first video frame, the intermediate frame, and the second video frame.

[0084] The first video frame is rendered earlier than the second video frame.

[0085] In one implementation, if it is determined that the third video frame is successfully rendered within a set time after the frame loss event, the first video frame, the intermediate frame, the second video frame, and the third video frame are displayed sequentially.

[0086] Furthermore, if it is determined that frame dropping events are detected continuously, a new intermediate frame is obtained based on the intermediate frame and the second video frame; if frame dropping events are detected continuously, it means that the third video frame fails to render again within the next set time period; the first video frame, the intermediate frame, the new intermediate frame, and the second video frame are displayed in sequence.

[0087] In one application scenario, intermediate frames can be added to the buffer corresponding to shared memory. If the rendering time of the currently rendered third video frame reaches twice the set time, the frame drop event triggered by the currently rendered video frame is detected again. The intermediate frames and the second video frame in the shared memory buffer are input into the frame prediction model to obtain new intermediate frames, so that the intermediate frames in the shared memory buffer can be used as the new first video frame.

[0088] Similarly, if the same video frame triggers frame dropping events consecutively, new intermediate frames can be generated continuously.

[0089] The structure of the frame prediction model is explained below. (See also...) Figure 3The diagram shown is an example of the structure of a frame prediction model. The frame prediction model includes an encoder network and a decoder network. The encoder network is constructed based on multiple cascaded encoders. The decoder network is constructed based on multiple cascaded decoders. The input to the frame prediction model is multiple target video frames (i.e., the first video frame and the second video frame), and the output is the intermediate frame. The frame prediction model can also be called a multi-frame interpolation algorithm.

[0090] Furthermore, the number of target video frames input to the model can be multiple, without any limitation.

[0091] Furthermore, during model training, a loss function can be used to calculate the difference between the intermediate frames and the corresponding reference frames, and the model parameters can be optimized based on this difference to obtain a well-trained frame prediction model. Optionally, the optical flow distillation loss function can be used. Alternatively, the feature space geometric consistency loss function can be combined with model training. The feature space geometric consistency loss function is used to calculate the difference between the feature maps of the intermediate frames and the reference frames, and the model parameters can then be optimized based on this difference, so that the feature maps of the intermediate frames can more accurately reflect the geometric structure of the feature maps of the reference frames.

[0092] Traditionally, motion estimation and motion compensation (MEMC) algorithms are used with a dynamic imaging system to insert a motion compensation frame between two original frames. However, this method relies on a complex network structure and has a high inference latency.

[0093] In this embodiment, the frame prediction model includes multiple cascaded encoders and multiple cascaded decoders. Intermediate features can be extracted through the cascaded encoders, and intermediate optical flow estimates can be generated through the cascaded decoders, thereby generating intermediate frames. The network structure is simple and has a more compact model structure. Compared with traditional optical flow algorithms, it eliminates the optical flow extraction process and can improve inference speed.

[0094] The following is combined Figure 4 The device architecture of the electronic equipment used to apply image processing methods is described. (See also...) Figure 4 The diagram shown is an example of a device architecture.

[0095] Figure 4 In this context, the device architecture includes the following layers: application layer, framework layer, hardware abstraction layer (HAL), driver layer, and hardware layer.

[0096] The application layer is the top layer of the electronic device and contains the app. It provides interfaces and services for interacting with the device, enabling users to easily use the device's functions.

[0097] The framework layer sits below the application layer and includes SurfaceFlinger. It provides the application layer with a common framework and toolset, helping developers build applications more efficiently. The framework layer offers a set of predefined components, libraries, and templates for handling common programming tasks such as user interface design, database access, and network communication. This allows developers to focus on implementing business logic without having to write all the code from scratch.

[0098] The Hardware Abstraction Layer (HAL) is an abstraction layer located between the operating system or software system and the underlying hardware. It includes a Hardware Composer (HWComposer) and Intermediate Feature Refine (IFR) plugins. The HAL hides the details of the underlying hardware. Its main goal is to provide a standard, unified hardware access interface for the software system, allowing software to be developed and run independently of specific hardware platforms. Thus, regardless of changes in the underlying hardware, as long as the HAL remains unchanged, the upper-layer software can continue to run with little or no modification.

[0099] The driver layer sits below the hardware abstraction layer and interacts directly with the hardware device. It contains the Device Driver Model (DRM). Drivers are key software components that control the operation of hardware devices. They are responsible for translating requests from the software system into instructions that the hardware device can understand and for handling the hardware device's responses.

[0100] The hardware layer is the foundational layer of the device, composed of various physical components such as chips, modules, and sensors. The hardware layer includes a data processing unit (DPU), an image enhancement chip, a display panel, a Mobile Industry Processor Interface (MIPI) transmitter (TX), and a MIPI receiver.

[0101] The following is combined Figure 2 , Figure 4 and Figure 5 For an explanation of the frame prediction method, please refer to [link / reference]. Figure 5 The diagram shown is an interaction diagram of a frame prediction method. The implementation process of this method includes:

[0102] Step 501: The APP renders the video to be processed.

[0103] The app stores the rendered video frames into the corresponding buffer.

[0104] Step 502: SurfaceFlinger detects a frame drop event.

[0105] In one implementation, the APP continuously renders images of the video to be processed during operation. SurfaceFlinger records the rendering status of each video frame. If isFramePending occurs, i.e. a frame dropping event is detected, it is determined that the APP rendering has resulted in frame dropping.

[0106] Step 503: SurfaceFlinger detects two target video frames that have been rendered but not yet output.

[0107] Furthermore, within shared memory, a corresponding BufferQueue can be set as a cache queue for the app. BufferQueue can also be represented as Buffer-Tx. BufferQueue contains multiple Buffers.

[0108] After rendering each video frame, the app can place each frame into a different buffer, and delete the frame from its corresponding buffer after it is displayed on the panel. A buffer can hold only one video frame. In one application scenario, for smooth playback, a BufferQueue typically contains three buffers, thus holding a maximum of three target video frames.

[0109] In one implementation, the capacity of Buffer-Tx can be checked. If there are at least two unprocessed buffers in Buffer-Tx, that is, if it is determined that there are at least two target video frames that have been rendered but not output, then in subsequent steps, the image with missing frames can be completed based on multiple target video frames in Buffer-Tx.

[0110] Furthermore, if each buffer in the APP contains only one usable video frame, then the traditional method of repeatedly displaying that video frame can be used to handle frame dropping.

[0111] Step 504: SurfaceFlinger calls the IFR Plugin to send frame interpolation instructions to the image enhancement chip.

[0112] The frame interpolation instruction can include the video resolution. The IFR Plugin is an interface used to invoke the frame prediction model for frame interpolation.

[0113] Step 505: The image enhancement chip acquires the first video frame and the second video frame.

[0114] The image enhancement chip can directly access the shared memory to obtain the first and second video frames from each buffer in the shared memory.

[0115] Step 506: The image enhancement chip inputs the first video frame and the second video frame into the frame prediction model to obtain the intermediate frame.

[0116] Step 507: The image enhancement chip returns the intermediate frame to SurfaceFlinger via the IFR Plugin.

[0117] In one implementation, the image enhancement chip can input the first video frame and the second video frame into a frame prediction model to obtain a rendered intermediate layer, i.e., a rendered intermediate frame, and store the rendered intermediate frame in a shared memory buffer. SurfaceFlinger and HWComposer process the rendered intermediate frames in the buffer sequentially to obtain the processed intermediate frames.

[0118] See Figure 6 The diagram shown is a detailed flowchart of an image display method. The specific implementation process of this method includes:

[0119] Step 601: The APP renders the video to be processed.

[0120] Step 602: SurfaceFlinger determines whether a frame dropping event has been detected. If yes, proceed to step 603; otherwise, proceed to step 606.

[0121] Step 603: SurfaceFlinger determines whether each Buffer corresponding to the APP contains at least two target video frames. If so, proceed to step 604; otherwise, proceed to step 606.

[0122] Step 604: SurfaceFlinger calls the IFR Plugin to send frame interpolation instructions to the image enhancement chip.

[0123] Step 605: The image enhancement chip inputs multiple target video frames from each buffer into the frame prediction model to obtain the rendered intermediate frames.

[0124] Furthermore, the rendered intermediate frames are stored in the corresponding buffer of the APP.

[0125] Step 606: SurfaceFlinger performs graphic composition on the rendered video frames.

[0126] SurfaceFlinger retrieves rendered video frames from the app's corresponding buffer for compositing. The video frames in the buffer can be intermediate frames or frames rendered by the app.

[0127] Step 607: HWComposer performs layer compositing on the video frames after graphic composition.

[0128] Step 608: The monitor displays the composited video frames.

[0129] The following is combined with Figure 7 The timing of frame interpolation is explained. (See reference...) Figure 7 The image shown is a timing diagram for a type of frame interpolation. Figure 7 In this setup, the app is used for rendering, SurfaceFlinger is used for graphics compositing, the image enhancement chip is used to generate intermediate frames, and the panel is used for display. Vsync i represents the i-th cycle, and frame j represents the j-th video frame.

[0130] Since the rendering of frame 3 takes up two cycles, if the rendering of frame 3 times out, SurfaceFlinger will detect the frame drop event in Vsync4, and then call the image enhancement chip to generate frame 1.5 (i.e., the intermediate frame) based on frame 1 and frame 2 using the frame prediction model. Frame 1, frame 1.5 and frame 2 can be displayed on the screen in sequence.

[0131] It should be noted that, due to the addition of the image enhancement chip, one frame of display latency will be sacrificed, i.e., compared to... Figure 1 In comparison, we can see that the overall display delay is one frame. For example, frame 1 is actually displayed on the monitor / screen only at Vsync4.

[0132] The following is combined with Figure 8 The timing of continuous frame interpolation is explained. (See reference...) Figure 8 The diagram shown is a timing diagram for a type of continuous frame interpolation. Figure 8 In the middle, since the rendering of frame 3 takes up three consecutive cycles, the rendering of frame 3 times out again. SurfaceFlinger will detect the frame drop event again in Vsync 5, call the image enhancement chip, use the frame prediction model, generate frame 1.75 (i.e., the new intermediate frame) based on frame 1.5 and frame 2, and display frame 1.5, frame 1.75 and frame 2 in sequence.

[0133] Thus, the embodiments of this application can perform frame replacement when a single frame is lost, and also use the same principle to perform continuous frame replacement based on the already generated intermediate frames when frames are lost continuously.

[0134] In traditional technologies, electronic devices typically lack image enhancement chips. When a frame drop event is detected during rendering, the previous video frame is usually simply repeated, resulting in poor video smoothness. However, in this embodiment, an image enhancement chip is added at the hardware level. Upon detecting a frame drop event, the image enhancement chip can be invoked, employing a frame prediction model to obtain an intermediate frame between the first and second video frames based on multiple rendered but not yet output target video frames. This intermediate frame is then used to complete the frame interpolation. This enables image / animation completion, reducing the stuttering experienced by the user due to rendering frame drops to a certain extent, allowing normal display tasks to be completed. Under the same CPU rendering resource conditions, rendering efficiency is improved, and the smoothness of video playback and the user's viewing experience are enhanced. Furthermore, because the frame prediction model has a simple structure, it can be integrated into the image enhancement chip, resulting in lower power consumption during operation.

[0135] Based on the same inventive concept, this disclosure also provides an image processing apparatus. Since the principle of the above-described apparatus and device in solving the problem is similar to that of an image processing method, the implementation of the above-described apparatus can refer to the implementation of the method, and repeated details will not be elaborated further. This apparatus can be applied to electronic devices. This disclosure does not limit the type of electronic device; it can be any suitable type of device, such as terminal devices and servers, etc., which will not be elaborated further in this disclosure. The apparatus embodiment can be implemented by software, or by hardware, or a combination of software and hardware. Taking software implementation as an example, as a logically defined apparatus, it is formed by the processor of the electronic device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0136] See Figure 9 The diagram shown is a structural block diagram of an image processing apparatus according to an embodiment of this disclosure. In some embodiments, the image processing apparatus of this disclosure includes:

[0137] The detection unit 901 is used to, during the rendering process of the video to be processed, if a frame dropping event is detected, acquire the first and second video frames that have been rendered and are to be displayed; the first and second video frames are two adjacent video frames of the video to be processed; the frame dropping event indicates a rendering timeout.

[0138] The prediction unit 902 is used to generate an intermediate frame between the first video frame and the second video frame based on the motion information of the pixels between the first video frame and the second video frame.

[0139] Display unit 903 is used to display the first video frame, intermediate frame and second video frame.

[0140] In one embodiment, the prediction unit 902 is used to:

[0141] Multiple cascaded encoders are used to extract intermediate features between the first and second video frames;

[0142] Multiple decoders are cascaded to obtain intermediate optical flow estimates for the first and second video frames based on intermediate features, and intermediate frames are generated based on the intermediate optical flow estimates.

[0143] In this model, intermediate optical flow estimation represents the motion information of pixels; multiple cascaded encoders and multiple cascaded decoders constitute the frame prediction model.

[0144] In one embodiment, the detection unit 901 is used for:

[0145] According to the set duration, the rendering status of the currently rendered third video frame is detected periodically.

[0146] If, based on the detection results, it is determined that the third video frame failed to render successfully within the set time period, then a frame drop event is detected.

[0147] In one embodiment, the display unit 903 is used for:

[0148] If the third video frame is successfully rendered within the set time after the frame drop event, the first video frame, the intermediate frame, the second video frame, and the third video frame will be displayed in sequence.

[0149] The first video frame is rendered earlier than the second video frame.

[0150] In one embodiment, the display unit 903 is used for:

[0151] If consecutive frame dropping events are detected, a new intermediate frame is obtained based on the intermediate frame and the second video frame; consecutive frame dropping events indicate that the third video frame fails to render again within the next set time period.

[0152] The first video frame, the intermediate frame, the new intermediate frame, and the second video frame are displayed sequentially.

[0153] The first video frame is rendered earlier than the second video frame.

[0154] In one aspect, this disclosure provides an electronic device, including:

[0155] Processor; and

[0156] The memory stores computer instructions that cause the processor to perform the steps of the methods provided in the various alternative implementations of any of the image processing described above.

[0157] In one aspect, this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to perform the steps of the methods provided in various alternative implementations of any of the above-described image processing methods.

[0158] On one hand, this disclosure provides a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the steps of the method provided in various alternative implementations of any of the above-described image processing methods.

[0159] The image processing method in this embodiment includes, during the rendering of a video to be processed, if a frame drop event is detected, acquiring a first video frame and a second video frame that have been rendered and are to be displayed; the first video frame and the second video frame are two adjacent video frames of the video to be processed; the frame drop event indicates a rendering timeout; generating an intermediate frame between the first video frame and the second video frame based on the motion information of the pixels between the first video frame and the second video frame; and displaying the first video frame, the intermediate frame, and the second video frame. In this way, when frame drop occurs, an intermediate frame between the first video frame and the second video frame can be generated in a timely manner based on the motion information of the pixels between the first video frame and the second video frame for frame interpolation, thereby improving the smoothness of the video.

[0160] In this embodiment of the disclosure, an electronic device is also provided, including:

[0161] Processor; and

[0162] The memory stores computer instructions that cause the processor to execute the methods of any of the above-described embodiments.

[0163] In this embodiment of the disclosure, a computer-readable storage medium is provided, storing computer instructions for causing a computer to perform the methods of any of the above embodiments.

[0164] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the method described in any of the above embodiments.

[0165] Figure 10 A schematic diagram of the structure of an electronic device 1000 is shown. (See attached diagram.) Figure 10 As shown, the electronic device 1000 includes a processor 1010 and a memory 1020, and optionally may also include a power supply 1030, a display unit 1040, and an input unit 1050.

[0166] The processor 1010 is the control center of the electronic device 1000. It connects various components through various interfaces and lines, and performs various functions of the electronic device 1000 by running or executing software programs and / or data stored in the memory 1020, thereby performing overall monitoring of the electronic device 1000.

[0167] In this embodiment of the present disclosure, the processor 1010 executes the steps in the above embodiments when it calls the computer program stored in the memory 1020.

[0168] Optionally, the processor 1010 may include one or more processing units; preferably, the processor 1010 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1010. In some embodiments, the processor and memory may be implemented on a single chip; in some embodiments, they may also be implemented separately on independent chips.

[0169] The memory 1020 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, various applications, etc.; the data storage area may store data created based on the use of the electronic device 1000, etc. In addition, the memory 1020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0170] The electronic device 1000 also includes a power supply 1030 (such as a battery) that supplies power to various components. The power supply can be logically connected to the processor 1010 through a power management system, thereby enabling the management of charging, discharging, and power consumption.

[0171] The display unit 1040 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device 1000. In this embodiment, it is mainly used to display the display interfaces of various applications in the electronic device 1000, as well as text, images, and other objects displayed on the display interfaces. The display unit 1040 may include a display panel 1041. The display panel 1041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0172] The input unit 1050 can be used to receive information such as numbers or characters input by the user. The input unit 1050 may include a touch panel 1051 and other input devices 1052. The touch panel 1051, also known as a touch screen, can collect touch operations on or near the touch panel 1051 (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 1051).

[0173] Specifically, the touch panel 1051 can detect user touch operations and the signals generated by these operations, convert them into touch point coordinates, send them to the processor 1010, and receive and execute commands from the processor 1010. Furthermore, the touch panel 1051 can be implemented using various types of sensors, including resistive, capacitive, infrared, and surface acoustic wave sensors. Other input devices 1052 can include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0174] Of course, the touch panel 1051 can cover the display panel 1041. When the touch panel 1051 detects a touch operation on or near it, it transmits the information to the processor 1010 to determine the type of touch event. Subsequently, the processor 1010 provides corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 10 In this embodiment, the touch panel 1051 and the display panel 1041 are two separate components to realize the input and output functions of the electronic device 1000. However, in some embodiments, the touch panel 1051 and the display panel 1041 can be integrated to realize the input and output functions of the electronic device 1000.

[0175] The electronic device 1000 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity sensor, etc. Of course, depending on the specific application, the electronic device 1000 may also include other components such as a camera. Since these components are not the focus of this disclosure, therefore... Figure 10 It is not shown in the text and will not be described in detail here.

[0176] Those skilled in the art will understand that Figure 10 This is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than shown, or a combination of certain components, or different components.

[0177] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this disclosure, the functions of each module (or unit) can be implemented in one or more software or hardware components.

Claims

1. An image processing method, characterized in that, The method includes: During the rendering of the video to be processed, if a frame drop event is detected, the first and second video frames that have been rendered and are ready to be displayed are obtained; the first and second video frames are two adjacent video frames of the video to be processed; the frame drop event indicates a rendering timeout. Based on the motion information of pixels between the first video frame and the second video frame, an intermediate frame between the first video frame and the second video frame is generated; Display the first video frame, the intermediate frame, and the second video frame.

2. The method according to claim 1, characterized in that, The step of generating an intermediate frame between the first video frame and the second video frame based on the motion information of pixels between the first video frame and the second video frame includes: Multiple cascaded encoders are used to extract intermediate features between the first video frame and the second video frame; Multiple decoders are cascaded to obtain intermediate optical flow estimates for the first video frame and the second video frame based on the intermediate features, and the intermediate frame is generated based on the intermediate optical flow estimates. The intermediate optical flow estimation represents the motion information of the pixel; the cascaded multiple encoders and cascaded multiple decoders constitute the frame prediction model.

3. The method according to claim 1 or 2, characterized in that, The method includes: According to the set duration, the rendering status of the currently rendered third video frame is detected periodically. If, based on the detection results, it is determined that the third video frame has not been successfully rendered within the set time period, then a frame dropping event is detected.

4. The method according to claim 3, characterized in that, The display of the first video frame, the intermediate frame, and the second video frame includes: If it is determined that the third video frame is successfully rendered within a set time after the frame drop event, then the first video frame, the intermediate frame, the second video frame, and the third video frame are displayed in sequence. The rendering time of the first video frame is earlier than the rendering time of the second video frame.

5. The method according to claim 3, characterized in that, The display of the first video frame, the intermediate frame, and the second video frame includes: If it is determined that frame dropping events are detected continuously, a new intermediate frame is obtained based on the intermediate frame and the second video frame; the continuous detection of frame dropping events indicates that the third video frame fails to render again within the next set time period. The first video frame, the intermediate frame, the new intermediate frame, and the second video frame are displayed sequentially. The rendering time of the first video frame is earlier than the rendering time of the second video frame.

6. An image processing apparatus, characterized in that, The device includes: The detection unit is used to, during the rendering process of the video to be processed, if a frame dropping event is detected, acquire the first and second video frames that have been rendered and are to be displayed; the first and second video frames are two adjacent video frames of the video to be processed; the frame dropping event indicates a rendering timeout. The prediction unit is configured to generate an intermediate frame between the first video frame and the second video frame based on the motion information of the pixels between the first video frame and the second video frame. The display unit is used to display the first video frame, the intermediate frame, and the second video frame.

7. The apparatus according to claim 6, characterized in that, The prediction unit is used for: Multiple cascaded encoders are used to extract intermediate features between the first video frame and the second video frame; Multiple decoders are cascaded to obtain intermediate optical flow estimates for the first video frame and the second video frame based on the intermediate features, and the intermediate frame is generated based on the intermediate optical flow estimates. The intermediate optical flow estimation represents the motion information of the pixel; the cascaded multiple encoders and cascaded multiple decoders constitute the frame prediction model.

8. An electronic device, characterized in that, include: processor; as well as A memory storing computer instructions for causing the processor to perform the method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer contains computer instructions for causing the computer to perform the method according to any one of claims 1 to 5.

10. A computer program product, characterized in that, Includes computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is executed in a processor of an electronic device, the processor in the electronic device is the method according to any one of claims 1 to 5.

Images