Picture frame processing method and apparatus

Abstract

The application provides a picture frame processing method and device. The method comprises the following steps: capturing a picture frame, and putting the picture frame into a first queue in a display memory; extracting the picture frame from the first queue, calculating an optical flow between two adjacent picture frames in front and behind through a shader on a GPU, interpolating the two adjacent picture frames in front and behind by using the optical flow to obtain a new picture frame, and recording the new picture frame as an intermediate frame; putting the intermediate frame and a picture frame behind the two adjacent picture frames in front and behind into a second queue in the display memory; extracting the picture frame from the second queue, encoding the picture frame, and sending the picture frame to a terminal, or displaying the picture frame in a covering window according to a time corresponding to the picture frame from early to late, so that the fluency of an application / video picture can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer, in particular to a picture frame processing method and device. BACKGROUND

[0002] When a user uses an application / video player to play a video, the smoothness of the picture presented by the application / video player is sometimes unsatisfactory due to the hardware processing or the application production level, the era, and the like.

[0003] Therefore, how to improve the smoothness of the application / video picture becomes a technical problem to be solved. SUMMARY

[0004] In view of the technical problems in the prior art, the embodiments of the present application provide a picture frame processing method and device.

[0005] In a first aspect, the embodiments of the present application provide a picture frame processing method, comprising:

[0006] capturing a picture frame and putting the picture frame into a first queue in a video memory;

[0007] extracting the picture frame from the first queue, calculating the optical flow between two adjacent picture frames before and after by a shader on a GPU, interpolating the two adjacent picture frames before and after by the optical flow to obtain a new picture frame, denoted as an intermediate frame, and putting the intermediate frame and the picture frame after the two adjacent picture frames before and after into a second queue in the video memory;

[0008] extracting the picture frame from the second queue, encoding the picture frame and sending the picture frame to a terminal, or displaying the picture frame in a covering window in a time sequence from early to late, wherein the covering window is a topmost window and covers a display window, and the captured picture frame is from the display window.

[0009] In a second aspect, the embodiments of the present application further provide a picture frame processing device, comprising:

[0010] a capturing unit configured to capture a picture frame and put the picture frame into a first queue in a video memory;

[0011] a first processing unit configured to extract the picture frame from the first queue, calculate the optical flow between two adjacent picture frames before and after by a shader on a GPU, interpolate the two adjacent picture frames before and after by the optical flow to obtain a new picture frame, denoted as an intermediate frame, and put the intermediate frame and the picture frame after the two adjacent picture frames before and after into a second queue in the video memory;

[0012] The second processing unit is configured to extract the picture frame from the second queue, encode the picture frame, and send the picture frame to a terminal, or display the picture frame in a covering window in a time sequence from early to late according to the corresponding time, wherein the covering window is a topmost window and covers a display window, and the captured picture frame is from the display window.

[0013] The picture frame processing method and device provided by the embodiments of the present application, in the first aspect, the interpolation is performed by the shader on the GPU, and the source data used for the interpolation and the result data generated by the interpolation are stored in the video memory, so that the interpolation process can be operated in the GPU all the time, the powerful computing capability of the GPU can be fully utilized, the interpolation process can be asynchronously and in parallel processed by using the queue, and finally the present solution can quickly generate the interpolated picture frame; in the second aspect, the interpolated picture frame is displayed in the covering window covering the display window, so that the user can see the picture frame with higher frame rate without awareness, and the intermediate frame generated by the interpolation can be prevented from also entering the interpolation processing logic to make the solution enter an infinite loop, the whole solution can be applied to cloud applications, local applications and local video players, and by the first aspect and the second aspect, the frame rate of the application / video can be doubled without awareness of the user, so that the fluency of the application / video picture can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 A flowchart of an embodiment of a picture frame processing method provided by the embodiments of the present application;

[0015] Figure 2 A structural diagram of an embodiment of a picture frame processing device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0016] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of description and illustration, and do not serve to limit the protection scope of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportions. The flowcharts show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or one or more operations can be removed from the flowcharts under the guidance of the content of the present application.

[0017] In addition, the described embodiments are only some embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0018] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0019] Referring to Figure 1 As shown in the flowchart of a picture frame processing method provided by the embodiments of the present application, the method comprises:

[0020] S10, capturing picture frames and putting the picture frames into a first queue in the video memory;

[0021] In the embodiments, it should be noted that the whole scheme can be applied to cloud applications (such as cloud games), local applications (such as local games) and local video players: for cloud applications, steps S10, S11 and S12 are executed on a cloud server, and the capturing of picture frames in step S10 refers to the capturing of cloud application pictures rendered by cloud applications; for local applications / local video players, steps S10, S11 and S12 are executed on a local terminal, and the capturing of picture frames in step S10 refers to the capturing of local application pictures / video pictures displayed by local application windows / video player windows. After capturing the picture frames, the picture frames are sequentially put into the first queue in the video memory according to the capturing order. In order to reduce the amount of calculation, when the picture frames are put into the first queue, the picture frames can be subjected to grayscale processing to obtain grayscale images, and the grayscale images are put into the first queue.

[0022] S11, extracting picture frames from the first queue, calculating the optical flow between two adjacent picture frames before and after through a shader on the GPU, and using the optical flow to interpolate the two adjacent picture frames before and after to obtain a new picture frame, denoted as an intermediate frame, and putting the intermediate frame and the latter one of the two adjacent picture frames before and after into a second queue in the video memory;

[0023] It should be noted that in the embodiment, the shader is pre-compiled and runs on the GPU, the source data of the entire process of generating the intermediate frame comes from the GPU, the calculation process is executed on the GPU, and the processing result is also stored in the GPU, that is, the entire processing process does not involve other hardware / programs except the execution subject and the GPU, and therefore the powerful calculation capability of the GPU can be fully utilized to quickly generate the intermediate frame. In addition, the source data used in the interpolation process and the intermediate frame generated by the interpolation process exist in the second queue, that is, if the first picture frame of the two adjacent picture frames does not exist in the second queue, the intermediate frame and the second picture frame of the two adjacent picture frames are not only put into the second queue, but also the first picture frame of the two adjacent picture frames is put into the second queue.

[0024] S12, extracting the picture frame from the second queue, encoding the picture frame and sending it to the terminal, or displaying the picture frame in the overlay window in the order from the earliest to the latest according to the corresponding time, wherein the overlay window is the topmost window and covers the display window, and the captured picture frame comes from the display window.

[0025] It should be noted that in the embodiment, for the cloud application, the process of extracting the picture frame from the second queue and encoding the picture frame and sending it to the terminal in step S12 can include: if the encoder cannot perform the format conversion process internally, the picture frame can be converted into an image format that can be processed by the encoder by using a pre-compiled and running shader on the GPU (the shader can be the same as or different from the shader in step S11), and the converted image is put into a third queue in the video memory, and then the image in the third queue is encoded by using the encoder and sent to the terminal; or if the encoder can perform the format conversion process internally, the picture frame in the second queue can be directly converted and encoded by using the encoder, and then the encoding result is sent to the terminal. After receiving the picture frame sent by the cloud server, the terminal needs to decode and display the picture frame. Since the terminal receives the interpolated picture frame, the frame rate of the cloud application picture displayed by the terminal can be greatly improved.

[0026] For the local application / local video player, the process of displaying the picture frames in the order of time from early to late in the overlay window is performed in step S12. The size of the overlay window needs to ensure that it completely covers the local application window / local video player window, so that the user sees the picture frames after the interpolation in the overlay window, thereby greatly improving the frame rate of the application / video picture displayed in the local application window / local video player window. In addition, in order to avoid the picture frames after the interpolation being captured by the execution subject of step S10 and causing the interpolation process to fall into an infinite loop, the type of the overlay window (such as registering the overlay window as an Overlay window) and the hook system of the video card capture callback can be set to make the overlay window not participate in the system video card capture callback. Thus, the execution subject of step S10 can only capture the application / video picture in the local application window / local video player window, and cannot capture the picture in the overlay window, so that the entire interpolation process does not fall into an infinite loop.

[0027] The picture frame processing method provided by the embodiments of the present application has the following advantages. First, the interpolation is performed by the shader on the GPU, and the source data used for the interpolation and the result data generated by the interpolation are stored in the video memory, so that the interpolation process can be operated in the GPU all the time, the powerful computing capability of the GPU can be fully utilized, and the use of the queue enables the interpolation process to be asynchronously and parallelly processed, so that the scheme can quickly generate the picture frames after the interpolation. Second, the picture frames after the interpolation are displayed in the overlay window overlaid on the display window, so that the user can see the picture frames with higher frame rate without any awareness, and the intermediate frames generated by the interpolation can be prevented from entering the interpolation processing logic to make the scheme fall into an infinite loop. The entire scheme can be applied to the cloud application, the local application, and the local video player, and the frame rate of the application / video can be doubled without any awareness of the user by the first aspect and the second aspect, so that the fluency of the application / video picture can be improved.

[0028] On the basis of the foregoing method embodiments, the calculation of the optical flow between the two adjacent picture frames before and after by the shader on the GPU, and the interpolation of the two adjacent picture frames before and after by the optical flow to obtain a new picture frame can include the following steps.

[0029] The first shader on the GPU is used to calculate the forward optical flow and the backward optical flow between the two adjacent picture frames before and after, the forward optical flow and the backward optical flow are bidirectionally and consistently checked and corrected, and the motion amplitude between the two adjacent picture frames before and after is calculated according to the corrected forward optical flow.

[0030] The second shader on the GPU is used to interpolate the two adjacent picture frames before and after according to the corrected forward optical flow, the corrected backward optical flow, and the motion amplitude to obtain a new picture frame.

[0031] It should be noted that the forward optical flow between the picture frame A and the picture frame B includes the displacement corresponding to each pixel position in the picture frame A, and each displacement represents the displacement from the corresponding pixel position in the picture frame A to the corresponding position in the picture frame B. The backward optical flow between the picture frame A and the picture frame B includes the displacement corresponding to each pixel position in the picture frame B, and each displacement represents the displacement from the corresponding pixel position in the picture frame B to the corresponding position in the picture frame A. The pyramid Lucas-Kanade optical flow method can be used to calculate the forward optical flow and the backward optical flow between the two adjacent picture frames, and the pyramid level in the method can be set to 5, and the iteration number on each level can be set to 8.

[0032] The bidirectional consistency check and correction includes the forward consistency check, the backward consistency check and the correction for inconsistency. The forward consistency check includes: for each displacement forward in the forward optical flow, calculating the displacement reverse_backward of the corresponding position of the displacement forward in the backward optical flow, and evaluating the reliability of the displacement forward according to the sum of the displacement forward and the displacement reverse_backward (ideally, the sum of the displacement forward and the displacement reverse_backward is 0, so the length of the sum of the displacement forward and the displacement reverse_backward can be used to evaluate the reliability of the displacement forward, and the greater the length of the sum of the displacement forward and the displacement reverse_backward, the lower the reliability of the displacement forward). The backward consistency check is consistent with the process of the forward consistency check, which will not be described here. Then for each position, the reliability result (such as the length of the sum of the displacement forward and the displacement reverse_backward) of the forward consistency check and the reliability result of the backward consistency check can be obtained, and the maximum of the two reliability results is taken as the reliability consistency of the optical flow estimation of the position. It is judged whether the consistency is greater than the adaptive threshold adaptiveThreshold. If the consistency is greater than the adaptive threshold adaptiveThreshold, the forward optical flow and the backward optical flow of the pixel point of the position are corrected, and the correction includes multiplying the current optical flow by a decay factor decay as a new optical flow.

[0033] adaptiveThreshold = consistencyThreshold × (1 + flowMag × 0.5),

[0034] consistencyThreshold is a basic threshold parameter, which can be understood as the maximum error that can be tolerated when there is "almost no motion" or "very small motion", and flowMag is the average optical flow magnitude, which represents the intensity of the motion of the pixel points at this position, and can be the average of the forward optical flow length and the backward optical flow length of the pixel points at this position.

[0035] decay = saturate(1 - (consistency - adaptiveThreshold) * 5),

[0036] where saturate(X) represents limiting X to the range [0, 1].

[0037] The motion amplitude between the two adjacent frames includes the lengths of the displacements in the corrected forward optical flow. The interpolation process can use existing algorithms, such as the bicubic interpolation algorithm.

[0038] On the basis of the foregoing method embodiment, the method can further include:

[0039] obtaining a control instruction generated by the user manipulating the overlay window;

[0040] converting the control instruction into a control instruction for the display window, and sending the control instruction for the display window to the system.

[0041] In this embodiment, it should be noted that for the local application / local video player, in order to make the user have no abnormal feeling when manipulating, the user's manipulation of the overlay window needs to be transparently transmitted to the local application / local video player, and therefore the control instruction generated by the user manipulating the overlay window needs to be converted into a control instruction for the display window, and the control instruction for the display window needs to be sent to the system, that is, the display window still responds to the user's manipulation, and the overlay window is only used to display the interpolated picture frame.

[0042] Referring to FIG. 1, Figure 2 FIG. 1 is a structural schematic diagram of a picture frame processing device provided by an embodiment of the present application, which includes:

[0043] a capturing unit 20 configured to capture picture frames and put the picture frames into a first queue in a display memory;

[0044] a first processing unit 21 configured to extract the picture frames from the first queue, calculate the optical flow between two adjacent picture frames by using a shader on a GPU, interpolate the two adjacent picture frames by using the optical flow to obtain a new picture frame, denoted as an intermediate frame, and put the intermediate frame and a latter one of the two adjacent picture frames into a second queue in the display memory;

[0045] The second processing unit 22 is configured to extract the picture frame from the second queue, encode the picture frame, and send the picture frame to a terminal, or display the picture frame in a covering window in a time sequence from early to late according to the corresponding time, wherein the covering window is a topmost window and covers a display window, and the captured picture frame is from the display window.

[0046] The picture frame processing device provided in the embodiments of the present application has the following advantages. In the first aspect, the interpolation is performed by the shader on the GPU, and the source data used for the interpolation and the result data generated by the interpolation are stored in the video memory, so that the interpolation process can be operated in the GPU all the time, the powerful computing capability of the GPU can be fully utilized, the interpolation process can be asynchronously and in parallel processed by using the queue, and finally the scheme can quickly generate the interpolated picture frame. In the second aspect, the interpolated picture frame is displayed in the covering window covering the display window, so that the user can see the picture frame with higher frame rate without awareness, and the intermediate frame generated by the interpolation can be prevented from also entering the interpolation processing logic to make the scheme enter an infinite loop. The whole scheme can be applied to cloud applications, local applications and local video players, and the frame rate of the application / video can be doubled without awareness of the user by using the first aspect and the second aspect, so that the fluency of the application / video picture can be improved.

[0047] On the basis of the foregoing device embodiments, the first processing unit can be configured to:

[0048] The first shader on the GPU is configured to calculate the forward optical flow and the backward optical flow between the two adjacent picture frames, perform bidirectional consistency check and correction on the forward optical flow and the backward optical flow, and calculate the motion amplitude between the two adjacent picture frames according to the corrected forward optical flow.

[0049] The second shader on the GPU is configured to perform interpolation on the two adjacent picture frames according to the corrected forward optical flow, the corrected backward optical flow and the motion amplitude to obtain a new picture frame.

[0050] On the basis of the foregoing device embodiments, the device can further include:

[0051] The third processing unit is configured to acquire a control instruction generated by the user operating the covering window, convert the control instruction into a control instruction for the display window, and send the control instruction for the display window to the system.

[0052] The picture frame processing device provided in the embodiments of the present application has the following advantages. In the first aspect, the interpolation is performed by the shader on the GPU, and the source data used for the interpolation and the result data generated by the interpolation are stored in the video memory, so that the interpolation process can be operated in the GPU all the time, the powerful computing capability of the GPU can be fully utilized, the interpolation process can be asynchronously and in parallel processed by using the queue, and finally the scheme can quickly generate the interpolated picture frame. In the second aspect, the interpolated picture frame is displayed in the covering window covering the display window, so that the user can see the picture frame with higher frame rate without awareness, and the intermediate frame generated by the interpolation can be prevented from also entering the interpolation processing logic to make the scheme enter an infinite loop. The whole scheme can be applied to cloud applications, local applications and local video players, and the frame rate of the application / video can be doubled without awareness of the user by using the first aspect and the second aspect, so that the fluency of the application / video picture can be improved.

[0053] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A picture frame processing method, characterized by, The method comprises the following steps: capturing picture frames and putting the picture frames into a first queue in the video memory; extracting the picture frames from the first queue, calculating the optical flow between two adjacent picture frames by using a shader on the GPU, interpolating the two adjacent picture frames by using the optical flow to obtain a new picture frame, denoted as an intermediate frame, and putting the intermediate frame and the latter one of the two adjacent picture frames into a second queue in the video memory; extracting the picture frames from the second queue and displaying the picture frames in the order of time from early to late in a cover window, wherein the cover window is the topmost window and covers a display window, and the captured picture frames come from the display window.

2. The method of claim 1, wherein, The method of calculating the optical flow between two adjacent picture frames by using the shader on the GPU, interpolating the two adjacent picture frames by using the optical flow to obtain a new picture frame comprises the following steps: calculating the forward optical flow and the backward optical flow between the two adjacent picture frames by using a first shader on the GPU, performing bidirectional consistency check and correction on the forward optical flow and the backward optical flow, and calculating the motion amplitude between the two adjacent picture frames according to the corrected forward optical flow; interpolating the two adjacent picture frames according to the corrected forward optical flow, the corrected backward optical flow and the motion amplitude by using a second shader on the GPU to obtain the new picture frame.

3. The method of claim 1 or 2, wherein, The method further comprises the following steps: obtaining a control instruction generated by a user manipulating the cover window; converting the control instruction into a control instruction for the display window and sending the control instruction for the display window to the system.

4. A picture frame processing apparatus, characterized by comprising: The method comprises the following steps: a capturing unit is configured to capture picture frames and put the picture frames into a first queue in the video memory; a first processing unit is configured to extract the picture frames from the first queue, calculate the optical flow between two adjacent picture frames by using a shader on the GPU, interpolate the two adjacent picture frames by using the optical flow to obtain a new picture frame, denoted as an intermediate frame, and put the intermediate frame and the latter one of the two adjacent picture frames into a second queue in the video memory; a second processing unit is configured to extract the picture frames from the second queue and display the picture frames in the order of time from early to late in a cover window, wherein the cover window is the topmost window and covers a display window, and the captured picture frames come from the display window.

5. The apparatus of claim 4, wherein, The first processing unit is configured to: calculate the forward optical flow and the backward optical flow between the two adjacent picture frames by using a first shader on the GPU, perform bidirectional consistency check and correction on the forward optical flow and the backward optical flow, and calculate the motion amplitude between the two adjacent picture frames according to the corrected forward optical flow; interpolate the two adjacent picture frames according to the corrected forward optical flow, the corrected backward optical flow and the motion amplitude by using a second shader on the GPU to obtain the new picture frame.

6. The apparatus of claim 4 or 5, wherein, The method further comprises the following steps: a third processing unit is configured to obtain a control instruction generated by a user manipulating the cover window, convert the control instruction into a control instruction for the display window, and send the control instruction for the display window to the system.

Images