Picture frame processing method and device

Abstract

The invention provides a picture frame processing method and device, and the method comprises the steps: capturing a picture frame, and putting the picture frame into a first queue in a video memory; the second queue is used for extracting picture frames from the first queue, calculating an optical flow between two front and back adjacent picture frames through a shader on the GPU, carrying out interpolation on the two front and back adjacent picture frames by utilizing the optical flow to obtain a new picture frame, recording the new picture frame as an intermediate frame, and putting the intermediate frame and the next picture frame in the two front and back adjacent picture frames into the video memory; and extracting the picture frames from the second queue, coding the picture frames and then sending the coded picture frames to the terminal, or displaying the picture frames in the overlay window according to the corresponding time sequence, so that the fluency of the application / video picture can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer, in particular to a picture frame processing method and device. BACKGROUND

[0002] When a user uses an application / video player to play a video, the smoothness of the picture presented by the application / video player is sometimes unsatisfactory due to reasons such as hardware processing or application production level, era, etc.

[0003] Therefore, how to improve the smoothness of the application / video picture becomes a technical problem to be solved. SUMMARY

[0004] In view of the technical problems existing in the prior art, the embodiments of the present application provide a picture frame processing method and device.

[0005] In a first aspect, the embodiments of the present application provide a picture frame processing method, comprising: capturing a picture frame and putting the picture frame into a first queue in a video memory; extracting the picture frame from the first queue, calculating an optical flow between two adjacent picture frames before and after through a shader on a GPU, interpolating the two adjacent picture frames before and after by using the optical flow to obtain a new picture frame, denoted as an intermediate frame, and putting the intermediate frame and a picture frame after the two adjacent picture frames before and after into a second queue in the video memory; extracting the picture frame from the second queue, encoding the picture frame and sending the picture frame to a terminal, or displaying the picture frame in a covering window in a sequence from early to late according to a corresponding time, wherein the covering window is a topmost window and covers a display window, and the captured picture frame is from the display window.

[0006] In a second aspect, the embodiments of the present application further provide a picture frame processing device, comprising: a capturing unit configured to capture a picture frame and put the picture frame into a first queue in a video memory; a first processing unit configured to extract the picture frame from the first queue, calculate an optical flow between two adjacent picture frames before and after through a shader on a GPU, interpolate the two adjacent picture frames before and after by using the optical flow to obtain a new picture frame, denoted as an intermediate frame, and put the intermediate frame and a picture frame after the two adjacent picture frames before and after into a second queue in the video memory; a second processing unit configured to extract the picture frame from the second queue, encode the picture frame and send the picture frame to a terminal, or display the picture frame in a covering window in a sequence from early to late according to a corresponding time, wherein the covering window is a topmost window and covers a display window, and the captured picture frame is from the display window.

[0007] The picture frame processing method and device provided by the embodiment of the present application, in the first aspect, the interpolation is performed by the shader on the GPU, and the source data used for interpolation and the result data generated by interpolation are stored in the video memory, so that the interpolation process can be operated in the GPU all the time, the powerful computing capability of the GPU can be fully utilized, and the use of the queue enables the interpolation process to be asynchronously and parallelly processed, so that the present solution can quickly generate the picture frame after interpolation; in the second aspect, the picture frame after interpolation is displayed by using the overlay window overlaid on the display window, so that the user can see the picture frame with higher frame rate without awareness, and the intermediate frame generated by interpolation can be prevented from also entering the interpolation processing logic to make the solution enter into an infinite loop, the whole solution can be applied to cloud application, local application and local video player, and by the first aspect and the second aspect, the frame rate of the application / video can be doubled without awareness of the user, so that the fluency of the application / video picture can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 A flowchart of an embodiment of a picture frame processing method provided by the embodiment of the present application; Figure 2 A structure diagram of an embodiment of a picture frame processing device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0009] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme of the embodiment of the present application will be described clearly and completely below in combination with the drawings in the embodiment of the present application. It should be understood that the drawings in the present application only serve the purpose of description and illustration, and do not limit the protection scope of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportion. The flowchart shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowchart or removed from the flowchart under the guidance of the content of the present application by those skilled in the art.

[0010] In addition, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0011] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0012] Reference Figure 1 FIG. 1 is a flow chart of a method for processing a picture frame according to an embodiment of the present application, the method comprising: S10, capturing the picture frame and placing the picture frame into the first queue in the video memory; In this embodiment, it should be noted that the entire solution can be applied to cloud applications (such as cloud games), local applications (such as local games) and local video players: for cloud applications, steps S10, S11 and S12 are executed on the cloud server, and capturing the screen frame in step S10 refers to capturing the cloud application screen rendered by the cloud application; for local applications / local video players, steps S10, S11 and S12 are executed on the local terminal, and capturing the screen frame in step S10 refers to capturing the local application screen / video screen displayed in the local application window / video player window. After capturing the screen frame, the screen frame needs to be placed in the first queue in the video memory in the order of capture. In order to reduce the amount of calculation, when placing the screen frame in the first queue, the screen frame can be grayscale processed to obtain a grayscale image, and the grayscale image can be placed in the first queue.

[0013] S11, extracting a picture frame from the first queue, calculating the optical flow between two adjacent pictures using a shader on the GPU, interpolating the two adjacent pictures using the optical flow to obtain a new picture frame, which is recorded as an intermediate frame, and placing the intermediate frame and the latter of the two adjacent pictures into a second queue in the video memory; In this embodiment, it should be noted that the shader is pre-compiled and runs on the GPU. The source data for the entire process of generating intermediate frames comes from the GPU, the computation is performed on the GPU, and the results are also stored on the GPU. In other words, the entire processing process involves no other hardware or programs besides the execution entity and the GPU, thus fully utilizing the powerful computing power of the GPU to quickly generate intermediate frames. In addition, the second queue contains the source data used for the interpolation process and the intermediate frames generated by the interpolation process. In other words, if the first frame of the two adjacent frames does not exist in the second queue, not only the intermediate frame and the second frame of the two adjacent frames need to be placed in the second queue, but also the first frame of the two adjacent frames needs to be placed in the second queue.

[0014] S12. Extract the picture frames from the second queue, encode the picture frames and send them to the terminal, or display the picture frames in an overlay window in order of corresponding time from first to last, wherein the overlay window is the topmost window and covers the display window, and the captured picture frames come from the display window.

[0015] It should be noted that, for the cloud application, the process of extracting the picture frame from the second queue, encoding the picture frame and sending the encoded picture frame to the terminal in step S12 can include: if the format conversion process cannot be performed inside the encoder, the picture frame can be converted into an image format that can be processed by the encoder by using a shader (which can be the same shader as in step S11 or a different shader) pre-compiled and running on the GPU, and the converted image can be put into a third queue in the video memory, and then the image in the third queue can be encoded by using the encoder and sent to the terminal; or if the format conversion process can be performed inside the encoder, the picture frame in the second queue can be directly converted and encoded by using the encoder, and then the encoding result can be sent to the terminal. After receiving the picture frame sent by the cloud server, the terminal needs to decode and display the picture frame. Since the terminal receives the picture frame after the frame insertion, the frame rate of the cloud application picture displayed by the terminal can be greatly improved.

[0016] For the local application / local video player, the process of displaying the picture frame in the order of time from early to late in the overlay window in step S12 is performed. The size of the overlay window needs to be ensured to completely cover the local application window / local video player window, so that the user sees the picture frame after the frame insertion in the overlay window, thereby greatly improving the frame rate of the application / video picture displayed by the local application window / local video player window. In addition, in order to avoid the picture frame after the frame insertion being captured by the execution subject of step S10 and causing the frame insertion process to fall into an infinite loop, the type of the overlay window (such as registering the overlay window as an Overlay window) / hook system video card capture callback can be set to make the overlay window not participate in the system video card capture callback, so that the execution subject of step S10 can only capture the application / video picture in the local application window / local video player window, and cannot capture the picture in the overlay window, thereby preventing the entire frame insertion process from falling into an infinite loop.

[0017] The frame processing method provided in the embodiment of the present application, firstly, performs interpolation through the shader on the GPU, and the source data used for interpolation and the result data generated by the interpolation are stored in the video memory, so that the interpolation process can be operated entirely in the GPU, which can fully utilize the powerful computing power of the GPU. At the same time, the use of queues enables the interpolation process to be processed asynchronously and in parallel, ultimately enabling the present solution to quickly generate interpolated frame images; secondly, an overlay window covering the display window is used to display the interpolated frame images, so that the user can see the frame images with a higher frame rate without noticing it, and the intermediate frames generated by the interpolation can be prevented from entering the interpolation processing logic and causing the solution to enter an infinite loop. The entire solution can be applied to cloud applications, local applications and local video players. Through the first and second aspects, the frame rate of the application / video can be doubled without the user noticing, thereby improving the smoothness of the application / video screen.

[0018] Based on the aforementioned method embodiment, calculating the optical flow between two adjacent frames by a shader on a GPU and interpolating the two adjacent frames using the optical flow to obtain a new frame may include: The forward optical flow and the backward optical flow between two adjacent frames are calculated by the first shader on the GPU, the forward optical flow and the backward optical flow are checked and corrected in both directions, and the motion amplitude between the two adjacent frames is calculated based on the corrected forward optical flow. The second shader on the GPU interpolates the two adjacent frames according to the corrected forward optical flow, the corrected backward optical flow and the motion amplitude to obtain a new frame.

[0019] In this embodiment, it should be noted that the forward optical flow between image frame A and image frame B includes the displacement corresponding to each pixel position in image frame A, where each displacement represents the movement from the corresponding pixel position in image frame A to the corresponding position in image frame B. The backward optical flow between image frame A and image frame B includes the displacement corresponding to each pixel position in image frame B, where each displacement represents the movement from the corresponding pixel position in image frame B to the corresponding position in image frame A. The forward and backward optical flows between two adjacent image frames can be calculated using the pyramid Lucas-Kanade optical flow method. In this algorithm, the number of pyramid levels can be set to 5, and the number of iterations at each level can be set to 8.

[0020] The bidirectional consistency check and correction includes: forward consistency check, backward consistency check and correction for inconsistency, wherein the forward consistency check includes: for each displacement forward in the forward optical flow, calculating the displacement reverse_backward of the corresponding position of the forward in the backward optical flow, and evaluating the reliability of the forward according to the sum of the forward and the reverse_backward (ideally, the sum of the forward and the reverse_backward is 0, so the length of the sum of the forward and the reverse_backward can be used to evaluate the reliability of the forward, the greater the length of the sum of the forward and the reverse_backward, the lower the reliability of the forward). The backward consistency check is consistent with the forward consistency check process, which will not be described here. Then for each position, the reliability result of the forward consistency check (such as the length of the sum of the forward and the reverse_backward) and the reliability result of the backward consistency check can be obtained, and the maximum of the two reliability results is taken as the reliability consistency of the optical flow estimation of the position. It is judged whether the consistency is greater than the adaptive threshold adaptiveThreshold, if the consistency is greater than the adaptiveThreshold, the forward optical flow and the backward optical flow of the pixel point of the position are corrected, and the correction includes multiplying the current optical flow by a decay factor decay as a new optical flow.

[0021] adaptiveThreshold = consistencyThreshold x (1 + flowMag x 0.5), Wherein, the consistencyThreshold is a basic threshold parameter, which can be understood as the maximum error that can be tolerated when there is "almost no motion" or "very small motion", and the flowMag is the average optical flow magnitude, which represents the intensity of the motion of the pixel point of the position, and can be the average of the length of the forward optical flow and the length of the backward optical flow of the pixel point of the position.

[0022] decay = saturate(1 - (consistency - adaptiveThreshold) x 5), Wherein, saturate(X) means limiting X to the range of [0, 1].

[0023] The motion magnitude between the two adjacent frames of the front and back includes the length of each displacement in the corrected forward optical flow. The interpolation process can use existing algorithms, such as bicubic interpolation algorithm.

[0024] On the basis of the foregoing method embodiments, the method can further comprise: obtaining a control instruction generated by user control of the overlay window; converting the control instruction into a control instruction for the display window, and sending the control instruction for the display window to the system.

[0025] In this embodiment, it should be noted that, for the local application / local video player, in order to make the user feel no abnormality when controlling, the user control of the overlay window needs to be transparently transmitted to the local application / local video player, and thus the control instruction generated by user control of the overlay window needs to be converted into a control instruction for the display window, and the control instruction for the display window is sent to the system, that is, the display window still responds to user control, and the overlay window is only used to display the picture frame after interpolation.

[0026] Referring to Figure 2 FIG. 1 shows a structure schematic diagram of a picture frame processing device provided by an embodiment of the present application, which comprises: a capturing unit 20 configured to capture picture frames and put the picture frames into a first queue in a display memory; a first processing unit 21 configured to extract the picture frames from the first queue, calculate the optical flow between two adjacent picture frames before and after through a shader on a GPU, interpolate the two adjacent picture frames before and after to obtain a new picture frame, denoted as an intermediate frame, and put the intermediate frame and the picture frame after the two adjacent picture frames before and after into a second queue in the display memory; a second processing unit 22 configured to extract the picture frames from the second queue, encode the picture frames and send them to a terminal, or display the picture frames in the order of time from early to late in an overlay window, wherein the overlay window is a topmost window and covers a display window, and the captured picture frames come from the display window.

[0027] The picture frame processing device provided by the embodiment of the present application has the following advantages. First, interpolation is performed through a shader on a GPU, and the source data used for interpolation and the result data generated by interpolation are stored in a display memory, so that the interpolation process can be operated in the GPU all the time, the powerful computing capability of the GPU can be fully utilized, and the interpolation process can be asynchronously and parallelly processed due to the use of the queue, so that the scheme can quickly generate the picture frame after interpolation. Second, the picture frame after interpolation is displayed in an overlay window covering a display window, so that the user can see the picture frame with higher frame rate without awareness, and the intermediate frame generated by interpolation can be prevented from entering the interpolation processing logic to make the scheme enter an infinite loop, and the whole scheme can be applied to a cloud application, a local application and a local video player. Through the first aspect and the second aspect, the frame rate of the application / video can be doubled without awareness of the user, so that the fluency of the application / video picture can be improved.

[0028] On the basis of the foregoing device embodiments, the first processing unit can be configured to: calculate the forward optical flow and the backward optical flow between the two adjacent picture frames by a first shader on the GPU, perform bidirectional consistency check and correction on the forward optical flow and the backward optical flow, and calculate the motion amplitude between the two adjacent picture frames according to the corrected forward optical flow; interpolate the two adjacent picture frames according to the corrected forward optical flow, the corrected backward optical flow and the motion amplitude by a second shader on the GPU to obtain a new picture frame.

[0029] On the basis of the foregoing device embodiments, the device can further include: a third processing unit configured to acquire a control instruction generated by a user manipulating the overlay window, convert the control instruction into a control instruction for the display window, and send the control instruction for the display window to the system.

[0030] The picture frame processing device provided by the embodiments of the present application has the same implementation process and the same achievable effect as the picture frame processing method provided by the embodiments of the present application, and thus will not be described here again.

[0031] The above merely describes specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A picture frame processing method, characterized by, The method comprises the following steps: capturing picture frames and putting the picture frames into a first queue in the video memory; extracting the picture frames from the first queue, calculating the optical flow between two adjacent picture frames by using a shader on the GPU, interpolating the two adjacent picture frames by using the optical flow to obtain a new picture frame, denoted as an intermediate frame, and putting the intermediate frame and the latter one of the two adjacent picture frames into a second queue in the video memory; extracting the picture frames from the second queue, encoding the picture frames and sending them to a terminal, or displaying the picture frames in the order of time from early to late in a cover window, wherein the cover window is the topmost window and covers a display window, and the captured picture frames come from the display window.

2. The method of claim 1, wherein, The method of calculating the optical flow between two adjacent picture frames by using a shader on the GPU, interpolating the two adjacent picture frames by using the optical flow to obtain a new picture frame comprises the following steps: calculating the forward optical flow and the backward optical flow between the two adjacent picture frames by using a first shader on the GPU, performing bidirectional consistency check and correction on the forward optical flow and the backward optical flow, and calculating the motion amplitude between the two adjacent picture frames according to the corrected forward optical flow; interpolating the two adjacent picture frames according to the corrected forward optical flow, the corrected backward optical flow and the motion amplitude by using a second shader on the GPU to obtain the new picture frame.

3. The method of claim 1 or 2, wherein, The method further comprises the following steps: obtaining a control instruction generated by a user manipulating the cover window; converting the control instruction into a control instruction for the display window, and sending the control instruction for the display window to the system.

4. A picture frame processing apparatus, characterized by comprising: The method comprises the following steps: a capturing unit is configured to capture picture frames and put the picture frames into a first queue in the video memory; a first processing unit is configured to extract the picture frames from the first queue, calculate the optical flow between two adjacent picture frames by using a shader on the GPU, interpolate the two adjacent picture frames by using the optical flow to obtain a new picture frame, denoted as an intermediate frame, and put the intermediate frame and the latter one of the two adjacent picture frames into a second queue in the video memory; a second processing unit is configured to extract the picture frames from the second queue, encode the picture frames and send them to a terminal, or display the picture frames in the order of time from early to late in a cover window, wherein the cover window is the topmost window and covers a display window, and the captured picture frames come from the display window.

5. The apparatus of claim 4, wherein, The first processing unit is configured to: calculate the forward optical flow and the backward optical flow between the two adjacent picture frames by using a first shader on the GPU, perform bidirectional consistency check and correction on the forward optical flow and the backward optical flow, and calculate the motion amplitude between the two adjacent picture frames according to the corrected forward optical flow; interpolate the two adjacent picture frames according to the corrected forward optical flow, the corrected backward optical flow and the motion amplitude by using a second shader on the GPU to obtain the new picture frame.

6. The apparatus of claim 4 or 5, wherein, The method further comprises the following steps: a third processing unit is configured to obtain a control instruction generated by a user manipulating the cover window, convert the control instruction into a control instruction for the display window, and send the control instruction for the display window to the system.

Images