A video stutter compensation method and related device

Abstract

The application relates to the technical field of video communication, and provides a video lag compensation method and related equipment. The method comprises the following steps: acquiring picture motion data, visual data and playing environment data of a video playing end at a current time; calculating a lag risk value of the video playing end based on the playing environment data; generating a predicted video frame according to the picture motion data and the visual data when the lag risk value meets a lag occurrence condition; generating a final playing video frame based on the predicted video frame according to a current decoding queue of the video playing end, rendering the final playing video frame, and playing a rendering result at a future time. The method can improve the playing fluency of video lag compensation.

Description

Technical Field

[0001] This application relates to the field of video communication technology, and in particular to a video stuttering compensation method and related equipment. Background Technology

[0002] Currently, there are two main background technologies in the field of real-time video playback: Passive retransmission and error correction-based packet loss mitigation technology (referencing common WebRTC protocol frameworks such as RFC 8829): This technology primarily relies on a combination of "jitter buffer" and "packet loss concealment (PLC)". Its core components include forward error correction (FEC), negative acknowledgment / retransmission (NACK / RTX), and picture loss indication (PLI). When packet loss or delay exceeding a preset threshold is detected, the system triggers a retransmission mechanism or reduces the bitrate. The relationship between these components is as follows: the buffer acts as a buffer zone to absorb jitter, and FEC provides static redundancy. However, this logic strictly adheres to the "post-hoc remediation" principle; retransmission is limited by network round-trip time (RTT). In cases of sudden network degradation, compensation data often lags behind the playback timeline. The overall reliance on "increasing bandwidth redundancy and sacrificing latency" to achieve reliability, without establishing a mechanism to proactively utilize the motion trends of video images, results in noticeable video pauses or momentary black screens even under deep weak network conditions.

[0003] Deep learning-based video frame interpolation technology primarily extracts features from the local motion regions of two adjacent frames and synthesizes intermediate frames using a shallow deep learning model to achieve real-time frame interpolation under limited computing power. While this approach improves the continuity of video playback to some extent, its application is still mainly based on pixel-level image processing, resulting in insufficient adaptability to real-time audio and video transmission scenarios. Specifically, this method does not fully utilize structured motion information such as motion vectors obtained during video decoding, leading to low computational resource utilization efficiency. Furthermore, the frame interpolation process does not adaptively adjust to network transmission conditions; under network fluctuations or frame loss, discontinuities or artifacts can easily occur between the predicted and actual images, making it difficult to reliably meet the requirements of low latency and high robustness in real-time audio and video scenarios.

[0004] In existing technologies, traditional video processing solutions primarily focus on optimizing video encoding parameters and motion compensation processes using machine learning or deep learning methods to improve encoding / decoding efficiency and video quality. Their application is mainly limited to the encoding stage. While these solutions involve motion estimation and compensation, they are typically based on linear motion information from a single frame or adjacent frames, failing to utilize multiple continuous motion vectors to construct a non-linear motion trend model. This makes them ill-suited for complex motion scenarios such as target acceleration or changes in direction. Furthermore, existing technologies do not incorporate playback buffer status and network fluctuations to predict stuttering beforehand, and lack a smooth switching mechanism between predicted and actual frames. When directly applied to real-time audio and video playback scenarios, they are prone to abrupt transitions during the frame recovery phase, making it difficult to effectively improve playback smoothness in weak network environments.

[0005] Therefore, it is evident that current video stuttering compensation methods suffer from poor playback smoothness. Summary of the Invention

[0006] This application provides a video stuttering compensation method and related equipment, which can solve the problem of poor playback smoothness in video stuttering compensation.

[0007] Firstly, this application provides a video stuttering compensation method, which includes: Acquire the video playback terminal's screen motion data, visual data, and playback environment data at the current moment; screen motion data is used to describe the movement of objects in the current playback screen, visual data is used to describe the overall state of the current playback screen, and playback environment data is used to describe the operating status of the video playback terminal. The video playback risk value is calculated based on playback environment data; the stuttering risk value is used to describe the degree of risk of video playback stuttering on the video playback terminal. When the stuttering risk value meets the stuttering occurrence conditions, a predicted video frame is generated based on the screen motion data and visual data; the predicted video frame is the screen that needs to be played at a future moment; Based on the current decoding queue at the video playback end, the final playback video frame is generated based on the predicted video frame. The final video frames are rendered, and the rendered results are played at a future time.

[0008] Optionally, playback environment data includes the decoding queue length; The risk value of video playback stuttering is calculated based on playback environment data, including: Calculate the remaining playable time based on the length of the decoding queue in the playback environment data; The risk of buffering during video playback is calculated based on the remaining playable time.

[0009] Optionally, the remaining playable time is calculated based on the length of the decoding queue for the playback environment data, including: Through the formula: ; Calculate remaining playback time ; in, Indicates the length of the decoding queue. Indicates the frame rate of the current video stream; The risk of buffering during video playback is calculated based on the remaining playable time, including: Through the formula: ; Calculate the risk value of stuttering ; in, , , These are the weighting coefficients. Represents a nonlinear mapping function. Indicates the remaining playback time. Indicates the probability of data arrival. Indicates the computational resource consumption for generating the prediction: ; in, This represents the average time taken to predict video frame generation. This represents the time budget available for predicting compensation.

[0010] Optionally, the condition for a lag to occur is: the lag risk value is greater than the risk threshold.

[0011] Optionally, predictive video frames are generated based on motion data and visual data, including: Calculate motion prediction vector fields based on image motion data; Feature extraction is performed on visual data to obtain fine-grained features; By fusing the motion prediction vector field and fine-grained features, a dense motion vector field is obtained; The latest decoded frame is back-mapped and filled using a dense motion vector field to generate a predicted video frame.

[0012] Optionally, the motion prediction vector field and fine-grained features are fused to obtain a dense motion vector field, including: Through the formula: ; Calculate dense motion vector field ; in, For confidence weighting plot, Represents the motion prediction vector field. Indicates fine-grained features. is the x-coordinate of the pixel. y is the ordinate of the pixel. This represents the residual mapping function.

[0013] Optionally, based on the current decoding queue at the video playback end, the final playback video frame is generated based on the predicted video frame, including: When the current decoding queue is empty, the predicted video frame will be used as the final video frame to be played. When the current decoding queue contains a predicted video frame and a real video frame arrives at the current decoding queue, the final playback video frame is generated based on the predicted video frame and the real video frame.

[0014] Optionally, the final playback video frames are generated based on the predicted video frames and the real video frames, including: Calculate the structural similarity between predicted video frames and real video frames; Determine whether the structural similarity is less than the similarity threshold; If so, the actual video frame will be used as the final video frame to be played. Otherwise, the predicted video frames and the actual video frames will be weighted and fused to obtain the final playback video frames.

[0015] Secondly, this application provides a video stuttering compensation device, comprising: The acquisition module is used to acquire the motion data, visual data, and playback environment data of the video playback terminal at the current moment. The motion data is used to describe the movement of objects in the current playback screen, the visual data is used to describe the overall state of the current playback screen, and the playback environment data is used to describe the operating status of the video playback terminal. The calculation module is used to calculate the stuttering risk value of the video playback terminal based on the playback environment data; the stuttering risk value is used to describe the degree of risk of video playback stuttering on the video playback terminal; The first generation module is used to generate predicted video frames based on screen motion data and visual data when the stuttering risk value meets the stuttering occurrence conditions; the predicted video frames are the screens that need to be played in the future. The second generation module is used to generate the final playback video frame based on the predicted video frame according to the current decoding queue of the video playback terminal. The rendering module is used to render the final video frames and play the rendered results at a future time.

[0016] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the video stuttering compensation method described above.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned video stuttering compensation method.

[0018] The above-mentioned solution in this application has the following beneficial effects: In the embodiments of this application, the system acquires the current motion data, visual data, and playback environment data of the video playback terminal. Then, it calculates the stuttering risk value of the video playback terminal based on the playback environment data. When the stuttering risk value meets the stuttering occurrence condition, it generates a predicted video frame based on the motion data and visual data. Finally, based on the current decoding queue of the video playback terminal, it generates the final playback video frame based on the predicted video frame and renders it, playing the rendered result at a future time. The calculation of the stuttering risk value allows for a description of the current stuttering risk of the video playback terminal, facilitating timely handling of stuttering situations. Generating the predicted video frame based on the current motion data and visual data considers both motion and visual information, effectively improving the accuracy of the predicted video frame. Generating the final playback video frame based on the current decoding queue before rendering avoids screen jumps and flickering caused by directly switching predicted video frames, thus improving the smoothness of video stuttering compensation playback.

[0019] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a video stuttering compensation method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a video stuttering compensation device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0022] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0023] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0024] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0025] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0026] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0028] To address the issue of poor playback smoothness in existing video stuttering compensation methods, this application provides a video stuttering compensation method. This method acquires the current motion data, visual data, and playback environment data of the video playback terminal. Then, it calculates a stuttering risk value based on the playback environment data. When the stuttering risk value meets the stuttering occurrence condition, it generates a predicted video frame based on the motion data and visual data. Finally, based on the current decoding queue of the video playback terminal, it generates a final playback video frame from the predicted video frame and renders it, playing the rendered result at a future time. The calculation of the stuttering risk value allows for a description of the current stuttering risk at the video playback terminal, facilitating timely handling of stuttering situations. Generating predicted video frames based on the current motion and visual data considers both motion and visual information, effectively improving the accuracy of the predicted video frames. Generating the final playback video frame based on the current decoding queue before rendering avoids screen jumps and flickering caused by directly switching predicted video frames, thus improving the playback smoothness of the video stuttering compensation.

[0029] The following is an exemplary description of the video stuttering compensation method provided in this application.

[0030] like Figure 1 As shown, the video stuttering compensation method provided in this application includes the following steps: Step 11: Obtain the motion data, visual data, and playback environment data of the video playback terminal at the current moment.

[0031] The aforementioned video playback end refers to devices such as mobile phones and computers that play videos. The aforementioned screen motion data is used to describe the motion of objects in the currently playing screen, including data such as macroblock motion vectors, segmentation modes, and reference frame indices of the currently playing screen. The aforementioned visual data is used to describe the overall state of the currently playing screen, including data such as the original video frames of the currently playing screen. The aforementioned playback environment data is used to describe the operating status of the video playback end, including data such as decoding queue length, playback buffer level, decoding queue length, network packet loss retransmission rate, and the CPU / GPU load rate of the current device.

[0032] In some embodiments of this application, video playback terminal control scripts, control systems, etc., can be used to obtain image motion data, visual data, and playback environment data.

[0033] Step 12: Calculate the stuttering risk value of the video playback terminal based on the playback environment data.

[0034] The above-mentioned stuttering risk value is used to describe the degree of risk of video playback stuttering on the video playback end.

[0035] In some embodiments of this application, the step of calculating the stuttering risk value of the video playback terminal based on playback environment data includes: The first step is to calculate the remaining playable time based on the length of the decoding queue in the playback environment data.

[0036] Specifically, through the formula: ; Calculate remaining playback time .

[0037] in, This indicates the length of the decoding queue (i.e., the number of remaining frames to be played in the decoding queue). This indicates the frame rate of the current video stream.

[0038] The second step is to calculate the risk of buffering on the video playback device based on the remaining playable time.

[0039] Specifically, through the formula: ; Calculate the risk value of stuttering .

[0040] in, , , These are the weighting coefficients. This represents a nonlinear mapping function that transforms physical state variables with different dimensions into a unified dimensionless risk value. Indicates the remaining playback time. Indicates the probability of data arrival. Indicates the computational resource consumption for generating the prediction: ; in, This represents the average time taken to predict video frame generation. This represents the time budget available for predicting compensation.

[0041] For example, the weighting coefficient is used to adjust the relative importance of different risk factors in the stuttering risk score. It can be dynamically adjusted according to the playback scenario, network status, or system load, and can be adaptively updated based on the current network stability and system load. For example, it can be increased when there are large network fluctuations. The weight is increased when the system's computing power is strained. The weights. Generally speaking, the weighting coefficients should satisfy: .

[0042] Step 13: When the stuttering risk value meets the stuttering occurrence conditions, generate a predicted video frame based on the screen motion data and visual data.

[0043] The predicted video frames above represent the footage that will be played at a future moment. A future moment is the point after the current moment where the video stuttering will occur.

[0044] Specifically, the condition for a lag to occur is that the lag risk value is greater than the risk threshold.

[0045] It should be noted that the specific steps for generating predicted video frames based on image motion data and visual data are as follows: First, the motion prediction vector field is calculated based on the image motion data.

[0046] The aforementioned motion prediction vector field is used to describe the motion trend of objects in a video frame.

[0047] Specifically, algorithms such as Kalman filtering can be used to process the motion data in the image to obtain the motion prediction vector field.

[0048] Then, feature extraction is performed on the visual data to obtain fine-grained features.

[0049] The fine-grained features described above are used to describe the image differences between video frames in a video.

[0050] Specifically, obtain the latest decoded complete video frame and perform... Downsampling at a higher rate yields a low-resolution feature map. Subtraction and other methods are used to calculate the difference or gradient features between the map and the previous frame, and the results are used as fine-grained features.

[0051] Then, the motion prediction vector field and fine-grained features are fused to obtain a dense motion vector field.

[0052] Specifically, through the formula: ; Calculate dense motion vector field .

[0053] in, For confidence weighting plot, Represents the motion prediction vector field. Indicates fine-grained features. is the x-coordinate of the pixel. y is the ordinate of the pixel. This represents the residual mapping function from features to motion vectors.

[0054] Finally, the latest decoded frame is back-mapped and filled using a dense motion vector field to generate a predicted video frame.

[0055] The aforementioned latest decoded frame is the latest video frame decoded by the video playback device up to the current moment.

[0056] For example, the latest decoded frame can be back-mapped and padded using a Transformer model and a dense motion vector field to generate a predicted video frame.

[0057] Step 14: Generate the final playback video frame based on the predicted video frame according to the current decoding queue of the video playback terminal.

[0058] The aforementioned current decoding queue is the queue of video frames waiting to be decoded and rendered at the current moment.

[0059] Specifically, when the current decoding queue is empty (i.e., there are no real video frames that can be decoded in the current decoding queue, and the real video frames are the video frames that are actually played by the video playback end), the predicted video frame will be used as the final video frame to be played.

[0060] When the current decoding queue contains a predicted video frame and the actual video frame arrives at the current decoding queue (that is, after the predicted video frame is generated, it is placed in the queue to wait for decoding and rendering, and then the actual video frame is received, at which point the current decoding queue contains both predicted and actual video frames, and the actual video frame enters the queue after the predicted video frame), the final playback video frame is generated based on the predicted and actual video frames.

[0061] When there are only real video frames in the current decoding queue (i.e., when the stuttering risk value is less than or equal to the risk threshold, it means that there is no stuttering risk on the video playback end, and there is no need to generate predicted video frames), the real video frames are directly used as the final playback video frames.

[0062] It should be noted that the steps for generating the final playback video frame based on the predicted video frame and the real video frame are as follows: calculate the structural similarity between the predicted video frame and the real video frame; determine whether the structural similarity is less than the similarity threshold; if so, use the real video frame as the final playback video frame; otherwise, perform a weighted fusion of the predicted video frame and the real video frame to obtain the final playback video frame.

[0063] For example, the structural similarity index (SSIM) between predicted video frames and real video frames can be calculated, and this index can be used as the structural similarity. The expression for the weighted fusion is as follows: ; in, This indicates the final video frame played. Indicates the weighted weight. These are real video frames. To predict video frames.

[0064] In some embodiments of this application, the above process is as follows: after detecting a risk of stuttering, predictive video frames are continuously generated and added to the decoding queue. After receiving the real video frame, in order to avoid screen jumps and flickering caused by direct switching, a transition process is initiated to weightedly fuse the real video frame and the predictive video frame in the queue, and the final playback video frame after fusion is played. After the final playback video frame is played, the system switches to the real video frame and continues to play it.

[0065] Step 15: Render the final video frame and play the rendered result at a future time.

[0066] For example, after obtaining the final video frame, it is encapsulated into a directly renderable texture object, stamped with a predicted timestamp, and written to a prediction compensation buffer for rendering. Upon reaching a future time, the final video frame is decoded and rendered using the video playback client's built-in decoding tools, and then played.

[0067] It is worth mentioning that calculating the stuttering risk value can describe the current stuttering risk on the video playback end, making it easier to handle stuttering situations in a timely manner. Based on the current screen motion data and visual data, predictive video frames are generated, taking into account information from screen motion and visual aspects, which effectively improves the accuracy of predicted video frames. The final playback video frame is generated based on the current decoding queue before rendering, avoiding screen jumps and flickering caused by directly switching predicted video frames, thus improving the playback smoothness of video stuttering compensation.

[0068] The video stuttering compensation device provided in this application is described below as an example.

[0069] like Figure 2 As shown, this application embodiment provides a video stuttering compensation device, the video stuttering compensation device 200 including: The acquisition module 201 is used to acquire the screen motion data, visual data and playback environment data of the video playback terminal at the current moment; the screen motion data is used to describe the movement of objects in the screen, the visual data is used to describe the overall state of the screen, and the playback environment data is used to describe the operating state of the video playback terminal. Calculation module 202 is used to calculate the stuttering risk value of the video playback terminal based on playback environment data; the stuttering risk value is used to describe the degree of risk of video playback stuttering on the video playback terminal; The first generation module 203 is used to generate a predicted video frame based on the screen motion data and visual data when the stuttering risk value meets the stuttering occurrence conditions; the predicted video frame is the screen that needs to be played in the future. The second generation module 204 is used to generate the final playback video frame based on the predicted video frame according to the current decoding queue of the video playback end. The rendering module 205 is used to render the final video frames and play the rendering results at a future time.

[0070] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0071] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0072] like Figure 3 As shown, an embodiment of this application provides a terminal device, wherein the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 3 The diagram shows only one processor, a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 executes the computer program D102 to implement the steps in any of the above method embodiments.

[0073] Specifically, when the processor D100 executes the computer program D102, it acquires the current motion data, visual data, and playback environment data of the video playback terminal. Then, based on the playback environment data, it calculates the stuttering risk value of the video playback terminal. When the stuttering risk value meets the stuttering occurrence conditions, it generates a predicted video frame based on the motion data and visual data. Finally, based on the current decoding queue of the video playback terminal, it generates the final playback video frame based on the predicted video frame and renders the final playback video frame, playing the rendered result at a future time. The calculation of the stuttering risk value describes the current stuttering risk of the video playback terminal, facilitating timely handling of stuttering situations. Generating the predicted video frame based on the current motion data and visual data considers motion and visual information, effectively improving the accuracy of the predicted video frame. Generating the final playback video frame based on the current decoding queue before rendering avoids direct switching of the predicted video frame, which can cause screen jumps and flickering, thus improving the smoothness of video stuttering compensation playback.

[0074] The processor D100 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0075] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may be an external storage device of the terminal device D10, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device D10. Furthermore, the memory D101 may include both internal and external storage units of the terminal device D10. The memory D101 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory D101 can also be used to temporarily store data that has been output or will be output.

[0076] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0077] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0078] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to the video stuttering compensation method apparatus / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0079] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0080] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0081] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention.

Claims

1. A video stuttering compensation method, characterized in that, include: Acquire the video playback device's current motion data, visual data, and playback environment data; The motion data is used to describe the movement of objects in the currently playing screen, the visual data is used to describe the overall state of the currently playing screen, and the playback environment data is used to describe the operating status of the video playback terminal. The video playback terminal's stuttering risk value is calculated based on the playback environment data. The stuttering risk value is used to describe the degree of risk of video playback stuttering occurring on the video playback terminal; When the stuttering risk value meets the stuttering occurrence condition, a predicted video frame is generated based on the screen motion data and the visual data; the predicted video frame is the screen that needs to be played at a future time. Based on the current decoding queue of the video playback terminal, the final playback video frame is generated based on the predicted video frame; The final video frame is rendered, and the rendered result is played at the future time.

2. The video stuttering compensation method according to claim 1, characterized in that, The playback environment data includes the decoding queue length; The calculation of the stuttering risk value of the video playback terminal based on the playback environment data includes: The remaining playable time is calculated based on the length of the playback environment data decoding queue. The risk value of buffering at the video playback end is calculated based on the remaining playable time.

3. The video stuttering compensation method according to claim 2, characterized in that, The calculation of the remaining playable time based on the length of the playback environment data decoding queue includes: Through the formula: Calculate remaining playback time ; in, Indicates the length of the decoding queue. Indicates the frame rate of the current video stream; The calculation of the video playback latency risk value based on the remaining playable time includes: Through the formula: Calculate the risk value of stuttering ; in, , , These are the weighting coefficients. Represents a nonlinear mapping function. Indicates the remaining playback time. Indicates the probability of data arrival. Indicates the computational resource consumption for generating the prediction: in, This represents the average time taken to predict video frame generation. This represents the time budget available for predicting compensation.

4. The video stuttering compensation method according to claim 1, characterized in that, The condition for the lag to occur is that the lag risk value is greater than the risk threshold.

5. The video stuttering compensation method according to claim 4, characterized in that, The step of generating predicted video frames based on the motion data and the visual data includes: Calculate the motion prediction vector field based on the aforementioned image motion data; Feature extraction is performed on the visual data to obtain fine-grained features; The motion prediction vector field and the fine-grained features are fused to obtain a dense motion vector field; The dense motion vector field is used to backmap and fill the latest decoded frame to generate a predicted video frame.

6. The video stuttering compensation method according to claim 5, characterized in that, The process of fusing the motion prediction vector field and the fine-grained features to obtain a dense motion vector field includes: Through the formula: Calculate dense motion vector field ; in, For confidence weighting plot, Represents the motion prediction vector field. Indicates fine-grained features. is the x-coordinate of the pixel. y is the ordinate of the pixel. This represents the residual mapping function.

7. The video stuttering compensation method according to claim 6, characterized in that, The step of generating the final playback video frame based on the predicted video frame according to the current decoding queue of the video playback terminal includes: When the current decoding queue is empty, the predicted video frame is used as the final playback video frame; When the current decoding queue contains a predicted video frame and a real video frame arrives at the current decoding queue, the final playback video frame is generated based on the predicted video frame and the real video frame.

8. The video stuttering compensation method according to claim 7, characterized in that, The step of generating the final playback video frame based on the predicted video frame and the real video frame includes: Calculate the structural similarity between predicted video frames and real video frames; Determine whether the structural similarity is less than a similarity threshold; If so, the actual video frame will be used as the final video frame to be played. Otherwise, the predicted video frames and the actual video frames are weighted and fused to obtain the final playback video frames.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the video stuttering compensation method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the video stuttering compensation method as described in any one of claims 1 to 8.

Images