A video processing method, apparatus, device, and storage medium

Abstract

An embodiment of the present application discloses a video processing method, apparatus, device, and storage medium. The method includes: in response to the decoding of the first video frame in the video to be processed being completed, obtaining a speed change parameter in a decoding queue corresponding to the video to be processed, where the speed change parameter is used to indicate whether the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed, obtaining a rendering parameter of the video to be processed, where the rendering parameter is used to indicate the time interval between video frames, and if the speed change parameter indicates that the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed, and the time interval between the video frames indicated by the rendering parameter meets the rendering frame dropping condition, then performing rendering frame dropping processing on the first video frame. It can be seen that on the premise of ensuring the continuity of the video to be processed, by performing frame dropping processing on the decoded video frames, the video delay can be reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to a video processing method, a video processing device, a computer device, and a computer-readable storage medium. Background Art

[0002] With the progress of scientific and technological research, videos have been widely used in people's daily lives. In low-latency video scenarios (such as live streaming, video calls, cloud gaming services, etc.), the video communication engine processes situations such as packet loss, out-of-order, and delayed arrival through caching to ensure that video frames can be smoothly output to the decoder. It has been found that during video processing, affected by factors such as network fluctuations and decoder performance fluctuations, the video latency will increase accordingly, resulting in a poor video interaction experience. Summary of the Invention

[0003] Embodiments of this application provide a video processing method, device, equipment, and computer-readable storage medium, which can reduce video latency.

[0004] On the one hand, embodiments of this application provide a video processing method, including:

[0005] In response to the decoding of the first video frame in the video to be processed being completed, obtain the speed change parameter in the decoding queue corresponding to the video to be processed, where the speed change parameter is used to indicate whether the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed;

[0006] Obtain the rendering parameter of the video to be processed, where the rendering parameter is used to indicate the time interval between video frames;

[0007] If the speed change parameter indicates that the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed, and the time interval between the video frames indicated by the rendering parameter meets the rendering frame dropping condition, perform rendering frame dropping processing on the first video frame.

[0008] On the other hand, embodiments of this application provide a video processing device, which includes:

[0009] An obtaining unit, configured to obtain the speed change parameter in the decoding queue corresponding to the video to be processed in response to the decoding of the first video frame in the video to be processed being completed, where the speed change parameter is used to indicate whether the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed;

[0010] And configured to obtain the rendering parameter of the video to be processed, where the rendering parameter is used to indicate the time interval between video frames;

[0011] A processing unit, which is configured to perform rendering frame dropping processing on a first video frame if a speed change parameter indicates that the writing speed of video frames in a decoding queue is greater than the decoding speed of video frames, and a time interval between video frames indicated by a rendering parameter meets a rendering frame dropping condition.

[0012] In an implementation, a speed change parameter in a decoding queue corresponding to a video to be processed includes a number of frames sent, and the number of frames sent is used to indicate the number of video frames in the decoding queue whose writing time interval is less than a first time interval threshold; the processing unit is further configured to:

[0013] If the number of frames sent is greater than or equal to a preset number of frames, it is determined that the writing speed of video frames in the decoding queue is greater than the decoding speed of video frames.

[0014] In an implementation, the processing unit is further configured to:

[0015] If the number of frames sent is less than the preset number of frames, and a time interval between video frames indicated by a rendering parameter meets a rendering frame dropping condition, obtain a cache parameter, where the cache parameter is used to indicate whether cache resources corresponding to the decoding queue meet decoding requirements;

[0016] If the cache parameter indicates that cache resources corresponding to the decoding queue do not meet decoding requirements, determine a rendering frame dropping frequency based on the cache parameter;

[0017] Perform rendering frame dropping processing on video frames decoded within a target time period according to the rendering frame dropping frequency.

[0018] In an implementation, the cache parameter includes an input cache time consumption of a second video frame, and the second video frame is the last video frame enqueued in the decoding queue; the processing unit is further configured to:

[0019] If the input cache time consumption of the second video frame is greater than a cache time consumption threshold, it is determined that cache resources corresponding to the decoding queue do not meet decoding requirements.

[0020] In an implementation, the cache parameter includes an input cache time consumption of a second video frame, and the second video frame is the last video frame enqueued in the decoding queue; the second video frame is transferred from an input cache to the decoding queue; the processing unit is configured to obtain the cache parameter, specifically:

[0021] Obtain the time when the second video frame first requests to be added to the input cache and the time when the second video frame is successfully transferred to the decoding queue;

[0022] Calculate the input cache time consumption of the second video frame according to the time when the second video frame first requests to be added to the input cache and the time when the second video frame is successfully transferred to the decoding queue.

[0023] In one implementation, the caching parameter includes the input caching time of the second video frame, where the second video frame is the last video frame enqueued in the decoding queue; the processing unit is configured to determine the rendering frame dropping frequency based on the caching parameter, specifically:

[0024] Determine the time interval to which the input caching time of the second video frame belongs, where different time intervals correspond to different frame dropping frequencies;

[0025] Determine the frame dropping frequency corresponding to the time interval to which the input caching time of the second video frame belongs as the rendering frame dropping frequency.

[0026] In one implementation, the processing unit is further configured to:

[0027] If the caching parameter indicates that the caching resource corresponding to the decoding queue meets the decoding requirement, then render the first video frame and output the first video frame.

[0028] In one implementation, the processing unit is further configured to:

[0029] If the number of frames sent is less than or equal to the preset number of frames, and the time interval between the video frames indicated by the rendering parameter meets the rendering frame dropping condition, then obtain the decoding performance parameter, where the decoding performance parameter is used to indicate whether the decoding performance of the decoder meets the decoding requirement;

[0030] If the decoding performance parameter indicates that the decoding performance of the decoder does not meet the decoding requirement, then determine the rendering frame dropping frequency as the first frequency;

[0031] Perform rendering frame dropping processing on the video frames decoded within the first time period according to the first frequency.

[0032] In one implementation, the decoding performance parameter includes the number of decoded frames in the queue at the first moment and the threshold of the number of decoded frames in the queue at the first moment, where the number of decoded frames in the queue at the first moment is used to indicate the number of video frames in the decoding queue at the first moment, and the threshold of the number of decoded frames in the queue at the first moment is determined based on the decoding performance of the decoder at the first moment; the processing unit is further configured to:

[0033] If the number of decoded frames in the queue at the first moment is greater than the threshold of the number of decoded frames in the queue at the first moment, then determine that the decoding performance of the decoder at the first moment does not meet the decoding requirement.

[0034] In one implementation, the decoding performance parameter further includes the number of decoded frames in the queue at the second moment and the threshold of the number of decoded frames in the queue at the second moment, where the second moment is after the first moment; the processing unit is further configured to:

[0035] If the number of decoded frames in the queue at the second moment is greater than the threshold of the number of decoded frames in the queue at the second moment, then determine the rendering frame dropping frequency as the second frequency, where the second frequency is greater than the first frequency;

[0036] Render the video frames decoded within the second time period at the second frequency. The second time period is after the first time period.

[0037] In one embodiment, the processing unit is further configured to:

[0038] If the number of decoded video frames at the second moment is less than or equal to the decoded video frame number threshold at the second moment, determine the rendering frame dropping frequency as the third frequency, where the third frequency is less than the first frequency;

[0039] Render the video frames decoded within the second time period at the third frequency.

[0040] In one embodiment, the decoding queue contains M video frames, where M is an integer greater than 1; the processing unit is configured to obtain the speed change parameter in the decoding queue corresponding to the video to be processed, specifically:

[0041] Obtain the time when the M video frames are written into the decoding queue;

[0042] Based on the time when the M video frames are written into the decoding queue, calculate the write time interval between adjacent video frames among the M video frames;

[0043] According to the write time interval between adjacent video frames among the M video frames and the first time interval threshold, count the number of frames sent.

[0044] In one embodiment, the processing unit is further configured to:

[0045] If the speed change parameter indicates that the write speed of the video frames in the decoding queue is less than or equal to the video frame decoding speed, render the first video frame and output the first video frame; or,

[0046] If the time interval between the video frames indicated by the rendering parameter does not meet the rendering frame dropping condition, render the first video frame and output the first video frame.

[0047] In one embodiment, the rendering parameter of the video to be processed includes a rendering time interval, which is used to indicate the time interval between the time when the first video frame is decoded and the time when the third video frame is rendered. The third video frame is the video frame with the shortest time interval between the rendering time and the time when the first video frame is decoded among the rendered video frames of the video to be processed; the time interval between the video frames meeting the rendering frame dropping condition includes that the rendering time interval is less than or equal to the second time interval threshold.

[0048] Correspondingly, the present application provides a computer device, which includes:

[0049] A memory, in which a computer program is stored;

[0050] A processor for loading a computer program to implement the above video processing method.

[0051] Correspondingly, the present application provides a computer-readable storage medium storing a computer program, which is adapted to be loaded and executed by a processor to implement the above video processing method.

[0052] Correspondingly, the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device implements the above video processing method.

[0053] In an embodiment of the present application, in response to the decoding of the first video frame in the video to be processed being completed, a speed change parameter in the decoding queue corresponding to the video to be processed is obtained. The speed change parameter is used to indicate whether the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed. A rendering parameter of the video to be processed is obtained. The rendering parameter is used to indicate the time interval between video frames. If the speed change parameter indicates that the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed, and the time interval between the video frames indicated by the rendering parameter meets the rendering frame dropping condition, then rendering frame dropping processing is performed on the first video frame. It can be seen that on the premise of ensuring the continuity of the video to be processed, by performing frame dropping processing on the decoded video frames, the video delay can be reduced. Description of the Drawings

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0055] Figure 1a It is a scene architecture diagram of a video processing system provided by an embodiment of the present application;

[0056] Figure 1b It is a process architecture diagram of a video processing solution provided by an embodiment of the present application;

[0057] Figure 2 It is a flowchart of a video processing method provided by an embodiment of the present application;

[0058] Figure 3 It is a flowchart of another video processing method provided by an embodiment of the present application;

[0059] Figure 4Schematic structural diagram of a video processing device provided by an embodiment of the present application;

[0060] Figure 5 Schematic structural diagram of a computer device provided by an embodiment of the present application. Specific implementation manners

[0061] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of the present application.

[0062] First, a brief introduction to the terms of the related technologies involved in the present application will be given:

[0063] Receiving frame rate: The receiving frame rate refers to the number of video frames received by the decoder per second, and the unit is frames per second (abbreviated as fps).

[0064] Decoding frame buffering: Decoding frame buffering refers to the phenomenon that the decoder starts to output decoded images only after receiving a certain number of video frames; for example, in a low-latency video service scenario with a requirement of 1080P@60fps, the normal video latency is 1000 / 60 = 16.66ms. When the number of decoded frame buffers is 2 (that is, the number of video frames to be decoded in the decoder's decoding queue is 2), the video latency will increase to 2 * 16.66 = 33.32ms.

[0065] Input buffer: The input buffer refers to the video frame buffer allocated by the computer device for the decoder. The video frames received by the computer device will be passed to the decoder through the input buffer for decoding. Specifically, after receiving a video frame, the computer device will first request to add the video frame to the input buffer. After being successfully added to the input buffer, the video frame will be passed from the input buffer to the decoder's decoding queue for decoding.

[0066] Rendering frame dropping: Rendering frame dropping refers to the computer device discarding a decoded video frame after completing the decoding of a video frame and not rendering and presenting the decoded video frame.

[0067] Based on the above technologies, an embodiment of the present application provides a video processing solution that can reduce video latency. Figure 1a Scene architecture diagram of a video processing system provided by an embodiment of the present application. As Figure 1aAs shown in the figure, the video processing system may include: a terminal device 101 and a server 102. The video processing method provided in the embodiments of the present application may be executed by the terminal device 101. Among them, the terminal device 101 includes, but is not limited to: smart phones (such as Android phones, IOS phones, etc.), tablet computers, portable personal computers, smart home appliances, vehicle-mounted terminals, wearable devices, unmanned aerial vehicles, and other devices with video playback functions. The embodiments of the present application do not make any limitations in this regard. The server 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content delivery network), and big data and artificial intelligence platforms. The embodiments of the present application do not make any limitations in this regard.

[0068] It should be noted that the terminal device 101 and the server 102 may be directly or indirectly connected through wired communication or wireless communication. This application does not make any restrictions in this regard. The numbers of the terminal device 101 and the server 102 are only for illustration and do not constitute an actual limitation of this application; for example, the video processing system may further include a terminal device 103, or a server 104.

[0069] Figure 1b It is a flow architecture diagram of a video processing solution provided in the embodiments of the present application. As Figure 1b shown, the general process of the video processing solution is as follows:

[0070] (1) The terminal device 101 receives the video to be processed sent by the server 102. In one implementation, the video frames in the video to be processed are sent by the server 102 to the terminal device 101 in real time. That is to say, the terminal device 101 can receive the video frames sent by the server 102 while decoding and rendering the received video frames. During the process of decoding the video frames by the terminal device 101, relevant parameters will be updated in real time. These parameters include: input buffer time consumption, number of frames sent, number of decoded frames stored, and rendering time interval; among them, the input buffer time consumption can be used to indicate whether the buffer resources corresponding to the decoding queue meet the decoding requirements; the number of frames sent can be used to indicate whether the writing speed of the video frames in the decoding queue corresponding to the decoder is greater than the video frame decoding speed; the number of decoded frames stored can be used to indicate whether the decoding performance of the decoder meets the decoding requirements; the rendering time interval can be used to indicate whether the video to be processed can ensure continuous playback after rendering and dropping frames for the decoded video frames.

[0071] The following details the process of decoding the video frames and the acquisition of each parameter:

[0072] In one implementation, after receiving the video frame sent by the server 102, the terminal device 101 requests to add the video frame to the input buffer (i.e., the buffer resource corresponding to the decoding queue). After being successfully added to the input buffer, the video frame is transferred (written) from the input buffer to the decoding queue corresponding to the decoder. When the video frame is successfully transferred to the decoding queue, the terminal device 101 records the input buffer time of the video frame. The input buffer time of the video frame is used to indicate the time elapsed from the first request to add the video frame to the input buffer until the video frame is successfully transferred to the decoding queue; that is, the input buffer time of the video frame is calculated based on the time when the video frame is first requested to be added to the input buffer and the time when the video frame is successfully transferred to the decoding queue. In addition, the terminal device 101 can also count the number of video frames in the decoding queue in real time to obtain the decoded frame count; calculate the time interval between adjacent video frames in the decoding queue based on the time when the video frames in the decoding queue are written into the decoding queue, and then count the frame sending count; adjacent video frames refer to video frames with adjacent arrangement positions in the decoding queue; for example, assume that the decoding queue includes video frame 1 - video frame 3, and video frame 1 - video frame 3 are arranged in ascending order of serial number, then video frame 1 and video frame 2 are adjacent frames to each other, and video frame 2 and video frame 3 are adjacent frames to each other.

[0073] Further, when the video frame decoding is completed, the terminal device 101 can calculate the rendering time interval based on the time when the video frame completes decoding and the time when the third video frame is rendered; where the third video frame is the video frame with the shortest time interval between the time when it is rendered and the time when the currently completed decoded video frame completes decoding among the already rendered video frames; for example, assume that the time when video frame 1 completes decoding is the 3rd second, and the already rendered video frames include: video frame 2 is rendered at the 0.5th second, video frame 3 is rendered at the 1.1th second, video frame 4 is rendered at the 1.9th second, video frame 5 is rendered at the 2.1th second, and video frame 6 is rendered at the 2.7th second; then for video frame 1, the third video frame is video frame 6, and the rendering time interval = 3 - 2.7 = 0.3 second. It should be noted that the numerical values and time units in the above example are only for illustration, and in actual applications, the unit of the rendering time interval can be milliseconds.

[0074] (2) In response to the completion of decoding of the first video frame in the video to be processed, the terminal device 101 obtains the speed change parameter in the decoding queue corresponding to the video to be processed. The speed change parameter is used to indicate whether the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed. In one implementation, the speed change parameter in the decoding queue corresponding to the video to be processed includes the number of frames sent. If the number of frames sent is greater than or equal to the preset number of frames (such as 2), the terminal device 101 determines that the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed. In another implementation, the terminal device 101 can directly count the writing speed and the video frame decoding speed of the decoding queue within a preset time period (such as within the first 10 seconds before the current moment), and use them as the speed change parameter in the decoding queue.

[0075] (3) The terminal device 101 obtains the rendering parameter of the video to be processed. The rendering parameter is used to indicate the time interval between video frames. Through the time interval between video frames, the terminal device 101 can determine whether the playback continuity of the video to be processed can be guaranteed after performing the rendering frame dropping process on the first video frame. In one implementation, the rendering parameter includes the rendering time interval. If the rendering time interval between the first video frame and the third video frame is less than or equal to the second time interval threshold (such as 50 ms), the terminal device 101 determines that the playback continuity of the video to be processed can be guaranteed after performing the rendering frame dropping process on the first video frame (that is, the time interval between video frames meets the rendering frame dropping condition). It should be noted that the second time interval threshold is used to ensure the continuity of the video. The second time interval threshold can be a fixed value or calculated based on the target decoding frame rate corresponding to the video to be processed; for example, the second time interval threshold = 1000 / target decoding frame rate * 2.

[0076] (4) If the speed change parameter indicates that the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed (such as the number of frames sent is greater than or equal to the preset number of frames), and the time interval between the video frames indicated by the rendering parameter meets the rendering frame dropping condition (such as the rendering time interval is less than or equal to the second time interval threshold), the terminal device 101 performs the rendering frame dropping process on the first video frame.

[0077] Accordingly, if the speed change parameter indicates that the writing speed of video frames in the decoding queue is less than or equal to the video frame decoding speed (such as the number of frames sent is less than the preset number of frames), and the time interval between video frames indicated by the rendering parameter meets the rendering frame dropping condition, the terminal device 101 may obtain a cache parameter, where the cache parameter is used to indicate whether the cache resources corresponding to the decoding queue meet the decoding requirements. In one implementation, the cache parameter includes the input cache time of the second video frame, and the second video frame is the last video frame to enter the decoding queue; for example, assuming that the decoding queue includes video frames 1 - 5, and the order of entry of video frames 1 - 5 is the same as the sequence number, the second video frame refers to video frame 5. If the input cache time of the second video frame is greater than the cache time threshold (such as 10 ms), the terminal device 101 determines that the cache resources corresponding to the decoding queue do not meet the decoding requirements. Further, if the cache parameter indicates that the cache resources corresponding to the decoding queue do not meet the decoding requirements, the terminal device 101 determines the rendering frame dropping frequency based on the cache parameter (such as the terminal device 101 may determine the rendering frame dropping rate based on the correspondence between the cache parameter and the rendering frame dropping rate), and performs rendering frame dropping processing on the video frames decoded within the target time period according to the rendering frame dropping frequency. The target time period may be a preset fixed time period or a time period dynamically determined by the terminal device based on relevant parameters; for example, assuming that the fourth video frame is a video frame after the second video frame, when the fourth video frame is added to the decoding queue, if the terminal device 101 detects that the input cache time of the fourth video frame is less than the cache time threshold, the terminal device 101 may determine the time period from when the second video frame is added to the decoding queue to when the fourth video frame is added to the decoding queue as the target time period.

[0078] If the speed change parameter indicates that the writing speed of video frames in the decoding queue is less than or equal to the video frame decoding speed (such as the number of frames sent is less than the preset number of frames), and the time interval between video frames indicated by the rendering parameter meets the rendering frame-drop condition, the terminal device 101 can also obtain a decoding performance parameter, which is used to indicate whether the decoding performance of the decoder meets the decoding requirements. In one implementation, the decoding performance parameter includes the number of decoded frames in the decoding queue at the first moment, and the number of decoded frames at the first moment is used to indicate the number of video frames in the decoding queue at the first moment; if the number of decoded frames at the first moment is greater than the decoding frame number threshold at the first moment, the terminal device 101 determines that the decoding performance of the decoder at the first moment does not meet the decoding requirements. Further, if the decoding performance parameter indicates that the decoding performance of the decoder does not meet the decoding requirements, the terminal device 101 determines the rendering frame-drop frequency as the first frequency, and performs rendering frame-drop processing on the video frames decoded within the first time period according to the first frequency. The first time period can be a preset fixed time period, or a time period dynamically determined by the terminal device based on relevant parameters; in the subsequent process, the terminal device 101 can also adjust (increase or decrease) the first frequency based on the number of decoded frames in the decoding queue at the current moment.

[0079] In the embodiments of the present application, in response to the decoding completion of the first video frame in the video to be processed, obtain the speed change parameter in the decoding queue corresponding to the video to be processed, where the speed change parameter is used to indicate whether the writing speed of video frames in the decoding queue is greater than the video frame decoding speed, obtain the rendering parameter of the video to be processed, where the rendering parameter is used to indicate the time interval between video frames, and if the speed change parameter indicates that the writing speed of video frames in the decoding queue is greater than the video frame decoding speed, and the time interval between video frames indicated by the rendering parameter meets the rendering frame-drop condition, then perform rendering frame-drop processing on the first video frame. It can be seen that on the premise of ensuring the continuity of the video to be processed, by performing frame-drop processing on the decoded video frames, the video delay can be reduced.

[0080] Based on the above video processing solution, the embodiments of the present application propose a more detailed video processing method, and the video processing method proposed in the embodiments of the present application will be introduced in detail below with reference to the accompanying drawings.

[0081] Please refer to Figure 2 , Figure 2 which is a flowchart of a video processing method provided by the embodiments of the present application. This video processing method can be executed by a computer device, and the computer device can be Figure 1a the terminal device 101 shown in Figure 2 . As shown in

[0082] S201. In response to the completion of decoding of the first video frame in the video to be processed, obtain the speed change parameter in the decoding queue corresponding to the video to be processed.

[0083] The speed change parameter in the decoding queue corresponding to the video to be processed is used to indicate whether the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed. In one implementation, the speed change parameter in the decoding queue corresponding to the video to be processed includes the number of frames sent, and the number of frames sent is used to indicate the number of video frames in the decoding queue whose writing time interval is less than the first time interval threshold. If the number of frames sent is greater than or equal to the preset number of frames (such as 2), the computer device determines that the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed.

[0084] In one embodiment, the decoding queue contains M video frames, where M is an integer greater than 1. The specific implementation for the computer device to obtain the number of frames sent is as follows: obtain the time when the M video frames are written into the decoding queue, and based on the time when the M video frames are written into the decoding queue, calculate the writing time interval between adjacent video frames among the M video frames. Adjacent video frames refer to video frames that are adjacent in the arrangement position in the decoding queue. For example, assume that the decoding queue includes video frame 1 - video frame 3, and video frame 1 - video frame 3 are arranged in ascending order of serial numbers. Then video frame 1 and video frame 2 are adjacent frames to each other, and video frame 2 and video frame 3 are adjacent frames to each other. It can be understood that the M video frames correspond to M - 1 writing time intervals between adjacent video frames. Further, the computer device can count the number of frames sent according to the M - 1 writing time intervals between adjacent video frames among the M video frames and the first time interval threshold. Specifically, the number of frames sent = K + 1, where K is the number of writing time intervals greater than the first time interval threshold among the M - 1 writing time intervals between adjacent video frames, and K is a positive integer less than M. For example, assume that the decoding queue includes video frame 1 - video frame 3, and video frame 1 - video frame 3 are arranged in ascending order of serial numbers. The writing time interval between video frame 1 and video frame 2 is greater than the first time interval threshold, and the writing time interval between video frame 2 and video frame 3 is less than the first time interval threshold. Then the number of frames sent = 1 + 1 = 2.

[0085] S202. Obtain the rendering parameter of the video to be processed.

[0086] The rendering parameter is used to indicate the time interval between video frames. The time interval can specifically refer to the rendering time interval or the playing time interval. Through the time interval between video frames, the computer device can determine whether the playing continuity of the video to be processed can be ensured after performing frame dropping processing on the first video frame.

[0087] In one embodiment, the rendering parameter includes a rendering time interval. The rendering time interval is calculated based on the time when the first video frame is decoded and the time when the third video frame is rendered; wherein, the third video frame is the video frame with the shortest time interval between the time when it is rendered and the time when the first video frame is decoded among the rendered video frames; for example, assuming that the time when video frame 1 is decoded is the 3rd second, and the rendered video frames include: video frame 2 is rendered at the 0.5th second, video frame 3 is rendered at the 1.1th second, video frame 4 is rendered at the 1.9th second, video frame 5 is rendered at the 2.1th second, and video frame 6 is rendered at the 2.7th second; then for video frame 1, the third video frame is video frame 6, and the rendering time interval = 3 - 2.7 = 0.3 second. It should be noted that the numerical values and time units in the above example are only for illustration, and in actual applications, the unit of the rendering time interval can be milliseconds. Further, if the rendering time interval between the first video frame and the third video frame is less than or equal to the second time interval threshold (such as 50 ms), the computer device determines that the processed video can ensure playback continuity after performing frame dropping processing on the first video frame (i.e., the time interval between video frames meets the frame dropping condition for rendering). It should be noted that the second time interval threshold is used to ensure the continuity of the video, and the second time interval threshold can be a fixed value or calculated based on the target decoding frame rate corresponding to the video to be processed; for example, the second time interval threshold = 1000 / target decoding frame rate * 2 (indicating that the rendering frame rate of the processed video after frame dropping processing drops to half of the decoding frame rate of the video to be processed).

[0088] Optionally, the rendering parameter includes a playback time interval. The computer device can calculate the playback time interval based on the playback time of the first video frame in the video to be processed and the playback time of the target video frame in the video to be processed. If the playback time interval between the first video frame and the target video frame is less than or equal to a second time interval threshold (such as 50 ms), the computer device determines that the video to be processed can ensure playback continuity after performing rendering frame dropping processing on the first video frame (that is, the time interval between video frames meets the rendering frame dropping condition). Among them, the target video frame is the video frame with the shortest playback time interval in the rendered video frames in the video to be processed compared to the playback time of the first video frame in the video to be processed. For example, assume the playback time of video frame 1 in the video to be processed is the 2nd second, and the rendered video frames include: video frame 2 with a playback time of the 0.5th second in the video to be processed, video frame 3 with a playback time of the 1st second in the video to be processed, video frame 4 with a playback time of the 1.2th second in the video to be processed, and video frame 5 with a playback time of the 1.7th second in the video to be processed. Then, for video frame 1, the target video frame is video frame 5, and the playback time interval = 2 - 1.7 = 0.3 second. It should be noted that the numerical values and time units in the above example are only for illustration. In practical applications, the unit of the playback time interval can be milliseconds.

[0089] S203. If the speed change parameter indicates that the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed, and the time interval between the video frames indicated by the rendering parameter meets the rendering frame dropping condition, then perform rendering frame dropping processing on the first video frame.

[0090] In one implementation, the speed change parameter in the decoding queue corresponding to the video to be processed includes the number of frames sent, and the rendering parameter includes a rendering time interval. The rendering time interval is used to indicate the time interval between the time when the first video frame is decoded and the time when the third video frame is rendered. The third video frame is the video frame with the shortest time interval between the rendering time in the rendered video frames of the video to be processed and the time when the first video frame is decoded. If the number of frames sent is greater than or equal to a preset number of frames, and the rendering time interval is less than or equal to a second time interval threshold (that is, the time interval between video frames meets the rendering frame dropping condition), the computer device performs rendering frame dropping processing on the first video frame.

[0091] Optionally, for S consecutive video frames in the decoding queue whose writing time intervals are less than a first time interval threshold, where S is an integer greater than 1, the computer can retain any one of the S decoded video frames corresponding to these S video frames (such as the first video frame among the S video frames, the last video frame among the S video frames, etc.) for rendering, and discard the other S - 1 decoded video frames; the computer device can also discard all the S decoded video frames corresponding to these S video frames.

[0092] In an embodiment of the present application, in response to the decoding completion of the first video frame in the video to be processed, a speed change parameter in the decoding queue corresponding to the video to be processed is obtained. The speed change parameter is used to indicate whether the writing speed of the video frames in the decoding queue is greater than the video decoding speed. A rendering parameter of the video to be processed is obtained. The rendering parameter is used to indicate the time interval between video frames. If the speed change parameter indicates that the writing speed of the video frames in the decoding queue is greater than the video decoding speed, and the time interval between the video frames indicated by the rendering parameter meets the rendering frame dropping condition, then rendering frame dropping processing is performed on the first video frame. It can be seen that on the premise of ensuring the continuity of the video to be processed, by performing frame dropping processing on the decoded video frames, the video delay can be reduced.

[0093] Please refer to Figure 3 , Figure 3 which is a flowchart of another video processing method provided by an embodiment of the present application. This video processing method can be executed by a computer device, and the computer device can be the Figure 1a terminal device 101 shown in Figure 3 . As shown in

[0094] S301. In response to the decoding completion of the first video frame in the video to be processed, obtain the speed change parameter in the decoding queue corresponding to the video to be processed.

[0095] S302. Obtain the rendering parameter of the video to be processed.

[0096] For the specific implementation manners of step S301 and step S302, reference can be made to the implementation manners in Figure 2 step S201 and step S202, which will not be elaborated here.

[0097] In one implementation manner, if the speed change parameter indicates that the writing speed of the video frames in the decoding queue is greater than the video decoding speed (for example, the speed change parameter includes the number of frames sent, and the number of frames sent is greater than or equal to the preset number of frames), and the time interval between the video frames indicated by the rendering parameter meets the rendering frame dropping condition (for example, the rendering parameter includes the rendering time interval, and the rendering time interval is less than or equal to the second time interval threshold), then the computer device performs rendering frame dropping processing on the first video frame.

[0098] In another implementation manner, the speed change parameter in the decoding queue corresponding to the processed video includes the number of frames sent. If the number of frames sent is less than the preset number of frames, and the time interval between the video frames indicated by the rendering parameter meets the rendering frame dropping condition, then the computer device continues to execute step S303.

[0099] In yet another embodiment, the speed change parameter in the decoding queue corresponding to the processed video includes the number of frames sent. If the number of frames sent is less than a preset number of frames, and the time interval between the video frames indicated by the rendering parameter satisfies the rendering frame dropping condition, the computer device continues to execute step S306.

[0100] In yet another embodiment, if the speed change parameter indicates that the writing speed of the video frames in the decoding queue is less than or equal to the video frame decoding speed, the computer device may render the first video frame and output the first video frame.

[0101] In yet another embodiment, if the time interval between the video frames indicated by the rendering parameter does not satisfy the rendering frame dropping condition (for example, the rendering parameter includes a rendering time interval, and the rendering time interval is greater than the second time interval threshold), the computer device renders the first video frame and outputs the first video frame.

[0102] S303. Obtain cache parameters.

[0103] The cache parameter is used to indicate whether the cache resource corresponding to the decoding queue (that is, the input cache allocated by the computer device to the decoder) meets the decoding requirements. In one embodiment, the cache parameter includes the input cache time consumption of the second video frame, and the second video frame is the last video frame enqueued in the decoding queue; for example, assuming that the decoding queue includes video frames 1 - 5, and the enqueue order of video frames 1 - 5 is the same as the sequence number, then the second video frame refers to video frame 5. If the input cache time consumption of the second video frame is greater than the cache time consumption threshold (such as 10 ms), the computer device determines that the cache resource corresponding to the decoding queue does not meet the decoding requirements.

[0104] In one embodiment, the second video frame is transferred from the input cache to the decoding queue. The computer device obtains the time when the second video frame first requests to be added to the input cache, and the time when the second video frame is successfully transferred to the decoding queue, and calculates the input cache time consumption of the second video frame according to the time when the second video frame first requests to be added to the input cache and the time when the second video frame is successfully transferred to the decoding queue. Specifically, the input cache time consumption of the second video frame = the time when the second video frame is successfully transferred to the decoding queue - the time when the second video frame first requests to be added to the input cache.

[0105] S304. If the cache parameter indicates that the cache resource corresponding to the decoding queue does not meet the decoding requirements, determine the rendering frame dropping frequency based on the cache parameter.

[0106] In one embodiment, the cache parameter includes the input cache time of the second video frame, where the second video frame is the last video frame enqueued in the decoding queue. If the input cache time of the second video frame is greater than the cache time threshold (e.g., 10 ms), the computer device determines the rendering frame drop frequency based on the input cache time of the second video frame. Specifically, the computer device can determine the time interval to which the input cache time of the second video frame belongs, and determine the rendering frame drop frequency according to the corresponding relationship between the time interval and the frame drop frequency. In one implementation, the corresponding relationship between the time interval and the frame drop frequency can be as shown in Table 1 below:

[0107] Table 1

[0108] Time-consuming interval Frame-drop frequency Description (50ms, ∞) 1 / 2 Discard one video frame out of every two decoded video frames (40ms, 50ms] 1 / 3 Discard one video frame out of every three decoded video frames (30ms, 40ms] 1 / 4 Discard one video frame out of every four decoded video frames (20ms, 30ms] 1 / 5 Discard one video frame out of every five decoded video frames (10ms, 20ms] 1 / 6 Discard one video frame out of every six decoded video frames

[0109] As can be seen from Table 1, different time intervals correspond to different frame drop frequencies, and the frame drop frequency is proportional to the input cache time. It can be understood that the time intervals and frame drop frequencies in Table 1 are only for illustration and do not constitute the actual limitations of this application.

[0110] S305. Perform rendering frame drop processing on the video frames decoded within the target time period according to the rendering frame drop frequency.

[0111] The target time period can be a preset fixed time period or a time period dynamically determined by the terminal device based on relevant parameters; for example, assume the fourth video frame is the video frame after the second video frame. When the fourth video frame is added to the decoding queue, if the computer device detects that the input cache time of the fourth video frame is less than or equal to the cache time threshold, the computer device can determine the time period from when the second video frame is added to the decoding queue to when the fourth video frame is added to the decoding queue as the target time period.

[0112] Further, after the target time period, the computer device can obtain the input cache time of the video frame (e.g., the fourth video frame) that is the last one in the current decoding queue in the decoding (enqueue) order. The fourth video frame is the video frame after the second video frame. If the computer device detects that the input cache time of the fourth video frame is less than or equal to the cache time threshold, the computer device can stop performing rendering frame drop processing on the decoded video frames; correspondingly, if the computer device detects that the input cache time of the fourth video frame is greater than the cache time threshold, the computer device can determine the rendering frame drop frequency corresponding to the next time period after the target time period according to the implementation in steps S304 - S305, and perform rendering frame drop processing on the video frames decoded within the next time period after the target time period according to the rendering frame drop frequency.

[0113] Accordingly, after obtaining the cache parameter, if the cache parameter indicates that the cache resource corresponding to the decoding queue meets the decoding requirement (for example, the input cache time of the second video frame is less than or equal to the cache time threshold), the computer device renders the first video frame and outputs the first video frame (that is, no rendering frame dropping processing is required).

[0114] S306. Obtain the decoding performance parameter.

[0115] The decoding performance parameter is used to indicate whether the decoding performance of the decoder meets the decoding requirement. In one implementation, the decoding performance parameter includes the decoding backlog number at the first moment and the decoding backlog number threshold at the first moment. The decoding backlog number at the first moment is used to indicate the number of video frames in the decoding queue at the first moment. The decoding backlog number threshold at the first moment is determined based on the decoding performance of the decoder at the first moment. It should be noted that the decoding performance of the decoder may fluctuate over time, so the decoding backlog number thresholds at different moments may be different. It can be understood that the decoding performance is proportional to the decoding backlog number threshold; that is, the higher the decoding performance of the decoder, the larger the decoding backlog number threshold. If the decoding backlog number at the first moment is greater than the decoding backlog number threshold at the first moment, the computer device determines that the decoding performance of the decoder at the first moment does not meet the decoding requirement.

[0116] S307. If the decoding performance parameter indicates that the decoding performance of the decoder does not meet the decoding requirement, determine the rendering frame dropping frequency as the first frequency.

[0117] In one implementation, the decoding performance parameter includes the decoding backlog number at the first moment and the decoding backlog number threshold at the first moment. If the decoding backlog number at the first moment is greater than the decoding backlog number threshold at the first moment, the computer device determines the rendering frame dropping frequency as the first frequency. The first frequency can be a preset frequency or calculated based on the difference between the decoding backlog number at the first moment and the decoding backlog number threshold at the first moment. The first frequency is proportional to the difference between the decoding backlog number at the first moment and the decoding backlog number threshold at the first moment; that is, the more the decoding backlog number at the first moment exceeds the decoding backlog number threshold at the first moment, the larger the first frequency.

[0118] S308. Perform rendering frame dropping processing on the video frames decoded within the first time period according to the first frequency.

[0119] The first time period can be a preset fixed time period or a time period dynamically determined by the terminal device based on relevant parameters; for example, assuming that at the second moment, the computer device detects that the decoding backlog number at the second moment is less than or equal to the decoding backlog number threshold at the second moment, the first time period can be the time period corresponding to the first moment to the second moment.

[0120] Furthermore, the decoding performance parameter also includes the number of decoded frames at the second moment and the threshold of the number of decoded frames at the second moment. If the number of decoded frames at the second moment is greater than the threshold of the number of decoded frames at the second moment, the computer device determines the rendering frame loss frequency as the second frequency, and performs rendering frame loss processing on the video frames decoded in the second time period according to the second frequency, the second frequency is greater than the first frequency, and the second time period is after the first time period. According to the above method, if after the rendering frame loss processing is performed, the decoding performance parameter still indicates that the decoding performance of the decoder does not meet the decoding requirements, the computer device can further increase the frequency of rendering frame loss until the frequency of rendering frame loss reaches the frequency threshold (which can be determined by the rendering parameters of the video to be processed).

[0121] Correspondingly, if the number of decoded frames at the second moment is less than or equal to the threshold of the number of decoded frames at the second moment, the computer device determines the rendering frame loss frequency as the third frequency, and performs rendering frame loss processing on the video frames decoded in the second time period according to the third frequency; the third frequency is less than the first frequency, and the second time period is after the first time period. According to the above method, if after the rendering frame loss processing is performed, the decoding performance parameter indicates that the decoding performance of the decoder can meet the decoding requirements, the computer device can gradually reduce the rendering frame loss frequency until the rendering frame loss frequency drops to zero; or the rendering frame loss frequency can be directly reduced to zero (that is, the third frequency can be 0).

[0122] The embodiments of this application are Figure 2 On the basis of the embodiment, on the one hand, by sending the number of frames, it is judged whether the video frame writing speed of the decoding queue is greater than the video frame decoding speed. When it is detected that the video frame writing speed of the decoding queue is greater than the video frame decoding speed (that is, the number of sent frames is greater than or equal to the preset number of frames), the video delay is reduced by rendering the frame loss; on the other hand, by inputting the cache time consumption, it is judged whether the cache resources corresponding to the decoding queue meet the decoding requirements. When it is detected that the cache resources corresponding to the decoding queue do not meet the decoding requirements (that is, the input cache time consumption of the video frame is greater than the cache time consumption threshold), the video frame that has been decoded is rendered and lost by rendering the frame loss frequency to reduce the video delay; on the other hand, by decoding the number of hoarded frames, it is judged whether the decoding performance of the decoder meets the decoding requirements. When it is detected that the decoding performance of the decoder does not meet the decoding requirements (such as the number of decoded hoarded frames at the current moment is greater than the decoding hoarded frame threshold at the current moment), the video frame that has been decoded is rendered and lost according to the set frequency to reduce the video delay. In addition, while reducing the video delay, the present application also ensures the continuity of the video by rendering the time interval, further improving the video interaction experience.

[0123] The method of the embodiments of the present application is elaborated in detail above. To facilitate better implementation of the above solutions of the embodiments of the present application, correspondingly, the devices of the embodiments of the present application are provided below.

[0124] Please refer to Figure 4 , Figure 4 which is a schematic structural diagram of a video processing device provided by an embodiment of the present application. Figure 4 The shown video processing device can be mounted in a computer device, and the computer device can specifically be Figure 1a the terminal device 101 shown. The video processing device can be used to execute some or all of the functions in the method embodiments described above Figure 2 and Figure 3 . Please refer to Figure 4 . The video processing device includes:

[0125] An acquisition unit 401, configured to, in response to the decoding of the first video frame in the video to be processed being completed, acquire the speed change parameter in the decoding queue corresponding to the video to be processed, where the speed change parameter is used to indicate whether the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed;

[0126] and is further configured to acquire the rendering parameter of the video to be processed, where the rendering parameter is used to indicate the time interval between video frames;

[0127] A processing unit 402, configured to, if the speed change parameter indicates that the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed, and the time interval between the video frames indicated by the rendering parameter meets the rendering frame dropping condition, perform rendering frame dropping processing on the first video frame.

[0128] In an implementation manner, the speed change parameter in the decoding queue corresponding to the video to be processed includes the number of frames sent, where the number of frames sent is used to indicate the number of video frames in the decoding queue whose writing time interval is less than the first time interval threshold; the processing unit 402 is further configured to:

[0129] If the number of frames sent is greater than or equal to the preset number of frames, it is determined that the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed.

[0130] In an implementation manner, the processing unit 402 is further configured to:

[0131] If the number of frames sent is less than the preset number of frames, and the time interval between the video frames indicated by the rendering parameter meets the rendering frame dropping condition, acquire the buffer parameter, where the buffer parameter is used to indicate whether the buffer resource corresponding to the decoding queue meets the decoding requirement;

[0132] If the buffer parameter indicates that the buffer resource corresponding to the decoding queue does not meet the decoding requirement, determine the rendering frame dropping frequency based on the buffer parameter;

[0133] Perform rendering frame dropping processing on the video frames decoded within the target time period according to the rendering frame dropping frequency.

[0134] In one embodiment, the cache parameter includes the input cache time of the second video frame, and the second video frame is the last video frame enqueued in the decoding queue; the processing unit 402 is further configured to:

[0135] If the input cache time of the second video frame is greater than the cache time threshold, it is determined that the cache resources corresponding to the decoding queue do not meet the decoding requirements.

[0136] In one embodiment, the cache parameter includes the input cache time of the second video frame, and the second video frame is the last video frame enqueued in the decoding queue; the second video frame is transferred from the input cache to the decoding queue; the processing unit 402 is configured to obtain the cache parameter, specifically:

[0137] Obtain the time when the second video frame first requests to be added to the input cache and the time when the second video frame is successfully transferred to the decoding queue;

[0138] Calculate the input cache time of the second video frame according to the time when the second video frame first requests to be added to the input cache and the time when the second video frame is successfully transferred to the decoding queue.

[0139] In one embodiment, the cache parameter includes the input cache time of the second video frame, and the second video frame is the last video frame enqueued in the decoding queue; the processing unit 402 is configured to determine the rendering frame dropping frequency based on the cache parameter, specifically:

[0140] Determine the time interval to which the input cache time of the second video frame belongs, and different time intervals correspond to different frame dropping frequencies;

[0141] Determine the frame dropping frequency corresponding to the time interval to which the input cache time of the second video frame belongs as the rendering frame dropping frequency.

[0142] In one embodiment, the processing unit 402 is further configured to:

[0143] If the cache parameter indicates that the cache resources corresponding to the decoding queue meet the decoding requirements, render the first video frame and output the first video frame.

[0144] In one embodiment, the processing unit 402 is further configured to:

[0145] If the number of frames sent is less than or equal to the preset number of frames and the time interval between the video frames indicated by the rendering parameter meets the rendering frame dropping condition, obtain the decoding performance parameter, and the decoding performance parameter is used to indicate whether the decoding performance of the decoder meets the decoding requirements;

[0146] If the decoding performance parameter indicates that the decoding performance of the decoder does not meet the decoding requirement, the rendering frame drop frequency is determined as the first frequency;

[0147] Perform rendering frame drop processing on the video frames decoded within the first time period according to the first frequency.

[0148] In one implementation, the decoding performance parameter includes the number of decoded frames in the decoding queue at the first moment and the threshold value of the number of decoded frames at the first moment. The number of decoded frames at the first moment is used to indicate the number of video frames in the decoding queue at the first moment, and the threshold value of the number of decoded frames at the first moment is determined based on the decoding performance of the decoder at the first moment; the processing unit 402 is further configured to:

[0149] If the number of decoded frames in the decoding queue at the first moment is greater than the threshold value of the number of decoded frames at the first moment, it is determined that the decoding performance of the decoder at the first moment does not meet the decoding requirement.

[0150] In one implementation, the decoding performance parameter further includes the number of decoded frames in the decoding queue at the second moment and the threshold value of the number of decoded frames at the second moment, and the second moment is after the first moment; the processing unit 402 is further configured to:

[0151] If the number of decoded frames in the decoding queue at the second moment is greater than the threshold value of the number of decoded frames at the second moment, the rendering frame drop frequency is determined as the second frequency, and the second frequency is greater than the first frequency;

[0152] Perform rendering frame drop processing on the video frames decoded within the second time period according to the second frequency, and the second time period is after the first time period.

[0153] In one implementation, the processing unit 402 is further configured to:

[0154] If the number of decoded frames in the decoding queue at the second moment is less than or equal to the threshold value of the number of decoded frames at the second moment, the rendering frame drop frequency is determined as the third frequency, and the third frequency is less than the first frequency;

[0155] Perform rendering frame drop processing on the video frames decoded within the second time period according to the third frequency.

[0156] In one implementation, the decoding queue contains M video frames, where M is an integer greater than 1; the processing unit 402 is configured to obtain the speed change parameter in the decoding queue corresponding to the video to be processed, specifically:

[0157] Obtain the time when the M video frames are written into the decoding queue;

[0158] Based on the time when the M video frames are written into the decoding queue, calculate the write time interval between adjacent video frames among the M video frames;

[0159] According to the writing time intervals between adjacent video frames in M video frames and the first time interval threshold, count the number of frames sent.

[0160] In one embodiment, the processing unit 402 is further configured to:

[0161] If the speed change parameter indicates that the writing speed of the video frames in the decoding queue is less than or equal to the video frame decoding speed, render the first video frame and output the first video frame; or,

[0162] If the time interval between the video frames indicated by the rendering parameter does not meet the rendering frame-drop condition, render the first video frame and output the first video frame.

[0163] In one embodiment, the rendering parameter of the video to be processed includes a rendering time interval, and the rendering time interval is used to indicate the time interval between the time when the first video frame is decoded and the time when the third video frame is rendered. The third video frame is the video frame with the shortest time interval between the rendering time and the time when the first video frame is decoded among the rendered video frames of the video to be processed; the time interval between the video frames meeting the rendering frame-drop condition includes that the rendering time interval is less than or equal to the second time interval threshold.

[0164] According to an embodiment of the present application, Figure 2 and Figure 3 Some of the steps involved in the video processing method shown can be executed by each unit in the Figure 4 shown video processing apparatus. For example, Figure 2 The steps S201 and S202 shown in Figure 4 can be executed by the obtaining unit 401 shown in Figure 2 The step S203 shown in Figure 4 can be executed by the processing unit 402 shown in Figure 3 The steps S301 - S303 and S306 shown in Figure 4 can be executed by the obtaining unit 401 shown in Figure 4 The steps S304, S305, S307, and S308 can be executed by the processing unit 402 shown in Figure 4Each unit in the video processing device shown can be separately or entirely combined into one or several other units to form, or a certain one (or some) of the units can be further split into multiple smaller units with more specific functions to form. This can achieve the same operations without affecting the realization of the technical effects of the embodiments of this application. The above units are divided based on logical functions. In practical applications, the function of one unit can also be realized by multiple units, or the functions of multiple units can be realized by one unit. In other embodiments of this application, the video processing device can also include other units. In practical applications, these functions can also be assisted by other units and can be realized through the cooperation of multiple units.

[0165] According to another embodiment of this application, it can be achieved by running a computer program (including program code) that can execute the respective steps involved in the corresponding methods shown in Figure 2 and Figure 3 on a general computing device such as a computer device including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM), to construct a video processing device as shown in Figure 4 and to implement the video processing method of the embodiments of this application. The computer program can be recorded on, for example, a computer-readable recording medium, loaded into the above computing device through the computer-readable recording medium, and run therein.

[0166] Based on the same inventive concept, the principle of problem-solving and the beneficial effects of the video processing device provided in the embodiments of this application are similar to those of the video processing method in the method embodiments of this application. The principle and beneficial effects of the method implementation can be referred to. For the sake of brevity, they will not be elaborated here.

[0167] Please refer to Figure 5 , Figure 5 which is a schematic structural diagram of a computer device provided in the embodiments of this application. This computer device can be a terminal device or a server. As Figure 5As shown in the figure, the computer device at least includes a processor 501, a communication interface 502, and a memory 503. Among them, the processor 501, the communication interface 502, and the memory 503 can be connected through a bus or other means. Among them, the processor 501 (or the Central Processing Unit (CPU)) is the computing core and control core of the computer device. It can parse various instructions in the computer device and process various data of the computer device. For example, the CPU can be used to parse the power-on and power-off instructions sent by an object to the computer device and control the computer device to perform power-on and power-off operations; Another example is that the CPU can transmit various interactive data between the internal structures of the computer device, and so on. The communication interface 502 can optionally include standard wired interfaces, wireless interfaces (such as WI-FI, mobile communication interfaces, etc.). Controlled by the processor 501, it can be used to send and receive data; the communication interface 502 can also be used for the transmission and interaction of internal data of the computer device. The memory 503 (Memory) is the memory device in the computer device and is used to store programs and data. It can be understood that the memory 503 here can include both the built-in memory of the computer device and, of course, the extended memory supported by the computer device. The memory 503 provides a storage space, and this storage space stores the operating system of the computer device, which can include but is not limited to: Android system, Internetworking Operating System (IOS), etc. This application does not make any limitations in this regard.

[0168] The embodiment of this application also provides a computer-readable storage medium (Memory). The computer-readable storage medium is the memory device in the computer device and is used to store programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and, of course, the extended storage medium supported by the computer device. The computer-readable storage medium provides a storage space, and this storage space stores the processing system of the computer device. And, a computer program suitable for being loaded and executed by the processor 501 is also stored in this storage space. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory; optionally, it can also be at least one computer-readable storage medium located far from the aforementioned processor.

[0169] In one embodiment, the processor 501 executes the following operations by running the computer program in the memory 503:

[0170] In response to the completion of decoding of the first video frame in the video to be processed, obtain the speed change parameter in the decoding queue corresponding to the video to be processed, where the speed change parameter is used to indicate whether the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed;

[0171] Obtain the rendering parameter of the video to be processed, where the rendering parameter is used to indicate the time interval between video frames;

[0172] If the speed change parameter indicates that the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed, and the time interval between the video frames indicated by the rendering parameter meets the rendering frame dropping condition, then perform rendering frame dropping processing on the first video frame.

[0173] As an optional embodiment, the speed change parameter in the decoding queue corresponding to the video to be processed includes the number of frames sent, where the number of frames sent is used to indicate the number of video frames in the decoding queue whose writing time interval is less than the first time interval threshold;

[0174] The processor 501 also performs the following operations by running the computer program in the memory 503:

[0175] If the number of frames sent is greater than or equal to the preset number of frames, it is determined that the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed.

[0176] As an optional embodiment, the processor 501 also performs the following operations by running the computer program in the memory 503:

[0177] If the number of frames sent is less than the preset number of frames, and the time interval between the video frames indicated by the rendering parameter meets the rendering frame dropping condition, then obtain the cache parameter, where the cache parameter is used to indicate whether the cache resource corresponding to the decoding queue meets the decoding requirement;

[0178] If the cache parameter indicates that the cache resource corresponding to the decoding queue does not meet the decoding requirement, then determine the rendering frame dropping frequency based on the cache parameter;

[0179] Perform rendering frame dropping processing on the video frames decoded within the target time period according to the rendering frame dropping frequency.

[0180] As an optional embodiment, the cache parameter includes the input cache time of the second video frame, where the second video frame is the last video frame enqueued in the decoding queue; the processor 501 also performs the following operations by running the computer program in the memory 503:

[0181] If the input cache time of the second video frame is greater than the cache time threshold, it is determined that the cache resource corresponding to the decoding queue does not meet the decoding requirement.

[0182] As an alternative embodiment, the cache parameter includes the input cache time of the second video frame, where the second video frame is the last video frame enqueued in the decoding queue; the second video frame is transferred from the input cache to the decoding queue; a specific embodiment for the processor 501 to obtain the cache parameter is as follows:

[0183] Obtain the time when the second video frame first requests to be added to the input cache, and the time when the second video frame is successfully transferred to the decoding queue;

[0184] Calculate the input cache time of the second video frame according to the time when the second video frame first requests to be added to the input cache and the time when the second video frame is successfully transferred to the decoding queue.

[0185] As an alternative embodiment, the cache parameter includes the input cache time of the second video frame, where the second video frame is the last video frame enqueued in the decoding queue; a specific embodiment for the processor 501 to determine the rendering frame drop frequency based on the cache parameter is as follows:

[0186] Determine the time interval to which the input cache time of the second video frame belongs, and different time intervals correspond to different frame drop frequencies;

[0187] Determine the frame drop frequency corresponding to the time interval to which the input cache time of the second video frame belongs as the rendering frame drop frequency.

[0188] As an alternative embodiment, the processor 501 further performs the following operations by running the computer program in the memory 503:

[0189] If the cache parameter indicates that the cache resource corresponding to the decoding queue meets the decoding requirement, then render the first video frame and output the first video frame.

[0190] As an alternative embodiment, the processor 501 further performs the following operations by running the computer program in the memory 503:

[0191] If the number of frames sent is less than or equal to the preset number of frames, and the time interval between the video frames indicated by the rendering parameter meets the rendering frame drop condition, then obtain the decoding performance parameter, where the decoding performance parameter is used to indicate whether the decoding performance of the decoder meets the decoding requirement;

[0192] If the decoding performance parameter indicates that the decoding performance of the decoder does not meet the decoding requirement, then determine the rendering frame drop frequency as the first frequency;

[0193] Perform rendering frame drop processing on the video frames decoded within the first time period according to the first frequency.

[0194] As an alternative embodiment, the decoding performance parameter includes the number of buffered decoded video frames at a first moment and the threshold of the number of buffered decoded video frames at the first moment. The number of buffered decoded video frames at the first moment is used to indicate the number of video frames in the decoding queue at the first moment, and the threshold of the number of buffered decoded video frames at the first moment is determined based on the decoding performance of the decoder at the first moment;

[0195] The processor 501 further performs the following operations by running the computer program in the memory 503:

[0196] If the number of buffered decoded video frames at the first moment is greater than the threshold of the number of buffered decoded video frames at the first moment, it is determined that the decoding performance of the decoder at the first moment does not meet the decoding requirement.

[0197] As an alternative embodiment, the decoding performance parameter further includes the number of buffered decoded video frames at a second moment and the threshold of the number of buffered decoded video frames at the second moment, where the second moment is after the first moment; the processor 501 further performs the following operations by running the computer program in the memory 503:

[0198] If the number of buffered decoded video frames at the second moment is greater than the threshold of the number of buffered decoded video frames at the second moment, the rendering frame drop frequency is determined to be a second frequency, and the second frequency is greater than the first frequency;

[0199] Perform rendering frame dropping processing on the video frames decoded within a second time period according to the second frequency, where the second time period is after the first time period.

[0200] As an alternative embodiment, the processor 501 further performs the following operations by running the computer program in the memory 503:

[0201] If the number of buffered decoded video frames at the second moment is less than or equal to the threshold of the number of buffered decoded video frames at the second moment, the rendering frame drop frequency is determined to be a third frequency, and the third frequency is less than the first frequency;

[0202] Perform rendering frame dropping processing on the video frames decoded within a second time period according to the third frequency.

[0203] As an alternative embodiment, the decoding queue contains M video frames, where M is an integer greater than 1; a specific embodiment of the processor 501 for obtaining the speed change parameter in the decoding queue corresponding to the video to be processed is:

[0204] Obtain the time when M video frames are written into the decoding queue;

[0205] Based on the time when M video frames are written into the decoding queue, calculate the write time interval between adjacent video frames among the M video frames;

[0206] According to the write time interval between adjacent video frames among the M video frames and the first time interval threshold, count the number of dropped frames.

[0207] As an alternative embodiment, the processor 501 further performs the following operations by running the computer program in the memory 503:

[0208] If the speed change parameter indicates that the writing speed of the video frames in the decoding queue is less than or equal to the video frame decoding speed, then render the first video frame and output the first video frame; or,

[0209] If the time interval between the video frames indicated by the rendering parameter does not meet the rendering frame dropping condition, then render the first video frame and output the first video frame.

[0210] As an alternative embodiment, the rendering parameter of the video to be processed includes a rendering time interval, and the rendering time interval is used to indicate the time interval between the time when the first video frame is decoded and the time when the third video frame is rendered. The third video frame is the video frame with the shortest time interval between the time when it is rendered and the time when the first video frame is decoded among the rendered video frames of the video to be processed; the time interval between the video frames meeting the rendering frame dropping condition includes that the rendering time interval is less than or equal to the second time interval threshold.

[0211] Based on the same inventive concept, the principle of problem-solving and the beneficial effects of the computer device provided in the embodiments of the present application are similar to the principle of problem-solving and the beneficial effects of the video processing method in the method embodiments of the present application. The principle and beneficial effects of the method implementation can be referred to. For the sake of brief description, they will not be elaborated here.

[0212] The embodiments of the present application further provide a computer-readable storage medium, in which a computer program is stored, and the computer program is adapted to be loaded and executed by a processor to perform the video processing method in the above method embodiments.

[0213] The embodiments of the present application further provide a computer program product, which includes a computer program, and the computer program is adapted to be loaded and executed by a processor to perform the video processing method in the above method embodiments.

[0214] The embodiments of the present application further provide a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above video processing method.

[0215] The steps in the method embodiments of the present application can be adjusted, combined, and deleted according to actual needs.

[0216] The modules in the device embodiments of the present application can be combined, divided, and deleted according to actual needs.

[0217] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. The readable storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, etc.

[0218] The above-disclosed is only a preferred embodiment of the present application. Of course, it cannot be used to limit the scope of rights of the present application. Those of ordinary skill in the art can understand all or part of the processes of implementing the above embodiments, and the equivalent changes made according to the claims of the present application still fall within the scope covered by the application.

Claims

1. A video processing method, characterized in that, The method includes: In response to the completion of decoding of the first video frame in the video to be processed, obtaining a speed change parameter in the decoding queue corresponding to the video to be processed, where the speed change parameter is used to indicate whether the writing speed of video frames in the decoding queue is greater than the video frame decoding speed; the speed change parameter includes the number of frames sent, and the number of frames sent is used to indicate the number of video frames in the decoding queue with a writing time interval less than a first time interval threshold; Obtaining the rendering parameter of the video to be processed, where the rendering parameter is used to indicate the time interval between video frames; judging the playback continuity of the video to be processed after performing rendering frame dropping processing on the first video frame based on the time interval between video frames; If the speed change parameter indicates that the writing speed of video frames in the decoding queue is greater than the video frame decoding speed, and the time interval between video frames indicated by the rendering parameter meets the rendering frame dropping condition, then perform rendering frame dropping processing on the first video frame; If the number of frames sent is less than a preset number of frames, and the time interval between video frames indicated by the rendering parameter meets the rendering frame dropping condition, then obtain a cache parameter, where the cache parameter is used to indicate whether the cache resources corresponding to the decoding queue meet the decoding requirements; If the cache parameter indicates that the cache resources corresponding to the decoding queue do not meet the decoding requirements, then determine the rendering frame dropping frequency based on the cache parameter; Perform rendering frame dropping processing on the video frames decoded within a target time period according to the rendering frame dropping frequency.

2. The method according to claim 1, wherein The method further includes: If the number of frames sent is greater than or equal to the preset number of frames, then determine that the writing speed of video frames in the decoding queue is greater than the video frame decoding speed.

3. The method according to claim 1, characterized in that The cache parameter includes the input cache time consumption of a second video frame, where the second video frame is the last video frame enqueued in the decoding queue; the method further includes: If the input cache time consumption of the second video frame is greater than a cache time consumption threshold, then determine that the cache resources corresponding to the decoding queue do not meet the decoding requirements.

4. The method according to claim 1, wherein The cache parameter includes the input cache time consumption of a second video frame, where the second video frame is the last video frame enqueued in the decoding queue; the second video frame is transferred from the input cache to the decoding queue, and obtaining the cache parameter includes: Obtaining the time when the second video frame first requests to be added to the input cache, and the time when the second video frame is successfully transferred to the decoding queue; Calculating the input cache time consumption of the second video frame according to the time when the second video frame first requests to be added to the input cache and the time when the second video frame is successfully transferred to the decoding queue.

5. The method according to claim 1, wherein The cache parameter includes the input cache time consumption of a second video frame, where the second video frame is the last video frame enqueued in the decoding queue; determining the rendering frame dropping frequency based on the cache parameter includes: Determining the time consumption interval to which the input cache time consumption of the second video frame belongs, and different time consumption intervals correspond to different frame dropping frequencies; Determining the frame dropping frequency corresponding to the time consumption interval to which the input cache time consumption of the second video frame belongs as the rendering frame dropping frequency.

6. The method according to claim 1, characterized in that, The method further includes: If the cache parameter indicates that the cache resource corresponding to the decoding queue meets the decoding requirement, render the first video frame and output the first video frame.

7. The method according to claim 2, characterized in that, The method further includes: If the number of frames sent is less than or equal to the preset number of frames, and the time interval between the video frames indicated by the rendering parameter meets the rendering frame-drop condition, obtain a decoding performance parameter, where the decoding performance parameter is used to indicate whether the decoding performance of the decoder meets the decoding requirement; If the decoding performance parameter indicates that the decoding performance of the decoder does not meet the decoding requirement, determine the rendering frame-drop frequency as the first frequency; Perform rendering frame-drop processing on the video frames decoded within the first time period according to the first frequency.

8. The method according to claim 7, wherein The decoding performance parameter includes the number of decoded frames in the decoding queue at the first moment and the threshold of the number of decoded frames in the decoding queue at the first moment. The number of decoded frames in the decoding queue at the first moment is used to indicate the number of video frames in the decoding queue at the first moment, and the threshold of the number of decoded frames in the decoding queue at the first moment is determined based on the decoding performance of the decoder at the first moment; the method further includes: If the number of decoded frames in the decoding queue at the first moment is greater than the threshold of the number of decoded frames in the decoding queue at the first moment, determine that the decoding performance of the decoder at the first moment does not meet the decoding requirement.

9. The method according to claim 8, wherein The decoding performance parameter further includes the number of decoded frames in the decoding queue at the second moment and the threshold of the number of decoded frames in the decoding queue at the second moment, where the second moment is after the first moment; the method further includes: If the number of decoded frames in the decoding queue at the second moment is greater than the threshold of the number of decoded frames in the decoding queue at the second moment, determine the rendering frame-drop frequency as the second frequency, and the second frequency is greater than the first frequency; Perform rendering frame-drop processing on the video frames decoded within the second time period according to the second frequency, where the second time period is after the first time period.

10. The method according to claim 9, characterized in that, The method further includes: If the number of decoded frames in the decoding queue at the second moment is less than or equal to the threshold of the number of decoded frames in the decoding queue at the second moment, determine the rendering frame-drop frequency as the third frequency, and the third frequency is less than the first frequency; Perform rendering frame-drop processing on the video frames decoded within the second time period according to the third frequency.

11. The method according to claim 2, wherein The decoding queue contains M video frames, where M is an integer greater than 1; obtaining the speed change parameter in the decoding queue corresponding to the video to be processed includes: Obtain the time when the M video frames are written into the decoding queue; Based on the time when the M video frames are written into the decoding queue, calculate the write time interval between adjacent video frames among the M video frames; According to the write time interval between adjacent video frames among the M video frames and the first time interval threshold, count the number of frames sent.

12. The method according to claim 1, wherein The method further includes: If the speed change parameter indicates that the write speed of the video frames in the decoding queue is less than or equal to the video frame decoding speed, render the first video frame and output the first video frame; or, If the time interval between the video frames indicated by the rendering parameter does not meet the rendering frame-drop condition, render the first video frame and output the first video frame.

13. The method according to any one of claims 1 to 12, characterized in that, The rendering parameters of the video to be processed include a rendering time interval, which is used to indicate the time interval between the time when the first video frame is decoded and the time when the third video frame is rendered. The third video frame is the video frame with the shortest time interval between the rendering time and the decoding completion time of the first video frame among the rendered video frames of the video to be processed; the time interval between video frames satisfying the rendering frame-drop condition includes that the rendering time interval is less than or equal to a second time interval threshold.

14. A video processing device, characterized in that, The video processing device includes: An acquisition unit, configured to, in response to the decoding completion of the first video frame in the video to be processed, acquire a speed change parameter in the decoding queue corresponding to the video to be processed, where the speed change parameter is used to indicate whether the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed; the speed change parameter includes the number of frames sent, which is used to indicate the number of video frames with a writing time interval less than a first time interval threshold in the decoding queue; and is configured to acquire the rendering parameters of the video to be processed, where the rendering parameters are used to indicate the time interval between video frames; and determine the playback continuity of the video to be processed after performing rendering frame-drop processing on the first video frame based on the time interval between video frames; A processing unit, configured to, if the speed change parameter indicates that the writing speed of the video frames in the decoding queue is greater than the video frame decoding speed, and the time interval between video frames indicated by the rendering parameters satisfies the rendering frame-drop condition, perform rendering frame-drop processing on the first video frame; The processing unit is further configured to, if the number of frames sent is less than a preset number of frames, and the time interval between video frames indicated by the rendering parameters satisfies the rendering frame-drop condition, acquire a cache parameter, where the cache parameter is used to indicate whether the cache resources corresponding to the decoding queue meet the decoding requirements; if the cache parameter indicates that the cache resources corresponding to the decoding queue do not meet the decoding requirements, determine a rendering frame-drop frequency based on the cache parameter; and perform rendering frame-drop processing on the video frames decoded within a target time period according to the rendering frame-drop frequency.

15. A computer device, characterized in that, including: a memory and a processor; The memory, in which a computer program is stored; The processor is configured to load the computer program to implement the video processing method according to any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by the processor to implement the video processing method according to any one of claims 1-13.

Images