Video rendering control method and device, equipment and storage medium
By calculating the actual arrival time, expected ready time, and drop rate of video frames, the video rendering decision is optimized, solving the stuttering and jitter problems in the video rendering process and improving the user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI ZULE INFORMATION TECHNOLOGY CO LTD
- Filing Date
- 2026-01-21
- Publication Date
- 2026-04-28
AI Technical Summary
Existing technologies fail to effectively handle network fluctuations and latency changes during video rendering, resulting in stuttering, frame skipping, and sudden increases in latency, leading to a poor user experience.
By determining the actual arrival time, expected ready time, and drop ratio of video frames, the target maximum latency and current latency are calculated. Rendering decisions are then made in conjunction with the rendering time to optimize the rendering time and drop strategy of video frames.
It effectively reduces stuttering and jitter during video rendering, improving the user's subjective experience and video rendering effect.
Smart Images
Figure CN121940587A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of video processing technology, and in particular to a video rendering control method, apparatus, device, and storage medium. Background Technology
[0002] In scenarios requiring low latency and smooth performance, such as video calls, remote desktop control, cloud gaming, remote monitoring, and AR / VR streaming, real-time video systems primarily address network jitter and latency fluctuations through methods like bitrate control at the transmitting end and buffer adjustment at the receiving end to maintain image continuity and acceptable latency. However, most of these solutions focus on control during the transmission or decoding phases, neglecting decision-making and processing during the rendering phase.
[0003] Traditional rendering mechanisms often employ strategies such as fixed latency or simply displaying the latest frame. When network fluctuations, decoding rate changes, or rendering time fluctuations occur, stuttering, frame skipping, or sudden increases in latency are likely to occur, resulting in poor video rendering quality and a decline in the user's subjective experience (QoE). Summary of the Invention
[0004] This application provides a video rendering control method, apparatus, device, and storage medium to solve the technical problem in related technologies where poor video rendering results lead to a decline in user subjective experience. It can effectively improve video rendering results and enhance user subjective experience.
[0005] In a first aspect, embodiments of this application provide a video rendering control method, including: Determine the actual arrival time, expected ready time, and video frame drop ratio of multiple video frames to be rendered within a preset time window, and determine the target maximum delay based on the actual arrival time, the expected ready time, and the video frame drop ratio; The current delay time is determined based on the previously determined delay time and the target maximum delay. The target rendering time of the video frame to be rendered is obtained by adding the current delay time and the expected ready time. Based on the current time and the target rendering time, a rendering decision is made for the queue of frames to be rendered.
[0006] In a second aspect, embodiments of this application provide a video rendering control device, including a maximum delay module, a current delay module, a rendering time module, and a rendering decision module, wherein: The maximum delay module is used to determine the actual arrival time, expected ready time and video frame drop ratio of multiple video frames to be rendered within a preset time window, and to determine the target maximum delay based on the actual arrival time, the expected ready time and the video frame drop ratio. The current delay module is used to determine the current delay time based on the previously determined delay time and the target maximum delay. The rendering time module is used to add the current delay time and the expected ready time to obtain the target rendering time of the video frame to be rendered. The rendering decision module is used to perform rendering decision processing on the queue of frames to be rendered based on the current time and the target rendering time.
[0007] In a third aspect, embodiments of this application provide a video rendering control device, including: a memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video rendering control method as described in the first aspect.
[0008] In a fourth aspect, embodiments of this application provide a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the video rendering control method as described in the first aspect.
[0009] This application embodiment determines the actual arrival time, expected ready time, and video frame drop ratio of multiple video frames to be rendered within a preset time window. It then determines the target maximum delay based on the actual arrival time, expected ready time, and video frame drop ratio. The current delay time is determined based on the previously determined delay time and the target maximum delay. The current delay time and expected ready time are added together to obtain the target rendering time for the video frames to be rendered. Rendering decision processing is then performed on the queue of frames to be rendered based on the current time and the target rendering time. By making rendering decisions at the rendering level based on the actual arrival time, expected ready time, and video frame drop ratio of the video frames, the video rendering effect can be effectively improved, enhancing the user's subjective experience. Attached Figure Description
[0010] Figure 1 This is a flowchart of a video rendering control method provided in an embodiment of this application; Figure 2 This is a flowchart of another video rendering control method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of a video rendering control device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a video rendering control device provided in an embodiment of this application. Detailed Implementation
[0011] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but additional steps not included in the drawings may also be present. The above processes can correspond to methods, functions, procedures, subroutines, subroutines, etc.
[0012] Figure 1 A flowchart of a video rendering control method provided in an embodiment of this application is given. The video rendering control method provided in this embodiment of the application can be executed by a video rendering control device, which can be implemented by hardware and / or software and integrated into a video rendering control equipment.
[0013] The following description uses a video rendering control device executing a video rendering control method as an example. (Reference) Figure 1 The video rendering control method includes: S110: Determine the actual arrival time, expected ready time, and video frame drop ratio of multiple video frames to be rendered within a preset time window, and determine the target maximum delay based on the actual arrival time, expected ready time, and video frame drop ratio.
[0014] The actual arrival time provided in this application can be determined by the timestamp of the video frame to be rendered before decoding arriving at the video rendering control device. The expected ready time can be understood as the expected time when the video frame to be rendered can be rendered (e.g., the expected time to send the video frame to be rendered into the GPU / display pipeline). The video frame drop ratio can be understood as the proportion of video frames to be rendered received within a preset time window that are dropped.
[0015] Upon receiving a decoded video frame to be rendered, the frame can be placed in a queue for further processing. For example, a preset time window (e.g., 1-2 seconds) can be maintained to record the actual arrival time, expected ready time, and video frame drop rate of multiple received video frames within that window.
[0016] Furthermore, after each received video frame to be rendered (i.e., the current video frame to be rendered), the target maximum latency of the current video frame to be rendered is determined based on the actual arrival time, expected ready time, and video frame drop ratio of multiple video frames to be rendered within a preset time window (including the current video frame to be rendered and multiple previously received video frames to be rendered). The target maximum latency can be understood as the maximum allowable delay from the expected ready time to the actual rendering time of the video frame to be rendered.
[0017] S120: Determine the current delay time based on the previously determined delay time and the target maximum delay time.
[0018] For example, the delay time determined when the video frame to be rendered was last received (i.e., the previously determined delay time) is determined, and the current delay time is determined based on the previously determined delay time and the aforementioned determined target maximum delay. The current delay time is further determined based on the previously determined delay time, and its maximum value is limited by the target maximum delay. The current delay time can be understood as the estimated delay time from the expected ready time of the current video frame to be rendered to the rendering process.
[0019] In one embodiment, the current delay time corresponding to the current video frame to be rendered will be used as the previously determined delay time corresponding to the next received video frame to be rendered, and will participate in the process of determining the next current delay time.
[0020] S130: Add the current delay time and the expected ready time to obtain the target rendering time of the video frame to be rendered.
[0021] For example, the current delay time and the expected ready time are added together to obtain the target rendering time of the current video frame to be rendered. The target rendering time can be understood as the estimated time to render the current video frame to be rendered.
[0022] S140: Based on the current time and the target rendering time, perform rendering decision processing on the queue of frames to be rendered.
[0023] For example, the current time and the target rendering time of each video frame to be rendered in the queue of frames to be rendered are determined, and rendering decision processing is performed on the video frames to be rendered in the queue of frames to be rendered. The rendering decision processing may be to extract the video frames to be rendered from the queue of frames to be rendered (e.g., the video whose target rendering time is not earlier than the current time and is the closest to the current time) for rendering processing, or it may be to discard expired video frames to be rendered from the queue of frames to be rendered (e.g., videos whose target rendering time is earlier than the current time).
[0024] Optionally, if the video frame to be rendered cannot be determined in the queue of frames to be rendered based on the current time and the target rendering time, the earliest video frame to be rendered added to the queue can be determined as the video frame to be rendered, forcibly advancing the rendering process and reducing video playback stuttering.
[0025] This application makes rendering decisions at the rendering level based on the actual arrival time of video frames, the expected ready time, and the video frame drop rate. It recursively calculates the expected ready time from the actual arrival time and estimates the network jitter state by using the difference between the actual arrival time and the expected ready time. This accurately determines the video frames to be rendered and the optimal target rendering time, reducing the issues of uneven or visually fragmented images caused by fixed delays or fixed display of the latest frame in related technologies. It also reduces stuttering caused by sudden screen stops and jitter accumulation, effectively improving video rendering quality.
[0026] The above describes a process that, by determining the actual arrival time, expected ready time, and video frame drop ratio of multiple video frames to be rendered within a preset time window, a target maximum delay is determined based on these parameters. The current delay is determined based on the previously determined delay and the target maximum delay. The target rendering time for each video frame to be rendered is obtained by adding the current delay and the expected ready time together. Rendering decisions are then made on the queue of frames to be rendered based on the current time and the target rendering time. By making rendering decisions at the rendering level based on the actual arrival time, expected ready time, and video frame drop ratio of the video frames, the video rendering effect can be effectively improved, enhancing the user's subjective experience.
[0027] Based on the above embodiments, Figure 2 A flowchart of another video rendering control method provided in an embodiment of this application is given, which is a concretization of the above-described video rendering control method. (Reference) Figure 2 The video rendering control method includes: S210: Determine the acquisition time of the current frame to be rendered, as well as the acquisition time and ready time of the reference frame. Subtract the acquisition time of the reference frame from the acquisition time of the current frame to be rendered, and add the ready time of the reference frame to obtain the expected ready time of the current video frame to be rendered.
[0028] The reference frame provided in this application is the first frame to be rendered. For example, upon receiving the first video frame to be rendered, the first frame to be rendered is determined as the reference frame, and the acquisition time of the reference frame (e.g., the time when the reference frame is acquired at the acquisition end) and the ready time of the reference frame (i.e., the time when the reference frame can be rendered) are determined. The reference ready time can be the time when the first video frame to be rendered can be rendered (e.g., the time to complete the decoding process of the first video frame to be rendered, or the time to complete the decoding process and related preprocessing of the first video frame to be rendered).
[0029] For example, upon receiving the current video frame to be rendered, the acquisition time of the current frame to be rendered (e.g., the time when the acquisition end acquires the current frame to be rendered) is determined, along with the acquisition time and readiness time of a reference frame. The acquisition time of the current frame to be rendered is then subtracted from the acquisition time of the reference frame, and the readiness time of the reference frame is added back. The sum can be used as the expected readiness time of the current video frame to be rendered. Based on this, the expected readiness time provided in this application can be determined by the following formula:
[0030] in, The expected ready time for the current video frame to be rendered. The ready time of the reference frame. The capture time of the current frame to be rendered. The acquisition time of the reference frame is used as the reference time. This application obtains the expected ready time of the current video frame to be rendered by subtracting the acquisition time of the reference frame from the acquisition time of the current frame to be rendered, and then adding the ready time of the reference frame. This improves the accuracy of determining the target rendering time and enhances the quality of video rendering control.
[0031] In one embodiment, after receiving the first video frame to be rendered, a preset number of subsequent video frames to be rendered can be used as video frames to start the convergence phase (for example, if the preset number is set to 19, then 20 video frames to be rendered, including the first video frame to be rendered, are in the convergence phase). The preset number of video frames to be rendered is used to establish a time baseline, without sampling historical jitter delay, to ensure the accuracy of video rendering control.
[0032] S220: Determine the actual arrival time, expected ready time, and video frame drop ratio of multiple video frames to be rendered within a preset time window, and subtract the expected ready time from the actual arrival time to obtain the historical jitter delay of the multiple video frames to be rendered.
[0033] S230: Determine the experience score of multiple historical jitter delays based on historical jitter delays and video frame drop ratio, and determine the target maximum latency from multiple historical jitter delays based on the experience score.
[0034] In one possible embodiment, the actual arrival time of multiple video frames to be rendered within a preset time window is subtracted from the corresponding expected ready time to obtain the historical jitter delay corresponding to each video frame to be rendered within the preset time window.
[0035] Furthermore, based on the historical jitter latency and video frame drop ratio corresponding to each video frame to be rendered within a preset time window, a quality of experience (QoS) score for multiple historical jitter latencies is calculated. The quality of experience is negatively correlated with the historical jitter latency and video frame drop ratio; that is, the smaller the historical jitter latency and video frame drop ratio, the higher the quality of experience, and the better the user's video viewing experience. The multiple historical jitter latencies provided in this application can be the historical jitter latencies corresponding to multiple video frames to be rendered within a preset time window.
[0036] In one embodiment, the target maximum latency of the current video frame to be rendered is determined from multiple historical jitter delays based on experience scores within a preset time window. For example, the historical jitter delay with the lowest experience score among multiple historical jitter delays can be used as the target maximum latency of the current video frame to be rendered. This application obtains historical jitter delays by subtracting the expected ready time from the actual arrival time, determines experience scores for multiple historical jitter delays based on the historical jitter delays and the video frame drop ratio, and determines the target maximum latency from multiple historical jitter delays based on the experience scores. By determining the target maximum latency with the best user experience from multiple possible historical jitter delays, the user experience is effectively improved.
[0037] In one embodiment, the historical jitter latency and video frame drop ratio can be normalized, and then the experience scores of multiple historical jitter latency can be determined based on the normalized historical jitter latency and video frame drop ratio.
[0038] In one embodiment, the video rendering control method provided in this application determines multiple historical jitter delay experience scores based on historical jitter delay and video frame drop ratio. This can be achieved by multiplying a preset frame drop penalty factor by the video frame drop ratio to obtain a frame drop penalty score, and then adding each historical jitter delay to the frame drop penalty score to obtain multiple historical jitter delay experience scores. Based on this, the experience scores provided in this application can be expressed by the following formula:
[0039] in, Rate your experience To delay historical jitter, To preset the frame dropping penalty factor, The ratio of dropped video frames is used to calculate the frame drop penalty score. This application obtains the frame drop penalty score by multiplying a preset frame drop penalty factor by the video frame drop ratio, and adds multiple historical jitter delays to the frame drop penalty score to obtain multiple historical jitter delay experience scores, thereby accurately evaluating the user experience under each historical jitter delay and effectively improving the user experience.
[0040] S240: Determine the current delay time based on the previously determined delay time and the target maximum delay time.
[0041] In one possible embodiment, the video rendering control method provided in this application determines the current delay time based on the previously determined delay time and the target maximum delay time, including: if the target maximum delay time is greater than the previously determined delay time, determining the target maximum delay time as the current delay time; if the target maximum delay time is less than the previously determined delay time, reducing the previously determined delay time by a preset delay reduction margin to obtain the current delay time.
[0042] For example, the previously determined delay time is compared with the target maximum delay time. If the target maximum delay time is the same as the previously determined delay time, the previously determined delay time is set as the current delay time. If the target maximum delay time is greater than the previously determined delay time, the target maximum delay time is set as the current delay time. If the target maximum delay time is less than the previously determined delay time, the previously determined delay time is reduced by a preset delay reduction margin to obtain the current delay time. Optionally, the delay reduction margin can be a preset delay value (e.g., 0~5ms) or a preset proportional coefficient (e.g., 0~50%). For example, if the target maximum delay time is less than the previously determined delay time, the previously determined delay time is reduced by 25% to obtain the current delay time (i.e., 75% of the previously determined delay time). This application determines the current latency immediately when the target maximum latency is greater than the previously determined latency time, and slowly decreases the previously determined latency time to obtain the current latency time when the target maximum latency is less than the previously determined latency time. This allows for rapid response to network degradation and smooth adaptation to improved network conditions, reducing video jitter and improving user experience.
[0043] S250: Add the current delay time and the expected ready time to obtain the target rendering time of the video frame to be rendered.
[0044] S260: Based on the current time and the target rendering time, perform rendering decision processing on the queue of frames to be rendered.
[0045] In one possible embodiment, the video rendering control method provided in this application performs rendering decision processing on the queue of frames to be rendered based on the current time and the target rendering time, including: performing expired frame dropping processing on the queue of frames to be rendered based on the current time and the target rendering time; and determining the video frame to be rendered whose target rendering time is closest to the current time in the queue of frames to be rendered after the expired frame dropping processing as the next target rendering frame to be rendered.
[0046] For example, starting from the head of the queue to be rendered, the target rendering time of each video frame in the queue is compared with the current time to determine whether the target rendering time is before the current time. If the target rendering time is before the current time, the corresponding video frame to be rendered is considered an expired frame and is discarded, until the first video frame to be rendered with a target rendering time after the current time is found, thus completing the expired frame discarding process for the queue to be rendered.
[0047] Furthermore, the video frame whose target rendering time is closest to the current time in the queue of frames to be rendered after expired frame dropping can be determined as the next target rendering frame. This application accurately determines which video frames to be discarded and which to be displayed by processing expired frames in the queue of frames to be rendered after expired frame dropping, thereby reducing stuttering caused by sudden screen pauses and jitter, and improving the user experience.
[0048] In one possible embodiment, the video rendering control method provided in this application performs rendering decision processing on the queue of frames to be rendered based on the current time and the target rendering time, and further includes: when the next target rendering frame to be rendered is not determined, or when the arrival time difference between the current video frame to be rendered and the previous video frame to be rendered reaches a preset time interval, the video frame to be rendered with the earliest actual arrival time in the queue of frames to be rendered after expired frame dropping processing is determined as the next target rendering frame to be rendered.
[0049] For example, if no next target rendering frame is found in the queue of frames to be rendered after expired frame dropping, or if the arrival time difference between the current video frame to be rendered and the previous video frame to be rendered reaches a preset time interval, in order to reduce the possibility of screen stuttering caused by rendering stoppage, the video frame to be rendered with the earliest actual arrival time in the queue of frames to be rendered after expired frame dropping can be determined as the next target rendering frame to be rendered, thereby improving video rendering quality and enhancing video playback smoothness. The preset time interval can be a preset multiple (e.g., 2 to 4 times) of the video frame arrival interval, which can be the arrival time difference between the current video frame to be rendered and the previous video frame to be rendered, or the average arrival time difference of multiple video frames to be rendered within a preset time window.
[0050] In one possible embodiment, the video rendering control method provided in this application, after performing rendering decision processing on the queue of frames to be rendered based on the current time and the target rendering time, further includes: estimating the target rendering time of the next video frame to be rendered when there is only one video frame to be rendered in the queue; and discarding the video frame to be rendered when the target rendering time of the next video frame to be rendered is earlier than the target rendering time of the only video frame to be rendered in the queue.
[0051] For example, when it is detected that there is only one video frame to be rendered in the rendering queue, the target rendering time of the next video frame to be rendered is estimated, and the target rendering time of the next video frame to be rendered is compared with the target rendering time of the only existing video frame to be rendered. If the target rendering time of the next video frame to be rendered is later than the target rendering time of the only existing video frame to be rendered, the only existing video frame to be rendered is retained; if the target rendering time of the next video frame to be rendered is earlier than the target rendering time of the only existing video frame to be rendered, the only existing video frame to be rendered is discarded. This eliminates the situation where the target rendering times of the rendered video frames are reversed, causing stuttering at the end of video playback and improving video rendering quality.
[0052] The above describes a process that, by determining the actual arrival time, expected readiness time, and frame drop ratio of multiple video frames to be rendered within a preset time window, a target maximum delay is determined based on these factors. The current delay is then determined based on the previously determined delay and the target maximum delay. The current delay and the expected readiness time are added together to obtain the target rendering time for each video frame to be rendered. Rendering decisions are then made on the queue of frames to be rendered based on the current time and the target rendering time. By making rendering decisions at the rendering level based on the actual arrival time, expected readiness time, and frame drop ratio of video frames, the video rendering effect can be effectively improved, enhancing the user's subjective experience. Furthermore, by handling expired frames in the queue of frames to be rendered and determining the next target frame to be rendered from the queue after this process, the system accurately identifies the video frames to be discarded and those to be displayed, reducing stuttering caused by sudden screen pauses and jitter, thus improving the user experience.
[0053] Figure 3 A schematic diagram of a video rendering control device according to an embodiment of this application is provided. (Reference) Figure 3 The video rendering control device includes a maximum delay module 31, a current delay module 32, a rendering time module 33, and a rendering decision module 34.
[0054] The maximum delay module 31 is used to determine the actual arrival time, expected ready time, and video frame drop ratio of multiple video frames to be rendered within a preset time window, and to determine the target maximum delay based on the actual arrival time, expected ready time, and video frame drop ratio; the current delay module 32 is used to determine the current delay time based on the previously determined delay time and the target maximum delay; the rendering time module 33 is used to add the current delay time and the expected ready time to obtain the target rendering time of the video frames to be rendered; and the rendering decision module 34 is used to perform rendering decision processing on the queue of frames to be rendered based on the current time and the target rendering time.
[0055] The above describes a process that, by determining the actual arrival time, expected ready time, and video frame drop ratio of multiple video frames to be rendered within a preset time window, a target maximum delay is determined based on these parameters. The current delay is determined based on the previously determined delay and the target maximum delay. The target rendering time for each video frame to be rendered is obtained by adding the current delay and the expected ready time together. Rendering decisions are then made on the queue of frames to be rendered based on the current time and the target rendering time. By making rendering decisions at the rendering level based on the actual arrival time, expected ready time, and video frame drop ratio of the video frames, the video rendering effect can be effectively improved, enhancing the user's subjective experience.
[0056] In one possible embodiment, the video rendering control device further includes a time determination module, which is used for: Determine the acquisition time of the current frame to be rendered, as well as the acquisition time and ready time of the reference frame. Subtract the acquisition time of the reference frame from the acquisition time of the current frame to be rendered, and add the ready time of the reference frame to obtain the expected ready time of the current video frame to be rendered. The reference frame is the first frame to be rendered.
[0057] In one possible embodiment, the maximum delay module 31 determines the target maximum delay based on the actual arrival time, the expected readiness time, and the video frame drop ratio, including: Subtracting the actual arrival time from the expected ready time yields the historical jitter delay of multiple video frames to be rendered; Experience scores for multiple historical jitter latency are determined based on historical jitter latency and the proportion of dropped video frames; The target maximum latency is determined from multiple historical jitter latency values based on experience ratings.
[0058] In one possible embodiment, the maximum latency module 31 determines multiple historical jitter latency experience scores based on historical jitter latency and the proportion of dropped video frames, including: The frame drop penalty score is obtained by multiplying the preset frame drop penalty factor by the proportion of video frames dropped. The experience scores for multiple historical jitter delays are obtained by adding the frame drop penalty scores to the historical jitter delay scores.
[0059] In one possible embodiment, the current delay module 32 determines the current delay time based on the previously determined delay time and the target maximum delay, including: If the maximum delay of the target is greater than the previously determined delay time, the maximum delay of the target will be determined as the current delay time; If the target maximum delay is less than the previously determined delay time, the previously determined delay time is reduced by the preset delay reduction rate to obtain the current delay time.
[0060] In one possible embodiment, the rendering decision module 34 performs rendering decision processing on the queue of frames to be rendered based on the current time and the target rendering time, including: Based on the current time and the target rendering time, perform expired frame dropping processing on the queue of frames to be rendered; In the queue of frames to be rendered after expired and dropped frames are processed, the video frame whose target rendering time is closest to the current time is determined as the next target rendering frame to be rendered. If the next target rendering frame is not determined, or if the arrival time difference between the current video frame to be rendered and the previous video frame to be rendered reaches a preset time interval, the video frame to be rendered with the earliest actual arrival time in the queue of frames to be rendered after expired and dropped frames is determined as the next target rendering frame to be rendered.
[0061] In one possible embodiment, the rendering decision module 34 is further configured to: Estimate the target rendering time of the next video frame to be rendered when there is only one video frame to be rendered in the queue of frames to be rendered. If the target rendering time of the next video frame to be rendered is earlier than the target rendering time of the only video frame to be rendered that exists in the queue of video frames to be rendered, then discard the only video frame to be rendered that exists in the queue of video frames to be rendered.
[0062] It is worth noting that in the above-described embodiments of the video rendering control device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this application.
[0063] This application also provides a video rendering control device, which can integrate the video rendering control apparatus provided in this application. Figure 4 This is a schematic diagram of the structure of a video rendering control device provided in an embodiment of this application. (Reference) Figure 4 The video rendering control device includes: an input device 43, an output device 44, a memory 42, and one or more processors 41; the memory 42 is used to store one or more programs; when one or more programs are executed by one or more processors 41, the one or more processors 41 implement the video rendering control method provided in the above embodiments. The input device 43, output device 44, memory 42, and processors 41 can be connected via a bus or other means. Figure 4 Taking the example of a connection between China and Israel via a bus.
[0064] The memory 42, as a computing device-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the video rendering control method provided in any embodiment of this application (e.g., the maximum delay module 31, current delay module 32, rendering time module 33, and rendering decision module 34 in the video rendering control device). The memory 42 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory 42 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 42 may further include memory remotely located relative to the processor 41, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0065] Input device 43 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 44 may include display devices such as a display screen.
[0066] The processor 41 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 42, thereby realizing the video rendering control method described above.
[0067] The video rendering control device, equipment, and computer provided above can be used to execute the video rendering control method provided in any of the above embodiments, and have corresponding functions and beneficial effects.
[0068] This application embodiment also provides a storage medium for storing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to execute the video rendering control method provided in the above embodiment. The video rendering control method includes: determining the actual arrival time, expected ready time, and video frame drop ratio of multiple video frames to be rendered within a preset time window; determining a target maximum delay based on the actual arrival time, expected ready time, and video frame drop ratio; determining the current delay time based on the previously determined delay time and the target maximum delay; adding the current delay time and the expected ready time to obtain the target rendering time of the video frames to be rendered; and performing rendering decision processing on the queue of frames to be rendered based on the current time and the target rendering time.
[0069] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROMs, floppy disks, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disks or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. Furthermore, storage media may reside in a first computer system in which a program is executed, or may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). Storage media may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.
[0070] Of course, the computer-executable instructions stored in the storage medium provided in the embodiments of this application are not limited to the video rendering control method provided above, but can also execute related operations in the video rendering control method provided in any embodiment of this application.
[0071] The video rendering control device, equipment, and storage medium provided in the above embodiments can execute the video rendering control method provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the video rendering control method provided in any embodiment of this application.
[0072] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments provided herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.
Claims
1. A video rendering control method, characterized in that, include: Determine the actual arrival time, expected ready time, and video frame drop ratio of multiple video frames to be rendered within a preset time window, and determine the target maximum delay based on the actual arrival time, the expected ready time, and the video frame drop ratio; The current delay time is determined based on the previously determined delay time and the target maximum delay time; The target rendering time of the video frame to be rendered is obtained by adding the current delay time and the expected ready time. Based on the current time and the target rendering time, a rendering decision is made for the queue of frames to be rendered.
2. The video rendering control method according to claim 1, characterized in that, Before determining the actual arrival time, expected ready time, and video frame drop ratio of multiple video frames to be rendered within a preset time window, the method further includes: Determine the acquisition time of the current frame to be rendered, the acquisition time of the reference frame, and the ready time of the reference frame. Subtract the acquisition time of the reference frame from the acquisition time of the current frame to be rendered, and add the ready time of the reference frame to obtain the expected ready time of the current video frame to be rendered, wherein the reference frame is the first frame to be rendered.
3. The video rendering control method according to claim 1, characterized in that, Determining the target maximum delay based on the actual arrival time, the expected readiness time, and the video frame drop ratio includes: Subtracting the actual arrival time from the expected ready time yields the historical jitter delay of multiple video frames to be rendered; Based on the historical jitter latency and the video frame drop ratio, a number of experience scores for the historical jitter latency are determined. The target maximum latency is determined from multiple historical jitter delays based on the experience score.
4. The video rendering control method according to claim 3, characterized in that, The process of determining multiple experience scores for historical jitter latency based on the historical jitter latency and the video frame drop ratio includes: The frame drop penalty score is obtained by multiplying the preset frame drop penalty factor by the video frame drop ratio; The experience scores for the historical jitter delays are obtained by adding the historical jitter delay scores to the frame drop penalty scores.
5. The video rendering control method according to claim 1, characterized in that, The step of determining the current delay time based on the previously determined delay time and the target maximum delay includes: If the maximum target delay is greater than the previously determined delay time, the maximum target delay is determined as the current delay time. If the target maximum delay is less than the previously determined delay time, the previously determined delay time is reduced by a preset delay reduction margin to obtain the current delay time.
6. The video rendering control method according to claim 1, characterized in that, The step of performing rendering decision processing on the queue of frames to be rendered based on the current time and the target rendering time includes: Based on the current time and the target rendering time, perform expired frame dropping processing on the queue of frames to be rendered. In the queue of frames to be rendered after expired and dropped frames are processed, the video frame whose target rendering time is closest to the current time is determined as the next target rendering frame to be rendered. If the next target rendering frame is not determined, or if the arrival time difference between the current video frame to be rendered and the previous video frame to be rendered reaches a preset time interval, the video frame to be rendered with the earliest actual arrival time in the queue of frames to be rendered after expired frame dropping is determined as the next target rendering frame to be rendered.
7. The video rendering control method according to claim 1, characterized in that, After performing rendering decision processing on the queue of frames to be rendered based on the current time and the target rendering time, the process further includes: Estimate the target rendering time of the next video frame to be rendered when there is only one video frame to be rendered in the queue of frames to be rendered. If the target rendering time of the next video frame to be rendered is earlier than the target rendering time of the only video frame to be rendered in the queue of frames to be rendered, then the only video frame to be rendered in the queue of frames to be rendered is discarded.
8. A video rendering control device, characterized in that, This includes a maximum latency module, a current latency module, a rendering time module, and a rendering decision module, among which: The maximum delay module is used to determine the actual arrival time, expected ready time and video frame drop ratio of multiple video frames to be rendered within a preset time window, and to determine the target maximum delay based on the actual arrival time, the expected ready time and the video frame drop ratio. The current delay module is used to determine the current delay time based on the previously determined delay time and the target maximum delay. The rendering time module is used to add the current delay time and the expected ready time to obtain the target rendering time of the video frame to be rendered. The rendering decision module is used to perform rendering decision processing on the queue of frames to be rendered based on the current time and the target rendering time.
9. A video rendering control device, characterized in that, include: Memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video rendering control method as described in any one of claims 1-7.
10. A storage medium for storing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the video rendering control method as described in any one of claims 1-7.