Live video playing method and device, storage medium and computer program product
By calculating the download frame rate and transcoding output frame rate of live videos and adjusting the buffering time, the lag caused by network and transcoding in low-latency live broadcasts is solved, and the smoothness and user experience of live broadcasts are improved.
Patent Information
- Application Number
- CN202411004159.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-25
AI Technical Summary
The factors considered in the prior art when dealing with lag problems in low-latency live broadcast scenarios are too single, and cannot effectively solve the lag phenomenon during live broadcast of videos, especially when the network bandwidth is sufficient.
By obtaining the download frame rate and original frame rate of the live video, calculate the jitter time difference between the download frame rate and the transcoding output frame rate, determine the jitter delay of the live video, and adjust the buffering time according to the jitter delay, comprehensively considering network fluctuations and rotations The impact of code output on live broadcast delay, dynamically adjust the data length of the play buffer and frame chasing logic.
It effectively improves the fluency of low-latency live broadcasts, reduces the lag of live videos, and provides a better viewing experience.
Smart Images

Figure CN119071519A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of video live broadcast technology, and in particular to a live video playback method, device, storage medium and computer program product. Background Art
[0002] In low-latency live broadcast scenarios, in order to avoid video freezes as much as possible, related technologies mainly adjust the length of buffered video according to network status. However, in low-latency live broadcast scenarios, related technologies consider relatively few factors when dealing with freezes, and cannot effectively solve the freeze problem during live broadcasts. Summary of the invention
[0003] The present disclosure provides a live video playback method, device, storage medium and computer program product.
[0004] According to a first aspect of the present disclosure, a live video playback method is provided, the method comprising:
[0005] Obtain the current download frame rate of the live video and the original frame rate of the live video during the live broadcast process;
[0006] Based on the download frame rate and the original frame rate, obtaining a download frame rate jitter time difference;
[0007] Obtaining a transcoding jitter frame rate of the live video, and obtaining a transcoding output frame rate jitter time difference of the live video based on the transcoding jitter frame rate and the original frame rate;
[0008] Based on the download frame rate jitter time difference and the transcoded output frame rate jitter time difference, the jitter delay of the live video is obtained, and the buffering duration of the live video is determined based on the jitter delay.
[0009] According to a second aspect of the present disclosure, a live video playback device is provided, the device comprising:
[0010] A live data acquisition module is used to obtain the current download frame rate of the live video and the original frame rate of the live video during the live broadcast process;
[0011] A download frame rate jitter time difference acquisition module, used to obtain the download frame rate jitter time difference based on the download frame rate and the original frame rate;
[0012] A transcoding output frame rate jitter time difference acquisition module, used to acquire the transcoding jitter frame rate of the live video, and obtain the transcoding output frame rate jitter time difference of the live video based on the transcoding jitter frame rate and the original frame rate;
[0013] The buffering duration determination module is used to obtain the jitter delay of the live video based on the download frame rate jitter time difference and the transcoded output frame rate jitter time difference, and determine the buffering duration of the live video based on the jitter delay.
[0014] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.
[0015] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above method of the present disclosure is implemented.
[0016] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above method of the present disclosure when executed by a processor.
[0017] The live video playback method, device, storage medium and computer program product provided by the embodiments of the present disclosure obtain the download frame rate jitter time difference by obtaining the current download frame rate of the live video and the original frame rate of the live video during the live broadcast process. And based on the transcoding jitter frame rate and the original frame rate of the acquired live video, the transcoding output frame rate jitter time difference of the live video is obtained. The jitter delay of the live video obtained by the download frame rate jitter time difference and the transcoding output frame rate jitter time difference can be used to determine the buffering time length of the live video. In this way, by comprehensively considering the impact of download frame rate fluctuations and transcoding jitter on live broadcast delay, and by adjusting the buffering length of the video buffer buffering the video, the jamming problem during video live broadcast can be effectively solved. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:
[0019] Figure 1 A flowchart of a live video playback method provided by an exemplary embodiment of the present disclosure;
[0020] Figure 2 A flowchart of a live video playback method provided by another exemplary embodiment of the present disclosure;
[0021] Figure 3 A flowchart of a live video playback method provided by another exemplary embodiment of the present disclosure;
[0022] Figure 4 A schematic block diagram of functional modules of a live video playback device provided by an exemplary embodiment of the present disclosure;
[0023] Figure 5A structural block diagram of an electronic device provided for an exemplary embodiment of the present disclosure;
[0024] Figure 6 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0026] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0027] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0028] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0029] In low-latency live broadcast scenarios, related technologies mainly evaluate the terminal network quality and adjust the length of the video playback buffer to combat network fluctuations and reduce the jam rate. This makes the related technologies too single-minded in analyzing the jam problem in low-latency live broadcast scenarios, and does not consider the volatility of the video transcoding output frame rate; after the network amplification effect, it will aggravate the video frame rate fluctuation, causing users to encounter jams in low-latency live broadcasts even when the network bandwidth is sufficient and stable. Therefore, the related technologies have certain limitations in adjusting the playback buffer only based on network fluctuation factors, and cannot completely solve the jam problem.
[0030] Therefore, the disclosed embodiment integrates multiple factors such as network fluctuations and video transcoding output fluctuations, and dynamically adjusts the playback buffer data length and the playback strategy of the frame tracking logic, so as to more comprehensively improve the smoothness of low-latency live broadcast and bring users the best low-latency live broadcast viewing experience.
[0031] In the embodiment, the video source station transcodes the live video and forwards the transcoded live video stream through a CDN (Content Delivery Network) server, and the terminal receives and plays the live video stream forwarded by the CDN. Specifically, the live video can be downloaded through the playback network module in the terminal, and the video is cached in the playback buffer module, and the playback analysis control module in the terminal analyzes and processes the video data in the buffer.
[0032] Among them, the video source station: the server equipment located in the computer room, is responsible for producing and pushing low-latency live video streaming resources to the CDN server.
[0033] CDN server: A server device located in the computer room, responsible for forwarding and sending low-latency live video streaming resources.
[0034] Playback network module: located in the user's mobile video APP (application), used to download live video stream resources from the CDN server side and store them in the playback cache module.
[0035] Playback cache module: located in the video APP of the user terminal such as a mobile phone, used to store and retrieve playback buffer data, that is, to transfer video stream playback data.
[0036] Playback analysis control module: located in the user terminal such as the mobile video APP of the mobile phone, used to receive the collected data of the video frame rate stored (ie downloaded) by the playback cache module, and analyze and process the original data.
[0037] The disclosed embodiment may specifically include two stages: the first stage is data collection and processing, in which the playback analysis control module collects video download frame rate data, and combines it with the transcoding frame rate data to dynamically calculate and adjust the video buffer duration parameters and playback strategy; the second stage is dynamic compensation, in which the playback delay parameters and playback strategy are dynamically compensated according to the number of freezes and duration data during the actual playback process.
[0038] In the first stage, data collection and processing can be carried out first.
[0039] (1) In the case of low latency, during the playback of a live video on a terminal, obtain the original download frame rate data reported by the player on the terminal. The original download frame rate data may include vFPS and netFPS; where vFPS represents the original frame rate of the video resource, that is, the original frame rate of the live video; netFPS represents the download frame rate of the current video of the player.
[0040] (2) Calculate the jitter time difference corresponding to the download frame rate of the video relative to the original frame rate of the video.
[0041] In the embodiment, the downward jitter frame rate difference DFPS between the current video download frame rate and the original frame rate of the video can be obtained through the following formula (1):
[0042] DFPS = vFPS - netFPS (1)
[0043] In the embodiment, the jitter time difference dT corresponding to the download frame rate jitter difference DFPS can be calculated through the following formula (2):
[0044] dT = (1000 / vFPS) * DFPS (2)
[0045] Among them, the original frame rate of the video is the number of video frames played and displayed within 1000 ms. By calculation, the average playback duration of each frame can be obtained, and the corresponding jitter time difference can be calculated according to the number of jitter frames within 1000 ms.
[0046] (3) Obtain the smoothed average value f(sT) of the download frame rate jitter time difference.
[0047] In the implementation, after obtaining the above jitter time difference dT, the dT can be smoothed through the following formula (3).
[0048]
[0049] Among them, sT represents the influence degree of the download frame rate on the playback delay of the live video when playing the live video based on the original frame rate within a unit time; Vnet represents the download rate of the live video; Mbr represents the bit rate of the live video; when the network bandwidth resource is sufficient, that is, Vnet is not less than Mbr, the initial value of sT is the first dT value; α is a smoothing coefficient (the value range is 0 to 1), and α can be set as needed, specifically it can be an empirical value; dT is the current jitter time difference.
[0050] It should be noted that in the case of insufficient network bandwidth, that is, Vnet < Mbr, the current dT value is not taken as a calculation factor for sT data to avoid interference caused by the network quality itself to the download frame rate jitter; when the network bandwidth is sufficient, the smoothed average value sT of the current download frame rate jitter time difference is calculated by combining the smoothing coefficient with the current jitter time difference dT data and the sT data of the previous cycle.
[0051] Among them, the smoothed average of the download frame rate jitter time difference f(sT) represents the time that is missing from the playback time that should theoretically be obtained per second in the terminal environment compared to the actual data playback time, that is, the buffer data playback time required to cope with network jitter factors in the low-latency video live broadcast scenario.
[0052] (4) Calculate the jitter time difference voT of the video source transcoding output frame rate.
[0053] In the embodiment, the video source transcoding output frame rate jitter time difference voT can be calculated by the following formula (4).
[0054] voT=voFPS*(1000 / vFPS) (4)
[0055] Among them, voT represents the influence of transcoding jitter frame rate on live video playback delay when playing live video based on the original frame rate per unit time; voFPS is the average downward jitter frame rate of the video source transcoding output. voFPS is related to the server video transcoding performance. The voFPS value of the same source station is generally a fixed value. The voFPS value can be obtained by exchanging messages with the server through video APPs on mobile phones and other terminals; the video source transcoding output frame rate jitter time difference voT can be the buffer data playback time required to deal with the transcoding output frame rate jitter factor during low-latency live broadcast; vFPS represents the original frame rate of the video resource.
[0056] (5) Calculate the live video delay T.
[0057] In the embodiment, the live video delay T can be calculated by the following formula (5):
[0058] T=sT+voT (5)
[0059] Among them, the live video delay T represents the minimum delay for the live video to play without freezing in the low-latency live broadcast scenario, that is, the length of time the player buffer can play. T represents the resistance value to avoid playback freezing events caused by network jitter and transcoding output jitter. When the buffer of the live video contains at least T-length buffered video, the playback can avoid freezing caused by network jitter and transcoding jitter in the low-latency scenario.
[0060] In the second stage, motion compensation can be performed on the video buffered in the video buffer.
[0061] In the embodiment, the historical freeze events reported by the player can be collected, and the number of freezes N and the freeze duration Ti of each freeze in the historical freeze time can be counted. And according to the number of freezes N and the freeze duration Ti, it is determined whether to trigger the buffer data excessive delay compensation;
[0062] Specifically, the player does not freeze during the current cycle, and the video duration that can be played by the video buffer data is not less than the threshold value T during this period. η When the player delay T, that is, the buffer length corresponding to the playback time can be reduced by T γ Among them, in order to avoid the risk of delay causing jamming problems, T γ The value range can be 0~T η ; T η Can be experience value.
[0063] When the number of freezes reaches the threshold value M times, or the total freeze duration ∑Ti exceeds a certain threshold, that is, the duration of the freeze phenomenon that is more obvious to the human eye, delay compensation is triggered, and the player's delay T, that is, the buffer length corresponding to the playback time can be increased by T γ Among them, T γ The value interval may be half of the total freeze duration, that is, ∑Ti / 2, and may be specifically set as required, and the embodiment is not limited thereto.
[0064] In an embodiment, the playback of the live video may also be compensated by the player.
[0065] For example, when the actual video buffer data playable time durT is too long, the video can be fast-forwarded to reduce the delay of video playback; when the actual video buffer data playable time durT is too short, the video can be slow-forwarded to avoid playback stuttering caused by the small amount of video data cached in the buffer.
[0066] Specifically, the difference between the video playback delay T and the actual video buffer data playable time durT can be obtained to decide whether to use the fast forward or slow play mode to optimize the playback experience. The specific strategies are as follows:
[0067] When the difference between the playable duration durT in the buffer and the video delay T is greater than the threshold value ΔT1, the player can be set to use a faster fast-forward mode with a speed greater than 1x. This will shorten the time difference between the play screen and the live source screen by quickly consuming the buffer data, thereby achieving the purpose of low live delay; at the same time, ensure that the playable duration durT in the buffer is not less than the video delay T, so as to avoid freezes in the live video during playback.
[0068] When the difference between the buffer playable time durT and the video playback delay T is less than the threshold value ΔT2, the player is set to use a slow-down mode lower than 1x speed for playback. This can slow down the consumption of buffer data to buy time for downloading playback data, thereby achieving the purpose of low lag in live broadcast.
[0069] In other cases, the player plays at 1x speed, that is, plays the live video normally. For example, when the difference between the playable duration durT in the buffer and the video playback delay T is between ΔT2 and ΔT1, the live video is played normally. ΔT1 and ΔT2 can be set as needed, and can be empirical values.
[0070] Based on the above embodiments, the present disclosure also provides a method for playing live video. Figure 1 As shown, the method may include the following steps:
[0071] In step S110, the current download frame rate of the live video and the original frame rate of the live video during the live broadcast process are obtained.
[0072] In an embodiment, in a low-latency live video scenario, when a user watches a live video through a mobile phone or other terminal, the user may be affected by network fluctuations, and the video playback may freeze when the live video is played. Before the live video is played, it is necessary to download the live data and buffer a certain amount of video data in the player's buffer, so that the video can be played by reading the video data buffered in the buffer. When the video is played, the video is usually played at a fixed frame rate, such as 24 frames per second. Since the amount of data contained in each frame in the video frame may be different, such as the number of key frames is usually greater than the amount of data contained in non-key frames, when the video is played at a fixed number of frames, there may be a situation where the amount of data contained in some frames is too large, resulting in a reduction in the number of video frames downloaded (i.e., buffered) per unit time, so that the video data buffered in the buffer is insufficient to support the video to play the video at a fixed number of frames, thereby causing the video to freeze.
[0073] Therefore, in the embodiment, it is necessary to obtain the current download frame rate of the video during the live broadcast and the original frame rate of the live video, and then determine the degree of influence of the current download frame rate of the live video on the live video playback delay.
[0074] In step S120, the download frame rate jitter time difference is obtained based on the download frame rate and the original frame rate.
[0075] The download jitter time difference represents the influence of the download frame rate on the live video playback delay when the live video is played based on the original frame rate within a unit time.
[0076] In an embodiment, the frame rate difference between the original frame rate and the downloaded frame rate can be calculated by the above formula (1), and based on the frame rate difference and the original frame rate of the live video, the downloaded frame rate jitter time difference can be obtained. Specifically, the downloaded frame rate jitter time difference can be calculated by the above formula (2).
[0077] In the embodiment, if the download frame rate is lower than the original frame rate of the live video, it means that the number of video frames downloaded per unit time is lower than the number of video frames played per unit time, which will cause freezes during video playback. Therefore, when the download frame rate is lower than the original frame rate of the live video, a certain amount of video data needs to be buffered in advance in the video buffer to compensate for the small download frame rate. If only the download frame rate is considered, the buffer must at least meet the requirement of pre-buffering video data of the first target playback duration, and the video data of the first target playback duration is the above-mentioned download frame rate jitter time difference, so as to ensure smooth playback of the live video without freezes.
[0078] In step S130, the transcoding jitter frame rate of the live video is obtained, and the transcoding output frame rate jitter time difference of the live video is obtained based on the transcoding jitter frame rate and the original frame rate.
[0079] The transcoding output frame rate jitter time difference represents the influence of the transcoding jitter frame rate on the live video playback delay when the live video is played based on the original frame rate within a unit time.
[0080] In the embodiment, in a low-latency live video scenario, when a user watches a live video through a mobile phone or other terminal, the user may be affected by transcoding jitter, which may cause the video to freeze when playing the live video. Before playing the live video, the live data needs to be downloaded and a certain amount of video data needs to be buffered in the player's buffer, so that the video can be played by reading the video data buffered in the buffer.
[0081] When playing a video, it is usually played at a fixed frame rate such as 24 frames per second. Since the transcoding rates of different transcoding servers may be different, and the transcoding may jitter downward during the transcoding process, once the transcoding rate of the corresponding transcoding server is lower than the original frame rate of the live video, the number of video frames downloaded (i.e., buffered) per unit time will be reduced, that is, the buffer cannot download enough video frames per unit time, making the video data buffered in the buffer insufficient to support the video to play at a fixed frame rate, thereby causing video freeze.
[0082] In the embodiment, similarly, if there is a downward jitter frame rate in the video transcoding output, it will affect the number of video frames received by the terminal per unit time, and may cause the number of video frames downloaded by the terminal per unit time to be less than the number of video frames played per unit time, thereby causing a freeze in the video playback. Therefore, it is necessary to buffer a certain amount of video data in the video buffer in advance to compensate for the situation caused by the low download frame rate. If only the transcoding jitter rate of the live video is considered, the buffer must at least meet the requirement of pre-buffering the video data of the second target playback time, and the video data of the first target playback time is the jitter time difference of the transcoding output frame rate of the live video, so as to ensure the smooth playback of the live video without freezes.
[0083] It should be noted that the transcoding jitter frame rate of the live video can specifically be the average downward jitter frame rate of the transcoding output of the video source. By determining the push stream server corresponding to the live video, the average downward jitter frame rate corresponding to the push stream server when transcoding the live video is obtained, and the average downward jitter frame rate is used as the transcoding jitter frame rate of the live video. Specifically, it can be calculated by the above formula (4). By obtaining the average downward jitter frame rate of the transcoding output of the video source per unit time, the jitter time difference of the transcoding output frame rate that affects the live video can be obtained, and when the playback time corresponding to the video data buffered in the video buffer is ensured to be not less than the jitter time difference of the transcoding output frame rate, the smooth playback of the live video can be ensured, thereby avoiding the jamming phenomenon.
[0084] In step S140, the jitter delay of the live video is obtained based on the download frame rate jitter time difference and the transcoded output frame rate jitter time difference, and the buffering time length of the live video is determined based on the jitter delay.
[0085] In the embodiment, the above-mentioned effects of network fluctuations and transcoding jitters on the playback of live video are considered respectively, so that the above-mentioned two effects can be combined, that is, based on the download frame rate jitter time difference and the transcoding output frame rate jitter time difference, it is ensured that the buffering time in the video buffer is at least not less than the sum of the download frame rate jitter time difference and the transcoding output frame rate jitter time difference, so as to ensure the flow playback of live video and avoid the jamming phenomenon.
[0086] It should be noted that the buffering duration of the live video in the embodiment is the playback duration corresponding to the buffered data in the buffer.
[0087] The live video playback method provided by the embodiments of the present disclosure obtains the download frame rate jitter time difference by acquiring the current download frame rate of the live video and the original frame rate of the live video during the live broadcast. And based on the transcoding jitter frame rate and the original frame rate of the acquired live video, the transcoding output frame rate jitter time difference of the live video is obtained. The jitter delay of the live video obtained from the download frame rate jitter time difference and the transcoding output frame rate jitter time difference can be used to determine the buffering duration of the live video. In this way, by comprehensively considering the impact of download frame rate fluctuations and transcoding jitter on the live broadcast delay, and adjusting the buffering length of the video buffer for the video, the stuttering problem during video live broadcast can be effectively solved.
[0088] Based on the above embodiments, in another embodiment provided by the present disclosure, in order to avoid large fluctuations in the download frame rate and have a greater impact on the video buffered in the video buffer, the embodiment can also perform smoothing processing on the above download frame rate jitter time difference.
[0089] In the embodiment, specifically, in combination with the above formula (3) and the corresponding embodiment, by acquiring the download rate of the live video and the live video bit rate, when the download rate of the live video is less than the live video bit rate, the download frame rate jitter time difference in the current period is discarded. In this way, when the network bandwidth is insufficient, that is, when Vnet < Mbr, the download frame rate jitter time difference in the current period can be not used as a calculation factor for sT data, avoiding the interference of the network quality itself on the download frame rate jitter.
[0090] When the download rate of the live video is not less than the live video bit rate, the download frame rate jitter time difference in the current period can be smoothed based on the download frame rate jitter time difference in the current period and the download frame rate jitter time difference in the previous period adjacent to the current period. In this way, when the network bandwidth is sufficient, the smoothed average value of the download frame rate jitter time difference in the current period can be calculated by combining the smoothing coefficient with the download frame rate jitter time difference in the current period and the download frame rate jitter time difference in the previous period, thereby achieving the purpose of smoothing the download frame rate jitter time difference in the current period.
[0091] Therefore, when obtaining the jitter delay of the live video based on the download frame rate jitter time difference and the transcoding output frame rate jitter time difference, when the download rate of the live video is not less than the live video bit rate, the jitter delay of the live video can be obtained based on the smoothed download frame rate jitter time difference and the transcoding output frame rate jitter time difference. The jitter delay of the live video obtained in this way can be smoother, avoiding large fluctuations in the video buffered in the video buffer, resulting in excessive delay or stuttering of the live video.
[0092] It should be noted that a period in the embodiment can specifically be 1 second, and the embodiment is not limited thereto.
[0093] Based on the above embodiments, in another embodiment provided by the present disclosure, Figure 2 As shown, the method may further include the following steps:
[0094] In step S150, the freeze information of the live video in the current cycle is obtained.
[0095] The freeze information includes the number of freezes and the freeze duration.
[0096] In step S160, when there is no freeze in the live video in the current cycle and the playback duration of the buffered video in the video buffer is not less than the first preset duration, the playback duration of the buffered video in the video buffer is reduced.
[0097] In the embodiment, if there is no freeze in the live video in the current cycle, and the playback duration of the buffered video in the video buffer is not less than the first preset duration, it means that enough video data has been cached in the video buffer. Although this can avoid freezes during playback, it will cause a large delay in the live broadcast. Therefore, in this case, the video data in the video buffer can be appropriately reduced, for example, by fast-forwarding playback, or skipping unimportant content according to the video content, so as to achieve the purpose of reducing the buffered data in the buffer.
[0098] In step S170, when the number of freezes in the live video in the current cycle is greater than the first threshold, or the total freeze duration in the current cycle is greater than the second preset duration, the playback duration of the buffered video in the video buffer is increased.
[0099] In the embodiment, when the number of freezes of the live video in the current cycle is greater than the first threshold, or the total freeze duration in the current cycle is greater than the second preset duration, it means that the amount of video data buffered in the video buffer is insufficient to support the non-free playback of the live video, and it is necessary to increase the playback duration of the buffered video in the video buffer. For example, more buffering time can be created by slow-forward playback or repeated playback of buffered data to buffer more video data in the buffer, thereby achieving the purpose of non-free playback of the live video.
[0100] It should be noted that the size of the video data reduced or increased in the buffer in the embodiment can refer to the description of the above embodiment, which will not be repeated here.
[0101] Based on the above embodiments, in another embodiment provided by the present disclosure, Figure 3 As shown, the method may further include the following steps:
[0102] In step S191, the playback duration of the buffered video in the video buffer and the playback delay preset in the player corresponding to the live video are obtained.
[0103] In step S192, when the difference between the playback time of the buffered video in the video buffer and the playback delay is greater than a second threshold, the live video is fast-forwarded through the player.
[0104] In the embodiment, when the playback duration corresponding to the video data buffered in the video buffer is greater than the playback delay by a second threshold, it means that the video buffer has buffered too much video, which will further increase the video delay. At this time, the video length in the buffer can be consumed by fast forwarding playback to reduce the delay in live video playback. The second threshold can be the threshold value ΔT1 in the above embodiment.
[0105] In step S193, when the difference between the playback duration of the buffered video in the video buffer and the playback delay is less than a third threshold, the live video is played in slow motion through the player.
[0106] When the playback duration corresponding to the video data buffered in the video buffer is less than the playback delay by a third threshold, it means that the video buffer has too little video, which will cause a freeze in the live video playback. In this case, the video length in the buffer can be increased by slow-forward playback to avoid freeze in the live video playback. The third threshold may be the threshold value ΔT2 in the above embodiment.
[0107] In step S194, when the difference between the playback duration of the buffered video in the video buffer and the playback delay is between the third threshold and the second threshold, the live video is played normally through the player.
[0108] In an embodiment, when the difference between the buffer playable time durT and the video play delay T is between ΔT2 and ΔT1, the live video can be played at a normal rate, which will neither cause excessive delay to the live video nor cause freezes.
[0109] In the case of dividing each functional module according to each function, an embodiment of the present disclosure provides a live video playback device, which can be a server, a terminal, or a chip applied to a server. Figure 4 This is a schematic block diagram of the functional modules of a live video playback device provided by an exemplary embodiment of the present disclosure. Figure 4 As shown, the live video playback device includes:
[0110] The live data acquisition module 10 is used to acquire the current download frame rate of the live video and the original frame rate of the live video during the live broadcast process;
[0111] A download frame rate jitter time difference acquisition module 20, configured to obtain a download frame rate jitter time difference based on the download frame rate and the original frame rate;
[0112] A transcoding output frame rate jitter time difference acquisition module 30 is used to acquire the transcoding jitter frame rate of the live video, and obtain the transcoding output frame rate jitter time difference of the live video based on the transcoding jitter frame rate and the original frame rate;
[0113] The buffering duration determination module 40 is used to obtain the jitter delay of the live video based on the download frame rate jitter time difference and the transcoded output frame rate jitter time difference, and determine the buffering duration of the live video based on the jitter delay.
[0114] In another embodiment provided by the present disclosure, the download frame rate jitter time difference acquisition module is specifically used to:
[0115] Obtaining a frame rate difference between the original frame rate and the downloaded frame rate;
[0116] Based on the frame rate difference and the original frame rate, a download frame rate jitter time difference is obtained.
[0117] In another embodiment provided by the present disclosure, the device further includes a smoothing processing module, specifically configured to:
[0118] Obtaining the download rate and bit rate of the live video;
[0119] When the download rate of the live video is lower than the bit rate of the live video, the download frame rate jitter time difference in the current period is discarded;
[0120] When the download rate of the live video is not less than the live video bit rate, the download frame rate jitter time difference of the current period is smoothed based on the download frame rate jitter time difference of the current period and the download frame rate jitter time difference of the previous period adjacent to the current period.
[0121] In another embodiment provided by the present disclosure, the smoothing processing module is specifically used to:
[0122] When the download rate of the live video is not less than the bit rate of the live video, the jitter delay of the live video is obtained based on the download frame rate jitter time difference after the smoothing process and the transcoded output frame rate jitter time difference.
[0123] In another embodiment provided by the present disclosure, the device further includes: a transcoding jitter frame rate determination module, specifically configured to:
[0124] Determine a streaming server corresponding to the live video;
[0125] An average downward jitter frame rate corresponding to the streaming server when transcoding the live video is obtained, and the average downward jitter frame rate is used as the transcoding jitter frame rate of the live video.
[0126] In another embodiment provided by the present disclosure, the device further includes:
[0127] A freeze information acquisition module, used to acquire freeze information of the live video in the current cycle, wherein the freeze information includes the freeze number and freeze duration;
[0128] A playback duration reduction module, configured to reduce the playback duration of the buffered video in the video buffer when there is no freeze in the live video in the current cycle and the playback duration of the buffered video in the video buffer is not less than a first preset duration;
[0129] The playback duration increasing module is used to increase the playback duration of the buffered video in the video buffer when the number of freezes of the live video in the current cycle is greater than a first threshold, or the total freeze duration in the current cycle is greater than a second preset duration.
[0130] In another embodiment provided by the present disclosure, the device further includes: a playback control module, specifically configured to:
[0131] Obtaining the playback duration of the buffered video in the video buffer and the playback delay preset in the player corresponding to the live video;
[0132] When the difference between the playback duration and the playback delay is greater than a second threshold, fast-forwarding the live video through the player;
[0133] When the difference between the playback duration and the playback delay is less than a third threshold, playing the live video in slow motion by the player;
[0134] When the difference between the playback duration and the playback delay is between the third threshold and the second threshold, the live video is played normally through the player.
[0135] The live video playback device provided by the embodiment of the present disclosure obtains the download frame rate jitter time difference by obtaining the current download frame rate of the live video and the original frame rate of the live video during the live broadcast process. And based on the transcoding jitter frame rate and the original frame rate of the acquired live video, the transcoding output frame rate jitter time difference of the live video is obtained. The buffering time length of the live video can be determined by the jitter delay of the live video obtained by the download frame rate jitter time difference and the transcoding output frame rate jitter time difference. In this way, by comprehensively considering the impact of download frame rate fluctuations and transcoding jitter on live broadcast delay, and by adjusting the buffering length of the video buffer to buffer the video, the jamming problem during video live broadcast can be effectively solved.
[0136] An embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory for storing instructions executable by the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the above method disclosed in the embodiment of the present disclosure.
[0137] Figure 5 The structure diagram of an electronic device provided by an exemplary embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801 , and the processor 1801 can execute corresponding steps in the above method disclosed in the embodiment of the present disclosure.
[0138] The processor 1801 may also be referred to as a central processing unit (CPU), which may be an integrated circuit chip having signal processing capabilities. Each step in the method disclosed in the embodiment of the present disclosure may be completed by an integrated logic circuit of hardware in the processor 1801 or by instructions in the form of software. The processor 1801 may be a general-purpose processor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a memory 1802, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, or other mature storage media in the art. The processor 1801 reads the information in the memory 1802 and completes the steps of the method in combination with its hardware.
[0139] In addition, when various operations / processes according to the present disclosure are implemented by software and / or firmware, they can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, such as Figure 6 The computer system 1900 shown is installed with the programs constituting the software. When the various programs are installed, the computer system can perform various functions, including the functions described above. Figure 6 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure.
[0140] Computer system 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0141] like Figure 6 As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 to a random access memory (RAM) 1903. In the RAM 1903, various programs and data required for the operation of the computer system 1900 can also be stored. The computing unit 1901, the ROM 1902, and the RAM 1903 are connected to each other via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.
[0142] A plurality of components in the computer system 1900 are connected to the I / O interface 1905, including: an input unit 1906, an output unit 1907, a storage unit 1908, and a communication unit 1909. The input unit 1906 may be any type of device capable of inputting information to the computer system 1900, and the input unit 1906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 1907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1908 may include, but is not limited to, a disk, an optical disk. The communication unit 1909 allows the computer system 1900 to exchange information / data with other devices over a network such as the Internet, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.
[0143] The computing unit 1901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the above methods disclosed in the embodiments of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1908. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 1902 and / or communication unit 1909. In some embodiments, the computing unit 1901 may be configured to perform the above methods disclosed in the embodiments of the present disclosure in any other appropriate manner (e.g., by means of firmware).
[0144] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above method disclosed in the embodiment of the present disclosure.
[0145] The computer-readable storage medium in the disclosed embodiments may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specifically, the computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0146] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0147] The embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor, the above method disclosed in the embodiments of the present disclosure is implemented.
[0148] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer.
[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0150] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.
[0151] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.
[0152] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above features with the technical features with similar functions disclosed in the present disclosure (but not limited to).
[0153] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.
Claims
1. A live video playback method, characterized in that: The method comprises: Obtain the current download frame rate of the live video and the original frame rate of the live video during the live broadcast process; Based on the download frame rate and the original frame rate, obtaining a download frame rate jitter time difference; Obtaining a transcoding jitter frame rate of the live video, and obtaining a transcoding output frame rate jitter time difference of the live video based on the transcoding jitter frame rate and the original frame rate; Based on the download frame rate jitter time difference and the transcoded output frame rate jitter time difference, the jitter delay of the live video is obtained, and the buffering duration of the live video is determined based on the jitter delay.
2. The method according to claim 1, characterized in that The obtaining the download frame rate jitter time difference based on the download frame rate and the original frame rate includes: Obtaining a frame rate difference between the original frame rate and the downloaded frame rate; Based on the frame rate difference and the original frame rate, a download frame rate jitter time difference is obtained.
3. The method according to claim 1, characterized in that The method further comprises: Obtaining the download rate and bit rate of the live video; When the download rate of the live video is lower than the bit rate of the live video, the download frame rate jitter time difference in the current period is discarded; Alternatively, when the download rate of the live video is not less than the live video bit rate, the download frame rate jitter time difference of the current period is smoothed based on the download frame rate jitter time difference of the current period and the download frame rate jitter time difference of the previous period adjacent to the current period.
4. The method according to claim 3, characterized in that The obtaining the jitter delay of the live video based on the download frame rate jitter time difference and the transcoded output frame rate jitter time difference comprises: When the download rate of the live video is not less than the bit rate of the live video, the jitter delay of the live video is obtained based on the download frame rate jitter time difference after the smoothing process and the transcoded output frame rate jitter time difference.
5. The method according to claim 1, characterized in that The method further comprises: Determine a streaming server corresponding to the live video; An average downward jitter frame rate corresponding to the streaming server when transcoding the live video is obtained, and the average downward jitter frame rate is used as the transcoding jitter frame rate of the live video.
6. The method according to claim 1, characterized in that The method further comprises: Obtaining freeze information of the live video in the current period, wherein the freeze information includes the number of freezes and the freeze duration; When there is no freeze in the live video in the current cycle and the playback duration of the buffered video in the video buffer is not less than the first preset duration, reducing the playback duration of the buffered video in the video buffer; Alternatively, when the number of freezes of the live video in the current cycle is greater than a first threshold, or the total freeze duration in the current cycle is greater than a second preset duration, the playback duration of the buffered video in the video buffer is increased.
7. The method according to claim 1, characterized in that The method further comprises: Obtaining the playback duration of the buffered video in the video buffer and the playback delay preset in the player corresponding to the live video; When the difference between the playback duration and the playback delay is greater than a second threshold, fast-forwarding the live video through the player; Alternatively, when the difference between the playback duration and the playback delay is less than a third threshold, the live video is played in slow motion by the player; Alternatively, when the difference between the playback duration and the playback delay is between the third threshold and the second threshold, the live video is played normally through the player.
8. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 7.
10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.
Citation Information
Patent Citations
RTMP live code rate self-adaptive method
CN110121089A
Batch processing method for monitoring videos with multi-channel load balance
CN111757145A
Live steaming transcoding method and device
CN111866533A
Video lag prediction method and device, equipment and medium
CN114401447A
Video stream processing method, device, terminal device, and computer-readable storage medium
US20220201205A1
Cited By
Media stream real-time playing method and device, electronic equipment and storage medium
CN120075381A
A media stream real-time playing method and device, electronic equipment and storage medium
CN120075381B
