Live video rendering method and device and storage medium

Abstract

The invention provides a live video rendering method and device and a storage medium. Wherein the stream pulling end equipment can accurately calculate the live broadcast delay between the stream pulling end equipment and the stream pushing end equipment according to a first timestamp carried by supplementary enhancement information in the live broadcast code stream data and a second timestamp corresponding to the current moment, and correspondingly regulate and control the picture rendering of the live broadcast video based on the live broadcast delay; and the display effect of the live video can be improved.

Description

Technical Field

[0001] The present application relates to the field of video technology, and in particular to a live video rendering method, device and storage medium. Background Art

[0002] With the continuous development of network technology, the application scope of live video has expanded from industries such as video entertainment and e-commerce to online education, optoelectronic media, surveillance and security, and is widely used in all aspects of daily life. In the scenario of live video, the process of generating and transmitting live content can be called push streaming, and the process of receiving and playing live content can be called pull streaming. Users can receive live stream data through pull-end devices such as smartphones and tablets, and render videos based on the live stream data, thereby realizing live video playback.

[0003] However, the inventors have discovered through research that the existing live video rendering technology has the problem of poor display effect. Summary of the invention

[0004] The purpose of the present application is to solve at least one of the above-mentioned technical defects, especially the technical defect of poor display effect in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a live video rendering method, which is applied to a stream pulling end device, and the method includes:

[0006] Receive live streaming data sent by a streaming device; wherein the live streaming data includes video encoding data and supplementary enhancement information, and the supplementary enhancement information carries a first timestamp, which is an NTP timestamp corresponding to when the supplementary enhancement information is added to the live streaming data;

[0007] According to the live streaming data, respectively obtain the first timestamp and the video frame data corresponding to the video encoding data;

[0008] Obtain a second timestamp, where the second timestamp is the NTP timestamp corresponding to the current moment;

[0009] Calculate the live broadcast delay between the stream pulling end device and the stream pushing end device according to the first timestamp and the second timestamp;

[0010] A target rendering rate is determined according to the live broadcast delay, and the live broadcast video is rendered based on the target rendering rate and the video frame data.

[0011] In some embodiments, determining a target rendering rate according to the live broadcast delay includes:

[0012] If the live broadcast delay is greater than a preset time threshold, the first rendering rate is used as the target rendering rate, otherwise, the second rendering rate is used as the target rendering rate; wherein the first rendering rate is greater than the second rendering rate.

[0013] In some embodiments, obtaining the first timestamp and the video frame data corresponding to the video encoding data respectively according to the live streaming data includes:

[0014] The live streaming data is parsed using a preset parsing algorithm, and a plurality of network abstraction layer units and a unit type corresponding to each of the network abstraction layer units are obtained;

[0015] Based on a preset video decoding algorithm and any first abstraction layer unit, the video frame data is obtained; wherein the first abstraction layer unit refers to a network abstraction layer unit whose unit type is a video information type;

[0016] The first timestamp is obtained based on a preset supplementary information parsing algorithm and any second abstraction layer unit; wherein the second abstraction layer unit refers to a network abstraction layer unit whose unit type is a supplementary enhancement information type.

[0017] In some embodiments, after obtaining the video frame data based on a preset video decoding algorithm and any first abstraction layer unit, the method further includes:

[0018] The video frame data is written into the video buffer of the stream pulling end device.

[0019] In some embodiments, the calculating, according to the first timestamp and the second timestamp, the live broadcast delay between the stream pulling end device and the stream pushing end device includes:

[0020] Calculate the difference between the second timestamp and the first timestamp, and use the difference as the live broadcast delay.

[0021] In a second aspect, an embodiment of the present application provides a live video rendering method, which is applied to a streaming end device, and the method includes:

[0022] Generate video encoding data according to the video frame data and a preset encoding algorithm;

[0023] Obtaining original bitstream data based on the video encoding data;

[0024] When the timing period arrives, a first timestamp is obtained, and supplementary enhancement information is generated according to the first timestamp, and the supplementary enhancement information is added to the original code stream data to obtain live code stream data; wherein the first timestamp is the NTP timestamp corresponding to the current moment;

[0025] The live stream data is sent to the stream pulling end device, so that the stream pulling end device renders the live video based on the live video rendering method applied to the stream pulling end device described in any of the above embodiments.

[0026] In some embodiments, the original bitstream data includes bitstream parameter set data;

[0027] The adding the supplemental enhancement information to the original bitstream data comprises:

[0028] Inputting the original code stream data into a bit stream filter;

[0029] The supplemental enhancement information is inserted after the bitstream parameter set data by using the bitstream filter.

[0030] In a third aspect, an embodiment of the present application provides a live video rendering device, which is applied to a stream pulling end device, and the device includes:

[0031] A code stream receiving module, used for receiving live code stream data sent by a streaming end device; wherein the live code stream data includes video encoding data and supplementary enhancement information, and the supplementary enhancement information carries a first timestamp, which is an NTP timestamp corresponding to when the supplementary enhancement information is added to the live code stream data;

[0032] A first timestamp acquisition module, used to obtain the first timestamp and the video frame data corresponding to the video encoding data according to the live streaming data;

[0033] A second timestamp acquisition module, used to acquire a second timestamp, where the second timestamp is an NTP timestamp corresponding to the current moment;

[0034] A live broadcast delay calculation module, used to calculate the live broadcast delay between the stream pulling end device and the stream pushing end device according to the first timestamp and the second timestamp;

[0035] A rendering module is used to determine a target rendering rate according to the live broadcast delay, and render the live video based on the target rendering rate and the video frame data.

[0036] In a fourth aspect, an embodiment of the present application provides a live video rendering device, which is applied to a streaming end device, and the device includes:

[0037] A video encoding module, used to generate video encoding data according to video frame data and a preset encoding algorithm;

[0038] An original bitstream generating module, used for obtaining original bitstream data based on the video encoding data;

[0039] A live stream generation module, used for obtaining a first timestamp when a timing period arrives, generating supplementary enhancement information according to the first timestamp, and adding the supplementary enhancement information to the original stream data to obtain live stream data; wherein the first timestamp is an NTP timestamp corresponding to the current moment;

[0040] The sending module is used to send the live stream data to the stream pulling end device, so that the stream pulling end device renders the live video based on the live video rendering method applied to the stream pulling end device described in any of the above embodiments.

[0041] In a fifth aspect, an embodiment of the present application provides a storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the live video rendering method described in any of the above embodiments.

[0042] In the live video rendering method, device and storage medium provided in some embodiments of the present application, the stream pulling end device can accurately calculate the live broadcast delay between the stream pulling end device and the stream pushing end device based on the first timestamp carried by the supplementary enhancement information in the live broadcast code stream data and the second timestamp corresponding to the current moment, and adjust the screen rendering of the live video based on the live broadcast delay, thereby improving the display effect of the live video. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 A schematic diagram of an implementation of a live video rendering method in one embodiment;

[0045] Figure 2 One of the flowcharts of the live video rendering method in one embodiment;

[0046] Figure 3 This is a second flow chart of a live video rendering method in one embodiment;

[0047] Figure 4 This is one of the structural schematic diagrams of a live video rendering device in one embodiment;

[0048] Figure 5 This is a second structural diagram of a live video rendering device in one embodiment. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] The terms involved in this application are explained below.

[0051] SEI (Supplemental Enhancement Information): One of the features of the H.264 / H.265 video compression standard, providing a method to add information to the video bitstream. SEI does not include image data information, but includes supplementary information to the image data information or the video stream.

[0052] Digital retina: A video and image perception system that can improve visual recognition and processing capabilities in areas such as smart cities and smart security by mimicking the workings of the human visual system. Digital retina technology can encode the captured video with high quality at the video acquisition device end, extract and encode the visual features of the video, and divide the video content into two data streams, video encoding stream and feature encoding stream, and transmit them to the back-end system. This not only ensures efficient video storage, but also supports the query and analysis of big data, and realizes rapid understanding and response to scenes.

[0053] NTP (Network Time Protocol): A protocol used to synchronize computer system clocks. Its function is to ensure that the computer system clocks can be synchronized with the global standard time so that timestamps and event records can be accurately performed in network communications and data transmission.

[0054] BSF (BitStream Filter): Make specific modifications and adjustments to the encoded bitstream without decoding the bitstream.

[0055] VCL (Video Coding Layer): responsible for video content processing, especially video encoding and decoding.

[0056] NAL (Network Abstraction Layer): responsible for packaging and transmitting the encoded data in the format required by the network to ensure that the data is suitable for transmission on the channel and storage medium.

[0057] NALU (NAL Unit): The basic syntax of NAL. Taking H.264 as an example, the original bitstream consists of multiple NALUs. Each NALU includes a NAL header and a raw data byte stream (RBSP) from VCL.

[0058] SODB (String of Data Bits): The original encoded data, i.e. VCL data, without any additional data.

[0059] RBSP (Raw Byte Sequence Payload): A byte sequence with trailing bits (RBSP_trailing_bits) added after the SODB data. The trailing bits may include one bit of "1" or several bits of "0" to achieve byte alignment.

[0060] SPS (Sequence Parameter Set): Parameters for a continuously encoded video sequence, including but not limited to the number of video frames, number of reference frames, decoded image size and other information.

[0061] PPS (Picture Parameter Set): Parameters for a specific image, including but not limited to the number of slices, initial quantization parameters, filter coefficients, and other information.

[0062] The following describes the application scenarios of the live video rendering method provided by this application.

[0063] Please combine Figure 1 The live video rendering method provided in this application can be applied to Figure 1 The application environment shown. Among them, the streaming end device can collect video frame data through a video acquisition device, and use a preset encoding algorithm to encode the collected video frame data to obtain video encoding data. The streaming end device can also generate original code stream data based on the video encoding data. For example, the streaming end device can generate corresponding original code stream data according to the encoding format H.264 / H.265. It can be understood that the original code stream data of this application refers to code stream data that does not include supplementary enhancement information SEI.

[0064] The streaming device can insert SEI frames into the original bitstream data at regular intervals, and the bitstream data into which the SEI frames are inserted is the live bitstream data mentioned in the embodiment of the present application. The SEI frame carries a first timestamp, which refers to the NTP timestamp corresponding to when the streaming device inserts the SEI frame into the original bitstream data.

[0065] For example, at 14:30, the streaming device needs to insert an SEI frame into the original bitstream data. In this case, the streaming device can obtain the NTP timestamp Timestamp1 corresponding to 14:30, and generate an SEI frame based on it, so that the SEI frame carries Timestamp1. The streaming device can insert the SEI frame carrying Timestamp1 into the original bitstream data to obtain the live bitstream data.

[0066] The streaming device can send live streaming data to the streaming device based on wireless networks, streaming media services and other technologies.

[0067] When receiving the live stream data, the stream-pulling device can call a specific function to parse the live stream data. If the SEI frame is extracted from the live stream data, the stream-pulling device can parse the SEI frame according to the SEI syntax standard to obtain the first timestamp Timestamp1 carried by the SEI frame. In addition, the stream-pulling device can obtain the NTP timestamp corresponding to the current moment and obtain the second timestamp Timestamp2. The stream-pulling device can calculate the live delay based on Timestamp1 and Timestamp2, and adjust the video rendering according to the live delay.

[0068] For example, if the first timestamp carried by the SEI frame is the NTP timestamp corresponding to 14:30, and the second timestamp corresponding to the current time is the NTP timestamp corresponding to 14:32, the stream pulling end device can determine that the live broadcast delay is 2 seconds based on the first timestamp and the second timestamp.

[0069] In this way, the live broadcast delay between the push-stream device and the pull-stream device can be calculated and quantified more accurately, so that the pull-stream device can more accurately adjust the rendering rate of the live video based on the calculated live broadcast delay, making the live video picture displayed by the pull-stream device smoother and with less delay.

[0070] In some embodiments, the video rendering method provided by the present application can be combined with digital retina technology and video control rendering technology, that is, the push stream end device can generate two code streams, video coding stream and feature coding stream, based on digital retina technology, and use the two code streams as original code stream data. This can improve the perception and cognition of data, and can achieve efficient storage and convenient big data analysis and query.

[0071] Through the combination of the two technologies, not only can efficient video data processing and intelligent analysis be achieved through digital retina technology, but the video display effect can also be improved through video control and rendering technology, making the live video picture clearer and more detailed, thereby further improving the display effect.

[0072] The live video rendering method provided in an embodiment of the present application is described below.

[0073] In some embodiments, Figure 2 As shown, the present application provides a live video rendering method applied to a stream pulling end device, which specifically includes the following steps:

[0074] S202: receiving live streaming data sent by a streaming device;

[0075] S204: Obtaining video frame data corresponding to the first timestamp and the video encoding data respectively according to the live streaming data;

[0076] S206: Obtain a second timestamp, where the second timestamp is the NTP timestamp corresponding to the current moment;

[0077] S208: Calculate the live broadcast delay between the stream pulling end device and the stream pushing end device according to the first timestamp and the second timestamp;

[0078] S210: Determine a target rendering rate according to the live broadcast delay, and render the live broadcast video based on the target rendering rate and the video frame data.

[0079] The live stream data includes video encoding data and supplementary enhancement information, and the supplementary enhancement information carries a first timestamp, which is the NTP timestamp corresponding to when the push-stream end device adds the supplementary enhancement information to the live stream data. The video encoding data may be data obtained by encoding the video frame data. For the description of the live stream data, supplementary enhancement information and the first timestamp, please refer to the above text, which will not be repeated in this article.

[0080] The stream pulling end device can receive the live stream data so as to render the live video according to the live stream data. It should be noted that the stream pulling end device can receive the live stream data in any manner, and this document does not impose any specific restrictions on this. For example, the stream pulling end device can receive the live stream data via a wireless network.

[0081] Since the live stream data includes supplementary enhancement information and video encoding data, and the supplementary enhancement information carries the first timestamp, and the video encoding data is data obtained after encoding the video frame data, the stream pulling end device can obtain the first timestamp and the video frame data according to the live stream data. When the first timestamp is obtained, the stream pulling end device can obtain the NTP timestamp corresponding to the current moment and obtain the second timestamp.

[0082] Since the first timestamp is the NTP timestamp corresponding to when the streaming device inserts the supplementary enhancement information into the bitstream, and the second timestamp is the NTP timestamp corresponding to when the streaming device obtains the supplementary enhancement information from the bitstream, the streaming device can calculate the live broadcast delay between the streaming device and the streaming device based on the first timestamp and the second timestamp.

[0083] It should be noted that the streaming end device can calculate the live broadcast delay in any manner. For example, the streaming end device can be pre-set with a device calculation duration, and calculate the live broadcast delay based on the device calculation duration, the first timestamp, and the second timestamp.

[0084] To improve processing efficiency, in one example, the stream pulling end device may calculate the difference between the second timestamp and the first timestamp, and use the difference as the live broadcast delay.

[0085] When the live broadcast delay is determined, the streaming device can determine the target rendering rate according to the live broadcast delay, and render the live video according to the target rendering rate. In this way, the video rendering can be adjusted according to the live broadcast delay, thereby improving the display effect of the live video.

[0086] In some embodiments, determining a target rendering rate according to a live broadcast delay includes:

[0087] If the live broadcast delay is greater than the preset time threshold, the first rendering rate is used as the target rendering rate, otherwise, the second rendering rate is used as the target rendering rate; wherein the first rendering rate is greater than the second rendering rate.

[0088] The specific value of the preset time threshold can be determined according to actual conditions, and this document does not impose any specific restrictions on this. In one example, the preset time threshold can be 100 milliseconds.

[0089] Specifically, if the live broadcast delay between the push-stream device and the pull-stream device is less than or equal to the preset time threshold, the pull-stream device can render the video at a normal rendering rate. If the live broadcast delay between the push-stream device and the pull-stream device is greater than the preset time threshold, the pull-stream device can accelerate the rendering until the acquired video frame data is consumed. In this way, the display quality of the image can be improved, the jaggedness of the image can be reduced, the picture can be made smoother, more natural and clear, and the picture details can be richer, thereby further improving the display effect.

[0090] Furthermore, in some embodiments, the live streaming data may be generated based on digital retina technology. Thus, by combining digital retina technology and video control and rendering technology, not only can efficient video data processing and intelligent analysis be achieved, but also the display effect of live video can be improved, making the monitoring screen clearer and more detailed.

[0091] In some embodiments, obtaining the first timestamp and the video frame data corresponding to the video encoding data respectively according to the live stream data includes:

[0092] Step A1: using a preset parsing algorithm to parse the live stream data, and obtaining a plurality of network abstraction layer units and a unit type corresponding to each network abstraction layer unit;

[0093] Step A3: obtaining video frame data based on a preset video decoding algorithm and any first abstraction layer unit; wherein the first abstraction layer unit refers to a network abstraction layer unit whose unit type is a video information type;

[0094] Step A5: obtaining a first timestamp based on a preset supplementary information parsing algorithm and any second abstraction layer unit; wherein the second abstraction layer unit refers to a network abstraction layer unit whose unit type is a supplementary enhancement information type.

[0095] In this embodiment, the stream pulling end device can use a preset parsing algorithm to parse the live stream data, and extract multiple NALUs (network abstraction layer units) from the live stream data, and determine the unit type corresponding to each NALU. According to the unit type corresponding to each NALU, the stream pulling end device can divide each NALU into a first abstraction layer unit and a second abstraction layer unit. Among them, the first abstraction layer unit refers to the NALU corresponding to the video encoding data, which can be but is not limited to the NALU of the unit type SPS, PPS or video frame image. The second abstract unit refers to the NALU of the unit type SEI frame.

[0096] The stream pulling end device can read and decode data from any first abstract layer unit according to a preset video decoding algorithm, and obtain video frame data that can be used for video rendering based on the decoding result. It can be understood that in this application, the stream pulling end device can use any method to obtain video frame data, and this article does not make specific restrictions on this.

[0097] In one example, after obtaining the video frame data, the stream pulling end device may write the video frame data into a video buffer of the stream pulling end device for subsequent calls.

[0098] For any second abstraction layer unit, the stream pulling end device may parse the second abstraction layer unit based on a preset SEI syntax standard to extract the first timestamp from the second abstraction layer unit.

[0099] In some embodiments, Figure 3 As shown, the present application provides a live video rendering method applied to a streaming end device, which specifically includes the following steps:

[0100] S302: Generate video encoding data according to the video frame data and a preset encoding algorithm;

[0101] S304: Obtaining original bitstream data based on the video encoding data;

[0102] S306: When the timing period arrives, obtain the first timestamp, generate supplementary enhancement information according to the first timestamp, and add the supplementary enhancement information to the original bitstream data to obtain live bitstream data;

[0103] S308: Sending live stream data to the stream pulling end device, so that the stream pulling end device renders the live video based on the live video rendering method applied to the stream pulling end device provided in any of the above embodiments.

[0104] The video frame data may be the video frame data acquired by the video acquisition device, for example, may include key frames (I frames), forward prediction frames (P frames) and bidirectional interpolation frames (B frames). The first timestamp is the NTP timestamp corresponding to the current moment, that is, the NTP timestamp corresponding to when the supplementary enhancement information is generated and inserted into the original bitstream data.

[0105] In this embodiment, the streaming end device can generate video encoding data according to the video frame data and the preset encoding algorithm, and generate original bitstream data accordingly. In addition, the streaming end device can periodically insert supplementary enhancement information into the original bitstream data according to the preset timing cycle, and obtain live bitstream data.

[0106] For example, when the timing cycle is 1 minute, the streaming end device can generate the first supplementary enhancement information according to the NTP timestamp corresponding to 1:00 when 1:00 arrives, and insert the first supplementary enhancement information into the bitstream data. When 2:00 arrives, the streaming end device generates the second supplementary enhancement information according to the NTP timestamp corresponding to 2:00, and inserts the second supplementary enhancement information into the bitstream data. When 3:00 arrives, the streaming end device generates the third supplementary enhancement information according to the NTP timestamp corresponding to 3:00, and inserts the third supplementary enhancement information into the bitstream data, and so on.

[0107] The push-stream device can send live stream data to the pull-stream device, so that the pull-stream device can accurately calculate the live delay based on the supplementary enhancement information in the live stream data, and perform video rendering control accordingly.

[0108] It should be noted that the push-stream device can continuously send live stream data to the pull-stream device, and the sending and receiving of live stream data may not depend on the insertion of supplementary enhancement information. When the timing period does not arrive, the push-stream device is not triggered to insert supplementary enhancement information, and the push-stream device can also send live stream data to the pull-stream device.

[0109] In some embodiments, the original code stream data includes code stream parameter set data. For example, the code stream parameter set data may be SPS data or PPS data.

[0110] Add supplementary enhancement information to the original bitstream data, including:

[0111] Input the original code stream data into the bit stream filter;

[0112] The supplementary enhancement information is inserted after the bitstream parameter set data using a bitstream filter.

[0113] In this embodiment, the streaming end device can insert the supplementary enhancement information through the bitstream filter, so that the information insertion can be realized without decoding the original bitstream data. At the same time, the streaming end device can add the supplementary enhancement information to the SPS data / PPS data before the I frame. That is, in the live bitstream data, the SPS data / PPS data and the supplementary enhancement information can be arranged in sequence.

[0114] The live video rendering device provided in an embodiment of the present application is described below. The live video rendering device described below and the live video rendering method described above can refer to each other.

[0115] In some embodiments, the present application provides a live video rendering device 400, which can be applied to a streaming end device. Figure 4 As shown, the device 400 includes:

[0116] The code stream receiving module 402 is used to receive live code stream data sent by the streaming end device; wherein the live code stream data includes video encoding data and supplementary enhancement information, and the supplementary enhancement information carries a first timestamp, which is the NTP timestamp corresponding to when the supplementary enhancement information is added to the live code stream data;

[0117] A first timestamp acquisition module 404 is used to obtain the first timestamp and the video frame data corresponding to the video encoding data according to the live streaming data;

[0118] A second timestamp acquisition module 406 is used to acquire a second timestamp, where the second timestamp is an NTP timestamp corresponding to the current moment;

[0119] A live broadcast delay calculation module 408, configured to calculate a live broadcast delay between the stream pulling end device and the stream pushing end device according to the first timestamp and the second timestamp;

[0120] The rendering module 410 is used to determine a target rendering rate according to the live broadcast delay, and render the live video based on the target rendering rate and the video frame data.

[0121] In some embodiments, the rendering module 410 of the present application includes:

[0122] A rendering rate determination unit is used to use the first rendering rate as the target rendering rate if the live broadcast delay is greater than a preset time threshold, otherwise, use the second rendering rate as the target rendering rate; wherein the first rendering rate is greater than the second rendering rate.

[0123] In some embodiments, the first timestamp acquisition module 404 of the present application includes:

[0124] A data parsing unit, used to parse the live stream data using a preset parsing algorithm, and obtain a plurality of network abstraction layer units and a unit type corresponding to each of the network abstraction layer units;

[0125] A video decoding unit, configured to obtain the video frame data based on a preset video decoding algorithm and any first abstraction layer unit; wherein the first abstraction layer unit refers to a network abstraction layer unit whose unit type is a video information type;

[0126] A timestamp parsing unit is used to obtain the first timestamp based on a preset supplementary information parsing algorithm and any second abstraction layer unit; wherein the second abstraction layer unit refers to a network abstraction layer unit whose unit type is a supplementary enhancement information type.

[0127] In some embodiments, the live video rendering device 400 of the present application further includes:

[0128] The data buffer module is used to write the video frame data into the video buffer of the stream pulling end device.

[0129] In some embodiments, the live broadcast delay calculation module 408 of the present application includes:

[0130] A delay calculation unit is used to calculate the difference between the second timestamp and the first timestamp, and use the difference as the live broadcast delay.

[0131] In some embodiments, the present application provides a live video rendering device 500, which can be applied to a streaming end device. Figure 5 As shown, the device 500 includes:

[0132] The video encoding module 502 is used to generate video encoding data according to the video frame data and a preset encoding algorithm;

[0133] The original bitstream generating module 504 is used to obtain the original bitstream data based on the video encoding data;

[0134] The live stream generating module 506 is used to obtain a first timestamp when the timing period arrives, generate supplementary enhancement information according to the first timestamp, and add the supplementary enhancement information to the original stream data to obtain the live stream data; wherein the first timestamp is the NTP timestamp corresponding to the current moment;

[0135] The sending module 508 is used to send the live stream data to the stream pulling end device, so that the stream pulling end device renders the live video based on the live video rendering method applied to the stream pulling end device described in any of the above embodiments.

[0136] In some embodiments, the original bitstream data includes bitstream parameter set data. The live bitstream generation module 506 of the present application includes:

[0137] An input unit, used for inputting the original code stream data into a bit stream filter;

[0138] The information inserting unit is used to insert the supplementary enhancement information after the bitstream parameter set data by using the bitstream filter.

[0139] In one embodiment, the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the live video rendering method in any embodiment.

[0140] In one embodiment, the present application also provides a stream pulling end device, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the live video rendering method applied to the stream pulling end device in any embodiment.

[0141] Exemplarily, the stream-pulling end device of the present application may be, but is not limited to, a smart phone, a tablet computer, a laptop computer, a desktop computer, a wearable device, an IoT device, etc. This document does not impose any specific restrictions on this, and it only needs to have data receiving and video rendering functions.

[0142] In one embodiment, the present application also provides a streaming end device, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the live video rendering method applied to the streaming end device in any embodiment.

[0143] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. Herein, "one", "one", "said", "the" and "it" may also include plural forms, unless the context clearly indicates another way. A plurality refers to at least two cases, such as 2, 3, 5 or 8, etc. "And / or" includes any and all combinations of the relevant listed items.

[0144] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0145] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A live video rendering method, characterized in that: Applied to the flow-pulling end device, the method comprises: Receive live streaming data sent by a streaming device; wherein the live streaming data includes video encoding data and supplementary enhancement information, and the supplementary enhancement information carries a first timestamp, which is an NTP timestamp corresponding to when the supplementary enhancement information is added to the live streaming data; According to the live streaming data, respectively obtain the first timestamp and the video frame data corresponding to the video encoding data; Obtain a second timestamp, where the second timestamp is the NTP timestamp corresponding to the current moment; Calculate the live broadcast delay between the stream pulling end device and the stream pushing end device according to the first timestamp and the second timestamp; A target rendering rate is determined according to the live broadcast delay, and the live broadcast video is rendered based on the target rendering rate and the video frame data.

2. The method according to claim 1, characterized in that The determining a target rendering rate according to the live broadcast delay includes: If the live broadcast delay is greater than a preset time threshold, the first rendering rate is used as the target rendering rate, otherwise, the second rendering rate is used as the target rendering rate; wherein the first rendering rate is greater than the second rendering rate.

3. The method according to claim 1 or 2, characterized in that: The obtaining, according to the live stream data, the first timestamp and the video frame data corresponding to the video encoding data respectively includes: The live streaming data is parsed using a preset parsing algorithm, and a plurality of network abstraction layer units and a unit type corresponding to each of the network abstraction layer units are obtained; Based on a preset video decoding algorithm and any first abstraction layer unit, the video frame data is obtained; wherein the first abstraction layer unit refers to a network abstraction layer unit whose unit type is a video information type; The first timestamp is obtained based on a preset supplementary information parsing algorithm and any second abstraction layer unit; wherein the second abstraction layer unit refers to a network abstraction layer unit whose unit type is a supplementary enhancement information type.

4. The method according to claim 3, characterized in that After obtaining the video frame data based on the preset video decoding algorithm and any first abstraction layer unit, the method further includes: The video frame data is written into the video buffer of the stream pulling end device.

5. The method according to claim 1 or 2, characterized in that: The calculating, according to the first timestamp and the second timestamp, the live broadcast delay between the stream pulling end device and the stream pushing end device comprises: Calculate the difference between the second timestamp and the first timestamp, and use the difference as the live broadcast delay.

6. A live video rendering method, characterized in that: Applied to a streaming end device, the method comprises: Generate video encoding data according to the video frame data and a preset encoding algorithm; Obtaining original bitstream data based on the video encoding data; When the timing period arrives, a first timestamp is obtained, and supplementary enhancement information is generated according to the first timestamp, and the supplementary enhancement information is added to the original code stream data to obtain live code stream data; wherein the first timestamp is the NTP timestamp corresponding to the current moment; The live stream data is sent to a stream pulling end device, so that the stream pulling end device renders the live video based on the live video rendering method according to any one of claims 1 to 5.

7. The method according to claim 6, characterized in that The original code stream data includes code stream parameter set data; The adding the supplemental enhancement information to the original bitstream data comprises: Inputting the original code stream data into a bit stream filter; The supplemental enhancement information is inserted after the bitstream parameter set data by using the bitstream filter.

8. A live video rendering device, characterized in that: Applied to the flow-pulling end equipment, the device comprises: A code stream receiving module, used for receiving live code stream data sent by a streaming end device; wherein the live code stream data includes video encoding data and supplementary enhancement information, and the supplementary enhancement information carries a first timestamp, which is an NTP timestamp corresponding to when the supplementary enhancement information is added to the live code stream data; A first timestamp acquisition module, used to obtain the first timestamp and the video frame data corresponding to the video encoding data according to the live streaming data; A second timestamp acquisition module, used to acquire a second timestamp, where the second timestamp is an NTP timestamp corresponding to the current moment; A live broadcast delay calculation module, used to calculate the live broadcast delay between the stream pulling end device and the stream pushing end device according to the first timestamp and the second timestamp; A rendering module is used to determine a target rendering rate according to the live broadcast delay, and render the live video based on the target rendering rate and the video frame data.

9. A live video rendering device, characterized in that: Applied to the flow-pushing end equipment, the device comprises: A video encoding module, used to generate video encoding data according to video frame data and a preset encoding algorithm; An original bitstream generating module, used for obtaining original bitstream data based on the video encoding data; A live stream generation module, used for obtaining a first timestamp when a timing period arrives, generating supplementary enhancement information according to the first timestamp, and adding the supplementary enhancement information to the original stream data to obtain live stream data; wherein the first timestamp is an NTP timestamp corresponding to the current moment; A sending module is used to send the live stream data to a stream pulling end device, so that the stream pulling end device renders the live video based on the live video rendering method according to any one of claims 1 to 5.

10. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the live video rendering method as described in any one of claims 1 to 7.

Images