Live video processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a live video processing method and device, equipment, a storage medium and a program product, and relates to the technical field of streaming media. The method is applied to playing end equipment, and comprises the following steps: separating audio and video data in a cache, marking key video frames in the audio and video data, and analyzing supplementary enhancement information in the audio and video data; the supplementary enhancement information at least comprises synchronous clock information, and the synchronous clock information represents the time when the anchor end equipment collects the audio and video data; adjusting a cache control state according to the supplementary enhancement information; the cache control state is used for indicating to cache or discard audio and video data sent by the content distribution network server; processing the audio and video data according to the current video first frame state, and decoding and playing the processed audio and video data; the video first-frame state is used for indicating a first-frame state or a first-frame-free state. The method is used for achieving the effects of reducing the live broadcast delay and avoiding the influence of large delay on the online interaction effect.

Description

Technical Field

[0001] This application relates to the field of streaming media technology, and in particular, to a method, apparatus, device, storage medium, and program product for processing live video. Background Art

[0002] With the rapid development of Internet technology and the popularization of smart devices, video live streaming has become an important social and entertainment method. Users can watch the content shared by the anchor in real time through terminal devices and experience the fun of real-time interaction.

[0003] In the prior art, the processing of live video usually involves multiple links such as the acquisition, encoding, transmission, decoding, and playback of audio and video data. To ensure the smoothness and stability of playback, it is usually necessary to perform caching processing on the audio and video data. However, caching will increase the latency, making the time difference between the audio and video data at the playback end and the anchor end farther and farther, affecting the interactive experience of the live stream. Especially in the live online open classes with tens of thousands of people attending classes at the same time, the caching mechanism of ordinary live streams usually cannot meet the real-time and interactive teaching requirements.

[0004] Therefore, there is an urgent need for a more reasonable caching processing mechanism to reduce the live latency and avoid the large latency from affecting the online interactive teaching effect. Summary of the Invention

[0005] The method, apparatus, device, storage medium, and program product for processing live video provided in the embodiments of this application are used to achieve the effect of reducing the live latency and avoiding the large latency from affecting the online interactive effect.

[0006] In a first aspect, the embodiments of this application provide a method for processing live video. The method is applied to a playback end device, and the method includes:

[0007] Separate the audio and video data in the cache, mark the key video frames in the audio and video data, and parse out the supplementary enhancement information in the audio and video data; at least the supplementary enhancement information includes synchronous clock information, and the synchronous clock information represents the time when the anchor end device collects the audio and video data;

[0008] Adjust the cache control state according to the supplementary enhancement information; the cache control state is used to indicate whether to cache or discard the audio and video data sent by the content delivery network server;

[0009] Process the audio and video data according to the current video first frame state, and decode and play the processed audio and video data; the video first frame state is used to indicate the presence or absence of the first frame state.

[0010] In a possible implementation manner, the adjusting the cache control state according to the supplementary enhancement information includes:

[0011] Determine delay information according to the system clock information and the synchronous clock information in the supplementary enhancement information;

[0012] If it is determined that the delay information is greater than a first preset threshold, adjust the cache control state to prohibit caching;

[0013] If it is determined that the delay information is less than or equal to the first preset threshold, adjust the cache control state to allow caching.

[0014] In a possible implementation manner, when it is determined that the delay information is greater than the first preset threshold, the method further includes:

[0015] Determine and control the duration of prohibited caching according to the delay information and the target delay.

[0016] In a possible implementation manner, if the supplementary enhancement information further includes fast forward instruction information, after determining the delay information, the method further includes:

[0017] If it is determined that the delay information is greater than a second preset threshold, discard non-key video frames in the audio-visual data and the audio data corresponding to the non-key video frames; wherein, the second preset threshold is less than the first preset threshold;

[0018] If it is determined that the delay information is less than or equal to the second preset threshold, do not execute the fast forward instruction information.

[0019] In a possible implementation manner, the discarding of non-key video frames in the audio-visual data and the audio data corresponding to the non-key video frames includes:

[0020] Determine the difference between the delay information and the second preset threshold, and determine the number of non-key video frames to be discarded in the audio-visual data according to the difference;

[0021] Discard non-key video frames in the audio-visual data and the audio data corresponding to the non-key video frames according to the number of discarded frames.

[0022] In a possible implementation manner, the processing of the audio-visual data according to the current video first frame state includes:

[0023] If the current video first frame state indicates that the current is a state without a first frame, discard non-key video frames in the audio-visual data until a key video frame is encountered, and then adjust the video first frame state to a state with a first frame.

[0024] In a possible implementation manner, the decoding and playing of the processed audio-visual data includes:

[0025] Put the video data in the processed audio - video data into the video queue, and put the audio data in the processed audio - video data into the audio queue;

[0026] Perform video decoding processing on the video data, perform audio decoding processing on the audio data, display the video data and play the audio data synchronously.

[0027] In a possible implementation manner, before separating the audio - video data in the cache, the method further includes:

[0028] Establish a network connection between the playback device and the content delivery network server;

[0029] Receive and parse the audio - video data sent by the content delivery network server, and process the audio - video data according to the cache control status.

[0030] In a second aspect, an embodiment of the present application provides a processing device for live video. The device is applied to a playback device, and the device includes:

[0031] A first processing unit, configured to separate the audio - video data in the cache, mark the key video frames in the audio - video data, and parse the supplementary enhancement information in the audio - video data; at least the supplementary enhancement information includes synchronous clock information, and the synchronous clock information represents the time when the host device collects the audio - video data;

[0032] A second processing unit, configured to adjust the cache control status according to the supplementary enhancement information; the cache control status is used to indicate caching or discarding the audio - video data sent by the content delivery network server;

[0033] A third processing unit, configured to process the audio - video data according to the current video first - frame state, and decode and play the processed audio - video data; the video first - frame state is used to indicate the presence or absence of the first - frame state.

[0034] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0035] The memory stores computer - executable instructions;

[0036] The processor executes the computer - executable instructions stored in the memory, so that the processor executes the above - mentioned first aspect and / or various possible implementation manners of the first aspect.

[0037] In a fourth aspect, an embodiment of the present application provides a computer - readable storage medium, in which computer - executable instructions are stored. When the computer - executable instructions are executed by a processor, they are used to implement the above - mentioned first aspect and / or various possible implementation manners of the first aspect.

[0038] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which when executed by a processor, implements the above first aspect and / or various possible implementation manners of the first aspect.

[0039] The method, apparatus, device, storage medium and program product for processing live video provided by the embodiments of the present application achieve precise time synchronization of the audio-visual stream by parsing the supplementary enhancement information in the audio-visual data, especially the synchronous clock information, enabling the playback end to accurately understand the time when the host end captures the audio-visual data. Thus, according to the synchronous clock information, the cache control state can be adjusted more effectively, the cache policy can be dynamically optimized, and it is possible to more flexibly decide whether to cache or discard data during network fluctuations, reducing unnecessary data backlog and processing delay. In addition, processing the audio-visual data according to the state of the first video frame ensures that video decoding starts from a valid key frame and also avoids the problem of screen freeze caused by the lack of key frames. These measures work together to significantly reduce the end-to-end delay of live video and provide users with a more real-time viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application and used together with the specification to explain the principles of the present application.

[0041] Figure 1 It is a schematic framework diagram of a live video processing process provided by an embodiment of the present application;

[0042] Figure 2 It is a schematic framework diagram of a streaming process provided by an embodiment of the present application;

[0043] Figure 3 It is a schematic framework diagram of a CDN server processing process provided by an embodiment of the present application;

[0044] Figure 4 It is a schematic flowchart of a method for processing live video provided by an embodiment of the present application;

[0045] Figure 5 It is a schematic framework diagram of a pulling process provided by an embodiment of the present application;

[0046] Figure 6 It is a schematic structural diagram of a device for processing live video provided by an embodiment of the present application;

[0047] Figure 7 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

[0048] Through the above-mentioned drawings, specific embodiments of the present application have been shown, and more detailed descriptions will be provided hereinafter. These drawings and written descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Detailed Description of the Embodiments

[0049] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0050] An open class is an online teaching form that supports tens of thousands of people attending classes simultaneously. The main technical means is in the form of live streaming. Through a Content Delivery Network (CDN) server, the live video of the lecturer is distributed to everyone in the online classroom. Among them, the live video may include audio data such as the voice of the lecturer and video data such as the image of the lecturer, courseware materials, and other information. The purpose of live teaching is to enable more people to receive professional guidance and interactive teaching from excellent teachers without being restricted by geographical location, enhance the learning effect, and save learning costs.

[0051] Exemplarily, Figure 1 It is a schematic framework diagram of a live video processing process provided for an embodiment of the present application. As Figure 1As shown in the figure, the audio and video data generated by the host during the live broadcast will be processed by the streaming client and sent to the CDN server in real time in the form of streaming. Due to technical reasons, a fixed delay related to the configuration will be generated during this process, which can be marked as Delay1; after receiving the audio and video data, the CDN server will cache the data according to the pre-configured caching policy or distribute it to other associated CDN server nodes level by level. A fixed delay related to the configuration will also be generated during this process, which can be marked as Delay2; the pulling client connects to the CDN server node closest to itself, obtains the audio and video data pre-cached by the CDN server, and decodes and plays it. This process will not only generate a fixed delay related to the configuration but also an adjustable delay related to the caching policy, which can be marked as Delay3. The live broadcast process is to continuously repeat the process of "streaming - CDN - pulling". The audience can continuously enjoy the pictures and sounds from the host during this process. In addition, due to the uncertainty of network transmission and the time required for data transmission, a delay related to the distance between the host and the audience will also be generated during the live broadcast process, which can be marked as Delay0. Among them, the streaming client corresponds to the host device of the live teacher, and the pulling client corresponds to the playback device of the audience.

[0052] It can be seen that due to the uncertainty of network transmission, the delay in the live broadcast process at least includes the delay Delay0 related to network transmission, the delay Delay1 related to the streaming end, the delay Delay2 related to the CDN server configuration, and the delay Delay3 related to the pulling end. When evaluating how large the overall live broadcast delay is, the following formula (1) can be referred to for calculation:

[0053] Delay = Delay0 + Delay1 + Delay2 + Delay3 (1)

[0054] Among them, the delay Delay1 related to streaming, the delay Delay2 related to the CDN server configuration, and the delay Delay3 related to pulling are mainly caused by caching. Although caching can improve the smoothness of network transmission, the cost of caching is to increase the delay. Currently, the general live broadcast delay is usually 3 to 10 seconds, and sometimes even larger. These delays are usually beneficial in the traditional live broadcast process, but they are not suitable in the open class scenario with relatively high requirements for real-time performance. It usually cannot meet the real-time and interactive teaching requirements. Therefore, it is necessary to reduce the delay in live teaching to avoid excessive delay affecting the online interactive teaching effect.

[0055] Exemplarily, Figure 2 is a schematic framework diagram of a streaming process provided by an embodiment of the present application. As Figure 2As shown, when the host is conducting a live broadcast, the camera on the host device captures video, which incurs a fixed delay Video1, and the microphone captures audio, which incurs a fixed delay Audio1. Converting the video image into the live broadcast encoding format through video encoding incurs a fixed delay Video2, and converting the audio signal into the live broadcast encoding format through audio encoding incurs a fixed delay Audio2. Mixing and encapsulating the encoded video data and audio data into the format required by the live broadcast protocol incurs a fixed delay Push1, and connecting the host device to the CDN server incurs a fixed delay Push2. When the network anomaly causes the connection to be disconnected, reconnecting to the CDN server also incurs a fixed delay Push3. Among them, when there is no disconnection and reconnection, the delay Push3 is 0. When there are multiple disconnections and reconnections, the delay Push3 needs to be accumulated. When the host device sends data, it incurs a delay Push4 due to network influence. Based on this, the push stream delay Delay1 can be calculated with reference to the following formula (2):

[0056] Delay1 = MAX(Video1 + Video2, Audio1 + Video2) + Push1 + Push2 + SUM(Push3) + Push4 (2)

[0057] It can be understood that the acquisition and encoding of video data and the acquisition and encoding of audio data are carried out simultaneously. Therefore, when calculating the push stream delay Delay1, the one with the larger delay between the two is selected for calculation.

[0058] Exemplarily, Figure 3 is a schematic framework diagram of a CDN server processing process provided by an embodiment of the present application. As Figure 3 shown, the audio and video data sent by the host device will reach the media center of the CDN server through the upstream stream of the CDN server, and then be sent down to the downstream stream of the CDN server, and then be pulled and played by the playback device of the audience. Although the operators of each CDN server have their own deployment architectures, they are all roughly the same. In the live broadcast push stream scenario, the delay of the CDN server can be configured according to customer requirements. When it is necessary to be compatible with CDN server products of different manufacturers, it can be assumed that the CDN server always generates a fixed delay Delay2.

[0059] Optionally, in order to reduce the delay Delay2, the CDN server cache can be configured to only retain a group of video frame sequence caches, that is, the cache content starts caching from the most recent key frame until it encounters another new key frame and ends.

[0060] In summary, it can be seen from the above analysis that both the push delay Delay1 and the delay Delay2 of the CDN server are related to the configuration. When the configuration cannot be changed, in order to further reduce the delay, the embodiments of the present application consider controlling from the pull client to reduce the delay.

[0061] In the live video processing method provided by the present application, when the playback device receives the audio and video data sent by the CDN server, it will cache or discard the audio and video data according to the cache control state, and the cache control state is determined by the current delay size or the fast forward instruction. At the same time, it also processes the audio and video data according to the presence or absence of the first frame state; finally, it decodes and plays, thereby realizing more flexible control of the cache situation, which can not only meet the delay requirements but also improve the live broadcast effect.

[0062] The technical solution of the present application and how the technical solution of the present application solves the above technical problems will be described in detail below with specific embodiments. These specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0063] Figure 4 It is a schematic flowchart of a live video processing method provided by an embodiment of the present application. As Figure 4 shown, the live video processing method provided by the embodiment of the present application is applied to a playback device, and the method includes:

[0064] S401. Separate the audio and video data in the cache, mark the key video frames in the audio and video data, and parse the supplementary enhancement information in the audio and video data; the supplementary enhancement information includes at least synchronous clock information, and the synchronous clock information represents the time when the host device collects the audio and video data.

[0065] Exemplarily, in a live broadcast, the playback device is responsible for playing the live video of the host for the audience at the pull end. In order to improve the smoothness and stability of the playback, the playback device usually performs cache processing on the audio and video data. At the same time, in order to reduce the delay caused by caching, a more reasonable cache mechanism is required for cache control of the audio and video data. In the embodiment of the present application, the initial cache control state of the playback device is usually allowed to cache. After receiving the audio and video data sent by the CDN server for the first time, it will cache and process the data.

[0066] In a live stream, audio data and video data are usually transmitted mixed together, and the playback device needs to separate these two types of data for separate processing. This separation is usually achieved by parsing the container format of the stream, such as the streaming media format (Flash Video, FLV), etc. After separating the audio and video data in the buffer of the playback device, the key video frames in the audio and video data can be marked. Among them, the video data consists of a series of frames, and the key frame (I-frame) is a frame that can be decoded independently. Different from other frames (such as P-frames and B-frames), the I-frame does not depend on the data of other frames. In practical applications, the frequency of key video frames can be set on the host device. For example, a key frame can be set every 20 video frames, etc.

[0067] In addition, the playback device can also parse out the Supplemental Enhancement Information (SEI) in the audio and video data. The Supplemental Enhancement Information is text data inserted into the audio and video stream to convey additional information, which can be precisely synchronized with the relevant audio and video content and reach the client through the transmission link. In the embodiments of the present application, the Supplemental Enhancement Information at least includes synchronization clock information, and the synchronization clock information refers to the time when the host device collects the audio and video data. Through the synchronization clock information, the playback device can not only ensure that the audio data and video data are synchronized during playback, but also be used to adjust the cache control state.

[0068] S402. Adjust the cache control state according to the Supplemental Enhancement Information; the cache control state is used to indicate whether to cache or discard the audio and video data sent by the content delivery network server.

[0069] Exemplarily, the content delivery network server is also the CDN server. When the cache control state is allowed to cache, the playback device will cache the audio and video data sent by the CDN server. When the cache control state is prohibited from caching, the playback device will discard the audio and video data sent by the CDN server. Through the parsed Supplemental Enhancement Information, the playback device can adjust its cache control state, thereby determining whether to cache more data to cope with potential network latency or discard some data to reduce latency. By dynamically adjusting the cache policy, the playback device can achieve a balance between reducing latency and ensuring smooth playback, thereby improving the live broadcast effect.

[0070] Optionally, in a possible embodiment, adjusting the cache control state according to the Supplemental Enhancement Information may include:

[0071] S1. Determine the delay information according to the system clock information and the synchronization clock information in the Supplemental Enhancement Information;

[0072] S2. If it is determined that the delay information is greater than the first preset threshold, adjust the cache control status to prohibit caching;

[0073] S3. If it is determined that the delay information is less than or equal to the first preset threshold, adjust the cache control status to allow caching.

[0074] Exemplarily, the system clock information is the time synchronized with the host device, and the delay information is determined by comparing the system clock information and the synchronization clock information. Specifically, the delay information can be defined as the system clock time minus the synchronization clock time, and this difference reflects the transmission delay of the audio-visual data from the host side to the playback side. If the delay information is greater than the first preset threshold, it indicates that the current transmission delay is too large, which may cause obvious delays in the audio-visual on the playback device. In this case, adjusting the cache control status to prohibit caching can reduce the further accumulation of delays. By no longer caching new data, the playback device can process and play the currently received data faster, thus reducing the delay. If the delay information is less than or equal to the first preset threshold, it indicates that the current transmission delay is within an acceptable range. In this case, adjusting the cache control status to allow caching can accumulate more data under good network conditions to cope with possible network fluctuations, which helps to still play smoothly when the network conditions suddenly deteriorate.

[0075] Among them, the specific value of the first preset threshold in the embodiments of the present application is not limited and can be set according to actual live broadcast requirements. For example, it can be set to 1s in live teaching.

[0076] By dynamically adjusting the cache policy, the playback device can achieve a balance between reducing delays and ensuring smooth playback. When the delay is too large, the delay is reduced by prohibiting caching, and when the delay is within a controllable range, the playback stability and smoothness are improved by allowing caching. This method improves the user experience of live video playback.

[0077] Optionally, in a possible embodiment, when it is determined that the delay information is greater than the first preset threshold, the method provided in the embodiments of the present application may further include:

[0078] S21. Determine and control the duration of prohibited caching according to the delay information and the target delay.

[0079] Exemplarily, the target delay is the ideal delay level expected to be achieved, which is a preset value representing the expected delay of the audio-visual data transmitted from the host side to the playback side under normal circumstances. The target delay can be set according to user experience requirements and network conditions, and the embodiments of the present application do not limit it. For example, it can be set to 0.5s.

[0080] To more precisely control the caching status of received audio and video data, when it is determined that the delay information is greater than the first preset threshold, the duration for which caching needs to be prohibited can also be determined based on the gap between the delay information and the target delay. For example, if the delay information is much more than the target delay, the duration of prohibited caching may be longer to reduce the delay more quickly; if the difference is small, the duration of prohibited caching can be relatively short to avoid losing too much data and causing the live broadcast to be discontinuous.

[0081] By dynamically adjusting the duration of prohibited caching, not only can the delay be effectively reduced, but caching can also be quickly restored when the network conditions improve, thereby achieving a better balance between reducing the delay and ensuring smooth playback, and improving the stability of live video playback and the user experience.

[0082] Optionally, in a possible embodiment, if the supplementary enhancement information further includes a fast-forward instruction message, after determining the delay information, the method provided by the embodiments of the present application may further include:

[0083] S4. If it is determined that the delay information is greater than the second preset threshold, discard the non-key video frames and the audio data corresponding to the non-key video frames in the audio and video data; where the second preset threshold is less than the first preset threshold;

[0084] S5. If it is determined that the delay information is less than or equal to the second preset threshold, do not execute the fast-forward instruction message.

[0085] Exemplarily, the supplementary enhancement information may further include a fast-forward instruction message. The fast-forward instruction is usually a one-time instruction issued by the live streamer, used to instruct the playback device to take fast-forward measures under specific conditions to catch up with the live playback more quickly. Optionally, the fast-forward instruction can also be a periodic instruction set by the live streamer to trigger at a certain frequency. The embodiments of the present application do not limit this. Among them, the second preset threshold is a relatively low delay threshold used to trigger the fast-forward measure, which is less than the first preset threshold. For example, it can be 0.5s.

[0086] When the supplementary enhancement information further includes a fast-forward instruction message, if the delay information is greater than the second preset threshold, it indicates that the delay begins to increase but has not reached the level of automatically triggering prohibited caching. In this case, the playback device can discard the non-key video frames (such as P frames and B frames) in the audio and video data and the audio data corresponding to these frames, so as to achieve the purpose of reducing the delay. Among them, the non-key video frames can include video frames other than the key video frames, and these frames rely on other frames for decoding. Discarding them can reduce the data volume and thus speed up the playback progress. If the delay information is less than or equal to the second preset threshold, it means that the delay is very small and not sufficient to execute the fast-forward instruction. In this case, the fast-forward measure can be not executed, and the audio and video data can continue to be cached and played normally.

[0087] Based on the fast forward instruction, by discarding non-key frames and corresponding audio data, the latency can also be quickly reduced, thus getting closer to real-time playback, and further improving the smoothness of live video playback and the user experience.

[0088] Optionally, in a possible embodiment, discarding non-key video frames and the audio data corresponding to the non-key video frames in the audio-visual data may include:

[0089] S41. Determine the difference between the latency information and the second preset threshold, and determine the number of non-key video frames to be discarded in the audio-visual data according to the difference;

[0090] S42. Discard the non-key video frames and the audio data corresponding to the non-key video frames in the audio-visual data according to the number of discarded frames.

[0091] Exemplarily, first, calculate the difference between the latency information and the second preset threshold. This difference reflects the degree to which the current latency exceeds the second preset threshold. The larger the difference, the more serious the latency. According to the calculated difference, the number of non-key video frames to be discarded can be determined. For example, through a mapping function or algorithm, the difference is converted into a specific number of discarded frames. When the difference is large, more frames may be discarded to reduce the latency faster; when the difference is small, fewer frames may be discarded. Then, according to the determined number of discarded frames, the corresponding number of non-key video frames are discarded from the audio-visual data. At the same time, to maintain audio-visual synchronization, the audio data synchronized with these non-key video frames is discarded.

[0092] The embodiment of the present application dynamically adjusts the discard strategy according to the current latency situation. This refined control can not only achieve the purpose of reducing latency, but also better ensure the playback effect, improving the smoothness of live video playback and the user experience.

[0093] S403. Process the audio-visual data according to the current video first-frame status, and decode and play the processed audio-visual data; the video first-frame status is used to indicate the presence or absence of the first-frame status.

[0094] Exemplarily, the video first-frame status is used to indicate whether the first frame of the video has been received currently. If there is a first-frame status, it means that the video stream can be normally decoded and played; if there is no first-frame status, it may be necessary to wait or request the retransmission of key frames. The processed audio-visual data can be decoded by the playback device and then output to the user interface for playback.

[0095] Optionally, in a possible embodiment, processing the audio-visual data according to the current video first-frame status may include:

[0096] If the current video first-frame state indicates that there is no first frame currently, non-key video frames in the audio-visual data are discarded until a key video frame is encountered, at which point the video first-frame state is adjusted to the state of having a first frame.

[0097] Exemplarily, when the audio-visual data in the cache is first separated, the current video first-frame state usually indicates that there is no first frame currently. At this time, non-key video frames in the audio-visual data need to be discarded until a key video frame is encountered, at which point the video first-frame state is adjusted to the state of having a first frame, and data is no longer discarded, so as to decode and play the live video normally subsequently. During the process of continuously separating the audio-visual data in the cache, the video first-frame state is continuously monitored, which usually indicates the state of having a first frame. When the separation is interrupted and the separation of the audio-visual data in the cache starts again, the video first-frame state will become the state of having no first frame. In this case, non-key video frames in the audio-visual data need to be discarded again until a key video frame is encountered.

[0098] Adjusting whether to discard non-key video frames in the audio-visual data through the video first-frame state can not only ensure that the decoding of the video stream starts from a valid starting point, that is, from a key frame, to ensure the accuracy of decoding and the quality of video playback, but also avoid possible decoding errors and resource waste by discarding non-key frames received in the state of having no first frame.

[0099] Optionally, in a possible embodiment, decoding and playing the processed audio-visual data may include:

[0100] S10: Put the video data in the processed audio-visual data into the video queue, and put the audio data in the processed audio-visual data into the audio queue;

[0101] S20: Perform video decoding processing on the video data, perform audio decoding processing on the audio data, display the video data and synchronously play the audio data.

[0102] Exemplarily, the video part of the processed audio - video data is put into a dedicated video queue. The video queue is used to temporarily store video frames for decoding and display in chronological order. This queue mechanism helps to handle jitter and latency in the video stream, ensuring that video frames are decoded in the correct order. Similarly, the audio data is put into an audio queue. The audio queue is used to temporarily store audio data packets for decoding and playback in chronological order. By using the audio queue, the audio stream can be smoothly processed, avoiding interruptions or inconsistencies in audio playback. Then, in accordance with the playback order, video data is extracted from the video queue for decoding. The encoded video data is decoded into displayable image frames, and the decoded image frames are sent to the display device for playback. At the same time, in accordance with the playback order, audio data is extracted from the audio queue for decoding. The encoded audio data is decoded into playable audio signals, and the decoded audio signals are sent to the speaker or other audio output devices for playback. During playback, timestamps or other synchronization information are used to control the synchronous playback of audio and video.

[0103] By using queues and synchronization mechanisms, the embodiments of the present application can effectively handle delays and jitters in audio - video data streams, ensuring smooth decoding and synchronous playback of audio and video, which is beneficial to improving the user experience.

[0104] The method for processing live video provided by the embodiments of the present application is applied to the playback - end device. By parsing the supplementary enhancement information in the audio - video data, especially the synchronous clock information, precise time synchronization of the audio - video stream is achieved, enabling the playback - end device to accurately know the time when the audio - video data is collected at the host - end device. Thus, according to the synchronous clock information, the cache control state can be adjusted more effectively, and the cache policy can be dynamically optimized. It can more flexibly decide whether to cache or discard data during network fluctuations, reducing unnecessary data backlog and processing delays. In addition, by processing the audio - video data according to the state of the first video frame, it is ensured that video decoding starts from a valid key frame, and the problem of video freeze caused by the lack of key frames is also avoided. These measures work together to significantly reduce the end - to - end latency of live video, providing a more real - time viewing experience for users.

[0105] Optionally, based on the above - mentioned embodiments, in a possible embodiment, before separating the audio - video data in the cache, the method provided by the embodiments of the present application may further include:

[0106] S100. Establish a network connection between the playback - end device and the content delivery network server;

[0107] S200. Receive and parse the audio - video data sent by the content delivery network server, and process the audio - video data according to the cache control state.

[0108] Exemplarily, before processing the audio and video data in the cache, the playback device needs to first receive the audio and video data sent by the distribution network server. Accordingly, it is necessary to first establish a stable network connection with the CDN server. For example, the connection is established through Internet protocols (such as TCP / IP, etc.). The establishment of this connection is a prerequisite for the transmission of audio and video data. When there is a normal connection between the playback device and the content distribution network server, the data can be transmitted from the CDN server to the playback device. When the connection between the playback device and the CDN server is disconnected due to network anomalies, the reconnection mechanism will be triggered, and the playback device will reconnect to the CDN server.

[0109] Once the network connection is successfully established, the playback device can receive the audio and video data from the CDN server. After the received audio and video data is parsed and confirmed, the audio and video data will be cached or discarded according to the current cache control status. Among them, in the initial state, when the playback device first receives the audio and video data from the CDN server, the cache control status is usually set to allow caching.

[0110] The embodiment of the present application ensures the reliability of data transmission by establishing a stable network connection, guarantees the effective transmission of audio and video data from the CDN server to the playback device, and optimizing the storage and use of data by processing data according to the cache control status provides support for subsequent decoding and playback, helps improve the efficiency and quality of streaming media playback, and reduces latency and stuttering.

[0111] Next, in combination with the pulling process and the method for processing live video provided by the embodiment of the present application, the latency caused by the pulling process will be introduced in detail.

[0112] Exemplarily, Figure 5 is a schematic framework diagram of a pulling process provided by an embodiment of the present application. As Figure 5 shown, before implementing the pulling, it is first necessary to establish a network connection between the playback device and the CDN server so that the playback device can receive the audio and video data sent by the CDN server. Among them, when the playback device connects to the CDN server, a fixed latency Pull1 will be generated. When the connection is disconnected due to network anomalies, a fixed latency Pull2 will be generated when reconnecting to the CDN server. When there is no disconnection and reconnection, the latency Pull2 is 0. When there are multiple disconnections and reconnections, the latency Pull2 is the accumulated latency.

[0113] After establishing a network connection, when receiving and parsing audio-visual data, latency Pull3 may occur due to network influence. The parsed audio-visual data first passes through the cache controller, and in the embodiments of the present application, the cache controller can be used to adjust the cache control state. Specifically, when the cache controller is used for the first time, SW1 is set to allow caching (SW1_Flag = true), and the reception and processing of audio-visual data start. At the same time, when the cache controller is used for the first time, it is fed back to the first frame controller that SW2 is in the state of no first frame (SW2_flag = false). When the cache controller is used subsequently, the cache control state is adjusted according to the value of the feedback state SW1_Flag of the synchronization controller. When SW1 is set to prohibit caching (SW1_Flag = false), the cache control state is adjusted to prohibit caching, and thus the received audio-visual data is discarded; when SW1 is set to prohibit caching, SW2 is continuously fed back to be in the state of no first frame (SW2_flag = false) until SW1 is set to allow caching (SW1_Flag = true), at which point the data discarding stops. Although discarding data affects the coherence of the audio-visual data, it can minimize latency and avoid meaningless data accumulation.

[0114] After caching or discarding the received audio-visual data according to the cache control state, the playback device separates the audio-visual data in the cache and checks the status of the audio and video data, corresponding to latency Pull4. Specifically, through separation, the key video frames in the audio-visual data can be marked, and the supplementary enhancement information in the video can be parsed. Among them, the supplementary enhancement information can be a synchronization data block synchronized with the audio-visual data, and this synchronization data block can include two types of information. One is the synchronization clock information, and the other is the fast forward instruction information. After obtaining the synchronization clock information, the synchronization controller compares it with its own system time information to determine the live broadcast latency information. If the latency is greater than the first preset threshold (such as 1 second), it will feedback SW1_Flag = false to the cache controller to adjust the cache control state to prohibit caching, so as to discard the received audio-visual data to reduce the latency until the latency is reduced below 0.5 seconds; if the latency is less than or equal to the first preset threshold (such as 1 second), it will feedback SW1_Flag = true to the cache controller to maintain the state of allowing caching. When the synchronization controller also obtains the fast forward instruction information, the synchronization controller will also evaluate whether to reduce the latency in a fast forward manner according to the current latency. If the latency is greater than the second preset threshold (such as 0.5 seconds), it will feedback to SW3 to start fast forward (SW3_Flag = true), causing the playback device to discard the non-key video frames and the audio data corresponding to the non-key video frames in the audio-visual data, achieving a reduction in latency to below 0.5 seconds; if it is determined that the latency is less than or equal to 0.5 seconds, no fast forward processing will be performed.

[0115] The separated audio and video data will pass through the first-frame controller. When the first-frame controller is used for the first time, SW2 is in the state of no first frame (SW2_Flag = false), and non-key video frames in the audio and video data start to be discarded. When a key video frame is encountered, SW2 is adjusted to the state of having a first frame (SW2_Flag = true), and data is no longer discarded. Further, the audio and video data will be processed by the fast-forward controller. The initial state of the fast-forward controller is the off state (SW3_Flag = false). After receiving the fast-forward instruction information fed back by the synchronization controller, it will enter the fast-forward state (SW3_Flag = true), thereby discarding non-key video frames and the corresponding audio data, only allowing key video frames to enter the video queue, and after adjusting the corresponding audio data to be muted, controlling it to enter the audio queue. When the video data enters the video queue, a fixed delay Video1 will be generated. When the audio data enters the audio queue, a fixed delay Audio1 will be generated. In addition, a fixed delay Video2 will be generated during video decoding, a fixed delay Audio2 will be generated during audio decoding, a fixed delay Video3 will be generated for displaying images, and a fixed delay Audio3 will be generated for playing sounds.

[0116] In summary, the pull stream delay Delay3 can be calculated with reference to the following formula (3):

[0117] Delay3 = Pull1 + SUM(Pull2) + Pull3 + Pull4 + MAX(Video1 + Video2 + Video3, Audio1 + Audio2 + Audio3) (3)

[0118] It can be understood that the decoding and playing of video data and the decoding and playing of audio data are carried out simultaneously. Therefore, when calculating the pull stream delay Delay3, the larger delay of the two is selected for calculation.

[0119] In the embodiments of the present application, the push stream delay can be understood as a fixed delay Delay1, which is related to the configuration and is usually set within 300 ms. In order to be compatible with the current mainstream live CDN servers, the corresponding delay of the CDN server can also be understood as a fixed delay Delay2, which is related to the configuration of the CDN server. Usually, only a set of video frame sequences is retained (the cached content starts caching from the nearest key frame until it encounters another new key frame and ends). The pull stream delay Delay3 has both fixed delays such as Pull1, Pull2, Pull3, Video1 + Video2 + Video3, Audio1 + Audio2 + Audio3, and adjustable delays such as Pull4. Therefore, in the embodiments of the present application, the synchronization clock information and fast forward instruction information in the supplementary enhancement information are used to adjust the states of the cache controller SW1, the first frame controller SW2, and the fast forward controller SW3, so as to control the data discarded in the audio and video data, achieve the purpose of reducing the delay, and reduce the overall delay of the open class to a nearly real-time state. Compared with the ordinary live broadcast technology, the method for processing live video in the embodiments of the present application can reduce the network delay of the open class. The overall delay affected by the network can be controlled within about 1 second, which not only reduces the network delay, but also maintains the advantages of low cost, good compatibility, and large supported scale of the live broadcast, and can meet the teaching requirements of real-time online interaction.

[0120] Figure 6 It is a schematic structural diagram of a device for processing live video provided by an embodiment of the present application. As Figure 6 shown, the device 60 for processing live video provided in this embodiment includes a first processing unit 601, a second processing unit 602, and a third processing unit 603.

[0121] Among them, the first processing unit 601 is used to separate the audio and video data in the cache, mark the key video frames in the audio and video data, and parse the supplementary enhancement information in the audio and video data; at least the synchronization clock information is included in the supplementary enhancement information, and the synchronization clock information represents the time when the host device collects the audio and video data;

[0122] The second processing unit 602 is used to adjust the cache control state according to the supplementary enhancement information; the cache control state is used to indicate whether to cache or discard the audio and video data sent by the content distribution network server;

[0123] The third processing unit 603 is used to process the audio and video data according to the current video first frame state, and decode and play the processed audio and video data; the video first frame state is used to indicate the state of having a first frame or not having a first frame.

[0124] The device provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effects are similar, and will not be elaborated here in this embodiment.

[0125] In a possible implementation, the second processing unit 602 is specifically configured to:

[0126] Determine delay information according to the system clock information and the synchronization clock information in the supplementary enhancement information;

[0127] If it is determined that the delay information is greater than the first preset threshold, adjust the cache control state to prohibit caching;

[0128] If it is determined that the delay information is less than or equal to the first preset threshold, adjust the cache control state to allow caching.

[0129] In a possible implementation, when it is determined that the delay information is greater than the first preset threshold, the second processing unit 602 is further configured to:

[0130] Determine and control the duration of prohibited caching according to the delay information and the target delay.

[0131] In a possible implementation, if the supplementary enhancement information further includes fast forward instruction information, after determining the delay information, the second processing unit 602 is further configured to:

[0132] If it is determined that the delay information is greater than the second preset threshold, discard the non-key video frames and the audio data corresponding to the non-key video frames in the audio-visual data; wherein, the second preset threshold is less than the first preset threshold;

[0133] If it is determined that the delay information is less than or equal to the second preset threshold, do not execute the fast forward instruction information.

[0134] In a possible implementation, the second processing unit 602 is specifically configured to:

[0135] Determine the difference between the delay information and the second preset threshold, and determine the number of non-key video frames to be discarded in the audio-visual data according to the difference;

[0136] Discard the non-key video frames and the audio data corresponding to the non-key video frames in the audio-visual data according to the number of discarded frames.

[0137] In a possible implementation, the third processing unit 603 is specifically configured to:

[0138] If the current video first frame state indicates that the current is a state without a first frame, discard the non-key video frames in the audio-visual data until a key video frame is encountered, and then adjust the video first frame state to a state with a first frame.

[0139] In a possible implementation, the third processing unit 603 is specifically configured to:

[0140] Put the video data in the processed audio - video data into the video queue, and put the audio data in the processed audio - video data into the audio queue;

[0141] Perform video decoding processing on the video data, perform audio decoding processing on the audio data, display the video data and play the audio data synchronously.

[0142] In a possible implementation manner, before separating the audio - video data in the buffer, the first processing unit 601 is further configured to:

[0143] Establish a network connection between the playback device and the content distribution network server;

[0144] Receive and parse the audio - video data sent by the content distribution network server, and process the audio - video data according to the cache control status.

[0145] The device provided by the embodiments of the present application can execute the method provided by the above - mentioned method embodiments. The implementation principles and technical effects are similar, and will not be elaborated here.

[0146] It should be noted that it should be understood that the division of each module of the above - mentioned device is only a division of logical functions. In actual implementation, it can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also all be implemented in the form of hardware; or some modules can be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. In addition, it can also be stored in the memory of the above - mentioned device in the form of program code, and called and executed by a certain processing element of the above - mentioned device to perform the functions of the above - mentioned data processing modules. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together or independently implemented. Here, the processing element can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above - mentioned method or each of the above - mentioned modules can be completed by the integrated logic circuit in the processor element in hardware or the instructions in software form.

[0147] Figure 7 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 7 shown, the electronic device 70 provided by this embodiment includes: at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. Among them, the processor 701, the memory 702, and the communication component 703 are connected through a bus 704.

[0148] In the specific implementation process, at least one processor 701 executes the computer - executable instructions stored in the memory 702, so that at least one processor 701 executes the above - mentioned method.

[0149] For the specific implementation process of the processor 701, reference can be made to the above method embodiments. Their implementation principles and technical effects are similar, and will not be elaborated here in this embodiment.

[0150] In the above embodiments, it should be understood that the processor may be a central processing unit (Central Processing Unit, CPU for short), or other general-purpose processors, digital signal processors (Digital Signal Processor, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0151] The memory may include high-speed memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0152] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the buses in the drawings of this application are not limited to only one bus or one type of bus.

[0153] This application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0154] This application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the processor executes the computer-executable instructions, the above method is implemented.

[0155] The above-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible by a general-purpose or special-purpose computer.

[0156] An exemplary readable storage medium is coupled to the processor, enabling the processor to read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in a device.

[0157] The division of units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be indirect couplings or communication connections through some interfaces, devices, or units, and can be in electrical, mechanical, or other forms.

[0158] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] In addition, the functional units in various embodiments of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0160] If a function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs and other various media that can store program codes.

[0161] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When this program is executed, it executes the steps including the above method embodiments; and the aforementioned storage medium includes: ROMs, RAMs, magnetic disks, or optical discs and other various media that can store program codes.

[0162] Finally, it should be noted that: after considering the specification and practicing the invention disclosed herein, those skilled in the art will readily think of other implementation manners of the present invention. The present invention is intended to cover any variations, uses, or adaptive changes of the present invention. These variations, uses, or adaptive changes follow the general principles of the present invention and include common general knowledge or conventional technical means in the technical field not disclosed in the present invention. It is not limited to the exact structures already described and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. A method for processing live video, characterized in that, The method is applied to a playback device, and the method includes: Separating the audio - video data in the cache, marking the key video frames in the audio - video data, and parsing out the supplementary enhancement information in the audio - video data; at least the synchronization clock information is included in the supplementary enhancement information, and the synchronization clock information represents the time when the host device collects the audio - video data; Adjusting the cache control state according to the supplementary enhancement information; the cache control state is used to indicate whether to cache or discard the audio - video data sent by the content delivery network server; Processing the audio - video data according to the current video first - frame state, and decoding and playing the processed audio - video data; the video first - frame state is used to indicate the state of having a first frame or not having a first frame.

2. The method according to claim 1, characterized in that, The adjusting the cache control state according to the supplementary enhancement information includes: Determining the delay information according to the system clock information and the synchronization clock information in the supplementary enhancement information; If it is determined that the delay information is greater than a first preset threshold, adjusting the cache control state to prohibit caching; If it is determined that the delay information is less than or equal to the first preset threshold, adjusting the cache control state to allow caching.

3. The method according to claim 2, wherein When it is determined that the delay information is greater than the first preset threshold, the method further includes: Determining and controlling the duration of prohibited caching according to the delay information and the target delay.

4. The method according to claim 2, wherein If the supplementary enhancement information further includes a fast - forward instruction information, after determining the delay information, the method further includes: If it is determined that the delay information is greater than a second preset threshold, discarding the non - key video frames in the audio - video data and the audio data corresponding to the non - key video frames; wherein, the second preset threshold is less than the first preset threshold; If it is determined that the delay information is less than or equal to the second preset threshold, not executing the fast - forward instruction information.

5. The method according to claim 4, wherein The discarding the non - key video frames in the audio - video data and the audio data corresponding to the non - key video frames includes: Determining the difference between the delay information and the second preset threshold, and determining the number of non - key video frames to be discarded in the audio - video data according to the difference; Discarding the non - key video frames in the audio - video data and the audio data corresponding to the non - key video frames according to the number of discarded frames.

6. The method according to claim 1, wherein The processing the audio - video data according to the current video first - frame state includes: If the current video first - frame state indicates that there is no first frame currently, discarding the non - key video frames in the audio - video data until a key video frame is encountered, and then adjusting the video first - frame state to having a first frame.

7. The method according to any one of claims 1 to 6, characterized in that, The decoding and playing the processed audio - video data includes: Putting the video data in the processed audio - video data into a video queue, and putting the audio data in the processed audio - video data into an audio queue; Performing video decoding processing on the video data, performing audio decoding processing on the audio data, displaying the video data and synchronously playing the audio data.

8. The method according to claim 7, wherein Before separating the audio - video data in the cache, the method further includes: Establishing a network connection between the playback device and the content delivery network server; Receive and parse the audio-visual data sent by the content delivery network server, and process the audio-visual data according to the cache control status.

9. A processing device for live video, characterized in that, The device is applied to a playback device, and the device includes: A first processing unit, configured to separate the audio-visual data in the cache, mark the key video frames in the audio-visual data, and parse the supplementary enhancement information in the audio-visual data; at least the synchronization clock information is included in the supplementary enhancement information, and the synchronization clock information represents the time when the host device collects the audio-visual data; A second processing unit, configured to adjust the cache control status according to the supplementary enhancement information; the cache control status is used to indicate caching or discarding the audio-visual data sent by the content delivery network server; A third processing unit, configured to process the audio-visual data according to the current video first-frame status, and decode and play the processed audio-visual data; the video first-frame status is used to indicate the presence or absence of a first-frame status.

10. An electronic device, characterized in that, Comprising: A memory, a processor; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory, so that the processor executes the method according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, Computer execution instructions are stored in the computer-readable storage medium, and when the computer execution instructions are executed by the processor, they are used to implement the method according to any one of claims 1-8.

12. A computer program product, characterized in that, Comprising a computer program, which when executed by a processor implements the method according to any one of claims 1-8.

Images