Live video processing method and device, equipment, storage medium and program product

Abstract

Embodiments of the present application provide a live video processing method and device, equipment, storage medium and program product, relating to the technical field of streaming media. The method is applied to a player device, comprising: separating audio and video data in a cache, marking key video frames in the audio and video data, and analyzing supplementary enhancement information in the audio and video data; the supplementary enhancement information at least includes synchronization clock information, and the synchronization clock information represents the time when the anchor device collects the audio and video data; according to the supplementary enhancement information, the cache control state is adjusted; the cache control state is used to indicate whether to cache or discard the audio and video data sent by a content distribution network server; according to the current video first frame state, the audio and video data is processed, and the processed audio and video data is decoded and played; the video first frame state is used to indicate whether there is a first frame state or not. The method is used to reduce the live delay and avoid the effect of large delay on online interaction.

Description

Technical Field

[0001] This application relates to the field of streaming media technology, and in particular to a method, apparatus, device, storage medium, and program product for processing live video. Background Technology

[0002] With the rapid development of internet technology and the widespread adoption of smart devices, live video streaming has become an important form of social interaction and entertainment. Users can watch content shared by streamers in real time through their devices and experience the fun of real-time interaction.

[0003] In existing technologies, live video processing typically involves multiple stages, including audio and video data acquisition, encoding, transmission, decoding, and playback. To ensure smooth and stable playback, audio and video data caching is usually required. However, caching increases latency, causing the time difference between the audio and video data on the playback end and the broadcaster end to widen, affecting the interactive experience of the live stream. Especially in live online open courses with tens of thousands of students attending simultaneously, the caching mechanisms of ordinary live streams are usually insufficient to meet the requirements of real-time and interactive teaching.

[0004] Therefore, a more reasonable caching mechanism is urgently needed to reduce live streaming latency and avoid significant delays affecting the effectiveness of online interactive teaching. Summary of the Invention

[0005] The live video processing method, apparatus, device, storage medium, and program products provided in this application are used to reduce live video latency and avoid the effect of large latency on online interaction.

[0006] In a first aspect, embodiments of this application provide a method for processing live video, the method being applied to a playback device, the method comprising:

[0007] The audio and video data in the cache are separated, key video frames in the audio and video data are marked, and supplementary enhancement information in the audio and video data is parsed out; the supplementary enhancement information includes at least synchronization clock information, which represents the time when the broadcaster's device collects the audio and video data;

[0008] Based on the supplementary enhancement information, adjust the cache control state; the cache control state is used to indicate whether to cache or discard audio and video data sent by the content delivery network server;

[0009] Based on the current first frame state of the video, the audio and video data are processed, and the processed audio and video data is decoded and played; the first frame state of the video is used to indicate whether there is a first frame state or not.

[0010] In one possible implementation, adjusting the cache control state based on the supplementary enhancement information includes:

[0011] The delay information is determined based on the system clock information and the synchronization clock information in the supplementary enhancement information;

[0012] If it is determined that the delay information is greater than the first preset threshold, then the cache control state is adjusted to disable caching;

[0013] If the delay information is determined to be less than or equal to the first preset threshold, then the cache control state is adjusted to allow caching.

[0014] In one possible implementation, when it is determined that the delay information is greater than a first preset threshold, the method further includes:

[0015] Based on the delay information and the target delay, determine and control the duration for which caching is prohibited.

[0016] In one possible implementation, if the supplementary enhancement information also includes fast-forward instruction information, then after determining the delay information, the method further includes:

[0017] If the delay information is determined to be greater than the second preset threshold, then the non-critical video frames and the corresponding audio data in the audio and video data are discarded; wherein, the second preset threshold is less than the first preset threshold;

[0018] If the delay information is determined to be less than or equal to the second preset threshold, then the fast-forward instruction information is not executed.

[0019] In one possible implementation, discarding non-critical video frames and their corresponding audio data from the audio and video data includes:

[0020] Determine the difference between the delay information and the second preset threshold, and determine the number of non-critical video frames to be discarded in the audio and video data based on the difference;

[0021] According to the stated number of discarded frames, discard non-critical video frames and the corresponding audio data in the audio and video data.

[0022] In one possible implementation, processing the audio and video data according to the current first frame state of the video includes:

[0023] If the current video first frame status indicates that there is no first frame, then non-critical video frames in the audio and video data are discarded until a critical video frame is encountered, at which point the video first frame status is adjusted to have a first frame.

[0024] In one possible implementation, decoding and playing the processed audio and video data includes:

[0025] The video data from the processed audio and video data is placed into the video queue, and the audio data from the processed audio and video data is placed into the audio queue.

[0026] The video data is processed by video decoding, the audio data is processed by audio decoding, the video data is displayed, and the audio data is played synchronously.

[0027] In one possible implementation, the method further includes, before separating the audio and video data in the buffer:

[0028] Establish a network connection between the playback device and the content distribution network server;

[0029] Receive and parse the audio and video data sent by the content delivery network server, and process the audio and video data according to the cache control state.

[0030] Secondly, embodiments of this application provide a live video processing apparatus, which is applied to a playback device, and the apparatus includes:

[0031] The first processing unit is used to separate the audio and video data in the cache, mark the key video frames in the audio and video data, and parse the supplementary enhancement information in the audio and video data; the supplementary enhancement information includes at least synchronization clock information, which represents the time when the broadcaster's device collects the audio and video data;

[0032] The second processing unit is used to adjust the cache control state according to the supplementary enhancement information; the cache control state is used to indicate whether to cache or discard the audio and video data sent by the content delivery network server.

[0033] The third processing unit is used to process the audio and video data according to the current first frame state of the video, and to decode and play the processed audio and video data; the first frame state of the video is used to indicate whether there is a first frame state or not.

[0034] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0035] The memory stores computer-executed instructions;

[0036] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0037] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0038] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0039] The live video processing method, apparatus, device, storage medium, and program product provided in this application embodiment achieve precise time synchronization of audio and video streams by parsing supplementary enhancement information in audio and video data, especially synchronization clock information. This allows the playback end to accurately understand the time when the broadcaster collects audio and video data, thereby more effectively adjusting the cache control state based on the synchronization clock information and dynamically optimizing the cache strategy. It can more flexibly decide to cache or discard data when the network fluctuates, reducing unnecessary data backlog and processing delays. In addition, processing audio and video data according to the state of the first frame of the video ensures that video decoding starts from a valid keyframe, and also avoids screen tearing problems caused by missing keyframes. These measures work together to significantly reduce the end-to-end latency of live video, providing users with a more real-time viewing experience. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] Figure 1 A schematic diagram illustrating a framework for a live video processing procedure provided in an embodiment of this application;

[0042] Figure 2 A schematic diagram of a streaming process provided for an embodiment of this application;

[0043] Figure 3 A schematic diagram illustrating the framework of a CDN server processing procedure provided in an embodiment of this application;

[0044] Figure 4 A flowchart illustrating a live video processing method provided in an embodiment of this application;

[0045] Figure 5 A schematic diagram of a pull-stream process provided for an embodiment of this application;

[0046] Figure 6 A schematic diagram of the structure of a live video processing device provided in an embodiment of this application;

[0047] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0048] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0049] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0050] Open courses are a form of online teaching that supports tens of thousands of people attending classes simultaneously. The primary technical means is live streaming, where the lecturer's live video is distributed to everyone in the online classroom via a Content Delivery Network (CDN) server. The live video may include audio data such as the lecturer's voice, as well as video data such as the lecturer's image, courseware, and other information. The purpose of live-streamed teaching is to allow more people, regardless of geographical location, to receive professional guidance and interactive teaching from excellent teachers, enhancing learning effectiveness and saving on learning costs.

[0051] For example, Figure 1 This is a schematic diagram illustrating a framework for a live video processing procedure provided in an embodiment of this application. Figure 1As shown, the audio and video data generated by the live streamer is processed by the push client and sent to the CDN server in real time via push streaming. Due to technical reasons, this process generates a fixed latency related to the configuration, which can be labeled Delay1. After receiving the audio and video data, the CDN server caches the data according to the pre-configured caching strategy or distributes it to other related CDN server nodes level by level. This process also generates a fixed latency related to the configuration, which can be labeled Delay2. The pull client connects to the nearest CDN server node, obtains the pre-cached audio and video data from the CDN server, and decodes and plays it. This process generates not only a fixed latency related to the configuration but also an adjustable latency related to the caching strategy, which can be labeled Delay3. The live streaming process is a continuous repetition of the "push-CDN-pull" process, during which viewers continuously enjoy the video and audio from the streamer. In addition, due to the uncertainty of network transmission and the time required for data transmission, the live streaming process also generates a latency related to the distance between the streamer and the viewers, which can be labeled Delay0. Among them, the push streaming client corresponds to the broadcasting device of the live teacher, and the pull streaming client corresponds to the playback device of the viewer.

[0052] As can be seen, due to the uncertainty of network transmission, the latency of the live streaming process includes at least the latency Delay0 related to network transmission, the latency Delay1 related to the push end, the latency Delay2 related to CDN server configuration, and the latency Delay3 related to the pull end. When assessing the overall live streaming latency, the following formula (1) can be used for calculation:

[0053] Delay = Delay0 + Delay1 + Delay2 + Delay3 (1)

[0054] The delays related to streaming (Delay1), CDN server configuration (Delay2), and streaming (Delay3) are mainly caused by caching. While caching can improve network transmission smoothness, it comes at the cost of increased latency. Currently, typical live streaming latency is 3-10 seconds, sometimes even longer. While this latency is generally beneficial in traditional live streaming, it is unsuitable for scenarios like open courses where real-time performance is crucial, as it often fails to meet the requirements of real-time, interactive teaching. Therefore, it is necessary to reduce latency in live teaching to avoid excessive latency affecting the effectiveness of online interactive teaching.

[0055] For example, Figure 2 This is a schematic diagram illustrating a streaming process as provided in an embodiment of this application. Figure 2As shown, when a broadcaster is live, the camera on the broadcaster's device captures video, which generates a fixed delay Video1; the microphone captures audio, which generates a fixed delay Audio1; video is encoded into a live broadcast encoding format, which generates a fixed delay Video2; audio is encoded into a live broadcast encoding format, which generates a fixed delay Audio2; the encoded video and audio data are mixed and packaged into the format required by the live broadcast protocol, which generates a fixed delay Push1; the broadcaster's device connecting to the CDN server generates a fixed delay Push2; when a network anomaly causes the connection to be lost, reconnecting to the CDN server will generate a fixed delay Push3. Among them, when there is no network disconnection and reconnection, the delay Push3 is 0; when there are multiple network disconnections and reconnections, the delay Push3 needs to be accumulated. When the broadcaster's device sends data, it will generate a delay Push4 due to network influence. Based on this, the push delay Delay1 can be calculated with reference to the following formula (2):

[0056] Delay1=MAX(Video1+Video2,Audio1+Video2)+Push1+Push2+SUM(Push3)+Push4(2)

[0057] Understandably, video data acquisition and encoding are performed simultaneously with audio data acquisition and encoding. Therefore, when calculating the streaming delay Delay1, the one with the larger delay is selected for the calculation.

[0058] For example, Figure 3 This is a schematic diagram illustrating the framework of a CDN server processing procedure provided in an embodiment of this application. Figure 3 As shown, the audio and video data sent by the broadcaster's device will travel from the CDN server's uplink stream to the CDN server's media center, then be sent down to the CDN server's downlink stream, and finally be retrieved and played by the viewer's playback device. Although each CDN server operator has its own deployment architecture, they are largely similar. In live streaming scenarios, the CDN server latency can be configured according to customer requirements. When compatibility with CDN server products from different vendors is required, it can be assumed that the CDN server will always generate a fixed latency, Delay2.

[0059] Optionally, to reduce latency Delay2, the CDN server cache can be configured to retain only one set of video frame sequences, that is, the cached content starts from the most recent keyframe and ends when another new keyframe is encountered.

[0060] In summary, the analysis shows that both the streaming delay Delay1 and the CDN server delay Delay2 are related to the configuration. In the case that the configuration cannot be changed, in order to further reduce the latency, this embodiment of the application considers controlling it from the streaming client to reduce the latency.

[0061] The live video processing method provided in this application allows the playback device to cache or discard audio and video data according to the cache control state when it receives audio and video data sent by the CDN server. The cache control state is determined by the current latency or fast-forward command. At the same time, the audio and video data is also processed according to whether there is a first frame. Finally, decoding and playback are performed. This achieves more flexible control over the caching situation, which can meet latency requirements and improve the live streaming effect.

[0062] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0063] Figure 4 This is a flowchart illustrating a live video processing method provided in an embodiment of this application. Figure 4 As shown in the embodiment of this application, the live video processing method is applied to a playback device, and the method includes:

[0064] S401. Separate the audio and video data in the buffer, mark the key video frames in the audio and video data, and parse out the supplementary enhancement information in the audio and video data; the supplementary enhancement information includes at least the synchronization clock information, which represents the time when the broadcaster's device collects the audio and video data.

[0065] For example, in live streaming, the playback device is responsible for playing the streamer's live video to viewers from the streaming end. To improve playback smoothness and stability, the playback device typically caches audio and video data. Simultaneously, to reduce latency caused by caching, a more reasonable caching mechanism is needed to control the caching of audio and video data. In this embodiment, the initial caching control state of the playback device is typically enabled. Upon first receiving audio and video data from the CDN server, it caches and processes the data.

[0066] In live streaming, audio and video data are typically transmitted together. The playback device needs to separate these two types of data for separate processing. This separation is usually achieved by parsing the stream's container format, such as streaming media formats (Flash Video, FLV). After separating the buffered audio and video data, the playback device can mark key video frames within the audio and video data. Video data consists of a series of frames, where key frames (I-frames) are independently decodeable frames. Unlike other frames (such as P-frames and B-frames), I-frames do not depend on the data of other frames. In practical applications, the frequency of key video frames can be set on the broadcaster's device; for example, a key frame can be set every 20 video frames.

[0067] Furthermore, the playback device can also parse Supplemental Enhancement Information (SEI) from the audio and video data. SEI is text data inserted into the audio and video stream to convey additional information. It can be precisely synchronized with the relevant audio and video content and reaches the client via the transmission link. In this embodiment, the SEI includes at least synchronization clock information. The synchronization clock information refers to the time when the broadcasting device collects the audio and video data. By using the synchronization clock information, the playback device can not only ensure that the audio and video data remain synchronized during playback, but also adjust the buffer control state.

[0068] S402. Adjust the cache control state based on the supplementary enhancement information; the cache control state is used to indicate whether to cache or discard audio and video data sent by the content delivery network server.

[0069] For example, a Content Delivery Network (CDN) server, when its caching control is enabled, allows the playback device to cache the audio and video data sent by the CDN server; when its caching control is disabled, it discards the data. By parsing the supplementary enhancement information, the playback device can adjust its caching control state, deciding whether to cache more data to address potential network latency or discard some data to reduce latency. By dynamically adjusting the caching strategy, the playback device can strike a balance between reducing latency and ensuring smooth playback, thereby improving the live streaming experience.

[0070] Optionally, in one possible embodiment, adjusting the cache control state based on supplementary enhancement information may include:

[0071] S1. Determine the delay information based on the system clock information and the synchronization clock information in the supplementary and enhanced information;

[0072] S2. If it is determined that the delay information is greater than the first preset threshold, then adjust the cache control state to disable caching;

[0073] S3. If the delay information is determined to be less than or equal to the first preset threshold, then adjust the cache control state to allow caching.

[0074] For example, the system clock information is the time it takes to synchronize with the broadcaster's device, and the latency information is determined by comparing the system clock information and the synchronization clock information. Specifically, the latency information can be defined as the system clock time minus the synchronization clock time; this difference reflects the transmission delay of audio and video data from the broadcaster to the playback device. If the latency information is greater than a first preset threshold, it indicates that the current transmission delay is too large, which may cause significant delays in the audio and video on the playback device. In this case, adjusting the buffer control state to disable buffering can reduce further latency accumulation. By no longer buffering new data, the playback device can process and play the currently received data more quickly, thereby reducing latency. If the latency information is less than or equal to the first preset threshold, it indicates that the current transmission delay is within an acceptable range. In this case, adjusting the buffer control state to enable buffering allows more data to be accumulated under good network conditions to cope with possible network fluctuations, helping to maintain smooth playback even when network conditions suddenly deteriorate.

[0075] In this embodiment, the specific value of the first preset threshold is not limited, and it can be set according to the actual live streaming needs. For example, it can be set to 1 second in live teaching.

[0076] By dynamically adjusting the caching strategy, the playback device can strike a balance between reducing latency and ensuring smooth playback. When the latency is too high, caching is disabled to reduce latency, while when the latency is within a controllable range, caching is allowed to improve the stability and smoothness of playback. This method improves the user experience of live video playback.

[0077] Optionally, in one possible embodiment, when the delay information is determined to be greater than a first preset threshold, the method provided in this application embodiment may further include:

[0078] S21. Based on the delay information and the target delay, determine and control the duration for which caching is prohibited.

[0079] For example, the target latency is the desired level of latency, which is a preset value representing the expected latency of audio and video data transmission from the broadcaster to the playback end under normal circumstances. The target latency can be set according to user experience requirements and network conditions. This application embodiment does not impose any restrictions. For example, it can be set to 0.5s.

[0080] To more precisely control the caching status of received audio and video data, when the latency information is determined to be greater than the first preset threshold, the duration of caching can be determined based on the difference between the latency information and the target latency. For example, if the latency information is significantly greater than the target latency, the duration of caching prohibition may be longer in order to reduce latency more quickly; if the difference is small, the duration of caching prohibition can be relatively short to avoid losing too much data and causing discontinuity in the live broadcast.

[0081] By dynamically adjusting the duration of the cache-disable period, not only can latency be effectively reduced, but the cache can also be quickly restored when network conditions improve. This achieves a better balance between reducing latency and ensuring smooth playback, thereby improving the stability of live video playback and the user experience.

[0082] Optionally, in one possible embodiment, if the supplementary enhancement information also includes fast-forward instruction information, then after determining the delay information, the method provided in this application embodiment may further include:

[0083] S4. If the delay information is determined to be greater than the second preset threshold, then discard the non-critical video frames and the audio data corresponding to the non-critical video frames in the audio and video data; wherein, the second preset threshold is less than the first preset threshold.

[0084] S5. If the delay information is determined to be less than or equal to the second preset threshold, then the fast forward instruction information is not executed.

[0085] For example, the supplementary enhancement information may also include fast-forward instruction information. A fast-forward instruction is typically a one-time instruction issued by the broadcaster, instructing the playback device to take fast-forward measures under specific conditions to catch up with live playback more quickly. Optionally, the fast-forward instruction may also be a periodic instruction set by the broadcaster and triggered at a certain frequency; this embodiment of the application does not limit this. The second preset threshold is a lower delay threshold used to trigger the fast-forward measure; it is less than the first preset threshold, for example, it could be 0.5 seconds.

[0086] When the supplementary enhancement information also includes fast-forward instruction information, if the latency information is greater than the second preset threshold, it indicates that the latency has started to increase, but has not yet reached the level that automatically triggers the disabling of buffering. In this case, the playback device can discard non-critical video frames (such as P-frames and B-frames) and their corresponding audio data from the audio and video data, thereby reducing latency. Non-critical video frames can include video frames other than critical video frames; these frames depend on other frames for decoding, and discarding them reduces the amount of data, thus speeding up playback. If the latency information is less than or equal to the second preset threshold, it indicates that the latency is very small and insufficient to execute the fast-forward instruction. In this case, fast-forward measures can be omitted, and normal buffering and playback of audio and video data can continue.

[0087] Based on the fast-forward command, by discarding non-keyframes and their corresponding audio data, latency can be quickly reduced, thus getting closer to real-time playback and improving the smoothness of live video playback and user experience.

[0088] Optionally, in one possible embodiment, discarding non-critical video frames and their corresponding audio data may include:

[0089] S41. Determine the difference between the delay information and the second preset threshold, and determine the number of non-critical video frames to be discarded in the audio and video data based on the difference.

[0090] S42. Discard non-critical video frames and their corresponding audio data in the audio and video data according to the number of discarded frames.

[0091] For example, firstly, the difference between the latency information and a second preset threshold is calculated. This difference reflects the degree to which the current latency exceeds the second preset threshold; the larger the difference, the more severe the latency. Based on the calculated difference, the number of non-critical video frames to be discarded can be determined. For example, the difference can be converted into a specific number of frames to be discarded using a mapping function or algorithm. When the difference is large, more frames may be discarded to reduce latency more quickly; while when the difference is small, fewer frames may be discarded. Then, according to the determined number of frames to be discarded, the corresponding number of non-critical video frames are discarded from the audio and video data. At the same time, in order to maintain audio and video synchronization, the audio data synchronized with these non-critical video frames is discarded.

[0092] The embodiments of this application dynamically adjust the discard strategy according to the current latency. This fine-grained control can not only reduce latency, but also better ensure playback effect, improve the smoothness of live video playback and user experience.

[0093] S403. Based on the current video first frame status, process the audio and video data, and decode and play the processed audio and video data; the video first frame status is used to indicate whether there is a first frame status or not.

[0094] For example, the video first frame status is used to indicate whether the first frame of the video has been received. If there is a first frame status, it means that the video stream can be decoded and played normally; if there is no first frame status, it may be necessary to wait or request a retransmission of the keyframe. The processed audio and video data can be decoded by the playback device and then output to the user interface for playback.

[0095] Optionally, in one possible embodiment, processing audio and video data based on the current state of the first video frame may include:

[0096] If the current video first frame status indicates that there is no first frame, then non-critical video frames in the audio and video data are discarded until a critical video frame is encountered, at which point the video first frame status is adjusted to have a first frame.

[0097] For example, when the audio and video data in the buffer is separated for the first time, the current first frame state usually indicates a state without a first frame. At this time, non-critical video frames in the audio and video data need to be discarded until a critical video frame is encountered. Then, the first frame state is adjusted to a state with a first frame, and no more data is discarded, so that the live video can be decoded and played normally subsequently. During the continuous separation of audio and video data in the buffer, the first frame state of the video is continuously monitored, and it usually indicates a state with a first frame. When the separation is interrupted and the separation of audio and video data in the buffer is restarted, the first frame state of the video will change to a state without a first frame. In this case, non-critical video frames in the audio and video data need to be discarded again until a critical video frame is encountered.

[0098] Adjusting the first frame state of the video to decide whether to discard non-critical video frames in the audio and video data not only ensures that the decoding of the video stream starts from a valid starting point, i.e., from a key frame, thus guaranteeing the accuracy of decoding and the quality of video playback, but also avoids potential decoding errors and resource waste by discarding non-critical frames received in the absence of a first frame.

[0099] Optionally, in one possible embodiment, decoding and playing the processed audio and video data may include:

[0100] S10. Put the video data in the processed audio and video data into the video queue, and put the audio data in the processed audio and video data into the audio queue.

[0101] S20. Perform video decoding on the video data, perform audio decoding on the audio data, display the video data and play the audio data synchronously.

[0102] For example, the video portion of the processed audio and video data is placed into a dedicated video queue. This queue temporarily stores video frames for decoding and display in chronological order. This queuing mechanism helps handle jitter and latency in the video stream, ensuring that video frames are decoded in the correct order. Similarly, audio data is placed into an audio queue, which temporarily stores audio data packets for decoding and playback in chronological order. Using the audio queue allows for smooth processing of the audio stream, avoiding interruptions or inconsistencies during audio playback. Then, according to the playback order, video data is extracted from the video queue for decoding, converting the encoded video data into displayable image frames. The decoded image frames are then sent to a display device for playback. Simultaneously, according to the playback order, audio data is extracted from the audio queue for decoding, converting the encoded audio data into playable audio signals. The decoded audio signals are then sent to speakers or other audio output devices for playback. During playback, timestamps or other synchronization information are used to control the synchronized playback of audio and video.

[0103] The embodiments of this application can effectively handle latency and jitter in audio and video data streams by using queues and synchronization mechanisms, ensuring smooth decoding and synchronized playback of audio and video, which is beneficial to improving user experience.

[0104] The live video processing method provided in this application is applied to a playback device. By parsing supplementary enhancement information in the audio and video data, especially the synchronization clock information, it achieves precise time synchronization of the audio and video streams. This allows the playback device to accurately understand the time when the broadcaster collects the audio and video data, thereby more effectively adjusting the cache control state based on the synchronization clock information and dynamically optimizing the cache strategy. It can more flexibly decide to cache or discard data when the network fluctuates, reducing unnecessary data backlog and processing delays. In addition, processing the audio and video data according to the state of the first frame of the video ensures that video decoding starts from a valid keyframe, and also avoids screen tearing problems caused by missing keyframes. These measures work together to significantly reduce the end-to-end latency of live video, providing users with a more real-time viewing experience.

[0105] Optionally, based on the above embodiments, in one possible embodiment, before separating the audio and video data in the buffer, the method provided in this application embodiment may further include:

[0106] S100. Establish a network connection between the playback device and the content distribution network server;

[0107] S200 receives and parses the audio and video data sent by the content delivery network server, and processes the audio and video data according to the cache control state.

[0108] For example, before processing the cached audio and video data, the playback device needs to receive the audio and video data sent by the content delivery network server. Therefore, a stable network connection needs to be established with the CDN server, for example, via an Internet protocol (such as TCP / IP). Establishing this connection is a prerequisite for audio and video data transmission. Data can only be transmitted from the CDN server to the playback device when the connection between the playback device and the content delivery network server is normal. If the connection between the playback device and the CDN server is lost due to a network anomaly, a reconnection mechanism will be triggered, and the playback device will reconnect to the CDN server.

[0109] Once the network connection is successfully established, the playback device can receive audio and video data from the CDN server. After the received audio and video data is parsed and confirmed, it will be cached or discarded according to the current cache control state. Initially, when the playback device receives audio and video data from the CDN server for the first time, the cache control state is usually set to allow caching.

[0110] This application embodiment ensures the reliability of data transmission by establishing a stable network connection, ensuring the effective transmission of audio and video data from the CDN server to the playback device. Furthermore, processing data according to the cache control state optimizes data storage and usage, providing support for subsequent decoding and playback. This helps improve the efficiency and quality of streaming media playback and reduces latency and stuttering.

[0111] The following section provides a detailed explanation of the delay caused by the streaming process, combining the streaming process with the live video processing method provided in the embodiments of this application.

[0112] For example, Figure 5 This is a schematic diagram illustrating a pull-stream process provided in an embodiment of this application. Figure 5 As shown, before implementing streaming, a network connection needs to be established between the playback device and the CDN server so that the playback device can receive audio and video data sent by the CDN server. The connection between the playback device and the CDN server incurs a fixed latency Pull1. When a network anomaly causes the connection to drop, reconnecting to the CDN server incurs a fixed latency Pull2. If no network disconnection and reconnection occurs, Pull2 is 0. If multiple network disconnections and reconnections occur, Pull2 becomes the accumulated latency.

[0113] After establishing a network connection, when receiving and parsing audio and video data, network interference will cause a delay (Pull3). The parsed audio and video data first passes through a buffer controller. In this embodiment, the buffer controller can be used to adjust the buffer control state. Specifically, when the buffer controller is used for the first time, SW1 is enabled for buffering (SW1_Flag=true), and audio and video data is received and processed. At the same time, when the buffer controller is used for the first time, SW2 is fed back to the first frame controller as having no first frame (SW2_flag=false). In subsequent uses of the buffer controller, the buffer control state is adjusted according to the value of the SW1_Flag fed back by the synchronization controller. When SW1 is disabled for buffering (SW1_Flag=false), the buffer control state is adjusted to disable buffering, thereby discarding the received audio and video data; when SW1 is disabled for buffering, SW2 is continuously fed back as having no first frame (SW2_flag=false) until SW1 is enabled for buffering (SW1_Flag=true), at which point data discarding stops. Although discarding data will affect the continuity of audio and video data, it can minimize latency and avoid meaningless data accumulation.

[0114] After caching or discarding received audio and video data according to the cache control state, the playback device separates the cached audio and video data and checks the status of the audio and video data, corresponding to a delay of Pull4. Specifically, by separating the data, key video frames in the audio and video data can be marked, and supplementary enhancement information in the video can be parsed. This supplementary enhancement information can be a synchronization data block synchronized with the audio and video data. This synchronization data block can include two types of information: synchronization clock information and fast-forward instruction information. After obtaining the synchronization clock information, the synchronization controller compares it with its own system time information to determine the live broadcast delay information. If the delay is greater than a first preset threshold (e.g., 1 second), it will send SW1_Flag=false to the cache controller, adjusting the cache control state to disable caching, thereby discarding the received audio and video data to reduce the delay until it is reduced to below 0.5 seconds. If the delay is less than or equal to the first preset threshold (e.g., 1 second), it will send SW1_Flag=true to the cache controller to maintain the caching enabled state. When the synchronization controller also receives the fast-forward instruction information, it will also evaluate whether to reduce the latency by fast-forwarding based on the current latency. If the latency is greater than the second preset threshold (e.g., 0.5 seconds), it will feed back to SW3 to start fast-forwarding (SW3_Flag=true), causing the playback device to discard non-critical video frames and the audio data corresponding to the non-critical video frames in the audio and video data, thereby reducing the latency to below 0.5 seconds; if it is determined that the latency is less than or equal to 0.5 seconds, fast-forwarding will not be performed.

[0115] The separated audio and video data passes through the first-frame controller. The first time the first-frame controller is used, SW2 is in a no-first-frame state (SW2_Flag=false), discarding non-critical video frames. When a critical video frame is encountered, SW2 is adjusted to have a first frame (SW2_Flag=true), and data is no longer discarded. Next, the audio and video data passes through the fast-forward controller. The fast-forward controller is initially in a closed state (SW3_Flag=false). Upon receiving a fast-forward command from the synchronization controller, it enters a fast-forward state (SW3_Flag=true), discarding non-critical video frames and their corresponding audio data, allowing only critical video frames to enter the video queue, and muting the corresponding audio data before controlling its entry into the audio queue. When video data enters the video queue, a fixed delay (Video1) is generated. When audio data enters the audio queue, a fixed delay (Audio1) is generated. In addition, video decoding generates a fixed delay (Video2), audio decoding generates a fixed delay (Audio2), displaying images generates a fixed delay (Video3), and playing sound generates a fixed delay (Audio3).

[0116] In summary, the pull delay Delay3 can be calculated using the following formula (3):

[0117] Delay3=Pull1+SUM(Pull2)+Pull3+Pull4+MAX(Video1+Video2+Video3, Audio1+Audio2+Audio3) (3)

[0118] Understandably, video data decoding and playback and audio data decoding and playback are performed simultaneously. Therefore, when calculating the streaming delay Delay3, the one with the larger delay is selected for the calculation.

[0119] In this embodiment, the push stream latency can be understood as a fixed latency Delay1, which is configuration-dependent and typically set within 300ms. To ensure compatibility with current mainstream live streaming CDN servers, the corresponding latency of the CDN server can also be understood as a fixed latency Delay2, which is configuration-dependent and typically only retains one set of video frame sequences (cached content starts from the most recent keyframe and ends when another new keyframe is encountered). The pull stream latency Delay3 includes both fixed latency such as Pull1, Pull2, Pull3, Video1+Video2+Video3, and Audio1+Audio2+Audio3, as well as adjustable latency such as Pull4. Therefore, this embodiment utilizes the synchronization clock information and fast-forward instruction information in the supplementary enhancement information to adjust the states of the cache controller SW1, the first frame controller SW2, and the fast-forward controller SW3, thereby controlling the discarding of data in the audio and video data to reduce latency and bring the overall latency of the open course down to near real-time. Compared with ordinary live streaming technology, the live video processing method of this application embodiment can reduce the network latency of open courses. The overall latency affected by the network can be controlled to about 1 second. It not only reduces network latency, but also maintains the advantages of low cost, good compatibility and large scale support for live streaming, and can meet the requirements of real-time online interactive teaching.

[0120] Figure 6 This is a schematic diagram of a live video processing device provided in an embodiment of this application. Figure 6 As shown, the live video processing device 60 provided in this embodiment includes a first processing unit 601, a second processing unit 602, and a third processing unit 603.

[0121] The first processing unit 601 is used to separate the audio and video data in the buffer, mark the key video frames in the audio and video data, and parse the supplementary enhancement information in the audio and video data; the supplementary enhancement information includes at least synchronization clock information, which represents the time when the broadcaster's device collects the audio and video data;

[0122] The second processing unit 602 is used to adjust the cache control state according to the supplementary enhancement information; the cache control state is used to indicate whether to cache or discard the audio and video data sent by the content delivery network server.

[0123] The third processing unit 603 is used to process audio and video data according to the current video first frame state, and to decode and play the processed audio and video data; the video first frame state is used to indicate whether there is a first frame state or not.

[0124] The apparatus provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0125] In one possible implementation, the second processing unit 602 is specifically used for:

[0126] Determine the delay information based on the system clock information and the synchronization clock information in the supplementary and enhanced information;

[0127] If the delay information is determined to be greater than the first preset threshold, then the cache control state is adjusted to disable caching;

[0128] If the delay information is determined to be less than or equal to the first preset threshold, the cache control state is adjusted to allow caching.

[0129] In one possible implementation, the second processing unit 602, upon determining that the delay information is greater than a first preset threshold, further configures itself to:

[0130] Based on the latency information and the target latency, determine and control the duration for which caching is prohibited.

[0131] In one possible implementation, if the supplementary enhancement information also includes fast-forward instruction information, then after determining the delay information, the second processing unit 602 is further configured to:

[0132] If the delay information is determined to be greater than the second preset threshold, then the non-critical video frames and the audio data corresponding to the non-critical video frames in the audio and video data are discarded; wherein, the second preset threshold is less than the first preset threshold;

[0133] If the delay information is determined to be less than or equal to the second preset threshold, then the fast-forward instruction information will not be executed.

[0134] In one possible implementation, the second processing unit 602 is specifically used for:

[0135] Determine the difference between the delay information and the second preset threshold, and based on the difference, determine the number of non-critical video frames to be discarded in the audio and video data;

[0136] Based on the number of frames discarded, discard non-critical video frames and the audio data corresponding to non-critical video frames from the audio and video data.

[0137] In one possible implementation, the third processing unit 603 is specifically used for:

[0138] If the current video first frame status indicates that there is no first frame, then non-critical video frames in the audio and video data are discarded until a critical video frame is encountered, at which point the video first frame status is adjusted to have a first frame.

[0139] In one possible implementation, the third processing unit 603 is specifically used for:

[0140] The video data from the processed audio and video data is placed into the video queue, and the audio data from the processed audio and video data is placed into the audio queue.

[0141] The system performs video decoding on video data, audio decoding on audio data, displays the video data, and plays the audio data synchronously.

[0142] In one possible implementation, before separating the audio and video data in the buffer, the first processing unit 601 is further configured to:

[0143] Establish a network connection between the playback device and the content distribution network server;

[0144] It receives and parses audio and video data sent by the content delivery network server, and processes the audio and video data according to the cache control state.

[0145] The apparatus provided in this application embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described in detail here.

[0146] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. These modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. Furthermore, they can be stored as program code in the device's memory, and the data processing modules can be called and executed by a specific processing element. The implementation of other modules is similar. These modules can be fully or partially integrated together, or implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0147] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 7 As shown, the electronic device 70 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, memory 702, and communication component 703 are connected via a bus 704.

[0148] In a specific implementation, at least one processor 701 executes computer execution instructions stored in memory 702, causing at least one processor 701 to perform the above-described method.

[0149] The specific implementation process of processor 701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0150] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0151] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0152] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0153] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0154] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0155] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0156] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0157] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0158] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0159] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0160] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0161] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0162] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for processing a live video, characterized by, The method is applied to a playing terminal device, and the method comprises: Separating audio and video data in a cache, marking key video frames in the audio and video data, and parsing supplementary enhancement information in the audio and video data; the supplementary enhancement information at least comprises synchronization clock information, and the synchronization clock information represents a time at which a host terminal device collects the audio and video data; Determining delay information according to system clock information and the synchronization clock information in the supplementary enhancement information; If it is determined that the delay information is greater than a first preset threshold, adjusting a cache control state to be prohibited from caching; If it is determined that the delay information is less than or equal to the first preset threshold, adjusting the cache control state to be allowed to cache; the cache control state is used to indicate whether to cache or discard audio and video data sent by a content distribution network server; Processing the audio and video data according to a current video first frame state, and decoding and playing the processed audio and video data; the video first frame state is used to indicate whether there is a first frame state or not; If the supplementary enhancement information further comprises fast-forward instruction information, after the delay information is determined, the method further comprises: If it is determined that the delay information is greater than a second preset threshold, discarding non-key video frames in the audio and video data and audio data corresponding to the non-key video frames; the second preset threshold is less than the first preset threshold; If it is determined that the delay information is less than or equal to the second preset threshold, not executing the fast-forward instruction information.

2. The method of claim 1, wherein, When it is determined that the delay information is greater than the first preset threshold, the method further comprises: According to the delay information and a target delay, determining and controlling a length of time during which caching is prohibited.

3. The method of claim 1, wherein, The discarding of the non-key video frames in the audio and video data and the audio data corresponding to the non-key video frames comprises: Determining a difference between the delay information and the second preset threshold, and according to the difference, determining a discard frame number of the non-key video frames in the audio and video data; According to the discard frame number, discarding the non-key video frames in the audio and video data and the audio data corresponding to the non-key video frames.

4. The method of claim 1, wherein, The processing of the audio and video data according to the current video first frame state comprises: If the current video first frame state indicates that there is no first frame state at present, discarding the non-key video frames in the audio and video data until a key video frame is encountered, and adjusting the video first frame state to be a first frame state.

5. The method according to any one of claims 1-4, characterized in that, The decoding and playing of the processed audio and video data comprises: Putting video data in the processed audio and video data into a video queue, and putting audio data in the processed audio and video data into an audio queue; Performing video decoding processing on the video data, performing audio decoding processing on the audio data, displaying the video data, and synchronously playing the audio data.

6. The method of claim 5, wherein, Before the separation of the audio and video data in the cache, the method further comprises: Establishing a network connection between the playing terminal device and the content distribution network server; Receiving and parsing audio and video data sent by the content distribution network server, and processing the audio and video data according to a cache control state.

7. A processing apparatus of a live video, characterized by, The device is applied to a playing terminal device, and the device comprises: The first processing unit is configured to separate audio and video data in the cache, mark out key video frames in the audio and video data, and parse out supplementary enhancement information in the audio and video data; the supplementary enhancement information at least includes synchronization clock information, and the synchronization clock information represents a time at which the anchor terminal device collects the audio and video data; The second processing unit is configured to determine delay information according to system clock information and the synchronization clock information in the supplementary enhancement information; if it is determined that the delay information is greater than a first preset threshold, the cache control state is adjusted to be prohibited from caching; if it is determined that the delay information is less than or equal to the first preset threshold, the cache control state is adjusted to be allowed to cache; and the cache control state is used to indicate whether to cache or discard audio and video data sent by a content distribution network server. The third processing unit is configured to process the audio and video data according to a current video first frame state, and decode and play the processed audio and video data; and the video first frame state is used to indicate whether there is a first frame state or not. If the supplementary enhancement information further includes fast-forward instruction information, after the delay information is determined, the second processing unit is further configured to: if it is determined that the delay information is greater than a second preset threshold, discard non-key video frames in the audio and video data and audio data corresponding to the non-key video frames; and if it is determined that the delay information is less than or equal to the second preset threshold, do not execute the fast-forward instruction information; wherein the second preset threshold is less than the first preset threshold.

8. An electronic device, comprising: comprising: a memory and a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the processor executes the method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method in any one of claims 1-6.

10. A computer program product, characterised in that, The computer program is executed by the processor to implement the method in any one of claims 1-6.

Images