Video data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a video data processing method, device and equipment and a storage medium, which are applied to an artificial intelligence technology and a cloud technology, and the method comprises the following steps: through a native conversion interface in an audio and video real-time communication architecture, converting a video coding sequence and an audio coding sequence into a video coding sequence and an audio coding sequence; storing in an external media buffer area of the audio and video real-time communication architecture; according to the playing time respectively corresponding to the video coding sequence and the audio coding sequence in the external media buffer area, performing synchronous playing condition verification on the video coding sequence and the audio coding sequence in the external media buffer area to obtain a verification result; and when the video coding sequence and the audio coding sequence in the external media buffer area meet a synchronous playing condition, transmitting the video coding sequence and the audio coding sequence in the external media buffer area to the native media buffer area through the native conversion interface according to the limited buffer duration. According to the invention, the playing quality and the network jitter resistance of the audio and video data can be improved.

Description

Technical Field

[0001] The present application relates to fields such as artificial intelligence technology and cloud technology, and in particular to a video data processing method, apparatus, device and storage medium. Background Art

[0002] WebRTC (Web Real-Time Communications), a real-time audio and video communication technology, has been widely used in scenarios such as real-time audio and video interaction and conferencing. In practice, real-time audio and video communication technology strictly pursues extremely low latency. In situations such as network packet loss and latency jitter, it uses a strategy of rapidly chasing frames or even dropping frames to ensure real-time audio and video playback. However, this can cause the video to fast-forward, jump, or even become out of sync with the audio and video, making it impossible to ensure smooth, synchronized audio and video playback, thereby reducing the playback quality of audio and video data and its ability to withstand network jitter. Summary of the Invention

[0003] The embodiments of the present application provide a video data processing method, apparatus, device, and storage medium, which can improve the playback quality and resistance to network jitter of audio and video data.

[0004] An embodiment of the present application provides a method for processing video data, including:

[0005] When receiving the video code sequence and the audio code sequence sent by the framing module in the real-time audio and video communication architecture, the video code sequence and the audio code sequence are stored in the external media buffer of the real-time audio and video communication architecture through the native conversion interface in the real-time audio and video communication architecture;

[0006] Determining a limited buffering duration of the external media buffer for the video code sequence according to a reception time of a historical video code sequence, wherein the historical video code sequence is a video code sequence whose reception time is before the reception time of the video code sequence;

[0007] Verifying the synchronous playback conditions of the video code sequence and the audio code sequence in the external media buffer according to the playback times corresponding to the video code sequence and the audio code sequence in the external media buffer, and obtaining a verification result;

[0008] When the above verification result indicates that the video encoding sequence and audio encoding sequence in the above external media buffer meet the synchronous playback conditions, the video encoding sequence and audio encoding sequence in the above external media buffer are transmitted to the native media buffer in the above audio and video real-time communication architecture through the above native conversion interface according to the above limited buffering time.

[0009] An embodiment of the present application provides a video data processing device, including:

[0010] A storage module is configured to, upon receiving the video code sequence and the audio code sequence sent by the framing module in the real-time audio and video communication architecture, store the video code sequence and the audio code sequence in an external media buffer of the real-time audio and video communication architecture through a native conversion interface in the real-time audio and video communication architecture;

[0011] a determination module, configured to determine a limited buffering duration of the external media buffer for the video coding sequence based on a reception time of a historical video coding sequence; the historical video coding sequence being a video coding sequence whose reception time is before the reception time of the video coding sequence;

[0012] a verification module, configured to verify the synchronous playback conditions of the video code sequence and the audio code sequence in the external media buffer according to the playback times respectively corresponding to the video code sequence and the audio code sequence in the external media buffer, and obtain a verification result;

[0013] A transmission module is used to transmit the video coding sequence and audio coding sequence in the external media buffer to the native media buffer in the above-mentioned audio and video real-time communication architecture through the above-mentioned native conversion interface according to the above-mentioned limited buffering time when the above-mentioned verification result indicates that the video coding sequence and audio coding sequence in the above-mentioned external media buffer meet the synchronous playback conditions.

[0014] On one hand, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0015] On one hand, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0016] On one hand, an embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0017] The present application includes at least the following beneficial effects: (1) When a computer device receives a video coding sequence and an audio coding sequence, the video coding sequence and the audio coding sequence are stored in an external media buffer through a native conversion interface in an audio and video real-time communication architecture. When the video coding sequence and the audio coding sequence in the external media buffer meet the synchronous playback conditions, the video coding sequence and the audio coding sequence in the external media buffer are adaptively sent to the native media buffer in the audio and video real-time communication architecture according to the limited buffering time of the external media buffer, thereby achieving synchronous playback of audio data and video data, avoiding fast forwarding, jumping, or even audio and video synchronization of the video screen, and improving the playback quality of audio and video data. (2) Since the limited buffering time of the external media buffer is determined based on the reception time of the historical video coding sequence, the reception time of the historical video coding sequence can reflect the network jitter to a certain extent, that is, the limited buffering time is dynamically determined based on the network jitter; therefore, transmitting the video coding sequence and the audio coding sequence in the external media buffer according to the limited buffering time of the external media buffer is conducive to improving the anti-network jitter capability of audio and video data. (3) By interacting with the external media buffer (storage, reading) through the native conversion interface in the audio and video real-time communication architecture, there is no need to modify the native audio and video real-time communication architecture or customize a private audio and video real-time communication architecture, which reduces the technical requirements and costs of audio and video communication and is conducive to the application, promotion and popularization of the audio and video real-time communication architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 This is a schematic diagram of a video data processing system provided by this application;

[0020] Figure 2 This is the application scenario intention of a video data processing method provided by this application;

[0021] Figure 3 This is a flowchart of a video data processing method provided by this application;

[0022] Figure 4 This is a schematic diagram of a scenario for updating the limited buffering time of an external media buffer provided by the present application;

[0023] Figure 5This is a flowchart of another video data processing method provided by the present application;

[0024] Figure 6 This is a schematic diagram of a scenario of transmitting an audio coding sequence and an audio coding sequence based on limiting the buffering time provided by the present application;

[0025] Figure 7 This is a schematic diagram of a video interface provided by this application;

[0026] Figure 8 is a structural diagram of a video data processing device provided in an embodiment of the present application;

[0027] Figure 9 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0029] The embodiments of the present application may involve artificial intelligence technology and fields such as autonomous driving and smart transportation, that is, the embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0030] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0031] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning. This application mainly involves machine learning technology in artificial intelligence technology, which uses machine learning technology to determine the limited buffering time of the video coding sequence in the external media buffer based on the reception time of the historical video coding sequence. When the video coding sequence and audio coding sequence in the external media buffer meet the synchronous playback conditions, the video coding sequence and audio coding sequence in the external media buffer are adaptively sent to the native media buffer in the audio and video real-time communication architecture according to the limited buffering time of the external media buffer. This can achieve synchronous playback of audio data and video data, avoid fast forwarding, jumping, or even audio and video synchronization in the video picture, and improve the playback quality of audio and video data.

[0032] In order to facilitate a clearer understanding of this application, the video data processing system implementing this application is first introduced. Figure 1 As shown, the video data processing system includes a server 10 and a terminal cluster. The terminal cluster may include one or more terminals, and the number of terminals is not limited here. Figure 1 As shown, taking the terminal cluster including 4 terminals as an example, the terminal cluster can specifically include terminal 1a, terminal 2a, terminal 3a, and terminal 4a; it can be understood that terminal 1a, terminal 2a, terminal 3a, and terminal 4a can all be connected to the server 10 through the network, so that each terminal can exchange data with the server 10 through the network connection.

[0033] It is understandable that the server can be an independent physical server, or a server cluster or distributed system composed of at least two physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms and other basic cloud computing services. The terminal can specifically refer to a vehicle-mounted terminal, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a screen speaker, a smart home appliance, a smart TV, a smart watch, an aircraft, etc., but is not limited to this. Each terminal and server can be directly or indirectly connected by wired or wireless communication. At the same time, the number of terminals and servers can be one or at least two, and this application does not limit this.

[0034] One or more target applications are installed in the terminal. The target applications herein may refer to applications that have video data processing capabilities (e.g., video data playback). For example, target applications may include independent applications, web applications, and mini-programs within a host application. Specifically, target applications may include various applications deployed on a terminal (e.g., an in-vehicle terminal) for purposes such as conferencing, education, messaging, travel, audiobooks, advertising, and navigation.

[0035] For example, when the terminal is an in-vehicle terminal, the target application may be a navigation application, the video coding sequence may be obtained by encoding the corresponding video data on the driving section of the vehicle where the in-vehicle terminal is located, and the video data is used to reflect the driving conditions and driving environment on the driving section, etc. The audio coding sequence may be obtained by encoding the voice data on the driving section, and the voice data may be a voice broadcast of the driving conditions on the driving section, etc. Through this application, the in-vehicle terminal can achieve synchronous playback of voice data and video data, and further, the vehicle can achieve automatic driving, assisted driving, etc. based on the voice data and video data.

[0036] The terminal may include an encoding device and a decoding device. The decoding device may generate video data through a target application, encode video frames and audio frames in the video data to obtain a video code sequence and an audio code sequence, and send the video code sequence and the audio code sequence to the server 10. The server 10 may be configured to send the video code sequence and the audio code sequence to the decoding device. The decoding device may be configured to perform framing, smoothing, decoding, and other processing on the video code sequence and the audio code sequence to obtain video data, and play the video data in the target application.

[0037] It should be noted that the audio and video real-time communication architecture in this application can be a communication architecture built based on real-time audio and video communication technology, and the audio and video real-time communication architecture can be a module in the target application in the terminal. The audio and video real-time communication architecture includes a native media buffer, which is used to smooth the video coding sequence and the audio coding sequence. In actual applications, it is found that the native media buffer has the following defects: (1) The buffering mechanism of the native media buffer is as follows: strictly pursue the ultimate low latency, and in the case of network packet loss, delay jitter, etc., the real-time performance of audio and video playback is guaranteed by the strategy of fast frame chasing or even frame loss. However, the video screen may fast forward, jump, or even be out of sync with the audio and video, and it is impossible to ensure smooth, smooth, and synchronous playback of audio and video, which reduces the playback quality of audio and video data and the ability to resist network jitter. (2) In the low-latency live broadcast scenario, it is necessary to keep the audio and video smooth, smooth, and synchronous playback as low as possible. Therefore, the native media buffer in the real-time audio and video communication architecture cannot meet the requirements for low-latency playback of audio and video while ensuring smooth, synchronized playback. In other words, the native media buffer in the real-time audio and video communication architecture cannot be directly applied to low-latency live streaming scenarios. If you want to apply the native media buffer in the real-time audio and video communication architecture to low-latency live streaming scenarios, you need to modify the native WebRTC SDK (software development kit) or customize a private real-time audio and video communication architecture, which has high technical requirements. In particular, in H5 WebRTC scenarios, you can only use the Web JS interface to call the browser kernel's built-in audio and video real-time communication architecture (i.e., the open source native audio and video real-time communication architecture implementation, i.e., the native WebRTC SDK), and you cannot modify the internal implementation logic. Therefore, directly using the native WebRTC SDK or the browser H5 WebRTC for low-latency live streaming has significantly poor resistance to network jitter and playback quality.

[0038] It should be noted that there are generally two main reasons why the native media buffer causes frame tracking, lag, and audio-video asynchrony:

[0039] 1. In weak network conditions (poor network quality between the encoder and decoder), due to packet loss, retransmission, delay, and jitter, data packets (audio and video) may not arrive within the sent time interval at the receiving end (i.e., the decoder) or may even arrive out of order. Data and video data are affected differently by weak network conditions. Audio frames are relatively small, consisting of a single audio packet, so they only require retransmission and reordering for normal decoding. Video frames, on the other hand, are larger (exceeding the MTU size) and require multiple video packets to form a single frame. Key frames, in particular, often require dozens of video packets. Therefore, under the same network jitter, the reception interval of video frames fluctuates much more than that of audio frames. This can easily lead to audio frames arriving early for playback, while video frames with the same timestamp (playback time) require more time to be reframed before decoding due to packet loss and retransmission. As a result, native WebRTC is prone to frame tracking, lag, and audio-video desynchronization.

[0040] 2. Even under normal network conditions, when the video content has little motion or is still, the key frames (i.e., I-frames, using intra-frame coding) used in video encoding have a large data volume, while the subsequent P-frames (forward reference coded inter-frame residuals with very small data volumes) have a very small data volume. This results in I-frames accounting for a very high bitrate in the entire video stream. This causes the sender (i.e., the encoding device) to use a large time window to send I-frames for bitrate smoothing. Consequently, the I-frames arrive at the receiver much later than the corresponding timestamp (playback time), causing playback lag in WebRTC (a real-time audio and video communication framework).

[0041] Currently, customizing WebRTC's Jitterbuffer, NetEq, and audio / video synchronization logic to improve jitter resistance and smooth, synchronized playback is often done. This approach requires certain technical skills and is unsuitable for most users who directly use the native WebRTC SDK and browser H5 WebRTC scenarios. Furthermore, H5 WebRTC directly calls the browser's native WebRTC, preventing modifications to the internal logic. This limitation severely impacts the promotion and application of WebRTC.

[0042] Based on this, this application adds an external media buffer, which interacts (stores and reads) with the external media buffer through the native conversion interface in the audio and video real-time communication architecture. There is no need to modify the native audio and video real-time communication architecture or customize a private audio and video real-time communication architecture, which reduces the technical requirements and costs of audio and video communication and is conducive to the application and popularization of the audio and video real-time communication architecture. It is suitable for a variety of scenarios, such as low-latency playback (browser live broadcast scenarios, social application live broadcast scenarios), real-time audio and video interaction, conferencing, content distribution and other scenarios.

[0043] In addition, when the video coding sequence and audio coding sequence in the external media buffer meet the synchronous playback conditions, the video coding sequence and audio coding sequence in the external media buffer are adaptively sent to the native media buffer in the audio and video real-time communication architecture according to the limited buffering time of the external media buffer, which can achieve synchronous playback of audio data and video data, avoid fast forwarding, jumping, or even audio and video synchronization in the video screen, and improve the playback quality of audio and video data. Since the limited buffering time of the external media buffer is determined based on the reception time of the historical video coding sequence, the reception time of the historical video coding sequence can reflect the network jitter to a certain extent, that is, the limited buffering time is dynamically determined based on the network jitter; therefore, transmitting the video coding sequence and audio coding sequence in the external media buffer according to the limited buffering time of the external media buffer is conducive to improving the anti-network jitter capability of audio and video data.

[0044] It should be noted that native media buffers can include JitterBuffer and NetEq. JitterBuffer is used to store coded video sequences, while NetEq is used to store coded audio sequences. External media buffers can include video buffers and audio buffers. The video buffer can be used to cache video coded sequences, while the audio buffer can be used to cache audio data coded sequences.

[0045] It should be noted that the external media buffer refers to a buffer set up outside the real-time audio and video communication framework. This external media buffer is not controlled by the real-time audio and video communication framework. In other words, deploying an external media buffer and changing its size do not require modifying the real-time audio and video communication framework. The native media buffer, on the other hand, is deployed within the real-time audio and video communication framework and is controlled by it. Changing the size of the native media buffer requires modifying the real-time audio and video communication framework.

[0046] It should be noted that the real-time audio and video communication architecture may include a framing module, a native conversion interface, a native media buffer, a decoding module, and a playback module.

[0047] The framing module can be used to perform framing processing on the received video data packets and audio data packets respectively to obtain a video coding sequence and an audio coding sequence.

[0048] Among them, the native conversion interface can be an interface defined in the audio and video real-time communication architecture, that is, the native conversion interface can be an interface that comes with the audio and video real-time communication architecture, and the native conversion interface can communicate with each module in the audio and video real-time communication architecture. The native conversion interface can be used to process the data after encoding and before decoding in the audio and video real-time communication architecture. Specifically, the native conversion interface can be used to store the video encoding sequence and audio encoding sequence obtained by the framing module in the external media buffer. The native conversion interface can be used to store the video encoding sequence obtained by the framing module in the video buffer, and cache the audio encoding sequence obtained by the framing module in the audio buffer. The external media buffer is used to synchronize the video encoding sequence and the audio encoding sequence, that is, when the video encoding frames and audio encoding frames stored in the external media buffer meet the synchronous playback conditions, the video encoding sequence and audio encoding sequence in the external media buffer are transferred to the native media buffer through the native conversion interface.

[0049] The native media buffer can be used to smooth video code sequences and audio code sequences. A video code sequence can include one or more video code frames, and an audio code sequence can include one or more audio code frames. Specifically, the native media buffer can detect network packet loss, latency jitter, and other anomalies based on the video and audio code sequences. If network packet loss, latency jitter, and other anomalies are present, the native media buffer uses fast frame tracking and frame loss strategies to ensure real-time playback of audio and video data.

[0050] It should be noted that, because the video and audio code sequences have already been synchronized in the external media buffer, the video and audio code sequences transmitted to the native media buffer are free from anomalies such as network packet loss and latency jitter. This shows that the video and audio code sequences transmitted to the native media buffer will not trigger the native media buffer's corresponding fast frame tracking and frame loss strategies. This can avoid fast-forwarding, jumps, and even audio and video asynchrony in the video, thereby improving the playback quality of audio and video data.

[0051] Among them, the decoding module can obtain the video coding sequence and the audio coding sequence from the native media buffer, decode the video coding sequence to obtain a video sequence, and decode the audio coding sequence to obtain an audio sequence.

[0052] The playback module can be used to render the video sequence, obtain the video image, and play the video image and the audio sequence synchronously.

[0053] Figure 1 The video data processing system in can be used for the video data processing method in this application, such as Figure 2As shown, the encoding device is Figure 1 The terminal 1a and the decoding device are Figure 1 Taking the terminal 2a in FIG. 1 as an example, the video data processing method may include the following steps S21 to S25:

[0054] S21. Framing: Terminal 2a can receive video data packets and audio data packets sent by terminal 1a via the network. The video data packets can be video Real-time Transport Protocol (RTP) packets, and the audio data packets can be audio RTP packets. Terminal 2a can use a framing module in the real-time audio and video communication architecture to perform framing on the video data packets to obtain a video code sequence 21a, and perform framing on the audio data packets to obtain an audio code sequence 22a.

[0055] The video coding sequence 21a may include one or more video coding frames, and the audio coding sequence 22a may include one or more audio coding frames, such as Figure 2 In the example, a video coding sequence 21a includes five video coding frames and an audio coding sequence 22a includes five audio coding frames. The video coding sequence 21a may include five video coding frames, labeled as video coding frame 211b, video coding frame 212b, video coding frame 213b, video coding frame 214b, and video coding frame 215b; the audio coding sequence 22a may include five audio coding frames, labeled as audio coding frame 221c, audio coding frame 222c, audio coding frame 223c, audio coding frame 224c, and audio coding frame 225c. The corresponding playback times of video coding frame 211b, video coding frame 212b, video coding frame 213b, video coding frame 214b and video coding frame 215b are t1, t2, t3, t4, t5 respectively; the corresponding playback times of audio coding frame 221c, audio coding frame 222c, audio coding frame 223c, audio coding frame 224c and audio coding frame 225c are t1, t2, t3, t4, t5 respectively.

[0056] from Figure 2 It can be seen that due to network jitter between terminal 21a and terminal 2a, the video coding sequence and audio coding sequence have delay jitter, and the delay jitter can be caused by different reception time intervals between adjacent frames. Figure 2, the first receiving time interval is different from the second receiving time interval. The first receiving time interval is the interval between the receiving time of the video coding frame 213b and the receiving time of the video coding frame 212b, and the second receiving time interval is the interval between the receiving time of the video coding frame 213b and the receiving time of the video coding frame 214b. Similarly, the third receiving time interval is different from the fourth receiving time interval. The third receiving time interval is the interval between the receiving time of the audio coding frame 223c and the receiving time of the audio coding frame 222c, and the fourth receiving time interval is the interval between the receiving time of the audio coding frame 223c and the receiving time of the audio coding frame 224c. If video code sequence 21a is sent directly to JitterBuff and audio code sequence 22a is sent directly to NetEq, JitterBuff will discard some video code frames in video code sequence 21a or use a frame-chasing strategy to ensure the real-time transmission of the video code sequence. NetEq will discard some audio code frames in audio code sequence 22a or use a frame-chasing strategy to ensure the real-time transmission of the audio code sequence. This can cause the video screen to fast-forward, jump, or even audio and video synchronization, resulting in relatively low playback quality of the audio and video data. Based on this, it is necessary to perform the following step S22 on video code sequence 21a and audio code sequence 22a.

[0057] S22, external smooth synchronization; Terminal 2a can store the video coding sequence 21a in the video buffer and the audio coding sequence 22a in the audio buffer through the native conversion interface in the audio and video real-time communication architecture, and determine the external media buffer (i.e., the video buffer) for the video coding sequence according to the reception time of the historical video coding sequence. According to the playback duration of the video coding sequence 21a in the video buffer and the corresponding playback duration of the audio coding sequence 22a in the audio buffer, the video coding sequence 21a and the audio coding sequence 22a are subjected to synchronous playback condition verification to obtain a verification result. When the verification result indicates that the video coding sequence 21a in the video buffer and the audio coding sequence 22a in the audio buffer meet the asynchronous playback condition, the video coding sequence 21a in the video buffer and the audio coding sequence 22a in the audio buffer can be waited for to meet the synchronous playback condition. Then, according to the limited buffering duration, the video coding sequence 21a in the video buffer is transferred to JitterBuffer, and the audio coding sequence 22a in the audio buffer is transferred to NetEq. When the verification result indicates that the video code sequence 21a in the video buffer and the audio code sequence 22a in the audio buffer meet the synchronous playback conditions, the video code sequence 21a in the video buffer is transferred to JitterBuffer and the audio code sequence 22a in the audio buffer is transferred to NetEq through the native conversion interface according to the limited buffering time. Figure 2 The transmission time intervals between adjacent video coding frames transmitted from the video buffer are relatively uniform, and the transmission time intervals between adjacent audio coding frames transmitted from the audio buffer are relatively uniform; the delay jitter between the video coding sequence 21a and the audio coding sequence 22a can be eliminated through the external media buffer (i.e., the video buffer and the audio buffer), thereby achieving synchronization between the video coding sequence 21a and the audio coding sequence 22a.

[0058] S23, internal smooth synchronization: After the video code sequence 21a is stored in the JitterBuffer and the audio code sequence 22a is stored in the NetEq, the terminal 2a can detect whether there is delay jitter in the video code sequence 21a and the audio code sequence 22a. Since the external media buffer has synchronized the video code sequence 21a and the audio code sequence 22a, if there is delay jitter in the video code sequence 21a and the audio code sequence 22a, strategies such as frame dropping and frame chasing will not be triggered. The video code sequence 21a in the JitterBuffer and the audio code sequence 22a in the NetEq are directly sent to the decoding module.

[0059] S24. Decoding: The video coding frames in the video coding sequence 21a are decoded by a decoding module to obtain a video sequence, and the audio coding frames in the audio coding sequence 22a are decoded to obtain an audio sequence; the video sequence may include a video frame obtained by decoding each video coding frame, and the audio sequence may include an audio frame obtained by decoding each audio coding frame.

[0060] S25, play, send the video sequence and audio sequence to the play module in the audio and video real-time communication architecture, the play module can be used to render the video frames in the video sequence, obtain the video picture (video data), and synchronously play the video picture and audio sequence (audio data).

[0061] In summary, when the video coding sequence and audio coding sequence in the external media buffer meet the synchronous playback conditions, the video coding sequence and audio coding sequence in the external media buffer are adaptively sent to the native media buffer in the audio and video real-time communication architecture according to the limited buffering time of the external media buffer, so as to realize the synchronous playback of audio data and video data, avoid fast forwarding, jumping or even audio and video out of sync in the video screen, and improve the playback quality of audio and video data.

[0062] Further, see Figure 3 , is a flow chart of a video data processing method provided by an embodiment of the present application. Figure 3 As shown, this method can be Figure 1 It can be executed by any terminal in the terminal cluster in Figure 1 It can also be executed by the server in Figure 1 The terminals and servers in the terminal cluster are used to perform the video data processing method in collaboration. In this application, the devices used to perform the video data processing method can be collectively referred to as computer devices. The method may include the following steps:

[0063] S101. When a video coding sequence and an audio coding sequence are received from a framing module in a real-time audio and video communication architecture, the video coding sequence and the audio coding sequence are stored in an external media buffer of the real-time audio and video communication architecture through a native conversion interface in the real-time audio and video communication architecture.

[0064] In the present application, after receiving the video coding sequence and audio coding sequence sent by the framing model in the audio and video real-time communication architecture, the computer device stores the above-mentioned video coding sequence and the above-mentioned audio coding sequence in the external media buffer of the above-mentioned audio and video real-time communication architecture through the native conversion interface in the above-mentioned audio and video real-time communication architecture. There is no need to change the native audio and video real-time communication architecture or customize a private audio and video real-time communication architecture, which reduces the technical requirements and costs of audio and video communication and is conducive to the application, promotion and popularization of the audio and video real-time communication architecture.

[0065] The size of the external media buffer can be determined based on the network quality and physical distance between the encoding device and the decoding device (i.e., the computer device). The external media buffer can include a video buffer and an audio buffer. The video buffer can be used to store video code sequences, and the audio buffer can be used to store audio code sequences. Alternatively, the external media buffer is an independent buffer, that is, the audio code sequence and the video code sequence are stored in the same buffer.

[0066] In one embodiment, the above step S101 may include the following steps S31 to S33:

[0067] S31. Obtain the architecture type to which the above-mentioned audio and video real-time communication architecture belongs.

[0068] Specifically, the computer device can obtain the architecture type of the audio and video real-time communication mechanism from the installation file of the audio and video real-time communication architecture. The architecture type may include an open source software development kit type and a web page type. For example, an open source software development kit type audio and video real-time communication architecture may include a WebRTC SDK; a web page type audio and video real-time communication architecture may include H5 WebRTC.

[0069] S32: Determine a native conversion interface in the real-time audio and video communication architecture according to the architecture type to which the real-time audio and video communication architecture belongs.

[0070] Specifically, since the corresponding native conversion interfaces of audio and video real-time communication architectures of different architecture types are inconsistent, the computer device can determine the native conversion interface in the audio and video real-time communication architecture according to the architecture type to which the audio and video real-time communication mechanism belongs.

[0071] S33. The video encoding sequence and the audio encoding sequence are stored in the external media buffer of the audio and video real-time communication architecture through the determined native conversion interface.

[0072] Specifically, when the external media buffer is an independent buffer, the computer device can store the video code sequence and the audio code sequence in the external media buffer of the audio and video real-time communication architecture through the obtained native conversion interface. When the external media buffer includes a video buffer and an audio buffer, the computer device can store the video code sequence in the video buffer and the audio code sequence in the audio buffer through the obtained native conversion interface.

[0073] Optionally, the above step S32 may include the following steps S321 and S322:

[0074] S321. When the architecture type of the above-mentioned audio and video real-time communication architecture belongs to the open source software development kit type, the frame conversion interface in the audio and video real-time communication architecture of the open source software development kit type is determined as the native conversion interface in the above-mentioned audio and video real-time communication architecture.

[0075] Specifically, when the architecture type of the above-mentioned audio and video real-time communication architecture belongs to the open source software development kit type, the frame conversion (Frame Transformer) interface in the audio and video real-time communication architecture of the open source software development kit type can be determined as the native conversion interface of the audio and video real-time communication architecture.

[0076] S322: When the architecture type of the real-time audio and video communication architecture is a web page type, the encoded conversion interface in the web page interface of the real-time audio and video communication architecture of the web page type is determined as the native conversion interface in the real-time audio and video communication architecture.

[0077] Specifically, when the architecture type of the above-mentioned audio and video real-time communication architecture belongs to the web page type, the encoded conversion interface (Encoded Transformer) interface in the web page interface (Web API) of the web page type audio and video real-time communication architecture is determined as the native conversion interface in the audio and video real-time communication architecture.

[0078] S102: Determine a limited buffering duration of the external media buffer for the video coding sequence according to a reception time of a historical video coding sequence; the historical video coding sequence is a video coding sequence whose reception time is before the reception time of the video coding sequence.

[0079] In the present application, since the video coding sequence is more susceptible to network jitter, the computer device can determine the limited buffering time of the external media buffer for the video coding sequence based on the reception time of the historical video coding sequence, and the historical video coding sequence is a video coding sequence whose reception time is before the reception time of the video coding sequence.

[0080] Among them, the limited buffering time is used to limit the transmission speed of the video coding sequence in the external media buffer. The receiving time of the historical video coding sequence can reflect the network jitter between the encoding device and the decoding device, as well as the delay jitter of the historical video coding sequence to a certain extent. For example, when the network jitter between the encoding device and the decoding device is relatively serious, then the time interval between the receiving time of the historical video coding sequence fluctuates greatly, which causes the historical video coding sequence to have a relatively large delay jitter. Similarly, when the network jitter between the encoding device and the decoding device is relatively mild, then the time interval between the receiving time of the historical video coding sequence is relatively stable (i.e., uniform), which causes the historical video coding sequence to have a relatively small delay jitter or no delay jitter. Therefore, it can be said that the limited buffering time is dynamically determined based on the network jitter and the delay jitter of the video coding sequence, which is conducive to improving the anti-network jitter capability of audio and video data.

[0081] In one embodiment, the above step S102 may include the following steps S41 to S43:

[0082] S41: When the limited buffering time corresponding to the external media buffer satisfies an update condition, the position of the sliding window in the historical video coding sequence when the limited buffering time was last updated is obtained.

[0083] Specifically, the first limited buffering duration corresponding to the external media buffer may be initialized. Thereafter, the limited buffering duration corresponding to the external media buffer may be updated based on the historical video coding sequence. During the first update, a sliding window may be used to obtain the historical video coding sequence whose reception time is closest to the current reception time. The limited buffering duration corresponding to the external media buffer may be updated based on the reception time of the obtained historical video coding sequence. When the update is not the first time, the computer device may obtain the position of the sliding window in the historical video coding sequence when the limited buffering duration was last updated.

[0084] S42. Starting from the above-mentioned stop position, the above-mentioned sliding window is moved in the above-mentioned historical video coding sequence according to the sliding step size.

[0085] Specifically, the computer device may move the sliding window within the historical video coding sequence, starting from the stop position and according to the sliding step size, in a direction closer to the video coding sequence corresponding to the current reception time. In other words, the reception time of the historical video coding frame closest to the video coding frame in the external media buffer is used to update the limited buffering time of the external media buffer for the video coding sequence. The reception time of the closest historical video coding frame can better reflect the current network jitter, thereby improving the accuracy of updating the limited buffering time.

[0086] S43: Update the limited buffering time of the external media buffer for the video coding sequence according to the reception time of the historical video coding frame currently contained in the sliding window.

[0087] Specifically, the computer device can adaptively update the limited buffering time of the external media buffer for the video encoding sequence based on the reception time of the historical video encoding frame currently contained in the above sliding window, which is conducive to the limited buffering time being able to reflect the network jitter between the encoding device and the decoding device in real time.

[0088] Optionally, step S41 may include the following steps S411 to S415:

[0089] S411: Obtain historical log data of a device for receiving the video coding sequence and the audio coding sequence.

[0090] The computer device can obtain the historical log data of the device (i.e., the decoding device) used to receive the above-mentioned video coding sequence and audio coding sequence. The historical log data can reflect the delay jitter corresponding to the video coding frames received by the decoding device within the historical time period, as well as the network jitter between the encoding device and the decoding device.

[0091] S412: Determine the network stability of the device based on the historical log data.

[0092] Specifically, the computer device can determine the network stability of the decoding device based on the historical log data. For example, if the delay jitter and network jitter are severe, the network stability of the decoding device is relatively low; if the delay jitter and network jitter are relatively mild, the network stability of the decoding device is relatively high.

[0093] S413: Generate an update time interval corresponding to the limited buffering time length according to the above network stability.

[0094] Specifically, the computer device can generate an update time interval corresponding to the limited buffer time based on the network stability. For example, the higher the network stability, the network between the decoding device and the encoding device is in the same state for a long time, and the longer the update time interval, which means the update frequency of the limited buffer time is reduced; the lower the network stability, the greater the fluctuation of the network state between the decoding device and the encoding device, and therefore, the smaller the update time interval, which means the update frequency of the limited buffer time is increased.

[0095] S414: Determine the update time of this update limit buffering time according to the above update time interval and the update time of the last update limit buffering time.

[0096] Specifically, the computer device may sum the update time interval and the update time of the last update limit buffer time length to obtain the update time of the current update limit buffer time length.

[0097] S415. When the time reaches the update time of the current limited buffering time, determine that the limited buffering time corresponding to the external media buffer meets the update condition, and obtain the stop position of the sliding window in the historical video encoding sequence when the limited buffering time was last updated.

[0098] Specifically, when the time reaches the update time of this updated limited buffering time, it is determined that the limited buffering time corresponding to the above-mentioned external media buffer meets the update conditions, and the stop position of the sliding window in the historical video encoding sequence when the limited buffering time was last updated is obtained.

[0099] For example, Figure 4As shown, at time T1, the external media buffer 41a includes a video code sequence 42a, the sliding window 43a includes a historical video code sequence 44a, and the sliding window also includes a historical video code sequence 45a. The historical video code sequence 44a and the historical video code sequence 45a may be video code sequences transmitted to the native media buffer. The computer device can calculate a limited buffering duration 46a for the external media buffer based on the reception time of the historical video code sequence 44a within the sliding window. When the video code sequence 42a and the audio code sequence within the external media buffer meet the synchronous playback conditions, the computer device can transmit the video code sequence 42a and the audio code sequence from the external media buffer to the native media buffer according to the limited buffering duration 46a. At time T2, the external media buffer includes a video code sequence 47a, indicating that the video code sequence 42a has been transmitted to the native media buffer. Therefore, the video code sequence 42a can be recorded as a historical video code sequence. Assuming the sliding step size is the duration corresponding to five video code frames, the computer device can slide the window 43a toward the direction of the video code frame 47. At this time, the sliding window 43a includes the 2nd to 5th video coding frames in the video coding sequence 42a. The computer device can calculate the limited buffering duration 48a based on the reception time of the video coding frames in the sliding window 43a. When the video coding sequence 47a and the audio coding sequence in the external media buffer meet the synchronous playback conditions, the computer device can transmit the video coding sequence 47a and the audio coding sequence in the external media buffer to the native media buffer according to the limited buffering duration 48a.

[0100] S103: Verify the synchronous playback conditions of the video code sequence and the audio code sequence in the external media buffer according to the playback times respectively corresponding to the video code sequence and the audio code sequence in the external media buffer, and obtain a verification result.

[0101] In the present application, the computer device can verify whether the video code sequence and the audio code sequence in the external media buffer meet the synchronous playback condition based on the playback time corresponding to the video code sequence and the audio code sequence in the external media buffer, and obtain a verification result. The verification result reflects that the video code sequence and the audio code sequence in the external media buffer meet the synchronous playback condition, or the verification result reflects that the video code sequence and the audio code sequence in the external media buffer do not meet the synchronous playback condition.

[0102] It should be noted that the video coding sequence and audio coding sequence in the external media buffer satisfying the synchronous playback condition may mean that: the playback time of the video coding sequence and the playback time of the audio coding sequence in the external media buffer are aligned, that is, the video coding frames and audio coding frames with the same playback time have arrived in the external media buffer. The video coding sequence and audio coding sequence in the external media buffer not satisfying the synchronous playback condition may mean that: the playback time of the video coding sequence and the playback time of the audio coding sequence in the external media buffer are not aligned, such as one of the video coding frames and audio coding frames with the same playback time has not arrived in the external media buffer.

[0103] The video encoding frames and the audio encoding frames are arranged in the media buffer according to the play order (ie, play time).

[0104] S104. When the above verification result indicates that the video coding sequence and the audio coding sequence in the above external media buffer meet the synchronous playback conditions, the video coding sequence and the audio coding sequence in the above external media buffer are transmitted to the native media buffer in the above audio and video real-time communication architecture through the above native conversion interface according to the above limited buffering time.

[0105] In the present application, when the verification result indicates that the video coding sequence and the audio coding sequence in the external media buffer do not meet the synchronous playback condition, it is necessary to wait for the video coding frames and the audio coding frames of the same playback time to arrive in the external media buffer to determine that the video coding sequence and the audio coding sequence in the external media buffer meet the synchronous playback condition. When the verification result indicates that the video coding sequence and the audio coding sequence in the external media buffer meet the synchronous playback condition, the video coding sequence and the audio coding sequence in the external media buffer are transferred to the native media buffer in the above-mentioned audio and video real-time communication architecture according to the limited buffering duration through the above-mentioned native conversion interface. By interacting with the external media buffer through the native conversion interface, there is no need to make additional modifications to the audio and video real-time communication architecture, or customize a private audio and video real-time communication architecture, thereby reducing the technical requirements and costs of audio and video communications, and is conducive to the application, promotion and popularization of the audio and video real-time communication architecture.

[0106] The present application includes at least the following beneficial effects: (1) When a computer device receives a video coding sequence and an audio coding sequence, the video coding sequence and the audio coding sequence are stored in an external media buffer through a native conversion interface in an audio and video real-time communication architecture. When the video coding sequence and the audio coding sequence in the external media buffer meet the synchronous playback conditions, the video coding sequence and the audio coding sequence in the external media buffer are adaptively sent to the native media buffer in the audio and video real-time communication architecture according to the limited buffering time of the external media buffer, thereby achieving synchronous playback of audio data and video data, avoiding fast forwarding, jumping, or even audio and video synchronization of the video screen, and improving the playback quality of audio and video data. (2) Since the limited buffering time of the external media buffer is determined based on the reception time of the historical video coding sequence, the reception time of the historical video coding sequence can reflect the network jitter to a certain extent, that is, the limited buffering time is dynamically determined based on the network jitter; therefore, transmitting the video coding sequence and the audio coding sequence in the external media buffer according to the limited buffering time of the external media buffer is conducive to improving the anti-network jitter capability of audio and video data. (3) By interacting with the external media buffer (storage, reading) through the native conversion interface in the audio and video real-time communication architecture, there is no need to modify the native audio and video real-time communication architecture or customize a private audio and video real-time communication architecture, which reduces the technical requirements and costs of audio and video communication and is conducive to the application, promotion and popularization of the audio and video real-time communication architecture.

[0107] Further, see Figure 5 , is a flow chart of a video data processing method provided by an embodiment of the present application. Figure 5 As shown, this method can be Figure 1 It can be executed by any terminal in the terminal cluster in Figure 1 It can also be executed by the server in Figure 1 The terminals and servers in the terminal cluster are used to perform the video data processing method in collaboration. In this application, the devices used to perform the video data processing method can be collectively referred to as computer devices. The method may include the following steps:

[0108] S201. When a video coding sequence and an audio coding sequence are received from a framing module in a real-time audio and video communication architecture, the video coding sequence and the audio coding sequence are stored in an external media buffer of the real-time audio and video communication architecture through a native conversion interface in the real-time audio and video communication architecture.

[0109] S202: Determine a limited buffering duration of the external media buffer for the video coding sequence according to a reception time of a historical video coding sequence; the historical video coding sequence is a video coding sequence whose reception time is before the reception time of the video coding sequence.

[0110] In one embodiment, the reception time of the historical video coding sequence includes the reception time of each video coding frame in the historical video coding sequence;

[0111] Optionally, step S202 may include the following steps S51 to S52:

[0112] S51 : Determine a receiving time interval between two adjacent video encoding frames according to the receiving times corresponding to each two adjacent video encoding frames in the historical video encoding sequence.

[0113] Specifically, the computer device may determine the reception time interval between two adjacent video coding frames based on the reception times corresponding to each two adjacent video coding frames in the historical coding sequence. Adjacent video coding frames may refer to two video coding frames with adjacent playback times. For example, the computer device may perform a difference processing on the reception times corresponding to the two adjacent video coding frames to obtain the reception time interval between the two adjacent video coding frames.

[0114] S52: Determine a limited buffering duration of the external media buffer for the video coding sequence according to a reception time interval between every two adjacent video coding frames in the historical video coding sequence.

[0115] Specifically, the computer device may average the reception time intervals between every two adjacent video encoding frames in the historical video encoding sequence to obtain the limited buffering duration of the external media buffer for the video encoding sequence.

[0116] It should be noted that there is a positive correlation between the receiving interval and the limited buffering period. A longer receiving interval indicates more severe network jitter between the encoder and decoder, and a longer limited buffering period. A shorter receiving interval indicates less network jitter between the encoder and decoder, and a shorter limited buffering period. Therefore, the limited buffering period can, to a certain extent, reflect network jitter.

[0117] In one embodiment, the limited buffer time includes a first limited buffer time and a second limited buffer time, and the first limited buffer time is greater than the second limited buffer time.

[0118] Optionally, step S52 may include the following steps S521 to S524:

[0119] S521 . Perform weighted averaging processing on the reception time interval between every two adjacent video coding frames in the historical video coding sequence according to the data amount corresponding to the video coding frames in the historical video coding sequence to obtain a first average time interval.

[0120] Specifically, the computer device can perform weighted averaging processing on the receiving time interval between each adjacent video coding frame in the historical video coding sequence according to the data amount corresponding to the video coding frame in the historical video coding sequence to obtain a first average time interval; the data amount corresponding to the video coding frame can refer to the size of the video coding frame.

[0121] S522: Determine a first limited buffering duration of the external media buffer for the video coding sequence according to the first average time interval.

[0122] Specifically, the computer device may determine a first limited buffering duration of the external media buffer for the video coding sequence according to the first average time interval, where the first limited buffering duration is the maximum limited buffering duration.

[0123] For example, the computer device may calculate the first limited buffering time length using the following formula (1):

[0124] T max =K2*WA(S i , t i ) (1)

[0125] In formula (1), T max is the first limit buffering time, WA is the weighted average processing, S i is the amount of data corresponding to the i-th video encoding frame, t i It may refer to the receiving time interval between the receiving time of the i-th video coding frame and the receiving time of the (i+1)-th video coding frame. K2 is a configuration parameter and is an empirical value.

[0126] S523: Perform averaging processing on the reception time interval between every two adjacent video encoding frames in the historical video encoding sequence to obtain a second average time interval.

[0127] Specifically, the computer device may accumulate and sum the reception time intervals between each adjacent video encoding frame in the historical video encoding sequence to obtain a total reception time interval, and divide the total reception time interval by the number of reception time intervals to obtain a second average time interval.

[0128] S524: Determine a second limited buffering duration of the external media buffer for the video coding sequence according to the second average time interval.

[0129] Specifically, the computer device may determine a second limited buffering duration of the external media buffer for the video coding sequence according to the second average time interval, where the second limited buffering duration may refer to a minimum limited buffering duration.

[0130] For example, the computer device may calculate the first limited buffering time length using the following formula (2):

[0131] T min =K1*Avg(t i ) (2)

[0132] In formula (2), T min The first limit is the buffering time. K1 is a configuration parameter, which is an empirical value. Avg is an averaging process.

[0133] S203: Verify the synchronous playback conditions of the video code sequence and the audio code sequence in the external media buffer according to the playback times respectively corresponding to the video code sequence and the audio code sequence in the external media buffer, and obtain a verification result.

[0134] In one embodiment, the above step S203 may include the following steps S61 to S63:

[0135] S61. Determine a target video encoding frame whose playback time is the earliest playback time based on the playback time corresponding to the video encoding sequence in the external media buffer. Specifically, the computer device can determine the target video encoding frame whose playback time is the earliest playback time based on the playback time corresponding to the video encoding sequence in the external media buffer. For example, the video encoding frames are arranged in the video buffer in descending order of playback time, i.e., the playback time of the video encoding frame arranged at the head of the queue in the video buffer is the earliest playback time, and the playback time of the video encoding frame arranged at the tail of the queue is the latest playback time. In other words, the target video encoding frame can be the video encoding frame arranged at the head of the queue in the video buffer.

[0136] S62: Determine a target audio code frame whose play time is the earliest play time according to the play time corresponding to the audio code sequence in the external media buffer.

[0137] Specifically, the computer device may determine a target audio code frame whose playback time is the earliest playback time based on the playback time corresponding to the audio code sequence in the external media buffer. For example, the audio code frames are arranged in the audio buffer in descending order of playback time, i.e., the playback time of the audio code frame arranged at the head of the queue in the audio buffer is the earliest playback time, and the playback time of the audio code frame arranged at the tail of the queue is the latest playback time. In other words, the target audio code frame may be the audio code frame arranged at the head of the queue in the audio buffer.

[0138] S63. Verify the synchronous playback conditions of the video code sequence and the audio code sequence in the external media buffer according to the playback time corresponding to the target audio code frame and the playback time corresponding to the target video code frame, and obtain a verification result.

[0139] Specifically, the computer device may verify the synchronous playback conditions of the video coding sequence and the audio coding sequence in the external media buffer according to the playback time corresponding to the target audio coding frame and the playback time corresponding to the target audio coding frame to obtain a verification result.

[0140] Optionally, step S63 may include the following steps S631 to S633:

[0141] S631: Determine a playback time interval between the playback time corresponding to the target audio coding frame and the playback time corresponding to the target video coding frame.

[0142] Specifically, the computer device may perform a difference processing on the playback time corresponding to the target audio coding frame and the playback time corresponding to the target video coding frame to obtain the playback time difference, and determine the absolute value of the playback time difference as the playback time interval.

[0143] S632: When the playback time interval is less than the time interval threshold, determine that the video encoding sequence and the audio encoding sequence in the external media buffer satisfy a synchronous playback condition.

[0144] Specifically, when the playback time interval is greater than or equal to the time interval threshold, it indicates that one of the audio coding frame and the video coding frame with the same playback time is not in the external media buffer. Therefore, it is determined that the video coding sequence and the audio coding sequence in the external media buffer do not meet the synchronous playback condition. When the playback time interval is less than the time interval threshold, it indicates that the audio coding frame and the video coding frame with the same playback time are both in the external media buffer. Therefore, it is determined that the video coding sequence and the audio coding sequence in the external media buffer meet the synchronous playback condition.

[0145] For example, when the playback time interval between the playback time corresponding to the target audio coding frame and the playback time corresponding to the above-mentioned target video coding frame satisfies the following formula (3), the video coding sequence and the audio coding sequence in the external media buffer can meet the synchronous playback condition. Formula (3) can be expressed as:

[0146] abs(A j -V j ) <K3 (3)

[0147] Among them, A in formula (3) jIndicates the playback time of the audio encoding frame at the head of the queue corresponding to the audio encoding frame in the external media buffer, V j The playback time of the most recently transmitted video encoding frame (ie, the target encoding video frame) in the external media buffer, abs is the absolute value, K3 may refer to the playback time interval threshold, which may be an empirical value.

[0148] S633: Generate a verification result reflecting that the video encoding sequence and the audio encoding sequence in the external media buffer satisfy a synchronous playback condition.

[0149] Specifically, when the video coding sequence and the audio coding sequence in the external media buffer satisfy the synchronous playback condition, a verification result reflecting that the video coding sequence and the audio coding sequence in the external media buffer satisfy the synchronous playback condition can be generated.

[0150] In one embodiment, step S203 may include: arranging the video coded frames in the video coded sequence in a first queue in the external media buffer in descending order of play time; and arranging the audio coded frames in the audio coded sequence in a second queue in the external media buffer in descending order of play time. The computer device may sequentially calculate the play time interval between the play time of the video coded frames and the play time of the audio coded frames in the first queue and the second queue that have the same play time interval. When the calculated play time interval is less than a time interval threshold, the audio coded frames and the video coded frames corresponding to the play time interval in the external media buffer are transferred to the native media buffer. When the calculated play time interval is greater than or equal to the time interval threshold, it may indicate that a frame has been lost, or that a transmission delay is relatively large, resulting in one of the video coded frames and audio coded frames with the same play time not arriving in the media buffer. Therefore, after the corresponding frame arrives and the play time interval is less than the time interval threshold, the audio coded frames and the video coded frames corresponding to the play time interval in the external media buffer are transferred to the native media buffer, and the play time interval between the play time of the next video coded frame and the play time of the next audio coded frame in the first queue and the second queue that have the same play time interval is continuously calculated. The above steps are repeated until all video encoding frames and audio encoding frames are transferred to the native media buffer.

[0151] S204: When the verification result indicates that the video coding sequence and the audio coding sequence in the external media buffer satisfy a synchronous playback condition, obtain a playback duration corresponding to the current video coding sequence in the external media buffer.

[0152] In the present application, when the above-mentioned verification result indicates that the video coding sequence and the audio coding sequence in the above-mentioned external media buffer satisfy the synchronous playback condition, the playback duration corresponding to the current video coding sequence in the external media buffer can be determined based on the playback time corresponding to each video coding frame in the video coding sequence in the external media buffer. For example, the playback duration corresponding to the current video coding sequence in the external media buffer can be obtained by taking the difference between the latest playback time and the earliest playback time corresponding to the video coding frames of the current video coding sequence in the external media buffer.

[0153] S205: Determine a transmission speed of the video encoding sequence in the external media buffer according to the playback duration and the limited buffering duration.

[0154] In this application, since video encoding frames are more easily affected by network jitter, the computer device can determine the transmission speed of the video encoding sequence in the external media buffer based on the playback time and the limited buffering time. The transmission speed of the audio encoding sequence is consistent with the transmission speed of the video encoding sequence, thereby adaptively determining the transmission speed of the video encoding sequence based on network jitter and delay jitter, thereby improving the ability of audio and video data to resist network jitter.

[0155] In one embodiment, the limited buffer time includes a first limited buffer time and a second limited buffer time, and the first limited buffer time is greater than the second limited buffer time.

[0156] Optionally, step S205 may include the following steps S71 to S74:

[0157] S71. Obtain the original transmission speed of the above-mentioned audio and video real-time communication architecture.

[0158] It should be noted that the original transmission speed in the audio real-time communication architecture can be predefined, or the original transmission speed can be determined based on the network jitter in the historical time period between the encoding device and the decoding device, the delay jitter of the video encoding frame, etc.

[0159] S72. When the playback duration is greater than the first limited buffer duration, multiply the original transmission speed by a first value to determine the transmission speed of the video coding sequence in the external media buffer; the first value is a number greater than 1.

[0160] Specifically, when the playback duration is greater than the first limited buffer duration, it indicates that the number of video encoding frames stored in the external media buffer is relatively large. Therefore, the computer device can multiply the product of the above-mentioned original transmission speed and the first value as the transmission speed of the video encoding sequence in the above-mentioned external media buffer, so as to accelerate the transmission of the video encoding sequence and the audio encoding sequence to the native media buffer and reduce the playback delay of the audio and video data.

[0161] S73: When the playback duration is less than or equal to the first limited buffering duration and greater than the second limited buffering duration, the original transmission speed is determined as the transmission speed of the video encoding sequence in the external media buffer.

[0162] Specifically, when the above-mentioned playback time is less than or equal to the above-mentioned first limit buffer time and greater than the above-mentioned second limit buffer time, it indicates that the number of video encoding frames stored in the external media buffer is moderate, and the above-mentioned original transmission speed is determined as the transmission speed of the video encoding sequence in the above-mentioned external media buffer.

[0163] S74. When the playback duration is less than or equal to the second limited buffer duration, the product of the original transmission speed and a second value is determined as the transmission speed of the video coding sequence in the external media buffer; the second value is a positive number less than 1.

[0164] Specifically, when the above-mentioned playback time is less than or equal to the above-mentioned second limited buffer time, it indicates that the number of video encoding frames stored in the external media buffer is relatively small. The product of the above-mentioned original transmission speed and the second value is determined as the transmission speed of the video encoding sequence in the above-mentioned external media buffer, so as to achieve deceleration in transmitting the video encoding sequence and the audio encoding sequence to the native media buffer, which can avoid congestion in the transmission of a large number of video encoding frames and audio encoding frames and improve transmission efficiency.

[0165] For example, Figure 6As shown, the external media buffer includes a video buffer 63a and an audio buffer 64a. The video buffer 63a includes a video code sequence 65a, and the audio buffer includes an audio code sequence 66a. The native media buffer includes a historical video code sequence 61a and a historical audio code sequence 62a. The computer device can determine a first limited buffering duration based on the reception time interval between the corresponding reception times of each two adjacent video code frames in the historical video code sequence 61a and the amount of data corresponding to each video code frame, and determine a second limited buffering duration based on the reception time interval between the corresponding reception times of each two adjacent video code frames in the historical video code sequence 61a. When the video coding sequence 65a in the video buffer 63a and the audio coding sequence 66a in the audio buffer meet the synchronous playback conditions, the computer device can obtain the playback duration (i.e., the total playback duration) corresponding to all video coding frames in the video coding sequence 65a in the video buffer 63a. When the playback duration is less than or equal to the second limit buffer duration, the frame transmission speed can be reduced, that is, the video coding sequence 65a in the video buffer 63a and the audio coding sequence 66a in the audio buffer 64a are transmitted to the native media buffer at a reduced speed. When the playback duration is greater than the second limit buffer duration and less than or equal to the first limit buffer duration, the frame transmission speed can be normal, that is, the video coding sequence 65a in the video buffer 63a and the audio coding sequence 66a in the audio buffer 64a are transmitted to the native media buffer at the original transmission speed. When the playback duration is greater than the first limited buffer duration, the frame delivery can be accelerated, that is, the video coding sequence 65a in the video buffer 63a and the audio coding sequence 66a in the audio buffer 64a are accelerated to be transmitted to the native media buffer to achieve synchronous transmission of the video coding sequence 65a and the audio coding sequence 66a.

[0166] S206 : Transmit the video code sequence and the audio code sequence in the external media buffer to the native media buffer in the audio and video real-time communication architecture through the native conversion interface at the transmission speed.

[0167] In this application, the computer device can transmit the video encoding sequence and audio encoding sequence in the external media buffer to the native media buffer in the audio and video real-time communication architecture in sequence according to the transmission speed and playback order through the native conversion interface.

[0168] It should be noted that this application can be applied to the H5 WebRTC scenario of browser live broadcast scenarios and social applications. This application can improve the anti-network jitter capability of H5 WebRTC and improve the smoothness of playback. The weak network packet loss test results of this application show that 1. The anti-weak network packet loss capability of this application in the H5 WebRTC scenario is improved from 5% to more than 20%; 2. For the jamming caused by the excessive size of the corresponding I frame, after using this application, it is measured that there is no jamming for a long time. Figure 7 As shown in the figure, the present application (H5 WebRTC superimposed on the anti-jitter method based on the native conversion interface and external media buffer) is compared with the native H5 WebRTC. Under 20% packet loss, the native H5 WebRTC playback is severely stuck, while the present application plays smoothly with almost no stuck. Figure 7 In the example, the present application and the native H5 WebRTC are both playing the same video data. At the same playback time, the audio and video real-time communication architecture of the present application is playing video screen 71a, and the native H5 WebRTC is playing video screen 72a. Video screen 71a includes characters 73a and 74a, as well as the words spoken by character 74a: "Xiaohong, I'm here." Video screen 72a includes character 73a, as well as the words spoken by character 74a: "Xiaohong, I'm here." Comparing video screen 71a and video screen 72a, it can be seen that character 74a does not appear in video screen 72a, but the words spoken appear first, that is, the audio and video in video screen 72a are out of sync. That is, the audio and video in the video screen played by the audio and video real-time communication architecture in the present application are synchronized, while the audio and video in the native H5 WebRTC are out of sync.

[0169] The present application includes at least the following beneficial effects: (1) When a computer device receives a video coding sequence and an audio coding sequence, the video coding sequence and the audio coding sequence are stored in an external media buffer through a native conversion interface in an audio and video real-time communication architecture. When the video coding sequence and the audio coding sequence in the external media buffer meet the synchronous playback conditions, the video coding sequence and the audio coding sequence in the external media buffer are adaptively sent to the native media buffer in the audio and video real-time communication architecture according to the limited buffering time of the external media buffer, thereby achieving synchronous playback of audio data and video data, avoiding fast forwarding, jumping, or even audio and video synchronization of the video screen, and improving the playback quality of audio and video data. (2) Since the limited buffering time of the external media buffer is determined based on the reception time of the historical video coding sequence, the reception time of the historical video coding sequence can reflect the network jitter to a certain extent, that is, the limited buffering time is dynamically determined based on the network jitter; therefore, transmitting the video coding sequence and the audio coding sequence in the external media buffer according to the limited buffering time of the external media buffer is conducive to improving the anti-network jitter capability of audio and video data. (3) By interacting with the external media buffer (storage, reading) through the native conversion interface in the audio and video real-time communication architecture, there is no need to modify the native audio and video real-time communication architecture or customize a private audio and video real-time communication architecture, which reduces the technical requirements and costs of audio and video communication and is conducive to the application, promotion and popularization of the audio and video real-time communication architecture.

[0170] See Figure 8 , is a structural diagram of a video data processing device provided in an embodiment of the present application. Figure 8 As shown, the video data processing device may include:

[0171] The storage module 811 is configured to, upon receiving the video code sequence and the audio code sequence sent by the framing module in the real-time audio and video communication architecture, store the video code sequence and the audio code sequence in the external media buffer of the real-time audio and video communication architecture through the native conversion interface in the real-time audio and video communication architecture;

[0172] A determination module 812 is configured to determine a limited buffering duration of the external media buffer for the video code sequence based on a reception time of a historical video code sequence, wherein the historical video code sequence is a video code sequence whose reception time is before the reception time of the video code sequence;

[0173] Verification module 813, configured to verify the synchronous playback conditions of the video code sequence and the audio code sequence in the external media buffer according to the playback times corresponding to the video code sequence and the audio code sequence in the external media buffer, and obtain a verification result;

[0174] The transmission module 814 is used to transmit the video coding sequence and audio coding sequence in the external media buffer to the native media buffer in the above-mentioned audio and video real-time communication architecture through the above-mentioned native conversion interface according to the above-mentioned limited buffering time when the above-mentioned verification result indicates that the video coding sequence and audio coding sequence in the above-mentioned external media buffer meet the synchronous playback conditions.

[0175] Optionally, the transmission module 814 is specifically configured to:

[0176] Get the playback duration of the current video encoding sequence in the external media buffer;

[0177] Determining a transmission speed of the video encoding sequence in the external media buffer according to the playback duration and the limited buffering duration;

[0178] The video encoding sequence and the audio encoding sequence in the external media buffer are transmitted to the native media buffer in the audio and video real-time communication architecture through the native conversion interface at the transmission speed.

[0179] Optionally, the above-mentioned limited buffer time includes a first limited buffer time and a second limited buffer time, and the above-mentioned first limited buffer time is greater than the above-mentioned second limited buffer time;

[0180] Optionally, the transmission module 814 is specifically configured to:

[0181] Get the original transmission speed of the above audio and video real-time communication architecture;

[0182] When the playback duration is greater than the first limited buffer duration, the transmission speed of the video code sequence in the external media buffer is determined by multiplying the original transmission speed by a first value, wherein the first value is a number greater than 1;

[0183] When the playback duration is less than or equal to the first limited buffering duration and greater than the second limited buffering duration, the original transmission speed is determined as the transmission speed of the video encoding sequence in the external media buffer;

[0184] When the playback duration is less than or equal to the second limited buffer duration, the product of the original transmission speed and the second value is determined as the transmission speed of the video encoding sequence in the external media buffer; the second value is a positive number less than 1.

[0185] Optionally, the reception time of the above historical video coding sequence includes the reception time of each video coding frame in the above historical video coding sequence;

[0186] Optionally, the determination module 812 is specifically configured to:

[0187] Determine the reception time interval between two adjacent video encoding frames according to the reception times corresponding to each two adjacent video encoding frames in the historical video encoding sequence;

[0188] The limited buffering duration of the external media buffer for the video coding sequence is determined according to the reception time interval between every two adjacent video coding frames in the historical video coding sequence.

[0189] Optionally, the above-mentioned limited buffer time includes a first limited buffer time and a second limited buffer time, and the above-mentioned first limited buffer time is greater than the above-mentioned second limited buffer time;

[0190] Optionally, the determination module 812 is specifically configured to:

[0191] performing weighted averaging processing on the reception time interval between every two adjacent video coding frames in the historical video coding sequence according to the data amount corresponding to the video coding frames in the historical video coding sequence to obtain a first average time interval;

[0192] Determining a first limited buffering duration of the external media buffer for the video coding sequence according to the first average time interval;

[0193] Averaging the received time intervals between every two adjacent video encoding frames in the historical video encoding sequence to obtain a second average time interval;

[0194] A second limited buffering duration of the external media buffer for the video coding sequence is determined according to the second average time interval.

[0195] Optionally, the verification module 813 is specifically configured to:

[0196] Determine, based on the playback time corresponding to the video code sequence in the external media buffer, a target video code frame whose playback time is the earliest playback time; and determine, based on the playback time corresponding to the audio code sequence in the external media buffer, a target audio code frame whose playback time is the earliest playback time.

[0197] According to the playback time corresponding to the target audio code frame and the playback time corresponding to the target video code frame, the synchronous playback condition verification is performed on the video code sequence and the audio code sequence in the external media buffer to obtain a verification result.

[0198] Optionally, the verification module 813 is specifically configured to:

[0199] Determine a playback time interval between a playback time corresponding to the target audio coding frame and a playback time corresponding to the target video coding frame;

[0200] When the playback time interval is less than the time interval threshold, determining that the video code sequence and the audio code sequence in the external media buffer satisfy a synchronization playback condition;

[0201] A verification result is generated to reflect that the video encoding sequence and the audio encoding sequence in the external media buffer satisfy a synchronous playback condition.

[0202] Optionally, the determination module 812 is specifically configured to:

[0203] When the limited buffering time corresponding to the external media buffer satisfies the update condition, the position of the sliding window in the historical video encoding sequence when the limited buffering time was last updated is obtained;

[0204] Starting from the above-mentioned stop position, the above-mentioned sliding window is moved in the above-mentioned historical video coding sequence according to the sliding step size;

[0205] The limited buffering time of the external media buffer for the video coding sequence is updated according to the reception time of the historical video coding frame currently contained in the sliding window.

[0206] Optionally, the determination module 812 is specifically configured to:

[0207] Obtaining historical log data of a device for receiving the video code sequence and the audio code sequence;

[0208] Determine the network stability of the above-mentioned devices based on the above-mentioned historical log data;

[0209] Based on the above network stability, an update time interval corresponding to the limited buffering time is generated;

[0210] Determine the update time of the current update limit buffer time based on the above update time interval and the update time of the last update limit buffer time;

[0211] When the time reaches the update time of this update limit buffering time, it is determined that the limit buffering time corresponding to the above external media buffer meets the update condition, and the stop position of the sliding window in the historical video encoding sequence when the limit buffering time was last updated is obtained.

[0212] Optionally, the storage module 811 is specifically configured to:

[0213] Obtain the architecture type to which the above audio and video real-time communication architecture belongs;

[0214] Determine a native conversion interface in the audio and video real-time communication architecture according to the architecture type to which the audio and video real-time communication architecture belongs;

[0215] The video encoding sequence and the audio encoding sequence are stored in the external media buffer of the audio and video real-time communication architecture through the determined native conversion interface.

[0216] Optionally, the storage module 811 is specifically configured to:

[0217] When the architecture type of the above-mentioned audio and video real-time communication architecture belongs to the open source software development kit type, the frame conversion interface in the audio and video real-time communication architecture of the open source software development kit type is determined as the native conversion interface in the above-mentioned audio and video real-time communication architecture;

[0218] When the architecture type of the above-mentioned audio and video real-time communication architecture is a web page type, the encoded conversion interface in the web page interface of the web page type audio and video real-time communication architecture is determined as the native conversion interface in the above-mentioned audio and video real-time communication architecture.

[0219] The present application includes at least the following beneficial effects: (1) When a computer device receives a video coding sequence and an audio coding sequence, the video coding sequence and the audio coding sequence are stored in an external media buffer through a native conversion interface in an audio and video real-time communication architecture. When the video coding sequence and the audio coding sequence in the external media buffer meet the synchronous playback conditions, the video coding sequence and the audio coding sequence in the external media buffer are adaptively sent to the native media buffer in the audio and video real-time communication architecture according to the limited buffering time of the external media buffer, thereby achieving synchronous playback of audio data and video data, avoiding fast forwarding, jumping, or even audio and video synchronization of the video screen, and improving the playback quality of audio and video data. (2) Since the limited buffering time of the external media buffer is determined based on the reception time of the historical video coding sequence, the reception time of the historical video coding sequence can reflect the network jitter to a certain extent, that is, the limited buffering time is dynamically determined based on the network jitter; therefore, transmitting the video coding sequence and the audio coding sequence in the external media buffer according to the limited buffering time of the external media buffer is conducive to improving the anti-network jitter capability of audio and video data. (3) By interacting with the external media buffer (storage, reading) through the native conversion interface in the audio and video real-time communication architecture, there is no need to modify the native audio and video real-time communication architecture or customize a private audio and video real-time communication architecture, which reduces the technical requirements and costs of audio and video communication and is conducive to the application, promotion and popularization of the audio and video real-time communication architecture.

[0220] See Figure 9 , is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 9 As shown, the above-mentioned computer device 1000 may refer to a terminal or a server, including: a processor 1001, a network interface 1004 and a memory 1005. In addition, the above-mentioned computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. In some embodiments, the user interface 1003 may include a display screen (DiSPlay), a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatileMeMory), such as at least one disk storage. The memory 1005 may optionally also be at least one storage device away from the aforementioned processor 1001. As Figure 9 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a computer program.

[0221] exist Figure 9 In the computer device 1000 shown, the network interface 1004 can provide network communication functions; the user interface 1003 is mainly used to provide an input interface; and the processor 1001 can be used to call the computer program stored in the memory 1005 to implement:

[0222] When receiving the video code sequence and the audio code sequence sent by the framing module in the real-time audio and video communication architecture, the video code sequence and the audio code sequence are stored in the external media buffer of the real-time audio and video communication architecture through the native conversion interface in the real-time audio and video communication architecture;

[0223] Determining a limited buffering duration of the external media buffer for the video code sequence according to a reception time of a historical video code sequence, wherein the historical video code sequence is a video code sequence whose reception time is before the reception time of the video code sequence;

[0224] Verifying the synchronous playback conditions of the video code sequence and the audio code sequence in the external media buffer according to the playback times corresponding to the video code sequence and the audio code sequence in the external media buffer, and obtaining a verification result;

[0225] When the above verification result indicates that the video encoding sequence and audio encoding sequence in the above external media buffer meet the synchronous playback conditions, the video encoding sequence and audio encoding sequence in the above external media buffer are transmitted to the native media buffer in the above audio and video real-time communication architecture through the above native conversion interface according to the above limited buffering time.

[0226] Optionally, the processor 1001 may be configured to call a computer program stored in the memory 1005 to implement:

[0227] Get the playback duration of the current video encoding sequence in the external media buffer;

[0228] Determining a transmission speed of the video encoding sequence in the external media buffer according to the playback duration and the limited buffering duration;

[0229] The video encoding sequence and the audio encoding sequence in the external media buffer are transmitted to the native media buffer in the audio and video real-time communication architecture through the native conversion interface at the transmission speed.

[0230] Optionally, the above-mentioned limited buffer time includes a first limited buffer time and a second limited buffer time, and the above-mentioned first limited buffer time is greater than the above-mentioned second limited buffer time;

[0231] Optionally, the processor 1001 may be configured to call a computer program stored in the memory 1005 to determine a transmission speed of the video encoding sequence in the external media buffer according to the playback duration and the limited buffering duration, including:

[0232] Get the original transmission speed of the above audio and video real-time communication architecture;

[0233] When the playback duration is greater than the first limited buffer duration, the transmission speed of the video code sequence in the external media buffer is determined by multiplying the original transmission speed by a first value, wherein the first value is a number greater than 1;

[0234] When the playback duration is less than or equal to the first limited buffering duration and greater than the second limited buffering duration, the original transmission speed is determined as the transmission speed of the video encoding sequence in the external media buffer;

[0235] When the playback duration is less than or equal to the second limited buffer duration, the product of the original transmission speed and the second value is determined as the transmission speed of the video encoding sequence in the external media buffer; the second value is a positive number less than 1.

[0236] Optionally, the reception time of the above historical video coding sequence includes the reception time of each video coding frame in the above historical video coding sequence;

[0237] Optionally, the processor 1001 may be configured to call a computer program stored in the memory 1005 to determine a limited buffering duration of the external media buffer for the video coding sequence based on a reception time of the historical video coding sequence, including:

[0238] Determine the reception time interval between two adjacent video encoding frames according to the reception times corresponding to each two adjacent video encoding frames in the historical video encoding sequence;

[0239] The limited buffering duration of the external media buffer for the video coding sequence is determined according to the reception time interval between every two adjacent video coding frames in the historical video coding sequence.

[0240] Optionally, the above-mentioned limited buffer time includes a first limited buffer time and a second limited buffer time, and the above-mentioned first limited buffer time is greater than the above-mentioned second limited buffer time;

[0241] Optionally, the processor 1001 may be configured to call a computer program stored in the memory 1005 to determine a limited buffering duration of the external media buffer for the video coding sequence based on a reception time interval between every two adjacent video coding frames in the historical video coding sequence, including:

[0242] performing weighted averaging processing on the reception time interval between every two adjacent video coding frames in the historical video coding sequence according to the data amount corresponding to the video coding frames in the historical video coding sequence to obtain a first average time interval;

[0243] Determining a first limited buffering duration of the external media buffer for the video coding sequence according to the first average time interval;

[0244] Averaging the received time intervals between every two adjacent video encoding frames in the historical video encoding sequence to obtain a second average time interval;

[0245] A second limited buffering duration of the external media buffer for the video coding sequence is determined according to the second average time interval.

[0246] Optionally, the processor 1001 may be configured to call a computer program stored in the memory 1005 to implement synchronous playback condition verification of the video code sequence and the audio code sequence in the external media buffer according to the playback times corresponding to the video code sequence and the audio code sequence in the external media buffer, respectively, to obtain a verification result, including:

[0247] Determine, based on the playback time corresponding to the video code sequence in the external media buffer, a target video code frame whose playback time is the earliest playback time; and determine, based on the playback time corresponding to the audio code sequence in the external media buffer, a target audio code frame whose playback time is the earliest playback time.

[0248] According to the playback time corresponding to the target audio code frame and the playback time corresponding to the target video code frame, the synchronous playback condition verification is performed on the video code sequence and the audio code sequence in the external media buffer to obtain a verification result.

[0249] Optionally, the processor 1001 may be configured to call a computer program stored in the memory 1005 to implement synchronous playback condition verification of the video code sequence and the audio code sequence in the external media buffer according to the playback time corresponding to the target audio code frame and the playback time corresponding to the target video code frame, and obtain a verification result, including:

[0250] Determine a playback time interval between a playback time corresponding to the target audio coding frame and a playback time corresponding to the target video coding frame;

[0251] When the playback time interval is less than the time interval threshold, determining that the video code sequence and the audio code sequence in the external media buffer satisfy a synchronization playback condition;

[0252] A verification result is generated to reflect that the video encoding sequence and the audio encoding sequence in the external media buffer satisfy a synchronous playback condition.

[0253] Optionally, the processor 1001 may be configured to call a computer program stored in the memory 1005 to determine a limited buffering duration of the external media buffer for the video coding sequence based on a reception time of the historical video coding sequence, including:

[0254] When the limited buffering time corresponding to the external media buffer satisfies the update condition, the position of the sliding window in the historical video encoding sequence when the limited buffering time was last updated is obtained;

[0255] Starting from the above-mentioned stop position, the above-mentioned sliding window is moved in the above-mentioned historical video coding sequence according to the sliding step size;

[0256] The limited buffering time of the external media buffer for the video coding sequence is updated according to the reception time of the historical video coding frame currently contained in the sliding window.

[0257] Optionally, the processor 1001 may be configured to call a computer program stored in the memory 1005 to implement, when the limited buffering time corresponding to the external media buffer satisfies an update condition, obtaining a stop position of the sliding window in the historical video coding sequence when the limited buffering time was last updated, including:

[0258] Obtaining historical log data of a device for receiving the video code sequence and the audio code sequence;

[0259] Determine the network stability of the above-mentioned devices based on the above-mentioned historical log data;

[0260] Based on the above network stability, an update time interval corresponding to the limited buffering time is generated;

[0261] Determine the update time of the current update limit buffer time based on the above update time interval and the update time of the last update limit buffer time;

[0262] When the time reaches the update time of this update limit buffering time, it is determined that the limit buffering time corresponding to the above external media buffer meets the update condition, and the stop position of the sliding window in the historical video encoding sequence when the limit buffering time was last updated is obtained.

[0263] Optionally, the processor 1001 may be configured to call a computer program stored in the memory 1005 to implement storing the video encoding sequence and the audio encoding sequence in an external media buffer of the real-time audio and video communication architecture through a native conversion interface in the real-time audio and video communication architecture, including:

[0264] Obtain the architecture type to which the above audio and video real-time communication architecture belongs;

[0265] Determine a native conversion interface in the audio and video real-time communication architecture according to the architecture type to which the audio and video real-time communication architecture belongs;

[0266] The video encoding sequence and the audio encoding sequence are stored in the external media buffer of the audio and video real-time communication architecture through the determined native conversion interface.

[0267] Optionally, the processor 1001 may be configured to call a computer program stored in the memory 1005 to determine a native conversion interface in the real-time audio and video communication architecture according to the architecture type to which the real-time audio and video communication architecture belongs, including:

[0268] When the architecture type of the above-mentioned audio and video real-time communication architecture belongs to the open source software development kit type, the frame conversion interface in the audio and video real-time communication architecture of the open source software development kit type is determined as the native conversion interface in the above-mentioned audio and video real-time communication architecture;

[0269] When the architecture type of the above-mentioned audio and video real-time communication architecture is a web page type, the encoded conversion interface in the web page interface of the web page type audio and video real-time communication architecture is determined as the native conversion interface in the above-mentioned audio and video real-time communication architecture.

[0270] The present application includes at least the following beneficial effects: (1) When a computer device receives a video coding sequence and an audio coding sequence, the video coding sequence and the audio coding sequence are stored in an external media buffer through a native conversion interface in an audio and video real-time communication architecture. When the video coding sequence and the audio coding sequence in the external media buffer meet the synchronous playback conditions, the video coding sequence and the audio coding sequence in the external media buffer are adaptively sent to the native media buffer in the audio and video real-time communication architecture according to the limited buffering time of the external media buffer, thereby achieving synchronous playback of audio data and video data, avoiding fast forwarding, jumping, or even audio and video synchronization of the video screen, and improving the playback quality of audio and video data. (2) Since the limited buffering time of the external media buffer is determined based on the reception time of the historical video coding sequence, the reception time of the historical video coding sequence can reflect the network jitter to a certain extent, that is, the limited buffering time is dynamically determined based on the network jitter; therefore, transmitting the video coding sequence and the audio coding sequence in the external media buffer according to the limited buffering time of the external media buffer is conducive to improving the anti-network jitter capability of audio and video data. (3) By interacting with the external media buffer (storage, reading) through the native conversion interface in the audio and video real-time communication architecture, there is no need to modify the native audio and video real-time communication architecture or customize a private audio and video real-time communication architecture, which reduces the technical requirements and costs of audio and video communication and is conducive to the application, promotion and popularization of the audio and video real-time communication architecture.

[0271] It should be understood that the computer device described in the embodiments of the present application can perform the description of the video data processing method described in the corresponding embodiments above, and can also perform the description of the video data processing device described in the corresponding embodiments above, and the description thereof will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated here.

[0272] The collection and processing of the relevant data in this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in practice, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0273] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the above-mentioned computer-readable storage medium stores a computer program executed by the video data processing device mentioned above, and the above-mentioned computer program includes program instructions. When the above-mentioned processor executes the above-mentioned program instructions, it can execute the description of the above-mentioned video data processing method in the corresponding embodiment above. Therefore, it will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0274] As an example, the above program instructions may be deployed on a computer device for execution, or deployed on at least two computer devices at one location for execution, or executed on at least two computer devices distributed at at least two locations and interconnected via a communication network. The at least two computer devices distributed at at least two locations and interconnected via a communication network may constitute a blockchain network.

[0275] The computer-readable storage medium may be the video data processing device provided in any of the aforementioned embodiments or the central storage unit of the computer device, such as a hard disk or a memory card of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SMART Media card, SMC), a secure digital (SD) card, a flash memory card, etc. equipped on the computer device. Furthermore, the computer-readable storage medium may also include both the central storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0276] The terms "first," "second," and the like in the description, claims, and drawings of the embodiments of the present application are used to distinguish between contents in different media, rather than to describe a specific order. In addition, the terms "including" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other steps and units inherent to these processes, methods, apparatuses, products, or devices.

[0277] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0278] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0279] The methods and related devices provided by the embodiments of the present application are described with reference to the method flow charts and / or structural diagrams provided by the embodiments of the present application. Specifically, each process and / or block in the method flow charts and / or structural diagrams, as well as the combination of processes and / or blocks in the flow charts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable network-connected device to generate a machine, so that the instructions executed by the processor of the computer or other programmable network-connected device generate instructions for implementing the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable network-connected device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the function specified in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable network connected device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 The flow or flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0280] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A video data processing method, characterized in that: include: When receiving the video code sequence and the audio code sequence sent by the framing module in the real-time audio and video communication architecture, storing the video code sequence and the audio code sequence in the external media buffer of the real-time audio and video communication architecture through the native conversion interface in the real-time audio and video communication architecture; Determining a limited buffering duration of the external media buffer for the video coding sequence according to a reception time of a historical video coding sequence, wherein the historical video coding sequence is a video coding sequence whose reception time is before the reception time of the video coding sequence; Verifying the synchronous playback conditions of the video code sequence and the audio code sequence in the external media buffer according to the playback times respectively corresponding to the video code sequence and the audio code sequence in the external media buffer to obtain a verification result; When the verification result indicates that the video encoding sequence and the audio encoding sequence in the external media buffer meet the synchronous playback conditions, the video encoding sequence and the audio encoding sequence in the external media buffer are transmitted to the native media buffer in the audio and video real-time communication architecture through the native conversion interface according to the limited buffering time.

2. The method according to claim 1, wherein The method of transmitting the video code sequence and the audio code sequence in the external media buffer to the native media buffer in the audio and video real-time communication architecture through the native conversion interface according to the limited buffering time includes: Obtaining the playback duration corresponding to the current video encoding sequence in the external media buffer; Determining a transmission speed of the video encoding sequence in the external media buffer according to the playback duration and the limited buffering duration; The video encoding sequence and the audio encoding sequence in the external media buffer are transmitted to the native media buffer in the audio and video real-time communication architecture through the native conversion interface at the transmission speed.

3. The method according to claim 2, wherein The limited buffer time includes a first limited buffer time and a second limited buffer time, and the first limited buffer time is greater than the second limited buffer time; The determining, according to the playback duration and the limited buffering duration, a transmission speed of the video encoding sequence in the external media buffer includes: Obtaining the original transmission speed of the audio and video real-time communication architecture; When the playback duration is greater than the first limited buffering duration, the product of the original transmission speed and a first value is multiplied to determine the transmission speed of the video coding sequence in the external media buffer; the first value is a number greater than 1; When the playback duration is less than or equal to the first limited buffer duration and greater than the second limited buffer duration, determining the original transmission speed as the transmission speed of the video coding sequence in the external media buffer; When the playback duration is less than or equal to the second limited buffer duration, the product of the original transmission speed and a second value is determined as the transmission speed of the video encoding sequence in the external media buffer; the second value is a positive number less than 1.

4. The method according to claim 1, wherein The reception time of the historical video coding sequence includes the reception time of each video coding frame in the historical video coding sequence; The determining, based on the reception time of the historical video coding sequence, the limited buffering time of the external media buffer for the video coding sequence includes: Determining a reception time interval between two adjacent video encoding frames according to the reception times corresponding to each two adjacent video encoding frames in the historical video encoding sequence; The limited buffering duration of the external media buffer for the video coding sequence is determined according to the reception time interval between every two adjacent video coding frames in the historical video coding sequence.

5. The method according to claim 4, wherein The limited buffer time includes a first limited buffer time and a second limited buffer time, and the first limited buffer time is greater than the second limited buffer time; The determining, based on the reception time interval between each two adjacent video coding frames in the historical video coding sequence, a limited buffering duration of the external media buffer for the video coding sequence includes: performing weighted averaging processing on the reception time interval between every two adjacent video coding frames in the historical video coding sequence according to the data amount corresponding to the video coding frames in the historical video coding sequence to obtain a first average time interval; Determining, according to the first average time interval, a first limited buffering duration of the external media buffer for the video coding sequence; Averaging the received time intervals between every two adjacent video coding frames in the historical video coding sequence to obtain a second average time interval; A second limited buffering duration of the external media buffer for the video coding sequence is determined according to the second average time interval.

6. The method according to claim 1, wherein The step of verifying the synchronous playback conditions of the video code sequence and the audio code sequence in the external media buffer according to the playback times respectively corresponding to the video code sequence and the audio code sequence in the external media buffer to obtain a verification result includes: Determining a target video encoding frame whose playback time is the earliest playback time according to the playback time corresponding to the video encoding sequence in the external media buffer; Determining, based on the playback time corresponding to the audio code sequence in the external media buffer, a target audio code frame whose playback time is the earliest playback time; According to the playback time corresponding to the target audio coding frame and the playback time corresponding to the target video coding frame, a synchronous playback condition verification is performed on the video coding sequence and the audio coding sequence in the external media buffer to obtain a verification result.

7. The method according to claim 6, wherein The performing synchronous playback condition verification on the video code sequence and the audio code sequence in the external media buffer according to the playback time corresponding to the target audio code frame and the playback time corresponding to the target video code frame to obtain a verification result includes: Determine a playback time interval between a playback time corresponding to the target audio coding frame and a playback time corresponding to the target video coding frame; When the playback time interval is less than the time interval threshold, determining that the video encoding sequence and the audio encoding sequence in the external media buffer satisfy a synchronous playback condition; A verification result is generated to reflect that the video encoding sequence and the audio encoding sequence in the external media buffer satisfy a synchronous playback condition.

8. The method according to claim 1, wherein The determining, based on the reception time of the historical video coding sequence, the limited buffering time of the external media buffer for the video coding sequence includes: When the limited buffering time length corresponding to the external media buffer area satisfies the update condition, obtaining the stop position of the sliding window in the historical video coding sequence when the limited buffering time length was last updated; Starting from the stop position, the sliding window is moved in the historical video coding sequence according to the sliding step size; The limited buffering duration of the external media buffer for the video coding sequence is updated according to the reception time of the historical video coding frame currently contained in the sliding window.

9. The method according to claim 8, wherein When the limited buffering time corresponding to the external media buffer satisfies the update condition, obtaining the stay position of the sliding window in the historical video coding sequence when the limited buffering time was last updated includes: Obtaining historical log data of a device for receiving the video code sequence and the audio code sequence; Determining the network stability of the device based on the historical log data; Generating an update time interval corresponding to a limited buffering time length according to the network stability; Determine the update time of the current update limit buffer time length according to the update time interval and the update time of the last update limit buffer time length; When the time reaches the update time of the current limited buffering time, it is determined that the limited buffering time corresponding to the external media buffer meets the update condition, and the stop position of the sliding window in the historical video encoding sequence when the limited buffering time was last updated is obtained.

10. The method according to claim 1, wherein The step of storing the video encoding sequence and the audio encoding sequence in an external media buffer of the audio and video real-time communication architecture through a native conversion interface in the audio and video real-time communication architecture comprises: Obtaining the architecture type to which the audio and video real-time communication architecture belongs; Determining a native conversion interface in the audio and video real-time communication architecture according to the architecture type to which the audio and video real-time communication architecture belongs; The video encoding sequence and the audio encoding sequence are stored in an external media buffer of the audio and video real-time communication architecture through the determined native conversion interface.

11. The method according to claim 10, wherein The determining, according to the architecture type to which the audio and video real-time communication architecture belongs, a native conversion interface in the audio and video real-time communication architecture includes: When the architecture type of the audio and video real-time communication architecture belongs to an open source software development kit type, determining the frame conversion interface in the audio and video real-time communication architecture of the open source software development kit type as the native conversion interface in the audio and video real-time communication architecture; When the architecture type of the audio and video real-time communication architecture is a web page type, the encoded conversion interface in the web page interface of the audio and video real-time communication architecture of the web page type is determined as the native conversion interface in the audio and video real-time communication architecture.

12. A video data processing device, characterized in that: include: A storage module is configured to, upon receiving a video code sequence and an audio code sequence sent by a framing module in the real-time audio and video communication architecture, store the video code sequence and the audio code sequence in an external media buffer of the real-time audio and video communication architecture through a native conversion interface in the real-time audio and video communication architecture; a determination module, configured to determine a limited buffering duration of the external media buffer for a video coding sequence based on a reception time of a historical video coding sequence, wherein the historical video coding sequence is a video coding sequence whose reception time is before the reception time of the video coding sequence; a verification module, configured to verify the synchronous playback conditions of the video code sequence and the audio code sequence in the external media buffer according to the playback times respectively corresponding to the video code sequence and the audio code sequence in the external media buffer, and obtain a verification result; A transmission module is used to transmit the video coding sequence and audio coding sequence in the external media buffer to the native media buffer in the audio and video real-time communication architecture through the native conversion interface according to the limited buffering time when the verification result indicates that the video coding sequence and audio coding sequence in the external media buffer meet the synchronous playback conditions.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images