Real-time media internal video and audio coordination method based on Android platform

Abstract

According to the invention, by setting the first-stage cache region, delay jitter caused to the video and audio packets when the network environment is not good is relieved, the video and audio packets are disordered to reach the receiving end in a UDP transmission mode adopted for ensuring the real-time performance of the video and audio streams, and in the first-stage cache region, the sequence numbers of the video and audio packets are compared, unpacking, sorting and framing are performed on out-of-order packets caused by unreliable transmission, then when a packet loss phenomenon occurs, a packet loss type is judged, a flow and congestion control strategy is adopted for packet loss caused by network congestion, and a network environment and the packet loss phenomenon are detected through SR packets and RR packets in RTCP packets; the receiving end calculates the packet loss rate and the loopback time through the SR packet and feeds back the calculated information to the sending end through the RR packet, and the sending end adjusts the sending rate and reduces the packet loss rate according to the feedback information and the improved throughput model, so that the video and audio in the real-time media of the Android platform are synchronized.

Description

technical field [0001] The invention relates to a method for coordinating video and audio in real-time media, in particular to a method for coordinating video and audio in real-time media based on an Android platform, and belongs to the technical field of real-time video and audio coordination. Background technique [0002] With the popularization of Android mobile terminals and the rapid development of the mobile Internet, people's requirements for the quality of video and audio mobile communication are increasing day by day. The internal synchronization and coordination effect is good. However, for a long time, the research and development of video and audio synchronization in real-time media has mainly focused on professional video equipment and PC terminals, which have the disadvantages of high equipment cost and poor convenience. However, the research and development of mobile video and audio synchronization in the existing technology still stays on the video player an...

AI Technical Summary

Problems solved by technology

[0010] First, for a long time, the research and development of video and audio synchronization in real-time media has mainly focused on professional video equipment and PCs, which have the disadvantages of high equipment cost and poor convenience. It still stays on the video player and media transmission framework, and does not solve the real-time video and audio out-of-sync problems caused by the limitations of Android device performance, mobile network interference, insufficient and unstable bandwidth, etc.;

[0011] Second, the traditional method of synchronously controlling playback based on audio time stamps mainly takes the audio stream as the mainstream, and the video stream as the secondary stream. Every time a video frame arrives, it is compared with the time stamp of the audio frame being played or to be played until it meets the ±80ms limit. This method is likely to play the next video frame because the newly arrived video frame also conforms to the synchronization range. When the media packet is lost due to network congestion, the video frame is synchronized with the timestamp of the audio frame. The number of comparisons will increase significantly, the system consumes a lot, and there is no feedback adjustment mechanism to reduce packet loss, which seriously affects the synchronization between video and audio. These disadvantages of the existing technology have a fatal negative impact on the video and audio synchronization of the Android platform. ;

[0012] Thirdly, the existing technology only sets buffers at the receiving end to adjust the delay jitter and output rate for real-time media synchronization processing. There is no corresponding processing mechanism for the media out-of-synch caused by packet loss. If When the network is congested, the original sending rate is still used, which will lead to more serious network load and increased packet loss rate, which is not conducive to subsequent synchronization processing;

[0013] Fourth, the transmission of real-time video and audio adopts the connectionless UDP method. Due to the different network topology, the adjacent data packets will choose different links, resulting in a certain time range. The data packets sent later arrive at the receiving end. This best-effort delivery has the disadvantage of not guaranteeing that the data packets can arrive at the receiving end in order. The sending end cannot even know whether the sent data packets are delivered correctly and completely, so the out-of-order delivery must first be sent. RTP data packets are sorted, and audio packets are encapsulated into one RTP packet by using the NALU of an audio frame. However, the video packet may be fragmented because the NALU length of one frame is too large, that is, the NALU of one frame is encapsulated. into multiple RTP packets, and the receiving end needs to receive a complete frame before it can be decoded normally. If the received data packets are directly decoded and played, it will cause blurred screens, image flickering, or cause no playback at all. ;

[0014] Fifth, the reason for the out-of-sync in the real-time media stream is mainly due to the large network load, which leads to network congestion, resulting in the loss of data packets, data interference errors or large delay jitter. The method of setting buffers at the end to alleviate the problem of delay jitter, but when network congestion occurs, transmitting too many data packets will cause a sharp drop in network throughput. If the sender continues to send a large amount of data, it will only make the end-to-end data packets The delay increases and the packet loss rate rises rapidly. Since the transmission of real-time streaming media mostly adopts the UDP method, it is only a best-effort delivery of data packets, and there is no confirmation retransmission mechanism, resulting in unrecoverable data packet loss. Currently, video and audio synchronization, Most of the receiving end handles packet loss by repeatedly playing the same frame. In the case of a small amount of packet loss, the user will not notice the impact on synchronization during the viewing process, but it is still handled in this way in the case of a large amount of packet loss. , which will cause the video and audio playback to have an obvious sense of stuttering, which will bring users a poor viewing experience.

Images