Synchronous playback method, system and device for audio and video, and storage medium

Abstract

Disclosed in the present disclosure are a synchronous playback method, system and device for audio and video, and a storage medium. The method comprises: a first terminal performing separation and decoding on an audio and video stream to be played, so as to obtain an audio stream and a video stream; by means of a real-time audio and video interactive connection established with a second terminal, acquiring a network delay between the first terminal and the second terminal; performing delayed rendering and playback on the video stream according to a preset first delay time; and sending the audio stream, the first delay time and the network delay to the second terminal, so that the second terminal performs delayed rendering and playback on the audio stream on the basis of the first delay time and the network delay.

Description

Method, system, device and storage medium for synchronous playback of audio and video

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese Patent Application No. 202311740334.8 filed in China on December 18, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the technical field of audio and video processing, and in particular to a method, system, device and storage medium for synchronous playback of audio and video. Background Art

[0004] With the continuous development of science and technology, urban large screens have become an important part of urban construction. Currently, cities often set up ultra-high-definition public large screens in downtown areas, promoting the integration and innovation of ultra-high-definition video in multiple fields.

[0005] Because large screens are located in downtown areas, they are often silent to avoid noise pollution from outdoor audio playback, resulting in a poor experience for the audience. Therefore, how to immerse the audience in the 3D audio experience on large screens has become an urgent problem to be solved. Summary of the Invention

[0006] The purpose of the embodiments of the present disclosure is to provide a method, system, device and storage medium for synchronous playback of audio and video, which can effectively realize the synchronous playback of audio and video, allowing on-site users to listen to the corresponding audio content while watching large-screen video content, and there is no time delay between the video being watched and the audio content being listened to.

[0007] The present disclosure provides a method for synchronously playing audio and video, which is applied to a first terminal. The method includes:

[0008] Separate and decode the audio and video streams to be played to obtain audio stream and video stream;

[0009] Acquire a network delay between the first terminal and the second terminal through a real-time audio and video interactive connection established with the second terminal;

[0010] Performing delayed rendering and playback of the video stream according to a preset first delay time;

[0011] The audio stream, the first delay time, and the network delay are sent to the second terminal, so that the second terminal performs delayed rendering and playback of the audio stream based on the first delay time and the network delay.

[0012] As an improvement to the above solution, the establishment of the audio and video interactive connection includes:

[0013] Creating a real-time audio and video interactive room and generating an audio and video interactive link so that the second terminal can join the audio and video interactive room through the audio and video interactive link;

[0014] When the second terminal joins the audio and video interactive room, a real-time audio and video interactive connection is established with the second terminal through a real-time audio and video network service.

[0015] As an improvement to the above solution, the step of creating a real-time audio and video interactive room and generating an audio and video interactive link so that a second terminal can join the audio and video interactive room through the audio and video interactive link includes:

[0016] Create a real-time audio and video interactive room and generate an audio and video interactive link;

[0017] Generating an identification code including the audio and video interactive link according to the audio and video interactive link;

[0018] The identification code is displayed on the display screen of the first terminal, so that the second terminal can join the audio and video interactive room by scanning the identification code.

[0019] As an improvement to the above solution, obtaining the network delay between the first terminal and the second terminal includes:

[0020] In response to the delay request sent by the second terminal, calculating the round-trip delay between the first terminal and the second terminal using a standard RTT algorithm;

[0021] Calculate a network delay between the first terminal and the second terminal based on the round-trip delay.

[0022] As an improvement to the above solution, the network delay is the sum of a first network delay and a second network delay; wherein, the first network delay is the network delay between the first terminal and the real-time audio and video network service, and the second network delay is the network delay between the second terminal and the real-time audio and video network service.

[0023] As an improvement to the above solution, the second terminal can calculate a second delay time based on the first delay time and the network delay, and delay rendering and playing the audio stream according to the second delay time; wherein the second delay time is the difference between the first delay time and the network delay.

[0024] The present disclosure also provides a method for synchronously playing audio and video, which is applied to a second terminal. The method includes:

[0025] Establishing a real-time audio and video interactive connection with a first terminal; wherein the first terminal is capable of separating and decoding the audio and video stream to be played to obtain an audio stream and a video stream, and delaying the rendering and playing of the video stream according to a preset first delay time;

[0026] receiving the audio stream, the first delay time, and the network delay between the first terminal and the second terminal sent by the first terminal;

[0027] The audio stream is delayed rendered and played according to the first delay time and the network delay.

[0028] As an improvement to the above solution, performing delayed rendering and playing of the audio stream according to the first delay time and the network delay includes:

[0029] Calculate the difference between the first delay time and the network delay to obtain a second delay time;

[0030] The audio stream is rendered and played with delay according to the second delay time.

[0031] The present disclosure also provides a device for synchronously playing audio and video, which is applied to a first terminal. The device includes:

[0032] The audio and video stream decoding module is used to separate and decode the audio and video streams to be played to obtain audio streams and video streams;

[0033] a network delay calculation module, configured to obtain the network delay between the first terminal and the second terminal through a real-time audio and video interactive connection established with the second terminal;

[0034] A video stream playing module, configured to perform delayed rendering and playing of the video stream according to a preset first delay time;

[0035] A data sending module is used to send the audio stream, the first delay time and the network delay to the second terminal, so that the second terminal performs delayed rendering and playback of the audio stream based on the first delay time and the network delay.

[0036] The present disclosure also provides a device for synchronously playing audio and video, which is applied to a second terminal. The device includes:

[0037] a connection establishment module for establishing a real-time audio and video interactive connection with a first terminal; wherein the first terminal is capable of separating and decoding the audio and video streams to be played to obtain an audio stream and a video stream, and delaying the rendering and playback of the video stream according to a preset first delay time;

[0038] a data receiving module, configured to receive the audio stream, the first delay time, and the network delay between the first terminal and the second terminal sent by the first terminal;

[0039] An audio stream playing module is used to perform delayed rendering and playing of the audio stream according to the first delay time and the network delay.

[0040] The embodiment of the present disclosure also provides a system for synchronously playing audio and video, including a first terminal and a second terminal; wherein,

[0041] The first terminal is configured to separate and decode the audio and video streams to be played to obtain an audio stream and a video stream, obtain a network delay between the first terminal and the second terminal through a real-time audio and video interactive connection established with the second terminal, delay the playback of the video stream according to a preset first delay time, and send the audio stream, the first delay time, and the network delay to the second terminal;

[0042] The second terminal is configured to delay playing the audio stream based on the first delay time and the network delay.

[0043] An embodiment of the present disclosure also provides a synchronous playback device for audio and video, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the synchronous playback method for audio and video as described in any one of the above items.

[0044] An embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the method for synchronous playback of audio and video as described in any one of the above.

[0045] Compared with the prior art, the method, system, device and storage medium for synchronous playback of audio and video disclosed in the present disclosure establishes a real-time audio and video interactive connection between a first terminal and a second terminal, wherein the first terminal sends two types of data, namely, an audio stream and a time delay, to the second terminal, and the second terminal performs delayed rendering and playback of the audio stream based on a delay algorithm. Compared with the solution in the prior art in which the audio listening terminal only obtains the transmitted audio and plays the audio in real time, the solution disclosed in the present disclosure effectively takes into account the influence of network delay and can achieve synchronous playback of the video of the first terminal and the audio of the second terminal. Moreover, compared with the solution that only performs delayed rendering and playback processing on the video playing terminal or the audio listening terminal, the embodiment of the present disclosure improves both the first terminal and the second terminal, wherein the first terminal delays the playback of the video stream, and the second terminal delays the playback of the audio stream based on a delay algorithm, thereby achieving synchronous playback of audio and video, solving the problem of asynchronous audio and video playback, and effectively providing an immersive experience of synchronously listening to three-dimensional audio for the audience watching the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] FIG1 is a flow chart of a method for synchronously playing audio and video according to an embodiment of the present disclosure;

[0047] FIG2 is a schematic diagram of data flow between an outdoor large screen and a user's mobile terminal in an embodiment of the present disclosure;

[0048] FIG3 is a schematic diagram of a display screen of a first terminal in an embodiment of the present disclosure;

[0049] FIG4 is a schematic diagram of a display screen of a second terminal in an embodiment of the present disclosure;

[0050] FIG5 is a schematic diagram of the structure of the audio processing component of the second terminal in an embodiment of the present disclosure;

[0051] FIG6 is a flow chart of another method for synchronously playing audio and video provided in an embodiment of the present disclosure;

[0052] FIG7 is a schematic structural diagram of an apparatus for synchronously playing audio and video provided by an embodiment of the present disclosure;

[0053] FIG8 is a schematic structural diagram of another apparatus for synchronously playing audio and video provided by an embodiment of the present disclosure;

[0054] FIG9 is a schematic structural diagram of a system for synchronously playing audio and video according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0055] The following will be combined with the accompanying drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0056] In the description of the present disclosure, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present disclosure.

[0057] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of such features. Throughout this disclosure, unless otherwise specified, "plurality" means two or more.

[0058] In the description of this disclosure, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood broadly. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in this disclosure based on the specific circumstances.

[0059] 1 is a flow chart of a method for synchronously playing audio and video provided by an embodiment of the present disclosure. The present disclosure provides a method for synchronously playing audio and video, which is executed by a first terminal and includes steps S11 to S14:

[0060] S11, separating and decoding the audio and video streams to be played to obtain an audio stream and a video stream;

[0061] S12. Acquire a network delay between the first terminal and the second terminal through a real-time audio and video interactive connection established with the second terminal;

[0062] S13, delay rendering and playing the video stream according to a preset first delay time;

[0063] S14: Send the audio stream, the first delay time, and the network delay to the second terminal, so that the second terminal performs delayed rendering and playback of the audio stream based on the first delay time and the network delay.

[0064] It should be noted that the first terminal is a video stream playback terminal, which can be, for example, a large screen located in a public area of ​​a city, or a television, etc., without specific limitation here. The second terminal is an audio stream playback terminal, which can be, for example, a user's mobile terminal, including a mobile phone, tablet, or smartwatch, etc., without specific limitation here.

[0065] Existing technologies offer solutions that stream audio to mobile phones and other audio listening devices for rendering and playback, allowing users to listen to the audio on their phones while watching videos. However, existing technologies fail to account for network distribution delays and channel switching issues associated with large-screen live broadcasts. This often results in the audio and video being out of sync between the audience's mobile phone or other audio listening device and the video, resulting in a poor viewing experience for the audience.

[0066] In order to solve the problem in the prior art that the use of silent mode for playback will lead to a poor experience for the live audience, or the problem of audio and video asynchrony caused by not considering network delay and other issues when using other audio listening devices to listen to audio, in the embodiment of the present disclosure, after receiving the audio and video stream to be played sent by the audio and video resource station, the first terminal separates and decodes the audio and video stream to be played to obtain a video stream and an audio stream, and directly renders and plays the video stream on the display screen of the first terminal with a delay time of the preset first delay time A1. When the user watching the video wants to listen to the audio at the same time, a data channel for real-time audio and video interactive connection with the first terminal is established through the second terminal, and the audio stream is obtained from the first terminal through the data channel, and the first delay time A1 and the network delay T between the first terminal and the second terminal are obtained at the same time. The second terminal renders and plays the audio stream with a delay based on the first delay time A1 and the network delay T.

[0067] Preferably, the second terminal can calculate the second delay time based on the first delay time and the network delay, and delay rendering and playing the audio stream according to the second delay time; wherein, the second delay time A2 is the difference between the first delay time and the network delay, that is, A2 = A1-T.

[0068] In an embodiment of the present disclosure, the first terminal establishes a data channel for real-time audio and video interactive connection with the second terminal. Afterwards, when the user watching the video wants to listen to the audio at the same time, the audio stream is obtained from the first terminal through the data channel, and a delay request is sent to the first terminal through the data channel. The first terminal responds to the delay request sent by the second terminal, calculates the network delay T between the first terminal and the second terminal, and returns the network delay T and the first delay time A1. The second terminal calculates the second delay time A2 based on the first delay time A1 and the network delay T, and delays the rendering and playback of the audio stream. The delay time is the second delay time A2, and the delay time is adjusted in real time with the value obtained by real-time communication.

[0069] It should be noted that the manner in which the first terminal and the second terminal establish a real-time audio and video interactive connection can adopt relevant means of establishing a real-time interactive connection in the prior art, which does not affect the beneficial effects achieved by the present disclosure.

[0070] In addition, when establishing a data communication mechanism between the first terminal and the second terminal, the nearest first terminal can be automatically matched in the second terminal APP list through the positioning method of the Global Positioning System (GPS); or the audio data of the nearest first terminal can be pushed through the application (Application, APP), and the user can choose to listen / reject, thereby realizing a solution for quickly establishing a data channel and optimizing the user's viewing and listening experience.

[0071] Taking the first terminal as an outdoor large screen and the second terminal as a user mobile terminal as an example for explanation, see Figure 2, which is a schematic diagram of the data flow between the outdoor large screen and the user mobile terminal in the disclosed embodiment. The set-top box terminal of the outdoor large screen obtains 8K AVS3 HDR Vivid ultra-high-definition video ultra-high-definition video stream resources from the video source station through the edge node content distribution network (Content Delivery Network, CDN) through HLS / HTTP-flv and other technologies, and decodes and synchronizes the HDR&Audio Vivid audio and video streams on the set-top box end of the outdoor large screen to obtain Audio Vivid audio stream and HDR Vivid video stream, and directly delays the rendering and playback of the HDR Vivid video stream on the large screen end, with a delay time of A1. When a user is watching an outdoor large screen, a real-time audio and video interactive connection is established between the mobile terminal's software development kit (SDK) and the outdoor large screen set-top box SDK, forming a real-time audio and video interactive connection data channel. Through this RTC Data Channel, the user obtains the audio stream and delay time A1 corresponding to the video stream played by the outdoor large screen. Simultaneously, the user's mobile terminal SDK initiates a delay request to the outdoor large screen, obtaining the network delay T between the outdoor large screen and the mobile terminal, and calculating the delay time A2. Furthermore, the Audio Vivid audio stream is rendered and played with a delay of A2.

[0072] By adopting the technical means of the embodiment of the present disclosure, a real-time audio and video interactive connection is established between a first terminal and a second terminal, and two types of data, namely audio stream and time delay, are sent from the first terminal to the second terminal, and the second terminal performs delayed rendering and playback of the audio stream based on a delay algorithm. Compared with the solution in the prior art in which the audio listening terminal only obtains the transmitted audio and plays the audio in real time, the solution of the present disclosure effectively takes into account the influence of network delay and can achieve synchronous playback of the video of the first terminal and the audio of the second terminal. Moreover, compared with the solution of only performing delayed rendering and playback processing on the video playing terminal or the audio listening terminal, the embodiment of the present disclosure improves both the first terminal and the second terminal. The first terminal delays the playback of the video stream, and the second terminal delays the playback of the audio stream based on a delay algorithm, thereby achieving synchronous playback of audio and video, solving the problem of asynchronous audio and video playback, and effectively providing the live audience watching the video with an immersive experience of synchronously listening to three-dimensional audio.

[0073] As a preferred implementation, the embodiment of the present disclosure is further implemented on the basis of the above embodiment, and the establishment of the audio and video interactive connection includes:

[0074] S121: Create a real-time audio and video interactive room and generate an audio and video interactive link so that a second terminal can join the audio and video interactive room through the audio and video interactive link;

[0075] S122: When the second terminal joins the audio and video interactive room, a real-time audio and video interactive connection is established with the second terminal through a real-time audio and video network service.

[0076] In the embodiment of the present disclosure, when the first terminal receives the audio and video stream to be played, a real-time audio and video (RTC) interactive room exclusive to the first terminal is created, and a corresponding audio and video interactive link is generated. Any second terminal can join the real-time audio and video interactive room through the audio and video interactive link. The first terminal establishes a real-time audio and video interactive data channel with the second terminal through the real-time audio and video network service to transmit relevant data.

[0077] Preferably, step S121, i.e., creating a real-time audio and video interactive room and generating an audio and video interactive link so that the second terminal can join the audio and video interactive room through the audio and video interactive link, includes:

[0078] Create a real-time audio and video interactive room and generate an audio and video interactive link;

[0079] Generating an identification code including the audio and video interactive link according to the audio and video interactive link;

[0080] The identification code is displayed on the display screen of the first terminal, so that the second terminal can join the audio and video interactive room by scanning the identification code.

[0081] Preferably, the identification code includes but is not limited to a QR code, a bar code or a pattern code.

[0082] Refer to Figures 3 and 4, Figure 3 is a schematic diagram of the display screen of the first terminal in the embodiment of the present disclosure, and Figure 4 is a schematic diagram of the display screen of the second terminal in the embodiment of the present disclosure. In the embodiment of the present disclosure, when the first terminal receives the audio and video stream to be played, it creates a real-time audio and video interaction room exclusive to the first terminal, generates a corresponding audio and video interaction link, and creates an identification code exclusive to the first terminal based on the audio and video interaction link, such as a QR code, and synchronously displays the QR code at a specific position on the display screen of the first terminal, such as the lower right corner, to guide the user to scan and listen.

[0083] The user watching the video uses the scanning function of a second terminal, such as any app in a mobile terminal such as a mobile phone, to scan the QR code displayed by the first terminal, extract the link information of the QR code, join the first terminal to establish a real-time audio and video room, and establish a real-time audio and video interactive connection data channel with the first terminal through the real-time audio and video interactive connection service network deployed in the cloud to obtain the real-time audio stream corresponding to the video currently played by the first terminal, as well as data such as the first delay time and network delay, and then delay rendering and playback of the audio stream.

[0084] By adopting the technical means of the embodiments of the present disclosure, a real-time interactive connection is established between the first terminal and the second terminal by scanning a QR code by the second terminal, so that the establishment of the real-time interactive connection will not be affected by the distance between the two devices, thereby ensuring the success rate of the second terminal in obtaining and playing audio stream data and improving the user experience.

[0085] As a preferred implementation, the embodiment of the present disclosure is further implemented on the basis of any of the above embodiments. In step S12, that is, obtaining the network delay between the first terminal and the second terminal includes:

[0086] S123. In response to the delay request sent by the second terminal, calculate the round-trip delay between the first terminal and the second terminal using an RTT standard algorithm;

[0087] S124. Calculate the network delay between the first terminal and the second terminal according to the round-trip delay.

[0088] Preferably, step S124 specifically includes: obtaining half of the round-trip delay to obtain the network delay between the first terminal and the second terminal.

[0089] In the embodiment of the present disclosure, the delay request is a round-trip time (RTT) request. The round-trip time refers to the total time it takes for a network request to travel from a starting point to a destination and then back to the destination. The example of the present disclosure utilizes the RTT standard algorithm (Jacobson / Karels algorithm). The characteristic of this algorithm [RFC6298] is that it introduces the difference between the latest RTT sample rtts and the smoothed srtt as a parameter for calculation.

[0090] When the first terminal receives the delay request sent by the second terminal, it uses the RTT standard algorithm to calculate the round-trip delay RTT between the first terminal and the second terminal, and the network delay T between the first terminal and the second terminal is RTT / 2.

[0091] It should be noted that network latency is closely related to RTT latency. In addition to the network transmission latency, RTT latency also includes the terminal processing latency. Therefore, although network latency is not necessarily exactly equal to half of the RTT latency, the terminal processing latency is a fixed value and can be considered and eliminated by delaying the video rendering delay time A1.

[0092] Preferably, the network delay T is the sum of the first network delay T1 and the second network delay T2; wherein, the first network delay T1 is the network delay between the first terminal and the real-time audio and video network service, and the second network delay T2 is the network delay between the second terminal and the real-time audio and video network service.

[0093] The calculating the network delay between the first terminal and the second terminal includes:

[0094] Calculate the network delay between the first terminal and the real-time audio and video network service as the first network delay; calculate the second network delay as the network delay between the second terminal and the real-time audio and video network service as the second network delay;

[0095] Then, sending the audio stream, the first delay time, and the network delay to the second terminal includes:

[0096] The audio stream, the first delay time, the first network delay, and the second network delay are sent to the second terminal.

[0097] That is, in the embodiment of the present disclosure, the first terminal, in response to the delay request, calculates the network delay between the first terminal and the real-time audio and video network service, i.e., the first network delay T1, and the network delay between the second terminal and the real-time audio and video network service, i.e., the second network delay T2, and sends the first network delay T1 and the second network delay T2 together to the second terminal. The second terminal calculates the second delay A2 = A1 - (T1 + T2) based on the first delay A1, the first network delay T1, and the second network delay T2.

[0098] Preferably, calculating the network delay between the first terminal and the real-time audio and video network service as the first network delay includes:

[0099] Calculate the round-trip delay between the first terminal and the real-time audio and video network service using an RTT standard algorithm;

[0100] Calculating the first network delay based on a round-trip delay between the first terminal and the real-time audio and video network service;

[0101] The calculating, as the second network delay, the network delay between the second terminal and the real-time audio and video network service includes:

[0102] Calculate the round-trip delay between the second terminal and the real-time audio and video network service using an RTT standard algorithm;

[0103] The second network delay is calculated according to the round-trip delay between the second terminal and the real-time audio and video network service.

[0104] Specifically, the round-trip delay RTT between the first terminal and the second terminal is the sum of the first round-trip delay RTT1 and the second round-trip delay RTT2; wherein, the first round-trip delay RTT1 is the round-trip delay between the first terminal and the real-time audio and video network service, and the second round-trip delay RTT2 is the round-trip delay between the second terminal and the real-time audio and video network service. Then, the first network delay T1 between the first terminal and the real-time audio and video network service is equal to RTT1 / 2, and the second network delay T2 between the second terminal and the real-time audio and video network service is equal to RTT2 / 2. The first terminal transmits the first network delay and the second network delay to the second terminal through the data channel of the real-time audio and video interactive connection, and the second terminal calculates the second delay time A2 is equal to A1-(RTT1+RTT2) / 2.

[0105] Taking the example of the first terminal being an outdoor large screen and the second terminal being a user's mobile terminal, the set-top box terminal of the outdoor large screen obtains 8K AVS3 HDR Vivid ultra-high-definition video stream resources from the video source station through the edge node CDN through technologies such as HLS / HTTP-flv, decodes and synchronizes the HDR&Audio Vivid audio and video streams on the set-top box end of the outdoor large screen, obtains the Audio Vivid audio stream and the HDR Vivid video stream, and performs delayed rendering and playback of the HDR Vivid video stream directly on the large screen end, with a delay time of A1. In addition, when the outdoor large screen set-top box receives the audio and video stream, it creates an RTC real-time audio and video interactive room exclusive to the large screen, and uses the exclusive room link of the large screen to create an exclusive outdoor large screen QR code, and synchronously displays the audio playback QR code link in the lower right corner of the outdoor large screen to guide users to scan and listen. On-site audiences use the QR code scanning function of any app on their mobile terminals to scan the QR code displayed on the outdoor large screen, extract the link information of the QR code, join the large screen to establish a WebRTC real-time audio and video room, and the user's mobile SDK establishes an ICE (Interactive Connectivity Establishment) interactive connection with the outdoor large screen set-top box SDK through the real-time audio and video (WebRTC) network service deployed in the cloud, realizing a peer-to-peer connection between the large screen SDK and the mobile SDK, and obtains the audio stream and delay time A1 corresponding to the video stream played by the outdoor large screen through the RTC Data Channel. At the same time, the user's mobile terminal SDK initiates a delay request to the outdoor large screen. The outdoor large screen estimates the first network delay between the user's mobile SDK and the RTC network service as T1=RTT1 / 2, and obtains the second network delay between the outdoor large screen SDK and the RTC network as T2=RTT2 / 2 through WebRTC parameters, and transmits the value to the user's mobile SDK through the RTC Data Channel. Referring to Figure 5, it is a structural diagram of the audio processing component of the second terminal in the embodiment of the present disclosure. The Network-aware Enhanced Quality (NetEQ) module is a key audio processing component. The NetEQ module of the mobile SDK delays the playback of the received audio. The delay time is A2=A1-(RTT1+RTT2) / 2, and the delay time is adjusted in real time with the value obtained by real-time communication. In the embodiment of the present disclosure, the user's mobile SDK sends a message to obtain the RTT time by real-time communication with the large-screen SDK, and uses RTCDataChannel to transmit audio data. The NetEQ module of the mobile SDK uses a delay algorithm to decode and play the audio data sent by the large-screen end, so as to achieve the effect of separating the user-end audio and the large-screen end and rendering and playing synchronously.

[0106] By adopting the technical means of the embodiments of the present invention, the RTT standard algorithm is used to calculate the network delay between the first terminal and the second terminal, and the delayed rendering time of the audio is determined in combination with the delay time of the delayed video rendering. By performing delayed rendering processing on both ends of the audio and video, the effect of synchronous playback of audio and video can be truly achieved.

[0107] 6 is a flow chart of another method for synchronously playing audio and video provided by an embodiment of the present disclosure. The present disclosure further provides a method for synchronously playing audio and video, which is executed by a second terminal and includes steps S21 to S23:

[0108] S21. Establishing a real-time audio and video interactive connection with a first terminal; wherein the first terminal is capable of separating and decoding the audio and video stream to be played to obtain an audio stream and a video stream, and delaying the rendering and playing of the video stream according to a preset first delay time;

[0109] S22. Receive the audio stream, the first delay time, and the network delay between the first terminal and the second terminal sent by the first terminal;

[0110] S23. Perform delayed rendering and playback on the audio stream according to the first delay time and the network delay.

[0111] Preferably, after step S21 and before step S22, the method further includes:

[0112] S21′, sending a delay request to the first terminal; wherein the delay request is used to trigger the first terminal to calculate the network delay between the first terminal and the second terminal;

[0113] Preferably, step S23 is specifically as follows:

[0114] Calculate the difference between the first delay time and the network delay to obtain a second delay time;

[0115] The audio stream is rendered and played with delay according to the second delay time.

[0116] It should be noted that the first terminal is a video stream playback terminal, which can be, for example, a large screen installed in a public area of ​​a city, or a television, etc., and is not specifically limited here. The second terminal is an audio stream playback terminal, which can be, for example, a user's mobile terminal, including a mobile phone, tablet, or smartwatch, etc., and is not specifically limited here.

[0117] In an embodiment of the present disclosure, after receiving the audio and video stream to be played sent by the audio and video resource station, the first terminal separates and decodes the audio and video stream to be played to obtain a video stream and an audio stream, and directly delays the rendering and playback of the video stream on the display screen of the first terminal, with the delay time being the preset first delay time A1. When the user watching the video wants to listen to the audio at the same time, a data channel for real-time audio and video interactive connection with the first terminal is established through the second terminal, and the audio stream is obtained from the first terminal through the data channel. At the same time, a delay request is sent to the first terminal. The first terminal responds to the delay request sent by the second terminal, calculates the network delay T between the first terminal and the second terminal, and returns the network delay T and the first delay time A1. The second terminal calculates the second delay time A2 based on the first delay time A1 and the network delay T, and delays the rendering and playback of the audio stream, with the delay time being the second delay time A2. The delay time is adjusted in real time with the value obtained from the real-time communication.

[0118] Preferably, calculating the second delay time according to the first delay time and the network delay includes: calculating the difference between the first delay time and the network delay to obtain the second delay time.

[0119] Specifically, in the embodiment of the present disclosure, the second delay time A2 is the difference between the first delay time and the network delay, that is, A2 = A1 - T.

[0120] By adopting the technical means of the embodiment of the present disclosure, a real-time audio and video interactive connection is established between a first terminal and a second terminal, and two types of data, namely audio stream and time delay, are sent from the first terminal to the second terminal, and the second terminal performs delayed rendering and playback of the audio stream based on a delay algorithm. Compared with the solution in the prior art in which the audio listening terminal only obtains the transmitted audio and plays the audio in real time, the solution of the present disclosure effectively takes into account the influence of network delay and can achieve synchronous playback of the video of the first terminal and the audio of the second terminal. Moreover, compared with the solution of only performing delayed rendering and playback processing on the video playing terminal or the audio listening terminal, the embodiment of the present disclosure improves both the first terminal and the second terminal. The first terminal delays the playback of the video stream, and the second terminal delays the playback of the audio stream based on a delay algorithm, thereby achieving synchronous playback of audio and video, solving the problem of asynchronous audio and video playback, and effectively providing the live audience watching the video with an immersive experience of synchronously listening to three-dimensional audio.

[0121] As a preferred embodiment, the embodiment of the present disclosure is further implemented on the basis of the above embodiment, and step S21, i.e., establishing a real-time audio and video interactive connection with the first terminal, includes:

[0122] Scanning an identification code on the display screen of the first terminal to enter the real-time audio and video interactive room created by the first terminal; wherein the identification code is generated by the first terminal after creating the real-time audio and video interactive room and generating a real-time audio and video interactive link based on the real-time audio and video interactive link;

[0123] A real-time audio and video interactive connection is established with the first terminal through a real-time audio and video network service.

[0124] Preferably, the identification code includes but is not limited to a QR code, a bar code or a pattern code.

[0125] In an embodiment of the present disclosure, when the first terminal receives the audio and video stream to be played, it creates a real-time audio and video interaction room exclusive to the first terminal, generates a corresponding audio and video interaction link, and creates an identification code exclusive to the first terminal based on the audio and video interaction link, such as a QR code, and synchronously displays the QR code at a specific location on the display screen of the first terminal, such as the lower right corner, to guide the user to scan and listen.

[0126] The user watching the video uses the scanning function of a second terminal, such as any app in a mobile terminal such as a mobile phone, to scan the QR code displayed by the first terminal, extract the link information of the QR code, join the first terminal to establish a real-time audio and video room, and establish a real-time audio and video interactive connection data channel with the first terminal through the real-time audio and video interactive connection service network deployed in the cloud to obtain the real-time audio stream corresponding to the video currently played by the first terminal, as well as data such as the first delay time and network delay, and then delay rendering and playback of the audio stream.

[0127] By adopting the technical means of the embodiments of the present disclosure, a real-time interactive connection is established between the first terminal and the second terminal by scanning a QR code by the second terminal, so that the establishment of the real-time interactive connection will not be affected by the distance between the two devices, thereby ensuring the success rate of the second terminal in obtaining and playing audio stream data and improving the user experience.

[0128] 7 is a schematic diagram of the structure of a synchronous audio and video playback device provided by an embodiment of the present disclosure. The present disclosure also provides a synchronous audio and video playback device 30, which is applied to a first terminal. The device 30 includes:

[0129] The audio and video stream decoding module 31 is used to separate and decode the audio and video streams to be played to obtain audio streams and video streams;

[0130] a network delay calculation module 32, configured to obtain the network delay between the first terminal and the second terminal through a real-time audio and video interactive connection established with the second terminal;

[0131] The video stream playing module 33 is used to delay rendering and play the video stream according to a preset first delay time;

[0132] The data sending module 34 is configured to send the audio stream, the first delay time, and the network delay to the second terminal, so that the second terminal performs delayed rendering and playback of the audio stream based on the first delay time and the network delay.

[0133] By adopting the technical means of the embodiment of the present disclosure, a real-time audio and video interactive connection is established between a first terminal and a second terminal, and two types of data, namely audio stream and time delay, are sent from the first terminal to the second terminal, and the second terminal performs delayed rendering and playback of the audio stream based on a delay algorithm. Compared with the solution in the prior art in which the audio listening terminal only obtains the transmitted audio and plays the audio in real time, the solution of the present disclosure effectively takes into account the influence of network delay and can achieve synchronous playback of the video of the first terminal and the audio of the second terminal. Moreover, compared with the solution of only performing delayed rendering and playback processing on the video playing terminal or the audio listening terminal, the embodiment of the present disclosure improves both the first terminal and the second terminal. The first terminal delays the playback of the video stream, and the second terminal delays the playback of the audio stream based on a delay algorithm, thereby achieving synchronous playback of audio and video, solving the problem of asynchronous audio and video playback, and effectively providing the live audience watching the video with an immersive experience of synchronously listening to three-dimensional audio.

[0134] It should be noted that the audio and video synchronous playback device 30 provided in the embodiment of the present disclosure is used to execute all the process steps of the audio and video synchronous playback method applied to the first terminal in the above embodiment. The working principles and beneficial effects of the two correspond one to one, so they will not be repeated here.

[0135] 8 is a schematic diagram showing the structure of another apparatus for synchronously playing audio and video provided in an embodiment of the present disclosure. The present disclosure also provides an apparatus 40 for synchronously playing audio and video, which is applied to a second terminal. The apparatus 40 includes:

[0136] A connection establishment module 41 is configured to establish a real-time audio and video interactive connection with a first terminal; wherein the first terminal is capable of separating and decoding the audio and video streams to be played to obtain an audio stream and a video stream, and delaying the rendering and playback of the video stream according to a preset first delay time;

[0137] a data receiving module 42, configured to receive the audio stream, the first delay time, and the network delay between the first terminal and the second terminal sent by the first terminal;

[0138] The audio stream playing module 43 is used to perform delayed rendering and playing of the audio stream according to the first delay time and the network delay.

[0139] It should be noted that the audio and video synchronous playback device 40 provided in the embodiment of the present disclosure is used to execute all the process steps of the audio and video synchronous playback method applied to the second terminal in the above embodiment. The working principles and beneficial effects of the two correspond one to one, so they will not be repeated here.

[0140] 9 is a structural diagram of a synchronous audio and video playback system provided by an embodiment of the present disclosure. The present disclosure also provides an audio and video synchronous playback system 50, including a first terminal 51 and a second terminal 52; wherein,

[0141] The first terminal 51 is configured to separate and decode the audio and video streams to be played to obtain an audio stream and a video stream, obtain the network delay between the first terminal and the second terminal through a real-time audio and video interactive connection established with the second terminal, delay the playback of the video stream according to a preset first delay time, and send the audio stream, the first delay time, and the network delay to the second terminal;

[0142] The second terminal 52 is configured to delay playing the audio stream based on the first delay time and the network delay.

[0143] It should be noted that the first terminal is used to execute the audio and video synchronous playback method applied to the first terminal in the above embodiment, and the second terminal is used to execute the audio and video synchronous playback method applied to the second terminal in the above embodiment, which will not be repeated here.

[0144] An embodiment of the present disclosure also provides a synchronous playback device for audio and video, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the synchronous playback method for audio and video as described in any of the above embodiments.

[0145] An embodiment of the present disclosure further provides a computer-readable storage medium, which includes a stored computer program. When the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the method for synchronous playback of audio and video as described in any one of the above embodiments.

[0146] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0147] The above is a preferred embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present disclosure. These improvements and modifications are also considered to be within the scope of protection of the present disclosure.

Claims

1. A method for synchronously playing audio and video, applied to a first terminal, the method comprising: Separate and decode the audio and video streams to be played to obtain audio streams and video streams; Acquire a network delay between the first terminal and the second terminal through a real-time audio and video interactive connection established with the second terminal; Performing delayed rendering and playing of the video stream according to a preset first delay time; The audio stream, the first delay time, and the network delay are sent to the second terminal, so that the second terminal performs delayed rendering and playback of the audio stream based on the first delay time and the network delay.

2. The method for synchronously playing audio and video according to claim 1, wherein: The establishment of the audio and video interactive connection includes: Creating a real-time audio and video interactive room, and generating an audio and video interactive link, so that the second terminal can join the audio and video interactive room through the audio and video interactive link; When the second terminal joins the audio and video interactive room, a real-time audio and video interactive connection is established with the second terminal through a real-time audio and video network service.

3. The method for synchronously playing audio and video according to claim 2, wherein: The step of creating a real-time audio and video interactive room and generating an audio and video interactive link so that a second terminal can join the audio and video interactive room through the audio and video interactive link includes: Create a real-time audio and video interactive room and generate an audio and video interactive link; According to the audio and video interactive link, generating an identification code including the audio and video interactive link; The identification code is displayed on a display screen of the first terminal, so that the second terminal can join the audio and video interactive room by scanning the identification code.

4. The method for synchronously playing audio and video according to claim 1, wherein: The acquiring the network delay between the first terminal and the second terminal includes: In response to the delay request sent by the second terminal, calculating the round-trip delay between the first terminal and the second terminal using an RTT standard algorithm; A network delay between the first terminal and the second terminal is calculated according to the round-trip delay.

5. The method for synchronously playing audio and video according to claim 1 or 4, wherein: The network delay is the sum of the first network delay and the second network delay; wherein the first network delay is the network delay between the first terminal and the real-time audio and video network service, and the second network delay is the network delay between the second terminal and the real-time audio and video network service.

6. A method for synchronously playing audio and video, applied to a second terminal, the method comprising: Establishing a real-time audio and video interactive connection with a first terminal; wherein the first terminal is capable of separating and decoding the audio and video stream to be played to obtain an audio stream and a video stream, and delaying the rendering and playing of the video stream according to a preset first delay time; receiving the audio stream, the first delay time, and a network delay between the first terminal and the second terminal sent by the first terminal; The audio stream is delayed rendered and played according to the first delay time and the network delay.

7. The method for synchronously playing audio and video according to claim 6, wherein: The delaying rendering and playing the audio stream according to the first delay time and the network delay includes: Calculate the difference between the first delay time and the network delay to obtain a second delay time; The audio stream is rendered and played after delay according to the second delay time.

8. A synchronous audio and video playback system, comprising a first terminal and a second terminal; wherein: The first terminal is used to separate and decode the audio and video streams to be played to obtain an audio stream and a video stream, and obtain the network delay between the first terminal and the second terminal through a real-time audio and video interactive connection established with the second terminal; delay the video stream according to a preset first delay time, and send the audio stream, the first delay time and the network delay to the second terminal; The second terminal is used to delay playing the audio stream based on the first delay time and the network delay.

9. A synchronous playback device for audio and video, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the synchronous playback method for audio and video as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, the computer-readable storage medium comprising a stored computer program, wherein: When the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the method for synchronously playing audio and video as described in any one of claims 1 to 7.

Images