Synchronous playing system and method, electronic device, storage medium and chip system

Abstract

The application provides a synchronous playing system and method, an electronic device, a storage medium and a chip system, relates to the technical field of distributed hardware, and can realize audio and picture synchronization in a cross-device data broadcasting application scenario. The method can be executed based on one device of a playing end, and includes the following steps: receiving a first timestamp; the first timestamp indicates an actual playing time of an mth first data frame in the first device; m is greater than or equal to 1 and is an integer; reading a second timestamp corresponding to a second data frame to be played; the second timestamp is used to indicate a playing time sequence of each second data frame in a second data stream; calculating a first playing time difference according to the first timestamp and the second timestamp; and determining a synchronous playing strategy executed for the second data frame to be played based on the first playing time difference.

Description

Technical Field

[0001] This application relates to the field of distributed hardware technology, and in particular to a synchronous playback system and method, electronic device, storage medium and chip system. Background Technology

[0002] A distributed operating system integrates the capabilities of various terminal devices, enabling hardware collaboration and resource sharing to provide users with a smooth, all-scenario experience. Based on distributed hardware, a distributed operating system can provide users with various distributed services, such as distributed cameras, distributed screens, and distributed audio. These distributed services typically include audio / video transmission capabilities. For example, in audio / video broadcasting scenarios, the source end encodes and compresses the original audio / video stream, sends it to the playback end through the transmission module, and then decodes, restores, and plays it back.

[0003] In audio and video casting scenarios based on distributed hardware, due to differences in sampling rates and post-processing times between audio and video data, as well as network jitter, audio and video playback may become out of sync when playing across devices, affecting the user experience. Summary of the Invention

[0004] This application provides a synchronous playback system and method, electronic device, storage medium and chip system, which can realize synchronous audio and video playback in distributed business scenarios with cross-device broadcasting.

[0005] In a first aspect, embodiments of this application provide a synchronous playback system, which includes a first device, a second device, and a third device; the third device is used to transmit a first data stream to the first device for playback, and to transmit a second data stream to the second device for playback; the first data stream and the second data stream are two data streams to be played synchronously; the first data stream includes multiple first data frames; the second data stream includes multiple second data frames; the first device is used to obtain a first timestamp and send the first timestamp to the third device; the third device is used to receive the first timestamp and forward it to the second device; the second device is used to receive the first timestamp, read the second timestamp corresponding to the second data frame to be played, and calculate a first playback time difference based at least on the first timestamp and the second timestamp, and determine a synchronous playback strategy to be executed for the second data frame to be played based on the first playback time difference.

[0006] In one feasible implementation, when the second device performs the operation of reading the second timestamp corresponding to the second data frame to be played, it is specifically used to: retrieve the next second data frame to be played from the buffer queue at predetermined time intervals and read the second timestamp.

[0007] In one feasible implementation, when the second device performs the operation of calculating the first playback time difference based at least on the first timestamp and the second timestamp, it is specifically used to: if it is determined that there is a mapping relationship between the next second data frame to be played and the m-th first data frame, calculate the first playback time difference based at least on the first timestamp corresponding to the m-th first data frame and the second timestamp corresponding to the next second data frame to be played; wherein, the mapping relationship is determined based on the sequence number and / or the playback timestamp PTS.

[0008] In one feasible implementation, the second device is further configured to identify the data type of the second data frame to be played and feed it back to the third device; the first device is further configured to identify the data type of the first data frame to be played and feed it back to the third device; the third device is further configured to receive the respective data types fed back by the first device and the second device and determine the synchronization management strategy.

[0009] Secondly, embodiments of this application also provide a synchronous playback system, which includes a first device, a second device, and a third device; the third device transmits a first data stream to the first device for playback and transmits a second data stream to the second device for playback; the first data stream includes multiple first data frames; the second data stream includes multiple second data frames; the first device is used to determine latency information; the latency information includes transmission latency and decoding latency; wherein, the transmission latency represents the duration of transmission of the first data frame from the third device to the first device; the decoding latency is the decoding duration occupied by one first data frame; the third device is used to receive the latency information, obtain the transmission time of the first data frame, and at least based on the transmission time and latency information, obtain an estimated value of the playback time of the first data frame on the first device; before sending the second data frame, the estimated value is added to the second data frame as a fifth timestamp; the second device is used to read the fifth timestamp of the second data frame to be played, calculate the difference between the fifth timestamp and the current time to obtain the second playback time difference; and determine a synchronous playback strategy to be executed for the second data frame to be played based on the second playback time difference.

[0010] Thirdly, embodiments of this application also provide a synchronous playback system, which includes a fourth device, a fifth device, and a sixth device; the fourth device and the fifth device respectively transmit a fourth data stream and a fifth data stream to the sixth device for playback; the fourth device, the fifth device, and the sixth device are all equipped with a control center, and the control center reserves an interface for setting synchronization group information; the fourth device and the fifth device are used to set the synchronization group information to their local control center, and to send the set synchronization group information to the sixth device; the synchronization group information includes synchronization reference information and stream information, the synchronization reference information includes source device identity information and playback device identity information; the stream information includes data type information of the data streams sent by each source device; the sixth device is used to receive the synchronization group information sent by the fourth device and the fifth device respectively, and to determine a synchronous playback strategy for the data stream to be played, at least based on the synchronization group information.

[0011] Fourthly, this application provides a synchronous playback method. This method is applied to a second device, which receives and plays a second data stream sent by a third device. The third device also transmits a first data stream to the first device for playback. The first and second data streams are two data streams to be played synchronously. The first data stream includes multiple first data frames, and the second data stream includes multiple second data frames. The method includes: receiving a first timestamp; the first timestamp represents the actual playback time of the m-th first data frame on the first device; m ≥ 1 and is an integer; reading the second timestamp corresponding to the second data frame to be played; the second timestamp represents the playback sequence of each second data frame in the second data stream; calculating a first playback time difference based at least on the first and second timestamps; and determining a synchronous playback strategy to be executed for the second data frame to be played based on the first playback time difference. This method uses the actual playback time of each data frame in the first data stream as a reference to calculate the playback time difference between the second and first data streams. The playback time difference can be used to determine whether the playback progress of the second data stream is ahead, behind, or synchronized with the first data stream, thereby adjusting the playback progress of the second data stream and achieving synchronous playback.

[0012] In one feasible implementation, receiving the first timestamp includes: receiving the first timestamp forwarded by the third device; the first timestamp is generated by the first device and sent to the third device. The forwarding of the first timestamp by the third device, compared to other information transmission methods such as the first device directly sending it to the second device, can reduce the transmission pressure on the data link and reduce the latency occupied by transmitting this crucial first timestamp information.

[0013] In one feasible implementation, reading the second timestamp corresponding to the second data frame to be played includes: retrieving the next second data frame to be played from the buffer queue at predetermined time intervals and reading the second timestamp. The trigger condition for reading the second timestamp is that it is triggered once at predetermined time intervals. The frequency of reading the second timestamp can be controlled by setting and adjusting the time interval, thereby controlling the frequency of calculating the playback time difference. The higher the frequency, the more granular the time difference determination is.

[0014] In one feasible implementation, calculating the first playback time difference, at least based on the first timestamp and the second timestamp, includes: if it is determined that there is a mapping relationship between the next second data frame to be played and the m-th first data frame, calculating the first playback time difference, at least based on the first timestamp corresponding to the m-th first data frame and the second timestamp corresponding to the next second data frame to be played; wherein, the mapping relationship is determined based on the sequence number and / or the playback timestamp (PTS). Calculating the playback time difference based on the first and second timestamps can reflect the difference in playback progress between the first and second data streams to a certain extent.

[0015] In one feasible implementation, the first playback time difference is calculated based at least on the first timestamp and the second timestamp, including: calculating the first playback time difference based on the first timestamp, the second timestamp, the first processing duration, and the second processing duration; wherein, the first processing duration is an estimated value of the processing time occupied from when a first data frame is retrieved from the buffer queue to when it is played; and the second processing duration is an estimated value of the processing time occupied from when a second data frame is retrieved from the buffer queue to when it is played. Further introducing two parameters, the first processing duration and the second processing duration, to calculate the playback time difference can more accurately reflect the difference in playback progress between the first data stream and the second data stream.

[0016] In one feasible implementation, the synchronous playback strategy includes at least one of waiting, displaying, and catching up. Based on a first playback time difference, determining the synchronous playback strategy to be executed for the second data frame to be played includes: comparing the first playback time difference with a predetermined first threshold interval; if the first playback time difference is less than the minimum value of the threshold interval, determining that the playback of the first data stream lags behind the second data stream, and determining that the synchronous playback strategy to be executed for the second data frame to be played is waiting; if the first playback time difference is greater than the maximum value of the threshold interval, determining that the playback of the first data stream is ahead of the second data stream, and determining that the synchronous playback strategy to be executed for the second data frame to be played is catching up; if the first playback time difference is within the threshold interval, determining that the playback of the first data stream and the playback of the second data stream are relatively synchronized, and determining that the synchronous playback strategy to be executed for the second data frame to be played is displaying.

[0017] In one feasible implementation, waiting includes one or more of reducing the playback rate, waiting for a first predetermined duration, and dropping frames; wherein the first predetermined duration is determined based on the difference between the minimum value in a threshold interval and the first playback time difference; catching up includes increasing the playback rate and / or dropping frames.

[0018] Fifthly, embodiments of this application also provide a synchronous playback method, which is applied to a third device. The third device is used to transmit a first data stream to the first device for playback and to transmit a second data stream to the second device for playback. The first data stream and the second data stream are two data streams to be played synchronously. The first data stream includes multiple first data frames; the second data stream includes multiple second data frames. The method includes: receiving delay information; the delay information includes transmission delay and decoding delay; wherein the transmission delay represents the duration of transmission of the m-th first data frame from the third device to the first device; the decoding delay is the decoding duration occupied by the m-th first data frame; m ≥ 1 and is an integer; obtaining the transmission time of the m-th first data frame; obtaining an estimated value of the playback time of the m-th first data frame on the first device based at least on the transmission time and delay information; before sending the second data frame, adding the estimated value as a fifth timestamp to the second data frame; the fifth timestamp is used to calculate the difference between the second data frame and the current time to obtain the second playback time difference, thereby determining the synchronous playback strategy to be executed for the second data frame to be played. This method estimates the playback time of each data frame in the first data stream based on the historical latency information of the first data stream. Using this estimate as a reference, the second device can determine the difference in playback progress by comparing the estimate with the current time, thereby achieving synchronized playback.

[0019] In one feasible implementation, obtaining an estimated playback time for the first data frame on the first device, based at least on the transmission time and delay information, includes: obtaining an estimated playback time for the m-th first data frame on the first device, based at least on the transmission time and delay information of the m-th first data frame; adding the estimated value as a fifth timestamp to the second data frame, including: adding the estimated value as a fifth timestamp to the second data frame that has a mapping relationship with the m-th first data frame, wherein the mapping relationship is determined based on the sequence number and / or playback timestamp (PTS). Two data frames with a mapping relationship, i.e., two data frames that should be played synchronously, have their fifth timestamp added to the second data frame that has a mapping relationship with the m-th first data frame. This is equivalent to marking the estimated playback time of the m-th first data frame on the corresponding second data frame. Before playback, the second device only needs to compare the fifth timestamps on each second data frame with the current time to determine the difference in playback progress, thereby achieving synchronous playback.

[0020] In one feasible implementation, an estimated playback time for the first data frame on the first device is obtained based at least on the transmission time and delay information. This includes obtaining an estimated playback time for the first data frame on the first device based on the transmission time, delay information, a third timestamp, and a fourth timestamp. The third timestamp is a playback timestamp (PTS) added to the first data frame to indicate the playback sequence of each first data frame in the first data stream; the fourth timestamp is a playback timestamp (PTS) added to the second data frame to indicate the playback sequence of each second data frame in the second data stream. This method provides a feasible scheme for calculating the estimated playback time, and the calculated estimated value can relatively accurately reflect the difference in playback progress between the first and second data streams.

[0021] In a sixth aspect, embodiments of this application also provide a synchronous playback method, which is applied to a second device, the second device being used to receive and play a second data stream sent by a third device; wherein, the third device is also used to transmit a first data stream to the first device for playback; the first data stream and the second data stream are two data streams to be played synchronously, the first data stream including a plurality of first data frames; the second data stream including a plurality of second data frames; the method includes: reading a fifth timestamp corresponding to the second data frame to be played; wherein, the fifth timestamp is an estimated value of the playback time of the first data frame on the first device, obtained at least based on the transmission time and delay information; wherein, the delay information includes transmission delay and decoding delay; the transmission delay represents the duration of the first data frame being transmitted from the third device to the first device; the decoding delay is the decoding duration occupied by one first data frame; the transmission time is the time when the third device sends the first data frame; calculating the difference between the fifth timestamp and the current time to obtain a second playback time difference; and determining a synchronous playback strategy to be executed for the second data frame to be played based on the second playback time difference.

[0022] In one feasible implementation, the synchronous playback strategy includes at least one of waiting, displaying, and catching up. Based on a second playback time difference, determining the synchronous playback strategy to be executed for the second data frame to be played includes: comparing the second playback time difference with a predetermined first threshold interval; if the second playback time difference is less than the minimum value of the threshold interval, determining that the playback of the first data stream lags behind the second data stream, and determining that the synchronous playback strategy to be executed for the second data frame to be played is waiting; if the second playback time difference is greater than the maximum value of the threshold interval, determining that the playback of the first data stream is ahead of the second data stream, and determining that the synchronous playback strategy to be executed for the second data frame to be played is catching up; if the second playback time difference is within the threshold interval, determining that the playback of the first data stream and the playback of the second data stream are relatively synchronized, and determining that the synchronous playback strategy to be executed for the second data frame to be played is displaying.

[0023] In a seventh aspect, embodiments of this application also provide a synchronous playback method, which is applied to a sixth device. The sixth device is used to receive a fourth data stream sent by a fourth device and a fifth data stream sent by a fifth device, and to synchronously play the fourth data stream and the fifth data stream. The method includes: receiving synchronization group information sent by the fourth device and the fifth device respectively; the synchronization group information includes synchronization reference information and stream information, the synchronization reference information including source device identity information and playback device identity information; the stream information including data type information of the data streams sent by each source device; and determining a synchronous playback strategy for the data streams to be played, at least based on the synchronization group information.

[0024] Eighthly, this application also provides an electronic device for transmitting data via a data transmission interface. The electronic device includes: one or more processors; a memory; at least one application program; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method as described in any one of the fourth to seventh aspects above.

[0025] Ninthly, the present application also provides a storage medium including computer instructions, which, when executed on an electronic device, cause the electronic device to perform the method as described in any one of the fourth to seventh aspects above.

[0026] In a tenth aspect, the present application also provides a chip system, comprising: a communication interface for inputting and / or outputting data; and a processor for executing a computer-executable program, causing a device equipped with the chip system to perform the method as described in any one of the fourth to seventh aspects above. Attached Figure Description

[0027] Figure 1 This is a system architecture example diagram for a one-to-one broadcast application scenario;

[0028] Figure 2 This is a schematic diagram of the audio and video data transmission process architecture based on WebRTC technology in related technologies;

[0029] Figure 3 This is a feasible product hardware architecture example of the electronic device provided in the embodiments of this application;

[0030] Figures 4(a)-4(e) These are example diagrams of device architecture for one-to-many broadcast application scenarios;

[0031] Figures 5(a)-5(d) These are example diagrams of one-to-many broadcast application scenarios;

[0032] Figures 6(a)-6(c)These are examples of device architecture diagrams for many-to-one broadcast application scenarios;

[0033] Figure 7 This is a system architecture example diagram for a one-to-two broadcast application scenario;

[0034] Figure 8 This is a system architecture example diagram for adding an audio-visual synchronization module in a one-to-many broadcast application scenario;

[0035] Figure 9 This is a schematic diagram of the latency feedback link between video playback devices and audio playback devices in a one-to-many broadcast application scenario;

[0036] Figure 10 This is a schematic diagram of video frame processing strategies in a one-to-many broadcast application scenario;

[0037] Figure 11 This is a schematic diagram of multiple thread time nodes in the video post-processing workflow;

[0038] Figure 12 This is a system architecture example diagram for a one-to-many broadcast application scenario using the second solution;

[0039] Figure 13 This is a system architecture example diagram for playback synchronization in a many-to-many broadcast application scenario;

[0040] Figure 14 This is a typical system architecture example diagram for a one-to-many broadcast application scenario;

[0041] Figure 15 A flowchart illustrating one embodiment of the synchronized playback method;

[0042] Figure 16 A flowchart illustrating another embodiment of the synchronized playback method;

[0043] Figure 17 A flowchart illustrating another embodiment of the synchronized playback method;

[0044] Figure 18 This is a typical system architecture example diagram for a many-to-one broadcast application scenario;

[0045] Figure 19 A flowchart illustrating yet another embodiment of the synchronized playback method;

[0046] Figure 20 This is a schematic diagram of the system architecture of a playback system in a one-to-many broadcast application scenario. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0049] In data broadcasting applications such as audio and video, asynchronous playback of sound and video can affect the viewing experience and reduce user experience, so it is necessary to maintain audio-visual synchronization.

[0050] Currently, in one-to-one broadcast scenarios, audio and video data can be encoded and transmitted in a single stream to maintain playback synchronization. One-to-one broadcast means that both the source and playback ends consist of only one device; one device (the source) transmits data to the other device (the playback end) for playback. For example, ... Figure 1 As shown, the source end consists of only one device A', and the playback end consists of only one device B'. Device A' transmits audio and video to device B', which then plays the video and audio. In this one-to-one broadcast scenario, the source device encodes and transmits the video and audio in one go, while the playback end decodes and broadcasts them. Because audio and video are encoded and transmitted in one go, their correspondence is one-to-one, meaning the playback timing of audio frames and their corresponding image frames can remain consistent, thus eliminating the asynchrony caused by audio and video transmission delays and differences in audio and video frame processing times.

[0051] Web Real-Time Communication (WebRTC) is a technology that enables web browsers to conduct real-time voice calls or video chats. It provides a solution for real-time audio and video transmission, including audio and video capture, encoding / decoding, network transmission, and display. Audio and video capture is handled by distributed services. For example... Figure 2As shown, WebRTC provides functions including audio and video encoding and decoding, audio and video capture and rendering, transmission, flow control, and audio-visual synchronization. The transmission protocol uses RTP / RTCP (Real-time Transport Protocol) / (Real-time Control Protocol). RTP packages media data, while RTCP handles QoS (Quality of Service) information feedback. The sending end adds redundant information after encoding, and the receiving end uses this redundant information to recover erroneous bits, resolving bit errors during transmission. The flow control module is deployed at the sending end, calculating the appropriate sending rate and encoding bitrate based on the QoS information fed back from the receiving end. The receiving end deploys a jitter buffer to address data packet jitter and out-of-order delivery issues.

[0052] from Figure 2 As can be seen, in audio and video data transmission based on WebRTC technology, on the sending side, audio and video frames are encoded and then packaged using the RTP / RTCP protocol. After smoothing, the audio and video data packets are sent through a single transmission channel. On the receiving side, they are received through another single transmission channel, and then audio and video are decoded separately. In this scheme, audio and video data are transmitted through a single transmission channel.

[0053] In a one-to-one broadcast scenario, the source device and the playback device can transmit audio and video data through a single transmission channel. Since there is only one playback device, it is relatively easy to achieve synchronized playback of audio and video. Therefore, the above-mentioned WebRTC-based solution can be applied to a one-to-one broadcast scenario.

[0054] However, in distributed business scenarios, there are often cross-device broadcasting requirements, such as one-to-many or many-to-one broadcasting. For example, if the playback end includes two or more devices, data transmission from one source device to the other two or more playback devices typically requires separate transmission channels. In such cases, audio and video transmission solutions based on WebRTC technology are unsuitable for cross-device broadcasting or more complex distributed business scenarios.

[0055] In view of this, this application proposes a synchronous playback method that can realize the synchronous playback of streaming media data such as audio and video in cross-device broadcasting application scenarios, thereby solving the above problems.

[0056] The method provided in this application can be applied to various application scenarios where data is transmitted from the source end to the playback end for playback, such as video casting to a display device and / or audio casting to an audio playback device. For example, one typical application scenario of this method is one-to-many casting in a distributed business scenario. Furthermore, this method can also be applied to many-to-one casting scenarios or many-to-many casting scenarios.

[0057] In the embodiments of this application, for clarity, the end that sends data is regarded as the source end, which includes at least one electronic device; the end that receives and plays data is regarded as the playback end, which includes at least one electronic device.

[0058] The device on the source side (e.g., a third device) can be an electronic device capable of storing and transmitting data, such as a mobile phone, a personal computer (PC) (e.g., a laptop), a tablet PC, wearable electronic devices (e.g., smartwatches, bracelets, glasses, collars), a smart speaker, a personal digital assistant (PDA), augmented reality (AR) devices, virtual reality (VR) devices, robots, smart toys, learning machines, etc. The device on the playback side (e.g., the first device, the second device) can be an electronic device that supports playing at least one data format, such as supporting audio and / or video playback. Specifically, it can be a television, a smart screen, a set-top box, a mobile phone, a tablet PC, a personal computer, wearable electronic devices, augmented reality devices, virtual reality devices, learning machines, robots, and various other products that support playing video, audio, and other data formats.

[0059] As shown in Figure 4 Figure 3 An example of a feasible product hardware architecture for an electronic device is shown. The electronic device 300 may include a processor 310, an external memory interface 320, internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, antenna 1, antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, a headphone jack 370D, buttons 390, a motor 393, an indicator 392, a display screen 393, and a subscriber identification module (SIM) card interface 394, etc.

[0060] It is understood that the structure illustrated in the embodiments of the present invention is merely an example and does not constitute a limitation on the electronic device 300. The electronic device in the embodiments of this application may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0061] For example, the electronic device on the source side may include only a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, etc.

[0062] The electronic device on the playback side, taking an audio playback device as an example, may only include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, antenna 1, antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 370A, buttons 390, and a display screen 394. Taking a video playback device as an example, it may only include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, antenna 1, antenna 2, a mobile communication module 350, a wireless communication module 360, and a display screen 394. Optionally, it may also include an audio module 370, a speaker 370A, and buttons 390.

[0063] Specifically, processor 310 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0064] The processor 310 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 310 is a cache memory. This memory can store instructions or data that the processor 310 has just used or that are used repeatedly. If the processor 310 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 310, and thus improves the efficiency of the system.

[0065] In some embodiments, the processor 310 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0066] USB port 330 is a USB standard compliant interface, which can be a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 330 can be used to connect a charger to charge electronic device 300, and can also be used for data transfer between electronic device 300 and other devices, such as data transfer from a source device to a playback device. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0067] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 300. In other embodiments of this application, the electronic device 300 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0068] The charging management module 340 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 340 receives charging input from the wired charger via a USB interface 330. In some wireless charging embodiments, the charging management module 340 receives wireless charging input via the wireless charging coil of the electronic device 300. While charging the battery 342, the charging management module 340 can also supply power to the electronic device via the power management module 341.

[0069] The power management module 341 connects the battery 342, the charging management module 340, and the processor 310. The power management module 341 receives input from the battery 342 and / or the charging management module 340, providing power to the processor 310, internal memory 321, display screen 394, camera 393, and wireless communication module 360. The power management module 341 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 341 may be located within the processor 310. In other embodiments, the power management module 341 and the charging management module 340 may be housed in the same device.

[0070] The data transmission function of electronic device 300 can be realized through antenna 1, antenna 2, mobile communication module 350, wireless communication module 360, modem processor and baseband processor, etc.

[0071] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 300 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0072] The mobile communication module 350 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 300. The mobile communication module 350 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 350 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 350 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 350 may be housed in the processor 310. In some embodiments, at least some functional modules of the mobile communication module 350 and at least some modules of the processor 310 may be housed in the same device.

[0073] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 370A, receiver 370B, etc.) or displays images or videos through a display screen 394. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 310 and may be housed in the same device as the mobile communication module 350 or other functional modules.

[0074] The wireless communication module 360 ​​can provide solutions for wireless communication applications on the electronic device 300, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 360 ​​can be one or more devices integrating at least one communication processing module. The wireless communication module 360 ​​receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signal, and sends the processed signal to processor 310. The wireless communication module 360 ​​can also receive signals to be transmitted from processor 310, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0075] In some embodiments, antenna 1 of electronic device 300 is coupled to mobile communication module 350, and antenna 2 is coupled to wireless communication module 360, enabling electronic device 300 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0076] Electronic device 300 implements display functions through a GPU, a display screen 393, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 393 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 310 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0077] Display screen 393 is used to display images, videos, etc. Display screen 393 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 300 may include 3 or N displays 393, where N is a positive integer greater than 3.

[0078] Electronic device 300 can achieve video playback function through video codec, GPU, display screen 393 and application processor.

[0079] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when the electronic device 300 is selecting a frequency, the DSP is used to perform Fourier transforms on the frequency energy.

[0080] Video codecs are used to compress or decompress digital video. Electronic device 300 may support one or more video codecs. Thus, electronic device 300 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 3, MPEG2, MPEG3, MPEG4, etc.

[0081] Optionally, the NPU (Neural-Network Processor) is a computational processor that, by drawing inspiration from the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, rapidly processes input information and can continuously learn on its own. The NPU can enable applications such as intelligent cognition in electronic devices, for example, intelligent recognition of data types in data streams.

[0082] The external storage interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 300. The external memory card communicates with the processor 310 through the external storage interface 320 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0083] Internal memory 321 can be used to store computer executable program code, which includes instructions. Internal memory 321 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 300 (such as audio data, phonebook, etc.). Furthermore, internal memory 321 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 310 executes various functional applications and data processing of electronic device 300 by running instructions stored in internal memory 321 and / or instructions stored in memory located in the processor.

[0084] Electronic device 300 can implement audio functions, such as music playback, through audio module 370, speaker 370A, receiver 370B, microphone 370C, headphone jack 370D, and application processor.

[0085] The audio module 370 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. The audio module 370 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 370 may be located in the processor 310, or some functional modules of the audio module 370 may be located in the processor 310.

[0086] The speaker 370A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. Electronic device 300 can listen to music or make hands-free calls through the speaker 370A.

[0087] The receiver 370B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 300 answers a telephone call or voice message, the receiver 370B can be brought close to the listener's ear to hear the voice.

[0088] Microphone 370C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 370C, inputting the sound signal into microphone 370C. Electronic device 300 may have at least one microphone 370C. In some embodiments, electronic device 300 may have two microphones 370C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 300 may have three, four, or more microphones 370C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0089] The 370D headphone jack is used to connect wired headphones. The 370D headphone jack can be a USB 330 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0090] Buttons 390 include a power button, volume buttons, etc. Buttons 390 can be mechanical buttons or touch-sensitive buttons. Electronic device 300 can receive button input and generate key signal inputs related to user settings and function control of electronic device 300.

[0091] Motor 391 can generate vibration alerts. Motor 391 can be used to indicate successful data transmission or for playback feedback. For example, different vibration feedback effects can correspond to whether data transmission was successful. Different playback scenarios (e.g., synchronized playback, video ahead of audio, audio ahead of video, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0092] Indicator 392 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0093] The SIM card interface 394 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 394 to make contact with and separate from the electronic device 300. The electronic device 300 can support 3 or N SIM card interfaces, where N is a positive integer greater than 3. The SIM card interface 394 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 394 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 394 is also compatible with different types of SIM cards. The SIM card interface 394 is also compatible with external memory cards. The electronic device 300 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 300 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 300 and cannot be separated from the electronic device 300.

[0094] The following provides exemplary descriptions of typical application scenarios involved in this application.

[0095] One-to-many broadcasting means that the source end includes only one device, while the playback end includes two or more devices. Data is transmitted from the source end to the two or more playback end devices through two or more transmission channels for playback. For example, as shown in Figure 4(a), device A is the source device, and devices B and C are the playback end devices. Device A transmits audio data (sound) to device B through one transmission channel for audio playback, and transmits video data (image) to device C through another transmission channel for video playback.

[0096] The number of devices included in the playback end can be greater than 2. Figure 4(a) only shows the case with 2 devices. In practical applications, the number of devices in the playback end can be 3 or more. For example, as shown in Figure 4(b), the playback end can include device B, device C, and device D. Device A transmits data to devices B through 3 transmission channels respectively. Device B can be a speaker used to play audio. Devices C and D are both displays, used to display different parts of the same frame. The two displays are stitched together to display the complete image. As another example, as shown in Figure 4(c), the playback end can include device C, device D, device E, and device E. Data from one side of device A is transmitted to devices B through 4 transmission channels respectively. Devices C and D are both speakers, used to play the left and right channels respectively. Devices D and E are both displays, used to stitch together to display the complete image in the video frame.

[0097] Furthermore, the data transmitted from the source to the playback end through different transmission channels can be of different types or the same type. For example, as shown in Figure 4(d), both transmission channels transmit video data (images), such as devices B and C being displays used to stitch together a complete image; or they can be used to independently display the same complete image, but placed in different locations within the application scenario for multi-view viewing. As shown in Figure 4(e), both transmission channels transmit audio data (sound), with devices B and C being speakers, one transmission channel transmitting the left channel data and the other transmitting the right channel data. Similarly, the number of playback devices can be three or more, not limited to two.

[0098] In practical application scenarios, as shown in Figure 5(a), in a distributed business scenario, one-to-many broadcasting can be implemented with the source device being a mobile terminal device 501, such as a mobile phone or tablet computer. The playback end includes a display screen 502 and an audio playback device 503, such as a smart screen or television, and the audio playback device can be a speaker, a smart assistant, or a robot with audio playback functionality. For example, the mobile phone transmits audio to the speaker for playback and transmits video to the display screen for playback.

[0099] For example, as shown in Figure 5(b), in a one-to-many projection scenario, the mobile terminal 504 can transmit images to display devices 5051-5054 via four transmission channels. Each display device 5051-5054 displays a portion of the image, and the four display devices are stitched together to display the complete image on a large screen. The relative latency of audio and video playback is only perceptible when it exceeds a certain level. For example, synchronous playback of audio and video allows for a relative latency of -185 to 90 ms. In the case of displaying a large, complete image by stitching together four display devices, since the image played on each device is compared with the images from the other three devices, the asynchrony between images becomes more apparent through comparison. Therefore, the precision of synchronization control in this scenario needs to be higher than the precision of synchronous playback control in the audio and single-channel video scenario. In this scenario where multiple images are stitched together to complete an image, the allowable latency time is shorter. For example, the relative latency time may need to be controlled below the inter-frame interval, such as within 33 ms or 16.7 ms.

[0100] For example, as shown in Figure 5(c) in the one-to-many casting scenario, a smart wearable device 506 can transmit audio to the left channel 507 and right channel 508 of the car audio system. For instance, the smart wearable device could be a smartwatch, transmitting audio to different channels of the car audio system based on a streaming media transmission protocol. This application scenario could involve setting up a one-click casting function key or button on a smartwatch while driving a car, motorcycle, or electric vehicle, allowing the driver to cast audio with a single click and play music through the car speakers. It should be noted that the method provided in this application does not limit the data transmission channel and can therefore be used to solve various distributed service scenarios where smart terminals (such as watches, mobile phones, etc.) cast audio to car audio systems. In contrast, in related solutions, taking Bluetooth transmission as an example, in the connected working mode, multiple CIS (Connected Isochronous Streams) need to belong to the same channel group CIG (Connected Isochronous Groups). To achieve synchronous processing, the CIG assigns different delay parameters to each CIS, allowing them to process synchronously after their respective time periods. CIS within the same CIG have the same timestamp. In connectionless operation mode, multiple BIS (Broadcast Isochronous Streams) must belong to the same broadcast group (BIG). Therefore, this related scheme requires data streams to belong to the same channel group or the same broadcast group, which restricts the channels for data transmission. However, the method provided in this application embodiment is not subject to this restriction.

[0101] For example, as shown in Figure 5(d), in a one-to-many broadcast scenario, the source device can be a smart wearable device 509, and the playback devices, in addition to car audio systems 510 and 511, can also include a mobile terminal 512, such as a smartphone. For instance, the smart wearable device could be a smartwatch. During driving, the smartwatch monitors the user's heart rate and other data, generating a video file that dynamically displays the heart rate data and supports display on the mobile terminal. It also generates a voice prompt accompanying the heart rate data, transmitting the video data to the linked mobile phone in real time for display, and transmitting the voice prompt to the car audio system for playback. For example, when the video displays an abnormal heart rate, the car audio system simultaneously plays a voice announcement, such as "Your heart rate is abnormal; we recommend stopping to rest or seeking help as soon as possible." In this scenario, the control precision for synchronized playback is lower than in the scenario shown in Figure 5(a), and the playback delay of the voice relative to the video can be less than 2 seconds.

[0102] In a many-to-one broadcast scenario, the source end contains different devices, while the playback end contains only one device. Data is transmitted from the different devices at the source end to the playback end through two or more transmission channels for playback. For example, ... Figures 6(a)-6(c) As shown, devices B and C transmit video data (images) and audio data (sounds) to device A via different transmission channels, respectively, for playback. Alternatively, devices B and C transmit images to device A via two separate transmission channels for playback. Or, devices B and C transmit the left and right audio channels to device A via two separate transmission channels for playback. Similarly, the number of devices at the source end can be greater than two, but is not limited to two.

[0103] Optionally, the software environment for implementing the method provided in this application embodiment can be a distributed operating system, such as HarmonyOS. For example, a distributed hardware management framework can be deployed in the distributed operating system, which can be used to provide information management capabilities for the distributed hardware subsystem.

[0104] In the embodiments of this application, the transmission channel can be one or more of various channels that enable point-to-point data transmission based on the transmission medium. For example, optionally, the transmission channel can be a channel, a data link, etc.

[0105] For example, data transmission between devices can be implemented based on one or more wireless projection protocols such as Miracast, DNLA (Digital Living Network Alliance), WiDi (Intel Wireless Display), and AirPlay.

[0106] It should be noted that, in the embodiments of this application, the data to be broadcast can be various data formats that support playback on the playback device, such as streaming media data that is mainly audio and video data. Streaming media data can include one or more of the following data streams: audio stream, video stream, text stream, image stream, animation stream, etc. For example, in some embodiments, the data to be broadcast can be audio and / or video data. For example, broadcasting two or more audio streams, or broadcasting two or more video streams, or broadcasting at least one audio stream and at least one video stream.

[0107] The video data can be one or more of the following video formats: MPEG (Moving Pictures Experts Group), AVI (Audio Video Interleave), WMV (Windows Media Video), QuickTime, RealVideo, Flash, MPEG-4, and newAVI. The audio data can be one or more of the following audio formats: MP3 (Moving Picture Experts Group Audio Layer III), WMA (Windows Media Audio), AAC (Advanced Audio Coding), WAV (WAVE from audio format Sound), and APE (Monkey's Audio).

[0108] The data to be broadcast can also be media files, and is not limited to containing only one type of stream. It may contain video, audio, subtitles, or even multiple video streams, multiple audio streams (multi-channel), dual subtitles, etc.

[0109] In other embodiments, it is possible that the data to be broadcast may be a combination of audio data and / or video data with document, image, and other format data. The following examples mainly use audio data and video data for illustration.

[0110] The synchronous playback method provided in this application, in a one-to-many broadcast scenario, uses the playback time of the streaming media on one of the multiple devices (the first device) as a reference to control the playback of streaming media on other devices, thereby achieving playback synchronization among the multiple devices. The reference playback time can be the actual playback time or an estimated playback time estimated by considering the transmission and decoding delays of the first device. For example, a playback time difference can be obtained based at least on the actual playback time of each data frame on one device and the playback timestamps of each data frame to be played on other devices. This playback time difference is then used to adjust the playback of the data frames to be played on other devices to achieve synchronization. Alternatively, the estimated playback time of each data frame on one device can be used as a reference to obtain the estimated playback time of the data frames to be played on other devices under synchronized conditions. A timestamp corresponding to the estimated playback time is added to each data frame before data transmission, thereby guiding other devices on the playback end to perform playback control according to the estimated playback time to achieve synchronization.

[0111] In a many-to-one scenario, after synchronizing the clocks between multiple source devices and one playback device, the synchronization information is set in the frame header through a reserved interface in the control center via the business layer settings, thus achieving synchronization in the many-to-one scenario.

[0112] The following section explains how to achieve synchronized playback between different devices in a one-to-many broadcasting scenario.

[0113] In this application embodiment, for one-to-many broadcast scenarios, at least two solutions are proposed to achieve playback synchronization between different playback devices. These two solutions can be implemented independently or in combination.

[0114] In the first solution, the actual playback time of one of the playback devices is used as a reference time to control the playback time of other devices to achieve synchronization.

[0115] In the second solution, before sending data, the source end obtains an estimated playback timestamp based on information such as transmission latency and decoding latency fed back by one of the playback devices. Other playback devices can directly compare the estimated playback timestamp with the current time to control playback and achieve synchronization.

[0116] For example, taking a one-to-two broadcast scenario as an example, Figure 7 The system architecture for a one-to-two broadcast scenario is illustrated. This system architecture includes a source end and a playback end. The source end device A performs video encoding on the image frame data stream and audio encoding on the audio frame data stream in two separate paths. Then, the data is transmitted to the playback end devices B and C through two transmission channels. After video decoding on device B, the video is sent to the image display module for displaying the video image through a distributed screen. After audio decoding on device C, the audio is sent to the sound playback module for playing the sound through a distributed audio system.

[0117] In this scenario, audio and video are encoded and transmitted separately through different transmission channels. On the one hand, the data processing steps such as encoding and decoding of audio and video take different amounts of time. On the other hand, the communication status may be different when transmitted through different channels, and network jitter may cause the start time to be more random. Therefore, it is quite difficult to synchronize the playback timestamps between different devices in the case of audio and video separation.

[0118] For example, with Figure 7 Taking the system architecture shown below as an example, let's elaborate on the first solution:

[0119] The PTS (Presentation Time Stamp) corresponding to the audio or video frames obtained after decoding at the playback end is the original timestamp generated by the source encoder, not the actual playback time. Therefore, it is necessary to know the actual playback time of each frame data on one of the devices, and achieve synchronization between multiple devices based on the actual playback time.

[0120] Taking the data streams transmitted through two transmission channels as an example, namely audio data stream and video data stream, in this case, considering that people are more sensitive to images and sounds than to video, after the audio playback device at the playback end receives the audio data stream and buffers it to a certain length, the audio frame can be played normally first. Then, the actual playback time of the audio frame is used as a reference to adjust the playback of the video frame on the video playback device to achieve audio-visual synchronization.

[0121] Specifically, such as Figure 8 As shown, audio-visual synchronization modules are deployed in the distributed hardware subsystems of the source device and the playback device, respectively, and the audio-visual synchronization modules on different devices play different roles.

[0122] In some embodiments, source device A adds a first audio-visual synchronization module, which includes at least a multi-channel management module; playback device B adds a second audio-visual synchronization module, which includes at least a frame rate control module; and playback device C adds a third audio-visual synchronization module, which includes at least a latency feedback module.

[0123] Optionally, since video and audio processing take different amounts of time, a data caching module can be added to the audio-visual synchronization module on the playback end to ensure smooth playback and offset the impact of uneven packet transmission and reception caused by network jitter. In some embodiments, a data caching module may not be included.

[0124] For example, the second audio-visual synchronization module includes a first status feedback module, a first data buffer module, and a frame rate control module. The third audio-visual synchronization module includes a second status feedback module, a second data buffer module, and a latency feedback module.

[0125] In other embodiments, the second audio-visual synchronization module and the third audio-visual synchronization module may not have a data buffer module, or the first state feedback module and the frame rate control module may be integrated into the same module, or the second state feedback module and the latency feedback module may be integrated into the same module. As long as the corresponding functions are achieved, various specific module settings can be obtained through formal modifications, which will not be listed one by one in this application specification.

[0126] Similarly, in specific implementations, the synchronization policy management, multiplexing management module, and status monitoring module in the source device can be integrated into the same module or into two modules. For example, synchronization policy management and multiplexing management can be integrated into a single management module, or the status monitoring module and synchronization policy management can be integrated into the same module, or the three modules of synchronization policy management, multiplexing management, and status monitoring can be integrated into the same module, etc. Other specific implementation methods will not be elaborated in this application specification.

[0127] The synchronization strategy management module determines the synchronization strategy based on the data type to be played.

[0128] The synchronization policy management module allows you to deploy a policy library or set up an interface to access it. The policy library stores synchronization policies corresponding to various data types. Data types include information about the data format to be played on different devices. For example, the data to be played may contain only audio, only video, both audio and video, or other formats in addition to audio and video.

[0129] For latency and stuttering, human hearing is more sensitive than vision. Generally, we try to maintain normal audio output. In some embodiments, for data types that include both audio and video data, the synchronization strategy adopted is audio-centric and video-subordinate. That is, the playback progress of the video frame follows the audio frame, with the audio frame playback as the reference.

[0130] When two or more data streams are video, or two or more data streams are audio, one device can be selected as the master among multiple devices on the playback end, and the other devices as slaves.

[0131] The following examples primarily use one audio data stream and one video data stream as illustrations. For other data types, the corresponding synchronization strategies can be adapted to these examples.

[0132] The video playback device (Device B), through the first status feedback module, identifies the data type of the current data stream to be played as video, monitors the connection status of the transmission channel from Device A to Device B, and feeds back the data type information and connection status information to the status monitoring module in the source device A. Similarly, the audio playback device (Device C), through the second status feedback module, identifies the data type of the current data stream to be played as audio, monitors the current connection status of the transmission channel, and feeds back the data type information and connection status information to the status monitoring module in the source device A.

[0133] The status monitoring module in device A first determines whether to continue data transmission based on the connection status information of the transmission channel fed back by devices B and C. The connection status of the transmission channel refers to its communication status, i.e., whether data can be sent normally. For example, if the connection status information indicates that communication has been lost, then synchronized playback of subsequent data is obviously impossible. In this case, the system can wait for communication to be restored, temporarily pausing playback on each device and notifying the user that the network has been lost or that playback has failed. Once normal communication is restored on the transmission channel, the appropriate synchronization strategy can then be adopted.

[0134] When the connection status indicates normal communication, the status monitoring module updates the data type information of the data streams to be played in real time based on the data type information of the two data streams fed back by device B and device C, and inputs it to the synchronization strategy management module.

[0135] Different data types may employ different synchronization strategies. For example, if one channel is audio and the other is video, or if both channels are video, the specific synchronization strategies may differ. Therefore, the source device must first determine the data type to be played, and then use the synchronization strategy management module to determine the corresponding synchronization strategy.

[0136] After the synchronization strategy is determined, the synchronization strategy management module will trigger the corresponding synchronization strategy. For example, it will trigger the multiplexing management module to receive the actual playback time information of each audio frame sent by device C.

[0137] To reduce the number of data transmissions between the playback device and the source device, device C can proactively send feedback on the actual playback time of audio frames to device A via a latency feedback module. For example, the actual playback time of the m-th audio frame that was just played can be denoted as m_clock and sent to device A.

[0138] After receiving m_clock, device A immediately forwards it to device B.

[0139] Device B, through its frame rate control module, uses the actual playback time m_clock of the received audio frame as a reference time and compares it with the playback timestamp (Vedio Presentation Time Stamp, Vpts) of the corresponding video frame. Based on the comparison result, it controls the playback progress of the video frame.

[0140] Optionally, a first data caching module is used to cache video data, smooth out encoding / decoding latency and network jitter, and the video frames in the queue serve as input to the frame rate control module. A second data caching module caches audio data, smoothing out the impact of encoding / decoding latency and network jitter on playback time.

[0141] like Figure 8As shown, the audio playback device (device C) obtains the actual playback time m_clock of the audio frame through the latency feedback module, and feeds back the actual playback time to the multi-channel management module of the source device A. The multi-channel management module then feeds back m_clock to the frame rate control module of the video playback device (device A).

[0142] like Figure 10 As shown, in one possible implementation, in the audio playback device, the latency feedback module writes the actual playback time of each frame of audio playback into the master clock m_clock. Of course, in other embodiments, the actual playback time of each of the several audio frames can be written all at once after several audio frames have been played, to reduce the latency caused by frequent writing. Alternatively, it can be written every few audio frames, selecting only one audio frame's actual playback time to write; for example, only one audio frame's actual playback time is written out of every two audio frames.

[0143] like Figure 9 As shown, in distributed business scenarios, there is no direct communication channel between different playback devices. The video playback device struggles to obtain the actual playback time information of audio frames from the audio playback device. If the audio and video playback devices were to directly provide latency feedback, a corresponding transmission channel would need to be established, or the source end would need to perform two forwarding operations. For example, the video playback device would first send a request (one forwarding), and then the audio playback device would provide playback time information (a second forwarding). Establishing a new link or performing two forwarding operations would put significant pressure on the link. Therefore, in some embodiments of this application, the latency feedback module of the audio playback device actively sends audio playback time information to the source end, which then forwards this information to other devices. This single forwarding allows other devices to obtain the audio playback time information from the audio playback device, reducing link pressure.

[0144] It should be noted that in other embodiments, it is not excluded that a transmission channel is established between the video playback device and the audio playback device, or that the audio playback device or other devices can obtain the audio playback information through two forwardings at the source end.

[0145] After the video playback device obtains the actual playback time of the audio frame, the frame rate control module retrieves a frame image from the buffer queue, reads the playback timestamp Vpts of the frame image, and calculates the playback time difference diff based at least on Vpts and m_clock.

[0146] like Figure 10 As shown, after the frame rate control module calculates the diff, it determines the processing strategy for the current video frame to be played based on the diff, such as whether to continue waiting, perform frame dropping, or send it directly to the display.

[0147] Optionally, in some embodiments, the frame rate control module retrieves a frame from the buffer queue, which can be done by retrieving a frame at fixed intervals according to a set frame rate. The interval t0 is determined based on the frame rate Fr, where t0 = n * 1 / Fr, and n is an integer equal to or greater than 1. n controls the proportion of video frames retrieved from the buffer queue. If n = 1, it is equivalent to retrieving frames one by one from the buffer queue. The playback time difference diff is calculated for each video frame in the buffer queue to determine whether to send it for display or whether to drop frames, etc.; n = 2 means that one frame is retrieved from every 2 frames in the buffer queue to calculate the playback time difference diff; n = 3 means that one frame is retrieved from every 3 frames in the buffer queue to calculate the playback time difference diff. The larger n is, the lower the precision of the synchronization control, and the synchronization effect may decrease as the value of n increases. In some embodiments, n ≤ 3.

[0148] For example, if n=1 and the frame rate is 30fps (frames per second), then t0=1*1 / 30≈0.033s≈33ms, meaning that one frame is retrieved from the buffer queue every 33ms. This is equivalent to retrieving frames one by one, calculating the playback time difference with the audio frame for each frame, and then determining whether to send it for display. If the frame rate is 60fps, then t0=1*1 / 60≈0.017s≈16.7ms, meaning that one frame is retrieved from the buffer queue every 16.7ms.

[0149] It should be noted that the next video frame retrieved may not originally be intended to be played synchronously with the m-th audio frame received. Therefore, the difference calculation for obtaining the playback time difference should be based on audio and video frames with a mapping relationship. Video and audio frames with a mapping relationship refer to audio and video frames that exhibit synchronized playback timing. Specifically, the mapping relationship can be determined based on at least one of the following: playback timestamp (PTS) and sequence number.

[0150] Because video and audio frame rates are generally different, for example, if device C writes a clock signal for each audio frame played, the frequency of sending the clock signal may be the same as or close to the frame rate. With an audio frame rate of 50fps, a clock signal would be sent every 20ms; for instance, the actual playback time of the (m+1)th audio frame would be m+1_clock. However, in a video playback device, taking a video frame rate of 30fps as an example, a frame is retrieved from the buffer queue every 33ms. Therefore, the frequency of retrieving video frames and the frequency of receiving the clock signal are different.

[0151] The comparison operation to determine whether a mapping relationship exists can be triggered by retrieving video frames. That is, for each retrieved video frame, a mapping relationship is identified with the latest received clock, for example, by reading the sequence number and determining whether a mapping relationship exists based on the sequence number. This requires adding not only the PTS (Pulse Test Set) but also the sequence number to the source device.

[0152] As one possible implementation, in addition to adding a sequence number or PTS, a sequence number or PTS that has a mapping relationship with the current audio / video frame or audio frame can also be recorded. That is, in some embodiments, for each data frame or every few data frames, the sequence number or PTS of other data frames that have a mapping relationship with the current data frame can be recorded. In this way, at the playback end, there is no need to perform difference calculation; the sequence number or PTS with the mapping relationship can be read directly to determine whether there is a mapping relationship.

[0153] For example, on device A, before sending the video data stream, for each video frame, in addition to adding the Vpts, a sequence number is added, along with information about other sequence numbers that have a mapping relationship with it. This can be added in the form of an array. For example, if the sequence number of the current video frame is 80, and the sequence numbers of at least one audio frame that has a mapping relationship with it are 98 and 99, then the added sequence number array can be [80:98,99], where 80 represents the sequence number of the video frame itself, and 98 and 99 represent the sequence numbers of the two audio frames that have a mapping relationship with this video frame. At the playback end, after the audio data stream packets and video data stream packets are decoded, the sequence number or sequence number array corresponding to each data frame can be read.

[0154] Furthermore, as another possible implementation, the PTS (Playback Time Stamp) is used to represent the playback timing and is itself a type of sequence information. Therefore, it is possible to determine the mapping relationship between audio frames and video frames using the playback timestamp PTS. The source end encodes the data before sending it, and most encoders generate a playback timestamp PTS for each frame during encoding. This timestamp is used to mark the playback order of each frame, and based on this playback order, the mapping relationship between video frames and audio frames can be determined. That is, the mapping relationship is determined based on the Vpts (Video Presentation Time Stamp) and Apts (Audio Playback Timestamp).

[0155] For example, before sending data, the source device adds a playback timestamp (PTS) to each data frame. Generally, because the video frame rate and audio sampling rate are different, the video frame and audio frame are not in a one-to-one mapping relationship. Assuming that the audio is sampled at 20ms per frame and the video frame rate is 30fps, that is, one frame is generated every 33ms, and assuming that the timestamp of the audio start sampling time is 80, then the approximate increasing pattern of the PTS of video and audio frames is as follows: Video frame VPTS: 109, 142, 175, 208, 241, 274...; Audio frame APTS: 80, 100, 120, 140, 160, 180... If at the playback end, it can be determined that the clock frequency of the PTS is consistent with the time reference (e.g., Greenwich Mean Time), then the mapping relationship between the audio frame and the video frame can be determined directly by calculating the difference in timestamps. Alternatively, based on the playback order indicated by the playback timestamp, combined with information such as time base or clock frequency (crystal oscillator frequency), the desired playback time information can be obtained. Based on the desired playback time information, the mapping relationship between audio frames and video frames can also be determined.

[0156] For example, audio frames and video frames whose PTS difference is less than a predetermined mapping threshold are considered to have a mapping relationship. For instance, the specified threshold could be 33. In the sequence above, the first audio frame has a PTS of 80, and the first video frame has a PTS of 109; the difference between them is less than 33, so a mapping relationship can be considered between them. The mapping threshold is not unique; for example, in a video frame rate of 60fps, the mapping threshold could be 20.

[0157] It should be noted that in this embodiment, device C needs to simultaneously report the original Apts of the audio frame along with the actual playback time clock. That is, the Apts are carried within the information used to send the clock. In this way, the video playback device can first read the Apts of the audio frame, compare the Vpts with the Apts, and if the difference between the Vpts and Apts is less than 33ms, then a mapping relationship is considered to exist between them. For example, after conversion to time information, if the difference is less than 33ms, then a mapping relationship is considered to exist.

[0158] In practical applications, it's possible that the next video frame retrieved from the buffer queue may not have a mapping relationship with the latest received m_clock. In this case, the sequence number or PTS can be used to determine the order of the audio frames mapped to the currently retrieved video frame and the currently received m_clock. Both the sequence number and PTS can express the playback order. Therefore, by comparing the sequence numbers or PTS of video and audio frames, even if the mapping relationship cannot be determined, the order can be identified. If the currently received m-th audio frame should be played before the currently retrieved video frame (i.e., the audio frame that should be played synchronously with the current video frame is after the current m-th audio frame), then the video playback device can wait to receive the subsequent audio frames. If the m-th audio frame should be played after the currently retrieved video frame, then the audio frame mapped to the current video frame can be searched among the multiple received clocks, its actual playback time can be read, and then the difference can be calculated.

[0159] It should be noted that PTS is added at the source end to represent the relative order of playback of each data frame, but it is not the actual playback time. Therefore, Vpts and Apts can be used to determine mapping relationships, but cannot be directly used for synchronization control.

[0160] It should be noted that the above serial number is different from the Presentation Time Stamp (PTS) and should not be understood as an equivalent technical concept.

[0161] Thus, assuming there is a mapping relationship between the nth video frame and the mth audio frame retrieved from the buffer queue, the following explains how to calculate the playback time difference (diff). All calculations below are based on the mapping relationship between the audio and video frames; for the sake of brevity, these details will not be repeated.

[0162] Optionally, as a possible implementation, in some embodiments, the playback time difference diff can be determined based on Vpts and m_clock, for example:

[0163] diff = Vpts - m_clock

[0164] Here, m_clock represents the actual playback time of the m-th audio frame. Each time an audio frame is played, its timestamp is written to the master clock. Vpts is the playback timestamp of the n-th frame retrieved from the video buffer queue.

[0165] Optionally, as another feasible implementation, in another part of the embodiments, the determination of the playback time difference diff may also incorporate factors such as post-processing duration.

[0166] For example, diff can be calculated using the following formula:

[0167] diff = Vpts - m_clock + (Y3 - X3)

[0168] Where X3 is the post-processing duration (second processing duration) for the video, and Y3 is the post-processing duration (first processing duration) for the audio.

[0169] For example, X3 can represent the time required for a single image frame to be drawn, rendered, composed, and displayed. In other words, it represents the total execution time of multiple threads involved in drawing, rendering, image frame composition, and image frame display for a single image frame. This time can be obtained by calculating the time consumed by each thread node. For instance, X3 = T1 + T2 + T3 + T4, where T1 is the drawing time, T2 is the rendering time, T3 is the image frame composition time, and T4 is the image frame display time. An image frame is a video frame.

[0170] In this embodiment of video playback via WMS (Window Manager Service), WMS provides image rendering services, etc. X3 can be understood as the time from image drawing and rendering to final display after the image is sent to WMS. X3 can be obtained by accessing WMS.

[0171] For example, Y3 can be understood as the time from when audio data is sent to the audio framework, processed by the audio driver, to when it is finally played out. This can be obtained by calculating the time consumed by each thread during audio playback. Specifically, as one possible implementation, Y3 can be obtained by accessing the audio framework.

[0172] For example, Y3 = T5 + T6 + T7 + T8 + T9 + T10, where T6 to T10 represent the mixing process, sound effect processing, volume processing, sending to the driver, sending to the DSP (Digital Signal Processor), and the duration of playback by the speaker, respectively.

[0173] For example, the duration of a single thread can be obtained as follows: When the thread starts running, the system time is obtained as the start time. After the thread finishes running, the system time is obtained again at the end of the thread's execution. This time minus the start time is the computation time. A start time and an end time are recorded, and the difference between the two is the program running time. In this embodiment, the post-processing of video and audio may involve multiple threads. Optionally, the start time of the first thread executing its corresponding operation can be used as the start time. After the last thread finishes executing, the system time is obtained again, and the difference between the two is the running time of multiple threads, which is used as the post-processing duration. For example, when the drawing thread starts executing the drawing action, the system time is recorded as the start time, and then the time when the image frame begins to display is recorded as the end time. The difference between the two is X3, for example, as shown below. Figure 11 As shown, X3 = t3 - t1.

[0174] It should be noted that due to differences in operating systems and hardware devices, the workflows for audio and video post-processing may differ. The above is merely an illustrative example. For instance, rendering is not a necessary step in video post-processing; when the device hardware is powerful enough, rendering is unnecessary. Similarly, sound effects processing and volume control are optional in practical applications.

[0175] In application scenarios where the processing flow differs from the examples above, the post-processing duration can be understood as the length of time from when a video or audio frame is determined to be sent for broadcast until it is actually played. As one possible implementation, the system time can be recorded when broadcast is determined as the start time, and the start of playback can be used as the end time to obtain Y3 or X3.

[0176] When the exact broadcast time is difficult to determine, a more universally applicable method for determining the post-processing duration is to retrieve a video or audio frame from the buffer queue as the start time and use the actual playback of the video or audio frame as the end time, thus obtaining X3 or Y3. In other words, the post-processing duration (e.g., the first processing duration or the second processing duration) can be an estimate of the processing time from when a data frame is retrieved from the buffer queue to when it is played.

[0177] It's important to note that Y3 and X3 can be constants determined by averaging multiple played video or audio frames. These can be written into the program as fixed parameters and do not need to be calculated before each frame is played during actual operation. For example, in the SRS (simple RTMP Server) 4.0 code architecture, Y3 = 40ms. When calculating the diff, Y3 is a fixed constant of 40ms.

[0178] It should be noted that the above implementation of determining X3 and Y3 is only an example. More generally, X3 can be the time length determined by taking the video frame retrieved from the buffer queue as the start time and playing it as the end time, and Y3 can be the time length determined by taking the audio frame retrieved from the buffer queue as the start time and playing it as the end time.

[0179] In this embodiment, the timestamps used to calculate the diff, such as Vpts or Apts, can be converted into time. Due to differences in encoding and decoding formats, in some encoding and decoding formats, the timestamps Vpts or Apts may be represented as a series of values ​​to inform the playback end of the relative playback order of each frame, and cannot directly represent time. Therefore, it is necessary to convert the timestamps into time and then use the above calculation formula to obtain the diff.

[0180] For example, the PTS (Person-Time Stamp) of multiple video frames in a video might be: 480 640 560 520 600 800 720 680 760 960. These values ​​do not represent time. In FFmpeg, the PTS needs to be converted into time information based on the respective time bases (time_base) of the audio and video before the calculation is performed using the formula described above. For instance, the playback time Tv of an audio frame is calculated as: Tv = time_base * PTS, where time_base represents the time scale corresponding to the value 1 in the PTS.

[0181] After obtaining the diff, a playback control strategy is determined based on the diff for the currently retrieved video frame. Playback control strategies may include sending the frame to the display, waiting, dropping frames, reducing playback speed, or increasing playback speed.

[0182] When video and audio are played together, if the playback time difference is within a certain threshold, the human senses will not noticeably perceive the asynchrony, and therefore the stuttering will not be obvious, thus not degrading the audiovisual experience. However, if the playback time difference exceeds a certain threshold, more noticeable stuttering may occur, reducing the audiovisual experience. In this embodiment, a certain delay is allowed for audio and video playback synchronization, and the delay must be within an acceptable range.

[0183] In this embodiment of the application, the playback control strategy for the current video frame is determined by comparing the diff with a predetermined threshold range (first threshold interval).

[0184] For example, setting min as the maximum threshold for audio lagging behind video and max as the maximum threshold for audio leading video, an optional and feasible playback control strategy could be:

[0185] If diff < min, it means the video frame is faster than the audio frame. At this time, the video frame needs to wait for a few milliseconds to wait for the master clock. For example, the waiting time can be (min - diff) milliseconds.

[0186] If diff > max, it means the video frame is slower than the audio frame. At this time, the video frame needs to catch up. The catching-up strategy can be discarding video frames, or increasing the playback rate, etc.

[0187] If min ≤ diff ≤ max, it means within the synchronization threshold range, no obvious asynchrony will occur, and the video frame can be directly sent for display.

[0188] Optionally, the value of min can specifically be -190ms to -180ms, and the value of max can be 85ms to 95ms.

[0189] The following lists a specific embodiment to exemplarily illustrate how to achieve playback synchronization between different devices at the playback end in a one-to-many casting scenario by using the first solution.

[0190] Combined with Figure 10 and Figure 8 As shown, device B and device C at the playback end respectively feedback the data type information of the data to be played to the status monitoring module in the source device A through the first status feedback module and the second status feedback module. For example, the data type feedback by device B is video, and the data type feedback by device C is audio.

[0191] The status monitoring module in device A inputs this data type information into the synchronization policy management module, and the synchronization policy management module selects the corresponding synchronization policy from the policy library according to this data type information.

[0192] In an actual distributed service scenario, due to the increase in the number of devices at the playback end and the source end, the combination of data types is diverse. The setting of the synchronization policy management module can achieve the selection of different synchronization policies for different data types, so it can be applied to more complex application scenarios.

[0193] Optionally, when the data types of multiple transmission channels include at least one audio channel and at least one video channel, one of the audio channels is the main one and the others are the secondary ones. Exemplarily, when the data type to be played currently is one audio channel and another video channel, the synchronization policy adopted is to perform master-slave setting, set the audio as the master, the video as the slave, and the video performs frame rate control with reference to the audio playback time.

[0194] Optionally, when the data types of the multiple transmission channels include multiple channels of the same type, such as two channels being video or two channels being audio, one channel can be selected as the master, which is equivalent to selecting one of the playback devices as the master and the other devices as slaves.

[0195] Optionally, when there are multiple playback devices, the audio frames are cached for a fixed number of frames and then played automatically, eliminating the need for video and audio to wait for each other. In this embodiment, the audio frame caching time can be set to be greater than that in a single-device scenario, i.e., greater than the caching time in a one-to-one broadcast application scenario. For example, if the caching time for a single playback device is 2 minutes, then the caching time of the data caching module in a one-to-many broadcast scenario should be set to be greater than 2 minutes.

[0196] After determining the synchronization strategy, the multi-channel management module listens to the actual playback time information m_clock sent by the latency feedback module of device C, and forwards the actual playback time information to the frame rate control module of device B as the input for video playback frame rate control.

[0197] In this embodiment, the video frame rate is 30 frames / s. The frame rate control module retrieves a frame image corresponding to the current audio frame from the buffer queue every 33ms, reads its playback timestamp Vpts, for example, reads the playback timestamp Vpts of the m-th video frame from the buffer queue, and then calculates diff based on the following formula:

[0198] diff = Vpts - m_clock + (Y3 - X3)

[0199] After calculating the playback time difference diff, the diff is compared with a predetermined threshold range to determine whether the current video frame should be sent for display.

[0200] The higher the frequency of retrieving images from the buffer queue, the higher the accuracy of synchronization control. Calculating the diff frame-by-frame improves the accuracy of synchronization control; a smaller diff results in lower relative latency and better synchronization. The accuracy of synchronization can be controlled by adjusting the frequency of retrieving image frames from the buffer queue. A diff smaller than the inter-frame interval, or even close to zero, enables inter-frame synchronization.

[0201] In this embodiment, min is set to -185ms and max is set to 90ms.

[0202] If diff < min, then wait. For example, if diff is -205ms, then wait -185ms - (-205)ms = 20ms before displaying.

[0203] If -185ms ≤ diff ≤ 90ms, the image is sent directly to the display driver. Sending the image to the display driver means passing the captured frame to the display screen for display.

[0204] If diff > 90ms, the video needs to catch up with the audio. In this case, the control strategy can be to drop frames or increase the video playback rate.

[0205] Optionally, to ensure the continuity of playback data and avoid issues such as swallowing sound or abrupt changes in the picture, try to avoid using frame dropping for synchronization. Instead, consider using a playback speed variation method for synchronization.

[0206] For example, optionally, if a video frame arrives too late (greater than 90ms) and a delayed frame arrives at the same time, the frame can be dropped and the next frame can be displayed directly; or, if the video frame arrives more than 185ms early and waits for the next scheduling, but this may cause the buffer to be full, then the frame needs to be dropped. Frame dropping prioritizes P-frames (predictive-frames) and B-frames (bi-directional interpolated prediction frames).

[0207] When there is a delay in the arrival of audio frames or when the audio is played slowly, video frames are allowed to be displayed earlier, with a maximum advance margin of 185ms.

[0208] Alternatively, if the audio arrives too late, all audio frames in the queue will have been sent, inevitably resulting in intermittent sound. To ensure a good user experience and prevent intermittent sound, when the audio queue data volume is less than a certain value, the audio playback rate is reduced, and the waiting time for audio frames is increased. Once the sending queue exceeds a certain value, normal playback speed is restored, minimizing frame drop. Slow playback is implemented without significantly affecting the user's perception; the specific slowdown ratio is determined based on the diff value.

[0209] Frame dropping results in information loss, so fast or slow playback is used for synchronization, which preserves more information and reduces data loss compared to frame dropping. However, in some situations, frame dropping is the only option.

[0210] Optionally, if the audio buffer queue is full and the first video frame has not yet arrived, the audio frame can only be dropped, or the playback failure will be returned.

[0211] The above solution uses the actual playback time of the audio playback device as a reference to control the playback time of other devices such as video playback devices. The following describes another solution for achieving synchronized playback across multiple devices in a one-to-many casting application scenario.

[0212] Synchronizing video frames with audio frames requires the video playback device to accurately know the playback time of the audio frames, thereby adjusting its own playback timing. In a single-device scenario, the video playback module can easily obtain the audio playback time information from the same device. However, in scenarios where audio and video are separate, the video playback device cannot obtain the playback time of the audio frames in real time. If the video frame obtains the playback time of the audio frame across devices, it needs to go through two cross-device forwardings at the source end, resulting in significant latency and unreliability. In the first solution mentioned above, the source end needs to perform one forwarding. However, in the second solution for a one-to-many scenario provided in this application embodiment, the source end does not need to perform forwarding; instead, the audio playback device only needs to provide a latency feedback to the source end, and the source end does not need to provide this latency feedback to the video playback device.

[0213] Specifically, also with Figure 7 The system architecture shown is used as an example for illustration. This solution can be implemented based on a control center deployed on both the source and playback devices. Specifically, as shown... Figure 12 As shown, for clarity, source device A has a first control center, audio playback device C has a second control center, and video playback device B has a third control center.

[0214] The first step is to synchronize the clocks of devices A, B, and C.

[0215] Performing clock synchronization first can offset or reduce asynchrony caused by different device clocks.

[0216] The audio playback device (Device C) determines the latency information of the audio frame and feeds it back to Device A. The latency information can be the transmission latency and decoding latency of the previous audio frame being played, or the average transmission latency and decoding latency of multiple audio frames that have been played, and is fed back to the first control center in the source device A.

[0217] In source device A, before encoding the video stream, a sequence number and a playback timestamp Vpts are added to each frame. After encoding by the encoder, the first control center estimates the playback time of the video frame based on the latency information of the audio frame, and obtains the estimated playback time Vcts' of the video frame. This estimated playback time is used to replace the original playback timestamp Vpts to offset the latency difference between the video frame and the audio frame, thereby achieving synchronized playback.

[0218] It should be noted that the video playback timestamp (Vpts) or audio playback timestamp (Apts) is used to guide the playback device to play the video in the order indicated by the timestamp. It is generated by the source device and is usually added to each frame of data by the source device before or during encoding.

[0219] The first control center first calculates Vcts' by combining the latency information fed back by the second control center, and then updates the playback timestamp of each video frame in the video stream to be transmitted to Vcts' before sending the video stream.

[0220] The following is a specific example to illustrate the point.

[0221] When the playback device is a single device, the video is synchronized with the audio on the same device, and the audio playback time (Acts) can be obtained. However, when the audio and video playback are on different devices, the video playback device cannot obtain the audio playback device's Acts. The audio playback time (Acts) can be understood as an estimate of the playback time.

[0222] In a distributed scenario, multiple devices first synchronize their clocks. Source device A transmits the video and audio data to be transmitted to playback devices B and C, respectively.

[0223] After a certain number of audio frames are buffered, they are played according to the set frame rate.

[0224] In some embodiments, the audio playback device C determines the latency information Tasd and then feeds it back to the device A.

[0225] Ignoring other negligible factors, the audio transmission and decoding latency (Tasd) is related to the audio playback time (Acts) and the audio transmission time as follows:

[0226] Tasd = Acts - Tas.

[0227] If the video playback device at the playback end cannot know Acts, the source device A can obtain the transmission time point Tas of the audio frame at the source end. Then, based on Tas and Tasd, the estimated value of Acts can be obtained.

[0228] Acts = Tas + Tasd

[0229] Acts is the estimated playback time of the audio frame at the source end, Tas is the transmission time of the audio frame at the source end, and Tasd is the audio frame transmission delay.

[0230] For example, in this embodiment, the audio frame transmission delay Tasd includes transmission delay and decoding delay, and can be averaged by the sum of the transmission delay and decoding delay of multiple audio frames played by the audio playback device over a period of time.

[0231] In other embodiments, the audio playback device may also be able to obtain Tas after determining the latency Tasd, and then calculate Acts on the audio playback device C, and then the device C directly feeds Acts back to the device A, instead of feeding back the latency information to the source device.

[0232] Tasd is a historical result and may not perfectly match the latency generated under current network conditions, therefore its reliability is not 100%, which may result in some errors in audio-visual synchronization. Optionally, the source device can perform filtering processing, filtering the Tasd values ​​of multiple audio frames to obtain the smoothest value, thereby reducing large errors caused by local fluctuations.

[0233] The playback time difference between video frames and audio frames with a mapping relationship can be calculated using the following formula:

[0234] diff=(Vcts–Acts)–(Vpts–Apts)

[0235] Vcts is the video frame delivery time, which can be understood as an estimate of the video frame playback time. Apts and Vpts are the playback timestamps that device A adds to the audio and video respectively before encoding.

[0236] There is a mapping relationship, meaning that video frames and audio frames have the same or similar sequence numbers. Alternatively, if the playback timestamps determine that the audio and video frames are played in the same time sequence, then the corresponding audio and video frames have a mapping relationship.

[0237] The goal of synchronized playback is: min≤diff≤max. In this embodiment, min is -186ms and max is 88ms.

[0238] Source device A obtains the transmission times of the current audio frame and video frame as Tas and Tvs, respectively;

[0239] Ignoring the error introduced by the current frame information, Acts = Tas + Tasd. Substituting this into the above calculation formula, the goal of synchronized playback is:

[0240] min≤Vcts-(Tas+Tasd)-(Vpts-Apts)≤max

[0241] Assuming that when diff = 0, Vcts = Vcts', that is, Vcts' - (Tas + Tasd) - (Vpts - Apts) = 0ms, then

[0242] Vcts'=Tas+Tasd+Vpts–Apts;

[0243] Among them, Apts and Vpts can be obtained by the source device A.

[0244] Vcts' is the estimated video playback time when diff=0.

[0245] In some embodiments, the playback timestamp Vpts of the current frame can be updated to Vcts' before device A sends a video frame.

[0246] In another embodiment, before device A sends video frames, Vcts' can be added to the original playback timestamp Vpts. This adds an estimated value instead of replacing the original Vpts. This avoids a series of problems caused by deleting the original Vpts.

[0247] After decoding, the video playback device (Device B) only needs to compare Vcts' with the current time to decide whether to send the video for display. For example, if the current time is Tn, and diff' = Vcts' - Tn, then by comparing diff' with [-186ms, 88ms], the corresponding processing strategy can be obtained, as detailed in the above embodiment, which will not be repeated here.

[0248] Optionally, in the second solution, X3 and Y3 can also be introduced to determine the playback time difference. The corresponding calculation formula is as follows:

[0249] diff=(Vcts–Acts)–(Vpts–Apts)+(Y3–X3)

[0250] So

[0251] Vcts'=Tas+Tasd+Vpts–Apts+(X3–Y3).

[0252] Similarly, in this embodiment, when there are multiple playback devices, the audio frames are cached for a fixed number of frames and then played automatically. The video and audio no longer wait for each other, and the caching time can be set to be longer than in a single-device scenario.

[0253] The above descriptions are mostly exemplified by cases where at least one channel is audio and at least one channel is video. Those skilled in the art can adapt these descriptions to solutions for one-to-many broadcast scenarios where all data streams to be transmitted are audio or video. For example, as shown in Figure 4(d), in an application scenario where both data streams are images, the playback time of one image stream can be used as a reference to control the playback time of the other device. For instance, according to Solution 1, the actual playback time of each image frame on device C can be used as a reference time to control the playback time of the corresponding image frames on device B; or, according to Solution 2, the playback time of the image frames on device B can be estimated based on the transmission and decoding delay of device C, and the playback timestamp can be updated to control the playback time of the images on device B, achieving synchronization between device B and device C. Similarly, for the scenario shown in Figure 4(e), corresponding solutions can also be adapted, which will not be elaborated upon in this application.

[0254] The above describes the synchronous playback control scheme for one-to-many scenarios. The following section will elaborate on the synchronous playback control scheme for many-to-one scenarios.

[0255] In scenarios where the source consists of multiple devices and the playback device is a single device, for example... Figure 13 In this scenario, device A sends audio, device B sends video, and the playback end is only device C. On device C, the video receiver can accurately obtain the audio playback time, but since the source devices are located on two separate devices, synchronization information between them cannot be easily obtained, requiring cross-device interaction.

[0256] In a many-to-one scenario, the audio-visual synchronization process requires the distributed hardware framework to first set up synchronization group information, and then the control center to synchronize the synchronization group information among multiple devices before data transmission and playback.

[0257] First, the three devices need to synchronize their clocks; that is, devices A and B must synchronize with device C. After clock synchronization, Apts and Vpts are added to the frame header information of devices A and B.

[0258] Adding timestamps after clock synchronization can reduce asynchrony caused by inconsistent device clocks.

[0259] Whether audio and video on different devices need to be synchronized cannot be determined by the devices themselves and requires configuration at the business layer.

[0260] A control center is set up at the business layer. Interfaces are reserved in the control centers corresponding to devices A, B, and C, and these interfaces are used to receive synchronization group information. The synchronization group information may include at least one of synchronization reference information and flow information.

[0261] Optionally, the synchronization reference information may include the identity information of each source device and the destination (playback) device, such as the identity information of source device A and device B and the identity information of playback device C.

[0262] Identity information can be codes, symbols, etc. that can uniquely identify a device. For example, identity information can be the device's Device ID, such as the IMEI (International Mobile Equipment Identity) code of a mobile phone.

[0263] Optionally, the stream information can include information about the data type currently being transmitted, such as whether the data type being transmitted is audio or video.

[0264] After the source device generates the synchronization group information, at least two devices on the source device set the synchronization group information to the local control center, and the source device synchronizes the synchronization group information to the control center of the playback device.

[0265] This section focuses on synchronizing the synchronization group information between multiple source devices and playback devices. There is no explicit time limit for the synchronization between source devices and between the source and playback devices.

[0266] Next, the playback device selects and retains data content that matches its own device identity information based on the identity information of the receiving device in the synchronization group information. Then, it filters out the data type sent by each device, determines the data type sent by each device at the source end, and then groups and divides the different devices at the source end according to the data type, and then performs audio-visual synchronization.

[0267] The following is a specific implementation example of a many-to-one broadcasting scenario.

[0268] For example, in a distributed business scenario, the control center of each device at the source and playback ends reserves the interface SetSyncGroup(vector<int>). <pair<deviceIDs,streamInfo> >) to receive synchronization group information.

[0269] Optionally, synchronization group information can be set in the frame header.

[0270] The first member of the vector is the synchronization reference, which is used to notify the control center of the current device ID of the sending end (i.e., the source end) and the device ID of the playback end after determining the device ID of the source end and the device ID of the playback end.

[0271] The second member is the stream information, which is used to notify the control center whether the data being sent is audio or video data.

[0272] Next, devices A and B set the synchronization group information to their local control center, and then devices A and B synchronize the synchronization group information to the control center of device C.

[0273] Synchronization of synchronization group information enables multiple devices at the source end to interact with each other before sending data, so as to know the data types and other information that each other needs to send, establish a mapping relationship between audio frames and video frames, and achieve synchronization of audio frames and image frames by synchronizing synchronization group information with the playback device.

[0274] Device C identifies the Device ID of the receiving end (playback end) based on the synchronization reference information in the synchronization group information. If the Device ID matches its own Device ID, it determines that the data belongs to itself and retains it.

[0275] Device C filters out the data type sent by each device based on the stream information in the synchronization group information, for example, such as... Figure 13 As shown, device C determines that device A is sending audio data and device B is sending video data, so it groups device A and device B and synchronizes the audio and video between them.

[0276] Optionally, since the receiving end only contains device C, device C can directly obtain Acts. In this case, the video data can be referenced to the audio data for processing according to min≤Vcts-Acts-(Vpts-Apts)≤max.

[0277] Similar to one-to-many broadcasting scenarios, those skilled in the art can adapt solutions to situations where the multiple data streams to be transmitted are all audio or all video, based on the above description. This application specification will not elaborate on these solutions one by one.

[0278] Based on the above implementation schemes for one-to-many and many-to-one scenarios, an implementation scheme for many-to-many scenarios can be adapted. For example, in the case where there are multiple source devices and multiple playback devices, the transmission time difference Δt between the multiple source devices can be determined first. When calculating the diff or estimating the video frame playback time, Δt can be introduced for correction, thereby offsetting or reducing the impact of asynchronous transmission times of the source devices on the playback device.

[0279] Based on the description of the above embodiments, it can be seen that the embodiments of this application provide a synchronous playback method in a distributed business scenario. This method can be applied to one-to-many broadcasting scenarios, such as... Figure 14As shown, in a one-to-many broadcast scenario, the playback end includes two or more devices, including a first device and a second device, and the source end includes a third device; the third device transmits a first data stream to the first device for playback and transmits a second data stream to the second device for playback; the first data stream includes multiple first data frames; the second data stream includes multiple second data frames.

[0280] like Figure 15 As shown, from the perspective of the second device, the method includes the following steps:

[0281] S141, Receive the first timestamp;

[0282] S142, Read the second timestamp corresponding to the second data frame to be played;

[0283] It should be noted that the step numbers are only for distinguishing different steps and do not restrict the execution order. For example, there is no explicit order of execution for S141 and S142. In practical applications, S142 may be executed before S141.

[0284] S143, calculate the first playback time difference based at least on the first timestamp and the second timestamp;

[0285] S144, based on the first playback time difference, determine the synchronous playback strategy to be executed for the second data frame to be played.

[0286] It should be noted that the first timestamp represents the actual playback time of the m-th first data frame on the first device, where m ≥ 1 and is an integer; the second timestamp is the playback timestamp PTS generated by the source third device, which is used to represent the playback sequence of each second data frame in the second data stream.

[0287] As described above, the first and second data streams can be of the same type or different types. For example, the first data stream can be an audio data stream, and the second data stream can be a video data stream. Alternatively, both the first and second data streams can be video data streams, for example, the first and second data streams can represent different parts of an image; or both the first and second data streams can be audio data streams, for example, the first data stream can be the left channel data, and the second data stream can be the right channel data, and so on.

[0288] Furthermore, there can be three or more playback devices, and the number of data streams is not limited to two. The processing method for three or more data streams can refer to the processing method for two data streams. For example, in the case of two data streams, the second data stream refers to the first data stream. Then, for three or more data streams, all data streams except the first data stream can refer to the playback time of the first data stream.

[0289] Optionally, receiving the first timestamp can be receiving a first timestamp forwarded by a third device; the first timestamp is generated by the first device and sent to the third device. By forwarding through the third device once, compared to establishing a data link between the second and first devices or forwarding twice through the source device, the link communication pressure can be reduced, and communication resources can be saved.

[0290] Optionally, the second timestamp corresponding to the second data frame to be played can be read by retrieving the next second data frame to be played from the buffer queue at predetermined time intervals and reading the second timestamp. The predetermined time interval is as described in the above embodiments, and the time interval should be determined based on the frame rate and be an inverse multiple of the frame rate.

[0291] Optionally, the first playback time difference can be calculated based at least on the first timestamp and the second timestamp. If it is determined that there is a mapping relationship between the next second data frame to be played and the m-th first data frame, the first playback time difference can be calculated based at least on the first timestamp corresponding to the m-th first data frame and the second timestamp corresponding to the next second data frame to be played.

[0292] Optionally, the first playback time difference can be calculated based at least on the first timestamp and the second timestamp, and can also be calculated based on the first timestamp, the second timestamp, the first processing duration, and the second processing duration; wherein, the first processing duration is an estimated value of the processing time occupied from when a first data frame is retrieved from the buffer queue to when it is played; the second processing duration is an estimated value of the processing time occupied from when a second data frame is retrieved from the buffer queue to when it is played.

[0293] Optionally, the synchronous playback strategy includes at least one of waiting, displaying, and catching up. Based on a first playback time difference, determining the synchronous playback strategy for the second data frame to be played can involve comparing the first playback time difference with a predetermined first threshold interval; if the first playback time difference is less than the minimum value of the threshold interval, determining that the playback of the first data stream lags behind the second data stream, and determining the synchronous playback strategy for the second data frame to be played as waiting; if the first playback time difference is greater than the maximum value of the threshold interval, determining that the playback of the first data stream is ahead of the second data stream, and determining the synchronous playback strategy for the second data frame to be played as catching up; if the first playback time difference is within the threshold interval, determining that the playback of the first data stream and the playback of the second data stream are relatively synchronized, and determining the synchronous playback strategy for the second data frame to be played as displaying.

[0294] Optionally, waiting includes one or more of reducing the playback rate, waiting for a first predetermined duration, and dropping frames; wherein the first predetermined duration is determined based on the difference between the minimum value in the threshold interval and the first playback time difference; catching up includes increasing the playback rate and / or dropping frames.

[0295] This application also provides a synchronous playback method in a distributed business scenario. This method can be applied to a one-to-many broadcast scenario. The introduction of the one-to-many broadcast scenario is as described above and will not be repeated here.

[0296] This method can be executed on a third-party device; see [link / reference]. Figure 16 As shown, the specific process can be as follows:

[0297] S151, Receive delay information;

[0298] The latency information includes transmission latency and decoding latency; where transmission latency represents the time it takes for the m-th first data frame to be transmitted from the third device to the first device; decoding latency is the decoding time occupied by the m-th first data frame; m≥1 and is an integer;

[0299] S152, obtain the transmission time of the m-th first data frame;

[0300] S153, at least based on the transmission time and delay information, obtain an estimated value of the playback time of the m-th first data frame on the first device;

[0301] S154, before sending the second data frame, add the estimated value as the fifth timestamp to the second data frame.

[0302] The fifth timestamp is used to calculate the difference with the current time to obtain the second playback time difference, in order to determine the synchronous playback strategy to be executed for the second data frame to be played.

[0303] Adding the fifth timestamp to the second data frame can be done by replacing the original video playback timestamp Vpts with the fifth timestamp, or by adding a separate fifth timestamp instead of replacing the original playback timestamp.

[0304] The step numbers in the embodiments of this application are only for distinguishing different steps and do not indicate a limitation on the execution sequence.

[0305] Optionally, the estimated playback time of the first data frame on the first device can be obtained at least based on the transmission time and delay information. This can be achieved by obtaining the estimated playback time of the m-th first data frame on the first device at least based on the transmission time and delay information of the m-th first data frame. Here, m ≥ 1 and is an integer. The mapping relationship is determined based on the sequence number and / or playback timestamp (PTS). The estimated value is added to the second data frame as the fifth timestamp. Alternatively, the estimated value can be added to the second data frame that has a mapping relationship with the m-th first data frame as the fifth timestamp.

[0306] Optionally, at least based on the transmission time and delay information, an estimated playback time for the first data frame on the first device can be obtained. This can be achieved by using the transmission time, delay information, a third timestamp, and a fourth timestamp. The third timestamp is a playback timestamp (PTS) added to the first data frame to indicate the playback sequence of each first data frame in the first data stream; the fourth timestamp is a playback timestamp (PTS) added to the second data frame to indicate the playback sequence of each second data frame in the second data stream.

[0307] This application also provides a synchronous playback method in a distributed service scenario. This method is applied to a one-to-many broadcast scenario and can be executed on a second device, such as... Figure 17 As shown, the method may include the following steps:

[0308] S161, Read the fifth timestamp corresponding to the second data frame to be played;

[0309] The fifth timestamp is an estimate of the playback time of the first data frame on the first device, obtained at least based on the transmission time and latency information. The latency information includes transmission latency and decoding latency. Transmission latency represents the time it takes for the first data frame to be transmitted from the third device to the first device; decoding latency is the decoding time occupied by one first data frame; and transmission time is the time it takes for the third device to send the first data frame.

[0310] S162, calculate the difference between the fifth timestamp and the current time to obtain the second playback time difference.

[0311] S163, based on the second playback time difference, determine the synchronous playback strategy to be executed for the second data frame to be played.

[0312] Optionally, the synchronous playback strategy includes at least one of waiting, displaying, and catching up. Specific synchronous playback strategies are described above and will not be repeated here.

[0313] This application also provides a synchronous playback method in a distributed business scenario, applied to a many-to-one broadcasting scenario. Referring to the foregoing description of the many-to-one broadcasting scenario, as follows... Figure 18 As shown, in a many-to-one broadcast scenario, the source end includes two or more devices, including a fourth device and a fifth device, and the playback end includes a sixth device; the fourth device and the fifth device transmit the fourth data stream and the fifth data stream to the sixth device for playback; the fourth device, the fifth device and the sixth device are all equipped with a control center, and the control center has a reserved interface for setting synchronization group information.

[0314] This method can be executed on a sixth device, such as Figure 19 As shown, the method may include the following steps:

[0315] S181, Receive synchronization group information sent by the fourth and fifth devices respectively;

[0316] Synchronization group information includes synchronization reference information and stream information. Synchronization reference information includes the identity information of the source device and the identity information of the playback device; stream information includes the data type information of the data streams sent by each source device.

[0317] S182, at least based on the synchronization group information, determine the synchronous playback strategy for the data stream to be played.

[0318] For specific implementation details, please refer to the descriptions of the corresponding scenarios above; they will not be repeated here.

[0319] This application also provides a playback device for a distributed service scenario. This playback device serves as a second device and is applied in a one-to-many broadcast scenario. In such a scenario... Figure 20 The second device shown includes a frame rate control module for performing the following operations:

[0320] Receive a first timestamp, read the second timestamp corresponding to the second data frame to be played, and calculate a first playback time difference based at least on the first timestamp and the second timestamp, and determine a synchronous playback strategy to be executed for the second data frame to be played based on the first playback time difference.

[0321] Optionally, the second device further includes: a second status feedback module and / or a second data buffer module; the second status feedback module is used at least to identify the data type of the second data frame to be played and to provide feedback to the third device; the second data buffer module is used at least to buffer at least one second data frame in the second data stream.

[0322] This application embodiment also provides a playback device in a distributed service scenario. The playback device, as a first device, is applied in a one-to-many broadcast scenario. In such a scenario... Figure 20 The first device shown includes a delay feedback module for performing the following operations:

[0323] Obtain the first timestamp and send it to the third device.

[0324] Wherein, the first timestamp represents the actual playback time of the m-th first data frame on the first device; the first timestamp is used to calculate the first playback time difference with the second timestamp, so as to determine the synchronous playback strategy to be executed for the second data frame to be played based on the first playback time difference; the second timestamp is used to represent the playback timing of each second data frame in the second data stream.

[0325] Optionally, the first device further includes: a first status feedback module and / or a first data buffer module; the first status feedback module is used at least to identify the data type of the first data frame to be played and to feed it back to the third device; the first data buffer module is used at least to buffer at least one first data frame in the first data stream.

[0326] This application also provides a source device for a distributed service scenario, which acts as a third device and is applied in a one-to-many broadcast scenario. For example... Figure 20 The third device shown includes a multiplexing management module for performing the following operations:

[0327] Receive the first timestamp and forward it to the second device; the first timestamp is obtained and sent by the first device.

[0328] Optionally, the third device may also include: a synchronization policy management module and / or a status monitoring module; the status monitoring module is used to receive the respective data types fed back by the two or more devices and input them into the synchronization policy management module; the synchronization policy management module is used to determine the synchronization management policy based on the respective data types fed back by the two or more devices.

[0329] This application also provides a playback system for distributed business scenarios, such as... Figure 20 As shown, the playback system is applied to a one-to-many broadcast scenario. The first device includes a latency feedback module for obtaining a first timestamp and sending the first timestamp to a third device. The third device includes a multiplexing management module for receiving the first timestamp and forwarding it to a second device. The second device includes a frame rate control module for receiving the first timestamp, reading the second timestamp corresponding to the second data frame to be played, and calculating a first playback time difference based at least on the first and second timestamps, and determining a synchronous playback strategy to be executed for the second data frame to be played based on the first playback time difference.

[0330] This application embodiment also provides a playback system for a distributed service scenario, which is applied to a one-to-many broadcast scenario. A first device is used to determine latency information; the latency information includes transmission latency and decoding latency; wherein the transmission latency represents the duration of transmission of a first data frame from a third device to the first device; the decoding latency is the decoding duration occupied by one first data frame; the third device is used to receive the latency information, obtain the transmission time of the first data frame, and, at least based on the transmission time and latency information, obtain an estimated value of the playback time of the first data frame on the first device; before sending a second data frame, the estimated value is added to the second data frame as a fifth timestamp; a second device is used to read the fifth timestamp of the second data frame to be played, calculate the difference between the fifth timestamp and the current time to obtain the second playback time difference; and, based on the second playback time difference, determine a synchronous playback strategy to be executed for the second data frame to be played.

[0331] This application also provides a playback system for a distributed business scenario, which is applied to a many-to-one broadcast scenario, such as... Figure 18 As shown, in a many-to-one broadcast scenario, the source end includes two or more devices, including a fourth device and a fifth device, and the playback end includes a sixth device. The fourth and fifth devices transmit the fourth and fifth data streams to the sixth device for playback, respectively. The fourth, fifth, and sixth devices are all equipped with a control center, which has a reserved interface for setting synchronization group information. The fourth and fifth devices are used to set the synchronization group information to their local control centers and to send the set synchronization group information to the sixth device. The synchronization group information includes synchronization reference information and stream information. The synchronization reference information includes the identity information of the source device and the identity information of the playback device. The stream information includes the data type information of the data streams sent by each source device. The sixth device is used to receive the synchronization group information sent by the fourth and fifth devices, and to determine the synchronization playback strategy for the data stream to be played, at least based on the synchronization group information.

[0332] This application also provides an electronic device, the electronic device including: one or more processors; a memory; at least one application program; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method as described in any of the above embodiments.

[0333] This application also provides a storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described in any of the above embodiments.

[0334] This application also provides a chip system, including a communication interface for inputting and / or outputting data; and a processor for executing a computer-executable program, causing a device equipped with the chip system to perform the methods described in any of the above embodiments.

[0335] It should be noted that the solutions provided in the embodiments of this application include any one or more of the synchronous playback system and method, electronic device, storage medium and chip system, or other forms of software products, hardware products or virtual service products obtained based on the technical concept of the method described in this application specification.

[0336] In the description of the apparatus in the embodiments of this application, various modules are constructed in the apparatus to implement the function of the method flow. The division of modules is only a logical division of functions. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0337] Specifically, in actual implementation, the various functional modules can be fully or partially integrated into a single physical entity, or they can be physically separated. These modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. For example, the detection module can be a separate processing element or integrated into a chip within the electronic device. The implementation of other modules is similar. Furthermore, these modules can be fully or partially integrated together, or implemented independently. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0338] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). Alternatively, these modules can be integrated together as a system-on-a-chip (SOC).

[0339] Specifically, in one embodiment of this application, the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to perform the method steps described in the embodiment of this application.

[0340] Specifically, in one embodiment of this application, the processor of the electronic device may be a device-on-a-chip (SoC), which may include a central processing unit (CPU) and may further include other types of processors, such as a graphics processing unit (GPU). Specifically, in one embodiment of this application, the processor of the electronic device may be a PWM control chip.

[0341] Specifically, in one embodiment of this application, the processor may include, for example, a CPU, DSP, microcontroller, or digital signal processor, and may also include a GPU, embedded neural network processing units (NPUs), and image signal processors (ISPs). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASICs, or one or more integrated circuits for controlling the execution of the program of the technical solution of this application. Furthermore, the processor may have the function of operating one or more software programs, which may be stored in a storage medium.

[0342] Specifically, in one embodiment of this application, the memory of the electronic device may be a read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), or other types of dynamic storage devices capable of storing information and instructions. It may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices. Alternatively, it may be any computer-readable medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer.

[0343] Specifically, in one embodiment of this application, the processor and memory can be combined into a single processing device, or more commonly, they are independent components. The processor executes program code stored in the memory to implement the method described in the embodiments of this application. In specific implementations, the memory can also be integrated into the processor, or it can be independent of the processor.

[0344] Furthermore, the devices, apparatuses, and modules described in the embodiments of this application can be implemented by computer chips or physical entities, or by products with certain functions.

[0345] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0346] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0347] The embodiments described in this application are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0348] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0349] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0350] It should also be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0351] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0352] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0353] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0354] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments of this application can be implemented using electronic hardware, computer software, or a combination of electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0355] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0356] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A synchronized playback system, characterized in that, The playback system includes a first device, a second device, and a third device; wherein the third device is used to transmit a first data stream to the first device for playback, and to transmit a second data stream to the second device for playback; the first data stream and the second data stream are two data streams to be played synchronously; the first data stream includes multiple first data frames; the second data stream includes multiple second data frames; The first device is used to obtain a first timestamp and send the first timestamp to the third device; the first timestamp represents the actual playback time of the m-th first data frame on the first device; m≥1 and is an integer; The third device is used to receive the first timestamp and forward it to the second device; The second device is configured to receive the first timestamp, read the second timestamp corresponding to the second data frame to be played, and calculate a first playback time difference based at least on the first timestamp and the second timestamp, and determine a synchronous playback strategy to be executed for the second data frame to be played based on the first playback time difference; the second timestamp is used to represent the playback timing of each second data frame in the second data stream; The step of calculating the first playback time difference based at least on the first timestamp and the second timestamp includes: The first playback time difference is calculated based on the first timestamp and the playback time of the nth second data frame on the second device; the first playback time difference is used to represent the difference between the playback time of the mth first data frame and the nth second data frame. There is a mapping relationship between the m-th first data frame and the n-th second data frame; the mapping relationship is determined based on the sequence number and / or playback timestamp (PTS); the existence of the mapping relationship indicates that the playback sequence presents a synchronized playback effect. The playback time of the nth second data frame on the second device is obtained based on the second timestamp corresponding to the nth second data frame.

2. The system as described in claim 1, characterized in that, The second device, when performing the operation of reading the second timestamp corresponding to the second data frame to be played, is specifically used for: At predetermined time intervals, the next second data frame to be played is retrieved from the buffer queue, and the second timestamp is read.

3. The system as described in claim 1, characterized in that, The second device, when performing the operation of calculating the first playback time difference based at least on the first timestamp and the second timestamp, is specifically used to: when it is determined that there is a mapping relationship between the next second data frame to be played and the m-th first data frame, calculate the first playback time difference based at least on the first timestamp corresponding to the m-th first data frame and the second timestamp corresponding to the next second data frame to be played.

4. The system as described in any one of claims 1-3, characterized in that, The second device is also used to identify the data type information of the second data frame to be played and feed it back to the third device; The first device is also used to identify the data type information of the first data frame to be played and feed it back to the third device; The third device is also used to receive data type information fed back by the first device and the second device respectively, and to determine the synchronization management strategy.

5. A method for synchronized playback, characterized in that, The method is applied to a second device, which receives and plays a second data stream sent by a third device; wherein the third device is also used to transmit a first data stream to the first device for playback; the first data stream and the second data stream are two data streams to be played synchronously, the first data stream includes multiple first data frames; the second data stream includes multiple second data frames; The method includes: Receive the first timestamp; the first timestamp represents the actual playback time of the m-th first data frame on the first device; m ≥ 1 and is an integer; Read the second timestamp corresponding to the second data frame to be played; the second timestamp is used to represent the playback sequence of each second data frame in the second data stream; Calculate the first playback time difference based at least on the first timestamp and the second timestamp; Based on the first playback time difference, a synchronous playback strategy is determined for the second data frame to be played. The step of calculating the first playback time difference based at least on the first timestamp and the second timestamp includes: The first playback time difference is calculated based on the first timestamp and the playback time of the nth second data frame on the second device; the first playback time difference is used to represent the difference between the playback time of the mth first data frame and the nth second data frame. There is a mapping relationship between the m-th first data frame and the n-th second data frame; the mapping relationship is determined based on the sequence number and / or playback timestamp (PTS); the existence of the mapping relationship indicates that the playback sequence presents a synchronized playback effect. The playback time of the nth second data frame on the second device is obtained based on the second timestamp corresponding to the nth second data frame.

6. The method as described in claim 5, characterized in that, Receive the first timestamp, including: The device receives the first timestamp forwarded by the third device; the first timestamp is generated by the first device and sent to the third device.

7. The method as described in claim 5, characterized in that, Read the second timestamp corresponding to the second data frame to be played, including: At predetermined time intervals, the next second data frame to be played is retrieved from the buffer queue, and the second timestamp is read.

8. The method according to any one of claims 5-7, characterized in that, Calculate the first playback time difference based at least on the first timestamp and the second timestamp, including: If it is determined that there is a mapping relationship between the next second data frame to be played and the m-th first data frame, the first playback time difference is calculated based at least on the first timestamp corresponding to the m-th first data frame and the second timestamp corresponding to the next second data frame to be played.

9. The method according to any one of claims 5-7, characterized in that, Calculate the first playback time difference based at least on the first timestamp and the second timestamp, including: Calculate the first playback time difference based on the first timestamp, the second timestamp, the first processing duration, and the second processing duration; Wherein, the first processing time is an estimated value of the processing time occupied from the time a first data frame is retrieved from the cache queue to the time it is played; the second processing time is an estimated value of the processing time occupied from the time a second data frame is retrieved from the cache queue to the time it is played.

10. The method according to any one of claims 5-7, characterized in that, The synchronous playback strategy includes at least one of waiting, displaying, and catching up; Based on the first playback time difference, a synchronous playback strategy is determined for the second data frame to be played, including: Compare the first playback time difference with a predetermined first threshold interval; If the first playback time difference is less than the minimum value of the first threshold interval, it is determined that the playback of the first data stream is lagging behind the second data stream, and the synchronous playback strategy executed for the second data frame to be played is determined to be waiting. If the first playback time difference is greater than the maximum value of the first threshold interval, it is determined that the playback of the first data stream is ahead of the second data stream, and the synchronous playback strategy executed for the second data frame to be played is determined to be catching up. If the first playback time difference falls within the first threshold range, it is determined that the playback of the first data stream and the playback of the second data stream are relatively synchronized, and the synchronous playback strategy executed for the second data frame to be played is determined to be to send it for display.

11. A method for synchronized playback, characterized in that, The method is applied to a third device, which is used to transmit a first data stream to the first device for playback and to transmit a second data stream to the second device for playback; wherein the first data stream and the second data stream are two data streams to be played synchronously, the first data stream includes multiple first data frames, and the second data stream includes multiple second data frames; The method includes: Receive latency information; the latency information includes transmission latency and decoding latency; wherein, the transmission latency represents the duration of transmission of the m-th first data frame from the third device to the first device; the decoding latency is the decoding duration occupied by the m-th first data frame; m≥1 and is an integer; Obtain the transmission time of the m-th first data frame; Based at least on the transmission time and the delay information, an estimated value of the playback time of the m-th first data frame on the first device is obtained; Before sending the second data frame, the estimated value is used as a fifth timestamp and added to the nth second data frame that has a mapping relationship with the mth first data frame; the fifth timestamp is used to calculate the second playback time difference with the current time to determine the synchronous playback strategy to be executed for the second data frame to be played; the mapping relationship is determined based on the sequence number and / or playback timestamp PTS; the existence of the mapping relationship indicates that a synchronous playback effect is presented in the playback sequence; The step of calculating the difference with the current time to obtain the second playback time difference includes: The second playback time difference is calculated based on the fifth timestamp and the current time; the second playback time difference is used to represent the difference in playback time between the m-th first data frame and the n-th second data frame. The current time is used to represent the playback time of the nth second data frame on the second device.

12. The method as described in claim 11, characterized in that, Based at least on the transmission time and the delay information, an estimated value for the playback time of the first data frame on the first device is obtained, including: Based at least on the transmission time of the m-th first data frame and the delay information, an estimated value of the playback time of the m-th first data frame on the first device is obtained.

13. The method as described in claim 11 or 12, characterized in that, Based at least on the transmission time and the delay information, an estimated value for the playback time of the first data frame on the first device is obtained, including: Based on the transmission time, the delay information, the third timestamp, and the fourth timestamp, an estimated value of the playback time of the first data frame on the first device is obtained; The third timestamp is a playback timestamp (PTS) added for the first data frame, used to represent the playback sequence of each first data frame in the first data stream. The fourth timestamp is a playback timestamp (PTS) added for the second data frame, used to indicate the playback sequence of each second data frame in the second data stream.

14. A method for synchronized playback, characterized in that, The method is applied to a sixth device, which is used to receive a fourth data stream sent by a fourth device and a fifth data stream sent by a fifth device, and to play the fourth data stream and the fifth data stream synchronously. The method includes: The system receives synchronization group information sent by the fourth device and the fifth device respectively; the synchronization group information includes synchronization reference information and stream information, the synchronization reference information includes source device identity information and playback device identity information; the stream information includes data type information of the data streams sent by each source device. Based at least on the synchronization group information, a synchronous playback strategy for the data stream to be played is determined; The determination of the synchronous playback strategy for the data stream to be played includes: Based on the difference in playback time between data frames in the fourth data stream and data frames in the fifth data stream that have a mapping relationship, a synchronous playback strategy to be executed for data frames in the fourth data stream or data frames in the fifth data stream is determined. The mapping relationship is determined based on the sequence number and / or playback timestamp (PTS); the existence of the mapping relationship indicates that the playback sequence presents a synchronized playback effect.

15. An electronic device, characterized in that, The electronic device includes: One or more processors; memory; at least one application program; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method as described in any one of claims 5-14.

16. A storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 5-14.

17. A chip system, characterized in that, include: A communication interface for inputting and / or outputting data; A processor for executing a computer-executable program, causing a device having the chip system mounted to perform the method as described in any one of claims 5-14.

Images