Synchronous playing system and method, electronic equipment, storage medium and chip system

Abstract

The invention provides a synchronous playing system and method, electronic equipment, a storage medium and a chip system, relates to the technical field of distributed hardware, and can realize sound and picture synchronization in a cross-equipment data projection application scene. The method can be executed based on a device of a playing end, and comprises the following steps: receiving a first timestamp; the first timestamp represents the actual playing time of the mth first data frame in the first device; m > = 1 and is an integer; reading a second timestamp corresponding to a second data frame to be played; the second timestamp is used for representing a playing time sequence of each second data frame in the second data stream; calculating a first playing time difference at least according to the first timestamp and the second timestamp; and based on the first playing time difference, determining a synchronous playing strategy executed for the to-be-played second data frame.

Description

Technical Field

[0001] The present application relates to the field of distributed hardware technology, and in particular to a synchronous playback system and method, electronic equipment, storage medium and chip system. Background Art

[0002] Distributed operating systems integrate the capabilities of various terminal devices to achieve hardware mutual assistance and resource sharing, providing users with a smooth full-scenario experience. Based on distributed hardware, distributed operating systems can provide users with various distributed services, such as distributed cameras, distributed screens, distributed audio, etc. Such distributed services generally include audio / video transmission capabilities. For example, in the audio and video broadcasting scenario, the source end encodes and compresses the original audio / video stream, sends it to the playback end through the transmission module, decodes and restores it, and plays it.

[0003] In the audio and video broadcasting scenario based on distributed hardware, due to differences in the sampling rate and post-processing time of audio and video data, as well as network jitter, audio and video playback may be out of sync when played across devices, affecting the user experience. Summary of the invention

[0004] The present application provides a synchronous playback system and method, electronic device, storage medium and chip system, which can realize synchronous playback of audio and video in a distributed business scenario of cross-device broadcasting.

[0005] In the first aspect, an embodiment of the present application provides a synchronous playback system, which includes a first device, a second device and a third device; the third device is used to transmit the first data stream to the first device for playback, and transmit the second data stream to the second device for playback; the first data stream and the second data stream are two data streams to be played synchronously; the first data stream includes multiple first data frames; the second data stream includes multiple second data frames; the first device is used to obtain a first timestamp and send the first timestamp to the third device; the third device is used to receive the first timestamp and forward it to the second device; the second device is used to receive the first timestamp, read the second timestamp corresponding to the second data frame to be played, and calculate the first playback time difference based on at least the first timestamp and the second timestamp, and determine the synchronous playback strategy to be executed for the second data frame to be played based on the first playback time difference.

[0006] In a feasible implementation, when the second device performs the operation of reading the second timestamp corresponding to the second data frame to be played, it is specifically used to: take out the next second data frame to be played from the cache queue at every predetermined time interval and read the second timestamp.

[0007] In a feasible implementation, the second device, when performing an operation of calculating the first playback time difference based on at least the first timestamp and the second timestamp, is specifically used to: when determining that there is a mapping relationship between the next second data frame to be played and the m-th first data frame, calculate the first playback time difference based on at least the first timestamp corresponding to the m-th first data frame and the second timestamp corresponding to the next second data frame to be played; wherein the mapping relationship is determined based on the serial number and / or the playback timestamp PTS.

[0008] In a feasible implementation, the second device is also used to identify the data type of the second data frame to be played, and feed it back to the third device; the first device is also used to identify the data type of the first data frame to be played, and feed it back to the third device; the third device is also used to receive the respective data types fed back by the first device and the second device, and determine the synchronization management strategy.

[0009] In a second aspect, an embodiment of the present application also provides a synchronous playback system, which includes a first device, a second device and a third device; the third device transmits the first data stream to the first device for playback, and transmits the second data stream to the second device for playback; the first data stream includes multiple first data frames; the second data stream includes multiple second data frames; the first device is used to determine delay information; the delay information includes transmission delay and decoding delay; wherein the transmission delay represents the duration of transmission of the first data frame from the third device to the first device; the decoding delay is the decoding duration occupied by a first data frame; the third device is used to receive the delay information, obtain the sending time of the first data frame, and obtain an estimated value of the playback time of the first data frame on the first device based on at least the sending time and the delay information; before sending the second data frame, the estimated value is added to the second data frame as a fifth timestamp; the second device is used to read the fifth timestamp of the second data frame to be played, calculate the difference between the fifth timestamp and the current time, and obtain a second playback time difference; based on the second playback time difference, determine the synchronous playback strategy to be executed for the second data frame to be played.

[0010] On the third aspect, an embodiment of the present application also provides a synchronous playback system, which includes a fourth device, a fifth device and a sixth device; the fourth device and the fifth device respectively transmit the fourth data stream and the fifth data stream to the sixth device for playback; the fourth device, the fifth device and the sixth device are all provided with a control center, and the control center reserves an interface for setting synchronization group information; the fourth device and the fifth device are used to set the synchronization group information to the local control center, respectively, and send the set synchronization group information to the sixth device, respectively; the synchronization group information includes synchronization reference information and stream information, the synchronization reference information includes source device identity information and playback device identity information; the stream information includes data type information of the data stream sent by each device on the source end; the sixth device is used to receive the synchronization group information sent by the fourth device and the fifth device, respectively, and determine the synchronous playback strategy for the data stream to be played based at least on the synchronization group information.

[0011] In the fourth aspect, the technical solution of the present application provides a synchronous playback method, which is applied to a second device, the second device is used to receive and play a second data stream sent by a third device; wherein the third device is also used to transmit a first data stream to the first device for playback; the first data stream and the second data stream are two data streams to be played synchronously, the first data stream includes multiple first data frames; the second data stream includes multiple second data frames; the method includes: receiving a first timestamp; the first timestamp indicates the actual playback time of the mth first data frame on the first device; m≥1 and is an integer; reading a second timestamp corresponding to the second data frame to be played; the second timestamp is used to indicate the playback timing of each second data frame in the second data stream; at least according to the first timestamp and the second timestamp, calculating the first playback time difference; based on the first playback time difference, determining the synchronous playback strategy executed for the second data frame to be played. The method uses the actual playback time of each data frame in the first data stream as a reference, calculates the playback time difference between the second data stream and the first data stream, and can judge whether the playback progress of the second data stream is ahead, behind or synchronized with the first data stream through the playback time difference, and then adjusts the playback progress of the second data stream, so as to achieve synchronous playback.

[0012] In a feasible implementation, receiving the first timestamp includes: receiving the first timestamp forwarded by the third device; the first timestamp is generated by the first device and sent to the third device. Compared with other information transmission methods such as the first device directly sending the first timestamp to the second device, the first timestamp is forwarded by the third device, which can reduce the transmission pressure of the data link and reduce the delay occupied by transmitting the key information of the first timestamp.

[0013] In a feasible implementation, reading the second timestamp corresponding to the second data frame to be played includes: taking out the next second data frame to be played from the cache queue at every predetermined time interval and reading the second timestamp. The trigger condition for reading the second timestamp is to trigger once at every predetermined time interval. The frequency of reading the second timestamp can be controlled by setting the adjustment time interval, thereby controlling the frequency of calculating the playback time difference. The higher the frequency, the more fine-grained the time is, which is equivalent to determining whether synchronization is performed.

[0014] In a feasible implementation, the first playback time difference is calculated based on at least the first timestamp and the second timestamp, including: when it is determined that there is a mapping relationship between the next second data frame to be played and the mth first data frame, the first playback time difference is calculated based on at least the first timestamp corresponding to the mth first data frame and the second timestamp corresponding to the next second data frame to be played; wherein the mapping relationship is determined based on the sequence number and / or the playback timestamp PTS. Calculating the playback time difference based on the first timestamp and the second timestamp can reflect the difference in playback progress between the first data stream and the second data stream to a certain extent.

[0015] In a feasible implementation, the first playback time difference is calculated based on at least the first timestamp and the second timestamp, including: calculating the first playback time difference based on the first timestamp, the second timestamp, the first processing time and the second processing time; wherein the first processing time is an estimated value of the processing time taken from the first data frame being taken out of the cache queue to the playback; the second processing time is an estimated value of the processing time taken from the second data frame being taken out of the cache queue to the playback. Further introducing the first processing time and the second processing time as two parameters to calculate the playback time difference can more accurately reflect the difference in playback progress between the first data stream and the second data stream.

[0016] In a feasible implementation, the synchronous playback strategy includes at least one of waiting, display and catching up; based on the first playback time difference, determining the synchronous playback strategy to be executed for the second data frame to be played, including: comparing the first playback time difference with a predetermined first threshold interval; when the first playback time difference is less than the minimum value of the threshold interval, determining that the playback of the first data stream lags behind the second data stream, and determining that the synchronous playback strategy to be executed for the second data frame to be played is waiting; when the first playback time difference is greater than the maximum value of the threshold interval, determining that the playback of the first data stream is ahead of the second data stream, and determining that the synchronous playback strategy to be executed for the second data frame to be played is catching up; when the first playback time difference belongs to the threshold interval, determining that the playback of the first data stream is relatively synchronized with the playback of the second data stream, and determining that the synchronous playback strategy to be executed for the second data frame to be played is display.

[0017] In a feasible implementation, waiting includes one or more of reducing the playback rate, waiting for a first predetermined time period, and dropping frames; wherein the first predetermined time period is determined based on the difference between the minimum value in the threshold interval and the first playback time difference; catching up includes increasing the playback rate and / or dropping frames.

[0018] In a fifth aspect, an embodiment of the present application also provides a synchronous playback method, which is applied to a third device, and the third device is used to transmit a first data stream to the first device for playback, and to transmit a second data stream to the second device for playback; wherein the first data stream and the second data stream are two data streams to be played synchronously, and the first data stream includes multiple first data frames; the second data stream includes multiple second data frames; the method includes: receiving delay information; the delay information includes transmission delay and decoding delay; wherein the transmission delay represents the duration of the mth first data frame being transmitted from the third device to the first device; the decoding delay is the decoding duration occupied by the mth first data frame; m≥1 and is an integer; obtaining the sending time of the mth first data frame; at least based on the sending time and the delay information, obtaining an estimated value of the playback time of the mth first data frame on the first device; before sending the second data frame, adding the estimated value as a fifth timestamp to the second data frame; the fifth timestamp is used to perform a difference calculation with the current time to obtain a second playback time difference, so as to determine the synchronous playback strategy to be executed for the second data frame to be played. This method estimates the playback time of each data frame in the first data stream based on the historical delay information of the first data stream. Using the estimated value as a reference, the second device can determine the difference in playback progress by comparing the estimated value with the current time, thereby achieving synchronous playback.

[0019] In a feasible implementation, obtaining an estimated value of the playback time of the first data frame on the first device at least based on the transmission time and delay information includes: obtaining an estimated value of the playback time of the mth first data frame on the first device at least based on the transmission time and delay information of the mth first data frame; adding the estimated value as the fifth timestamp to the second data frame, including: adding the estimated value as the fifth timestamp to the second data frame that has a mapping relationship with the mth first data frame, wherein the mapping relationship is determined based on the sequence number and / or the playback timestamp PTS. For two data frames that have a mapping relationship, that is, two data frames that should be played synchronously, adding the fifth timestamp to the second data frame that has a mapping relationship with the mth first data frame is equivalent to marking the estimated playback time of the mth first data frame on the corresponding second data frame. Before playing, the second device only needs to compare the fifth timestamp on each second data frame with the current time to determine the difference in playback progress, thereby achieving synchronous playback.

[0020] In a feasible implementation, at least according to the sending time and the delay information, the estimated value of the play time of the first data frame on the first device is obtained, including: according to the sending time, the delay information, the third timestamp and the fourth timestamp, the estimated value of the play time of the first data frame on the first device is obtained; wherein the third timestamp is the play timestamp PTS added to the first data frame, which is used to indicate the play timing of each first data frame in the first data stream; the fourth timestamp is the play timestamp PTS added to the second data frame, which is used to indicate the play timing of each second data frame in the second data stream. This method provides a feasible solution for calculating the estimated value of the play time, and the calculated estimated value can relatively accurately reflect the difference in the play progress of the first data stream and the second data stream.

[0021] In a sixth aspect, an embodiment of the present application also provides a synchronous playback method, which is applied to a second device, and the second device is used to receive and play a second data stream sent by a third device; wherein the third device is also used to transmit the first data stream to the first device for playback; the first data stream and the second data stream are two data streams to be played synchronously, and the first data stream includes multiple first data frames; the second data stream includes multiple second data frames; the method includes: reading a fifth timestamp corresponding to the second data frame to be played; wherein the fifth timestamp is an estimated value of the playback time of the first data frame on the first device, which is obtained at least based on the sending time and delay information; wherein the delay information includes transmission delay and decoding delay; the transmission delay indicates the duration of the first data frame being transmitted from the third device to the first device; the decoding delay is the decoding duration occupied by a first data frame; the sending time is the time when the third device sends the first data frame; the difference between the fifth timestamp and the current time is calculated to obtain a second playback time difference; based on the second playback time difference, determine the synchronous playback strategy to be executed for the second data frame to be played.

[0022] In a feasible implementation, the synchronous playback strategy includes at least one of waiting, display and catching up; based on the second playback time difference, determining the synchronous playback strategy to be executed for the second data frame to be played, including: comparing the second playback time difference with a predetermined first threshold interval; when the second playback time difference is less than the minimum value of the threshold interval, determining that the playback of the first data stream lags behind the second data stream, and determining that the synchronous playback strategy to be executed for the second data frame to be played is waiting; when the second playback time difference is greater than the maximum value of the threshold interval, determining that the playback of the first data stream is ahead of the second data stream, and determining that the synchronous playback strategy to be executed for the second data frame to be played is catching up; when the second playback time difference belongs to the threshold interval, determining that the playback of the first data stream is relatively synchronized with the playback of the second data stream, and determining that the synchronous playback strategy to be executed for the second data frame to be played is display.

[0023] In the seventh aspect, an embodiment of the present application also provides a synchronous playback method, which is applied to a sixth device, and the sixth device is used to receive a fourth data stream sent by a fourth device and a fifth data stream sent by a fifth device, and to synchronously play the fourth data stream and the fifth data stream; the method includes: receiving synchronization group information sent by the fourth device and the fifth device respectively; the synchronization group information includes synchronization reference information and stream information, the synchronization reference information includes source device identity information and playback device identity information; the stream information includes data type information of the data stream sent by each device on the source end; at least based on the synchronization group information, determine the synchronous playback strategy for the data stream to be played.

[0024] In an eighth aspect, the technical solution of the present application also provides an electronic device for sending data through a data transmission interface, the electronic device comprising: one or more processors; a memory; at least one application; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to execute a method as described in any one of the fourth to seventh aspects above.

[0025] In a ninth aspect, the technical solution of the present application also provides a storage medium, comprising computer instructions, which, when executed on an electronic device, enables the electronic device to execute a method as described in any one of the fourth to seventh aspects above.

[0026] In the tenth aspect, the technical solution of the present application also provides a chip system, including: a communication interface for inputting and / or outputting data; a processor for executing a computer executable program, so that a device equipped with the chip system executes a method as described in any one of the fourth to seventh aspects above. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a diagram showing an example of the system architecture for a one-to-one broadcasting application scenario;

[0028] Figure 2 This is a schematic diagram of the audio and video data transmission process architecture based on WebRTC technology in related technologies;

[0029] Figure 3 This is an example of a feasible product hardware architecture of the electronic device provided in the embodiments of the present application;

[0030] Figure 4(a)-Figure 4(e) They are example diagrams of device architecture for one-to-many broadcasting application scenarios;

[0031] Figure 5(a)-Figure 5(d) They are example diagrams of one-to-many broadcast application scenarios;

[0032] Figure 6(a)-Figure 6(c)They are example diagrams of device architecture for many-to-one broadcast application scenarios;

[0033] Figure 7 This is a diagram showing an example of the system architecture for a one-to-two broadcast application scenario;

[0034] Figure 8 This is a diagram showing an example of a system architecture in which an audio and video synchronization module is added to a one-to-many broadcasting application scenario;

[0035] Fig. 9 It is a schematic diagram of a link for delay feedback between a video playback device and an audio playback device in a one-to-many broadcasting application scenario;

[0036] Fig.10 It is a schematic diagram of the video frame processing strategy in a one-to-many broadcast application scenario;

[0037] Fig.11 It is a schematic diagram of multiple thread time nodes of the video post-processing process;

[0038] Fig.12 This is a diagram showing an example of the system architecture using the second solution in a one-to-many broadcasting scenario.

[0039] Fig.13 This is a diagram showing an example of a system architecture for synchronized playback in a many-to-many broadcasting application scenario;

[0040] Fig.14 This is a typical system architecture example diagram for a one-to-many broadcast application scenario;

[0041] Fig.15 A schematic diagram of a flow chart of an embodiment of a synchronous playback method;

[0042] Fig.16 A flowchart of another embodiment of a synchronous playback method;

[0043] Fig.17 A flowchart of another embodiment of a synchronous playback method;

[0044] Fig.18 This is a typical system architecture example diagram for a many-to-one broadcast application scenario;

[0045] Fig.19 A flowchart of another embodiment of a synchronous playback method;

[0046] Fig. 20 The system architecture diagram of the playback system in a one-to-many broadcast application scenario. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0048] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0049] In data broadcasting application scenarios such as audio and video, the asynchronous playback of sound and picture will affect the viewing experience and reduce the user experience, so it is necessary to keep the audio and picture synchronized.

[0050] Currently, in a one-to-one broadcast scenario, audio data and video data can be encoded and transmitted in one channel to maintain playback synchronization. One-to-one broadcast means that both the source and the playback end only include one device, and one device (the source end) transmits data to another device (the playback end) for playback. For example, Figure 1 As shown, the source end includes only one device A', and the playback end includes only one device B'. Device A' transmits audio and video to device B', and device B' plays the video and audio. In this one-to-one broadcasting scenario, the source end device encodes and transmits the video and audio in one channel, and the playback end decodes the video and audio and then broadcasts them. Since audio and video are encoded and transmitted in one channel, the corresponding relationship is one-to-one matching, that is, the playback timing of the audio frame and the corresponding image frame can be kept consistent, thereby eliminating the asynchrony caused by the audio and video transmission delay and the difference in audio and video frame processing time.

[0051] Web Real-Time Communication (WebRTC) is a technology that supports real-time voice calls or video chats in web browsers. It provides a solution to support real-time audio and video transmission, including audio and video acquisition, encoding and decoding, network transmission, and display. The audio and video acquisition is carried by distributed services. Figure 2As shown in the figure, the functions provided by WebRTC include audio and video encoding and decoding, audio and video acquisition and rendering, transmission, flow control, audio and video synchronization, etc. Among them, the transmission protocol uses RTP / RTCP (Realtime Transport Protocol) / (Real-time Control Protocol), RTP completes the packaging of media data, and RTCP is responsible for QoS (Quality of Service) information feedback; the sender adds redundant information after encoding, and the receiver recovers the error BIT through redundant information to solve the bit error during transmission. The flow control module is deployed at the sender, and the appropriate sending rate and encoding bit rate of the sender are calculated based on the QoS information fed back by the receiver. The receiver deploys jitterBuffer to solve the jitter and disorder problems of transmitted data packets.

[0052] from Figure 2 As can be seen from the figure, for audio and video data transmission based on WebRTC technology, on the sending side, the audio frames and video frames are encoded and packaged based on the RTP / RTCP protocol. After smoothing, the audio and video data packets are sent through a transmission channel. On the receiving side, they are received through a transmission channel, and then audio decoding and video decoding are performed respectively. In this solution, audio data and video data are transmitted through a transmission channel.

[0053] In a one-to-one broadcast scenario, audio and video data can be transmitted between the source device and the playback device through a transmission channel. Since there is only one device on the playback end, it is relatively easy to achieve audio and video playback synchronization. Therefore, the above solution based on WebRTC can be applied to one-to-one broadcast scenarios.

[0054] However, in distributed business scenarios, there are business needs for cross-device broadcasting, such as one-to-many broadcasting or many-to-one broadcasting. For example, when the playback end includes more than two devices, a device at the source end transmits data to more than two devices at the playback end, which generally needs to be transmitted separately through different transmission channels. In this case, audio and video transmission solutions such as those based on WebRTC technology cannot adapt to cross-device broadcasting or more complex distributed business scenarios.

[0055] In view of this, an embodiment of the present application proposes a synchronous playback method, which can achieve synchronous playback of streaming media data such as audio and video in cross-device broadcasting application scenarios, thereby solving the above-mentioned problems.

[0056] The method provided in the embodiment of the present application can be applied to various application scenarios in which data is transmitted from the source end to the playback end for playback, for example, various application scenarios such as video projection to a display device and / or audio projection to an audio playback device. Exemplarily, one of the typical application scenarios of the method can be one-to-many broadcasting in a distributed business scenario. In addition, the method can also be applied to many-to-one broadcasting scenarios or many-to-many broadcasting scenarios.

[0057] In the embodiments of the present application, for the sake of clarity, the end that sends data is regarded as the source end, which includes at least one electronic device; the end that receives and plays data is regarded as the playback end, which includes at least one electronic device.

[0058] Among them, the device on the source side (such as the third device) can be an electronic device that can store data and support sending data, such as a mobile phone, a personal computer (PC) (such as a laptop), a tablet computer (Tablet Personal Computer, Tablet PC), a wearable electronic device (such as a smart watch, a bracelet, glasses, a collar), a smart speaker, a personal digital assistant (PDA), an augmented reality technology (AR) device, a virtual reality (VR) device, a robot, a smart toy, a learning machine, etc. The device on the playback side (such as the first device, the second device) can be an electronic device that supports playing at least one data format, such as supporting audio playback and / or video playback, and can specifically be a TV, a smart screen, a set-top box, a mobile phone, a tablet computer, a personal computer, a wearable electronic device, an augmented reality technology device, a virtual reality device, a learning machine, a robot, and other products that support playing video, audio, and other formats of data.

[0059] As shown in Figure 4, Figure 3 An example of a feasible product hardware architecture of an electronic device is shown. The electronic device 300 may include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, an earphone interface 370D, a button 390, a motor 393, an indicator 392, a display screen 393, and a subscriber identification module (SIM) card interface 394, etc.

[0060] It is to be understood that the structure illustrated in the embodiment of the present invention is only an example and does not constitute a limitation on the electronic device 300. The electronic device in the embodiment of the present application may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0061] For example, the electronic device on the source side may only include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, etc.

[0062] The electronic device on the playback end side, taking the audio playback device as an example, may only include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 370A, a button 390, a display screen 394, etc. Taking the video playback device as an example, it may only include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, a display screen 394, etc., and optionally, it may also include an audio module 370, a speaker 370A, a button 390, etc.

[0063] Specifically, the processor 310 may include one or more processing units, for example, the processor 310 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0064] The processor 310 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 310 is a cache memory. The memory may store instructions or data that the processor 310 has just used or cyclically used. If the processor 310 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 310, and thus improves the efficiency of the system.

[0065] In some embodiments, the processor 310 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0066] The USB interface 330 is an interface that complies with the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 330 can be used to connect a charger to charge the electronic device 300, and can also be used to transmit data between the electronic device 300 and other devices, such as a source device transmitting data to a playback device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0067] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 300. In other embodiments of the present application, the electronic device 300 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0068] The charging management module 340 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 340 may receive charging input from a wired charger through the USB interface 330. In some wireless charging embodiments, the charging management module 340 may receive wireless charging input through a wireless charging coil of the electronic device 300. While the charging management module 340 is charging the battery 342, it may also power the electronic device through the power management module 341.

[0069] The power management module 341 is used to connect the battery 342, the charging management module 340 and the processor 310. The power management module 341 receives input from the battery 342 and / or the charging management module 340, and supplies power to the processor 310, the internal memory 321, the display screen 394, the camera 393, and the wireless communication module 360. The power management module 341 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 341 can also be set in the processor 310. In other embodiments, the power management module 341 and the charging management module 340 can also be set in the same device.

[0070] The data transmission function of the electronic device 300 can be implemented through the antenna 1, the antenna 2, the mobile communication module 350, the wireless communication module 360, the modem processor and the baseband processor.

[0071] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 300 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0072] The mobile communication module 350 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 300. The mobile communication module 350 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 350 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 350 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 350 can be set in the processor 310. In some embodiments, at least some of the functional modules of the mobile communication module 350 can be set in the same device as at least some of the modules of the processor 310.

[0073] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 370A, a receiver 370B, etc.), or displays an image or video through a display screen 394. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 310 and be set in the same device as the mobile communication module 350 or other functional modules.

[0074] The wireless communication module 360 ​​can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 300. The wireless communication module 360 ​​can be one or more devices integrating at least one communication processing module. The wireless communication module 360 ​​receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 310. The wireless communication module 360 ​​can also receive the signal to be sent from the processor 310, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0075] In some embodiments, the antenna 1 of the electronic device 300 is coupled to the mobile communication module 350, and the antenna 2 is coupled to the wireless communication module 360, so that the electronic device 300 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0076] The electronic device 300 implements the display function through a GPU, a display screen 393, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 393 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 310 may include one or more GPUs, which execute program instructions to generate or change display information.

[0077] The display screen 393 is used to display images, videos, etc. The display screen 393 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 300 may include 3 or N display screens 393, where N is a positive integer greater than 3.

[0078] The electronic device 300 can implement the video playback function through a video codec, a GPU, a display screen 393 and an application processor.

[0079] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 300 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0080] Video codecs are used to compress or decompress digital videos. The electronic device 300 may support one or more video codecs. Thus, the electronic device 300 may play or record videos in a variety of coding formats, such as moving picture experts group (MPEG) 3, MPEG2, MPEG3, MPEG4, etc.

[0081] Optionally, the NPU is a neural network (NN) computing processor that quickly processes input information by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, and can also continuously self-learn. The NPU can realize applications such as intelligent cognition of the electronic device 300, such as intelligent identification of data types in data streams.

[0082] The external memory interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 300. The external memory card communicates with the processor 310 through the external memory interface 320 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0083] The internal memory 321 can be used to store computer executable program codes, which include instructions. The internal memory 321 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 300 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 321 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 310 executes various functional applications and data processing of the electronic device 300 by running instructions stored in the internal memory 321, and / or instructions stored in a memory provided in the processor.

[0084] The electronic device 300 can implement audio functions, such as music playback, through the audio module 370, the speaker 370A, the receiver 370B, the microphone 370C, the headphone jack 370D, and the application processor.

[0085] The audio module 370 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 370 can also be used to encode and decode audio signals. In some embodiments, the audio module 370 can be arranged in the processor 310, or some functional modules of the audio module 370 can be arranged in the processor 310.

[0086] The speaker 370A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 300 can listen to music or listen to a hands-free call through the speaker 370A.

[0087] The receiver 370B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 300 receives a call or voice message, the voice can be received by placing the receiver 370B close to the human ear.

[0088] Microphone 370C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 370C to input the sound signal into the microphone 370C. The electronic device 300 can be provided with at least one microphone 370C. In other embodiments, the electronic device 300 can be provided with two microphones 370C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 300 can also be provided with three, four or more microphones 370C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0089] The earphone interface 370D is used to connect a wired earphone and can be a USB interface 330 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0090] The key 390 includes a power key, a volume key, etc. The key 390 may be a mechanical key or a touch key. The electronic device 300 may receive key input and generate key signal input related to user settings and function control of the electronic device 300.

[0091] Motor 391 can generate vibration prompts. Motor 391 can be used for successful data transmission prompts, and can also be used for playback feedback. For example, whether the data transmission is successful can correspond to different vibration feedback effects. Different playback situations (for example: synchronous playback, video ahead of audio, audio ahead of video, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0092] Indicator 392 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0093] The SIM card interface 394 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 300 by inserting it into the SIM card interface 394 or pulling it out from the SIM card interface 394. The electronic device 300 can support 3 or N SIM card interfaces, where N is a positive integer greater than 3. The SIM card interface 394 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 394 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 394 can also be compatible with different types of SIM cards. The SIM card interface 394 can also be compatible with external memory cards. The electronic device 300 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 300 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 300 and cannot be separated from the electronic device 300.

[0094] The following is an illustrative description of typical application scenarios involved in this application.

[0095] One-to-many broadcasting means that the source end includes only one device, while the playback end includes more than two devices. Data is transmitted from the source end to more than two devices at the playback end through more than two transmission channels, and then played. For example, as shown in FIG4(a), device A is the source end device, and devices B and C are the playback end devices. Device A transmits audio data (sound) to device B through one transmission channel, and audio is played through device B. It transmits video data (image) to device C through another transmission channel, and video is played through device C.

[0096] The number of devices included in the playback end can be greater than 2. FIG4(a) only shows the case where the number of devices is 2. In actual applications, the number of devices at the playback end can be 3 or more. For example, as shown in FIG4(b), the playback end can include device B, device C, and device D. Device A transmits data to device B to device D through three transmission channels. Device B can be a speaker for playing audio. Device C and device D are both display screens, which are respectively used to display different parts of the same picture. The two display screens are spliced ​​to display a complete image. For another example, as shown in FIG4(c), the playback end can include device C, device D, device D, and device E. The data on the device A side is transmitted to device B to device E through four transmission channels. Device C and device D are both speakers, which are respectively used to play the left channel and the right channel. Device D and device E are both display screens, which are used to splice and display the complete image in the video frame.

[0097] In addition, the data transmitted from the source end to the playback end through different transmission channels can be different types of data or the same type of data. For example, as shown in Figure 4(d), both transmission channels transmit video data (images). For example, device B and device C are both display screens, and the two are used to splice and display a complete image; or the two are used to independently display the same complete image, but are placed in different positions within the application scene for multi-viewing. As shown in Figure 4(e), both transmission channels transmit audio data (sound). Device B and device C are both speakers, one of which transmits left channel data, and the other transmits right channel data. Similarly, the number of devices on the playback end can be more than 3, not limited to 2.

[0098] In combination with actual application scenarios, as shown in FIG5(a), in a distributed business scenario, a one-to-many broadcast may be a source device that is a mobile terminal device 501, such as a mobile terminal that can be a mobile phone, a tablet computer, etc., and a playback end that includes a display screen 502 and an audio playback device 503, such as a smart screen, a TV, etc., and an audio playback device that can be a speaker, a smart assistant, a robot with an audio playback function, etc. Exemplarily, the mobile phone transmits the audio to the speaker to play the audio, and transmits the video to the display screen to play the video.

[0099] For example, as shown in FIG5(b), one-to-many broadcasting can also be that the mobile terminal 504 transmits the image to the display devices 5051-5054 through four transmission channels, and the display devices 5051-5054 each display a part of the image, and the four display devices are spliced ​​together to display the complete image of the large screen. The relative delay of audio and video playback will only be noticed by people when it exceeds a certain degree. For example, the synchronous playback of audio and video allows a relative delay of -185 to 90ms. In the case of displaying a larger complete image through splicing of four display devices, since the picture played on each device is compared with the pictures of the other three devices, the asynchrony between the images will be more obvious through comparison. Therefore, the accuracy of the synchronization control in this scenario needs to be higher than the accuracy of the synchronous playback control in the audio and one video scenario. In this scenario where multiple images are spliced ​​to complete the image, the delay time allowed is shorter, for example, the relative delay time may need to be controlled below the inter-frame interval, for example, within 33ms or 16.7ms.

[0100] For another example, as shown in FIG5(c) of the one-to-many broadcasting scenario, it can also be a smart wearable device 506, which transmits the audio to the left channel 507 and the right channel 508 of the car audio respectively. For example, the smart wearable device is a smart watch, which is transmitted to different channel devices of the car audio based on the streaming media transmission protocol. This application scenario can be that during the driving process of a car, motorcycle, or electric car, a function key or button for one-key delivery is set on the smart watch, and the driver can deliver the audio with one key and play music through the car speaker. It should be noted that the method provided in the embodiment of the present application does not limit the data transmission channel, and can therefore be used to solve various distributed business scenarios in which smart terminals (such as watches, mobile phones, etc.) deliver audio to the car audio. In comparison, in the related scheme, taking Bluetooth transmission as an example, in the connection working mode, multiple CIS (ConnectedIsochronous Stream) are required to belong to the same channel group CIG (ConnectedIsochronous Groups). In order to achieve synchronous processing, CIG will give each CIS different delay parameters, allowing them to be processed synchronously after their respective time periods. Among them, the CIS in the same CIG has the same timestamp. In the non-connected working mode, multiple BIS (Broadcast Isochronous Stream) must belong to the same broadcast group BIG (Broadcast Isochronous Groups). It can be seen that the relevant scheme requires that the data streams belong to the same channel group or the same broadcast group, and there are restrictions on the channels for data transmission, while the method provided in the embodiment of the present application is not subject to this restriction.

[0101] For another example, as shown in FIG5(d), in a one-to-many broadcasting scenario, the source device may be a smart wearable device 509, and the playback device may include a mobile terminal 512, such as a smart phone, in addition to the car audio 510 and 511. For example, the smart wearable device is a smart watch. During the user's driving, the smart watch monitors the user's heart rate and other data, generates a video format file that can dynamically display the heart rate data on the mobile terminal, and generates a voice prompt accompanying the heart rate data. The video format data is transmitted to the bound mobile phone in real time for display, and the voice is transmitted to the car audio for playback. For example, when the video shows an abnormal heart rate, the voice broadcast content is synchronously played through the car audio, such as "Your heart rate is abnormal. It is recommended to stop and rest or seek help as soon as possible." The synchronization in this scenario has a lower control accuracy for synchronous playback than the scenario shown in FIG5(a), and the playback delay time of the voice relative to the video can be less than 2s.

[0102] In the many-to-one broadcast scenario, the source end includes different devices, while the playback end only includes one device. Data is transmitted from different devices at the source end to the playback end through two or more transmission channels for playback. Figure 6(a)-Figure 6(c) As shown, device B and device C transmit video data (image) and audio data (sound) to device A through different transmission channels, respectively, and play through device A. Alternatively, device B and device C transmit images to device A through two transmission channels for playback. Alternatively, device B and device C transmit the left channel and the right channel of the sound to device A through two transmission channels for playback. Similarly, the number of devices included in the source end can be greater than 2, and is not limited to 2.

[0103] Optionally, the software environment for implementing the method provided in the embodiment of the present application may be a distributed operating system, such as a distributed operating system based on HarmonyOS. Exemplarily, a distributed hardware management framework may be deployed in the distributed operating system, and the distributed hardware management framework may be used to provide information management capabilities for distributed hardware subsystems.

[0104] In an embodiment of the present application, the transmission channel may be one or more of various channels that can realize point-to-point data transmission based on a transmission medium. For example, optionally, the transmission channel may be a channel, a data link, etc.

[0105] Exemplarily, data transmission between devices can be implemented based on one or more wireless screen projection protocols such as Miracast, DNLA (Digital Living Network Alliance), WiDi (Intel Wireless Display, wireless high-definition technology), and AirPlay.

[0106] It should be noted that, in the embodiments of the present application, the data to be broadcast may be in various data formats that support playback on a playback end device, for example, it may be streaming media data mainly based on audio and video data, and the streaming media data may include one or more data streams such as sound stream, video stream, text stream, picture stream, animation stream, etc. Exemplarily, in some embodiments, the data to be broadcast may be audio and / or video data, for example, broadcasting more than two audio channels, or broadcasting more than two videos, or broadcasting at least one audio channel and at least one video channel.

[0107] The video data may be one or more video formats such as MPEG (Moving Pictures Expert Group), AVI (Audio Video Interleave), WMV (Windows Media Video), QuickTime, RealVideo, Flash, Mpeg-4, newAVI, etc. The audio data may be one or more audio formats such as MP3 (Moving Picture Experts Group Audio LayerⅢ), WMA (Windows Media Audio), AAC (Advanced Audio Coding), WAV (WAVE form audio format Sound), APE (Monkey's Audio), etc.

[0108] The data to be broadcast may also be a media file, which is not limited to only one stream type, and may include video, audio, subtitles, or even multi-channel video, multi-channel audio (multi-channel), dual subtitles, etc.

[0109] In other embodiments, it is not excluded that the broadcast data may be a combination of audio data and / or video data and data in formats such as documents and pictures. The following illustrative description will be given using audio data and video data as examples.

[0110] The synchronous playback method provided by the embodiment of the present application, in a one-to-many broadcast scenario, uses the playback time of the streaming media on one of the multiple devices (the first device) of the playback end as a reference to control the playback of the streaming media on other devices of the playback end, thereby achieving playback synchronization between multiple devices of the playback end. The playback time used as a reference can be the actual playback time, or it can be an estimated playback time estimated by combining the transmission delay and decoding delay corresponding to the first device. For example, a playback time difference can be obtained based on at least the actual playback time of each data frame on one of the devices on the playback end and the playback timestamp of each data frame to be played by other devices, and the playback of the data frames to be played on other devices can be adjusted based on the playback time difference to achieve synchronization. Alternatively, the estimated playback time of each data frame on one of the devices on the playback end is used as a reference to obtain the estimated playback time of the data frames to be played on other devices in a synchronized situation, and a timestamp corresponding to the estimated playback time is added to each data frame before transmitting the data, thereby guiding other devices on the playback end to perform playback control according to the estimated playback time to achieve synchronization.

[0111] In a many-to-one scenario, after the clocks of multiple devices on the source end and one device on the playback end are synchronized, the synchronization information is set in the frame header through the service layer settings and reserved in the control center to achieve synchronization in the many-to-one scenario.

[0112] The following describes how to achieve synchronized playback between different devices on the playback end in a one-to-many broadcast scenario.

[0113] In the embodiment of the present application, for the one-to-many broadcasting scenario, at least two solutions are proposed to achieve playback synchronization between different devices at the playback end. These two solutions can be implemented independently or in combination.

[0114] In the first solution, the actual playback time of one of the devices on the playback end is used as a reference time to control the playback time of other devices to achieve synchronization.

[0115] In the second solution, before sending data, the source end will obtain an estimated value of the playback timestamp based on information such as the transmission delay and decoding delay fed back by one of the devices on the playback end. Other devices on the playback end can directly compare the estimated value of the playback timestamp with the current time to control the playback to achieve synchronization.

[0116] For example, taking a one-to-two broadcast scenario as an example, Figure 7 The system architecture in a one-to-two broadcasting scenario is shown, which includes a source end and a playback end. Device A at the source end performs video encoding on the image frame data stream and audio encoding on the audio frame data stream in two ways, and then transmits them to devices B and C at the playback end through two transmission channels. After device B performs video decoding, it is sent to the image display module, and the video image is displayed on the distributed screen. After device C performs audio decoding, it is sent to the sound playback module, and the sound is played through the distributed audio.

[0117] In this scenario, audio and video are encoded and transmitted separately and through different transmission channels. On the one hand, the data processing processes such as audio and video encoding and decoding take different amounts of time. On the other hand, when transmitted through different transmission channels, the communication status may be different and network jitter may cause the start time to be more random. Therefore, when audio and video are separated, it is difficult to synchronize the playback timestamps between different devices.

[0118] For example, Figure 7 Taking the system architecture shown in the figure as an example, the first solution is explained in detail:

[0119] The PTS (Presentation Time Stamp) corresponding to the audio frame or video frame obtained after decoding by the playback end is the original timestamp generated by the source encoder, not the actual playback time. Therefore, it is necessary to know the actual playback time of each frame data on one of the devices and achieve synchronization between multiple devices based on the actual playback time.

[0120] Taking the example of two transmission channels transmitting data streams, namely audio data stream and video data stream, respectively, in this case, considering that people have different sensitivities to images and sounds, and are often more sensitive to audio than video, after the audio playback device at the playback end receives the audio data stream and the cache reaches a certain length, the audio frame can be played normally first, and then the actual playback time of the audio frame is used as a reference to adjust the playback of the video frame on the video playback device to achieve synchronization of audio and video.

[0121] Specifically, Figure 8 As shown, the audio and video synchronization modules are deployed in the distributed hardware subsystems of the source device and the playback device respectively, and the audio and video synchronization modules on different devices play different roles.

[0122] In some embodiments, the source device A adds a first audio-visual synchronization module, which includes at least a multi-channel management module; the playback device B adds a second audio-visual synchronization module, which includes at least a frame rate control module; the playback device C adds a third audio-visual synchronization module, which includes at least a delay feedback module.

[0123] Optionally, since the video and audio processing processes take different amounts of time, a data cache module may be added to the audio and video synchronization module of the playback end to ensure smooth playback and offset the impact of uneven packet transmission and reception caused by network jitter. In some embodiments, the data cache module may not be provided.

[0124] Exemplarily, the second audio-visual synchronization module includes a first state feedback module, a first data buffer module and a frame rate control module. The third audio-visual synchronization module includes a second state feedback module, a second data buffer module and a delay feedback module.

[0125] In other embodiments, the second audio and video synchronization module and the third audio and video synchronization module may not be provided with a data cache module, or the first state feedback module and the frame rate control module may be integrated into the same module, or the second state feedback module and the delay feedback module may be integrated into the same module. On the premise of realizing the corresponding functions, a variety of specific module settings can be obtained through formal changes, which are not listed one by one in this application specification.

[0126] Similarly, in a specific implementation, the synchronization policy management, multi-channel management module and status monitoring module in the source device can also be integrated into the same module or into two modules for implementation. For example, the synchronization policy management and multi-channel management are integrated into a management module, or the status monitoring module and the synchronization policy management are integrated into the same module, or the synchronization policy management, multi-channel management and status monitoring modules are integrated into the same module, and so on. Other specific implementation methods will not be described in detail in this application specification.

[0127] The synchronization strategy management module determines the synchronization strategy according to the type of data to be played.

[0128] The synchronization policy management module can deploy a policy library or set an interface that can access the policy library. The policy library stores synchronization policies corresponding to various data types. The data type includes the data format information to be played by different devices, for example, the data to be played contains only audio format data, or only video format data, or both audio data and video data, or other formats of data in addition to audio and video data.

[0129] Human hearing is more sensitive than vision to delays and freezes, and generally, the audio output is maintained as normal as possible. In some embodiments, for data types containing audio data and video data, the synchronization strategy adopted is to use audio as the main and video as the secondary, that is, the audio frame playback is used as a reference, and the playback progress of the video frame follows the audio frame.

[0130] In the case where two or more data streams are both video, or two or more data streams are both audio, one device can be selected from multiple devices at the playback end as the master, and the other devices as slaves.

[0131] The following mainly uses one channel of audio data and one channel of video data as an example for exemplary description. For other data types, the corresponding synchronization strategies can be adaptively obtained by referring to the exemplary description.

[0132] The video playback device (device B) identifies the data type of the current data stream to be played as video through the first state feedback module, monitors the connection state of the transmission channel from device A to device B, and feeds back the data type information and connection state information to the state monitoring module in the source device A. Similarly, the audio playback device (device C) identifies the data type of the current data stream to be played as audio through the second state feedback module, monitors the current connection state of the transmission channel, and feeds back the data type information and connection state information to the state monitoring module in the source device A.

[0133] The status monitoring module in device A first determines whether to continue transmitting data based on the connection status information of the transmission channel fed back by devices B and C. The connection status of the transmission channel, that is, the communication status of the transmission channel, is whether data can be sent normally. For example, if the connection status information shows that the communication is disconnected, then it is obviously impossible to achieve synchronous playback of subsequent data. In this case, you can wait for the communication to be restored, pause the playback of each device first, and prompt the user that the network is disconnected or prompt that the playback has failed. After waiting for the transmission channel to resume normal communication, adopt the corresponding synchronization strategy.

[0134] When the connection status shows that the communication is normal, the status monitoring module updates the data type information of the current data to be played in real time according to the data type information of the two data streams fed back by device B and device C, and inputs it into the synchronization strategy management module.

[0135] Different data types may use different synchronization strategies. For example, if one channel is audio and the other is video, or if both channels are video, the specific synchronization strategies may be different. Therefore, the source device must first know the type of data to be played, and then determine the synchronization strategy corresponding to the data type through the synchronization strategy management module.

[0136] After the synchronization strategy is determined, the synchronization strategy management module will trigger the corresponding synchronization strategy, for example, triggering the multi-channel management module to receive the actual playback time information of each audio frame sent by device C.

[0137] In order to reduce the number of information transmissions between the playback device and the source device, device C can actively feedback the actual playback time of the audio frame to device A through the delay feedback module. For example, the actual playback time of the mth audio frame just played is recorded as m_clock and sent to device A.

[0138] After receiving m_clock, device A immediately forwards it to device B.

[0139] Device B uses the frame rate control module to compare the actual playback time m_clock of the received audio frame with at least the playback timestamp (Vpts) time of the corresponding video frame, and controls the playback progress of the video frame according to the comparison result.

[0140] Optionally, the first data cache module is used to cache video data, smooth out the encoding and decoding delay and network jitter, and the video frames in the queue will be used as input to the frame rate control module. The second data cache module caches audio data to smooth out the impact of encoding and decoding delay and network jitter on playback time.

[0141] like Figure 8As shown, the audio playback device (device C) obtains the actual playback time m_clock of the audio frame through the delay feedback module, and feeds back the actual playback time to the multi-channel management module of the source device A. The multi-channel management module then feeds back m_clock to the frame rate control module of the video playback device (device A).

[0142] like Fig.10 As shown, as an implementable method, in the audio playback device, every time a frame of audio is played, the delay feedback module writes the actual playback time of the played frame into the main clock m_clock. Of course, in other embodiments, the actual playback time of each audio frame in these audio frames can be written once after playing several audio frames to reduce the delay caused by frequent writing. Or write once every several audio frames, and only select the actual playback time of one audio frame to write, for example, only write the actual playback time of one audio frame in every 2 audio frames.

[0143] like Fig. 9 As shown, in a distributed business scenario, there is no transmission channel for direct communication between different devices on the playback end, and it is difficult for the video playback device to obtain the actual playback time information of the audio frame on the audio playback device. If the audio playback device and the video playback device are directly allowed to directly perform delay feedback, it is necessary to establish a corresponding transmission channel, or to perform two forwardings through the source end. For example, the video playback device first needs to send a request, which is a forwarding, and then the audio playback device feeds back the playback time information, which generates a second forwarding. New links or secondary forwarding will cause greater link pressure. Therefore, in some embodiments of the present application, the delay feedback module of the audio playback device actively sends the audio playback time information to the source end, and the source end forwards the audio playback time information to other devices. After one forwarding, other devices can obtain the audio playback time information of the audio playback device, thereby reducing link pressure.

[0144] It should be noted that in other embodiments, it is not excluded to establish a transmission channel between the video playback device and the audio playback device or to forward the information twice by the source end so that the video playback device and other devices can obtain the audio playback information.

[0145] After the video playback device obtains the actual playback time of the audio frame, the frame rate control module takes out an image frame from the cache queue, reads the playback timestamp Vpts of the image frame, and calculates the playback time difference diff based on at least Vpts and m_clock.

[0146] like Fig.10 As shown, after the frame rate control module calculates the diff, it determines the processing strategy for the current video frame to be played according to the diff, such as whether to continue waiting, perform frame drop processing, or directly send it for display.

[0147] Optionally, in some embodiments, the frame rate control module takes out a frame of image from the cache queue, which may be a frame of image taken out at fixed intervals according to a set frame rate (Frame Rate). The interval time t0 is determined according to the frame rate Fr, t0 = n*1 / Fr, n is an integer equal to or greater than 1, and n is used to control the proportion of video frames taken out from the cache queue. If n = 1, it is equivalent to taking out frame by frame from the cache queue, and each video frame in the cache queue must calculate the playback time difference diff to determine whether to send it for display, whether to drop the frame, etc.; n = 2, which means that one frame is taken out of every 2 frames in the cache queue to calculate the playback time difference diff; n = 3 means that one frame is taken out of every 3 frames in the cache queue to calculate the playback time difference diff. The larger n is, the lower the accuracy of synchronization control, and the synchronization effect may decrease as the value of n increases. In some embodiments, n≤3.

[0148] For example, if n=1 and the frame rate is 30fps (frames per second), then t0=1*1 / 30≈0.033s≈33ms, that is, a frame is taken out from the cache queue every 33ms, which is equivalent to taking out frame by frame, calculating the playback time difference with the audio frame frame by frame, and then judging whether to send it to display. If the frame rate is 60fps, then t0=1*1 / 60≈0.017s≈16.7ms, and a frame is taken out from the cache queue every 16.7ms.

[0149] It should be noted that the next video frame currently taken out may not be two data frames that need to be played synchronously with the mth audio frame currently received. Therefore, the difference calculation for the purpose of obtaining the playback time difference should be calculated based on the audio frame and video frame with a mapping relationship. Video frames and audio frames with a mapping relationship refer to audio frames and video frames that present a synchronous playback effect in the playback timing, which are considered to have a mapping relationship. Specifically, the mapping relationship can be determined based on at least one of the playback timestamp PTS and the sequence number.

[0150] Since the frame rates of video and audio are generally different, for example, if device C writes a clock every time it plays an audio frame, then the frequency of sending the clock may be the same as or close to the frame rate. For example, if the audio frame rate is 50fps, a clock is sent every 20ms, so the actual playback time of the m+1th audio frame is m+1_clock. On the video playback device, for example, if the video frame rate is 30fps, a frame is taken from the cache queue every 33ms, so the frequency of taking video frames and the frequency of receiving clocks are different.

[0151] The comparison operation to determine whether there is a mapping relationship can be triggered by taking out a video frame, that is, each time a video frame is taken out, the mapping relationship is identified with the latest received clock, such as reading the serial number and judging whether there is a mapping relationship based on the serial number. This requires that not only PTS but also the serial number should be added at the source device.

[0152] As an implementable method, in addition to adding a serial number or PTS, a serial number or PTS having a mapping relationship with the current audio and video frame or audio frame may also be recorded. That is, in some embodiments, the serial number or PTS of other data frames having a mapping relationship with the current data frame may be recorded for each data frame or every few data frames. In this way, there is no need to perform difference calculation on the playback end, and the serial number or PTS having a mapping relationship may be directly read to determine whether there is a mapping relationship.

[0153] For example, before sending the video data stream on device A, for each video frame, in addition to adding Vpts, a serial number is also added, and information about other serial numbers that have a mapping relationship with it is added, which can be added in the form of an array. For example, the serial number of the current video frame is 80, and the serial numbers of at least one audio frame that has a mapping relationship with it are 98 and 99 respectively. Then the added serial number array can be [80:98,99], where 80 represents the serial number of the video frame itself, and 98 and 99 represent the serial numbers of the two audio frames that have a mapping relationship with the video frame. At the playback end, after the audio data stream packet and the video data stream packet are decoded, the serial number or serial number array corresponding to each data frame can be read.

[0154] In addition, as another possible implementation method, PTS is used to indicate the playback timing, which is actually a kind of sequence information itself. Therefore, it is not ruled out that the mapping relationship between audio frames and video frames can be determined by the playback timestamp PTS. The source end will encode before sending data. Most encoders will generate a playback timestamp PTS for each frame during encoding. This timestamp is used to mark the playback order of each frame. Based on this playback order, the mapping relationship between video frames and audio frames can be determined. That is, the mapping relationship is determined based on Vpts and Apts (Audio Presentation Time Stamp, audio playback timestamp).

[0155] For example, before sending data, the source device adds a playback timestamp PTS to each data frame. Generally, due to the different sampling rates of video frame rate and audio, video frames and audio frames are not mapped one to one. Assuming that the audio is sampled for one frame at 20ms and the video frame rate is 30fps, that is, one frame is generated every 33ms, and assuming that the timestamp at the start of audio acquisition is 80, then the approximate increasing pattern of the PTS of the video frame and the audio frame is: video frame VPTS: 109, 142, 175, 208, 241, 274...; audio frame APTS: 80, 100, 120, 140, 160, 180..., if at the playback end, it can be determined that the clock frequency of PTS is consistent with the time reference benchmark (such as Greenwich Mean Time), then the mapping relationship between the audio frame and the video frame can be directly determined by calculating the difference in timestamps. Alternatively, the desired playback time information may be obtained based on the playback order indicated by the playback timestamp in combination with information such as the time base time_base or the clock frequency (crystal frequency). Based on the desired playback time information, the mapping relationship between the audio frame and the video frame may also be determined.

[0156] For example, the audio frame and the video frame whose PTS difference is less than a predetermined mapping threshold are considered to have a mapping relationship. For example, the specified threshold can be 33. For example, the PTS of the first audio frame in the above sequence is 80, and the PTS of the first video frame is 109. The difference between the two is less than 33, so it can be considered that there is a mapping relationship between the two. The mapping threshold is not unique. For example, when the video frame rate is 60fps, the mapping threshold can be 20.

[0157] It should be noted that in this implementation, device C needs to feed back the original Apts of the audio frame while feeding back the actual playback time clock, that is, the information used to send the clock carries Apts. In this way, the video playback device can first read the Apts of the audio frame, compare Vpts and Apts, and if the difference between VPts and Apts is less than , it is considered that the two have a mapping relationship. For example, after being converted into time information, if the difference is less than 33ms, it is considered that the two have a mapping relationship.

[0158] In actual applications, it is possible that the next video frame currently taken out from the cache queue has no mapping relationship with the latest received m_clock. In this case, the order relationship between the audio frame that has a mapping relationship with the currently taken out video frame and the currently received m_clock can be determined based on the sequence number or PTS. Both the sequence number and PTS can express the order information of the playback sequence. Therefore, by comparing the sequence numbers or PTS of the video frame and the audio frame, even if the mapping relationship cannot be determined, the order relationship can be identified. If the currently received mth audio frame should be played before the currently taken out video frame, that is, the audio frame that should be played synchronously with the current video frame is after the current mth audio frame, then the video playback device can wait for the subsequent audio frame to be received. If the mth audio frame should be played after the currently taken out video frame, then the audio frame that has a mapping relationship with the current video frame can be searched in the multiple clocks that have been received, and its actual playback time can be read, and then the difference calculation can be performed.

[0159] It should be noted that PTS is added by the source end to indicate the relative order of the playback timing of each data frame, but it is not the actual playback time. Therefore, Vpts and Apts can be used to determine the mapping relationship, but cannot be directly used for synchronization control.

[0160] It should be noted that the above-mentioned serial number is different from the presentation time stamp PTS (Presentation Time Stamp) and cannot be understood as an equivalent technical concept of the presentation time stamp PTS.

[0161] Assume that there is a mapping relationship between the nth video frame and the mth audio frame taken from the cache queue, and then explain how to calculate the play time difference diff. The following calculations are all performed based on the audio frame and the video frame with a mapping relationship, and for the sake of brevity, they will not be repeated.

[0162] Optionally, as a feasible implementation, in some embodiments, the play time difference diff can be determined based on Vpts and m_clock, for example:

[0163] diff=Vpts-m_clock

[0164] Among them, m_clock represents the actual playback time of the mth audio frame. Every time an audio frame is played, the timestamp of the playback frame is written into the master clock. Vpts is the playback timestamp of the nth frame taken out from the video cache queue.

[0165] Optionally, as another feasible implementation mode, in another part of the embodiments, the determination of the playback time difference diff may also include factors such as post-processing duration.

[0166] For example, diff can be calculated as follows:

[0167] diff=Vpts-m_clock+(Y3-X3)

[0168] Among them, X3 is the post-processing duration corresponding to the video (second processing duration), and Y3 is the post-processing duration corresponding to the audio (first processing duration).

[0169] Exemplarily, X3 can represent the time corresponding to the process of drawing, rendering, composing and displaying a frame of image, that is, the total execution time of multiple threads such as drawing, rendering, image frame composition and image frame display corresponding to a frame of image. This time can be obtained by calculating the time consumed by each thread node. For example, X3 = T1 + T2 + T3 + T4, where T1 is the drawing time, T2 is the rendering time, T3 is the image frame composition time, and T4 is the image frame display time. Image frame is video frame.

[0170] In the embodiment of playing the video through WMS (Window Manager Service), WMS is used to provide image rendering services, etc. X3 can be understood as the time from image drawing and rendering to final display after the image is sent to WMS. X3 can be obtained by accessing WMS.

[0171] Exemplarily, Y3 can be understood as the time from when the audio data is sent to the audio framework, through the audio driver, to when it is finally played, which can be obtained by calculating the time consumed by each thread during the audio playback process. Specifically, as an implementable method, Y3 can be obtained by accessing the audio framework.

[0172] For example, Y3=T5+T6+T7+T8+T9+T10, where T6 to T10 respectively represent the duration of mixing processing, sound effect processing, volume processing, sending to the driver, sending to the DSP (Digital Signal Processor), and speaker playback.

[0173] Exemplarily, the duration of a single thread can be obtained as follows: when the thread starts running, the system time is obtained as the start time. After the thread finishes running, the system time is obtained again at the end of the thread. This time minus the start time is the operation time. Record a start time and an end time. The subtraction of the two is the program running time. In an embodiment of the present application, the post-processing process of video and audio may include multiple threads. Optionally, the start time of the first thread to perform the corresponding operation can be used as the start time. After the last thread finishes executing, the system time is obtained again. The two are subtracted to obtain the running time of multiple threads as the post-processing duration. For example, when the drawing thread starts to execute the drawing action, the system time is recorded as the start time, and then when the image frame starts to be displayed, it is recorded as the end time. The two are subtracted to obtain X3. For example, Fig.11 As shown, X3=t3-t1.

[0174] It should be noted that due to differences in operating systems and hardware devices, the processes of audio post-processing and video post-processing may be different. The above is only an example. For example, rendering is not a necessary step in the video post-processing process. When the device hardware configuration is powerful enough, rendering is not required. Similarly, sound effect processing, volume processing, etc. are also optional in actual applications.

[0175] In application scenarios where the processing flow is different from the above example, the post-processing duration can be understood as the length of time from when the video frame or audio frame is determined to be sent to when the video frame or audio frame is actually played. As an implementation method, the system time can be recorded when the sending is determined as the start time, and the start of the playback can be used as the end time to obtain Y3 or X3.

[0176] When it is difficult to determine the time point for sending and broadcasting, a more universal way to determine the post-processing duration is to take out a video frame or an audio frame from the cache queue as the starting time, and take the actual playback of the video frame or the audio frame as the ending time, thereby obtaining X3 or Y3. That is, the post-processing duration (such as the first processing duration or the second processing duration) can be an estimated value of the processing duration from the retrieval of a data frame from the cache queue to the playback.

[0177] It should be noted that Y3 and X3 can be a constant determined by statistically averaging multiple played video frames or audio frames, and can be written into the program as fixed parameters. It is not necessary to calculate the post-processing duration once before each frame is played in actual operation. For example, in the SRS (simple Rtmp Server) 4.0 code architecture, Y3 = 40ms. When calculating diff, Y3 is a fixed constant of 40ms.

[0178] It should be noted that the above-mentioned implementation method for determining X3 and Y3 is only an example. More generally, X3 can be a time length determined by taking out a video frame from the cache queue as the starting moment and playing it as the end moment, and Y3 can be a time length determined by taking out an audio frame from the cache queue as the starting moment and playing the audio frame as the end moment.

[0179] In the embodiment of the present application, the timestamp used to calculate diff, such as Vpts or Apts, can be converted into time. Due to the differences in encoding formats and decoding formats, in some encoding and decoding formats, the timestamp Vpts or Apts may be represented by a series of numerical values, which are used to notify the playback end of the relative order of playback of each frame, and cannot directly represent the time. Therefore, it is necessary to convert the timestamp into time, and then use the above calculation formula to calculate to obtain diff.

[0180] For example, the PTS of multiple video frames in a video: 480 640 560 520 600 800 720 680 760960, where the values ​​do not represent time. In FFmpeg, it is necessary to convert PTS into time information according to the respective time bases (time_base) of audio and video, and then calculate it using the above formula. For example, the playback time of an audio frame Tv = time_base*PTS, where time_base represents the time scale corresponding to the value 1 in PTS.

[0181] After obtaining the diff, the play control strategy for the currently retrieved video frame is determined according to the diff. The play control strategy may include sending to display, waiting, dropping frames, reducing the play rate, increasing the play rate, etc.

[0182] When the video and audio are played together, if the playback time difference is within a certain threshold, human senses will not obviously perceive the asynchrony, so the sense of lag is not obvious, and the audio-visual experience will not be reduced; if the playback time difference exceeds a certain threshold, a more obvious lag may occur, reducing the audio-visual experience. In the embodiment of the present application, a certain delay can be allowed for the synchronization of audio and video playback, and the delay must be within an acceptable range.

[0183] In the embodiment of the present application, the playback control strategy for the current video frame is determined by comparing diff with a predetermined threshold range (first threshold interval).

[0184] For example, min is set as the maximum threshold of audio lagging behind video, and max is set as the maximum threshold of audio leading ahead of video. Optionally, a feasible playback control strategy may be:

[0185] If diff < min, it means the video frame is faster than the audio frame. In this case, the video frame needs to wait for a few milliseconds for the master clock. For example, the waiting time can be (min - diff) milliseconds.

[0186] If diff > max, it means the video frame is slower than the audio frame. In this case, the video frame needs to catch up. The catching-up strategy can be discarding video frames, or increasing the playback rate, etc.

[0187] If min ≤ diff ≤ max, it means within the synchronization threshold range, no obvious asynchrony will occur, and the video frame can be directly sent for display.

[0188] Optionally, the value of min can specifically be -190 ms to -180 ms, and the value of max can be 85 ms to 95 ms.

[0189] The following lists a specific embodiment to exemplarily illustrate how to achieve playback synchronization between different devices at the playback end in a one-to-many casting scenario using the first solution.

[0190] Combined with Fig.10 and Figure 8 As shown, device B and device C at the playback end respectively feedback the data type information to be played back to the status monitoring module in the source device A through the first status feedback module and the second status feedback module. For example, the data type feedback by device B is video, and the data type feedback by device C is audio.

[0191] The status monitoring module in device A inputs this data type information into the synchronization policy management module, and the synchronization policy management module selects the corresponding synchronization policy from the policy library according to this data type information.

[0192] In an actual distributed service scenario, due to the increase in the number of devices at the playback end and the source end, the combination of data types is diverse. The setting of the synchronization policy management module can achieve the selection of different synchronization policies for different data types, and thus can be applied to more complex application scenarios.

[0193] Optionally, when the data types of multiple transmission channels include at least one audio and at least one video, one of the audio channels is taken as the master and the others are taken as the slaves. Exemplarily, when the data types to be played back currently are one audio channel and another video channel, the synchronization policy adopted is to perform master-slave setting, set the audio as the master and the video as the slave, and the video performs frame rate control with reference to the audio playback time.

[0194] Optionally, when the data type of the multi-channel transmission channel includes multiple channels of the same type of data, such as two channels of data are video or two channels of data are audio, you can select one of them as the main channel, which is equivalent to selecting one of the devices on the playback end as the main channel and the other devices as slaves.

[0195] Optionally, when the playback end is a multiple device, the audio frame is automatically played after caching a fixed number of frames, and the video and audio no longer wait for each other. In this embodiment, the caching time of the audio frame can be set to be greater than the caching time in the single device scenario, that is, greater than the caching time in the one-to-one broadcast application scenario. For example, if the caching time of the playback end is a single device and is 2 minutes, then the caching time of the data caching module in the one-to-many broadcast scenario is set to be greater than 2 minutes.

[0196] After determining the synchronization strategy, the actual playback time information m_clock sent by the delay feedback module of device C is monitored by the multi-channel management module, and the actual playback time information is forwarded to the frame rate control module of device B as the input of the video playback frame rate control.

[0197] In this embodiment, the video frame rate is 30 frames / s. The frame rate control module takes out a frame of image corresponding to the current audio frame from the cache queue every 33ms, reads its play timestamp Vpts, for example, reads the play timestamp Vpts of the mth video frame from the cache queue, and then calculates diff based on the following formula:

[0198] diff=Vpts-m_clock+(Y3-X3)

[0199] After calculating the playback time difference diff, the diff is compared with a predetermined threshold range to determine whether the current video frame should be displayed.

[0200] The higher the frequency of taking images from the cache queue, the higher the accuracy of synchronization control. Among them, calculating diff frame by frame can improve the accuracy of synchronization control. The smaller the diff, the smaller the relative delay and the better the synchronization effect. By controlling the frequency of taking image frames from the cache queue, the accuracy of synchronization can be controlled. When diff is less than the inter-frame interval or even close to 0, inter-frame synchronization can be achieved.

[0201] In this embodiment, the minimum value is -185 ms and the maximum value is 90 ms.

[0202] If diff is less than min, then wait. For example, if diff is -205ms, then wait for -185ms-(-205)ms=20ms before displaying.

[0203] If -185ms≤diff≤90ms, the image is sent directly to the display. Sending means transferring the taken image frame to the display driver for display on the display.

[0204] If diff>90ms, the video needs to catch up with the audio. The control strategy adopted at this time can be to drop frames or increase the video playback rate.

[0205] Optionally, in order to ensure the continuity of the playback data and avoid problems such as swallowing of sound or picture jumps, try not to use the frame dropping method for synchronization, and consider using the playback rate change method for synchronization.

[0206] For example, optionally, if the video frame arrives too late (greater than 90ms) and the delayed frame arrives at the same time, the frame can be dropped and the next frame can be displayed directly; or, if the video frame is advanced by more than 185ms, waiting for the next scheduling may cause the buffer to be full, then the frame needs to be dropped, and the frame dropping prioritizes P frames (predictive-frame, forward predictive coding frames) and B frames (bi-directional interpolated prediction frame, bidirectional predictive interpolated coding frames).

[0207] When the arrival of audio frames is delayed or the audio is played slowly, video frames are allowed to be displayed in advance, with a maximum advance margin of 185ms.

[0208] Or, optionally, the audio arrives too late and all audio frames in the queue have been sent, which will inevitably cause intermittent sound. To ensure user experience and prevent intermittent sound, when the amount of audio queue data is less than a certain value, the audio playback rate is reduced, the waiting time for audio frames is increased, and the normal playback speed is restored after the sending queue exceeds a certain value, and frame loss is avoided as much as possible. Slow playback is based on the premise that it does not affect the user's obvious perception, and the specific slow playback ratio is determined according to the specific value of diff.

[0209] Frame dropping will cause information loss, so fast or slow playback can achieve synchronization, which can retain more information and reduce data loss compared to frame dropping. However, in some cases, frame dropping is the only option.

[0210] Optionally, if the audio cache queue is full and the first video frame has not arrived, the audio frame can only be dropped or the broadcast failure is returned.

[0211] The above solution uses the actual playback time of the audio playback device at the playback end as a reference to control the playback time of other devices such as the video playback device. The following describes another solution that can achieve multi-device synchronous playback in a one-to-many broadcast application scenario.

[0212] The synchronization of video frames to audio frames requires the video playback device to accurately know the time when the audio frames are played, and then adjust its own playback timing. If the playback end is in a single-device scenario, the video playback module can easily obtain the audio playback time information of the same device. However, in a scenario where audio and video are separated, the video playback device cannot obtain the playback time of the audio frame in real time. If the video frame obtains the playback time of the audio frame across devices, it needs to be forwarded across devices twice at the source end, which has a large delay and is unreliable. In the first solution mentioned above, the source end needs to forward once, while in the second solution in a one-to-many scenario provided in an embodiment of the present application, the source end does not need to forward, but the audio playback device only needs to provide a delay feedback to the source end, and the source end does not need to feed back the delay to the video playback device.

[0213] Specifically, Figure 7 The system architecture shown in FIG. 1 is used as an example to illustrate. The solution can be implemented based on a control center deployed on the source end and the playback end device. Specifically, Fig.12 As shown, for a clear description, the source device A is deployed with a first control center, the audio playback device C is deployed with a second control center, and the video playback device B is deployed with a third control center.

[0214] Devices A, B, and C must first synchronize their clocks.

[0215] Performing clock synchronization first can offset or reduce the asynchrony caused by different device clocks.

[0216] The audio playback device (device C) determines the delay information of the audio frame and feeds it back to device A. The delay information can be the transmission delay and decoding delay of the last audio frame played, or the average transmission delay and decoding delay of multiple audio frames that have been played, which are fed back to the first control center in the source device A.

[0217] In source device A, before encoding the video stream, a serial number and a playback timestamp Vpts are added to each frame. After encoding by the encoder, the first control center estimates the playback time of the video frame based on the delay information of the audio frame to obtain the estimated playback time Vcts' of the video frame. The estimated playback time is used to replace the original playback timestamp Vpts to offset the delay difference between the video frame and the audio frame, thereby achieving synchronous playback.

[0218] It should be noted that the video playback timestamp Vpts or the audio playback timestamp Apts is used to guide the playback end to play in sequence according to the timing indicated by the timestamp. It is generated by the source device and is generally added to each frame of data by the source device before or during encoding.

[0219] The first control center first calculates Vcts' in combination with the delay information fed back by the second control center, and then updates the play timestamp of each video frame in the video stream to be transmitted to Vcts' before the video stream is sent.

[0220] A specific embodiment is listed below to illustrate the invention.

[0221] When the playback end is a single device, the video is synchronized with the audio of the same device, and the audio delivery time Acts can be obtained. However, when the audio and video playback are on different devices, the video playback device cannot obtain the Acts of the audio playback device. The audio delivery time Acts can be understood as an estimated value of the playback time.

[0222] Multiple devices in a distributed scenario first synchronize their clocks. The source device A transmits the video data and audio data to be transmitted to the playback device B and device C respectively.

[0223] After a certain number of audio frames are cached, they are played at the set frame rate.

[0224] In some embodiments, the audio playback device C determines the delay information Tasd and then feeds it back to the device A.

[0225] Ignoring other factors that can be ignored, the relationship between audio transmission and decoding delay Tasd and audio playback time Acts and audio sending time is as follows:

[0226] Tasd = Acts - Tas.

[0227] When the video playback device at the playback end cannot obtain Acts, the source device A can obtain the sending time point Tas of the audio frame at the source end, and then the estimated value of Acts can be obtained based on Tas and Tasd.

[0228] Acts=Tas+Tasd

[0229] Acts is the estimated audio frame playback time at the source end, Tas is the audio frame sending time at the source end, and Tasd is the audio frame transmission delay.

[0230] Exemplarily, in this embodiment, the audio frame transmission delay Tasd includes transmission delay and decoding delay, and the sum of the transmission delay and decoding delay of multiple audio frames played by the audio playback device within a period of time can be used to calculate the average.

[0231] In other embodiments, the audio playback device may not need to feed back the delay information to the source device, but instead obtain Tas after determining the delay Tasd, and then calculate Acts on the audio playback device C, and then device C directly feeds back Acts to device A.

[0232] Tasd is a historical result, and may not be completely consistent with the delay generated under the current actual network status. Therefore, the credibility is not 100%, which may cause a certain error in audio and video synchronization. Optionally, the source device can perform filtering processing, filter the Tasd of multiple audio frames obtained, and filter out the smoothest value to reduce the large error caused by local fluctuations.

[0233] The play time difference between the video frame and the audio frame with a mapping relationship can be calculated by the following formula:

[0234] diff=(Vcts–Acts)–(Vpts–Apts)

[0235] Vcts is the video frame transmission time, which can be understood as an estimated value of the video frame playback time. Apts and Vpts are the playback timestamps added to the audio and video by device A before encoding.

[0236] There is a mapping relationship, that is, the video frame and the audio frame have the same or similar sequence number. Alternatively, according to the play timestamp, it can be determined that the two are audio frames and video frames played at the same time sequence, and the corresponding audio frames and video frames have a mapping relationship.

[0237] The goal of synchronous playback is: min≤diff≤max. In this embodiment, the min value is -186ms and the max value is 88ms.

[0238] The source device A obtains the sending time of the current audio frame and video frame as Tas and Tvs respectively;

[0239] Ignoring the error introduced by the current frame information, Acts = Tas + Tasd, and substituting it into the above calculation formula, the goal of synchronous playback is:

[0240] min≤Vcts-(Tas+Tasd)-(Vpts-Apts)≤max

[0241] Assume that when diff = 0, Vcts = Vcts', that is, Vcts'-(Tas+Tasd)-(Vpts-Apts) = 0ms, then

[0242] Vcts'=Tas+Tasd+Vpts–Apts;

[0243] Among them, Apts and Vpts can be obtained by the source device A.

[0244] Vcts' is the estimated video playback time when diff=0.

[0245] In some embodiments, before device A sends a video frame, the playback timestamp Vpts of the current frame may be updated to Vcts'.

[0246] In another embodiment, before device A sends a video frame, Vcts' can be added based on the original playback timestamp Vpts, so as not to replace the original Vpts but to add an estimated value, which can avoid a series of problems caused by deleting the original Vpts.

[0247] After decoding, the video playback device (device B) at the playback end only needs to compare Vcts' with the current time to decide whether to send it for display. For example, the current time is Tn, diff'=Vcts'-Tn, and then by comparing diff' with [-186ms, 88ms], the corresponding processing strategy can be obtained. The details can be seen in the above embodiment, which will not be repeated here.

[0248] Optionally, in the second solution, X3 and Y3 can also be introduced to determine the play time difference. The corresponding calculation formula is as follows:

[0249] diff=(Vcts–Acts)–(Vpts–Apts)+(Y3–X3)

[0250] So

[0251] Vcts'=Tas+Tasd+Vpts–Apts+(X3–Y3).

[0252] Similarly, in this embodiment, when the playback end is multiple devices, the audio frame is automatically played after caching a fixed number of frames, and the video and audio no longer wait for each other. The cache time can be set to be longer than the single-device scenario.

[0253] The above description is mostly exemplified by at least one channel being audio and at least one channel being video. Based on the above description, those skilled in the art can adaptively obtain solutions for the case where the multiple channels of data to be transmitted in a one-to-many broadcast scenario are all audio, or the multiple channels of data to be transmitted are all video. For example, as shown in FIG4(d), in an application scenario where both channels of data are images, the playback time of the playback device of one of the images can be used as a reference to control the playback time of the other device. For example, according to the above solution one, the actual playback time of each image frame on device C is used as a reference time to control the playback time of the corresponding image frame on device B; or, according to the above solution two, based on the transmission and decoding delay of device C, the playback time of the image frame on device B is estimated, and the playback timestamp is updated to control the playback time of the image on device B and achieve synchronization between device B and device C. By analogy, for the situation shown in FIG4(e), corresponding solutions can also be adaptively obtained, and this application specification does not elaborate one by one.

[0254] The above introduces the synchronous playback control solution in a one-to-many scenario. The following will continue to explain the synchronous playback control solution in a many-to-one scenario.

[0255] In the scenario where the source end is multiple devices and the playback end is a single device, for example Fig.13 In the example, device A sends audio, device B sends video, and the only player is device C. The video receiver on device C can accurately obtain the audio playback time, but the source is located on two devices, and the synchronization information between them cannot be easily obtained, so cross-device interaction is required.

[0256] In the many-to-one scenario, the audio and video synchronization process requires the distributed hardware framework to first set the synchronization group information, and then synchronize the synchronization group information between multiple devices through the control center, and then perform data transmission and playback.

[0257] First, the clocks of the three devices need to be synchronized, that is, device A and device B are synchronized to device C. After the clocks are synchronized, Apts and Vpts are added to the frame header information of device A and device B.

[0258] Timestamps are added after clock synchronization to reduce asynchrony caused by inconsistent device clocks.

[0259] Whether the audio and video on different devices need to be synchronized cannot be determined by each device individually, and business layer settings are required.

[0260] A control center is set at the service layer, and interfaces are reserved in the control centers corresponding to device A, device B, and device C, respectively, and the interfaces are used to receive synchronization group information. The synchronization group information may include at least one of synchronization reference information and flow information.

[0261] Optionally, the synchronization reference information may include the identity information of each source device and the destination (playing end) device, for example, the identity information of source device A, device B and the identity information of playing end device C.

[0262] The identity information may be a code or symbol that can uniquely identify the device. For example, the identity information may be the Device ID of the device, such as the Device ID of a mobile phone, which is also the IMEI (International Mobile Equipment Identity) code of the mobile phone.

[0263] Optionally, the stream information may include information about the type of data currently being transmitted, for example, whether the type of data currently being transmitted is audio or video.

[0264] After the source generates the synchronization group information, at least two devices at the source set the synchronization group information to the local control center, and the source device synchronizes the synchronization group information to the control center of the playback device.

[0265] Here, it is sufficient to synchronize the synchronization group information between multiple devices on the source end and the devices on the playback end. There is no clear sequence limit on the synchronization between the devices on the source end and the synchronization between the source end and the playback end.

[0266] Next, the playback device selects and retains the data content that is consistent with the identity information of its own device based on the identity information of the receiving (playing) device in the synchronization group information. Then, it filters out the data type sent by each device, determines the data type sent by each device on the source side, and then groups the different devices on the source side according to the data type, and then performs audio and video synchronization.

[0267] A specific implementation example in a many-to-one broadcasting scenario is listed below.

[0268] For example, in a distributed business scenario, in the control center of each device on the source end and the playback end, a reserved interface SetSyncGroup (vector <pair<deviceIDs,streamInfo> >) to receive sync group information.

[0269] Optionally, synchronization group information can be set in the frame header.

[0270] The first member of the vector is the synchronization reference, which is used to notify the control center of the device ID of the current sending end (ie, the source end) and the device ID of the playback end after determining the device ID of the source end and the device ID of the playback end.

[0271] The second member is the stream information, which is used to inform the control center whether the data type currently being sent is audio data or video data.

[0272] Next, device A and device B set the synchronization group information to the local control center, and then device A and device B synchronize the synchronization group information to the control center of device C.

[0273] The synchronization of the synchronization group information enables multiple devices at the source end to interact with each other before sending data to obtain information such as the type of data that each other needs to send, establish a mapping relationship between audio frames and video frames, and synchronize the synchronization group information with the playback end device so that the playback end device can achieve synchronization of audio frames and image frames.

[0274] Device C identifies the Device ID of the receiving end (playing end) based on the synchronization reference information in the synchronization group information. If it is consistent with its own Device ID, it determines that the data is received by itself and retains it;

[0275] Device C filters out the data type sent by each device according to the flow information in the synchronization group information, for example, Fig.13 As shown, device C determines that device A sends audio data and device B sends video data, so it divides device A and device B into groups and synchronizes the audio and video of the two.

[0276] Optionally, since the receiving end only includes device C, device C can directly obtain Acts. At this time, the video data can be processed with reference to the audio data according to min≤Vcts-Acts-(Vpts-Apts)≤max.

[0277] Similar to the one-to-many broadcast scenario, technical personnel in this field can, based on the above description, adaptively obtain solutions for the many-to-one broadcast scenario where the multiple channels of data to be transmitted are all audio, or the multiple channels of data to be transmitted are all video, and this application specification does not go into details one by one.

[0278] According to the implementation schemes for the one-to-many scenario and the many-to-one scenario described above, an implementation scheme for the many-to-many scenario can be adaptively obtained. For example, in a case where the source end is multiple devices and the playback end is multiple devices, the sending time difference Δt between the multiple devices on the source end can be determined first, and Δt can be introduced for correction when calculating the diff or estimating the video frame playback time, thereby offsetting or reducing the impact of the asynchronous sending time of the source device on the playback end.

[0279] According to the description of the above embodiments, it can be known that the embodiments of the present application provide a synchronous playback method in a distributed service scenario, which can be applied to a one-to-many broadcasting scenario, such as Fig.14As shown, in a one-to-many broadcasting scenario, the playback end includes more than two devices, the two or more devices include a first device and a second device, and the source end includes a third device; the third device transmits the first data stream to the first device for playback, and transmits the second data stream to the second device for playback; the first data stream includes multiple first data frames; the second data stream includes multiple second data frames.

[0280] like Fig.15 As shown, from the perspective of the second device, the method includes the following process:

[0281] S141, receiving a first timestamp;

[0282] S142, reading a second timestamp corresponding to a second data frame to be played;

[0283] It should be noted that the step numbers are only used to distinguish different steps and do not limit the execution order. For example, there is no clear order limit for the execution sequence of S141 and S142. In actual applications, S142 may be executed before S141.

[0284] S143, calculating a first playback time difference based at least on the first timestamp and the second timestamp;

[0285] S144: Determine a synchronous playback strategy to be executed for the second data frame to be played based on the first playback time difference.

[0286] It should be noted that the first timestamp represents the actual playback time of the mth first data frame on the first device, where m≥1 and is an integer; the second timestamp is the playback timestamp PTS generated by the third device at the source end, which is used to represent the playback timing of each second data frame in the second data stream.

[0287] According to the above introduction, the first data stream and the second data stream can be the same type of data or different types of data. For example, the first data stream can be an audio data stream and the second data stream can be a video data stream. Alternatively, the first data stream and the second data stream can both be video data streams, for example, the first data stream and the second data stream are different parts of an image; or, the first data stream and the second data stream can both be audio data streams, for example, the first data stream is left channel data and the second data stream is right channel data, and so on.

[0288] Furthermore, the number of devices at the playback end may be three or more, and the number of data streams is not limited to two. The processing method for more than three data streams can refer to the processing method for two data streams. For example, in the case of two data streams, the second data stream refers to the first data stream, and for more than three multi-channel data streams, the other data streams except the first data stream can refer to the playback time of the first data stream.

[0289] Optionally, receiving the first timestamp may be receiving the first timestamp forwarded by a third device; the first timestamp is generated by the first device and sent to the third device. Forwarding once by the third device can reduce link communication pressure and save communication resources compared to establishing a data link between the second device and the first device or forwarding twice by the source device.

[0290] Optionally, the second timestamp corresponding to the second data frame to be played can be read by taking out the next second data frame to be played from the cache queue at a predetermined time interval and reading the second timestamp. The predetermined time interval refers to the description in the above embodiment, and the time interval should be determined based on the frame rate and is a multiple of the inverse proportion of the frame rate.

[0291] Optionally, the first playback time difference is calculated based on at least the first timestamp and the second timestamp. When it is determined that there is a mapping relationship between the next second data frame to be played and the mth first data frame, the first playback time difference can be calculated based on at least the first timestamp corresponding to the mth first data frame and the second timestamp corresponding to the next second data frame to be played.

[0292] Optionally, the first playback time difference is calculated based on at least the first timestamp and the second timestamp, and the first playback time difference can also be calculated based on the first timestamp, the second timestamp, the first processing time and the second processing time; wherein the first processing time is an estimated value of the processing time taken from when a first data frame is taken out of the cache queue to when it is played; and the second processing time is an estimated value of the processing time taken from when a second data frame is taken out of the cache queue to when it is played.

[0293] Optionally, the synchronous playback strategy includes at least one of waiting, sending and displaying, and catching up. Based on the first playback time difference, the synchronous playback strategy to be executed for the second data frame to be played is determined, which can be to compare the first playback time difference with a predetermined first threshold interval; when the first playback time difference is less than the minimum value of the threshold interval, it is determined that the playback of the first data stream lags behind the second data stream, and the synchronous playback strategy to be executed for the second data frame to be played is determined to be waiting; when the first playback time difference is greater than the maximum value of the threshold interval, it is determined that the playback of the first data stream is ahead of the second data stream, and the synchronous playback strategy to be executed for the second data frame to be played is determined to be catching up; when the first playback time difference belongs to the threshold interval, it is determined that the playback of the first data stream is relatively synchronized with the playback of the second data stream, and the synchronous playback strategy to be executed for the second data frame to be played is determined to be sending and displaying.

[0294] Optionally, waiting includes one or more of reducing the playback rate, waiting for a first predetermined time period, and dropping frames; wherein the first predetermined time period is determined based on the difference between the minimum value in the threshold interval and the first playback time difference; catching up includes increasing the playback rate and / or dropping frames.

[0295] An embodiment of the present application also provides a synchronous playback method in a distributed business scenario, which can be applied to a one-to-many broadcast scenario. The introduction of the one-to-many broadcast scenario refers to the above description and will not be repeated here.

[0296] The method can be performed based on a third device, see Fig.16 As shown, the specific process can be as follows:

[0297] S151, receiving delay information;

[0298] The delay information includes transmission delay and decoding delay; wherein the transmission delay represents the duration of the mth first data frame being transmitted from the third device to the first device; the decoding delay is the decoding duration occupied by the mth first data frame; m≥1 and is an integer;

[0299] S152, obtaining the sending time of the mth first data frame;

[0300] S153, obtaining an estimated value of the playback time of the mth first data frame on the first device at least according to the transmission time and delay information;

[0301] S154, before sending the second data frame, adding the estimated value as a fifth timestamp to the second data frame.

[0302] The fifth timestamp is used to perform difference calculation with the current time to obtain a second playback time difference, so as to determine a synchronous playback strategy to be executed for the second data frame to be played.

[0303] Adding the fifth timestamp to the second data frame may be replacing the original video playback timestamp Vpts with the fifth timestamp, or may not replace the original playback timestamp but add another fifth timestamp.

[0304] The step numbers in the embodiments of the present application are only used to distinguish different steps and do not limit the execution sequence.

[0305] Optionally, an estimated value of the playback time of the first data frame on the first device is obtained at least based on the sending time and delay information, and an estimated value of the playback time of the mth first data frame on the first device can be obtained at least based on the sending time and delay information of the mth first data frame; wherein m≥1 and is an integer; the mapping relationship is determined based on the sequence number and / or the playback timestamp PTS; and the estimated value is added to the second data frame as the fifth timestamp, which can be adding the estimated value as the fifth timestamp to the second data frame having a mapping relationship with the mth first data frame.

[0306] Optionally, obtaining an estimated value of the play time of the first data frame on the first device at least based on the sending time and the delay information may be obtaining an estimated value of the play time of the first data frame on the first device based on the sending time, the delay information, the third timestamp and the fourth timestamp. The third timestamp is a play timestamp PTS added to the first data frame, which is used to indicate the play timing of each first data frame in the first data stream; the fourth timestamp is a play timestamp PTS added to the second data frame, which is used to indicate the play timing of each second data frame in the second data stream.

[0307] The embodiment of the present application also provides a synchronous playback method in a distributed service scenario, which is applied to a one-to-many broadcasting scenario. The method can be executed based on a second device, such as Fig.17 As shown, the method may include the following process:

[0308] S161, reading a fifth timestamp corresponding to a second data frame to be played;

[0309] The fifth timestamp is an estimated value of the playback time of the first data frame on the first device, which is obtained at least according to the sending time and the delay information. The delay information includes the transmission delay and the decoding delay. The transmission delay indicates the time length of the first data frame being transmitted from the third device to the first device; the decoding delay is the decoding time length occupied by one first data frame; and the sending time is the time length of the first data frame being sent by the third device.

[0310] S162, calculating the difference between the fifth timestamp and the current time to obtain a second playback time difference.

[0311] S163: Determine a synchronous playback strategy to be executed for the second data frame to be played based on the second playback time difference.

[0312] Optionally, the synchronous play strategy includes at least one of waiting, sending and displaying, and catching up. The specific synchronous play strategy refers to the above description, which will not be repeated here.

[0313] The embodiment of the present application also provides a synchronous playback method in a distributed service scenario, which is applied to a many-to-one broadcast scenario. Fig.18 As shown, in a many-to-one broadcasting scenario, the source end includes more than two devices, the more than two devices include a fourth device and a fifth device, and the playback end includes a sixth device; the fourth device and the fifth device respectively transmit the fourth data stream and the fifth data stream to the sixth device for playback; the fourth device, the fifth device and the sixth device are all equipped with a control center, and the control center reserves an interface for setting synchronization group information.

[0314] The method can be performed based on the sixth device, such as Fig.19 As shown, the method may include the following process:

[0315] S181, receiving synchronization group information respectively sent by the fourth device and the fifth device;

[0316] The synchronization group information includes synchronization reference information and stream information. The synchronization reference information includes the source device identity information and the playback device identity information; the stream information includes the data type information of the data stream sent by each source device.

[0317] S182: Determine a synchronous playback strategy for the data stream to be played based at least on the synchronization group information.

[0318] For specific implementation, please refer to the description of the embodiments in the corresponding scenarios above, which will not be repeated here.

[0319] The embodiment of the present application also provides a playback device in a distributed service scenario. The playback device is used as a second device in a one-to-many broadcast scenario. Fig. 20 The second device shown in , includes a frame rate control module, configured to perform the following operations:

[0320] A first timestamp is received, a second timestamp corresponding to a second data frame to be played is read, and a first play time difference is calculated based on at least the first timestamp and the second timestamp, and a synchronous play strategy to be executed for the second data frame to be played is determined based on the first play time difference.

[0321] Optionally, the second device also includes: a second state feedback module and / or a second data cache module; the second state feedback module is at least used to identify the data type of the second data frame to be played and feed it back to the third device; the second data cache module is at least used to cache at least one second data frame in the second data stream.

[0322] The embodiment of the present application also provides a playback device in a distributed service scenario. The playback device is used as a first device and is applied to a one-to-many broadcasting scenario. Fig. 20 The first device shown includes a delay feedback module, which is used to perform the following operations:

[0323] A first timestamp is obtained, and the first timestamp is sent to a third device.

[0324] Among them, the first timestamp represents the actual playback time of the mth first data frame on the first device; the first timestamp is used to calculate the first playback time difference with the second timestamp, so as to determine the synchronous playback strategy to be executed for the second data frame to be played based on the first playback time difference; the second timestamp is used to represent the playback timing of each second data frame in the second data stream.

[0325] Optionally, the first device also includes: a first state feedback module and / or a first data cache module; the first state feedback module is at least used to identify the data type of the first data frame to be played and feed it back to the third device; the first data cache module is at least used to cache at least one first data frame in the first data stream.

[0326] The embodiment of the present application also provides a source device in a distributed service scenario, which is used as a third device in a one-to-many broadcasting scenario. Fig. 20 The third device shown includes a multi-channel management module, which is used to perform the following operations:

[0327] The first timestamp is received and forwarded to the second device; the first timestamp is acquired and sent by the first device.

[0328] Optionally, the third device also includes: a synchronization policy management module and / or a status monitoring module; the status monitoring module is used to receive the respective data types fed back by more than two devices and input them into the synchronization policy management module; the synchronization policy management module is used to determine the synchronization management policy based on the respective data types fed back by more than two devices.

[0329] The present application also provides a playback system in a distributed service scenario, such as Fig. 20 As shown, the playback system is applied to a one-to-many broadcasting scenario, the first device includes a delay feedback module, which is used to obtain a first timestamp and send the first timestamp to a third device; the third device includes a multi-channel management module, which is used to receive the first timestamp and forward it to the second device; the second device includes a frame rate control module, which is used to receive the first timestamp, read the second timestamp corresponding to the second data frame to be played, and calculate the first playback time difference based on at least the first timestamp and the second timestamp, and determine the synchronous playback strategy to be executed for the second data frame to be played based on the first playback time difference.

[0330] The embodiment of the present application also provides a playback system in a distributed business scenario, which is applied to a one-to-many broadcast scenario. The first device is used to determine the delay information; the delay information includes transmission delay and decoding delay; wherein the transmission delay indicates the duration of the transmission of the first data frame from the third device to the first device; the decoding delay is the decoding duration occupied by a first data frame; the third device is used to receive the delay information, obtain the sending time of the first data frame, and obtain the estimated value of the playback time of the first data frame on the first device based on at least the sending time and the delay information; before sending the second data frame, the estimated value is added to the second data frame as the fifth timestamp; the second device is used to read the fifth timestamp of the second data frame to be played, calculate the difference between the fifth timestamp and the current time, and obtain the second playback time difference; based on the second playback time difference, determine the synchronous playback strategy to be executed for the second data frame to be played.

[0331] The embodiment of the present application also provides a playback system in a distributed service scenario, and the playback system is applied to a many-to-one broadcasting scenario, such as Fig.18 As shown, in a many-to-one broadcast scenario, the source end includes more than two devices, the more than two devices include a fourth device and a fifth device, and the playback end includes a sixth device; the fourth device and the fifth device respectively transmit the fourth data stream and the fifth data stream to the sixth device for playback; the fourth device, the fifth device and the sixth device are all provided with a control center, and the control center reserves an interface for setting synchronization group information; the fourth device and the fifth device are used to set the synchronization group information to the local control center respectively, and send the set synchronization group information to the sixth device respectively; the synchronization group information includes synchronization reference information and stream information, the synchronization reference information includes source device identity information and playback device identity information; the stream information includes data type information of the data stream sent by each device on the source end; the sixth device is used to receive the synchronization group information sent by the fourth device and the fifth device respectively, and determine the synchronization playback strategy for the data stream to be played based at least on the synchronization group information.

[0332] An embodiment of the present application also provides an electronic device, which includes: one or more processors; a memory; at least one application; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to execute a method as described in any of the above embodiments.

[0333] An embodiment of the present application further provides a storage medium, including computer instructions, which, when executed on an electronic device, enables the electronic device to execute a method as described in any of the above embodiments.

[0334] An embodiment of the present application also provides a chip system, including a communication interface for inputting and / or outputting data; and a processor for executing a computer executable program so that a device equipped with the chip system executes a method as described in any of the above embodiments.

[0335] It should be noted that the solution provided in the embodiments of the present application includes any one or more of a synchronous playback system and method, an electronic device, a storage medium and a chip system, or may also be other forms of software products, hardware products or virtual service products obtained based on the technical concept of the method described in the specification of this application.

[0336] In the description of the device in the embodiment of the present application, various modules for implementing the method flow functions are constructed in the device. The division of modules is merely a division of logical functions. When implementing the embodiment of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0337] Specifically, each functional module can be fully or partially integrated into one physical entity during actual implementation, or can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; or all can be implemented in the form of hardware; or some modules can be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the detection module can be a separately established processing element, or it can be integrated in a chip of an electronic device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in a processor element or instructions in the form of software.

[0338] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0339] Specifically, in one embodiment of the present application, the above-mentioned one or more computer programs are stored in the above-mentioned memory, and the above-mentioned one or more computer programs include instructions. When the above-mentioned instructions are executed by the above-mentioned device, the above-mentioned device executes the method steps described in the embodiment of the present application.

[0340] Specifically, in one embodiment of the present application, the processor of the electronic device may be a SOC device on a chip, which may include a central processing unit (CPU) and may further include other types of processors, such as a graphics processing unit (GPU). Specifically, in one embodiment of the present application, the processor of the electronic device may be a PWM control chip.

[0341] Specifically, in an embodiment of the present application, the processor involved may include, for example, a CPU, a DSP, a microcontroller or a digital signal processor, and may also include a GPU, an embedded neural network processor (Neural-network Process Units, NPU) and an image signal processor (Image Signal Processing, ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASICs, or one or more integrated circuits for controlling the execution of the program of the technical solution of the present application. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium.

[0342] Specifically, in one embodiment of the present application, the memory of the electronic device may be a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage devices, or may be any computer-readable medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer.

[0343] Specifically, in an embodiment of the present application, the processor and the memory can be combined into a processing device, more commonly, they are independent components, and the processor is used to execute the program code stored in the memory to implement the method described in the embodiment of the present application. In specific implementation, the memory can also be integrated into the processor, or independent of the processor.

[0344] Furthermore, the devices, apparatuses, and modules described in the embodiments of the present application may be implemented by computer chips or entities, or by products having certain functions.

[0345] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0346] In the several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0347] The embodiments described in this application are described with reference to the flowcharts and / or block diagrams of the methods, devices (apparatus), and computer program products according to the embodiments of the application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0348] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0349] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0350] It should also be noted that in the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0351] In the embodiments of the present application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, commodity or device including the element.

[0352] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0353] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0354] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments of the present application can be implemented by a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0355] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0356] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A synchronous playback system, characterized in that: The playback system includes a first device, a second device and a third device; wherein the third device is used to transmit a first data stream to the first device for playback, and transmit a second data stream to the second device for playback; the first data stream and the second data stream are two data streams to be played synchronously; the first data stream includes a plurality of first data frames; the second data stream includes a plurality of second data frames; The first device is used to obtain a first timestamp and send the first timestamp to the third device; The third device is used to receive the first timestamp and forward it to the second device; The second device is used to receive the first timestamp, read the second timestamp corresponding to the second data frame to be played, and calculate the first playback time difference based on at least the first timestamp and the second timestamp, and determine the synchronous playback strategy to be executed for the second data frame to be played based on the first playback time difference.

2. The system according to claim 1, characterized in that The second device, when performing the operation of reading the second timestamp corresponding to the second data frame to be played, is specifically used to: At every predetermined time interval, the next second data frame to be played is taken out from the cache queue and the second timestamp is read.

3. The system according to claim 1, characterized in that The second device, when performing the operation of calculating the first playback time difference based on at least the first timestamp and the second timestamp, is specifically used to: determine that when there is a mapping relationship between the next second data frame to be played and the m-th first data frame, calculate the first playback time difference based on at least the first timestamp corresponding to the m-th first data frame and the second timestamp corresponding to the next second data frame to be played; wherein the mapping relationship is determined based on the serial number and / or the playback timestamp PTS.

4. The system according to any one of claims 1 to 3, characterized in that: The second device is further used to identify data type information of the second data frame to be played, and feed back the information to the third device; The first device is further used to identify data type information of the first data frame to be played, and feed back the information to the third device; The third device is further used to receive data type information fed back by the first device and the second device respectively, and determine a synchronization management strategy.

5. A synchronous playback method, characterized in that: The method is applied to a second device, the second device is used to receive and play a second data stream sent by a third device; wherein the third device is also used to transmit a first data stream to the first device for playing; the first data stream and the second data stream are two data streams to be played synchronously, the first data stream includes a plurality of first data frames; the second data stream includes a plurality of second data frames; The method comprises: Receive a first timestamp; the first timestamp indicates the actual playback time of the mth first data frame on the first device; m≥1 and is an integer; Reading a second timestamp corresponding to a second data frame to be played; the second timestamp is used to indicate a playback timing of each second data frame in the second data stream; Calculating a first playback time difference based at least on the first timestamp and the second timestamp; Based on the first playback time difference, a synchronous playback strategy to be executed for the second data frame to be played is determined.

6. The method according to claim 5, characterized in that Receive a first timestamp, including: The first timestamp forwarded by the third device is received; the first timestamp is generated by the first device and sent to the third device.

7. The method according to claim 5, characterized in that Reading a second timestamp corresponding to a second data frame to be played includes: At every predetermined time interval, the next second data frame to be played is taken out from the cache queue and the second timestamp is read.

8. The method according to any one of claims 5 to 7, characterized in that Calculating a first playback time difference at least according to the first timestamp and the second timestamp includes: When it is determined that there is a mapping relationship between the next second data frame to be played and the m-th first data frame, the first playback time difference is calculated at least based on the first timestamp corresponding to the m-th first data frame and the second timestamp corresponding to the next second data frame to be played; wherein the mapping relationship is determined based on the serial number and / or the playback timestamp PTS.

9. The method according to any one of claims 5 to 7, characterized in that Calculating a first playback time difference at least according to the first timestamp and the second timestamp includes: Calculating a first playback time difference according to the first timestamp, the second timestamp, the first processing duration, and the second processing duration; Among them, the first processing time is an estimated value of the processing time taken from the fetching of a first data frame from the cache queue to the playback; the second processing time is an estimated value of the processing time taken from the fetching of a second data frame from the cache queue to the playback.

10. The method according to any one of claims 5 to 9, characterized in that The synchronous playback strategy includes at least one of waiting, sending and displaying, and catching up; Determining a synchronous playback strategy to be executed for the second data frame to be played based on the first playback time difference includes: comparing the first playback time difference with a predetermined first threshold interval; When the first play time difference is less than the minimum value of the first threshold interval, determining that the play of the first data stream lags behind the play of the second data stream, and determining that the synchronous play strategy to be executed for the second data frame to be played is waiting; When the first playback time difference is greater than the maximum value of the first threshold interval, it is determined that the playback of the first data stream is ahead of the second data stream, and the synchronous playback strategy executed for the second data frame to be played is determined to be catch-up; When the first playback time difference belongs to the first threshold interval, it is determined that the playback of the first data stream is relatively synchronized with the playback of the second data stream, and the synchronous playback strategy executed for the second data frame to be played is determined to be display.

11. A synchronous playback method, characterized in that: The method is applied to a third device, the third device is used to transmit a first data stream to the first device for playback, and transmit a second data stream to the second device for playback; wherein the first data stream and the second data stream are two data streams to be played synchronously, the first data stream includes a plurality of first data frames; the second data stream includes a plurality of second data frames; The method comprises: receiving delay information; the delay information includes transmission delay and decoding delay; wherein the transmission delay represents the duration of the mth first data frame being transmitted from the third device to the first device; the decoding delay is the decoding duration occupied by the mth first data frame; m≥1 and is an integer; Obtaining the sending time of the mth first data frame; Obtaining an estimated value of the playback time of the mth first data frame on the first device at least according to the sending time and the delay information; Before sending the second data frame, the estimated value is added to the second data frame as a fifth timestamp; the fifth timestamp is used to calculate the difference with the current time to obtain a second playback time difference, so as to determine the synchronous playback strategy to be executed for the second data frame to be played.

12. The method according to claim 11, characterized in that Obtaining an estimated value of the play time of the first data frame on the first device at least according to the sending time and the delay information, including: Obtaining an estimated value of a playback time of the mth first data frame on the first device at least according to a transmission time of the mth first data frame and the delay information; Adding the estimated value as a fifth timestamp to the second data frame comprises: The estimated value is added as a fifth timestamp to the second data frame having a mapping relationship with the mth first data frame; wherein the mapping relationship is determined based on a sequence number and / or a playback timestamp PTS.

13. The method according to claim 11 or 12, characterized in that Obtaining an estimated value of the play time of the first data frame on the first device at least according to the sending time and the delay information, including: Obtaining an estimated value of a play time of the first data frame on the first device according to the sending time, the delay information, the third timestamp, and the fourth timestamp; The third timestamp is a playback timestamp PTS added to the first data frame, and is used to indicate the playback timing of each first data frame in the first data stream; The fourth timestamp is a play timestamp PTS added to the second data frame, and is used to indicate the play timing of each second data frame in the second data stream.

14. A synchronous playback method, characterized in that: The method is applied to a sixth device, the sixth device being used to receive a fourth data stream sent by a fourth device and a fifth data stream sent by a fifth device, and synchronously play the fourth data stream and the fifth data stream; The method comprises: Receive synchronization group information respectively sent by the fourth device and the fifth device; the synchronization group information includes synchronization reference information and stream information, the synchronization reference information includes source device identity information and playback device identity information; the stream information includes data type information of the data stream sent by each source device; A synchronous playback strategy for the data stream to be played is determined based at least on the synchronization group information.

15. An electronic device, characterized in that: The electronic device comprises: One or more processors; a memory; at least one application; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform a method as described in any one of claims 5-14.

16. A storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 5 to 14.

17. A chip system, characterized in that: include: A communication interface for inputting and / or outputting data; A processor, used to execute a computer executable program so that a device equipped with the chip system executes a method as described in any one of claims 5 to 14.

Images