Multi-screen audio and video synchronization method, device, system and electronic equipment

Abstract

The present application provides a multi-screen audio and video synchronization method, device, system and electronic device. The method is applied to a first playback device. When it is necessary to achieve audio and video synchronization between multiple playback devices, the first playback device and the second playback device negotiate with each other according to a preset authoritative device negotiation algorithm to negotiate an authoritative end device, and other devices access the authoritative end device as clients. At this time, the client device can determine the difference in playback progress between the playback devices based on the time reference in the synchronization information obtained by interacting with the authoritative end device, calculate the correction rate based on the communication time of the synchronization information between the authoritative end device, and adjust the current playback rate according to the correction rate to achieve the effect of synchronization with the authoritative end device. The embodiment of the present application incorporates the communication time into the influencing factors of audio and video synchronization, which can effectively reduce the problem of audio and video playback asynchronism caused by network fluctuations.

Description

Technical Field

[0001] The present application relates to the field of audio and video data processing, and in particular to a multi-screen audio and video synchronization method, device, system and electronic equipment. Background Art

[0002] With the development of audio and video technology, the application scenarios of synchronously playing video sources on multiple screens (hereinafter collectively referred to as multi-screens) are becoming increasingly popular. Large-scale audio and video synchronous playback scenarios such as urban rail transit, computer room video surveillance, multimedia exhibition halls, and flight cockpits have an increasingly urgent need for synchronous playback of video sources on multiple display screens.

[0003] In order to ensure the effect of multi-screen synchronous playback in large-scale audio and video synchronous playback scenarios, limited by the performance of the equipment, a large server is usually used to centrally manage and process data synchronization solutions to achieve multi-screen synchronous playback. Among them, the large server centralized management and data processing synchronization solution is usually to establish a large high-performance streaming media server, and then the streaming media server uniformly sends the playback audio and video streams to each playback screen client, and realizes multi-screen synchronization through flow control or session interaction.

[0004] However, this synchronization solution based on centralized management and data processing by large servers has high requirements on server performance and network status, and is prone to problems such as asynchrony in audio and video playback due to network fluctuations or device differences. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a multi-screen audio and video synchronization method, device, system and electronic device to reduce the problem of audio and video playback being out of sync due to network fluctuations or device differences.

[0006] In a first aspect, an embodiment of the present application provides a multi-screen audio and video synchronization method, wherein the method is applied to a first playback device, and the method includes:

[0007] Establishing a communication connection between the first playback device and each of the other second playback devices according to a preset authoritative end device negotiation algorithm; wherein the preset authoritative end device negotiation algorithm is a communication algorithm for each playback device to automatically determine an authoritative end device according to a communication message and establish a communication connection with the authoritative end device;

[0008] Based on the communication connection, exchanging synchronization information with the target second playback device that establishes the communication connection, wherein the synchronization information includes: a time reference, and the time reference includes: a system time of the authoritative end device and a system time of the client device;

[0009] When the first playback device is a client device that establishes a communication connection with the authoritative end device, the audio and video stream correction rate is calculated according to the received time reference and the communication time of the synchronization information according to a preset correction rate calculation formula, and the audio and video playback rate of the playback task is adjusted according to the audio and video stream correction rate.

[0010] In a second aspect, an embodiment of the present application provides a multi-screen audio and video synchronization device, wherein the device is applied to a first playback device, and the device includes:

[0011] A negotiation module, used to establish a communication connection between the first playback device and other second playback devices according to a preset authoritative end device negotiation algorithm; wherein the preset authoritative end device negotiation algorithm is a communication algorithm for each playback device to automatically determine an authoritative end device according to a communication message and establish a communication connection with the authoritative end device;

[0012] An information interaction module, used for exchanging synchronization information with a target second playback device with which a communication connection is established based on the communication connection, wherein the synchronization information includes: a time reference, and the time reference includes: a system time of an authoritative end device and a system time of a client device;

[0013] A data processing module, used to calculate the correction rate of the audio and video stream according to a preset correction rate calculation formula based on the received time reference and the communication time of the synchronization information;

[0014] The playback control module is used to adjust the playback rate of the audio and video of the playback task according to the audio and video stream correction rate when the first playback device is a client device that establishes a communication connection with the authoritative terminal device.

[0015] In a third aspect, an embodiment of the present application provides a multi-screen audio and video synchronization system, the multi-screen audio and video synchronization system comprising: a first playback device and a second playback device, wherein:

[0016] The first playback device or the second playback device is used to determine the authoritative end device and the client device according to a preset authoritative end device negotiation algorithm, and establish a communication connection between the authoritative end device and the client device;

[0017] The preset authoritative end device negotiation algorithm is a communication algorithm for each playback device to automatically determine the authoritative end device according to the communication message and establish a communication connection with the authoritative end device; if the first playback device is an authoritative end device, the second playback device connected to the first playback device is a client device, and if the second playback device is an authoritative end device, the first playback device connected to the second playback device is a client device;

[0018] The first playback device is used to exchange synchronization information with the second playback device that has established a communication connection based on the communication connection, wherein the synchronization information includes: a time reference, and the time reference includes: a system time of an authoritative end device and a system time of a client device;

[0019] When the first playback device is a client device, or when the second playback device is a client device, the client device is used to calculate the audio and video stream correction rate according to the received time reference and the communication time of the synchronization information according to a preset correction rate calculation formula, and adjust the audio and video playback rate of the playback task according to the audio and video stream correction rate.

[0020] In a fourth aspect, an embodiment of the present application provides an electronic device, wherein the electronic device includes:

[0021] Processor; and

[0022] Memory for storing programs,

[0023] Among them, the program includes instructions, and when the instructions are executed by the processor, the processor executes the multi-screen audio and video synchronization method described in the first aspect.

[0024] In a fifth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to enable a computer to execute the multi-screen audio and video synchronization method described in the first aspect.

[0025] Beneficial effects of this application:

[0026] The present application provides a multi-screen audio and video synchronization method, device, system and electronic device, wherein the method is applied to a first playback device in a scenario of multiple audio and video playback devices, and in the multiple audio and video playback scenarios, there are other second playback devices in addition to the first playback device. That is, when it is necessary to achieve audio and video synchronization between the first playback device and multiple second playback devices, in an embodiment of the present application, the first playback device and the second playback device automatically negotiate according to a preset authoritative device negotiation algorithm, and negotiate an authoritative end device, and other playback devices except the authoritative end device establish a communication connection with the authoritative end device.

[0027] Then, based on the communication connection, the authoritative end device and the client device (in order to distinguish the playback devices connected by the communication connection, the playback devices connected to the authoritative end device are referred to as client devices in this article) exchange synchronization information, and based on the time reference carried in the interactive synchronization information and the communication time of the interactive synchronization information, calculate the audio and video stream correction rate according to the preset correction rate calculation formula. Finally, the client device adjusts the audio and video playback rate of the playback task according to the correction rate.

[0028] By using the embodiment of the present application, in a scenario where multiple playback devices require audio and video synchronization, each playback device can negotiate an authoritative end device, and then other devices access the authoritative end device as a client. At this time, the client device can determine whether the current playback progress is slower or faster than the playback progress of the authoritative end device based on the time reference in the synchronization information obtained by interacting with the authoritative end device, and calculate the correction rate that should be used to adjust the current playback rate based on the communication time of the synchronization information between the authoritative end to maintain synchronization with the authoritative end device. Then, adjust the current playback rate according to the correction rate to achieve the effect of playback synchronization with the authoritative end device.

[0029] Compared with the synchronization solution of the prior art that relies on large servers for unified management and processing, the embodiment of the present application is selected to include communication time in the influencing factors of audio and video synchronization. The playback devices can negotiate an authoritative end device by themselves, and then calculate the correction rate based on the communication time of the synchronization information to adjust the playback rate of the playback task, which can effectively reduce the problem of audio and video playback asynchrony caused by network fluctuations or device differences. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0031] Figure 1 A schematic diagram of a process flow of a multi-screen audio and video synchronization method provided by an embodiment of the present application is shown;

[0032] Figure 2 A schematic diagram of a process of authoritative end device negotiation provided in an embodiment of the present application is shown;

[0033] Figure 3 Another schematic diagram of the process of authoritative end device negotiation provided in an embodiment of the present application is shown;

[0034] Figure 4 Another schematic diagram of the process of negotiating with an authoritative end device to change an authoritative end device provided in an embodiment of the present application is shown;

[0035] Figure 5 A schematic diagram of a process for authoritative end device negotiation across regional networks provided by an embodiment of the present application is shown;

[0036] Figure 6 A schematic diagram of a process for synchronizing information between a client device and an authoritative terminal device provided in an embodiment of the present application is shown;

[0037] Figure 7A schematic diagram showing the contents of interactive synchronization information provided by an embodiment of the present application is shown;

[0038] Figure 8 A schematic diagram of a mapping relationship between system time and playback time provided in an embodiment of the present application is shown;

[0039] Fig. 9 A schematic diagram of an audio and video synchronization timeline provided in an embodiment of the present application is shown;

[0040] Fig.10 A schematic diagram showing a logical structure of the internal functional modules of the playback device provided in an embodiment of the present application;

[0041] Fig.11 A schematic diagram of the logical structure of a multi-screen audio and video synchronization device provided in an embodiment of the present application is shown;

[0042] Fig.12 A system interaction schematic diagram of a multi-screen audio and video synchronization system provided by an embodiment of the present application is shown;

[0043] Fig.13 A logical structure diagram of an electronic device provided in an embodiment of the present application for implementing the multi-screen audio and video synchronization method provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0044] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0045] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0046] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0047] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0048] In a first aspect, the present application provides a multi-screen audio and video synchronization method, which is applied to any electronic device with a multi-screen audio and video synchronization function, including but not limited to a personal mobile terminal, a computer or a server, etc. Figure 1 As shown, the method includes the following steps S11 to S14:

[0049] S11. Establish a communication connection between the first playback device and other second playback devices according to a preset authoritative end device negotiation algorithm.

[0050] The preset authoritative end device negotiation algorithm is a communication algorithm for each playback device to automatically determine the authoritative end device according to the communication message and establish a communication connection with the authoritative end device;

[0051] S12: Based on the communication connection, exchange synchronization information with the target second playback device with which the communication connection is established.

[0052] The synchronization information includes: a time reference, the time reference includes: a system time of the authority device and a system time of the client device;

[0053] S13: When the first playback device is a client device that establishes a communication connection with the authoritative terminal device, the correction rate of the audio and video stream is calculated according to a preset correction rate calculation formula based on the received time reference and the communication time of the synchronization information.

[0054] S14. Adjust the playback rate of the audio and video of the playback task according to the audio and video stream correction rate.

[0055] In the embodiment of the present application, when multiple playback devices need to perform audio and video synchronization, each playback device and other playback devices in the environment automatically determine an authoritative end device according to the communication message according to the preset authoritative end device negotiation algorithm, and then the remaining playback devices establish a communication connection with the authoritative end device. Then, based on the communication connection between the playback device and the playback device, the playback device and the playback device exchange synchronization information, and then when the first playback device is not an authoritative end device, the first playback device calculates the audio and video stream correction rate according to the preset correction rate calculation formula based on the received time reference and the communication time of the exchanged synchronization information, and then adjusts the playback rate of the playback task according to the audio and video stream correction rate.

[0056] By selecting the embodiment of the present application, the traditional multi-screen audio and video solution that relies on unified management and data processing of large server devices is abandoned. The playback devices negotiate with each other to determine an authoritative end device as a synchronization reference. Then other playback devices access the authoritative end device as client devices, and use the time reference of the authoritative end device as the synchronization reference. The communication time of the synchronization information is combined to determine the communication network time difference (i.e., network delay) between the authoritative end device and the client device. Then, the audio and video stream correction rate is calculated based on the time reference and the communication time, and the playback rate of its own playback task is adjusted according to the audio and video stream correction rate, so that each client device can catch up with or wait for the playback progress of the authoritative end device, so as to achieve the effect of synchronous audio and video playback of each playback device.

[0057] In an embodiment of the present application, the communication time is included in the influencing factors of audio and video synchronization, and the correction rate is calculated with the help of the communication time. In this way, the calculated correction rate can overcome the playback asynchrony caused by network fluctuations and reduce the problem of audio and video playback asynchrony caused by network playback.

[0058] The above steps S11 to S14 will be described in detail below with reference to specific examples:

[0059] In the embodiment of the present application, since the audio and video synchronization method is applied to the scenario of multi-screen synchronization, naturally, there will be multiple playback devices in the actual application scenario, and each playback device can be a playback device with a display screen and a speaker, or a playback device with only a display screen and no speaker, or even a playback device with only a speaker and no display screen. The specific form of the playback device can be flexibly adjusted according to the actual application process, and the present application does not strictly limit its specific form.

[0060] There will be many terms for playback devices in this article. In order to distinguish the various playback devices in this article, the various playback device terms involved in this article will be explained here:

[0061] First playback device: Since there is communication interaction between playback devices, a playback device in the communication interaction is defined as the first playback device in this article. That is, the first playback device is any playback device in the entire audio and video synchronization scenario, specifically referring to one party in the communication interaction.

[0062] Second playback device: other playback devices except the first playback device in the entire audio and video synchronization scenario, that is, other playback devices that need to interact with the first playback device can be collectively referred to as second playback devices. To put it simply, this application defines other playback devices except the first playback device as second playback devices. In the embodiment of the present application, the first playback device and the second playback device are only different in name, and the functions they can achieve should be the same, that is, they both have the ability to adjust their own playback progress based on the results of communication interaction.

[0063] Authoritative end device: A playback device that can be used as a synchronization reference benchmark after two or more playback devices execute the preset authoritative end device negotiation algorithm. In other words, the authoritative end device is a reference device that is self-selected from many playback devices.

[0064] Client device: a playback device that establishes a communication connection with an authoritative end device, which can be said to be other playback devices that do not belong to the authoritative end device after two or more playback devices execute the preset authoritative end device negotiation algorithm.

[0065] Target second playback device: The second playback device that needs to synchronize audio and video with the first playback device and has determined the identity of the client device or the authoritative end device after negotiating with the first playback device. At this time, there are several possibilities between the target second playback device and the first playback device (corresponding to possibilities 1 to 4 below):

[0066] Possibility 1: the target second playback device and the first playback device belong to the same local area network, and the target second playback device is a confirmed authoritative terminal device in the local area network. At this time, the first playback device accesses the target second playback device as a client device.

[0067] Possibility 2: The target second playback device and the first playback device belong to the same local area network, the first playback device is an authoritative terminal device that has been determined in the local area network, and the target second playback device accesses the first playback device as a client device.

[0068] Possibility 3: The target second playback device and the first playback device belong to the same local area network. The target second playback device and the first playback device are the first two playback devices that appear in the local area network. At this time, the first playback device and the target second playback device are neither authoritative end devices nor client devices. At this time, the two can determine each other's identity by themselves, specifically whether they are authoritative end devices or client devices.

[0069] Possibility 4: The first playback device and the target second playback device do not belong to the same local area network, and both the first playback device and the target second playback device are connected to a common authoritative terminal device as client devices. In this case, the first playback device and the target second playback device are both client devices connected to the common authoritative terminal device.

[0070] Based on the above possibilities 1 to 4, if you want to achieve multi-screen audio and video synchronization, you need to go through the following stages of processing:

[0071] Negotiation phase—>communication establishment phase—>interview phase—>correction rate calculation phase—>synchronization adjustment phase.

[0072] Among them, the core of the negotiation stage is that the first playback device and the second playback device discover each other and negotiate an authoritative end device. The core of the communication connection establishment stage is: establishing a communication connection between the authoritative end device and the client device, mainly establishing a connection-oriented communication connection, so as to realize the switching of the authoritative end device and the client device from a non-connection-oriented communication state to a connection-oriented communication state. The core of the interview stage is to exchange synchronization information based on the established connection-oriented communication connection. The correction rate calculation stage calculates the audio and video correction rate used to adjust the playback rate based on the exchanged synchronization information. Finally, the synchronization adjustment stage is for the client device to adjust its own audio and video playback rate based on the audio and video correction rate calculated in the correction rate calculation stage.

[0073] The following is a detailed description of the above stages:

[0074] In the negotiation phase, each playback device communicates and negotiates with other playback devices by executing a preset authoritative end device negotiation algorithm preset locally to determine an authoritative end device. In view of the above possibilities 1 to 3, when the first playback device and the second playback device are located in the same local area network, the first playback device and the second playback device can directly determine the authoritative end device based on the data packets sent in the same local area network. Among them, the data packets sent in the same local area network carry the relevant description information of the determined authoritative end device, which can specifically refer to the identification information of the authoritative end device.

[0075] Based on this, for possibility 1, the target second playback device can be directly determined as the authoritative end device and the first playback device as the client device. For possibility 2, the first playback device can be directly determined as the authoritative end device and the target second playback device can be directly determined as the client device.

[0076] For possibility 3, that is, when there is no authoritative end device in the current network, as an implementation method, Figure 2 As shown, each playback device obtains its own historical records to determine whether there is historical multicast data. If there is historical multicast data, the device priority of the local device is determined based on the historical multicast data, and a playback device with a high device priority is determined as the authoritative end device. Specifically, if there are multiple devices negotiating at the same time, the playback device with the highest device priority can be determined as the authoritative end device. The device priority can be determined based on the historical online time of the device. The longer the historical online time, the higher the corresponding device priority. If there is no historical multicast data, you can wait for multicast data through multiple rounds of queries. If there is a timeout in the query round (which means that there are no other playback devices in the current environment), the local device is determined as the authoritative end device, and the remaining playback devices naturally access the authoritative end device as client devices.

[0077] On this basis, in some embodiments, the first playback device and the second playback device can determine the authoritative end device and the client device by themselves. If the first playback device is a client device, the target second playback device is determined to be an authoritative end device. If the first playback device is an authoritative end device, the target second playback device is determined to be a client device. Since the first playback device and the second playback device are located in the same local area network, at this time, for the communication connection establishment stage, the client device and the authoritative end device can first establish a multicast communication connection.

[0078] However, the multicast communication connection at this time is not a connection-oriented communication connection, the communication quality cannot be guaranteed, and the communication delay between the two playback devices cannot be accurately determined. Based on this, it is also necessary to use the multicast communication connection to establish a connection-oriented point-to-point communication connection between the first playback device and the target second playback device.

[0079] As an implementation mode, the first playback device and the target second playback device establish a point-to-point communication connection in the following manner:

[0080] When the first playback device is an authoritative end device and the target second playback device is a client device, the target second playback device parses the multicast communication data packet in the multicast communication connection based on the multicast communication connection between the client device and the authoritative end device, and establishes a point-to-point communication connection between the first playback device and the target second playback device, that is, establishes a point-to-point communication connection between the authoritative end device and the client device;

[0081] Alternatively, when the first playback device is a client device and the target second playback device is an authoritative end device, based on the multicast communication connection between the client device and the authoritative end device, the multicast communication data packets in the multicast communication connection are parsed to establish a point-to-point communication connection between the authoritative end device and the client device.

[0082] Specifically, based on the established multicast communication connection, the authoritative end device sends a multicast communication data packet to the client device, and the multicast communication data packet carries the device information of the authoritative end device, including the identification information of the authoritative end device, the priority information of the authoritative end device, and the communication address information of the authoritative end device. Then, the client device parses the received multicast communication data packet, obtains various information carried in the multicast communication data packet, especially the communication address of the authoritative end device, and establishes a connection-oriented point-to-point connection with the authoritative end device, such as establishing a TCP (Transmission Control Protocol) connection.

[0083] In the embodiment of the present application, when executing step S12 to exchange synchronization information between the target second playback device and the first playback device (i.e., corresponding to the above-mentioned interview stage), it can be based on the multicast communication connection exchange synchronization information between the first playback device and the target second playback device, or it can be based on the point-to-point communication connection exchange synchronization information between the first playback device and the target second playback device. However, as a preferred implementation, for the situation in the same regional network, that is, for the above-mentioned possible 1 to possible 3, all of them are through the multicast communication connection exchange synchronization information, and for possible 4, there is another processing idea, which will be described in detail later. Here, it is mainly for possible 1 to possible 3 How to specifically exchange synchronization information through multicast communication connection:

[0084] As a possible implementation, the first playback device and the target second playback device may exchange synchronization information in the following manner:

[0085] When the first playback device is a client device, based on the multicast communication between the authoritative end device and the client device, the media playback control data sent by the authoritative end device is received, wherein the media playback control data includes: the time reference and the communication time of the synchronization information.

[0086] As a possible implementation, the first playback device and the target second playback device may exchange synchronization information in the following manner:

[0087] When the first playback device is an authoritative end device, based on the multicast communication connection between the authoritative end device and the client device, the device information of the authoritative end device is sent to each target second playback device in the multicast group to which the first playback device belongs (also referred to as each client device in the multicast group to which the first playback device belongs), so that each target second playback device in the multicast group to which the first playback device belongs verifies the authoritative end device based on the received device information of the authoritative end device.

[0088] That is to say, in the embodiment of the present application, after the authoritative end device and the client device establish a multicast communication connection, the authoritative end device will actively send media playback control data to the connected client device based on the multicast communication connection between the two to inform the client device of the specific time reference and the communication time of the synchronization information. Among them, the specific time reference may include the system time of the authoritative end device (the system time of the authoritative end device is referred to as the authoritative system time in this article) and the playback timestamp of the authoritative end device (the playback timestamp of the authoritative end device is referred to as the authoritative playback timestamp in this article).

[0089] When there is an authoritative device in the current network, Figure 3 As shown, the authoritative end device can also send a multicast data packet to each playback device in the same multicast group through the multicast communication connection. The multicast data packet carries the device information of the authoritative end device. The device information of the authoritative end device includes: device identification information of the authoritative end device, priority information of the authoritative end device, communication address information of the authoritative end device, etc. Among them, as described in the previous description of building a point-to-point communication connection, the communication address information of the authoritative end device is mainly used to build a point-to-point communication connection between the authoritative end device and the client device, and the device identification information and priority information of the authoritative end device are mainly used by the client device to verify the authoritative end device.

[0090] After receiving the multicast data packet sent by the authoritative end device, the client device verifies the received multicast data packet sent by the authoritative end device, mainly using the device identification information of the authoritative end device carried in the multicast communication data packet to perform security verification on the authoritative end device to determine whether the authoritative end device is a legal playback device, so as to avoid data leakage caused by accessing an illegal authoritative end device. If the verification passes, a connection-oriented point-to-point connection is established with the authoritative end device. If the verification fails, a request to change the authoritative end device is sent to the authoritative end device that sent the multicast data packet.

[0091] In the embodiment of the present application, the identification information of the device is information used to distinguish different devices, which can be the device number of each device, or the IMEI (International Mobile Equipment Identity) of the device, the MAC (Media Access Control Address) address of the device, the MEID (Mobile Equipment Identifier) ​​of the device, etc., which can uniquely identify the device.

[0092] In addition, secondary negotiation of the authoritative end device can be performed according to the priority information of the authoritative end device, so that when the authoritative end device changes, a new authoritative end device can be determined as soon as possible. Based on this, in some embodiments, when executing step S12, it can also be implemented in the following way:

[0093] When the first playback device is an authoritative end device, based on the multicast communication connection between the client device and the authoritative end device, device discovery data is sent to the multicast group to which the first playback device belongs, so that each target second playback device in the multicast group to which the first playback device belongs performs exception processing based on the device discovery data, and the exception processing includes: changing the authoritative end device.

[0094] The device discovery data is used to quickly determine a new authoritative end device based on the information carried in the received device discovery data and establish a communication connection with the new authoritative end device when the connection with the existing authoritative end device is interrupted. Specifically, the new authoritative end device is determined based on the device priority carried in the received device discovery data.

[0095] Exemplarily, when the pre-set working time of the authoritative end device expires, device discovery data can be sent to each playback device in the multicast group to which it belongs through the multicast communication connection between the first playback device and the target second playback device. The device discovery data carries the device information of the authoritative end device and the reason for the status change, so that other playback devices in the multicast group can change the authoritative end device according to the received device discovery data and establish a communication connection with the new authoritative end device.

[0096] Specifically, when the state of the authoritative end device changes, it can generate a change authoritative end event, and then indicate the specific reason for the state change in the change authoritative end event, and send the change authoritative end event to each playback device in the multicast group to which it belongs as device discovery data. Each playback device performs exception processing based on the received device discovery data, changes the authoritative end device to which it is connected, and forwards the device discovery data, so that each playback device in the multicast group can quickly complete the access change of the authoritative end device.

[0097] Exemplarily, when the authoritative end device loses power and loses response, each second playback device will time out from receiving multicast, internally generate an authoritative end change event and enter the event processing logic, use historical device discovery data to determine whether it is the highest priority, and if it is the highest priority, it will establish the authoritative end as itself. If there are other high-priority devices in the historical device discovery data, it will wait to receive multicast communication data and change the access of the authoritative end device according to the received multicast communication data, and access other high-priority authoritative end devices.

[0098] As an implementation method, an external user can modify the priority of a specified device. When the playback device detects that its priority has changed, and the priority of the device is the highest priority playback device among the current playback devices, the internal processing logic of the change authority end event within the playback device will trigger the above-mentioned change authority end event.

[0099] As an example, the processing flow chart of the playback device for the authoritative end device change event can be as follows: Figure 4 As shown, when each playback device receives an authoritative end change event, it determines whether the local device is an authoritative end device. If so, it further receives authoritative end multicast data not sent by itself, that is, receives authoritative end multicast data sent by other authoritative end devices, and performs authoritative end device verification, verifies the verification code and priority of the new authoritative end device, and if the priority of the new authoritative end device is higher than that of the local device, determines the new authoritative end device as a new authoritative end device in the current network, and accesses the new authoritative end device as a client device.

[0100] In addition, it can also receive a request to change the authoritative end, and determine whether the priority of the authoritative end carried in the request is lower than that of the local device, and then reply to refuse to change the authoritative end device. If it is higher than the local device, accept the change of the authoritative end device and reply to accept the change of the authoritative end device.

[0101] If it is determined that the local device is not an authoritative end device, that is, the local device is a client device, then the local device automatically updates its own network status according to the change authority end event, thereby changing its own network status. As a possible implementation method, when the authoritative end device is offline, when the multicast data communication with the authoritative end device times out, it can further determine the authoritative end device existing in the current network and automatically connect to the authoritative end device. If there is no authoritative end device in the current network, the local device's network status is switched from the client device role to the authoritative end device role, so that other playback devices can subsequently access as client devices.

[0102] From a comprehensive analysis, it can be seen that in the embodiment of the present application, if the authoritative end device and the client device are in the same local area network, a multicast communication connection can be first established between the authoritative end device and the client device through a non-connection-oriented communication method in the local area network, and then the authoritative end device sends device information, media broadcast control data, and device discovery data to other playback devices in the multicast group to which it belongs through the multicast communication connection, so that the playback devices with which there is a multicast communication connection can verify, change, and perform other processing on the authoritative end device based on the received data.

[0103] In some embodiments, for possibility 4, that is, the first playback device and the second playback device are not located in the same local area network, when executing step S11 to implement the negotiation phase and the communication connection establishment phase, a point-to-point communication connection between the first playback device and the second playback device can be established in the following manner:

[0104] A point-to-point communication connection is established between the first playback device and the target second playback device using a cross-network connection tool, wherein the cross-network connection tool includes: virtual networking and P2P (Peer to Peer) tunnel communication.

[0105] Specifically, when the communication IP address of the target second playback device is known, the cross-network connection tool, such as a virtual network VPN, is used to set the first playback device and the target second playback device under the same virtual network, and then a point-to-point connection is established between the first playback device and the target second playback device. Alternatively, a communication tunnel between the target second playback device and the first playback device is established based on the communication IP address of the target second playback device by using P2P tunnel communication, and the communication tunnel is used as a point-to-point communication connection between the target second playback device and the first playback device.

[0106] As an example, Figure 5 As shown, it is assumed that there are devices A, B, and C in regional network X. At this time, device B is the authoritative end device in regional network X, and devices A and C are client devices of the authoritative end device. There are devices D, E, and F in regional network Y, among which device F is the authoritative end device in regional network Y. At this time, devices D and E are both connected to device F as client devices. If you want to achieve multi-screen audio and video synchronization of playback devices in regional network X and regional network Y, you can use the above-mentioned cross-network connection tool to establish a point-to-point communication network connection between device C in regional network X and device D in regional network Y.

[0107] On this basis, two playback devices connected across the network determine an authoritative end device according to a preset authoritative end device negotiation algorithm, such as Figure 5 In the process, device C and device D determine that device D is the authoritative end device according to the preset authoritative end device negotiation algorithm. At this time, device C and device D are similar to the authoritative end device and client device in multicast communication to exchange media broadcast control data, device discovery data of the authoritative end device, and device information of the authoritative end device, so that device C, as a client device, forwards the device information of device D to device A and device B in regional network X. During the process, device C can also generate a change of authoritative end event, determine itself as a new authoritative end device, and inform device A and device B of the change of authoritative end event based on multicast communication in regional network X.

[0108] Then, based on the device discovery data of the authoritative end device and the change of the authoritative end event, device A and device B are connected to device C, which is the new authoritative end device, as client devices. Similarly, devices E and F in area Y are also connected to device D, which is the new authoritative end device, as client devices according to the same processing flow.

[0109] In some embodiments, taking the first playback device as an authoritative end device as an example, the authoritative end device receives the device information of the client device sent by the client device through the point-to-point communication connection established between the authoritative end device and the client device. That is to say, if the target second playback device is a client device, the first playback device receives the device information of the client device sent by the target second playback device based on the point-to-point communication connection between the first playback device and the target second playback device.

[0110] Among them, similar to the device information of the authoritative end device, the device information of the client device includes: identification information of the client device, communication address information of the client device and priority information of the client device. In this way, by selecting the embodiment of the present application, the authoritative end device can uniformly manage each connected client device according to the received device information of the client device, which helps to ensure the effective synchronization of the entire multi-screen audio and video.

[0111] The above is a description of the negotiation phase and the communication connection establishment phase between the playback devices. After the authoritative end device and the client device successfully establish a point-to-point communication connection, the authoritative end device and the client device enter the interview phase. In the embodiment of the present application, the interaction and synchronization of information between the authoritative end device and the client device is specifically subdivided into the following interaction processes:

[0112] Interaction process 1: The authoritative end device and the client device in the same regional network are connected based on multicast communication to exchange media broadcast control data, device information of the authoritative end device, and device discovery data.

[0113] Interaction process 2: The authoritative end device and the client device that are not in the same regional network are connected based on a point-to-point communication connection to exchange media broadcast control data, device information of the authoritative end device, and device discovery data.

[0114] Interaction process 3: The authoritative end device and the client device exchange media playback description information, network delay calculation requests and other information related to audio and video synchronization adjustment based on the point-to-point communication connection between each other.

[0115] That is to say, in the embodiment of the present application, the interactive synchronization information between the authoritative end device and the client device is a general term for all data interacting between the authoritative end device and the client device, specifically including: media broadcast control data, device information of the authoritative end device / client device, device discovery data, media playback description information, network delay calculation request, etc.

[0116] As an example, the authoritative end device sends media playback description information to the client device through a point-to-point communication connection established between the authoritative end device and the client device. This process corresponds to the above-mentioned interview stage. Figure 6 As shown, the media playback description information includes: the playback task currently being played, the stream data source address corresponding to the playback task, audio and video stream information, time reference and playback progress. Specifically, the stream data source address is the media source address of the playback task, which can be a local storage path or a cloud storage path. The time reference includes: system time, which is the system time of each device itself, and the playback progress specifically refers to the timestamp corresponding to the currently playing music frame or video frame.

[0117] Based on this, the communication time of the synchronization information specifically refers to the communication time when the network delay calculation request is exchanged between the authoritative end device and the client device. In this way, in some embodiments, the target network delay between the authoritative end device and the client device can be determined based on the network delay calculation request exchanged between the authoritative end device and the client device, that is, the above step S13 can be implemented by the following steps:

[0118] Based on the point-to-point communication connection between the first playback device (when the first playback device is a client device) and the target second playback device (in this case, the target second playback device is an authoritative terminal device), a network delay calculation request is sent to the target second playback device;

[0119] The first playback device receives authoritative synchronization information returned by the target second playback device in response to the network delay calculation request, wherein the authoritative synchronization information carries the system time of the target second playback device;

[0120] A target network delay between the first playback device and the target second playback device is determined according to the sending time of the network delay calculation request and the receiving time of the authoritative synchronization information.

[0121] As an example, Figure 7 As shown, on the left side of the client device is the client synchronization request information sent by the client device, and the information name of the client synchronization request information corresponds to "Request" in the figure: "NetworkDelay", and on the right side is the authoritative synchronization information replied by the authoritative device, and the information name of the authoritative synchronization information replied by the authoritative device corresponds to "Response" in the figure: "NetworkDelay". Among them, the client synchronization request information carries the client device's: device IP address (corresponding to Device in the figure), device system time (corresponding to DeviceTime in the figure), playback progress (corresponding to DevicePlayTime in the figure), and device verification code (corresponding to DevicePublicKey in the figure). The authoritative synchronization information carries the authoritative device's: device IP address (corresponding to Authority in the figure), device system time (corresponding to AuthorityTime in the figure), playback progress (corresponding to AuthorityPlayTime in the figure), and device verification code (corresponding to AuthorityPublicKey in the figure).

[0122] In the embodiment of the present application, any communication data sent by the playback device will generate a historical record, which can be recorded in a historical log. Based on this, the client device can obtain the sending time of the network delay calculation request sent by the client device and the receiving time of the authoritative synchronization information sent by the authoritative end device by reading the historical log, and the authoritative end device can obtain the receiving time of the network delay calculation request received by the authoritative end device and the sending time of the authoritative synchronization information sent by the authoritative end device by reading the historical log.

[0123] As an implementation method, the sending time of the network delay calculation request sent by the client device is determined as x, the receiving time of the network delay calculation request received by the authoritative device is determined as y, and the receiving time of the authoritative synchronization information sent by the authoritative device is determined as z. At this time, on the client device, the target network delay t between the client device and the authoritative device can be determined based on x, z, that is, t=(xz) / 2.

[0124] Based on this, the time difference between the client device and the authoritative device can be further calculated as d=y-(x+t), where d is the time difference between the client device and the authoritative device. At this time, the time axis on the client device and the time axis on the authoritative device can be as follows Figure 8 Each time audio and video synchronization is performed, the corresponding time difference d and target network delay t can be calculated according to the interactive synchronization information, and recalibration is required after a period of time, specifically by re-executing the above steps S11 and S12 to perform a calibration.

[0125] In the embodiment of the present application, the authoritative synchronization information sent by the authoritative end device also carries the first data mapping relationship map1 of the authoritative end device, and the first data mapping relationship map1 is specifically the mapping relationship between the authoritative system time o1 of the authoritative end device and the authoritative playback timestamp p1 of the authoritative end device, that is, map1=[o1,p1]. In the embodiment of the present application, for the purpose of distinction, the authoritative system time refers to the system time of the authoritative end device, the client system time refers to the system time of the client device, the authoritative playback timestamp refers to the timestamp of the currently played video frame of the authoritative end device, and the client playback timestamp refers to the timestamp of the currently played video frame of the client device.

[0126] Among them, the first data mapping relationship is a playback record data generated and maintained locally by the authoritative end device, or it can be a playback record data directly generated by the authoritative end device based on the current playback timestamp and system time. When the client device sends a network delay calculation request, the first data mapping relationship is directly filled into the specified field in the returned authoritative synchronization information. After the client device receives the authoritative synchronization information, it parses the authoritative synchronization information, obtains the first data mapping relationship, and records it.

[0127] Specifically, in some embodiments, when executing step S13, the audio and video stream correction rate can be calculated by the following steps:

[0128] receiving and recording the first data mapping relationship, and constructing a second data mapping relationship based on the target network delay, the client system time of the first playback device, and the client playback timestamp of the first playback device;

[0129] Based on the first data mapping relationship and the second data mapping relationship, the audio and video stream correction rate is calculated according to the preset correction rate calculation formula.

[0130] For ease of understanding, the first data mapping relationship is referred to as map1 (o1, p1) and the second data mapping relationship is referred to as map2 (o2+d, p2) in this article, where map1 (o1, p1) is a mapping relationship between the system time o1 of the authoritative end device recorded, stored, and managed by the authoritative end device and the current playback timestamp p1 of the current task of the authoritative end device. According to the first data mapping relationship, it can be directly determined that the playback timestamp corresponding to the authoritative end device at the system time o1 is p1.

[0131] In the embodiment of the present application, the second data mapping relationship map2 (o2+d, p2) is a mapping relationship between the sum of the client system time o2 and the target time difference d (o2+d) recorded, stored, and managed on the client device, and the playback timestamp p2 of the current playback task of the client device. Here, map refers to a mapping relationship. For example, map1 (o1, p1) means that when the system time of the authoritative end device is o1, the corresponding playback timestamp of the current playback task should be p1. Similarly, for map2, when the system time of the client device is o2+d, the corresponding playback timestamp of the current playback task should be p2.

[0132] At this time, you can Fig. 9As shown, from the time axis, before the correction starts, the relationship between the playback time axis of the authoritative device and the time axis of the authoritative device satisfies the mapping relationship of map1, and the relationship between the playback time axis on the client device and the time axis of the client device satisfies the mapping relationship of map[o2,p2]. If the client device and the authoritative device are to maintain audio and video synchronization, the playback timestamp p2 on the playback time axis of the client device must be the same as the playback timestamp p2 on the playback time axis of the authoritative device, and the system time of the client device plus the time difference between the two devices (i.e., o2+d) must be the same as the system time o1 of the authoritative device, so that audio and video synchronization between the two can be achieved.

[0133] Based on this standard, the preset correction rate calculation phase is entered, and the client device executes step S14 to adjust the playback rate of the audio and video playback task of the client device with the help of the correction rate, so as to catch up with or wait for the playback progress of the authoritative terminal device. Specifically, in some embodiments, the correction rate can be calculated by the following formula:

[0134]

[0135] Among them, R is the correction rate of this audio and video synchronization, o1 is the system time of the authoritative end device, o2 is the system time of the client device, d is the target network delay time difference between the authoritative end device and the client device, N is the number of mapping relationship pairs participating in the correction rate calculation in the first data mapping relationship, r' is the correction rate of the previous mapping relationship pair, r is the correction rate of the current mapping relationship pair, and the mapping relationship pair is a pair of mapping relationship data consisting of the authoritative system time and the authoritative playback timestamp.

[0136] in, is the second data mapping relationship mentioned above, the The value of is the client playback timestamp p2 of the client device at the client system time o2+d. Similarly, is the second data mapping relationship, the The value of is the authoritative playback timestamp p1 of the authoritative end device at the authoritative system time o1.

[0137] Based on this, when calculating the preset correction rate, the first data mapping relationship and the second data mapping relationship can be queried to obtain the corresponding customer playback timestamp and authoritative playback timestamp under each different customer system time and authoritative system time, and then substitute them into the above formula to calculate the corresponding correction rate R of this audio and video synchronization.

[0138] When the above formula is executed, the calculation can be performed according to the following code:

[0139] for(Author) / / Code description: This line of code means to poll the first data mapping relationship

[0140] {

[0141] r= r+ (Client[o2 +d] - Author[o1]) / Client[o2 +d] / / Code description: Client[o2+d] is the playback timestamp p2 corresponding to the client device when the local system time is o2+d, Author[o1] is the playback timestamp p1 corresponding to the authoritative device when the local system time is o1, the equal sign is assignment, and the calculation result on the right is assigned to r on the left

[0142] }

[0143] R = r_past / Author.num

[0144] As an implementation method, the authoritative end device may send the first mapping relationship of map1 of the local device to the client device, and the client device may traverse according to the received first mapping relationship, calculate the ratio of each element in the first mapping relationship mapped to the client device, and calculate the average value r_past / Author.num according to the traversal result, that is, the corresponding correction rate R. Based on this, when executing step S14, the API (Application Programming Interface) provided by the player of the client device may be called through the correction rate R, and then the audio and video playback rate of the API may be adjusted to achieve the final audio and video synchronization effect.

[0145] Among them, by selecting the embodiment of the present application, the streaming media transmission protocol (such as the prior art solution of synchronizing through I frames or P frames) itself is no longer used for audio and video synchronization. Specifically, by calling the API provided by the player to control the playback rate, the data buffer of the client device can be controlled, and audio and video synchronization is not achieved without being restricted by the streaming media transmission protocol. In addition, since each device negotiates the authoritative end device by itself, a centralized idea is adopted. When the authoritative end device is offline or changed, a new authoritative end device can be quickly determined again, which can adapt to more application scenarios and is not only suitable for local area networks, but also for the Internet across regional networks. In addition, since the interactive synchronization information can carry the playback task and the streaming media address of the playback task, it only needs to add the storage address of the local streaming media to the synchronization information, which can be applied not only to online file playback synchronization, but also to local streaming media synchronization.

[0146] The technical effect of autonomous negotiation between playback devices of the authoritative end device provided in the embodiment of the present application is due to the fact that both the functional modules of the authoritative end device and the functional modules of the client device are integrated in the playback device. In this way, during the negotiation process, the playback device can determine which identity it belongs to based on the negotiation results, and specifically call which identity's functional module to perform the corresponding action.

[0147] As an implementation method, the functional modules in each playback device can be as follows: Fig.10 As shown, it includes the following parts:

[0148] Authoritative device function module, client device function module, player. Among them, the authoritative end device function module is further divided into: external control module, authoritative end communication module, data processing module, client management module, playback data collection module. The client device function module is further divided into: external control module, client communication module, data processing module, synchronization and prediction algorithm module, playback data collection module, playback control module.

[0149] Among them, the external control module is actually an inter-process communication module based on HTTP (Hypertext Transfer Protocol). External data requests can access the authoritative end device function module through this module. The authoritative end communication module is used to discover connectable playback devices and establish a communication connection with the discovered communication device. Specifically, it can be to establish multicast communication or Internet communication. If necessary, it can also establish a TCP, multicast communication connection or HTTP connection with the client device. The client communication module is used to receive multicast data sent from the authoritative end device. If necessary, it can also establish a TCP, multicast communication connection or HTTP connection with the authoritative end device.

[0150] The data processing module is specifically used for processing network data, device security verification, and executing the preset authoritative end device negotiation algorithm and other data processing.

[0151] The client management module is used to record and / or manage the client devices that have been connected to itself.

[0152] The playback data collection module is used to collect the playback data of the playback task through the data API interface provided by the player, including obtaining the media source of the playback task (such as the playback source URL (uniform resource locator) address), audio and video stream information, and synchronization information, etc.

[0153] The synchronization and prediction algorithm module is used to calculate the network delay between the playback devices, calculate the correction rate based on the calculation results, and verify the synchronization effect based on the playback data.

[0154] The playback control module is used to control the playback rate of the audio and video stream of the player's data API interface according to the correction rate calculated by the synchronization and prediction algorithm module.

[0155] For example, if the first playback device is an authoritative end device and the other second playback devices are client devices according to the negotiation result of the preset authoritative end negotiation algorithm during the execution of step S11, the first playback device calls its own pre-integrated authoritative end device function module, and the various parts in the authoritative end device function module cooperate to perform the corresponding authoritative end device communication, data management, client management, playback data collection and other actions. The other second playback devices call their own pre-integrated client device function modules to perform the corresponding client device data processing, synchronization and prediction algorithm execution, playback data collection, playback control and other actions.

[0156] That is to say, in the embodiment of the present application, before executing step S11, each playback device can act as both an authoritative end device and a client device because it has pre-integrated the functional modules of the authoritative end device and the functional modules of the client device. Before autonomous negotiation, it can act as both an authoritative end device and a client device. The specific device can only be determined after the authoritative end device negotiation algorithm preset in step S11 is executed. After the identity to which it belongs is determined through negotiation, the built-in functional module matching the identity (authoritative end device or client device) executes the corresponding action to complete the audio and video synchronization.

[0157] In a second aspect, the present application provides a multi-screen audio and video synchronization device, wherein, Fig.11 As shown, the device is applied to a first playback device, and the device 11 includes:

[0158] The negotiation module 110 is used to establish a communication connection between the first playback device and other second playback devices according to a preset authoritative end device negotiation algorithm; wherein the preset authoritative end device negotiation algorithm is a communication algorithm for each playback device to automatically determine an authoritative end device according to a communication message and establish a communication connection with the authoritative end device;

[0159] The information interaction module 111 is used to exchange synchronization information with the target second playback device that establishes the communication connection based on the communication connection, wherein the synchronization information includes: a time reference, and the time reference includes: a system time of the authoritative end device and a system time of the client device;

[0160] The data processing module 112 is used to calculate the correction rate of the audio and video stream according to the received time reference and the communication time of the synchronization information according to the preset correction rate calculation formula;

[0161] The playback control module 113 is used to adjust the playback rate of the audio and video of the playback task according to the audio and video stream correction rate when the first playback device is a client device that establishes a communication connection with the authoritative terminal device.

[0162] In a third aspect, an embodiment of the present application provides a multi-screen audio and video synchronization system, wherein the multi-screen audio and video synchronization system can be as follows Fig.12 As shown, it includes: a first playback device and a second playback device, wherein:

[0163] The first playback device or the second playback device is used to determine the authoritative end device and the client device according to a preset authoritative end device negotiation algorithm, and to establish a communication connection between the authoritative end device and the client device.

[0164] The preset authoritative end device negotiation algorithm is a communication algorithm for each playback device to automatically determine the authoritative end device according to the communication message and establish a communication connection with the authoritative end device. If the first playback device is an authoritative end device, the second playback device connected to the first playback device is a client device; if the second playback device is an authoritative end device, the first playback device connected to the second playback device is a client device;

[0165] The first playback device is used to exchange synchronization information with the second playback device that has established a communication connection based on the communication connection, wherein the synchronization information includes: a time reference, and the time reference includes: a system time of an authoritative end device and a system time of a client device;

[0166] When the first playback device is a client device, or when the second playback device is a client device, the client device is used to calculate the audio and video stream correction rate according to the received time reference and the communication time of the synchronization information according to a preset correction rate calculation formula, and adjust the audio and video playback rate of the playback task according to the audio and video stream correction rate.

[0167] The information or names of the information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of the information or the information.

[0168] In a fourth aspect, the exemplary embodiments of the present application further provide an electronic device, comprising: at least one processor; and a memory in communication with the at least one processor. The memory stores a computer program that can be executed by the at least one processor, and when the computer program is executed by the at least one processor, the electronic device performs a method according to an embodiment of the present application.

[0169] In a fifth aspect, an exemplary embodiment of the present application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present application.

[0170] In a sixth aspect, an exemplary embodiment of the present application further provides a computer program product, comprising a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present application.

[0171] refer to Fig.13 , the structural block diagram of the electronic device 1300 that can be used as the server or client of the present application will now be described, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0172] like Fig.13 As shown, the electronic device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the electronic device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0173] A plurality of components in the electronic device 1300 are connected to the I / O interface 1305, including: an input unit 1306, an output unit 1307, a storage unit 1308, and a communication unit 1309. The input unit 1306 may be any type of device capable of inputting information to the electronic device 1300, and the input unit 1306 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 1307 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1308 may include but is not limited to a disk, an optical disk. The communication unit 1309 allows the electronic device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0174] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above. For example, in some embodiments, the aforementioned multi-screen audio and video synchronization method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1300 via the ROM 1302 and / or the communication unit 1309. In some embodiments, the computing unit 1301 may be configured to perform the aforementioned multi-screen audio and video synchronization method in any other appropriate manner (e.g., by means of firmware).

[0175] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0176] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0177] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0178] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0179] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0180] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A multi-screen audio and video synchronization method, characterized in that: The method is applied to a first playback device, and the method includes: Establishing a communication connection between the first playback device and each of the other second playback devices according to a preset authoritative end device negotiation algorithm; wherein the preset authoritative end device negotiation algorithm is a communication algorithm for each playback device to automatically determine an authoritative end device according to a communication message and establish a communication connection with the authoritative end device; Based on the communication connection, exchanging synchronization information with the target second playback device that establishes the communication connection, wherein the synchronization information includes: a time reference, and the time reference includes: a system time of the authoritative end device and a system time of the client device; When the first playback device is a client device that establishes a communication connection with the authoritative end device, the audio and video stream correction rate is calculated according to a preset correction rate calculation formula based on the received time reference and the communication time of the synchronization information, and the audio and video playback rate of the playback task is adjusted according to the audio and video stream correction rate; Wherein, the preset correction rate calculation formula is: Among them, R is the correction rate of this audio and video synchronization, o1 is the authoritative system time of the authoritative end device, o2 is the client system time of the client device, d is the target network delay time difference between the authoritative end device and the client device, N is the number of mapping relationship pairs participating in the correction rate calculation in the first data mapping relationship, r' is the correction rate of the previous mapping relationship pair, r is the correction rate of the current mapping relationship pair, and the mapping relationship pair is a pair of mapping relationship data consisting of the authoritative system time and the authoritative playback timestamp. The client playback timestamp corresponding to the client device at client system time o2+d, The authoritative playback timestamp of the authoritative end device at the authoritative system time o1; The first data mapping relationship is a mapping relationship constructed by the authoritative end device according to the authoritative system time of the authoritative end device and the authoritative playback timestamp of the authoritative end device.

2. The method according to claim 1, characterized in that The establishing of a communication connection between the first playback device and other second playback devices according to a preset authoritative end device negotiation algorithm includes: According to the preset authoritative end device negotiation algorithm, the client device in the first playback device and the target second playback device is determined. If the first playback device is a client device, the target second playback device is determined to be an authoritative end device. If the first playback device is an authoritative end device, the target second playback device is determined to be a client device, wherein a multicast communication connection is established between the client device and the authoritative end device.

3. The method according to claim 2, characterized in that The step of exchanging synchronization information with a target second playback device with which the communication connection is established based on the communication connection includes: When the first playback device is a client device, based on the multicast communication connection between the first playback device and the target second playback device, media playback control data sent by the target second playback device is received, wherein the media playback control data includes: the time reference and the communication time of the synchronization information.

4. The method according to claim 2, characterized in that: The method further comprises: When the first playback device is an authoritative end device, based on the multicast communication connection, the device information of the first playback device is sent to each target second playback device in the multicast group to which the first playback device belongs, so that each target second playback device in the multicast group to which the first playback device belongs verifies the authoritative end device based on the received device information.

5. The method according to claim 2, characterized in that: The method further comprises: When the first playback device is an authoritative end device, based on the multicast communication connection, device discovery data is sent to each target second playback device in the multicast group to which the first playback device belongs, so that each target second playback device in the multicast group to which the first playback device belongs performs exception processing based on the device discovery data, and the exception processing includes: changing the authoritative end device.

6. The method according to claim 2, characterized in that The method further comprises: When the first playback device is an authoritative end device and the target second playback device is a client device, the target second playback device parses the multicast communication data packets in the multicast communication connection based on the multicast communication connection to establish a point-to-point communication connection between the authoritative end device and the client device; When the first playback device is a client device and the target second playback device is an authoritative end device, based on the multicast communication connection, the multicast communication data packets in the multicast communication connection are parsed to establish a point-to-point communication connection between the authoritative end device and the client device.

7. The method according to claim 1, characterized in that If the first playback device and the target second playback device are not located in the same local area network, establishing a communication connection between the first playback device and each of the other second playback devices according to a preset authoritative end device negotiation algorithm includes: A point-to-point communication connection is established between the first playback device and the target second playback device using a cross-network connection tool, wherein the cross-network connection tool includes: virtual networking and P2P tunnel communication.

8. The method according to any one of claims 6 or 7, characterized in that: When the first playback device is an authoritative end device, the method further includes: Based on the point-to-point communication connection, device information of the client device sent by the client device is received.

9. The method according to any one of claims 6 or 7, characterized in that: When the first playback device is a client device, the method further includes: Based on a point-to-point communication connection, a network delay calculation request is sent to the authoritative end device; and authoritative synchronization information returned by the authoritative end device in response to the network delay calculation request is received, wherein the authoritative synchronization information carries the system time of the authoritative end device; The target network delay between the authoritative end device and the client device is determined according to the sending time of the network delay calculation request and the receiving time of the authoritative synchronization information.

10. The method according to claim 9, characterized in that The authoritative synchronization information also carries a first data mapping relationship of the authoritative terminal device, and the method further includes: receiving and recording the first data mapping relationship, and constructing a second data mapping relationship based on the target network delay, the client system time of the first playback device, and the client playback timestamp of the first playback device; Based on the first data mapping relationship and the second data mapping relationship, the audio and video stream correction rate is calculated according to the preset correction rate calculation formula.

11. A multi-screen audio and video synchronization device, characterized in that: The device is applied to a first playback device, and comprises: A negotiation module, used to establish a communication connection between the first playback device and other second playback devices according to a preset authoritative end device negotiation algorithm; wherein the preset authoritative end device negotiation algorithm is a communication algorithm for each playback device to automatically determine an authoritative end device according to a communication message and establish a communication connection with the authoritative end device; An information interaction module, used for exchanging synchronization information with a target second playback device with which a communication connection is established based on the communication connection, wherein the synchronization information includes: a time reference, and the time reference includes: a system time of an authoritative end device and a system time of a client device; A data processing module, used to calculate the correction rate of the audio and video stream according to a preset correction rate calculation formula based on the received time reference and the communication time of the synchronization information; A playback control module, configured to adjust the playback rate of the audio and video of the playback task according to the audio and video stream correction rate when the first playback device is a client device that establishes a communication connection with the authoritative terminal device; Wherein, the preset correction rate calculation formula is: Among them, R is the correction rate of this audio and video synchronization, o1 is the authoritative system time of the authoritative end device, o2 is the client system time of the client device, d is the target network delay time difference between the authoritative end device and the client device, N is the number of mapping relationship pairs participating in the correction rate calculation in the first data mapping relationship, r' is the correction rate of the previous mapping relationship pair, r is the correction rate of the current mapping relationship pair, and the mapping relationship pair is a pair of mapping relationship data consisting of the authoritative system time and the authoritative playback timestamp. The client playback timestamp corresponding to the client device at client system time o2+d, The authoritative playback timestamp of the authoritative end device at the authoritative system time o1; The first data mapping relationship is a mapping relationship constructed by the authoritative end device according to the authoritative system time of the authoritative end device and the authoritative playback timestamp of the authoritative end device.

12. A multi-screen audio and video synchronization system, characterized in that: The multi-screen audio and video synchronization system includes: a first playback device and a second playback device, wherein: The first playback device or the second playback device is used to determine the authoritative end device and the client device according to a preset authoritative end device negotiation algorithm, and establish a communication connection between the authoritative end device and the client device; The preset authoritative end device negotiation algorithm is a communication algorithm for each playback device to automatically determine the authoritative end device according to the communication message and establish a communication connection with the authoritative end device; if the first playback device is an authoritative end device, the second playback device connected to the first playback device is a client device, and if the second playback device is an authoritative end device, the first playback device connected to the second playback device is a client device; The first playback device is used to exchange synchronization information with the second playback device that has established a communication connection based on the communication connection, wherein the synchronization information includes: a time reference, and the time reference includes: a system time of an authoritative end device and a system time of a client device; When the first playback device is a client device, or when the second playback device is a client device, the client device is used to calculate the audio and video stream correction rate according to the received time reference and the communication time of the synchronization information according to a preset correction rate calculation formula, and adjust the audio and video playback rate of the playback task according to the audio and video stream correction rate; Wherein, the preset correction rate calculation formula is: Among them, R is the correction rate of this audio and video synchronization, o1 is the authoritative system time of the authoritative end device, o2 is the client system time of the client device, d is the target network delay time difference between the authoritative end device and the client device, N is the number of mapping relationship pairs participating in the correction rate calculation in the first data mapping relationship, r' is the correction rate of the previous mapping relationship pair, r is the correction rate of the current mapping relationship pair, and the mapping relationship pair is a pair of mapping relationship data consisting of the authoritative system time and the authoritative playback timestamp. The client playback timestamp corresponding to the client device at client system time o2+d, The authoritative playback timestamp of the authoritative end device at the authoritative system time o1; The first data mapping relationship is a mapping relationship constructed by the authoritative end device according to the authoritative system time of the authoritative end device and the authoritative playback timestamp of the authoritative end device.

13. An electronic device, characterized in that: The electronic device comprises: Processor; and Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 10.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 1-10.

Images