An audio synchronous playing method and related device

Abstract

The application provides an audio synchronous playing method and related device, which realizes that a master device periodically sends a data packet to a slave device in the process of playing the same audio on the master device and the slave device, the slave device can further adjust the expected playing time of the local audio stream according to the expected playing time of the data packet, so that the time difference of playing the same audio on the master device and the slave device is within a preset range, and the effect of synchronous playing of the audio is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data transmission, and particularly relates to an audio synchronous playing method and related device. BACKGROUND

[0002] With the continuous development of audio technology, in order to improve the human hearing experience, when playing audio, different audio output devices can be used to output audio of different channels, so as to achieve the purpose of playing multi-channel audio.

[0003] When multiple different audio output devices output the same audio data, how to ensure that the time of the multiple different audio output devices outputting the same audio data is consistent, and avoid the occurrence of playing delay between the multiple different audio output devices, needs to be further researched. SUMMARY

[0004] The present application provides an audio synchronous playing method and related device, which realizes that the master device and the slave device can dynamically negotiate in the process of playing audio, and realizes the effect of synchronous playing of audio.

[0005] In a first aspect, the present application provides an audio synchronous playing method, which comprises the following steps: a slave device receives a first audio stream, divides the first audio stream into multiple data packets with a first length, and determines an expected playing time of each data packet in the first audio stream; the slave device receives a first synchronization data packet sent by a master device, the first synchronization data packet comprising first audio data, a first sequence number of the first synchronization data packet, and a first expected playing time of the first synchronization data packet; the slave device determines a first local data packet based on the first audio data in the first synchronization data packet; the first local data packet comprises second audio data, a second sequence number of the first local data packet, and a second expected playing time of the first local data packet, and the second audio data partially matches or completely matches the first audio data; the slave device receives a second audio stream, which is an audio stream after the first audio stream; the slave device sets a starting playing time of the second audio stream as a third expected playing time based on a difference between the first sequence number and the second sequence number and a first offset between the second audio data and the first audio data; and the slave device plays the second audio stream based on the third expected playing time.

[0006] Through the method provided in the first aspect, the master device and the slave device can dynamically negotiate in the process of playing audio, and the effect of synchronous playing of audio can be realized.

[0007] In a possible implementation manner of the first aspect, the slave device sets a starting playing time of the second audio stream as a third expected playing time based on a difference between the first sequence number and the second sequence number and the first offset of the second audio data and the first audio data, and specifically includes: in a case where the difference between the first sequence number and the second sequence number is 0 and the first offset is greater than a first value, the slave device adds data of the first offset in front of the second audio stream to obtain a second updated audio stream; the slave device sets the starting playing time of the second updated audio stream as the third expected playing time; in a case where the difference between the first sequence number and the second sequence number is 0 and the first offset is less than a second value, the slave device removes data of the first offset in front of the second audio stream to obtain a second updated audio stream; the slave device sets the starting playing time of the second updated audio stream as the third expected playing time; and in a case where the difference between the first sequence number and the second sequence number is 0 and the first offset is greater than the second value and less than the first value, the slave device sets the starting playing time of the second audio stream as the third expected playing time.

[0008] In this way, in a case where the first sequence number and the second sequence number are the same, if the first offset is greater than the first value, that is, the front part of the audio content in the first local data packet and the rear part of the audio content in the first synchronization data packet are the same, the slave device adds data of the first offset in front of the second audio stream, so that the audio content in the local data packet and the synchronization data packet in the second audio stream is completely the same.

[0009] If the first offset is less than the first value, that is, the rear part of the audio content in the first local data packet and the front part of the audio content in the first synchronization data packet are the same, the slave device removes data of the first offset in front of the second audio stream, so that the audio content in the local data packet and the synchronization data packet in the second audio stream is completely the same.

[0010] If the first offset is equal to the first value, the master device and the slave device are synchronously played, and the slave device does not need to adjust the second audio stream.

[0011] In a possible implementation manner of the first aspect, the slave device sets the starting playing time of the second updated audio stream as the third expected playing time, and specifically includes: the slave device determines the starting playing time of the second updated audio stream as the third expected playing time based on the expected playing time of the last data packet in the first audio stream and the playing duration of each data packet; and the slave device plays the second audio stream based on the third expected playing time, and specifically includes: the slave device plays the second updated audio stream based on the third expected playing time.

[0012] In this way, in a case where the first sequence number and the second sequence number are the same, the slave device can adjust the second audio stream based on the first offset, so as to adjust the playing time of each byte in the second audio stream.

[0013] Or,

[0014] The slave device sets a start playing time of the second audio stream as the third expected playing time, specifically including: determining the start playing time of the second audio stream as the third expected playing time based on an expected playing time of a last data packet in the first audio stream of the slave device, and a playing duration of each data packet.

[0015] In this way, in the case that the first sequence number is the same as the second sequence number, the time of the second audio stream can also not be adjusted.

[0016] In combination with the first aspect, in a possible implementation manner, the slave device sets the start playing time of the second audio stream as the third expected playing time based on a difference between the first sequence number and the second sequence number, and a first offset between the second audio data and the first audio data, specifically including: in the case that the first sequence number and the second sequence number are not 0, and the first offset is greater than a first value, the slave device adds data of the first offset in front of the second audio stream to obtain a second updated audio stream; the slave device sets a start playing time of the second updated audio stream as the third expected playing time; in the case that the difference between the first sequence number and the second sequence number is not 0, and the first offset is less than a second value, the slave device removes data of the first offset in front of the second audio stream to obtain a second updated audio stream; the slave device sets a start playing time of the second updated audio stream as the third expected playing time; in the case that the difference between the first sequence number and the second sequence number is not 0, and the first offset is greater than the second value and less than the first value, the slave device sets the start playing time of the second audio stream as the third expected playing time.

[0017] In this way, in the case that the first sequence number is different from the second sequence number, if the first offset is greater than the first value, that is, the front part of the audio content in the first local data packet and the rear part of the audio content in the first synchronization data packet are the same, the slave device adds data of the first offset in front of the second audio stream, so that the audio content in the local data packet and the synchronization data packet in the second audio stream is completely the same.

[0018] If the first offset is less than the first value, that is, the rear part of the audio content in the first local data packet and the front part of the audio content in the first synchronization data packet are the same, the slave device removes data of the first offset in front of the second audio stream, so that the audio content in the local data packet and the synchronization data packet in the second audio stream is completely the same.

[0019] If the first offset is equal to the first value, the master device and the slave device are synchronously played, and the slave device does not need to adjust the second audio stream.

[0020] With reference to the first aspect, in a possible implementation manner, the slave device sets a start playing time of the second updated audio stream as the third expected playing time, and specifically includes: the slave device determines the start playing time of the second updated audio stream as the third expected playing time based on the expected playing time of the last data packet in the first audio stream, the playing time length of each data packet, and the difference between the first sequence number and the second sequence number; and the slave device plays the second audio stream based on the third expected playing time, and specifically includes: the slave device plays the second updated audio stream based on the third expected playing time.

[0021] In this way, in the case that the first sequence number is different from the second sequence number, the slave device can adjust the expected playing time of each byte in the second audio stream based on the first offset.

[0022] Alternatively,

[0023] The slave device sets the start playing time of the second audio stream as the third expected playing time, and specifically includes: the slave device determines the start playing time of the second audio stream as the third expected playing time based on the expected playing time of the last data packet in the first audio stream, the playing time length of each data packet, and the difference between the first sequence number and the second sequence number.

[0024] In this way, in the case that the first sequence number is different from the second sequence number, the slave device can adjust the expected playing time of each byte in the second audio stream based on the difference between the first sequence number and the second sequence number.

[0025] With reference to the first aspect, in a possible implementation manner, the slave device plays the second updated audio stream based on the third expected playing time, and specifically includes: the slave device divides the second updated audio stream into a plurality of data packets containing the first length, and determines the expected playing time of each data packet in the second updated audio stream based on the third expected playing time and the playing time length of each data packet; and the slave device plays the second updated audio stream based on the expected playing time of each data packet in the second updated audio stream.

[0026] With reference to the first aspect, in a possible implementation manner, the slave device plays the second audio stream based on the third expected playing time, and specifically includes: the slave device divides the second audio stream into a plurality of data packets containing the first length, and determines the expected playing time of each data packet in the second audio stream based on the third expected playing time and the playing time length of each data packet; and the slave device plays the second audio stream based on the expected playing time of each data packet in the second audio stream.

[0027] With reference to the first aspect, in a possible implementation manner, after the slave device plays the second audio stream based on the third expected playing time, the method further includes:

[0028] The slave device receives a second synchronization data packet sent by the master device, the second synchronization data packet comprising third audio data, a third sequence number of the second synchronization data packet, and a fourth expected playing time of the second synchronization data packet; the slave device determines a second local data packet based on the third audio data in the second synchronization data packet; the second local data packet comprises fourth audio data, a fourth sequence number of the second local data packet, and a fifth expected playing time of the second local data packet; and the fifth expected playing time and the fourth expected playing time are within a first preset value, and the third audio data and the fourth audio data are the same.

[0029] In this way, after the slave device adjusts the second audio stream, the difference between the expected times of the same audio content in the master device and the slave device is within the first preset value, so that the same audio content is played synchronously.

[0030] With reference to the first aspect, in a possible implementation, the plurality of data packets in the second audio stream comprises a third local data packet, and a start playing time of the third local data packet is a third expected playing time; the slave device plays the second updated audio stream based on the expected playing time of each data packet in the second updated audio stream, specifically comprising: the slave device plays the third local data packet based on the third expected playing time of the third local data packet.

[0031] With reference to the first aspect, in a possible implementation, the slave device plays the third local data packet based on the third expected playing time of the third local data packet, specifically comprising: the slave device acquires a system time; the slave device determines an actual playing time of the third local data packet based on the system time, a number of data packets in front of a playing cache queue where the third local data packet is located, and a playing duration of each data packet; when the actual playing time is greater than the third expected playing time and the difference between the actual playing time and the third expected playing time is greater than a first duration, the slave device discards the third local data packet; when the actual playing time is less than the third expected playing time and the difference between the actual playing time and the third expected playing time is greater than the first duration, the slave device adds data packets with the first duration before the second local data packet, the first duration being the difference between the system time and the third expected playing time; and when the actual playing time is less than or greater than the third expected playing time and the difference between the actual playing time and the third expected playing time is less than the first duration, the slave device plays the third local data packet based on the third expected playing time.

[0032] The system time can be understood as a current time.

[0033] In this way, the slave device can also adjust whether to play a data packet based on the actual playing time and the expected playing time of each data packet.

[0034] With reference to the first aspect, in a possible implementation manner, the slave device determines the first local data packet based on the first audio data in the first synchronization data packet, and specifically includes: the slave device determines a byte position corresponding to a highest peak in the first synchronization data packet; the slave device acquires one or more local peaks matching the highest peak and byte positions corresponding to the one or more local peaks in the first audio stream; the slave device determines a first local peak from the one or more local peaks, where an audio data segment where the first local peak is located is the same as the first audio data; and the slave device determines the first local data packet based on a byte position corresponding to the first local peak. In this way, the first local data packet is matched based on the peak, and the calculation amount of the slave device can be reduced.

[0035] In a second aspect, the present application provides an electronic device, which is a slave device. The slave device includes one or more processors, one or more memories, the one or more memories are coupled to the one or more processors, the one or more memories are configured to store computer program codes, the computer program codes include computer instructions, and the one or more processors invoke the computer instructions to enable the slave device to perform the audio synchronization playing method provided in any one of the first aspect.

[0036] In a third aspect, the present application provides a computer readable storage medium, which includes instructions, and when the instructions are run on the slave device, enable the slave device to perform the audio synchronization playing method provided in any one of the first aspect.

[0037] In a fourth aspect, the present application provides a computer program product, and when the computer program product is run on the slave device, enable the slave device to perform the audio synchronization playing method provided in any one of the first aspect.

[0038] For beneficial effects of the second aspect to the fourth aspect, refer to the description of the beneficial effects of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 A schematic diagram of an electronic device 100 provided by an embodiment of the present application playing stereo audio through a master device 200 and a slave device 300 by means of a two-way audio line;

[0040] Figure 2 And Figure 3 A schematic diagram of an electronic device 100 provided by an embodiment of the present application playing stereo audio through a master device 200 and a slave device 300 by means of wireless transmission;

[0041] Figure 4 A schematic diagram of a system architecture provided by an embodiment of the present application;

[0042] Figure 5A structural schematic diagram of an electronic device 100 provided by an embodiment of the present application;

[0043] Figure 6 A structural schematic diagram of a master device 200 provided by an embodiment of the present application;

[0044] Figure 7 A functional module schematic diagram of a master device 200 provided by an embodiment of the present application;

[0045] Figure 8 A functional module schematic diagram of a slave device 300 provided by an embodiment of the present application;

[0046] Figure 9A A schematic diagram of how a master device 200 divides data packets provided by an embodiment of the present application;

[0047] Figure 9B A schematic diagram of how a master device 200 timestamps data packets provided by an embodiment of the present application;

[0048] Figures 10-14 A schematic diagram of how a slave device 300 determines an audio segment matching a synchronization data packet i provided by an embodiment of the present application;

[0049] Figures 15A-15D A set of schematic diagrams provided by an embodiment of the present application;

[0050] Figures 16A-16D A set of schematic diagrams provided by an embodiment of the present application;

[0051] Figures 17A-17C A set of schematic diagrams provided by an embodiment of the present application;

[0052] Figures 18A-18C A set of schematic diagrams provided by an embodiment of the present application;

[0053] Figures 19A-19C A set of schematic diagrams provided by an embodiment of the present application;

[0054] Figures 20A-20C A set of schematic diagrams provided by an embodiment of the present application;

[0055] Figures 21A-21C A set of schematic diagrams provided by an embodiment of the present application;

[0056] Figures 22-24 A schematic diagram of how a slave device 300 plays an audio stream provided by an embodiment of the present application;

[0057] Figure 25 A flow schematic diagram of an audio synchronous playing method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; the "and / or" in the text only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0059] Hereinafter, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more features. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" is two or more than two.

[0060] The term "user interface (UI)" in the following embodiments of the present application is a medium interface for interaction and information exchange between an application program or an operating system and a user, which realizes the conversion between the internal form of information and the form that the user can accept. The commonly used form of user interface is graphic user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icon, button, menu, tab, text box, dialog box, status bar, navigation bar, Widget, etc. displayed in the display screen of an electronic device.

[0061] There are two ways to play multi-channel audio at present. Way one: wired transmission. Way two: wireless transmission.

[0062] First, the specific implementation of multi-channel audio playback through wired transmission in way one is introduced. The multi-channel audio playback can also be referred to as stereo audio playback. Multi-channel audio or stereo audio refers to audio channel information including at least two different channel information, such as left channel and right channel.

[0063] Figure 1 An exemplary schematic diagram of the electronic device 100 playing stereo audio through the master device 200 and the slave device 300 through a split audio line is shown.(No need to split the package).

[0064] As Figure 1As shown, the electronic device 100 is connected to the master device 200 through the first audio line and to the slave device 300 through the second audio line. The master device 200 can be referred to as a master device, and the slave device 300 can be referred to as a slave device.

[0065] Specifically, the electronic device 100 sends the audio stream to the master device 200 through the first audio line, and simultaneously, the electronic device 100 sends the audio stream to the slave device 300 through the second audio line.

[0066] It should be noted that the electronic device 100 simultaneously sends the audio stream to the master device 200 through the first audio line and to the slave device 300 through the second audio line.

[0067] The master device 200 and the slave device 300 directly play out the audio stream sent by the electronic device 100 through the split audio line through hardware (such as a loudspeaker or a speaker).

[0068] In this way, in an ideal case, the master device 200 and the slave device 300 simultaneously receive the audio stream sent by the electronic device 100, and the effect of synchronous playback can be achieved.

[0069] The first mode has the following defects:

[0070] Defect 1: This mode requires that the hardware devices of the master device 200 and the slave device 300 are quite or the same, and has a high requirement for hardware. At the same time, the volume, sound effect, and other algorithms need to be embedded in the hardware layer to avoid the problem that the software processing affects the master device 200 and the slave device 300 cannot play synchronously.

[0071] Defect 2: In the first mode, the electronic device 100 sends the audio stream to the master device 200 and the slave device 300 through the split audio line, and the master device 200 and the slave device 300 are loudspeakers or speakers, which have a single function and do not support other functions, such as Bluetooth playback or networked playback.

[0072] First, the specific implementation of the second mode for realizing multi-channel playback of audio through wireless transmission is introduced.

[0073] Figure 2 And Figure 3 An exemplary schematic diagram of the electronic device 100 playing stereo audio based on the wireless transmission mode through the master device 200 and the slave device 300 is shown.

[0074] As Figure 2As shown, electronic device 100 establishes a wireless connection with master device 200 via Bluetooth or Wi-Fi. Electronic device 100 also establishes a wireless connection with slave device 300 via Bluetooth or Wi-Fi. Subsequently, electronic device 100 sends an audio stream to master device 200 through the wireless connection. Upon receiving the audio stream, master device 200 segments the audio stream, adding a sequence number and timestamp (PTS) information to each audio data packet. Then, master device 200 sends the audio data packets to its playback buffer.

[0075] Simultaneously, electronic device 100 sends audio data packets tagged with serial numbers and timestamps to slave device 300 via a wireless connection. After receiving the audio data packets tagged with serial numbers and timestamps, slave device 300 plays the audio data packets based on the timestamps.

[0076] In such Figure 3 As shown, electronic device 100 establishes a wireless connection with master device 200 via Bluetooth or Wi-Fi, and master device 200 establishes a wireless connection with slave device 300 via Bluetooth or Wi-Fi. Then, electronic device 100 sends audio data packets to master device 200 through the wireless connection. After receiving the audio data packets, master device 200 plays the audio data packets based on timestamps.

[0077] Simultaneously, the master device 200 sends audio data packets to the slave device 300 via a wireless connection. After receiving the audio data packets sent by the master device 200, the slave device 300 sends the audio data packets to the playback buffer on the smart device 300.

[0078] The master device 200 and the slave device 300 read data from the local playback buffer and play the audio synchronously according to the timestamp in the audio data packet.

[0079] Method 2 has the following drawbacks:

[0080] Defect 1: Long audio data path and significant audio-visual latency. The audio data played by the slave device 300 needs to be processed by the master device 200, and data transmission between the master device 200 and the slave device 300 also takes time.

[0081] Defect 2: Wireless transmission is susceptible to interference from the surrounding environment (such as in scenarios with many electronic devices or near a wireless mouse and keyboard), leading to unstable playback and issues like stuttering and popping sounds. The network connection status of the master and slave devices can also affect playback performance (audio-video latency, stuttering, popping sounds, etc.).

[0082] Based on this, the embodiment of the present application provides an audio synchronous playing method, the method comprises the following steps:

[0083] Step one: the electronic device 100 respectively sends the audio stream to the master device 200 and the slave device 300.

[0084] Optionally, the electronic device 100 can send the audio stream to the master device 200 and the slave device 300 through wired connection or wireless connection respectively.

[0085] Step two: after the master device 200 and the slave device 300 respectively receive the audio stream sent by the electronic device 100, the master device 200 obtains the audio stream of the first sound channel from the audio stream, and the master device 200 obtains the audio stream of the second sound channel from the audio stream.

[0086] The master device 200 will package the audio stream of the first sound channel according to the preset rules, mark the sequence number, time stamp and the like, and obtain a data packet. The slave device 300 will also package the audio stream of the second sound channel according to the preset rules, mark the sequence number, time stamp and the like, and obtain a data packet.

[0087] Optionally, the electronic device 100 can also send different audio streams to the master device 200 and the slave device 300 respectively, for example, the electronic device 100 sends the audio stream of the first sound channel to the master device 200, and the electronic device 100 sends the audio stream of the second sound channel to the slave device 300.

[0088] Step three: after obtaining the data packet, the master device 200 and the slave device 300 send the data packet into the respective hardware modules for playing.

[0089] In this way, the master device 200 plays the audio stream of the first sound channel, and the slave device 300 plays the audio stream of the second sound channel, the audio content in the audio stream of the first sound channel and the audio stream of the second sound channel is the same, only the sound channel is different, that is, the first sound channel is different from the second sound channel, for example, the first sound channel is the left sound channel, and the second sound channel is the right sound channel. The effect of playing stereo audio through the master device 200 and the slave device 300 is realized.

[0090] Optionally, before starting to play the audio, the master device 200 and the slave device 300 can negotiate the sound channel information of the master device 200 and the slave device 300 playing the audio. For example, the master device 200 plays the audio stream of the first sound channel, and the slave device 300 plays the audio stream of the second sound channel.

[0091] The audio stream processed by the master device 200 in the following embodiments of the present application is the audio stream of the first sound channel, and the audio stream processed by the slave device 300 in the following embodiments of the present application is the audio stream of the second sound channel.

[0092] It should be noted that the embodiments of this application can also be applied to scenarios where the master device 200 and the slave device 300 play audio data of the same channel, so that the audio data played by the slave device 300 is synchronized with the audio data played by the master device 200. The embodiments of this application do not limit this.

[0093] During playback, due to differences in the latency of audio stream transmission in the data channel, latency in hardware processing of the master device 200 and slave device 300, data loss, etc., the master device 200 and slave device 300 may experience playback asynchrony. Therefore, the master device 200 and slave device 300 need to negotiate to achieve the effect of synchronous audio playback.

[0094] Specifically, the master device 200 sends a data packet (which can be referred to as a synchronization data packet) to the slave device 300 every preset number of data packets. The synchronization data packet includes the sequence number, timestamp, and PCM audio data of the data packet. After receiving the synchronization data packet sent by the master device 200 at fixed intervals, the slave device 300 performs correlation calculations based on its local data packets and the synchronization data packet to determine a local data packet that matches the synchronization data packet. The slave device 300 calculates the offset and timestamp difference between the matching local data packet and the synchronization data packet. Based on the offset and timestamp difference between the matching local data packet and the synchronization data packet, the slave device 300 adjusts the expected playback time of the audio stream. For details, please refer to the relevant descriptions in the subsequent embodiments; these will not be repeated here.

[0095] The audio synchronization playback method provided in this application embodiment enables dynamic playback synchronization adjustment when the same audio is output through different channels on different devices, ensuring the effect of synchronized audio playback on different devices.

[0096] Figure 4 An exemplary schematic diagram of a system architecture provided by an embodiment of this application is shown.

[0097] like Figure 4 As shown, electronic device 100 establishes a communication connection with host device 200. Then, electronic device 100 sends an audio stream to host device 200 through the data channel between them.

[0098] Electronic device 100 establishes a communication connection with slave device 300. Subsequently, electronic device 100 sends an audio stream to slave device 300 through the data channel between electronic device 100 and slave device 300.

[0099] Exemplarily, the electronic device 100 and the master device 200 or the slave device 300 can establish a communication connection in a wired manner or a wireless manner.

[0100] Exemplarily, the wireless connection established between the electronic device 100 and the master device 200 by using the wireless communication technology includes but is not limited to: a wireless local area network (WLAN) technology, a bluetooth, an infrared, a near field communication (NFC), a ZigBee, and other wireless communication technologies developed later.

[0101] Exemplarily, the wired connection established between the electronic device 100 and the master device 200 by using the wired communication technology includes but is not limited to: the electronic device 100 establishes a wired connection with the master device 200 and the slave device 300 respectively by using a two-in-one audio line.

[0102] The above connection manners are only used for explaining the present application, and should not be construed as limitation.

[0103] The electronic device 100 can establish a communication connection with more devices, for example, the electronic device 100 establishes a communication connection with the master device 200, the slave device 300 and the slave device 400, and the embodiments of the present application do not limit this.

[0104] Figure 5 Exemplarily, a structural schematic diagram of the electronic device 100 is shown.

[0105] The electronic device 100 can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, and a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, and / or a smart city device, and the specific type of the electronic device is not specially limited in the embodiments of the present application.

[0106] The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0107] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0108] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices or integrated in one or more processors.

[0109] The controller can generate operation control signals according to the instruction operation code and the timing signal, complete the control of fetching and executing instructions.

[0110] The processor 110 can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save instructions or data that have just been used or are used repeatedly by the processor 110. If the processor 110 needs to use the instructions or data again, it can directly call from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thereby improving the efficiency of the system.

[0111] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0112] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 can contain multiple sets of I2C bus. The processor 110 can be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces respectively. For example, the processor 110 can be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface, and the touch function of the electronic device 100 is realized.

[0113] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can contain multiple sets of I2S bus. The processor 110 can be coupled to the audio module 170 through the I2S bus, and communication between the processor 110 and the audio module 170 is realized. In some embodiments, the audio module 170 can deliver audio signals to the wireless communication module 160 through the I2S interface, and the function of answering a phone through a Bluetooth headset is realized.

[0114] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled through the PCM bus interface. In some embodiments, the audio module 170 can also deliver audio signals to the wireless communication module 160 through the PCM interface, and the function of answering a phone through a Bluetooth headset is realized. Both the I2S interface and the PCM interface can be used for audio communication.

[0115] The UART interface is a universal serial data bus, which is used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is usually used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface, and the Bluetooth function is realized. In some embodiments, the audio module 170 can deliver audio signals to the wireless communication module 160 through the UART interface, and the function of playing music through a Bluetooth headset is realized.

[0116] The MIPI interface can be used to connect the processor 110 and the display screen 194, the camera 193 and other peripheral devices. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), and the like. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to implement the photographing function of the electronic device 100. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the electronic device 100.

[0117] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 and the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, and the like. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, and the like.

[0118] The USB interface 130 is an interface that conforms to the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transmit data between the electronic device 100 and a peripheral device. It can also be used to connect a headset to play audio through the headset. The interface can also be used to connect other electronic devices, such as AR devices and the like.

[0119] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the electronic device 100. In other embodiments of the present application, the electronic device 100 can also use different interface connection methods or combinations of multiple interface connection methods in the above embodiments.

[0120] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input through the wireless charging coil of the electronic device 100. The charging management module 140 can charge the battery 142 while also providing power to the electronic device through the power management module 141.

[0121] The power management module 141 is configured to connect the battery 142 and the charging management module 140 to the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the display 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 can also be configured to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), and the like. In some embodiments, the power management module 141 can also be disposed in the processor 110. In some embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0122] The wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, and the like.

[0123] The antenna 1 and the antenna 2 are configured to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be configured to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some embodiments, the antennas can be used in combination with a tuning switch.

[0124] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G and the like applied to the electronic device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), and the like. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and perform filtering, amplification, and the like on the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 can also amplify signals modulated by the modem processor, and radiate the amplified signals as electromagnetic waves through the antenna 1. In some embodiments, at least part of the function modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least part of the function modules of the mobile communication module 150 and at least part of the modules of the processor 110 can be disposed in the same device.

[0125] The modem processor can include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is configured to demodulate a received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a microphone 170B, etc.), or displays an image or a video through the display 194. In some embodiments, the modem processor can be a separate device. In other embodiments, the modem processor can be independent of the processor 110 and disposed in the same device as the mobile communication module 150 or other functional modules.

[0126] The wireless communication module 160 can provide a wireless communication solution including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signal, and transmits the processed signal to the processor 110. The wireless communication module 160 can also receive a signal to be transmitted from the processor 110, perform frequency modulation, amplification, and convert the signal into electromagnetic wave radiation via the antenna 2.

[0127] In some embodiments, the antenna 1 and the mobile communication module 150 of the electronic device 100 are coupled, and the antenna 2 and the wireless communication module 160 are coupled, so that the electronic device 100 can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include a global positioning system (GPS), a global navigation satellite system (GLONASS), a beidu navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and / or a satellite based augmentation systems (SBAS).

[0128] The electronic device 100 implements a display function through a GPU, a display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs, which execute program instructions to generate or change display information.

[0129] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diodes (QLED), or the like. In some embodiments, the electronic device 100 can include one or N display screens 194, where N is a positive integer greater than 1.

[0130] The electronic device 100 can implement the photographing function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor.

[0131] The ISP is configured to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also optimize the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be disposed in the camera 193.

[0132] The camera 193 is configured to capture still images or videos. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, the electronic device 100 can include one or N cameras 193, where N is a positive integer greater than 1.

[0133] The digital signal processor is used to process digital signals, in addition to being able to process digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0134] The video codec is used to compress or decompress digital video. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0135] The NPU is a neural-network (NN) calculation processor, which can quickly process input information by drawing on the structure of a biological neural network, such as drawing on the transmission mode between human brain neurons, and can also constantly self-learn. Through the NPU, the electronic device 100 can realize intelligent cognitive applications, such as: image recognition, face recognition, voice recognition, text understanding, etc.

[0136] The internal memory 121 can include one or more random access memories (RAMs) and one or more non-volatile memories (NVMs).

[0137] The random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, such as the fifth generation DDR SDRAM commonly referred to as DDR5 SDRAM), etc.

[0138] The non-volatile memory can include a magnetic disk storage device, a flash memory.

[0139] According to the operation principle, the flash memory can include NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. According to the potential order of the storage unit, the flash memory can include single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. According to the storage specification, the flash memory can include universal flash storage (UFS), embedded multi media Card (eMMC), etc.

[0140] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of an operating system or other programs running, and can also be used to store data of users and application programs, etc.

[0141] The non-volatile memory can also store executable programs and data of users and application programs, etc., and can be loaded into the random access memory in advance for direct reading and writing by the processor 110.

[0142] The external memory interface 120 can be used to connect an external non-volatile memory, so as to expand the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120, so as to realize the data storage function. For example, files such as music and video are saved in the external non-volatile memory.

[0143] The electronic device 100 can realize the audio function through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, etc. For example, music playing, recording, etc.

[0144] The audio module 170 is used to convert digital audio information into analog audio signals, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or part of the function modules of the audio module 170 can be arranged in the processor 110.

[0145] The speaker 170A, also known as a "loudspeaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0146] The receiver 170B, also called "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a phone call or a voice message, the user can answer the voice by holding the receiver 170B close to the ear.

[0147] The microphone 170C, also called "microphone", "sound collector", is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can make a sound by holding the microphone 170C close to the mouth, and input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In some other embodiments, the electronic device 100 can be provided with two microphones 170C, in addition to collecting sound signals, the noise reduction function can also be realized. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C, in addition to collecting sound signals, noise reduction, the sound source can also be identified, and the directional recording function can also be realized.

[0148] The earphone interface 170D is used to connect a wired earphone. The earphone interface 170D can be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0149] The pressure sensor 180A is used to sense pressure signals, and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A can be provided on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can include at least two parallel plates made of conductive material. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change of the capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, the touch operation acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than a first pressure threshold value acts on a short message application icon, an instruction of viewing a short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold value acts on the short message application icon, an instruction of creating a new short message is executed.

[0150] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake photography. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of shaking of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and lets the lens offset the shaking of the electronic device 100 by reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and motion sensing game scenarios.

[0151] The barometer sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude, assists positioning and navigation by the air pressure value measured by the barometer sensor 180C.

[0152] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can detect the opening and closing of a flip cover by using the magnetic sensor 180D. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. In turn, according to the detected opening and closing state of the cover or the opening and closing state of the flip cover, the electronic device 100 can set features such as automatic unlocking of the flip cover.

[0153] The acceleration sensor 180E can detect the magnitude of acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. The acceleration sensor 180E can also be used to identify the posture of the electronic device, and can be applied to landscape / portrait switching, pedometers, and other applications.

[0154] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance by infrared or laser. In some embodiments, in a shooting scenario, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0155] The proximity light sensor 180G can include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The light-emitting diode can be an infrared light-emitting diode. The electronic device 100 emits infrared light outwardly by the light-emitting diode. The electronic device 100 detects infrared reflected light from nearby objects using the photodiode. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that a user is holding the electronic device 100 close to the ear for a call, so as to automatically turn off the screen to achieve the purpose of power saving. The proximity light sensor 180G can also be used for automatic unlocking and locking of the cover mode and pocket mode.

[0156] Ambient light sensor 180L is used to sense ambient light brightness. Electronic device 100 can adaptively adjust display screen 194 brightness according to sensed ambient light brightness. Ambient light sensor 180L can also be used to automatically adjust white balance when taking a picture. Ambient light sensor 180L can also cooperate with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent accidental touch.

[0157] Fingerprint sensor 180H is used to collect a fingerprint. Electronic device 100 can use collected fingerprint characteristics to implement fingerprint unlocking, access application lock, take a picture with a fingerprint, answer an incoming call with a fingerprint, and so on.

[0158] Temperature sensor 180J is used to detect temperature. In some embodiments, electronic device 100 uses temperature detected by temperature sensor 180J to implement a temperature processing strategy. For example, when temperature reported by temperature sensor 180J exceeds a threshold value, electronic device 100 reduces performance of a processor located near temperature sensor 180J to reduce power consumption and implement thermal protection. In another embodiment, when temperature is lower than another threshold value, electronic device 100 heats battery 142 to avoid abnormal shutdown of electronic device 100 caused by low temperature. In other embodiments, when temperature is lower than yet another threshold value, electronic device 100 boosts output voltage of battery 142 to avoid abnormal shutdown caused by low temperature.

[0159] Touch sensor 180K, also referred to as a "touch device". Touch sensor 180K can be disposed on display screen 194, and touch sensor 180K and display screen 194 together form a touch screen, also referred to as a "touch panel". Touch sensor 180K is used to detect a touch operation acting on or near it. Touch sensor 180K can pass detected touch operation to an application processor to determine a touch event type. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K can also be disposed on a surface of electronic device 100, which is different from the position of display screen 194.

[0160] Bone conduction sensor 180M can obtain a vibration signal. In some embodiments, bone conduction sensor 180M can obtain a vibration signal of a human body sound part vibration bone block. Bone conduction sensor 180M can also contact a human body pulse to receive a blood pressure pulsation signal. In some embodiments, bone conduction sensor 180M can also be disposed in a headset to form a bone conduction headset. Audio module 170 can analyze a voice signal based on the vibration signal of the sound part vibration bone block obtained by bone conduction sensor 180M to implement a voice function. Application processor can analyze heart rate information based on the blood pressure pulsation signal obtained by bone conduction sensor 180M to implement a heart rate detection function.

[0161] The keys 190 include a power key, a volume key, and the like. The keys 190 can be mechanical keys. Alternatively, the keys 190 can be touch keys. The electronic device 100 can receive a key input, and generate a key signal input related to a user setting and a function control of the electronic device 100.

[0162] The motor 191 can generate a vibration prompt. The motor 191 can be used for a call vibration prompt, and can also be used for a touch vibration feedback. For example, a touch operation applied to different applications (e.g., a photograph, audio playback, and the like) can correspond to different vibration feedback effects. A touch operation applied to different regions of the display screen 194 can also correspond to different vibration feedback effects. Different application scenarios (e.g., a time reminder, a received message, an alarm, a game, and the like) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0163] The indicator 192 can be an indicator light, and can be used to indicate a charging state, a power change, and can also be used to indicate a message, a missed call, a notification, and the like.

[0164] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or pulled out of the SIM card interface 195 to realize contact and separation with the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, and N is a positive integer greater than 1. The SIM card interface 195 can support a Nano SIM card, a Micro SIM card, a SIM card, and the like. The same SIM card interface 195 can simultaneously insert multiple cards. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with an external storage card. The electronic device 100 interacts with a network through the SIM card to realize a call and data communication function, and the like. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0165] Figure 6 An exemplary structural diagram of the master device 200 is shown.

[0166] The structure of the slave device 300 is similar to that of the master device 200, and the embodiments of the present application will not be described here.

[0167] It should be noted that, Figure 6 The master device 200 shown is only an example, and the master device 200 can have more or fewer components than those shown in Figure 6 the master device 200 can be combined into two or more components, or can have a different component configuration. Figure 6The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0168] like Figure 6 As shown, the main device 200 may include a processor 601, a memory 602, a wireless communication module 603, a button 605, an antenna 604, an audio module 606, a speaker 607, an indicator 608, and a bus 609. The processor 601, memory 602, wireless communication module 603, button 605, audio module 606, and indicator 608 can be connected via the bus 609.

[0169] The processor 601 can be used to read and execute computer-readable instructions. Specifically, the processor 601 mainly includes a controller, an arithmetic logic unit (ALU), and registers. The controller is primarily responsible for instruction decoding and issuing control signals for the operations corresponding to the instructions. The ALU is primarily responsible for storing register operands and intermediate operation results temporarily stored during instruction execution. Specifically, the hardware architecture of the processor 601 can be an application-specific integrated circuit (ASIC) architecture, a MIPS architecture, an ARM architecture, or an NP architecture, etc.

[0170] The processor 601 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 601 is a cache memory. This memory can store instructions or data that the processor 601 has just used or that are used repeatedly. If the processor 601 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 601, and thus improves the efficiency of the system.

[0171] Memory 602 is coupled to processor 601 and is used to store various software programs and / or multiple sets of instructions. Memory 602 can be used to store computer executable program code, which includes instructions. Processor 601 executes various functional applications and data processing of host device 200 by running the instructions stored in memory 602. Memory 602 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function (such as sound playback function, image playback function, etc.). The data storage area may store data created during the use of host device 200 (such as audio frames). In addition, memory 602 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0172] The wireless communication module 603 can provide a solution for wireless communication applied to the host device 200, including wireless local area networks (WLAN) (such as a wireless fidelity (Wi-Fi) network), Bluetooth (BT), and the like.

[0173] In some embodiments, the wireless communication module 603 can include a Bluetooth (BT) communication module 603A, a WLAN communication module 603B. One or more of the Bluetooth (BT) communication module 603A, the WLAN communication module 603B can listen to signals emitted by other devices, such as probe requests, scan signals, and the like, and can send response signals, such as probe responses, scan responses, and the like, so that other devices can discover the host device 200 and establish a wireless communication connection with other devices, and communicate with other devices through one or more of Bluetooth or WLAN communication technologies. For example, a Bluetooth long connection is established with one or more Bluetooth devices through Bluetooth technology, and interaction is performed with the one or more Bluetooth devices. For another example, a networking function is implemented through WLAN technology to communicate with a server, and the like. The Bluetooth (BT) communication module 603A can provide a solution for one or more of classic Bluetooth and / or Bluetooth Low Energy communication. The WLAN communication module 603B can include a solution for one or more of Wi-Fi direct, Wi-Fi LAN, or Wi-Fi softAP communication.

[0174] The antenna 604 can be used to transmit and receive electromagnetic wave signals. The antennas of different communication modules can be multiplexed or independent of each other to improve the utilization rate of the antennas.

[0175] The key 605 can include a power-on key. The key 605 can be a mechanical key or a touch key. Taking the host device 200 as an example of a smart speaker, the key 605 can further include one or more of a volume key, a previous song switching key, a next song switching key, a play / pause key, and the like. The host device 200 can receive a key input and generate a key signal input related to the function control of the host device 200. When the host device 200 is playing audio data broadcast by an electronic device, the host device 200 can receive an input of a user for the play / pause key. In response to the input, the host device 200 can send a pause instruction to the electronic device through a Wi-Fi direct connection established with the electronic device, to trigger the electronic device to pause the broadcast of audio data to the host device 200.

[0176] In some embodiments, the host device 200 can further include a battery, a charging management module, and a power management module. The battery can be used to provide power for the host device 200. The charging management module is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module can receive charging input from a wired charger through a USB interface. In some wireless charging embodiments, the charging management module can receive wireless charging input through a wireless charging coil of the host device 200. The charging management module can supply power to the electronic device through the power management module while charging the battery. The power management module is used to connect the battery, the charging management module, and the processor 601. The power management module receives input from the battery and / or the charging management module to supply power to the processor 601, the memory 602, the wireless communication module 603, and the like. The power management module can also be used to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), and the like. In other embodiments, the power management module can also be disposed in the processor 601. In other embodiments, the power management module and the charging management module can also be disposed in the same device.

[0177] The audio module 606 can be used to convert digital audio information into analog audio signals and to convert analog audio input into digital audio signals. The audio module 606 can also be used to encode and decode audio signals. In some embodiments, the audio module 606 can be disposed in the processor 601, or some functional modules of the audio module 606 can be disposed in the processor 601.

[0178] The speaker 607 can be connected to the audio module 606. The speaker 607 can be used to convert audio electrical signals into sound signals.

[0179] Optionally, the host device 200 can further include a microphone (not shown in the figure), which can be used to convert sound signals into electrical signals. When a user controls the host device 200 through voice instructions, the user can speak close to the microphone to input sound signals to the microphone. Figure 6

[0180] The indicator 608 can be an indicator light, which can be used to indicate charging status, power change, and can also be used to indicate messages, notifications, and the like.

[0181] Figure 7 An exemplary functional module schematic diagram of the host device 200 is shown.

[0182] The host device 200 includes, but is not limited to, an audio stream receiving module, an audio stream unpacking module, a playback time difference calculation module, an audio time synchronization module, and an audio playing module, and the like.

[0183] ​The audio stream receiving module is configured to receive an audio stream sent by the electronic device 100.

[0184] Optionally, the audio stream receiving module is further configured to receive a start playing time of the audio data sent by the electronic device 100. In some embodiments, the start playing time can also be referred to as a reference time.

[0185] After receiving the audio stream sent by the electronic device 100, the audio stream receiving module is further configured to send the audio stream to an audio stream packetizing module.

[0186] The audio stream packetizing module is further configured to divide the audio stream into individual data packets after receiving the audio stream, and mark the data packets with a sequence number and a time stamp. For details, reference can be made to the description of subsequent embodiments, which will not be repeated here.

[0187] After obtaining the data packets, the audio stream packetizing module is further configured to send the data packets marked with the sequence number and the time stamp to a playing time difference calculating module.

[0188] In some embodiments, the time stamp of the data packet can be referred to as an expected playing time of the data packet, and the expected playing time of the data packet can be understood as an expected start playing time of the data packet.

[0189] The playing time difference calculating module is configured to adjust an expected playing time of a subsequent received audio stream based on a difference between a time stamp of a data packet and an actual playing time of the data packet. For details, reference can be made to the description of subsequent embodiments, which will not be repeated here.

[0190] The playing time difference calculating module is further configured to send the data packet with the adjusted expected playing time to an audio playing module.

[0191] The audio playing module is configured to play the audio based on the data packet with the adjusted expected playing time after receiving the data packet with the adjusted expected playing time sent by the playing time difference calculating module.

[0192] The audio stream packetizing module is further configured to send the data packet marked with the sequence number and the time stamp to an audio time synchronizing module.

[0193] The audio time synchronizing module is configured to periodically send the data packet marked with the sequence number and the time stamp to the slave device 300, for example, the audio time synchronizing module sends one data packet to the slave device 300 every 50 data packets. The data packet marked with the sequence number and the time stamp sent by the audio time synchronizing module to the slave device 300 can be referred to as a synchronization data packet.

[0194] It should be noted that the one or more function modules described above can be used alone to complete the preset function, or two or more of them can be combined to achieve the preset function.

[0195] Figure 8 An exemplary function module diagram of the slave device 300 is shown.

[0196] The slave device 300 includes, but is not limited to, an audio stream receiving module, an audio stream packetizing module, a play time difference calculating module, an audio time synchronizing module, and an audio playing module.

[0197] The master device 200 includes, but is not limited to, an audio stream receiving module, an audio stream packetizing module, a play time difference calculating module, an audio time synchronizing module, and an audio playing module.

[0198] The audio stream receiving module is configured to receive the audio stream sent by the electronic device 100.

[0199] Optionally, the audio stream receiving module is further configured to receive a start playing time of the audio data sent by the electronic device 100. In some embodiments, the start playing time can also be referred to as a reference time.

[0200] After the audio stream receiving module receives the audio stream sent by the electronic device 100, the audio stream receiving module is further configured to send the audio stream to the audio stream packetizing module.

[0201] The audio stream packetizing module is further configured to divide the audio stream into data packets after receiving the audio stream, and mark the data packets with a sequence number and a time stamp. For details, reference can be made to the description of subsequent embodiments, which will not be described herein again.

[0202] After obtaining the data packets, the audio stream packetizing module is further configured to send the data packets marked with the sequence number and the time stamp to the play time difference calculating module. Meanwhile, the audio stream packetizing module is further configured to send the data packets marked with the sequence number and the time stamp to the audio time synchronizing module.

[0203] The audio time synchronizing module is configured to receive the synchronization data packet sent by the master device 200.

[0204] The audio time synchronizing module is further configured to determine whether the expected playing time of the synchronization data packet and the local data is consistent based on the synchronization data packet and the local data. If yes, the master device 200 and the slave device 300 can achieve synchronized audio playing. If not, the audio stream packetizing module needs to adjust the expected playing time of the local data packet based on the synchronization data packet, determine a playing time delay, so that the expected playing times of the same audio data are consistent, and the master device 200 and the slave device 300 achieve synchronized audio playing.

[0205] The audio time alignment module is also configured to send the playback time delays of the master device 200 and the slave device 300 to the audio stream packetizing module.

[0206] The audio stream packetizing module is also configured to adjust the expected playback time of the local audio stream based on the playback time delays of the master device 200 and the slave device 300 after receiving the playback time delays sent by the audio time alignment module. The specific adjustment strategy can be referred to the description of subsequent embodiments, which will not be described here again in the embodiments of the present application.

[0207] The audio stream packetizing module is also configured to send the data packets to the playback time difference calculation module.

[0208] The playback time difference calculation module is configured to increase or delete the data packets based on the difference between the expected playback time of the data packets and the actual playback time of the data packets. The specific adjustment strategy can be referred to the description of subsequent embodiments, which will not be described here again in the embodiments of the present application.

[0209] The playback time difference calculation module is also configured to send the playback-adjusted data packets to the audio playback module.

[0210] The audio playback module is configured to play the audio after receiving the playback-adjusted data packets sent by the playback time difference calculation module.

[0211] It should be noted that the one or more functional modules described above can be used alone to complete the preset function, or two or more of them can be combined to complete the preset function.

[0212] First, after receiving the audio stream sent by the electronic device 100, the master device 200 and the slave device 300 need to packetize the audio stream and mark it with a sequence number and a timestamp.

[0213] The following embodiments of the present application take the master device 200 as an example to illustrate how to mark the audio stream with a sequence number and a timestamp. The description of how the slave device 300 marks the audio stream with a sequence number and a timestamp can be referred to the description of how the master device 200 marks the audio stream with a sequence number and a timestamp.

[0214] The electronic device 100 sends the audio stream to the master device 200 and the slave device 300 in real time. After receiving the audio stream, the master device 200 divides the audio stream into a data packet according to a preset number of bytes, for example, 7680 bytes per data packet, and marks the data packet with a sequence number according to the time sequence of the received audio stream, that is, the sequence number gradually increases by 1, for example, data packet 1, data packet 2, data packet 2,..., data packet n, and so on.

[0215] Figure 9AAn exemplary diagram showing how the master device 200 divides the data packets is shown.

[0216] An exemplary data packet includes 7680 bytes.

[0217] The master device 200 divides the audio stream into data packets in the time order of the received audio stream. For example, the first byte to the 7680th byte, a total of 7680 bytes, is divided into data packet 1. The 7681st byte to the 15360th byte, a total of 7680 bytes, is divided into data packet 2. The 15361st byte to the 23040th byte, a total of 7680 bytes, is divided into data packet 3. The 23041st byte to the 30720th byte, a total of 7680 bytes, is divided into data packet 4. The 30721st byte to the 38400th byte, a total of 7680 bytes, is divided into data packet 5. The 38401st byte to the 46080th byte, a total of 7680 bytes, is divided into data packet 6. In this way, the master device 200 divides the received audio stream into data packets in sequence.

[0218] Figure 9B An exemplary diagram showing how the master device 200 timestamps the data packets is shown.

[0219] The master device 200 timestamps the data packets to indicate the playing time of each data packet, which can be referred to as the expected playing time of the data packet.

[0220] The expected playing time of each data packet is equal to the sequence number of the data packet multiplied by the average playing time of each data packet. The average playing time of each data packet is preset, for example, 10 ms.

[0221] The master device 200 and the slave device 300 negotiate the time to start playing the audio before the master device 200 and the slave device 300 start playing the audio, which can be referred to as the reference time. After determining the reference time, the master device 200 sends the reference time to the slave device 300 so that the slave device 300 obtains the reference time.

[0222] The master device 200 can timestamp each data packet based on the reference time and the preset playing time of each data packet.

[0223] For example, assuming a base time of 10:00:00, and the playback time of the aforementioned 7680 bytes as a data packet is 10ms, each data packet is timestamped according to this base time and the preset playback time of each data packet. Here, 10:00:00 represents 10 minutes, 0 seconds, and 0 milliseconds. The base time can also be expressed as 10:10:00:00, where 10:10:00:00 represents 10 hours, 10 minutes, 0 seconds, and 0 milliseconds. The following embodiments of this application only show the minutes, seconds, and milliseconds of the base time; the hours are not shown.

[0224] like Figure 9B As shown, the start time of data packet 1 is 10:00:00, therefore the expected playback time of data packet 1 is 10:00:00. Since the playback time of data packet 1 is 10ms, the playback start and end time of data packet 1 is 10:00:00-10:00:10. The start time of data packet 2 is 10:00:10, therefore the expected playback time of data packet 1 is 10:00:10. Since the playback time of data packet 2 is 10ms, the playback start and end time of data packet 2 is 10:00:10-10:00:20. The start time of data packet 3 is 10:00:20, therefore the expected playback time of data packet 3 is 10:00:20. Since the playback time of data packet 3 is 10ms, the playback start and end time of data packet 3 is 10:00:20-10:00:30. Data packet 4 starts playing at 10:00:30, therefore its expected playback time is 10:00:30. Since its playback time is 10ms, the playback duration for data packet 4 is 10:00:30-10:00:40. Similarly, data packet 5 starts playing at 10:00:40, therefore its expected playback time is 10:00:40. Since its playback time is 10ms, the playback duration for data packet 5 is 10:00:40-10:00:50. Likewise, data packet 6 starts playing at 10:00:50, therefore its expected playback time is 10:00:50. Since its playback time is 10ms, the playback duration for data packet 6 is 10:00:50-10:00:60.

[0225] In this way, the master device 200 and the slave device 300 can timestamp the data packets to obtain the expected playback time of each data packet.

[0226] After the master device 200 and the slave device 300 start playing audio based on local data packets, the progress of audio playback by the master device 200 and the slave device 300 is different due to differences in the reception of data streams (such as data loss) and differences in data processing between the master device 200 and the slave device 300.

[0227] Therefore, the master device 200 and the slave device 300 need to dynamically negotiate to synchronize the audio playback.

[0228] The master device 200 periodically sends a data packet to the slave device 300, which is used to determine whether the master device 200 and the slave device 300 can achieve synchronized audio playback. For example, the master device 200 can send a data packet to the slave device 300 every 50 data packets.

[0229] Here, the data packet periodically sent by the master device 200 to the slave device 300 can be referred to as a synchronization data packet. The synchronization data packet includes the sequence number of the synchronization data packet, the timestamp of the synchronization data packet, and the audio data.

[0230] After receiving the synchronization data packet sent by the master device 200, the slave device 300 needs to match a synchronization data packet from the local data packet, and determine the playback delay between the slave device 300 and the master device 200 based on the sequence number of the synchronization data packet and the sequence number of the local data packet, and the offset of the synchronization data packet and the local data packet. So that the slave device 300 can adjust the expected playback time of the audio stream received thereafter, so that the expected playback time of the same audio content in the slave device 300 and the master device 200 is the same, to achieve synchronized audio playback of the master device 200 and the slave device 300.

[0231] Here, the synchronization data packet includes the sequence number of the data packet in the slave device 300, the timestamp of the data packet, and the audio data.

[0232] First, the slave device 300 needs to determine the data packet in the local data packet that matches the synchronization data packet. If the sequence numbers of the local data packet and the synchronization data packet are inconsistent and / or the offset is not 0, the slave device 300 needs to adjust the expected playback time of the audio stream received thereafter.

[0233] In some embodiments, if the progress of the slave device 300 playing the audio is the same as the progress of the master device 200 playing the audio, the sequence number of the synchronization data packet and the sequence number of the matching local data packet are the same, and the offset is 0.

[0234] In some embodiments, if the progress of the slave device 300 playing the audio is different from the progress of the master device 200 playing the audio, the slave device 300 needs to determine the audio segment that matches the audio data in the synchronization data packet, thereby determining the sequence number of the local data packet that matches the synchronization data packet. Based on the sequence number of the synchronization data packet, the sequence number of the local data packet, and the offset, the expected playback time of the audio stream received thereafter is adjusted.

[0235] Next, how the slave device 300 determines the audio segment matching the audio data in the synchronization data packet is introduced.

[0236] First, the slave device 300 obtains a waveform graph i based on the audio segment in the synchronization data packet i, and determines the position of the highest peak in the waveform graph i, where the position of the highest peak is the position where the amplitude of the audio segment is the largest.

[0237] Then, the slave device 300 obtains a plurality of matching peaks from the locally cached audio segment, and obtains the sequence number of the data packet where each matching peak is located.

[0238] In order to reduce the amount of calculation of the slave device 300, the slave device 300 can filter the plurality of matching peaks based on the maximum amplitude value corresponding to the position of the highest peak in the synchronization data packet i, for example, the first peak value, and retain the plurality of matching peaks where the peak value is greater than the second peak value. Among them, the second peak value is less than or equal to the first peak value, and the difference between the second peak value and the first peak value is greater than a preset value. In this way, the slave device 300 filters out the matching peaks where the peak value is less than the second peak value, so as to reduce the amount of calculation of the slave device 300.

[0239] Here, the plurality of matching peaks obtained after filtering are referred to as first matching peaks. The first matching peaks include a plurality of peak values and the sequence number of the data packet where each peak value is located.

[0240] After the slave device 300 determines the first matching peaks, the slave device 300 can determine the audio segment containing the first data length of each peak in the first matching peaks from the audio data cached in the slave device 300 based on the length of the audio data in the synchronization data packet i, for example, the first data length. If the first matching peaks contain M matching peaks, the slave device 300 needs to obtain M audio segments with the first data length. Then, the slave device 300 calculates the correlation between the audio segment in the synchronization data packet i and the M audio segments in the slave device 300 in sequence, and obtains an audio segment with the highest correlation. Then, the audio segment is the audio segment in the slave device 300 matching the synchronization data packet i. After determining the audio segment in the slave device 300 matching the synchronization data packet i, the slave device 300 determines the sequence number of the data packet in the slave device 300 matching the synchronization data packet i based on the sequence number of the data packet where the matching peak in the matching audio segment is located.

[0241] Next, how the slave device 300 determines the audio segment matching the synchronization data packet i is described in detail.

[0242] Figures 10-14 A schematic diagram of the slave device 300 determining the audio segment matching the synchronization data packet i is shown.

[0243] Suppose the length of the audio data in the synchronization data packet i contains 7680 bytes, as shown in FIG. 3, the position of the highest peak in the synchronization data packet i is at the 3000th byte. Then in the synchronization data packet i, there are 2999 bytes before the highest peak and 4680 bytes after the highest peak. Figure 10

[0244] The device 300 determines the first matching peak from the audio data in the local cache. The first matching peak includes a plurality of to-be-matched peaks, such as to-be-matched peak a, to-be-matched peak b, to-be-matched peak c, …, to-be-matched peak j, and the like.

[0245] Then, as shown in FIG. 3, the device 300 determines the byte position of the matching peak a based on the sequence number of the local data packet where the matching peak a is located and the position of the matching peak a in the local data packet. The device 300 takes 2999 bytes before the byte position of the matching peak a and 4680 bytes after the byte position of the matching peak a from the audio data in the local cache to obtain a to-be-matched audio segment a. Figure 11

[0246] The device 300 calculates the correlation between the audio segment in the synchronization data packet i and the to-be-matched audio segment a to obtain a correlation value a.

[0247] Optionally, the device 300 can calculate the correlation between the audio segment in the synchronization data packet i and the to-be-matched audio segment a based on the amplitude of the correlation, and the method for calculating the correlation includes but is not limited to the average value, the variance, and the like features to obtain the correlation value a.

[0248] Optionally, the device 300 can obtain the waveform graph of the audio segment in the synchronization data packet i and the waveform graph of the to-be-matched audio segment a respectively, calculate the similarity between the waveform graph of the audio segment in the synchronization data packet i and the waveform graph of the to-be-matched audio segment a, and obtain the correlation value a.

[0249] The device 300 can also calculate the correlation between the audio segment in the synchronization data packet i and the to-be-matched audio segment a based on other manners, and the embodiments of the present application do not limit this.

[0250] In this way, as shown in FIG. 3, the device 300 determines the first matching peak from the audio data in the local cache, and the first matching peak includes a plurality of to-be-matched peaks, such as to-be-matched peak a, to-be-matched peak b, to-be-matched peak c, …, to-be-matched peak j, and the like. Figure 11 ​​As shown, the slave device 300 can calculate the byte position of the matching peak b based on the sequence number of the local data packet containing the matching peak b and the position of the matching peak b within that local data packet. From the audio data locally cached by the slave device 300, 2999 bytes before the byte position of the matching peak b and 4680 bytes after the byte position of the matching peak b are retrieved. This yields the audio segment b to be matched.

[0251] The correlation value b is obtained by calculating the correlation between the audio segment in the synchronization data packet i and the audio segment b to be matched from the device 300.

[0252] like Figure 11 As shown, the slave device 300 can calculate the byte position of the matching peak c based on the sequence number of the local data packet containing the matching peak c and the position of the matching peak c within that local data packet. From the audio data locally cached by the slave device 300, 2999 bytes before the byte position of the matching peak c and 4680 bytes after the byte position of the matching peak c are retrieved. This yields the audio segment c to be matched.

[0253] The correlation between the audio segment in the synchronization data packet i and the audio segment c to be matched is calculated from device 300, and the correlation value c is obtained.

[0254] And so on, such as Figure 11 As shown, the slave device 300 can calculate the byte position of the matching peak j based on the sequence number of the local data packet containing the matching peak j and the position of the matching peak j within that local data packet. From the audio data locally cached by the slave device 300, 2999 bytes before the byte position of the matching peak j and 4680 bytes after the byte position of the matching peak j are retrieved. This yields the audio segment j to be matched.

[0255] The correlation between the audio segment in the synchronization data packet i and the audio segment j to be matched is calculated from device 300, and the correlation value j is obtained.

[0256] After obtaining the correlation between the synchronization data packet i and multiple audio segments to be matched from device 300, the correlation values ​​are compared. The audio segment with the highest correlation value is the audio segment that matches the synchronization data packet i. For example, if the correlation value j is the largest, then device 300 can determine that the audio segment j is the audio segment that matches the synchronization data packet i.

[0257] After the slave device 300 determines the audio segment that matches the synchronization data packet i, the slave device 300 also needs to determine the sequence number of the local data packet that matches the synchronization data packet i based on the sequence number of the local data packet where the matching peak j is located. For example, the local data packet that matches the synchronization data packet i is local data packet j. Local data packet j and synchronization data packet i can be a complete match or a partial match. Partial match includes the following two cases: Case 1: The first part of the audio content of local data packet j is the same as the second part of the audio content of synchronization data packet i. Case 2: The second part of the audio content of local data packet j is the same as the first part of the audio content of synchronization data packet i.

[0258] Furthermore, the slave device 300 can determine the offset between the audio data contained in the local data packet j and the audio data contained in the synchronization data packet i based on the local data packet j.

[0259] For example, local data packet j includes bytes from the (7680*(j-1)+1)th byte to the 7680*jth byte, for a total of 7680 bytes.

[0260] Match the position of peak j in a byte position between (7680*(j-1)+1) bytes and 7680*j bytes.

[0261] like Figure 12 As shown, if the matching peak j is located at the (7680*(j-1)+3000)th byte position, then there are 2999 bytes between the (7680*(j-1)+1)th byte and the (7680*(j-1)+2999)th byte. There are 4680 bytes between the (7680*(j-1)+3001)th byte and the (7680*j)th byte. At this point, the offset between the audio data contained in the local data packet j and the audio data contained in the synchronization data packet i is exactly 0.

[0262] like Figure 13As shown, if the matching peak j is located at any byte position between the (7680*(j-1)+1)th byte and the (7680*(j-1)+2999)th byte, then the offset is less than 0. For example, if the matching peak j is located at the (7680*(j-1)+2900)th byte, then there are 2899 bytes between the (7680*(j-1)+1)th byte and the (7680*(j-1)+2899)th byte, which is less than 2999 bytes. Therefore, the slave device 300 needs to extract 100 bytes of audio data from the latter half of the (j-1)th local data packet and place it in the audio segment j to be matched. The interval between byte (7680*(j-1)+2901) and byte (7680*j) is 4790 bytes, which is greater than 4680 bytes. Therefore, device 300 needs to extract bytes (7680*(j-1)+2901) to (7680*(j-1)+7590) from the j-th local data packet and place them in the audio segment j to be matched. Therefore, the offset between the audio data contained in local data packet j and the audio data contained in synchronization data packet i is 100.

[0263] like Figure 14 As shown, if the matching peak j is located at any byte position between the (7680*(j-1)+3001)th byte and the (7680*j)th byte, then the offset is greater than 0. For example, if the matching peak j is located at the (7680*(j-1)+4000)th byte, then there are 3999 bytes between the (7680*(j-1)+1)th byte and the (7680*(j-1)+3999)th byte, which is greater than 2999 bytes. Therefore, the slave device 300 needs to take 2999 bytes backward from the (7680*(j-1)+4000)th byte, that is, from the (7680*(j-1)+1002)th byte to the (7680*(j-1)+4000)th byte, a total of 2999 bytes, and place them in the audio segment j to be matched. The interval between the (7680*(j-1)+4001)th byte and the (7680*j)th byte is 3690 bytes, which is less than 4680 bytes. Therefore, device 300 needs to extract the (7680*j+1)th byte to the (7680*j+1000)th byte, a total of 1000 bytes, from the (j+1)th local data packet and place it in the audio segment j to be matched. Therefore, the offset between the audio data contained in local data packet j and the audio data contained in synchronization data packet i is -1000.

[0264] After the slave device 300 determines the local data packet j matching the synchronization data packet i and the offset between the synchronization data packet i and the local data packet j, the slave device 300 needs to adjust the expected play time of the audio stream received later so that the expected play time of the same audio segment in the master device 200 and the slave device 300 is consistent.

[0265] First, how the slave device 300 adjusts the expected play time of the audio stream received later when the offset between the synchronization data packet i and the local data packet j is 0 is introduced.

[0266] When the offset is 0, it means that the audio data in the synchronization data packet i and the audio data in the local data packet j are consistent, i.e., completely matched.

[0267] If i < j, it means that the progress of playing the audio in the slave device 300 is later than that in the master device 200, and the time difference is approximately the play time of each data packet*(j-i). Therefore, when the slave device 300 labels the audio stream received later with the sequence number and the timestamp, the sequence number of each data packet is added by the value of (i-j), and the timestamp of each data packet is updated based on the new sequence number. In this way, the timestamp of each data packet of the audio stream received later is advanced by the play time of each data packet*(j-i).

[0268] The slave device 300 can further update the timestamp of each data packet by the following formula (1).

[0269] timestamp = syncTimestamp + (index - syncIndex)*PERIOD_TIME Formula (1)

[0270] As shown in formula (1), timestamp is the updated timestamp of each data packet, syncTimestamp is the reference time, index is the sequence number before the update of each data packet, syncIndex is the difference between the sequence number j and the sequence number i, for example, syncIndex is equal to 4, and PERIOD_TIME is the play time of each data packet, for example, 10 ms. Based on formula (1), the slave device 300 can update the timestamp of each data packet of the audio stream received later, i.e., the timestamp of each data packet of the audio stream received later is advanced by the play time of each data packet*(j-i).

[0271] For example, assume that after receiving the synchronization packet i sent by the master device 200, the slave device 300 confirms that the local packet matching the synchronization packet i is the local packet j, the synchronization packet i completely matches the local packet j, the offset of the synchronization packet i and the local packet j is 0, and i < j, then the slave device 300 needs to adjust the sequence number and the time stamp of the audio stream received thereafter.

[0272] For example, assume that i = 3 and j = 7, and the average playing time of each packet is 10 ms, then it indicates that the progress of playing the audio by the slave device 300 is later than the progress of playing the audio by the master device 200, and the time is about 40 ms. Therefore, the slave device 300 needs to advance the expected playing time of the audio stream received thereafter by 40 ms.

[0273] Figure 15A A diagram showing the sequence number and the time stamp of the packet of the audio stream one played by the slave device 300 before the adjustment of the sequence number and the time stamp of the local packet based on the synchronization packet i.

[0274] The packet 1 includes the 1st byte to the 7680th byte, and the expected playing time of the packet 1 is 10:00:00 to 10:00:10. The packet 2 includes the 7681st byte to the 15360th byte, and the expected playing time of the packet 2 is 10:00:10 to 10:00:20. The packet 3 includes the 15361st byte to the 23040th byte, and the expected playing time of the packet 3 is 10:00:20 to 10:00:30. The packet 4 includes the 23041st byte to the 30260th byte, and the expected playing time of the packet 4 is 10:00:30 to 10:00:40. The packet 5 includes the 30261st byte to the 38400th byte, and the expected playing time of the packet 5 is 10:00:40 to 10:00:50. The packet 6 includes the 38401st byte to the 46080th byte, and the expected playing time of the packet 6 is 10:00:50 to 10:00:60. The packet 7 includes the 46081st byte to the 53760th byte, and the expected playing time of the packet 7 is 10:00:60 to 10:00:70.

[0275] Figure 15B A diagram showing the sequence number and the time stamp of the packet of the audio stream two played by the slave device 300 before the adjustment of the sequence number and the time stamp of the local packet based on the synchronization packet i.

[0276] The audio stream two is the audio stream after the audio stream one.

[0277] Data packet 8 includes the 53761st byte to the 61440th byte, and the expected playing time of data packet 8 is 10:00:70 to 10:00:80. Data packet 9 includes the 61441st byte to the 69120th byte, and the expected playing time of data packet 9 is 10:00:80 to 10:00:90. Data packet 10 includes the 69121st byte to the 76800th byte, and the expected playing time of data packet 10 is 10:00:90 to 10:00:100. Data packet 11 includes the 76801st byte to the 84480th byte, and the expected playing time of data packet 11 is 10:00:100 to 10:00:110. Data packet 12 includes the 84481st byte to the 92160th byte, and the expected playing time of data packet 12 is 10:00:110 to 10:00:120.

[0278] The data length of the audio stream two can be the same as or different from the data length of the audio stream one, and the embodiments of the present application do not limit this.

[0279] Optionally, Figure 15B The schematic diagram shown can be understood as the expected playing time of the audio stream two before being updated from the slave device 300.

[0280] Figure 15C The schematic diagram of the master device 200 marking the data packets of the audio stream two with the sequence number and the time stamp is shown.

[0281] Data packet 4 includes the 53761st byte to the 61440th byte, and the expected playing time of data packet 4 is 10:00:30 to 10:00:40. Data packet 5 includes the 61441st byte to the 69210th byte, and the expected playing time of data packet 5 is 10:00:40 to 10:00:50. Data packet 6 includes the 69211st byte to the 76800th byte, and the expected playing time of data packet 6 is 10:00:50 to 10:00:60. Data packet 7 includes the 76801st byte to the 84480th byte, and the expected playing time of data packet 7 is 10:00:60 to 10:00:70. Data packet 8 includes the 84481st byte to the 92160th byte, and the expected playing time of data packet 8 is 10:00:70 to 10:00:80.

[0282] From Figure 15B And Figure 15C It can be seen that the audio segment in data packet 4 in the master device 200 and data packet 8 in the slave device 300 is the same, but the expected playing time of the audio segment in the slave device 300 is 40 ms later than the expected playing time of the audio segment in the master device 200.

[0283] The audio segment in the data packet 5 in the master device 200 and the data packet 9 in the slave device 300 is the same, but the expected playing time of the audio segment in the slave device 300 is 40 ms later than the expected playing time of the audio segment in the master device 200.

[0284] The audio segment in the data packet 6 in the master device 200 and the data packet 10 in the slave device 300 is the same, but the expected playing time of the audio segment in the slave device 300 is 40 ms later than the expected playing time of the audio segment in the master device 200.

[0285] The audio segment in the data packet 7 in the master device 200 and the data packet 11 in the slave device 300 is the same, but the expected playing time of the audio segment in the slave device 300 is 40 ms later than the expected playing time of the audio segment in the master device 200.

[0286] The audio segment in the data packet 8 in the master device 200 and the data packet 12 in the slave device 300 is the same, but the expected playing time of the audio segment in the slave device 300 is 40 ms later than the expected playing time of the audio segment in the master device 200.

[0287] Therefore, the expected playing time of the audio stream received after the slave device 300 needs to be adjusted, so that the expected playing time of the audio stream received after the slave device 300 is 40 ms earlier.

[0288] Specifically, after receiving the second audio stream, the slave device 300 adjusts the sequence number of the data packet when dividing the data packet, that is, subtracts the value of (j-i) from the original sequence number of the data packet. And update the timestamp of the data packet based on formula (1), which is the updated expected playing time.

[0289] Figure 15D The figure shows the sequence number and timestamp of the data packet of the second audio stream based on the difference between the sequence number i and the sequence number j after adjusting the sequence number and timestamp of the local data packet based on the synchronization data packet i.

[0290] For example, the sequence number i is 3, the sequence number j is 7, and the offset is 0. Then the slave device 300 needs to modify the sequence number of the original data packet 8 to the sequence number 4 after receiving the second audio stream, and the corresponding expected playing time of the original data packet 8 is 40 ms earlier. The sequence number of the original data packet 9 is modified to the sequence number 5, and the corresponding expected playing time of the original data packet 9 is 40 ms earlier.

[0291] As shown in Figure 15D The sequence number of the data packet of the second audio stream is updated based on the difference between the sequence number i and the sequence number j, and the expected playing time of the data packet is updated based on the updated sequence number.

[0292] The data packet 4 includes the 53761st byte to the 61440th byte, and the expected playing time of the data packet 5 is 10:00:30 to 10:00:40. The data packet 5 includes the 61441st byte to the 69210th byte, and the expected playing time of the data packet 5 is 10:00:40 to 10:00:50. The data packet 6 includes the 69211th byte to the 76800th byte, and the expected playing time of the data packet 6 is 10:00:50 to 10:00:60. The data packet 7 includes the 76801st byte to the 84480th byte, and the expected playing time of the data packet 7 is 10:00:60 to 10:00:70. The data packet 8 includes the 84481st byte to the 92160th byte, and the expected playing time of the data packet 8 is 10:00:70 to 10:00:80.

[0293] Optionally, Figure 15D The schematic diagram shown can be understood as the expected playing time of the audio stream two after being updated from the device 300.

[0294] By analogy, the expected playing time of the audio stream after the audio stream one is all advanced by 40 ms.

[0295] From Figure 15D and Figure 15B It can be seen that the audio segment in the data packet 8 before adjustment and the audio segment in the data packet 4 after adjustment are the same, but the expected playing time of the audio segment after adjustment is 40 ms earlier than the expected playing time before adjustment.

[0296] The audio segment in the data packet 9 before adjustment and the audio segment in the data packet 5 after adjustment are the same, but the expected playing time of the audio segment after adjustment is 40 ms earlier than the expected playing time before adjustment.

[0297] The audio segment in the data packet 10 before adjustment and the audio segment in the data packet 6 after adjustment are the same, but the expected playing time of the audio segment after adjustment is 40 ms earlier than the expected playing time before adjustment.

[0298] The audio segment in the data packet 11 before adjustment and the audio segment in the data packet 7 after adjustment are the same, but the expected playing time of the audio segment after adjustment is 40 ms earlier than the expected playing time before adjustment.

[0299] The audio segment in the data packet 12 before adjustment and the audio segment in the data packet 8 after adjustment are the same, but the expected playing time of the audio segment after adjustment is 40 ms earlier than the expected playing time before adjustment.

[0300] From Figure 15D and Figure 15CThe difference between the two can be seen that after the adjustment, the audio segment in the packet 4 in the master device 200 and the audio segment in the packet 4 in the slave device 300 are the same, and the expected playing time of the audio segment in the slave device 300 is also the same as the expected playing time of the audio segment in the master device 200.

[0301] The audio segment in the packet 5 in the master device 200 and the audio segment in the packet 5 in the slave device 300 are the same, and the expected playing time of the audio segment in the slave device 300 is also the same as the expected playing time of the audio segment in the master device 200.

[0302] The audio segment in the packet 6 in the master device 200 and the audio segment in the packet 6 in the slave device 300 are the same, and the expected playing time of the audio segment in the slave device 300 is also the same as the expected playing time of the audio segment in the master device 200.

[0303] The audio segment in the packet 7 in the master device 200 and the audio segment in the packet 7 in the slave device 300 are the same, and the expected playing time of the audio segment in the slave device 300 is also the same as the expected playing time of the audio segment in the master device 200.

[0304] The audio segment in the packet 8 in the master device 200 and the audio segment in the packet 8 in the slave device 300 are the same, and the expected playing time of the audio segment in the slave device 300 is also the same as the expected playing time of the audio segment in the master device 200.

[0305] Thus, after the slave device 300 adjusts the expected playing time of the audio stream two, the expected playing time of the audio stream two in the slave device 300 and the expected playing time of the audio stream two in the master device 200 are the same, and the slave device 300 and the slave device 300 can realize playing the audio stream two at the same time.

[0306] It should be noted that the slave device 300 can adjust the expected playing time of the audio stream after the audio stream two in the above manner.

[0307] In the case of the offset being 0, it is indicated that the audio data in the synchronization packet i and the audio data in the local packet j are consistent, i.e., completely matched.

[0308] If i>j, it is indicated that the progress of playing the audio in the slave device 300 is faster than the progress of playing the audio in the master device 200, and the fast time is about the playing duration of each packet*(i-j). Therefore, when the slave device 300 marks the sequence number and the timestamp of the audio stream received later, the sequence number of the packet is added by the value of (i-j), and the timestamp of the packet is updated based on the new sequence number of the packet. In this way, the timestamp of each packet divided by the audio stream received later will be delayed by the playing duration of each packet*(i-j).

[0309] The timestamp of the data packet can be further updated from device 300 using the following formula (2).

[0310] timestamp=syncTimestamp+(index+syncIndex)*PERIOD_TIME formula (2)

[0311] As shown in formula (2), timestamp is the timestamp after the data packet is updated, syncTimestamp is the base time, index is the sequence number before the data packet is updated, syncIndex is the difference between sequence number j and sequence number i, for example, syncIndex equals 4, and PERIOD_TIME is the playback duration of each data packet, for example, 10ms. Based on formula (2), the slave device 300 can update the timestamp of each data packet of the audio stream received later, that is, the timestamp of each data packet of the audio stream received later will be delayed by the playback duration of each data packet * (ij).

[0312] For example, suppose that after receiving a synchronization data packet i from the master device 200, the slave device 300 determines that the local data packet that matches the synchronization data packet i is the local data packet j. The synchronization data packet i and the local data packet j are completely matched, the offset between the synchronization data packet i and the local data packet j is 0, and i>j. Then the slave device 300 needs to adjust the sequence number and timestamp of the data packets in the audio stream received later.

[0313] For example, assuming i=7, j=3, and the average playback time of each data packet is 10ms, this means that the slave device 300 plays audio faster than the master device 200, by approximately 40ms. Therefore, the slave device 300 needs to delay the expected playback time of subsequently received audio streams by 40ms. Figure 16A This diagram illustrates how a slave device 300 adds sequence numbers and timestamps to data packets of audio stream 1 before adjusting the sequence numbers and timestamps of local data packets based on synchronization data packet i.

[0314] right Figure 16A For more information, please refer to the following: Figure 15A The descriptions in the embodiments of this application will not be repeated here.

[0315] Figure 16B This diagram illustrates how the slave device 300 adds sequence numbers and timestamps to the data packets of audio stream 2 before adjusting the sequence numbers and timestamps of local data packets based on synchronization data packet i.

[0316] Audio stream 2 is the audio stream that follows audio stream 1.

[0317] rightFigure 16B For more information, please refer to the following: Figure 15B The descriptions in the embodiments of this application will not be repeated here.

[0318] The data length of audio stream two can be the same as or different from that of audio stream one; this application does not limit this in its embodiments.

[0319] Optional, Figure 16B The diagram shown can be understood as the expected playback time of audio stream 2 before it is updated from device 300.

[0320] Figure 16C The diagram shows a master device 200 adding sequence numbers and timestamps to the data packets of audio stream two.

[0321] Data packet 12 includes bytes 53761 to 61440, with an expected playback time of 10:00:110 to 10:00:120. Data packet 13 includes bytes 61441 to 69210, with an expected playback time of 10:00:120 to 10:00:130. Data packet 14 includes bytes 69211 to 76800, with an expected playback time of 10:00:130 to 10:00:60. Data packet 15 includes bytes 76801 to 84480, with an expected playback time of 10:00:140 to 10:00:150. Data packet 16 includes bytes 84481 to 92160, and the expected playback time of data packet 16 is 10:00:150 to 10:00:160.

[0322] from Figure 16B and Figure 16C It can be seen that the audio segments in data packet 12 of master device 200 and data packet 8 of slave device 300 are the same, but the expected playback time of the audio segment in slave device 300 is 40ms faster than the expected playback time of the audio segment in master device 200.

[0323] The audio segments in data packet 13 of master device 200 and data packet 9 of slave device 300 are the same, but the expected playback time of the audio segment in slave device 300 is 40ms faster than the expected playback time of the audio segment in master device 200.

[0324] The audio segments in data packet 14 of master device 200 and data packet 10 of slave device 300 are the same, but the expected playback time of the audio segment in slave device 300 is 40ms faster than the expected playback time of the audio segment in master device 200.

[0325] The audio segment in packet 15 in master device 200 and packet 11 in slave device 300 is the same, but the expected play time of the audio segment in slave device 300 is 40 ms faster than the expected play time of the audio segment in master device 200.

[0326] The audio segment in packet 16 in master device 200 and packet 12 in slave device 300 is the same, but the expected play time of the audio segment in slave device 300 is 40 ms faster than the expected play time of the audio segment in master device 200.

[0327] Therefore, the expected play time of the audio stream received after slave device 300 needs to be adjusted so that the expected play time of the audio stream received after slave device 300 is delayed by 40 ms.

[0328] Specifically, slave device 300, after receiving audio stream two, when dividing into packets, will update the sequence number of the packets, i.e. add the value of (i-j) to the original sequence number of the packets. Slave device 300 will then timestamp the packets based on the updated sequence number of the packets, and the timestamp is the updated expected play time.

[0329] Figure 16D A diagram showing that after adjusting the sequence number and timestamp of the local packets based on the synchronization packet i, slave device 300 timestamps the packets of audio stream two based on the difference between sequence number i and sequence number j.

[0330] For example, sequence number i is 7, sequence number j is 3, and the offset is 0. Then slave device 300, after receiving audio stream two, needs to modify the sequence number of original packet 8 to sequence number 12, and correspondingly, the expected play time of original packet 8 is delayed by 40 ms. The sequence number of original packet 9 is modified to sequence number 13, and correspondingly, the expected play time of original packet 9 is delayed by 40 ms.

[0331] As shown in Figure 16D slave device 300 updates the sequence number of the packets of audio stream two based on the difference between sequence number i and sequence number j, and updates the expected play time of the packets based on the updated sequence number.

[0332] The data packet 12 includes the 53761st byte to the 61440th byte, and the expected playing time of the data packet 12 is 10:00:100 to 10:00:120. The data packet 13 includes the 61441st byte to the 69210th byte, and the expected playing time of the data packet 13 is 10:00:120 to 10:00:130. The data packet 14 includes the 69211th byte to the 76800th byte, and the expected playing time of the data packet 14 is 10:00:130 to 10:00:140. The data packet 15 includes the 76801st byte to the 84480th byte, and the expected playing time of the data packet 15 is 10:00:140 to 10:00:150. The data packet 16 includes the 84481st byte to the 92160th byte, and the expected playing time of the data packet 16 is 10:00:150 to 10:00:160.

[0333] Optionally, Figure 16D The schematic diagram shown can be understood as the expected playing time of the audio stream two after the update from the device 300.

[0334] By analogy, the expected playing time of the audio stream after the audio stream one from the device 300 is delayed by 40 ms.

[0335] From Figure 16D and Figure 16B It can be seen from the difference that the audio segment in the data packet 8 before the adjustment and the audio segment in the data packet 12 after the adjustment are the same, but the expected playing time of the audio segment after the adjustment is delayed by 40 ms than the expected playing time before the adjustment.

[0336] The audio segment in the data packet 9 before the adjustment and the audio segment in the data packet 13 after the adjustment are the same, but the expected playing time of the audio segment after the adjustment is delayed by 40 ms than the expected playing time before the adjustment.

[0337] The audio segment in the data packet 10 before the adjustment and the audio segment in the data packet 14 after the adjustment are the same, but the expected playing time of the audio segment after the adjustment is delayed by 40 ms than the expected playing time before the adjustment.

[0338] The audio segment in the data packet 11 before the adjustment and the audio segment in the data packet 15 after the adjustment are the same, but the expected playing time of the audio segment after the adjustment is delayed by 40 ms than the expected playing time before the adjustment.

[0339] The audio segment in the data packet 12 before the adjustment and the audio segment in the data packet 16 after the adjustment are the same, but the expected playing time of the audio segment after the adjustment is delayed by 40 ms than the expected playing time before the adjustment.

[0340] From Figure 16D and Figure 16CThe difference between them can be seen that after the adjustment, the audio segment in the packet 4 in the master device 200 and the audio segment in the packet 12 in the slave device 300 are the same, and the expected playing time of the audio segment in the slave device 300 is also the same as the expected playing time of the audio segment in the master device 200.

[0341] The audio segment in the packet 13 in the master device 200 and the audio segment in the packet 13 in the slave device 300 are the same, and the expected playing time of the audio segment in the slave device 300 is also the same as the expected playing time of the audio segment in the master device 200.

[0342] The audio segment in the packet 14 in the master device 200 and the audio segment in the packet 14 in the slave device 300 are the same, and the expected playing time of the audio segment in the slave device 300 is also the same as the expected playing time of the audio segment in the master device 200.

[0343] The audio segment in the packet 15 in the master device 200 and the audio segment in the packet 15 in the slave device 300 are the same, and the expected playing time of the audio segment in the slave device 300 is also the same as the expected playing time of the audio segment in the master device 200.

[0344] The audio segment in the packet 16 in the master device 200 and the audio segment in the packet 16 in the slave device 300 are the same, and the expected playing time of the audio segment in the slave device 300 is also the same as the expected playing time of the audio segment in the master device 200.

[0345] Thus, after the slave device 300 adjusts the expected playing time of the audio stream two, the expected playing time of the audio stream two in the slave device 300 is the same as the expected playing time of the audio stream two in the master device 200, and the slave device 300 and the slave device 300 can play the audio stream two at the same time.

[0346] It should be noted that the slave device 300 can adjust the expected playing time of the audio stream after the audio stream two in the above manner.

[0347] In the case where the offset is not 0, it means that the audio data in the synchronization packet i and the audio data in the local packet j are not completely matched, but only partially matched.

[0348] If the offset is greater than 0, as shown in the foregoing Figure 13 embodiment, the latter part of the audio segment in the local packet j-1 and the former part of the audio segment in the local packet j are matched with the audio segment in the synchronization packet i. The latter part of the audio segment in the local packet j is matched with the former part of the audio segment in the synchronization packet i+1. The data length of the latter part of the audio segment in the local packet j-1 can be referred to as the offset offset between the local packet j and the synchronization packet i.

[0349] Then, based on the offset between the local data packet j and the synchronization data packet i, the audio stream received later can be added with null bytes with a length of offset in front of the audio stream. In this way, the audio stream received later from the slave device 300 is moved backward by offset bytes, and the offset between the local data packet j+m and the synchronization data packet i+m is 0.

[0350] In the case that the offset is greater than 0, the size between the sequence number j of the local data packet j and the sequence number i of the synchronization data packet i also needs to be considered.

[0351] If the sequence number j is equal to the sequence number i, the slave device 300 can packet the audio stream added with the null bytes with a length of offset, and stamp the data packet with the sequence number and the time stamp.

[0352] Figures 17A-17C A schematic diagram of the slave device 300 adjusting the audio stream two is shown.

[0353] Figure 17A A schematic diagram of the slave device 300 stamping the received audio stream two with the time stamp and the sequence number in the case that the sequence number j is equal to the sequence number i is shown.

[0354] The introduction of Figure 17A can refer to the introduction of Figure 15B in the embodiments, and details are not described herein again.

[0355] It is assumed that the offset is 300 bytes, Figure 17B A schematic diagram of the master device 200 stamping the received audio stream two with the time stamp and the sequence number is shown.

[0356] As shown in Figure 17B , the expected playing time of the data packet 8 in the master device 200 is the same as that of the data packet 8 in the slave device 300, and the data packet 8 in the master device 200 includes the 53431th byte to the 61140th byte. It can be seen that the playing progress of the data packet 8 in the master device 200 is 300 bytes slower than that of the data packet 8 in the slave device 300.

[0357] The expected playing time of the data packet 9 in the master device 200 is the same as that of the data packet 9 in the slave device 300, and the data packet 9 in the master device 200 includes the 61141th byte to the 68820th byte. It can be seen that the playing progress of the data packet 9 in the master device 200 is 300 bytes slower than that of the data packet 9 in the slave device 300.

[0358] The expected playback time of data packet 10 in master device 200 is the same as that of data packet 10 in slave device 300. Data packet 10 in master device 200 includes bytes 68821 to 76500. It can be seen that the playback progress of data packet 10 in master device 200 is 300 bytes slower than that of data packet 10 in slave device 300.

[0359] The expected playback time of data packet 11 in master device 200 and data packet 11 in slave device 300 is the same. Data packet 11 in master device 200 includes bytes 76501 to 84180. It can be seen that the playback progress of data packet 11 in master device 200 is 300 bytes slower than the playback progress of data packet 11 in slave device 300.

[0360] The expected playback time of data packet 12 in master device 200 is the same as that of data packet 12 in slave device 300. Data packet 12 in master device 200 includes bytes 84181 to 91860. It can be seen that the playback progress of data packet 12 in master device 200 is 300 bytes slower than that of data packet 12 in slave device 300.

[0361] Figure 17C An exemplary diagram is shown in which device 300 adds 300 null bytes to the beginning of audio stream 2 and then adds a timestamp and serial number to audio stream 2.

[0362] After adding 300 null bytes to the beginning of audio stream 2 from device 300, the playback progress of each data packet is delayed by 300 bytes.

[0363] like Figure 17C As shown, data packet 8 in slave device 300 includes 300 empty bytes and bytes 53761 through 61140. Data packet 8 in master device 200 includes bytes 53431 through 61140.

[0364] Data packet 9 in slave device 300 includes bytes 61141 to 69120. Data packet 9 in master device 200 includes bytes 61141 to 68820. The playback progress of data packet 9 in slave device 300 and data packet 9 in master device 200 is the same.

[0365] Data packet 10 in slave device 300 includes bytes 69121 to 76800. Data packet 10 in master device 200 includes bytes 69121 to 76800. The playback progress of data packet 10 in slave device 300 is the same as that in master device 200.

[0366] The data packet 11 in the slave device 300 includes the 76801st byte to the 84480th byte. The data packet 11 in the master device 200 includes the 76801st byte to the 84480th byte. The playing progress of the data packet 11 in the slave device 300 is the same as that of the data packet 11 in the master device 200.

[0367] The data packet 12 in the slave device 300 includes the 84481st byte to the 92160th byte. The data packet 11 in the master device 200 includes the 84481st byte to the 92160th byte. The playing progress of the data packet 12 in the slave device 300 is the same as that of the data packet 12 in the master device 200.

[0368] If i is greater than j, the slave device 300 adds the offset data length of null bytes to the front of the received audio stream, which indicates that the playing progress of the slave device 300 is faster than that of the master device 200, and the time difference is approximately the playing duration of each data packet*(i-j). Therefore, when the slave device 300 labels the sequence number and the timestamp of the audio stream received later, the sequence number of each data packet is added by the value of (i-j), and the timestamp of each data packet is updated based on the new sequence number. In this way, the timestamp of each data packet of the audio stream received later is delayed by the playing duration of each data packet*(i-j).

[0369] The slave device 300 can further update the timestamp of each data packet based on the above formula (2). The embodiment of the present application will not be described here. Based on the formula (2), when the slave device 300 labels the sequence number and the timestamp of the audio stream after adding the offset data length of null bytes, the sequence number of each data packet is added by the value of (i-j), and in this way, the timestamp of each data packet of the audio stream after adding the offset data length of null bytes is delayed by the playing duration of each data packet*(i-j).

[0370] Figures 18A-18C Another set of diagrams for adjusting the audio stream two by the slave device 300 is shown.

[0371] For example, j is equal to 3, and i is equal to 7.

[0372] Figure 18A A diagram for labeling the sequence number and the timestamp of the data packet of the audio stream two by the slave device 300 before adjusting the sequence number and the timestamp of the local data packet based on the synchronization data packet i is shown.

[0373] For the introduction of Figure 18A , please refer to the introduction of Figure 16B in the embodiment of the present application, which will not be described here.

[0374] Figure 18BFig. 4 shows a diagram of the main device 200 marking the data packets of the audio stream two with a sequence number and a time stamp.

[0375] The data packet 12 comprises the 53431st byte to the 61440th byte, and the expected play time of the data packet 12 is from 10:00:100 to 10:00:120. The data packet 13 comprises the 61321st byte to the 68820th byte, and the expected play time of the data packet 13 is from 10:00:120 to 10:00:130. The data packet 14 comprises the 68821st byte to the 76500th byte, and the expected play time of the data packet 14 is from 10:00:130 to 10:00:140. The data packet 15 comprises the 76501st byte to the 84180th byte, and the expected play time of the data packet 15 is from 10:00:140 to 10:00:150. The data packet 16 comprises the 84181st byte to the 91860th byte, and the expected play time of the data packet 16 is from 10:00:150 to 10:00:160.

[0376] From Fig. 4 it can be seen that the play progress of the data packet 8 in the slave device 300 is 300 bytes faster than the play progress of the data packet 12 in the main device 200, but the expected play time of the data packet 8 in the slave device 300 is 40 ms faster than the expected play time of the data packet 12 in the main device 200. Figure 18B Figure 18A From Fig. 4 it can be seen that the play progress of the data packet 9 in the slave device 300 is 300 bytes faster than the play progress of the data packet 13 in the main device 200, but the expected play time of the data packet 9 in the slave device 300 is 40 ms faster than the expected play time of the data packet 13 in the main device 200.

[0377] From Fig. 4 it can be seen that the play progress of the data packet 10 in the slave device 300 is 300 bytes faster than the play progress of the data packet 14 in the main device 200, but the expected play time of the data packet 10 in the slave device 300 is 40 ms faster than the expected play time of the data packet 14 in the main device 200.

[0378] From Fig. 4 it can be seen that the play progress of the data packet 11 in the slave device 300 is 300 bytes faster than the play progress of the data packet 15 in the main device 200, but the expected play time of the data packet 11 in the slave device 300 is 40 ms faster than the expected play time of the data packet 15 in the main device 200.

[0379] From Fig. 4 it can be seen that the play progress of the data packet 12 in the slave device 300 is 300 bytes faster than the play progress of the data packet 16 in the main device 200, but the expected play time of the data packet 12 in the slave device 300 is 40 ms faster than the expected play time of the data packet 16 in the main device 200.

[0380] From Fig. 4 it can be seen that the play progress of the data packet 12 in the slave device 300 is 300 bytes faster than the play progress of the data packet 16 in the main device 200, but the expected play time of the data packet 12 in the slave device 300 is 40 ms faster than the expected play time of the data packet 16 in the main device 200. ​

[0381] Therefore, it is required that the slave device 300 appends null bytes of the offset data length to the audio stream after receiving the audio stream, and adjusts the expected play time of the audio stream to which the null bytes of the offset data length are appended, so that the expected play time of the audio stream to which the null bytes of the offset data length are appended is delayed by 40 ms.

[0382] Figure 18C A diagram showing that the audio stream two to which the null bytes of the offset data length are appended is sequenced and time-stamped in the slave device 300 is shown.

[0383] Figure 18C With Figure 18B Similarly, the difference is that the data packet 12 in the slave device 300 includes null bytes of 300 bytes and the 53761st byte and the 61440th byte. The data packet 12 in the slave device 300 has the same expected play time as the data packet 12 in the master device 200.

[0384] The data packet 13, the data packet 14, the data packet 15, the data packet 16 in the slave device 300 and the data packet 13, the data packet 14, the data packet 15, the data packet 16 in the master device 200 contain the same byte content, and have the same expected play time.

[0385] Optionally, Figure 18C The diagram shown can be understood as the expected play time of the audio stream two after being updated in the slave device 300.

[0386] By analogy, the expected play time of the audio stream after the audio stream one in the slave device 300 is delayed by 40 ms.

[0387] If i is less than j, the slave device 300 appends null bytes of the offset data length to the front of the received audio stream, which indicates that the slave device 300 plays audio slower than the master device 200 plays audio, and the time of being slow is approximately the play duration of each data packet*(j-i). Therefore, when the slave device 300 sequences and time-stamps the audio stream received later, the sequence number of each data packet is added by the value of (i-j), and the time stamp of the data packet is updated based on the new sequence number of the data packet. In this way, the time stamp of each data packet divided by the audio stream received later is advanced by the play duration of each data packet*(j-i).

[0388] The slave device 300 can further update the timestamp of the data packet according to the above formula (1). The embodiments of the present application will not be described herein. Based on formula (1), when the slave device 300 stamps the audio stream after adding the null bytes of the offset data length with the sequence number and the timestamp, the sequence number of the data packet is added by the value of (i-j), so that the timestamp of each data packet divided by the audio stream after adding the null bytes of the offset data length is advanced by the playing time of each data packet*(j-i).

[0389] Figures 19A-19C Another set of diagrams for adjusting the audio stream two by the slave device 300 is shown.

[0390] Figure 19A A diagram for the slave device 300 to stamp the data packet of the audio stream two with the sequence number and the timestamp before adjusting the sequence number and the timestamp of the local data packet based on the synchronization data packet i is shown.

[0391] For example, j is equal to 7 and i is equal to 3.

[0392] For Figure 19A , reference can be made to the description of Figure 15B in the foregoing embodiments, and the embodiments of the present application will not be described herein.

[0393] Figure 19B A diagram for the master device 200 to stamp the data packet of the audio stream two with the sequence number and the timestamp in the master device 200 is shown.

[0394] The data packet 4 includes the 53431th byte to the 61440th byte, and the expected playing time of the data packet 4 is from 10:00:30 to 10:00:40. The data packet 5 includes the 61321th byte to the 68820th byte, and the expected playing time of the data packet 5 is from 10:00:40 to 10:00:50. The data packet 6 includes the 68821th byte to the 76500th byte, and the expected playing time of the data packet 6 is from 10:00:50 to 10:00:60. The data packet 7 includes the 76501th byte to the 84180th byte, and the expected playing time of the data packet 7 is from 10:00:60 to 10:00:70. The data packet 8 includes the 84181th byte to the 91860th byte, and the expected playing time of the data packet 8 is from 10:00:70 to 10:00:80.

[0395] From Figure 19B and Figure 19A it can be seen that the playing progress of the data packet 8 in the slave device 300 is 300 bytes faster than that of the data packet 4 in the master device 200, but the expected playing time of the data packet 8 in the slave device 300 is 40 ms slower than that of the data packet 4 in the master device 200.

[0396] The play progress of the packet 9 in the slave device 300 is 300 bytes faster than the play progress of the packet 5 in the master device 200, but the expected play time of the packet 9 in the slave device 300 is 40 ms slower than the expected play time of the packet 5 in the master device 200.

[0397] The play progress of the packet 10 in the slave device 300 is 300 bytes faster than the play progress of the packet 6 in the master device 200, but the expected play time of the packet 10 in the slave device 300 is 40 ms slower than the expected play time of the packet 6 in the master device 200.

[0398] The play progress of the packet 11 in the slave device 300 is 300 bytes faster than the play progress of the packet 7 in the master device 200, but the expected play time of the packet 11 in the slave device 300 is 40 ms slower than the expected play time of the packet 7 in the master device 200.

[0399] The play progress of the packet 12 in the slave device 300 is 300 bytes faster than the play progress of the packet 8 in the master device 200, but the expected play time of the packet 12 in the slave device 300 is 40 ms slower than the expected play time of the packet 8 in the master device 200.

[0400] Therefore, the slave device 300 needs to add the null bytes of the offset data length to the audio stream after receiving the audio stream, and adjust the expected play time of the audio stream to which the null bytes of the offset data length are added, so that the expected play time of the audio stream to which the null bytes of the offset data length are added is 40 ms in advance.

[0401] Figure 19C A schematic diagram of the slave device 300 stamping the audio stream to which the null bytes of the offset data length are added with a sequence number and a timestamp is shown.

[0402] Figure 19C With Figure 18B Similarly, the difference is that the packet 4 in the slave device 300 includes 300 bytes of null bytes and the 53761st byte and the 61140th byte. The expected play time of the packet 12 in the slave device 300 is the same as that of the packet 12 in the master device 200.

[0403] The byte contents contained in the packet 5, the packet 6, the packet 7, the packet 8 in the slave device 300 and the packet 5, the packet 6, the packet 7, the packet 8 in the master device 200 are the same, and the expected play times are also the same.

[0404] Optionally, Figure 19CThe diagram shown can be understood as the expected playing time of the second audio stream after the update of the device 300.

[0405] In this way, the expected playing time of the audio stream after the first audio stream of the device 300 is advanced by 40 ms.

[0406] If the offset is less than 0, as described above Figure 14 In the embodiment shown, the latter part of the audio segment in the local data packet j and the former part of the audio segment in the local data packet j+1 are matched with the audio segment in the synchronization data packet i. The latter part of the audio segment in the local data packet j+1 is matched with the former part of the audio segment in the synchronization data packet i+1. The data length of the former part of the audio segment in the local data packet j+1 can be referred to as the offset between the local data packet j and the synchronization data packet i.

[0407] Then, based on the offset between the local data packet j and the synchronization data packet i, the audio segment with the data length of the offset can be removed from the audio stream received subsequently, the audio stream after the removal of the audio segment with the data length of the offset is reserved, and the audio stream after the removal of the audio segment with the data length of the offset is packetized and time-stamped. In this way, the audio stream received subsequently from the device 300 is moved forward by the number of bytes of the offset data length, and the offset between the local data packet j+m and the synchronization data packet i+m can be 0.

[0408] In the case where the offset is less than 0, the size between the sequence number j of the local data packet j and the sequence number i of the synchronization data packet i also needs to be considered.

[0409] If the sequence number j is equal to the sequence number i, the audio stream after the removal of the audio segment with the data length of the offset can be packetized by the slave device 300, and the data packets can be time-stamped with the sequence numbers and the timestamps.

[0410] If i is greater than j, it means that the progress of the audio played by the slave device 300 is faster than the progress of the audio played by the master device 200. Then, the slave device 300 discards the bytes with the data length of the offset from the front of the audio stream received. It can be determined that the progress of the audio played by the slave device 300 is faster than the progress of the audio played by the master device 200 for about the playing duration of each data packet*(i-j). Therefore, when the data packets of the audio stream after the removal of the bytes with the data length of the offset are time-stamped with the sequence numbers and the timestamps, the sequence numbers of the data packets are all added by the value of (i-j), and the timestamps of the data packets are updated based on the new sequence numbers of the data packets. In this way, the timestamps of each data packet in the audio stream after the removal of the bytes with the data length of the offset are all delayed by the playing duration of each data packet*(i-j).

[0411] The slave device 300 can further update the timestamp of the data packet according to the above formula (2). The embodiments of the present application will not be described herein. Based on the formula (2), when the slave device 300 stamps the sequence number and the timestamp on the audio stream after discarding the offset data length of bytes, the sequence number of the data packet is added by the value of (i-j), so that the timestamp of each data packet of the audio stream received later is delayed by the playing time of each data packet*(i-j).

[0412] Figures 20A-20C Another set of diagrams for adjusting the audio stream two by the slave device 300 is shown.

[0413] For example, i is equal to 7 and j is equal to 3.

[0414] Figure 20A A diagram for the slave device 300 to stamp the sequence number and the timestamp on the data packet of the audio stream two is shown before adjusting the sequence number and the timestamp of the local data packet based on the synchronization data packet i.

[0415] For Figure 20A For Figure 16B The introduction of the slave device 300 can refer to the introduction of the master device 200 in the embodiments of the present application, which will not be described herein.

[0416] Figure 20B A diagram for the master device 200 to stamp the sequence number and the timestamp on the data packet of the audio stream two is shown.

[0417] The data packet 12 includes the 54061th byte to the 61740th byte, and the expected playing time of the data packet 12 is from 10:00:110 to 10:00:120. The data packet 13 includes the 61741th byte to the 69420th byte, and the expected playing time of the data packet 13 is from 10:00:120 to 10:00:130. The data packet 14 includes the 69421th byte to the 77100th byte, and the expected playing time of the data packet 14 is from 10:00:130 to 10:00:140. The data packet 15 includes the 77101th byte to the 84780th byte, and the expected playing time of the data packet 15 is from 10:00:140 to 10:00:150. The data packet 16 includes the 84781th byte to the 92460th byte, and the expected playing time of the data packet 16 is from 10:00:150 to 10:00:160.

[0418] The slave device 300 can further update the timestamp of the data packet according to the above formula (2). The embodiments of the present application will not be described herein. Based on the formula (2), when the slave device 300 stamps the sequence number and the timestamp on the audio stream after discarding the offset data length of bytes, the sequence number of the data packet is added by the value of (i-j), so that the timestamp of each data packet of the audio stream received later is delayed by the playing time of each data packet*(i-j). Figure 20B and Figure 20AIt can be seen that the play progress of packet 8 in the slave device 300 is 300 bytes slower than the play progress of packet 12 in the master device 200, but the expected play time of packet 8 in the slave device 300 is 40 ms faster than the expected play time of packet 12 in the master device 200.

[0419] The play progress of packet 9 in the slave device 300 is 300 bytes slower than the play progress of packet 13 in the master device 200, but the expected play time of packet 9 in the slave device 300 is 40 ms faster than the expected play time of packet 13 in the master device 200.

[0420] The play progress of packet 10 in the slave device 300 is 300 bytes slower than the play progress of packet 14 in the master device 200, but the expected play time of packet 10 in the slave device 300 is 40 ms faster than the expected play time of packet 14 in the master device 200.

[0421] The play progress of packet 11 in the slave device 300 is 300 bytes slower than the play progress of packet 15 in the master device 200, but the expected play time of packet 11 in the slave device 300 is 40 ms faster than the expected play time of packet 15 in the master device 200.

[0422] The play progress of packet 12 in the slave device 300 is 300 bytes slower than the play progress of packet 16 in the master device 200, but the expected play time of packet 12 in the slave device 300 is 40 ms faster than the expected play time of packet 16 in the master device 200.

[0423] Therefore, the slave device 300 needs to remove the offset data length from the audio stream after receiving the audio stream, and adjust the expected play time of the audio stream after removing the offset data length, so that the expected play time of the audio stream after removing the offset data length is delayed by 40 ms.

[0424] Figure 20C A schematic diagram of the slave device 300 marking the audio stream after removing the offset data length with a sequence number and a timestamp is shown.

[0425] Figure 20C With Figure 20B Similarly, the difference is that the 54061st byte and the 61740th byte of packet 12 in the slave device 300. The expected play time of packet 12 in the slave device 300 is the same as that of packet 12 in the master device 200.

[0426] The byte contents of the data packet 13, the data packet 14, the data packet 15, the data packet 16 in the slave device 300 and the data packet 13, the data packet 14, the data packet 15, the data packet 6 in the master device 200 are the same, and the expected playing time is also the same.

[0427] Optionally, Figure 20C The schematic diagram shown can be understood as the expected playing time of the audio stream two after the update of the slave device 300.

[0428] By analogy, the expected playing time of the audio stream after the audio stream one in the slave device 300 is delayed by 40 ms.

[0429] If i is less than j, the slave device 300 removes the audio stream of the front part of the received audio stream by the offset data length, which indicates that the slave device 300 plays the audio slower than the master device 200 plays the audio, and the time of the slowness is approximately the playing time of each data packet*(j-i). Therefore, when the slave device 300 serializes and timestamps the audio stream after removing the offset data length, the serial numbers of the data packets are all added by the value of (i-j), and the timestamps of the data packets are updated based on the new serial numbers of the data packets. In this way, the timestamps of each data packet divided by the audio stream after receiving the audio stream later are all advanced by the playing time of each data packet*(j-i).

[0430] The slave device 300 can further update the timestamps of the data packets based on the above formula (1). The embodiments of the present application will not be described herein again. Based on the formula (1), when the slave device 300 serializes and timestamps the audio stream after removing the offset data length, the serial numbers of the data packets are all added by the value of (i-j), and in this way, the timestamps of each data packet divided by the audio stream after adding the offset data length of the empty byte later are all advanced by the playing time of each data packet*(j-i).

[0431] Figures 21A-21C Another set of schematic diagrams of the slave device 300 adjusting the audio stream two is shown.

[0432] Figure 21A A schematic diagram of the slave device 300 serializing and timestamping the data packets of the audio stream two before adjusting the serial numbers and the timestamps of the local data packets based on the synchronization data packet i is shown.

[0433] For example, j is equal to 7, and i is equal to 3.

[0434] For the introduction of Figure 21A , reference can be made to the introduction of Figure 15B in the embodiments of the present application, which will not be described herein again.

[0435] Figure 21BFig. 4 shows a diagram of the main device 200 marking the data packets of the audio stream two with a sequence number and a time stamp.

[0436] Data packet 4 comprises the 54061st byte to the 61740th byte, and the expected play time of data packet 4 is from 10:00:30 to 10:00:40. Data packet 5 comprises the 61741st byte to the 69420th byte, and the expected play time of data packet 5 is from 10:00:40 to 10:00:50. Data packet 6 comprises the 69421st byte to the 77100th byte, and the expected play time of data packet 6 is from 10:00:50 to 10:00:60. Data packet 7 comprises the 7720th byte to the 84780th byte, and the expected play time of data packet 7 is from 10:00:60 to 10:00:70. Data packet 8 comprises the 84781st byte to the 92460th byte, and the expected play time of data packet 8 is from 10:00:70 to 10:00:80.

[0437] From Fig. 4 it can be seen that the play progress of data packet 8 in the slave device 300 is 300 bytes faster than the play progress of data packet 4 in the main device 200, but the expected play time of data packet 8 in the slave device 300 is 40 ms slower than the expected play time of data packet 4 in the main device 200. Figure 21B Figure 21A From Fig. 4 it can be seen that the play progress of data packet 9 in the slave device 300 is 300 bytes faster than the play progress of data packet 5 in the main device 200, but the expected play time of data packet 9 in the slave device 300 is 40 ms slower than the expected play time of data packet 5 in the main device 200.

[0438] From Fig. 4 it can be seen that the play progress of data packet 10 in the slave device 300 is 300 bytes faster than the play progress of data packet 6 in the main device 200, but the expected play time of data packet 10 in the slave device 300 is 40 ms slower than the expected play time of data packet 6 in the main device 200.

[0439] From Fig. 4 it can be seen that the play progress of data packet 11 in the slave device 300 is 300 bytes faster than the play progress of data packet 7 in the main device 200, but the expected play time of data packet 11 in the slave device 300 is 40 ms slower than the expected play time of data packet 7 in the main device 200.

[0440] From Fig. 4 it can be seen that the play progress of data packet 12 in the slave device 300 is 300 bytes faster than the play progress of data packet 8 in the main device 200, but the expected play time of data packet 12 in the slave device 300 is 40 ms slower than the expected play time of data packet 8 in the main device 200.

[0441] From Fig. 4 it can be seen that the play progress of data packet 12 in the slave device 300 is 300 bytes faster than the play progress of data packet 8 in the main device 200, but the expected play time of data packet 12 in the slave device 300 is 40 ms slower than the expected play time of data packet 8 in the main device 200.

[0442] ​Therefore, it is necessary to remove the offset data length from the audio stream after the device 300 receives the audio stream, and adjust the expected play time of the audio stream after removing the offset data length, so that the expected play time of the audio stream after removing the offset data length is 40 ms in advance.

[0443] Figure 21C The schematic diagram of the device 300 marking the audio stream two after removing the offset data length with a serial number and a timestamp is shown.

[0444] Figure 21C With Figure 21B Similarly, the difference is that the data packet 4 in the slave device 300 includes the 54061th byte and the 61740th byte. The expected play time of the data packet 4 in the slave device 300 is the same as that of the data packet 4 in the master device 200.

[0445] The data packets 5, 6, 7, and 8 in the slave device 300 and the data packets 5, 6, 7, and 8 in the master device 200 contain the same byte content and have the same expected play time.

[0446] Optionally, Figure 21C The schematic diagram shown can be understood as the expected play time of the audio stream two after updating in the slave device 300.

[0447] By analogy, the expected play time of the audio stream after the audio stream one in the slave device 300 is 40 ms in advance.

[0448] After the slave device 300 adjusts the expected play time of the received data stream, the slave device 300 can place the data packets in the buffer area for playing. The slave device 300 can calculate the actual play time of each data packet based on the number of data packets in the play buffer queue. And adjust the buffered data packets based on the actual play time and the expected play time of each data packet.

[0449] Among them, the play buffer queue is a queue of data packets to be played in the hardware of the slave device 300. After the slave device 300 packetizes the audio stream, the slave device 300 places the data packets in the first buffer area. When the slave device 300 starts playing the audio, the slave device 300 obtains the data packets from the first buffer area and places the data packets in the play buffer area. The slave device 300 reads the data packets from the play buffer area in sequence and plays the data packets.

[0450] Figures 22-24 The schematic diagram of how the slave device 300 plays the audio stream is shown.

[0451] The actual playing time of each data packet can be determined based on the current system time, the number of data packets buffered in front of the data packet in the queue, and the average playing time of each data packet. For example, the current system time is 10:00:50, data packet 3 is about to be played, there are data packet 3, data packet 4, data packet 5, data packet 6 and data packet 7 in the playing buffer queue, and the actual playing time of data packet 7 is determined to be 10:00:120.

[0452] The foregoing embodiments introduce that the expected playing time of each data packet in the slave device 300 can be adjusted based on the sequence number and timestamp of the synchronization data packet sent by the master device 200, so that the playing time of the same audio data on the master device 200 and the slave device 300 is consistent.

[0453] However, due to different hardware processing times on the master device 200 or the slave device 300, the master device 200 or the slave device 300 can adjust the data packets in the buffered data packet queue based on the actual playing time and the expected playing time of each data packet.

[0454] The following embodiments of the present application illustrate how the slave device 300 adjusts the data packets in the buffered data packet queue based on the actual playing time and the expected playing time of each data packet.

[0455] Figure 22 An exemplary schematic diagram of the buffered data packet queue in the slave device 300 is shown.

[0456] The buffered data packet queue in the slave device 300 includes data packet 3, data packet 4, data packet 5, …, data packet m and data packet m+1. Among them, the slave device 300 plays the data packets in the data packet queue in the order of the arrangement of the data packets in the data packet queue.

[0457] If the slave device 300 determines that the actual playing time of data packet 3 is 10:00:50, and the expected playing time of data packet 3 is 10:00:100. The expected playing time of data packet 3 is 50ms different from the actual playing time of data packet 3, which is greater than the preset value (the first time length), and the preset value is greater than or equal to 0, for example, the preset value is 10ms, then the slave device 300 needs to delay playing data packet 3. Optionally, as Figure 23As shown, the slave device 300 can play the data packet 1 before the data packet 3, and the playing time of the data packet 1 is about 50 ms, which can be changed based on the difference between the expected playing time of the data packet 3 and the actual playing time of the data packet 3. Then the slave device 300 can start playing the data packet 1 at 10:00:50. After playing the data packet 1, the slave device 300 can start playing the data packet 3. In this way, the actual playing time of the data packet 3 played by the slave device 300 can be consistent with the expected playing time, so as to be consistent with the time of playing the data packet 3 by the master device 200.

[0458] If the slave device 300 determines that the actual playing time of the data packet 3 is 10:00:50, and the expected playing time of the data packet 3 is 10:00:30. The actual playing time of the data packet 3 is later than the expected playing time of the data packet 3, and the difference between the expected playing time of the data packet 3 and the actual playing time of the data packet 3 is 20 ms, which is greater than the preset value, and the preset value is greater than or equal to 0, for example, the preset value is 10 ms, as shown in the following table. Figure 24 As shown, the slave device 300 can discard the data packet 3 in the playing cache queue, and speed up the progress of playing the data packet by the slave device 300.

[0459] Optionally, if the difference between the expected playing time of the data packet 3 and the actual playing time of the data packet 3 is greater than the preset value but less than the playing time of the data packet 3, the slave device 300 can discard part of the data in the data packet 3. The length of the discarded part of the data in the data packet 3 is the difference between the expected playing time of the data packet 3 and the actual playing time of the data packet 3.

[0460] Figure 25 A flowchart of an audio synchronous playing method provided by an embodiment of the present application is shown.

[0461] S2501, the slave device receives a first audio stream, divides the first audio stream into a plurality of data packets with a first length, and determines the expected playing time of each data packet in the first audio stream.

[0462] S2502, the slave device receives a first synchronization data packet sent by the master device, and the first synchronization data packet includes first audio data, a first serial number of the first synchronization data packet, and a first expected playing time of the first synchronization data packet.

[0463] S2503, the slave device determines a first local data packet based on the first audio data in the first synchronization data packet, and the first local data packet includes second audio data, a second serial number of the first local data packet, and a second expected playing time of the first local data packet, and the second audio data partially matches or completely matches the first audio data.

[0464] S2504, receiving the second audio stream from the slave device, the second audio stream being an audio stream after the first audio stream.

[0465] S2505, setting, by the slave device, a start playing time of the second audio stream as a third expected playing time based on a difference between the first sequence number and the second sequence number and the first offset of the second audio data and the first audio data; playing, by the slave device, the second audio stream based on the third expected playing time.

[0466] In a possible implementation, the slave device sets the start playing time of the second audio stream as the third expected playing time based on the difference between the first sequence number and the second sequence number and the first offset of the second audio data and the first audio data, and specifically includes: in a case where the difference between the first sequence number and the second sequence number is 0 and the first offset is greater than a first value, the slave device adds data of the first offset in front of the second audio stream to obtain a second updated audio stream; the slave device sets the start playing time of the second updated audio stream as the third expected playing time; in a case where the difference between the first sequence number and the second sequence number is 0 and the first offset is less than a second value, the slave device removes data of the first offset in front of the second audio stream to obtain a second updated audio stream; the slave device sets the start playing time of the second updated audio stream as the third expected playing time; in a case where the difference between the first sequence number and the second sequence number is 0 and the first offset is greater than the second value and less than the first value, the slave device sets the start playing time of the second audio stream as the third expected playing time.

[0467] In this way, in a case where the first sequence number and the second sequence number are the same, if the first offset is greater than the first value, that is, the front part of the audio content in the first local data packet and the rear part of the audio content in the first synchronization data packet are the same, the slave device adds data of the first offset in front of the second audio stream, so that the audio content in the local data packet and the synchronization data packet in the second audio stream is completely the same.

[0468] If the first offset is less than the first value, that is, the rear part of the audio content in the first local data packet and the front part of the audio content in the first synchronization data packet are the same, the slave device removes data of the first offset in front of the second audio stream, so that the audio content in the local data packet and the synchronization data packet in the second audio stream is completely the same.

[0469] If the first offset is equal to the first value, the master device and the slave device are synchronously played, and the slave device does not need to adjust the second audio stream.

[0470] Specifically, refer to the embodiments shown in Figures 17A-17C which will not be described herein again.

[0471] In a possible implementation, the slave device sets the start playing time of the second updated audio stream to the third expected playing time, and specifically includes: the slave device determines the start playing time of the second updated audio stream to be the third expected playing time based on the expected playing time of the last data packet in the first audio stream and the playing duration of each data packet; and the slave device plays the second audio stream based on the third expected playing time, and specifically includes: the slave device plays the second updated audio stream based on the third expected playing time.

[0472] In this way, in the case where the first sequence number is the same as the second sequence number, the slave device can adjust the second audio stream based on the first offset, so as to adjust the playing time of each byte in the second audio stream.

[0473] Alternatively,

[0474] The slave device sets the start playing time of the second audio stream to the third expected playing time, and specifically includes: the slave device determines the start playing time of the second audio stream to be the third expected playing time based on the expected playing time of the last data packet in the first audio stream and the playing duration of each data packet.

[0475] In this way, in the case where the first sequence number is the same as the second sequence number, the time of the second audio stream can also not be adjusted.

[0476] In a possible implementation, the slave device sets the start playing time of the second audio stream to the third expected playing time based on the difference between the first sequence number and the second sequence number and the first offset between the second audio data and the first audio data, and specifically includes: in the case where the first sequence number and the second sequence number are not 0 and the first offset is greater than a first value, the slave device adds data of the first offset in front of the second audio stream to obtain a second updated audio stream; the slave device sets the start playing time of the second updated audio stream to the third expected playing time; in the case where the difference between the first sequence number and the second sequence number is not 0 and the first offset is less than a second value, the slave device removes data of the first offset in front of the second audio stream to obtain a second updated audio stream; the slave device sets the start playing time of the second updated audio stream to the third expected playing time; and in the case where the difference between the first sequence number and the second sequence number is not 0 and the first offset is greater than the second value and less than the first value, the slave device sets the start playing time of the second audio stream to the third expected playing time.

[0477] In this way, in the case where the first sequence number is different from the second sequence number, if the first offset is greater than the first value, that is, the front part of the audio content in the first local data packet and the rear part of the audio content in the first synchronization data packet are the same, the slave device makes the audio content in the local data packet and the synchronization data packet in the second audio stream completely the same by adding data of the first offset in front of the second audio stream. Specifically, reference can be made to Figures 18A-18C、 Figures 19A-19C The embodiments of the application are not described here again in the light of the embodiments shown in the drawings.

[0478] In the case where the first sequence number is different from the second sequence number, if the first offset is less than the first value, that is, the audio content in the latter part of the first local data packet and the audio content in the former part of the first synchronization data packet are the same, the slave device removes the data of the first offset from the front of the second audio stream, so that the audio content in the local data packet and the synchronization data packet in the second audio stream is completely the same. Specifically, reference can be made to Figures 20A-20C 、 Figures 21A-21C The embodiments of the application are not described here again in the light of the embodiments shown in the drawings.

[0479] In the case where the first sequence number is different from the second sequence number, if the first offset is equal to the first value, the master device and the slave device are synchronized in playing, and the slave device does not need to adjust the second audio stream. Specifically, reference can be made to Figures 15A-15D 、 Figures 16A-1 The embodiments of the application are not described here again in the light of the embodiments shown in the drawings.

[0480] In a possible implementation, the slave device sets the starting playing time of the second updated audio stream as the third expected playing time, specifically including: the slave device determines the starting playing time of the second updated audio stream as the third expected playing time based on the expected playing time of the last data packet in the first audio stream, the playing duration of each data packet, and the difference between the first sequence number and the second sequence number; and the slave device plays the second audio stream based on the third expected playing time, specifically including: the slave device plays the second updated audio stream based on the third expected playing time.

[0481] In this way, in the case where the first sequence number is different from the second sequence number, the slave device can adjust the second audio stream based on the first offset, and adjust the expected playing time of each byte in the second audio stream based on the difference between the first sequence number and the second sequence number.

[0482] Or,

[0483] The slave device sets the starting playing time of the second audio stream as the third expected playing time, specifically including: the slave device determines the starting playing time of the second audio stream as the third expected playing time based on the expected playing time of the last data packet in the first audio stream, the playing duration of each data packet, and the difference between the first sequence number and the second sequence number.

[0484] In this way, in the case where the first sequence number is different from the second sequence number, the slave device can adjust the expected playing time of each byte in the second audio stream based on the difference between the first sequence number and the second sequence number.

[0485] In a possible implementation, the slave device plays the second updated audio stream based on the third expected play time, specifically including: the slave device divides the second updated audio stream into a plurality of data packets containing the first length, and determines the expected play time of each data packet in the second updated audio stream based on the third expected play time and the play duration of each data packet; and the slave device plays the second updated audio stream based on the expected play time of each data packet in the second updated audio stream.

[0486] In a possible implementation, the slave device plays the second audio stream based on the third expected play time, specifically including: the slave device divides the second audio stream into a plurality of data packets containing the first length, and determines the expected play time of each data packet in the second audio stream based on the third expected play time and the play duration of each data packet; and the slave device plays the second audio stream based on the expected play time of each data packet in the second audio stream.

[0487] In a possible implementation, after the slave device plays the second audio stream based on the third expected play time, the method further includes:

[0488] The slave device receives the second synchronization data packet sent by the master device, the second synchronization data packet including third audio data, a third sequence number of the second synchronization data packet, and a fourth expected play time of the second synchronization data packet; the slave device determines a second local data packet based on the third audio data in the second synchronization data packet; the second local data packet includes fourth audio data, a fourth sequence number of the second local data packet, and a fifth expected play time of the second local data packet; wherein the difference between the fifth expected play time and the fourth expected play time is within the first preset value, and the third audio data is the same as the fourth audio data.

[0489] In this way, after the slave device adjusts the second audio stream, the difference between the expected times of the same audio content in the master device and the slave device is within the first preset value, so that the same audio content is played synchronously.

[0490] In a possible implementation, the plurality of data packets in the second audio stream include a third local data packet, and the start play time of the third local data packet is the third expected play time; the slave device plays the second updated audio stream based on the expected play time of each data packet in the second updated audio stream, specifically including: the slave device plays the third local data packet based on the third expected play time of the third local data packet.

[0491] In a possible implementation, the slave device plays the third local data packet based on the third expected play time of the third local data packet, specifically including: the slave device acquires a system time; the slave device determines an actual play time of the third local data packet based on the system time, a number of data packets in front of a play buffer queue where the third local data packet is located, and a play duration of each data packet; when the actual play time is greater than the third expected play time and a difference between the actual play time and the third expected play time is greater than a first duration, the slave device discards the third local data packet; when the actual play time is less than the third expected play time and the difference between the actual play time and the third expected play time is greater than the first duration, the slave device adds a data packet with the first duration before the second local data packet, and the first duration is a difference between the system time and the third expected play time; and when the actual play time is less than or greater than the third expected play time and the difference between the actual play time and the third expected play time is less than the first duration, the slave device plays the third local data packet based on the third expected play time.

[0492] The system time can be understood as a current time.

[0493] In this way, the slave device can also adjust whether to play a data packet based on an actual play time and an expected play time of each data packet. Specifically, reference can be made to the embodiment shown in Figures 22-24 The embodiments of the present application will not be repeated here.

[0494] In a possible implementation, the slave device determines the first local data packet based on the first audio data in the first synchronization data packet, specifically including: the slave device determines a byte position corresponding to a highest peak in the first synchronization data packet; the slave device acquires one or more local peaks matching the highest peak and byte positions corresponding to the one or more local peaks in the first audio stream from the first audio stream; the slave device determines a first local peak from the one or more local peaks, where an audio data segment where the first local peak is located is the same as the first audio data; and the slave device determines the first local data packet based on the byte position corresponding to the first local peak. In this way, the first local data packet is determined based on peak matching, which can reduce the amount of calculation of the slave device. Specifically, reference can be made to the embodiment shown in Figures 10-14 The embodiments of the present application will not be repeated here.

[0495] Through the method provided by the first aspect, the first aspect can achieve dynamic negotiation between the master device and the slave device in the process of playing audio, and achieve the effect of synchronously playing audio.

[0496] The embodiments of the present application can be combined in any manner to achieve different technical effects.

[0497] In the above embodiments, all or part of the processes can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the processes can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes described in the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available media can be magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as DVDs), or semiconductor media (such as solid state disks (SSD)), etc.

[0498] Those of ordinary skill in the art can understand that all or part of the processes in the above embodiments can be implemented by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The storage medium includes ROM or random access memory (RAM), magnetic disks or optical disks, and various media that can store program codes.

[0499] In summary, the above only describes the embodiments of the technical solutions of the present application, and is not used to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made according to the disclosure of the present application shall be included in the protection scope of the present application.

Claims

1. An audio synchronous playing method, characterized in that, The method comprises: receiving a first audio stream from a device, dividing the first audio stream into a plurality of data packets of a first length, and determining an expected playing time of each data packet in the first audio stream; receiving a first synchronization data packet sent by a master device, the first synchronization data packet comprising first audio data, a first sequence number of the first synchronization data packet, and a first expected playing time of the first synchronization data packet; determining a first local data packet based on the first audio data in the first synchronization data packet, the first local data packet comprising second audio data, a second sequence number of the first local data packet, and a second expected playing time of the first local data packet, the second audio data partially matching or completely matching the first audio data; receiving a second audio stream from the device, the second audio stream being an audio stream after the first audio stream; setting a starting playing time of the second audio stream to a third expected playing time based on a difference between the first sequence number and the second sequence number and a first offset between the second audio data and the first audio data; playing the second audio stream based on the third expected playing time.

2. The method of claim 1, wherein, The method comprises: in a case where the difference between the first sequence number and the second sequence number is 0 and the first offset is greater than a first value, adding data of the first offset to the second audio stream to obtain a second updated audio stream; setting the starting playing time of the second updated audio stream to the third expected playing time; in a case where the difference between the first sequence number and the second sequence number is 0 and the first offset is less than a second value, removing data of the first offset from the front of the second audio stream to obtain a second updated audio stream; setting the starting playing time of the second updated audio stream to the third expected playing time; in a case where the difference between the first sequence number and the second sequence number is 0 and the first offset is greater than the second value and less than the first value, setting the starting playing time of the second audio stream to the third expected playing time.

3. The method of claim 2, wherein, The method comprises: determining the starting playing time of the second updated audio stream to be the third expected playing time based on an expected playing time of a last data packet in the first audio stream and a playing duration of each data packet; playing the second audio stream based on the third expected playing time, specifically comprising: playing the second updated audio stream based on the third expected playing time; or setting the starting playing time of the second audio stream to the third expected playing time, specifically comprising: The slave device determines the start playing time of the second audio stream as the third expected playing time according to the expected playing time of the last data packet in the first audio stream, the playing time length of each data packet.

4. The method of claim 1, wherein, The slave device sets the start playing time of the second audio stream as the third expected playing time according to the difference between the first sequence number and the second sequence number and the first offset between the second audio data and the first audio data, and specifically includes the following steps. In the case that the difference between the first sequence number and the second sequence number is not 0 and the first offset is greater than a first value, the slave device adds data of the first offset in front of the second audio stream to obtain a second updated audio stream. The slave device sets the start playing time of the second updated audio stream as the third expected playing time. In the case that the difference between the first sequence number and the second sequence number is not 0 and the first offset is less than a second value, the slave device removes data of the first offset in front of the second audio stream to obtain a second updated audio stream. The slave device sets the start playing time of the second updated audio stream as the third expected playing time. In the case that the difference between the first sequence number and the second sequence number is not 0 and the first offset is greater than the second value and less than the first value, the slave device sets the start playing time of the second audio stream as the third expected playing time.

5. The method of claim 4, wherein, The slave device sets the start playing time of the second updated audio stream as the third expected playing time, and specifically includes the following steps. The slave device determines the start playing time of the second updated audio stream as the third expected playing time according to the expected playing time of the last data packet in the first audio stream, the playing time length of each data packet and the difference between the first sequence number and the second sequence number. The slave device plays the second audio stream according to the third expected playing time, and specifically includes the following steps. The slave device plays the second updated audio stream according to the third expected playing time. Or, The slave device sets the start playing time of the second audio stream as the third expected playing time, and specifically includes the following steps. The slave device determines the start playing time of the second audio stream as the third expected playing time according to the expected playing time of the last data packet in the first audio stream, the playing time length of each data packet and the difference between the first sequence number and the second sequence number.

6. The method according to claim 3 or 5, characterized in that, The slave device plays the second updated audio stream according to the third expected playing time, and specifically includes the following steps. The slave device divides the second updated audio stream into a plurality of data packets containing the first length and determines the expected playing time of each data packet in the second updated audio stream according to the third expected playing time and the playing time length of each data packet. The slave device plays the second updated audio stream according to the expected playing time of each data packet in the second updated audio stream.

7. The method according to any one of claims 1 to 5, characterized in that, The slave device plays the second audio stream according to the third expected playing time, and specifically includes the following steps. The slave device divides the second audio stream into a plurality of data packets containing the first length, and determines an expected playing time of each data packet in the second audio stream based on the third expected playing time and a playing duration of each data packet; The slave device plays the second audio stream based on the expected playing time of each data packet in the second audio stream.

8. The method according to any one of claims 1 to 5, characterized in that, After the slave device plays the second audio stream based on the third expected playing time, the method further comprises: The slave device receives a second synchronization data packet sent by the master device, the second synchronization data packet comprising third audio data, a third serial number of the second synchronization data packet, and a fourth expected playing time of the second synchronization data packet; The slave device determines a second local data packet based on the third audio data in the second synchronization data packet, the second local data packet comprising fourth audio data, a fourth serial number of the second local data packet, and a fifth expected playing time of the second local data packet; The difference between the fifth expected playing time and the fourth expected playing time is within a first preset value, and the third audio data is the same as the fourth audio data.

9. The method of claim 7, wherein, The plurality of data packets in the second audio stream comprises a third local data packet, and the start playing time of the third local data packet is the third expected playing time; The slave device plays the second audio stream based on the expected playing time of each data packet in the second audio stream, specifically comprising: The slave device plays the third local data packet based on the third expected playing time of the third local data packet.

10. The method of claim 9, wherein, The slave device plays the third local data packet based on the third expected playing time of the third local data packet, specifically comprising: The slave device acquires a system time; The slave device determines an actual playing time of the third local data packet based on the system time, the number of data packets in front of a playing cache queue where the third local data packet is located, and a playing duration of each data packet; When the actual playing time is greater than the third expected playing time and the difference between the actual playing time and the third expected playing time is greater than a first duration, the slave device discards the third local data packet; When the actual playing time is less than the third expected playing time and the difference between the actual playing time and the third expected playing time is greater than the first duration, the slave device adds data packets of the first duration before the third local data packet, the first duration being the difference between the system time and the third expected playing time; When the actual playing time is less than or greater than the third expected playing time and the difference between the actual playing time and the third expected playing time is less than the first duration, the slave device plays the third local data packet based on the third expected playing time.

11. The method according to any of claims 1-5 or claim 9 or claim 10, characterized by, The slave device determines a first local data packet based on the first audio data in the first synchronization data packet, specifically comprising: The slave device determines a byte position corresponding to a highest peak value in the first synchronization data packet; The slave device obtains one or more local peaks matching the highest peak and byte positions corresponding to the one or more local peaks in the first audio stream from the first audio stream; The slave device determines a first local peak from the one or more local peaks, wherein the first local peak is in the same audio data segment as the first audio data; The slave device determines the first local data packet based on the byte position corresponding to the first local peak.

12. An electronic device, being a slave device, characterized in that The slave device comprises one or more processors, one or more memories; the one or more memories are coupled to the one or more processors, the one or more memories are configured to store computer program codes, the computer program codes comprise computer instructions, and the one or more processors invoke the computer instructions to enable the slave device to perform the method in any one of claims 1-11.

13. A computer-readable storage medium comprising instructions, wherein: The instructions, when executed on the slave device, enable the slave device to perform the method in any one of claims 1-11.

14. A computer program product, characterised in that, The computer program product, when executed on the slave device, enables the slave device to perform the method in any one of claims 1-11.

Images