Multi-room multi-channel audio synchronization method, device, equipment and storage medium

Abstract

The present application relates to the field of audio processing technology, and discloses a multi-room multi-channel audio synchronization method, device, equipment and storage medium. The method determines the audio signal to be played in the main audio device, and uses a preset low-delay audio encoding strategy to encode the audio signal according to the channel to obtain audio encoding data; based on the user datagram protocol and the NACK retransmission mechanism, the audio encoding data is optimized to obtain an audio encoding sequence data packet; the audio encoding sequence data packet is controlled to be played synchronously in at least one slave audio device using the preset room synchronization control strategy and channel synchronization control strategy. In the present application, the audio signal is encoded by low-delay LC3 encoding, thereby minimizing the transmission delay, and the encoded audio signal is transmitted using the UDP protocol and the NACK retransmission mechanism, which reduces the network bandwidth occupancy and avoids the additional delay caused by transmission or retransmission.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a multi-room multi-channel audio synchronization method, device, equipment and storage medium. Background Art

[0002] In the field of real-time audio communication, especially multi-room and multi-channel audio systems, wireless or wired connections can be used. With the development of wireless technology, wireless audio devices in the system have become a trend, and wireless methods can simplify the lines.

[0003] In order to ensure the synchronization of audio signals between wireless audio devices, audio coding technologies such as AAC and MP3 and TCP protocols are currently used to reduce transmission delays. Although these coding schemes perform well in terms of compression rate, their encoding and decoding processes require the accumulation of audio frames for a certain period of time before processing, resulting in high algorithm delays. Similarly, although the TCP protocol can ensure the reliability of data transmission, its congestion control and retransmission mechanisms will cause significant delay increases when the network is unstable. Therefore, the currently used audio synchronization schemes still have problems with high audio delay and poor synchronization. Summary of the invention

[0004] The present application provides a multi-room multi-channel audio synchronization method, device, equipment and storage medium to solve the problems of high audio delay and poor synchronization in the audio system in the prior art.

[0005] The first aspect of the present application provides a multi-room multi-channel audio synchronization method, the method comprising: determining an audio signal to be played in a master audio device, and encoding the audio signal according to the channel using a preset low-delay audio encoding strategy to obtain audio encoding data; optimizing the audio encoding data based on a user datagram protocol and a NACK retransmission mechanism to obtain an audio encoding sequence data packet; using a preset room synchronization control strategy and a channel synchronization control strategy, controlling the audio encoding sequence data packet to be synchronously played in at least one slave audio device; wherein the room synchronization control strategy is a synchronization strategy for performing clock synchronization of multiple audio devices based on a distributed clock synchronization mechanism and performing clock synchronization of a master audio device and a slave audio device based on a PTP protocol; and the channel synchronization control strategy is a synchronization strategy for dynamically allocating audio transmission of each channel based on channel grouping.

[0006] In a feasible implementation, the method of encoding the audio signal according to channels using a preset low-delay audio encoding strategy to obtain audio encoding data includes: determining encoding parameters from the preset low-delay audio encoding strategy, the encoding parameters including frame length, bit rate and sampling rate; using an audio processing tool to separate the audio spectrum of each channel of the audio signal to obtain multiple channel audios; using an LC3 encoder and a parallel sub-frame processing mechanism to encode each channel audio according to the frame length, the bit rate and the sampling frequency to obtain audio encoding data corresponding to the audio signal.

[0007] In a feasible implementation manner, the LC3 encoder and the parallel subframe processing mechanism are used to encode each of the channel audios according to the frame length, the bit rate and the sampling frequency to obtain audio coding data corresponding to the audio signal, including: based on the bit rate, the frame length and the sampling rate, each of the channel audios is framed to obtain multiple audio frame signals, wherein the length of each audio frame signal is equal to the frame length and contains N sampling points, and the N sampling points are calculated based on the sampling rate; the parallel subframe processing mechanism is used to control the LC3 encoder to encode each of the audio frame signals, and splice them to obtain audio coding data corresponding to the audio signal.

[0008] In a feasible implementation, after determining the audio signal to be played in the main audio device, and before encoding the audio signal according to the channel using a preset low-delay audio encoding strategy to obtain audio encoding data, it also includes: using an adaptive noise suppression algorithm to suppress noise on the audio signal; using an adaptive filtering algorithm to cancel echoes on the printed audio signal; normalizing the audio signal after echo cancellation, and outputting an audio signal.

[0009] In a feasible implementation manner, the audio coding data is optimized based on the user datagram protocol and the NACK retransmission mechanism to obtain an audio coding sequence data packet, including: based on the data packet format defined in the user datagram protocol, adding a uniquely identified data packet header to the audio coding data of each audio frame signal, the data packet header including the frame sequence number, timestamp and redundancy mark; using the NACK retransmission mechanism and the sequence number, calculating the maximum number of retransmissions of the corresponding audio frame signal, and adding the maximum number of retransmissions and the NACK response mark to the audio coding data of the audio frame signal; sorting the audio frame signals after adding the data packet header, the maximum number of retransmissions and the NACK response mark in chronological order to obtain an audio coding sequence data packet.

[0010] In a feasible implementation manner, the use of a preset room synchronization control strategy and a channel synchronization control strategy to control the synchronous playback of the audio coding sequence data packet in at least one slave audio device includes: electing a master audio device through mutual voting through an election mechanism, and controlling the master audio device to send a synchronization instruction to multiple slave audio devices, wherein the synchronization instruction contains timestamp information, and the timestamp information includes the sending time of the master audio device and the receiving time of each slave audio device; calculating the clock deviation of each channel between the master audio device and each slave audio device according to a response reply returned by each slave audio device based on the synchronization instruction; and planning the audio coding sequence data packet to be played in the corresponding slave audio device based on the clock deviation of each channel.

[0011] In a feasible implementation, the clock deviation planning based on each channel plays the audio coding sequence data packet to the corresponding slave audio device, including: adjusting the bandwidth resources of each channel according to the priority allocation strategy of broadband resources based on the clock deviation of each channel; and sending the audio coding sequence data packet to the corresponding slave audio device for playback based on the adjusted bandwidth.

[0012] In a feasible implementation manner, the adjusting the bandwidth resources of each channel according to the priority allocation strategy of broadband resources based on the clock deviation of each channel includes: sorting the channels of each slave audio device according to the size of the clock deviation of each channel, wherein the channels include front channels, surround channels and bass channels; calculating the weighting coefficient of each channel according to the priority of the channel for bandwidth resources and the sorting of the channels; calculating the bandwidth resources of each channel based on the weighting coefficient and the bandwidth allocation bit rate formula; wherein the front channel and the bass channel have the highest priority.

[0013] In a feasible implementation, the clock deviation planning based on each channel plays the audio coding sequence data packet to the corresponding slave audio device, including: calculating the buffered audio size of the slave audio device based on the clock deviation of each channel; configuring a corresponding buffer based on the buffered audio size; monitoring the storage of the audio coding sequence data packet in the buffer, and controlling the corresponding slave audio device to play after the buffer is completely occupied.

[0014] In a feasible implementation manner, configuring a corresponding buffer based on the buffered audio size includes: using a delay compensation algorithm to calculate a corresponding buffer size based on the buffered audio size, wherein the buffer size is: ,in, is the adjusted buffer size, is the current buffer size, is the clock deviation, and r is the buffer adjustment factor.

[0015] In a feasible implementation manner, after using the preset room synchronization control strategy and channel synchronization control strategy to control the synchronous playback of the audio coding sequence data packet in at least one slave audio device, it also includes: detecting whether there is a NACK response message; if so, determining the target slave audio device based on the NACK response message, and polling and transmitting the audio coding sequence data packet to the target slave audio device according to the maximum number of retransmissions.

[0016] The second aspect of the present application provides a multi-room multi-channel audio synchronization device, including: an encoding module, used to determine the audio signal to be played in the main audio device, and use a preset low-delay audio encoding strategy to encode the audio signal according to the channel to obtain audio encoding data; an optimization module, used to optimize the audio encoding data based on the user datagram protocol and the NACK retransmission mechanism to obtain an audio encoding sequence data packet; a transmission module, used to control the audio encoding sequence data packet to be played synchronously in at least one slave audio device using a preset room synchronization control strategy and a channel synchronization control strategy; wherein the room synchronization control strategy is a synchronization strategy for performing clock synchronization of multiple audio devices based on a distributed clock synchronization mechanism and performing clock synchronization of a main audio device and a slave audio device based on a PTP protocol; the channel synchronization control strategy is a synchronization strategy for dynamically allocating audio transmission of each channel based on channel grouping.

[0017] A third aspect of the present application provides an audio device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the audio device executes the above-mentioned multi-room multi-channel audio synchronization method.

[0018] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the multi-room multi-channel audio synchronization method described above.

[0019] In the technical solution provided by the present application, the audio signal to be played in the main audio device is determined, and the audio signal is encoded according to the channel using the preset low-delay audio coding strategy (LC3) to obtain audio coding data; based on the user datagram (UDP) protocol and the NACK retransmission mechanism, the audio coding data is optimized to obtain an audio coding sequence data packet; the audio coding sequence data packet is controlled to be played synchronously in at least one slave audio device using the preset room synchronization control strategy and channel synchronization control strategy. In the present application, the audio signal is encoded by low-delay LC3 coding, thereby minimizing the transmission delay, and the encoded audio signal is transmitted using the UDP protocol and the NACK retransmission mechanism, which reduces the network bandwidth occupancy and avoids the additional delay caused by transmission or retransmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of an embodiment of a multi-room multi-channel audio synchronization method in an embodiment of the present application;

[0021] Figure 2 It is a schematic diagram of another embodiment of the multi-room multi-channel audio synchronization method in the embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of an embodiment of a multi-room multi-channel audio synchronization device in an embodiment of the present application;

[0023] Figure 4 It is a schematic diagram of another embodiment of the multi-room multi-channel audio synchronization device in the embodiment of the present application;

[0024] Figure 5 This is a schematic diagram of an embodiment of an audio device in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The embodiments of the present application provide a multi-room multi-channel audio synchronization method, apparatus, device and storage medium. By adopting low-latency LC3 coding as the audio coding scheme, the encoding and decoding delay can be significantly reduced compared to the traditional coding scheme. At the same time, it is also based on the reliable transmission mechanism of UDP, through the optimized retransmission algorithm, while ensuring reliability, the transmission delay is minimized to the greatest extent, and the audio synchronization optimization strategy based on multi-room and multi-channel scenarios is realized. Millisecond-level audio synchronization between devices. In multi-channel application scenarios, through innovative distributed clock synchronization and dynamic delay compensation mechanisms, accurate synchronization of each channel signal is ensured, providing reliable protection for the wireless home theater system.

[0026] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] It is understandable that the execution subject of the present application can be a multi-room multi-channel audio synchronization device, or a terminal or audio playback software or server, which is not limited here. The embodiment of the present application is explained by taking the server as the execution subject as an example. The server is a server in the local area network, and control software that can realize audio playback in multiple rooms is installed on the server, such as a certain music software. In this application, the server is used as the main audio device (which is set in one of the rooms), and other rooms or audio devices are used as slave audio devices.

[0028] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 , an embodiment of the multi-room multi-channel audio synchronization method in the embodiment of the present application includes:

[0029] 101. Determine an audio signal to be played in a primary audio device, and encode the audio signal according to channels using a preset low-latency audio encoding strategy to obtain audio encoding data.

[0030] In this embodiment, when the server obtains the audio signal to be played, it can be the audio output by the server's own music software, or it can be the audio transmitted to the server when the user connects to the server's audio software / platform through a smart phone, IPAD or other device, where the audio format can be advanced multi-channel formats such as Dolby AC-3, DTS-HD, Dolby TrueHD or Atmos.

[0031] It is understandable that the server is used as the main audio device and the speakers in the room are used as slave audio devices. The controller in the server detects whether the audio playback function is triggered, and the same audio signal is played in multiple rooms at the same time.

[0032] After determining the audio signal to be played, the audio signal is encoded. It should be noted that the encoding here is implemented by using a low-delay encoding method, such as an LC3 encoder. Furthermore, a parallel frame method combined with an LC3 encoder can be used to further reduce the encoding delay.

[0033] Specifically, encoding parameters are determined from a preset low-delay audio encoding strategy, the encoding parameters including frame length, bit rate and sampling rate; an audio processing tool is used to separate the audio spectrum of each channel of the audio signal to obtain multiple channel audios; an LC3 encoder and a parallel sub-frame processing mechanism are used to encode each channel audio according to the frame length, the bit rate and the sampling frequency to obtain audio encoding data corresponding to the audio signal.

[0034] It is understandable that the frame length, bit rate and sampling rate are obtained by analyzing the low-latency audio encoding strategy, and the specific values ​​of the frame length, bit rate and sampling rate are determined based on the sound effects played by the user on each slave audio device. Specifically, the frame length is determined according to the sound effects played and the parameters of the encoder. The frame length is usually short to reduce the encoding and decoding delays, and is usually between 10ms and 30ms.

[0035] The bit rate is determined based on the playback sound effect and the degree of compression of the audio data by the encoder, such as by calculating the balance between audio quality and transmission bandwidth.

[0036] When determining the sampling rate, the frequency resolution of the audio signal is used as the sampling rate. Common sampling rates include 44.1kHz, 48kHz, and 16kHz.

[0037] To separate the audio spectrum of each channel, use an audio processing tool (such as FFT) to separate the audio spectrum of each channel of the audio signal. The formula is:

[0038] ,

[0039] Where x[n] is the input signal, X[k] is the frequency domain signal, and N is the number of FFT points.

[0040] When encoding using the LC3 encoder and the parallel sub-frame processing mechanism, the LC3 encoder is first initialized, the frame length, bit rate and sampling rate are set, each channel audio is divided into multiple sub-frames according to the frame length, and then each sub-frame is LC3 encoded, and finally the encoded data of all sub-frames are combined into the final audio encoding data. Among them, when processing in parallel sub-frames, the audio frame is first divided into multiple sub-frames, and then the sub-frames are processed in parallel using a multi-threaded or parallel computing framework (such as OpenMP, CUDA), and finally the encoded data after parallel processing are merged.

[0041] 102. Based on the user datagram protocol and the NACK retransmission mechanism, the audio coding data is optimized to obtain an audio coding sequence data packet.

[0042] In this embodiment, the size of each data packet is determined, which usually depends on factors such as network bandwidth, delay and jitter; the audio encoding data is divided according to the size of the data packet, and each data packet contains a part of the audio encoding data; header information is added to each data packet, including a sequence number, a timestamp, a data packet size, etc. Furthermore, a checksum or a cyclic redundancy check field can be added for error detection.

[0043] 103. Using a preset room synchronization control strategy and a channel synchronization control strategy, control the audio coding sequence data packets to be synchronously played in at least one slave audio device.

[0044] It should be noted that the room synchronization control strategy is a synchronization strategy for clock synchronization of multiple audio devices based on a distributed clock synchronization mechanism and for clock synchronization of a master audio device and a slave audio device based on a PTP protocol; the channel synchronization control strategy is a synchronization strategy for dynamically allocating audio transmission of each channel based on channel grouping.

[0045] This step actually includes three parts: clock synchronization, channel grouping and dynamic allocation, and audio playback synchronization. Among them, clock synchronization is achieved through room synchronization control strategy, and channel grouping and dynamic allocation are achieved through channel synchronization control strategy.

[0046] When clock synchronization is achieved through a room synchronization control strategy, it includes: clock synchronization of multiple audio devices based on a distributed clock synchronization mechanism and clock synchronization of a master audio device and a slave audio device based on a PTP protocol.

[0047] It should be noted that the clock synchronization of multiple audio devices based on the distributed clock synchronization mechanism includes:

[0048] Implement the Network Time Protocol (NTP) or a similar distributed clock synchronization mechanism on each audio device; the audio device periodically sends time requests to the time server in the network and receives the response from the time server; adjust the local clock of the audio device according to the response of the time server to achieve clock synchronization between multiple audio devices; monitor clock drift and periodically perform clock correction to maintain synchronization accuracy.

[0049] The clock synchronization of the master audio device and the slave audio device based on the PTP protocol includes:

[0050] Configure the master clock of PTP (Precision Time Protocol) on the master audio device; configure the slave clock of PTP on the slave audio device, and set their master clock to the configured master audio device; PTP protocol achieves time synchronization by exchanging synchronization messages (Sync), follow messages (Follow_Up), delay request messages (Delay_Req) ​​and delay response messages (Delay_Resp); the slave audio device calculates its clock offset relative to the master audio device based on the received synchronization message, and adjusts its local clock to match the clock of the master audio device; monitor the PTP synchronization status and resynchronize when necessary.

[0051] Channel grouping and dynamic allocation are achieved through channel synchronization control strategy, including:

[0052] According to the channel configuration of the audio content (such as stereo, surround sound, etc.), the channels are divided into different groups; a unique identifier (ID) and the corresponding audio sending channel are assigned to each channel group; on the master audio device, according to the channel grouping and dynamic allocation strategy, audio sending tasks are generated, each task contains the audio data to be sent, the target channel group, the sending timestamp and other information; the audio device calculates the precise sending time of each audio sending task based on the clock synchronization result, and assigns the task to the corresponding slave audio device; after receiving the audio sending task, the slave audio device plays the audio data at the specified time point according to the local clock and synchronization information; monitors the audio playback synchronization status, and if an asynchronization is detected (such as delay, jitter, etc.), it is adjusted according to the synchronization control algorithm, such as increasing the buffer size, adjusting the sending time, etc.

[0053] After completing the above synchronization and allocation, on the master audio device, a playback task list is generated according to the playback order and timeline of the audio content; the playback task list contains information such as the playback time, duration, channel group, etc. of each audio clip; the master audio device calculates the precise playback time of each audio clip based on the clock synchronization result and the playback task list, and sends the playback instruction to the slave audio device; after receiving the playback instruction, the slave audio device starts playing the audio clip at the specified time point according to the local clock and synchronization information.

[0054] In an embodiment of the present application, the audio signal is encoded using low-latency LC3 encoding to minimize transmission delay, and the encoded audio signal is transmitted using the UDP protocol and NACK retransmission mechanism, thereby reducing network bandwidth occupancy and avoiding additional delays caused by transmission or retransmission.

[0055] See also Figure 2 Another embodiment of the multi-room multi-channel audio synchronization method in the embodiment of the present application includes:

[0056] 201. Obtain an audio signal to be played currently, and encode the audio signal using low-delay audio coding LC3.

[0057] In this step, encoding parameters are determined from a preset low-delay audio encoding strategy, wherein the encoding parameters include frame length, bit rate and sampling rate; an audio processing tool is used to separate the audio spectrum of each channel of the audio signal to obtain multiple channel audios; an LC3 encoder and a parallel sub-frame processing mechanism are used to encode each channel audio according to the frame length, the bit rate and the sampling frequency to obtain audio encoding data corresponding to the audio signal.

[0058] It should be noted that the encoding process is to perform frame processing on each channel audio based on the bit rate, the frame length and the sampling rate to obtain multiple audio frame signals, wherein the length of each audio frame signal is equal to the frame length and contains N sampling points, and the N sampling points are calculated based on the sampling rate; use the parallel subframe processing mechanism to control the LC3 encoder to encode each audio frame signal, and splice them to obtain audio encoding data corresponding to the audio signal.

[0059] In another feasible implementation, after determining the audio signal to be played in the main audio device, and before encoding the audio signal according to the channel using a preset low-delay audio encoding strategy to obtain audio encoding data, it also includes: using an adaptive noise suppression algorithm to suppress noise on the audio signal; using an adaptive filtering algorithm to echo-eliminate the printed audio signal; normalizing the audio signal after echo elimination, and outputting the audio signal.

[0060] The parallel sub-frame processing mechanism is used to control the LC3 encoder to encode each of the audio frame signals and splice them to obtain audio coding data corresponding to the audio signal. Specifically, the audio signal is collected by an audio collection device with a sampling frequency of 48kHz; noise suppression, echo elimination and automatic gain control are performed on the collected audio signal to ensure the quality of the audio signal; the audio signal is divided into ultra-short frames of 2.5ms, and each frame contains 120 sampling points; each frame of the audio signal is converted into a frequency domain signal by fast Fourier transform (FFT), and quantized and compressed, and Huffman coding is used for data compression processing.

[0061] In another feasible implementation, the use of the parallel subframe processing mechanism to control the LC3 encoder to encode each of the audio frame signals and splice them to obtain the audio encoding data corresponding to the audio signal is specifically implemented as follows:

[0062] First, the input audio frame signal is divided; specifically, the audio frame signal x[n] is sampled and divided into 2.5 ms frames, and each frame contains N=48×2.5=120 sampling points.

[0063] Then, the subframes are processed in parallel; specifically, each frame of data is divided into k subframes, the length of each subframe is M=N / k, and each subframe is processed in parallel.

[0064] Secondly, a discrete Fourier transform (DFT) is performed; specifically, a fast Fourier transform (FFT) is performed on each subframe, and the formula is: , k is the number of subframes.

[0065] 202. Based on the data packet format defined in the User Datagram Protocol, add a uniquely identified data packet header to the audio encoding data of each audio frame signal, where the data packet header includes a frame sequence number, a timestamp, and a redundancy flag.

[0066] Specifically, first generate a new data packet header for each audio frame signal and initialize its fields; then call the preset data packet header definition rules to determine the sequence number, timestamp and redundant mark, where the sequence number can be incremented from 0 and increased by 1 each time an audio frame signal is sent; the timestamp can use a high-precision timer (such as the clock_gettime function of POSIX) to obtain the current timestamp; the redundant mark is extracted by parsing the content field in each audio frame signal and extracting the frequency band without audio as redundant content; finally, the data packet header and audio frame data are packaged together to form a complete UDP data packet. During subsequent synchronous playback, the data packet header and audio frame data are extracted from the UDP data packet, and reassembled and played according to the sequence number and timestamp.

[0067] 203. Calculate the maximum number of retransmissions of the corresponding audio frame signal by using the NACK retransmission mechanism and the sequence number, and add the maximum number of retransmissions and the NACK response mark to the audio coding data of the audio frame signal.

[0068] 204. Arrange the audio frame signals after adding the data packet header, the maximum number of retransmissions and the NACK response mark in chronological order to obtain an audio coding sequence data packet.

[0069] The NACK retransmission mechanism here is used to return a NACK message to request retransmission of the lost data packet when data packet loss or damage is detected.

[0070] The calculated maximum number of retransmissions is a limit on the number of retransmissions of a data packet with each sequence number. When the number of retransmissions reaches a preset maximum number, the retransmission is abandoned.

[0071] The NACK response mark is used to mark that the retransmission request of the audio coding data is based on the NACK response, and the sequence number of the data packet currently requested to be retransmitted is carried in the feedback NACK message as the NACK response mark; after receiving the NACK message, the retransmission counter of the corresponding data packet is updated according to the sequence number, and prepare for retransmission.

[0072] 205. A master audio device is elected through mutual voting through an election mechanism, and the master audio device is controlled to send synchronization instructions to multiple slave audio devices.

[0073] The synchronization instruction includes timestamp information, and the timestamp information includes the sending time of the master audio device and the receiving time when the sending time arrives at each slave audio device.

[0074] In this embodiment, audio devices in multiple rooms broadcast messages to each other for voting. When one audio device receives broadcast messages from other audio devices that are greater than half of the total number of rooms, the audio device will broadcast its master device identity and status information to the network. After receiving the broadcast from the master device, other slave devices will confirm their identities and prepare to receive synchronization instructions.

[0075] When sending synchronization commands, the master device maintains a global timestamp to synchronize the playback time of all devices;

[0076] The slave device will periodically send time synchronization requests to the master device to ensure that its local time is consistent with that of the master device.

[0077] When the master device receives a play instruction, it calculates a unified play time point based on the global timestamp; the master device sends a play synchronization instruction to all slave devices, including key information such as the play start time and play duration;

[0078] After receiving the synchronization instruction, the slave device will adjust its playback time according to the instruction to ensure synchronization with the playback of the master device.

[0079] Furthermore, if the slave device fails to execute correctly after receiving the synchronization instruction (for example, due to network delay or device failure), it sends an error report to the master device; after receiving the error report, the master device can take corresponding remedial measures, such as resending the synchronization instruction or adjusting the playback strategy.

[0080] 206. Calculate the clock deviation of each channel between the master audio device and each slave audio device according to the response reply returned by each slave audio device based on the synchronization instruction.

[0081] The calculation of the clock deviation is actually obtained by exchanging timestamps:

[0082] First, the master device sends a synchronization message and records the sending time T1.

[0083] Then, the message is received from the device and the receiving time T2 is recorded.

[0084] Finally, the slave responds and the master records times T3 and T4.

[0085] Based on this, the clock deviation ΔT = ((T2−T1)−(T4−T3)) / 2.

[0086] 207. Based on the clock deviation plan of each channel, the audio coding sequence data packet is sent to the corresponding slave audio device for playing.

[0087] In this embodiment, the audio playback control based on the clock deviation is implemented in two ways: one is by adjusting the bandwidth resources; the other is by setting a buffer.

[0088] The methods of adjusting bandwidth resources include:

[0089] Sort the channels of each slave audio device according to the size of the clock deviation of each channel, wherein the channels include front channels, surround channels and bass channels;

[0090] Calculate the weighting coefficient of each channel according to the priority of using bandwidth resources by the channel and the order of the channels;

[0091] Based on the weighted coefficient and the bandwidth allocation bit rate formula, the bandwidth resources of each channel are calculated; wherein the front channel and the bass channel have the highest priority.

[0092] The bandwidth allocation bit rate formula is: , is the bit rate of the ith channel, is the weighting coefficient of the channel, B is the total bandwidth, and N is the total number of channels.

[0093] The method of setting a buffer zone includes:

[0094] Calculating the buffered audio size of the slave audio device based on the clock deviation of each channel;

[0095] Based on the buffered audio size, configure a corresponding buffer;

[0096] The storage of the audio coding sequence data packets in the buffer is monitored, and after the buffer is completely occupied, the corresponding slave audio device is controlled to play.

[0097] Wherein, configuring a corresponding buffer based on the buffered audio size includes:

[0098] Using a delay compensation algorithm, a corresponding buffer size is calculated based on the buffered audio size, wherein the buffer size is:

[0099] ,in, is the adjusted buffer size, is the current buffer size, is the clock deviation, and r is the buffer adjustment factor.

[0100] In another feasible implementation manner, after using the preset room synchronization control strategy and channel synchronization control strategy to control the audio coding sequence data packets to be synchronously played on at least one slave audio device, the method further includes:

[0101] Detect whether there is a NACK response message;

[0102] If so, the target slave audio device is determined based on the NACK response message, and the audio coding sequence data packet is polled and transmitted to the target slave audio device according to the maximum number of retransmissions.

[0103] In the embodiment of the present application, the technical problem of multi-room and multi-channel real-time audio synchronization is successfully solved by innovatively combining low-delay LC3 coding technology, optimized UDP transmission mechanism and distributed synchronization strategy. At the audio coding level, the LC3 coding scheme with an ultra-short frame length of 2.5ms is adopted, and the parallel subframe processing mechanism is used to control the total coding and decoding delay within 1.5ms, which is nearly 10 times the performance of the traditional audio coding scheme. At the network transmission level, through the carefully designed NACK-based UDP reliable transmission mechanism, combined with selective redundant transmission and intelligent congestion control strategy, the network transmission delay is reduced by more than 60% while ensuring transmission reliability. The scheme also innovatively introduces a distributed clock synchronization mechanism based on PTP and a device-level synchronization control strategy, and cooperates with adaptive buffer management and dynamic delay compensation technology to successfully control the audio synchronization difference between multiple devices within 1ms. In multi-channel application scenarios, efficient bandwidth utilization is achieved through channel grouping coding and dynamic bit allocation strategy, while ensuring the audio quality of each channel. The system is also equipped with a complete performance monitoring, fault recovery and exception handling mechanism to ensure the stability and reliability of the scheme in practical applications. Through the organic combination of these technological innovations, the system achieves excellent performance with end-to-end latency as low as 70ms, while ensuring excellent audio quality and multi-device synchronization, providing a comprehensive technical solution for the field of real-time audio communication. Especially in multi-channel application scenarios such as home theaters, this solution not only solves the latency and synchronization problems of wireless speakers, but also provides a more flexible deployment solution than traditional wired systems, significantly improving the user experience.

[0104] The above describes the multi-room multi-channel audio synchronization method in the embodiment of the present application. The following describes the multi-room multi-channel audio synchronization device in the embodiment of the present application. Figure 3 In the embodiment of the present application, an embodiment of a multi-room multi-channel audio synchronization device includes:

[0105] The encoding module 310 is used to determine the audio signal to be played in the main audio device, and encode the audio signal according to the channel using a preset low-delay audio encoding strategy to obtain audio encoding data;

[0106] The optimization module 320 is used to optimize the audio coding data based on the user datagram protocol and the NACK retransmission mechanism to obtain an audio coding sequence data packet;

[0107] The transmission module 330 is used to control the synchronous playback of the audio coding sequence data packets in at least one slave audio device by using a preset room synchronization control strategy and a channel synchronization control strategy; wherein the room synchronization control strategy is a synchronization strategy for performing clock synchronization of multiple audio devices based on a distributed clock synchronization mechanism and performing clock synchronization of a master audio device and a slave audio device based on a PTP protocol; and the channel synchronization control strategy is a synchronization strategy for dynamically allocating audio transmission of each channel based on channel grouping.

[0108] In the embodiment of the present application, the audio signal to be played in the main audio device is determined, and the audio signal is encoded according to the channel using the preset low-delay audio coding strategy (LC3) to obtain audio coding data; based on the user datagram (UDP) protocol and the NACK retransmission mechanism, the audio coding data is optimized to obtain an audio coding sequence data packet; and the audio coding sequence data packet is controlled to be played synchronously in at least one slave audio device using the preset room synchronization control strategy and channel synchronization control strategy. This solves the problem of high audio delay and poor synchronization in the audio system in the prior art.

[0109] See also Figure 4 Another embodiment of the multi-room multi-channel audio synchronization device in the embodiment of the present application includes:

[0110] The encoding module 310 is used to determine the audio signal to be played in the main audio device, and encode the audio signal according to the channel using a preset low-delay audio encoding strategy to obtain audio encoding data;

[0111] The optimization module 320 is used to optimize the audio coding data based on the user datagram protocol and the NACK retransmission mechanism to obtain an audio coding sequence data packet;

[0112] The transmission module 330 is used to control the synchronous playback of the audio coding sequence data packets in at least one slave audio device by using a preset room synchronization control strategy and a channel synchronization control strategy; wherein the room synchronization control strategy is a synchronization strategy for performing clock synchronization of multiple audio devices based on a distributed clock synchronization mechanism and performing clock synchronization of a master audio device and a slave audio device based on a PTP protocol; and the channel synchronization control strategy is a synchronization strategy for dynamically allocating audio transmission of each channel based on channel grouping.

[0113] Optionally, the encoding module 310 includes:

[0114] A determination unit 311, configured to determine encoding parameters from a preset low-delay audio encoding strategy, wherein the encoding parameters include frame length, bit rate, and sampling rate;

[0115] A separation unit 312, configured to separate the audio spectrum of each channel of the audio signal by using an audio processing tool to obtain multiple channel audios;

[0116] The encoding unit 313 is used to encode the audio of each channel according to the frame length, the bit rate and the sampling frequency by using the LC3 encoder and the parallel sub-frame processing mechanism to obtain the audio encoding data corresponding to the audio signal.

[0117] Optionally, the encoding unit 313 is specifically used for:

[0118] Based on the bit rate, the frame length and the sampling rate, perform frame processing on each of the channel audios to obtain multiple audio frame signals, wherein the length of each audio frame signal is equal to the frame length and includes N sampling points, and the N sampling points are calculated based on the sampling rate;

[0119] The parallel sub-frame processing mechanism is used to control the LC3 encoder to encode each of the audio frame signals and splice them to obtain audio encoding data corresponding to the audio signal.

[0120] Optionally, the encoding module 310 further includes a preprocessing unit 314, configured to:

[0121] Performing noise suppression on the audio signal using an adaptive noise suppression algorithm;

[0122] Using adaptive filtering algorithm to perform echo cancellation on printed audio signals;

[0123] The audio signal after echo cancellation is normalized and outputted.

[0124] Optionally, the optimization module 320 includes:

[0125] An adding unit 321 is used to add a uniquely identified data packet header to the audio coded data of each of the audio frame signals based on a data packet format defined in a user datagram protocol, the data packet header including a frame sequence number, a timestamp and a redundancy flag;

[0126] A first calculation unit 322 is used to calculate the maximum number of retransmissions of the corresponding audio frame signal by using the NACK retransmission mechanism and the sequence number, and add the maximum number of retransmissions and the NACK response mark to the audio coding data of the audio frame signal;

[0127] The sorting unit 323 is used to sort the audio frame signals after adding the data packet header, the maximum number of retransmissions and the NACK response mark in chronological order to obtain an audio coding sequence data packet.

[0128] Optionally, the transmission module 330 includes:

[0129] An election unit 331 is used to elect a master audio device through mutual voting by an election mechanism, and control the master audio device to send a synchronization instruction to multiple slave audio devices, wherein the synchronization instruction includes timestamp information, and the timestamp information includes the sending time of the master audio device and the receiving time of each slave audio device;

[0130] A second calculation unit 332, configured to calculate the clock deviation of each channel between the master audio device and each slave audio device according to the response reply returned by each slave audio device based on the synchronization instruction;

[0131] The transmission unit 333 is used to transmit the audio coding sequence data packets to the corresponding slave audio device for playback based on the clock deviation planning of each channel.

[0132] Optionally, the transmission unit 333 is specifically used for:

[0133] Based on the clock deviation of each channel, adjusting the bandwidth resources of each channel according to the priority allocation strategy of the broadband resources;

[0134] The audio coding sequence data packet is sent to the corresponding slave audio device for playing based on the adjusted bandwidth.

[0135] Optionally, the transmission unit 333 is specifically used for:

[0136] Sort the channels of each slave audio device according to the size of the clock deviation of each channel, wherein the channels include front channels, surround channels and bass channels;

[0137] Calculate the weighting coefficient of each channel according to the priority of using bandwidth resources by the channel and the order of the channels;

[0138] Based on the weighted coefficient and the bandwidth allocation bit rate formula, the bandwidth resources of each channel are calculated; wherein the front channel and the bass channel have the highest priority.

[0139] Optionally, the transmission unit 333 is specifically used for:

[0140] Calculating the buffered audio size of the slave audio device based on the clock deviation of each channel;

[0141] Based on the buffered audio size, configure a corresponding buffer;

[0142] The storage of the audio coding sequence data packets in the buffer is monitored, and after the buffer is completely occupied, the corresponding slave audio device is controlled to play.

[0143] Optionally, the transmission unit 333 is specifically used for:

[0144] Using a delay compensation algorithm, a corresponding buffer size is calculated based on the buffered audio size, wherein the buffer size is:

[0145] ,in, is the adjusted buffer size, is the current buffer size, is the clock deviation, and r is the buffer adjustment factor.

[0146] Optionally, the transmission unit 333 is further used for:

[0147] Detect whether there is a NACK response message;

[0148] If so, the target slave audio device is determined based on the NACK response message, and the audio coding sequence data packet is polled and transmitted to the target slave audio device according to the maximum number of retransmissions.

[0149] In the embodiments of the present application, in the present application, the audio signal is encoded by low-latency LC3 encoding, thereby minimizing the transmission delay, and the encoded audio signal is transmitted using the UDP protocol and the NACK retransmission mechanism, thereby reducing network bandwidth occupancy and avoiding additional delays caused by transmission or retransmission.

[0150] above Figure 3 and Figure 4 The multi-room multi-channel audio synchronization device in the embodiment of the present application is described in detail from the perspective of modular functional entities, and the audio device in the embodiment of the present application is described in detail from the perspective of hardware processing.

[0151] See also Figure 5As shown, the audio device includes a processor 500 and a memory 501 . The memory 501 stores machine executable instructions that can be executed by the processor 500 . The processor 500 executes the machine executable instructions to implement the multi-room multi-channel audio synchronization method.

[0152] Further, Figure 5 The audio device shown further includes a bus 502 and a communication interface 503 , and the processor 500 , the communication interface 503 and the memory 501 are connected via the bus 502 .

[0153] Among them, the memory 501 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), for example, at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 503 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 502 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0154] The processor 500 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 500. The above processor 500 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 501 , and the processor 500 reads the information in the memory 501 and completes the method steps of the above-mentioned embodiment in combination with its hardware.

[0155] The present application also provides an audio device, wherein the computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the multi-room multi-channel audio synchronization method in the above-mentioned embodiments.

[0156] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the multi-room multi-channel audio synchronization method provided above.

[0157] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0159] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-room multi-channel audio synchronization method, characterized in that: The method comprises: Determine an audio signal to be played in a main audio device, and encode the audio signal according to channels using a preset low-latency audio encoding strategy to obtain audio encoding data; Separate the audio spectrum of each channel of the audio signal to obtain multiple channel audio, perform frame processing on each channel audio, and obtain multiple audio frame signals; calculate the maximum number of retransmissions of the audio frame signal, and add the data packet header, the maximum number of retransmissions, and the NACK response mark to each of the audio frame signals based on the user datagram protocol, and sort them in chronological order to obtain an audio coding sequence data packet; The preset room synchronization control strategy and channel synchronization control strategy are used to control the audio coding sequence data packet to be played synchronously in at least one slave audio device; wherein the room synchronization control strategy is a synchronization strategy for performing clock synchronization of multiple audio devices based on a distributed clock synchronization mechanism and performing clock synchronization of a master audio device and a slave audio device based on a PTP protocol; and the channel synchronization control strategy is a synchronization strategy for adjusting the bandwidth resources of each channel according to a priority allocation strategy of broadband resources based on the clock deviation of each channel and sending the audio coding sequence data packet to the corresponding slave audio device for playback based on the adjusted bandwidth.

2. The multi-room multi-channel audio synchronization method according to claim 1, characterized in that: The method of encoding the audio signal according to the channel by using a preset low-delay audio encoding strategy to obtain audio encoding data includes: Determining encoding parameters from a preset low-delay audio encoding strategy, the encoding parameters including frame length, bit rate, and sampling rate; Separating the audio spectrum of each channel of the audio signal by using an audio processing tool to obtain multiple channel audio; The audio of each channel is encoded according to the frame length, the bit rate and the sampling rate by using the LC3 encoder and the parallel sub-frame processing mechanism to obtain audio encoding data corresponding to the audio signal.

3. The multi-room multi-channel audio synchronization method according to claim 2, characterized in that: The method of encoding each of the channel audio signals according to the frame length, the bit rate and the sampling rate by using the LC3 encoder and the parallel subframe processing mechanism to obtain the audio coding data corresponding to the audio signal includes: Based on the bit rate, the frame length and the sampling rate, perform frame processing on each of the channel audios to obtain multiple audio frame signals, wherein the length of each audio frame signal is equal to the frame length and includes N sampling points, and the N sampling points are calculated based on the sampling rate; The parallel sub-frame processing mechanism is used to control the LC3 encoder to encode each of the audio frame signals and splice them to obtain audio encoding data corresponding to the audio signal.

4. The multi-room multi-channel audio synchronization method according to claim 1, characterized in that: After determining the audio signal to be played in the main audio device, and before encoding the audio signal according to the channel using the preset low-delay audio encoding strategy to obtain the audio encoding data, the method further includes: Performing noise suppression on the audio signal using an adaptive noise suppression algorithm; Using adaptive filtering algorithm to perform echo cancellation on printed audio signals; The audio signal after echo cancellation is normalized and outputted.

5. The multi-room multi-channel audio synchronization method according to claim 3, characterized in that: The method of calculating the maximum number of retransmissions of the audio frame signal, and adding the data packet header, the maximum number of retransmissions, and the NACK response mark to each of the audio frame signals based on the user datagram protocol, and sorting them in chronological order to obtain an audio coding sequence data packet includes: Based on the data packet format defined in the user datagram protocol, a data packet header with a unique identifier is added to the audio coded data of each of the audio frame signals, the data packet header including a frame sequence number, a timestamp and a redundancy mark; Calculate the maximum number of retransmissions of the corresponding audio frame signal by using the NACK retransmission mechanism and the sequence number, and add the maximum number of retransmissions and the NACK response mark to the audio coding data of the audio frame signal; After adding the data packet header, the maximum number of retransmissions and the NACK response mark, the audio frame signals are sorted in chronological order to obtain an audio coding sequence data packet.

6. The multi-room multi-channel audio synchronization method according to claim 1, characterized in that: The method of using a preset room synchronization control strategy and a channel synchronization control strategy to control the synchronous playback of the audio coding sequence data packets in at least one slave audio device comprises: The master audio device is elected through mutual voting by an election mechanism, and the master audio device is controlled to send synchronization instructions to multiple slave audio devices, wherein the synchronization instructions contain timestamp information, and the timestamp information includes the sending time of the master audio device and the receiving time of each slave audio device; Calculating the clock deviation of each channel between the master audio device and each slave audio device according to the response reply returned by each slave audio device based on the synchronization instruction; The audio coding sequence data packets are sent to the corresponding slave audio devices for playback based on the clock deviation planning of each channel.

7. The multi-room multi-channel audio synchronization method according to claim 6, characterized in that: The step of planning the clock deviation of each channel to send the audio coding sequence data packets to the corresponding slave audio device for playback includes: Based on the clock deviation of each channel, adjusting the bandwidth resources of each channel according to the priority allocation strategy of the broadband resources; The audio coding sequence data packet is sent to the corresponding slave audio device for playing based on the adjusted bandwidth.

8. The multi-room multi-channel audio synchronization method according to claim 7, characterized in that: The adjusting the bandwidth resources of each channel according to the priority allocation strategy of the broadband resources based on the clock deviation of each channel includes: Sort the channels of each slave audio device according to the size of the clock deviation of each channel, wherein the channels include front channels, surround channels and bass channels; Calculate the weighting coefficient of each channel according to the priority of using bandwidth resources by the channel and the order of the channels; Based on the weighted coefficient and the bandwidth allocation bit rate formula, the bandwidth resources of each channel are calculated; wherein the front channel and the bass channel have the highest priority.

9. The multi-room multi-channel audio synchronization method according to claim 6, characterized in that: The step of planning the clock deviation of each channel to send the audio coding sequence data packets to the corresponding slave audio device for playback includes: Calculating the buffered audio size of the slave audio device based on the clock deviation of each channel; Based on the buffered audio size, configure a corresponding buffer; The storage of the audio coding sequence data packets in the buffer is monitored, and after the buffer is completely occupied, the corresponding slave audio device is controlled to play.

10. The multi-room multi-channel audio synchronization method according to claim 9, characterized in that: The configuring a corresponding buffer based on the buffered audio size includes: Using a delay compensation algorithm, a corresponding buffer size is calculated based on the buffered audio size, wherein the buffer size is: ,in, is the adjusted buffer size, is the current buffer size, is the clock deviation, and r is the buffer adjustment factor.

11. The multi-room multi-channel audio synchronization method according to claim 5, characterized in that: After using the preset room synchronization control strategy and channel synchronization control strategy to control the audio coding sequence data packets to be synchronously played in at least one slave audio device, the method further includes: Detect whether there is a NACK response message; If so, the target slave audio device is determined based on the NACK response message, and the audio coding sequence data packet is polled and transmitted to the target slave audio device according to the maximum number of retransmissions.

12. A multi-room multi-channel audio synchronization device, characterized in that: The device comprises: An encoding module, used for determining an audio signal to be played in a main audio device, and encoding the audio signal according to channels using a preset low-delay audio encoding strategy to obtain audio encoding data; The optimization module is used to separate the audio spectrum of each channel of the audio signal to obtain multiple channel audios, perform frame processing on each channel audio to obtain multiple audio frame signals; calculate the maximum number of retransmissions of the audio frame signal, and add the data packet header, the maximum number of retransmissions and the NACK response mark to each of the audio frame signals based on the user datagram protocol, and sort them in chronological order to obtain audio coding sequence data packets; A transmission module is used to control the synchronous playback of the audio coding sequence data packet in at least one slave audio device by using a preset room synchronization control strategy and a channel synchronization control strategy; wherein the room synchronization control strategy is a synchronization strategy for performing clock synchronization of multiple audio devices based on a distributed clock synchronization mechanism and performing clock synchronization of a master audio device and a slave audio device based on a PTP protocol; and the channel synchronization control strategy is a synchronization strategy for adjusting the bandwidth resources of each channel according to a priority allocation strategy of broadband resources based on the clock deviation of each channel and sending the audio coding sequence data packet to the corresponding slave audio device for playback based on the adjusted bandwidth.

13. An audio device, characterized in that: The audio device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the audio device to execute the multi-room multi-channel audio synchronization method according to any one of claims 1 to 11.

14. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by a processor, the multi-room multi-channel audio synchronization method according to any one of claims 1 to 11 is implemented.

Images