An audio synchronization method and related apparatus

Abstract

The application discloses an audio synchronization method and a related device. The method is applied to a microphone collection system including a first collection end and a second collection end, and includes the following steps: obtaining a first frequency domain signal collected by the first collection end and a second frequency domain signal collected by the second collection end; obtaining phase angle information of the first frequency domain signal and the second frequency domain signal at a plurality of preset frequencies, determining a phase difference average and a time difference average between the first frequency domain signal and the second frequency domain signal based on the phase angle information; and adjusting a collection time delay of the first collection end and the second collection end based on the phase difference average and the time difference average. In this way, the application can save the calculation cost and improve the audio synchronization efficiency.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to an audio synchronization method and related apparatus. Background Technology

[0002] In the field of audio acquisition, the varying distances between different acquisition devices and the sound source in a microphone acquisition system lead to asynchrony in the audio data acquired by these devices, affecting the subsequent mixing of the audio data. Current methods for synchronizing audio data from different acquisition devices primarily involve frame-by-frame adjustment. This approach requires real-time processing of the acquired audio data, resulting in high computational resource consumption and low efficiency. Therefore, developing a more efficient audio synchronization method with lower computational resource consumption is a pressing issue. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide an audio synchronization method and related apparatus that can save computational costs and improve audio synchronization efficiency.

[0004] To address the aforementioned technical problems, this application provides an audio synchronization method. This method is applied to a microphone acquisition system comprising a first acquisition end and a second acquisition end. The method includes: acquiring a first frequency domain signal acquired by the first acquisition end and a second frequency domain signal acquired by the second acquisition end; acquiring phase angle information of the first and second frequency domain signals at multiple preset frequencies; determining the average phase difference and average time difference between the first and second frequency domain signals based on the phase angle information; and adjusting the acquisition delay of the first and second acquisition ends based on the average phase difference and the average time difference.

[0005] To address the aforementioned technical problems, another technical solution adopted in this application is: providing an audio synchronization device, which is applied to a microphone system including a first acquisition end and a second acquisition end, comprising: an acquisition module for acquiring a first frequency domain signal acquired by the first acquisition end and a second frequency domain signal acquired by the second acquisition end; a processing module for acquiring phase angle information of the first frequency domain signal and the second frequency domain signal at multiple preset frequencies, and determining the average phase difference and average time difference between the first frequency domain signal and the second frequency domain signal based on the phase angle information; and a delay module for adjusting the acquisition delay of the first acquisition end and the second acquisition end based on the average phase difference and the average time difference.

[0006] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide an electronic device, including a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the audio synchronization method as described in the above technical solution.

[0007] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium storing program data thereon, wherein the program data, when executed by a processor, implements the audio synchronization method as described in the above technical solution.

[0008] The beneficial effects of this application are as follows: Unlike existing technologies, this application proposes an audio synchronization method for a microphone acquisition system including a first acquisition end and a second acquisition end. By acquiring a first frequency domain signal acquired by the first acquisition end and a second frequency domain signal acquired by the second acquisition end, the phase angle information of the first and second frequency domain signals at multiple preset frequencies is determined. Based on the phase angle information, the average phase difference and average time difference between the first and second frequency domain signals are determined. Therefore, adjustments to subsequently acquired audio data are completed solely through the first and second acquisition ends, eliminating the need for alignment processing of each frame of data acquired by the first and second acquisition ends, thus improving audio synchronization efficiency and saving computational costs. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0010] Figure 1 This is a flowchart illustrating one implementation of the audio synchronization method of this application;

[0011] Figure 2 This is a flowchart illustrating step S102 corresponding to one embodiment.

[0012] Figure 3 This is a flowchart illustrating step S102 in another embodiment.

[0013] Figure 4 This is a flowchart illustrating step S103 corresponding to another embodiment.

[0014] Figure 5 This is a schematic diagram of the structure of one embodiment of the audio synchronization device of this application;

[0015] Figure 6This is a schematic diagram of the structure of one embodiment of the electronic device of this application;

[0016] Figure 7 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0018] Please see Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the audio synchronization method proposed in this application. The audio synchronization method proposed in this application is applied to a microphone acquisition system, which includes a first acquisition end, a second acquisition end, and a receiving end coupled to the first and second acquisition ends. The first and second acquisition ends are respectively used to acquire audio data and send the acquired audio data to the receiving end. The number of second acquisition ends can be one or more. Because the distances between the first and second acquisition ends and the sound source are different, the audio data acquired by the first acquisition end is not synchronized with the audio data acquired by the second acquisition end. To synchronize the audio data acquired by each second acquisition end with the audio data acquired by the first acquisition end, the following method, with the processor of the receiving end as the execution entity, specifically includes:

[0019] S101: Acquire the first frequency domain signal acquired by the first acquisition terminal and the second frequency domain signal acquired by the second acquisition terminal.

[0020] In one embodiment, step S101 includes: responding to the audio data collected by the first acquisition terminal and the second acquisition terminal respectively, sending the corresponding audio data to the receiving terminal, and causing the receiving terminal to process the received audio data to obtain a first frequency domain signal of the current frame audio data collected by the first acquisition terminal and a second frequency domain signal of the current frame audio data collected by the second acquisition terminal. The current frame audio data refers to the audio data collected by the corresponding acquisition terminal at the current moment, and the specific duration of each frame audio data can be set according to actual needs.

[0021] In one implementation scenario, after receiving the audio data of the current frame collected by the first acquisition terminal and the second acquisition terminal respectively, a fast Fourier transform is performed on the audio data of the current frame corresponding to the first acquisition terminal to obtain a first frequency domain signal; and a fast Fourier transform is performed on the audio data of the current frame corresponding to the second acquisition terminal to obtain a second frequency domain signal.

[0022] In a specific application scenario, to reduce spectral leakage, the Hanning window function is used to process the current frame audio data before performing a Fast Fourier Transform on the current frame data corresponding to the first and second acquisition terminals. The specific calculation formula is as follows:

[0023] w[n]=0.5*(1-cos2πn / (N-1))

[0024] x_windowed[n] = x[n] * w[n]

[0025] Where w[n] represents the Hanning window function, N represents the total length of the Hanning window function, x[n] represents the audio data of the current frame, and x_windowed[n] represents the processed audio data of the current frame.

[0026] Furthermore, a Fast Fourier Transform is performed on the processed current frame audio data to obtain a first frequency domain signal and a second frequency domain signal. The formulas for the first or second frequency domain signal are shown below:

[0027]

[0028] f = k * fs / N

[0029] Where X[k] represents the first frequency domain signal or the second frequency domain signal, and A[k] represents the amplitude at frequency k. The sampling rate is defined as follows: f represents the phase angle at frequency k; k represents the frequency index, with values ​​k = 0, 1, 2, ..., N-1; f represents the actual frequency corresponding to k; and fs represents the sampling rate. In this embodiment, the sampling rate is set to 48KHz. Of course, in other embodiments, the specific value of the sampling rate can be set according to actual needs, and this application does not limit it in this regard.

[0030] S102: Obtain the phase angle information of the first frequency domain signal and the second frequency domain signal at multiple preset frequencies, and determine the average phase difference and average time difference between the first frequency domain signal and the second frequency domain signal based on the phase angle information.

[0031] In one implementation, please refer to Figure 2 , Figure 2 This is a flowchart illustrating step S102 in one embodiment. When multiple preset frequencies correspond to multiple first frequencies, step S102 involves acquiring phase angle information of the first frequency domain signal and the second frequency domain signal at the multiple preset frequencies, and determining the average phase difference between the first frequency domain signal and the second frequency domain signal based on the phase angle information. The process includes:

[0032] S201: Obtain the first phase angle corresponding to each first frequency in the first frequency domain signal, and obtain the second phase angle corresponding to each first frequency in the second frequency domain signal.

[0033] In one implementation scenario, step S201 includes: in order to determine the phase deviation between the first frequency domain signal and the second frequency domain signal, a plurality of first frequencies are predetermined, and a first phase angle corresponding to each first frequency in the first frequency domain signal and a second phase angle corresponding to each first frequency in the second frequency domain signal are obtained.

[0034] In a specific application scenario, to improve data accuracy and save computing resources, ten first frequencies are set, including 200Hz, 400Hz, 600Hz...2kHz. This allows us to obtain the first phase angles corresponding to the ten different first frequencies in the first frequency domain signal, and the second phase angles corresponding to the ten different first frequencies in the second frequency domain signal.

[0035] Optionally, in other specific application scenarios, to further improve the accuracy of the subsequent average phase difference, the number of first frequencies is set to be greater than 10, for example, 20 first frequencies including 100Hz, 200Hz, 300Hz...2KHz; or, to reduce resource consumption, the number of first frequencies is set to be less than 10, for example, 5 first frequencies including 400Hz, 800Hz, 1200Hz, 1600Hz and 2KHz.

[0036] S202: Obtain the first difference between the first phase angle and the second phase angle corresponding to each first frequency.

[0037] In one implementation scenario, step S202 includes: in response to obtaining a first phase angle and a second phase angle corresponding to each first frequency, obtaining a first difference between the corresponding first phase angle and the second phase angle for each first frequency. The number of obtained first differences is consistent with the number of first frequencies.

[0038] S203: The average value of the first difference corresponding to all first frequencies is taken as the average phase difference between the first frequency domain signal and the second frequency domain signal.

[0039] In one implementation scenario, step S203 includes: averaging the absolute values ​​of the first differences corresponding to all first frequencies to obtain the average phase difference between the first frequency domain signal and the second frequency domain signal. This average phase difference is used to indicate whether the frequencies corresponding to the first frequency domain signal and the second frequency domain signal are synchronized, which helps to determine whether there is a deviation in the acquisition of audio data by the first acquisition end and the second acquisition end.

[0040] In another implementation scenario, step S203 may also include: selecting the maximum value among all the first differences as the average phase difference between the first frequency domain signal and the second frequency domain signal.

[0041] In another implementation, please refer to Figure 3 , Figure 3 This is a flowchart illustrating step S102 in another embodiment. When multiple preset frequencies correspond to multiple second frequencies, step S102 involves obtaining phase angle information of the first frequency domain signal and the second frequency domain signal at the multiple preset frequencies, and determining the average time difference between the first frequency domain signal and the second frequency domain signal based on the phase angle information. This process includes:

[0042] S301: Obtain the third phase angle corresponding to each second frequency in the first frequency domain signal, and obtain the fourth phase angle corresponding to each second frequency in the second frequency domain signal.

[0043] In one implementation scenario, step S301 includes: pre-determining multiple second frequencies, and obtaining the third phase angle at each second frequency in the first frequency domain signal, and the fourth phase angle at each second frequency in the second frequency domain signal. In this implementation scenario, to improve the accuracy of subsequent judgments regarding whether there is a deviation in the acquisition of audio data by the first and second acquisition terminals, 20 second frequencies, including 1kHz, 2kHz, 3kHz...20kHz, are set.

[0044] Of course, in other implementation scenarios, the number and specific value of the second frequency can also be set according to the actual situation.

[0045] S302: Obtain the second difference between the third phase angle and the fourth phase angle corresponding to each second frequency.

[0046] In one implementation scenario, step S302 includes: for each second frequency, subtracting the corresponding third phase angle from the corresponding fourth phase angle to obtain a second difference value between the third phase angle and the fourth phase angle corresponding to each second frequency. The number of second differences is the same as the number of second frequencies.

[0047] S303: Based on the second frequency and the corresponding second difference, obtain the time difference corresponding to each second frequency.

[0048] In one implementation scenario, step S303 includes: obtaining the time difference between the first frequency domain signal and the second frequency domain signal at the second frequency based on the second frequency and the corresponding second difference.

[0049] In a specific implementation scenario, the formula for obtaining the time difference corresponding to each second frequency is as follows:

[0050]

[0051] Where f1 represents the current second frequency, Δt represents the second difference corresponding to the current second frequency, and Δt represents the time difference corresponding to the current second frequency.

[0052] S304: Based on the time differences corresponding to all second frequencies, obtain the average time difference between the first frequency domain signal and the second frequency domain signal.

[0053] In one implementation scenario, step S304 includes: in response to obtaining the time difference corresponding to all second frequencies, obtaining the average value of all time differences as the average time difference between the first frequency domain signal and the second frequency domain signal.

[0054] In another implementation scenario, step S304 further includes: determining a preset weight for each second frequency based on the frequency range in which the second frequency is located. Multiple frequency ranges are matched with their own preset weights, and the preset weights are negatively correlated with the values ​​within the frequency range. The specific range of each frequency range and its corresponding preset weight can be set according to actual conditions.

[0055] Furthermore, for each second frequency, the first product of the second difference and a preset weight is obtained. Based on the first product and preset weights corresponding to all second frequencies, the mean time difference between the first frequency domain signal and the second frequency domain signal is obtained.

[0056] In a specific application scenario, the response to the preset second frequency includes 20 frequency data points: 1kHz, 2kHz, 3kHz...20kHz. The preset weights for the second frequencies greater than or equal to 1kHz and less than or equal to 8kHz are set to 3; the preset weights for the second frequencies greater than or equal to 9kHz and less than or equal to 16kHz are set to 2; and the preset weights for the second frequencies greater than or equal to 17kHz and less than or equal to 20kHz are set to 1. The first product of each second difference and its corresponding preset weight is obtained. The ratio of the sum of all first products to the sum of all preset weights is taken as the average time difference between the first and second frequency domain signals.

[0057] S103: Adjust the acquisition delay of the first and second acquisition terminals based on the average phase difference and the average time difference.

[0058] In one embodiment, step S103 includes: in response to obtaining the average phase difference and average time difference between the first frequency domain signal and the second frequency domain signal, determining whether the average phase difference is less than or equal to a preset threshold. If not, it is considered that the phase difference of the audio data acquired by the first acquisition end and the second acquisition end in the low-frequency band is not within the threshold range, that is, it is determined that the sound sources acquired by the first acquisition end and the second acquisition end are different sound sources, and there is no need to perform synchronization processing on the audio data acquired by the first acquisition end and the second acquisition end. If yes, it indicates that the phase difference of the audio data acquired by the first acquisition end and the second acquisition end in the low-frequency band is within the threshold range, that is, it is considered that the sound sources acquired by the first acquisition end and the second acquisition end are the same sound source. At this time, it is necessary to adjust the acquisition delay of the first acquisition end and / or the second acquisition end according to the average time difference to synchronize the audio data acquired by the first acquisition end and the second acquisition end. In this embodiment, the preset threshold is set to π / 9. In other embodiments, the preset threshold can also be set according to the actual situation.

[0059] In another embodiment, when the average phase difference is less than or equal to a preset threshold, multiple frames of reference audio data after the current frame are acquired by the first and second acquisition terminals respectively, and the average time difference corresponding to each frame of reference audio data is acquired. It is determined whether the number of frames of reference audio data with an average time difference greater than the time threshold is greater than the frame number threshold. If so, the audio data acquired by the first and second acquisition terminals is considered to be out of sync, and the acquisition delay of the first and / or second acquisition terminals needs to be adjusted according to the average time difference. For example, when the average phase difference is less than or equal to the preset threshold, 100 frames of reference audio data acquired by the first and second acquisition terminals after the current frame are acquired, and 100 average time differences are determined based on these two sets of 100 frames of reference audio data. When the number of frames of reference audio data with an average time difference greater than 10 microseconds is greater than 95, the audio data acquired by the first and second acquisition terminals is considered to be out of sync. The specific process of acquiring the average time difference corresponding to the reference audio data can be referred to the corresponding embodiment described above, and will not be elaborated in detail here.

[0060] In one implementation scenario, when adjusting the acquisition latency of the first and / or second acquisition terminals based on the average time difference, if the average time difference is greater than 0, it indicates that the audio data acquired by the first acquisition terminal is faster. This allows the first acquisition terminal to delete portions of subsequently acquired audio data, or the second acquisition terminal to supplement subsequently acquired audio data, thus synchronizing the audio data acquired by the first and second acquisition terminals. If the average time difference is less than 0, it indicates that the audio data acquired by the first acquisition terminal is slower. This allows the first acquisition terminal to supplement subsequently acquired audio data, or the second acquisition terminal to delete portions of subsequently acquired audio data, thus synchronizing the audio data acquired by the first and second acquisition terminals.

[0061] In a specific application scenario, the first or second acquisition end deletes portions of subsequently acquired audio data, including deleting data of a first preset duration at the start position of the next acquired audio frame. The first or second acquisition end then supplements subsequently acquired audio data by adding predicted audio data of a first preset duration at the start position of the next acquired audio frame. The first preset duration is the same as the average time difference. By deleting or supplementing, the audio data acquired by the first and second acquisition ends is synchronized, avoiding real-time adjustments to the acquired audio data, reducing the difficulty of audio synchronization, and saving computational resources. Furthermore, the process of adding predicted audio data can be referred to as an audio augmentation method, which will not be elaborated upon in this application.

[0062] In another specific application scenario, enabling the first or second acquisition end to delete portions of subsequently acquired audio data can further include: uniformly deleting audio data within a preset number of frames acquired after the current frame, ensuring that the total duration of all deleted data within the preset number of frames is the same as the average time difference; for example, when the preset number of frames is set to 100 frames, for the 100 frames of audio data acquired after the current frame, data of a second preset duration at the start position of each frame of audio data is deleted. Enabling the first or second acquisition end to supplement subsequently acquired audio data can further include: uniformly supplementing audio data within a preset number of frames acquired after the current frame, ensuring that the total duration of all supplemented data within the preset number of frames is the same as the average time difference; for example, when the preset number of frames is 100, for the 100 frames of audio data acquired after the current frame, predicted audio data of a second preset duration is supplemented at the start position of each frame of audio data. This method improves the stability of the adjustment by deleting or adding audio data within a preset number of frames after the current frame, avoiding the simultaneous deletion or addition of a large amount of audio data. The second preset duration is the ratio of the average time difference to the preset number of frames.

[0063] Additionally, it should be noted that deletions or additions can be made at the beginning of each frame of audio data, or at the middle or end of each frame of audio data.

[0064] This application proposes an audio synchronization method for a microphone acquisition system including a first acquisition end and a second acquisition end. By acquiring a first frequency domain signal acquired by the first acquisition end and a second frequency domain signal acquired by the second acquisition end, the phase angle information of the first and second frequency domain signals at multiple preset frequencies is determined. Based on the phase angle information, the average phase difference and average time difference between the first and second frequency domain signals are determined. Therefore, adjustments to subsequently acquired audio data are completed solely through the first and second acquisition ends, eliminating the need for alignment processing of each frame of data acquired by the first and second acquisition ends, thus improving audio synchronization efficiency and saving computational costs.

[0065] In another implementation, please refer to Figure 4 , Figure 4 This is a flowchart illustrating step S103 corresponding to another implementation method. Specifically, step S103 includes:

[0066] S401: In response to the average phase difference being less than or equal to a preset threshold, determine the first target delay of the first acquisition end and the second target delay of the second acquisition end based on the average time difference.

[0067] In one embodiment, step S401 further includes: when the average phase difference is less than or equal to a preset threshold, i.e., the sound source acquired by the first acquisition end and the second acquisition end is the same sound source, acquiring the first initial delay of the first acquisition end and the second initial delay of the second acquisition end at the current moment, and determining the first target delay of the first acquisition end and the second target delay of the second acquisition end based on the first initial delay, the second initial delay and the average time difference. Specifically, when the first acquisition end and the second acquisition end are powered on and begin acquiring audio data, the corresponding first initial delay and second initial delay are both 0; and when the value of the first initial delay or the second initial delay increases, it is considered that the corresponding first acquisition end or the second acquisition end needs to slow down the acquisition of audio data, i.e., delete some data in the subsequently acquired audio data; when the value of the first initial delay or the second initial delay decreases, it is considered that the corresponding first acquisition end or the second acquisition end needs to speed up the acquisition of audio data, i.e., supplement the subsequently acquired audio data.

[0068] In one implementation scenario, in response to the mean time difference being greater than a first value and the second initial delay being greater than the mean time difference, a third difference between the second initial delay and the mean time difference is obtained, and this third difference is used as the second target delay, ensuring that the first target delay is consistent with the first initial delay. Alternatively, in response to the mean time difference being greater than a first value and the second initial delay being less than or equal to the mean time difference, a first sum of the first initial delay and the mean time difference is obtained, and this first sum is used as the first target delay, ensuring that the second target delay is consistent with the second initial delay.

[0069] Specifically, the first value mentioned above is 0. When the average time difference is greater than 0, it indicates that during subsequent audio data acquisition, a portion of the audio data acquired by the first acquisition end needs to be deleted, or the audio data acquired by the second acquisition end needs to be supplemented. In this case, if the second initial delay is greater than the average time difference, it is considered too large, and the second initial delay is preferentially reduced; that is, the third difference between the second initial delay and the average time difference is taken as the second target delay. If the second initial delay is less than or equal to the average time difference, it is considered too small, and the first initial delay is preferentially increased; that is, the first sum of the first initial delay and the average time difference is taken as the first target delay. In this implementation scenario, when the average time difference is greater than 0, based on the comparison result between the second initial delay and the average time difference, it is determined whether to adjust the first initial delay or the second initial delay, thereby avoiding multiple adjustments to the first or second initial delay during audio synchronization, which could lead to it being too large or too small, thus affecting the quality of the audio data acquired by the corresponding acquisition end.

[0070] In another implementation scenario, in response to the mean time difference being less than a first value and the first initial delay being greater than the absolute value of the mean time difference, a second sum of the first initial delay and the mean time difference is obtained, and this second sum is used as the first target delay, ensuring that the second target delay is consistent with the second initial delay. Alternatively, in response to the mean time difference being less than a first value and the first initial delay being less than or equal to the absolute value of the mean time difference, a fourth difference between the second initial delay and the mean time difference is obtained, and this fourth difference is used as the second target delay, ensuring that the first target delay is consistent with the first initial delay.

[0071] Specifically, when the first value is 0 and the average time difference is less than 0, if the first initial delay is greater than the absolute value of the average time difference, then the first initial delay is considered too large and the first initial delay is reduced firstly, that is, the second sum of the first initial delay and the average time difference is taken as the first target delay. If the first initial delay is less than or equal to the absolute value of the average time difference, then the first initial delay is considered too small and the second initial delay is increased firstly, that is, the fourth difference between the second initial delay and the average time difference is taken as the second target delay.

[0072] In another embodiment, step S401 may further include: when the average phase difference is less than or equal to a preset threshold, acquiring multiple frames of reference audio data after the current frame acquired by the first acquisition end and the second acquisition end respectively, and acquiring the average time difference corresponding to each frame of reference audio data respectively. Determining whether the number of frames of reference audio data with an average time difference greater than the time threshold is greater than the frame number threshold. If so, it is considered that the audio data acquired by the first acquisition end and the second acquisition end are out of sync, and a first target time delay of the first acquisition end and a second target time delay of the second acquisition end are determined. The specific process can be referred to the corresponding embodiment described above, and will not be elaborated in detail here.

[0073] S402: Adjust the audio data within a preset number of frames after the current moment acquired by the first acquisition terminal based on the first target time delay, and adjust the audio data within a preset number of frames after the current moment acquired by the second acquisition terminal based on the second target time delay.

[0074] In one implementation scenario, when the first target delay increases by a third preset duration compared to the first initial delay, the first acquisition end uniformly deletes audio data within a preset number of frames after the current acquisition time, ensuring that the total duration of all deleted data within the preset number of frames is the same as the third preset duration. Alternatively, when the second target delay decreases by a third preset duration compared to the second initial delay, the second acquisition end uniformly replenishes audio data within a preset number of frames after the current acquisition time, ensuring that the total duration of all replenished data within the preset number of frames is the same as the third preset duration. The specific implementation process can be referred to the corresponding implementation method described above.

[0075] In another implementation scenario, when the first target delay is reduced by a fourth preset duration compared to the first initial delay, the first acquisition end uniformly supplements the audio data within a preset number of frames after the current acquisition time, ensuring that the total duration of all supplemented data within the preset number of frames is the same as the fourth preset duration. Alternatively, when the second target delay is increased by a fourth preset duration compared to the second initial delay, the second acquisition end uniformly deletes the audio data within a preset number of frames after the current acquisition time, ensuring that the total duration of all deleted data within the preset number of frames is the same as the fourth preset duration.

[0076] It should be noted that after adjusting the audio data within a preset number of frames after the current moment based on the first target delay and the second target delay, the first target delay at this time is used as the first initial delay of the first acquisition end, and the second target delay is used as the second initial delay of the second acquisition end. The audio synchronization method proposed in this application is then re-executed based on the latest audio data acquired by the first acquisition end and the second acquisition end.

[0077] In another embodiment, in response to the microphone system including a receiver coupled to a first acquisition end and a second acquisition end, and the receiver having a channel mode including mono and multi-channel, the first and second acquisition ends can respectively perform audio processing on the acquired audio data when acquiring audio data. To avoid the impact on mixing effects caused by different audio processing methods corresponding to different acquisition ends, the audio synchronization method proposed in this application further includes:

[0078] Based on the channel mode of the receiving end, the audio processing methods of the first and second acquisition ends are determined. Specifically, when the channel mode is mono, the first and second acquisition ends use the same audio processing method; when the channel mode is multi-channel, the first and second acquisition ends use at least one audio processing method.

[0079] In one implementation scenario, when the channel mode is mono, the receiving end acquires audio data from both the first and second acquisition ends and mixes it. Since the first and second acquisition ends have various audio processing methods for the acquired audio data, such as original sound, light noise reduction, heavy noise reduction, and timbre enhancement, differences in their audio processing methods can easily affect the mixing effect. Therefore, when the first and second acquisition ends have different audio processing methods, and the channel mode is switched to mono, the audio processing method of the second acquisition end is switched to match that of the first acquisition end. Furthermore, when an instruction to adjust the audio processing method of the first acquisition end is received, both the first and second acquisition ends are simultaneously switched to the target audio processing method corresponding to that instruction.

[0080] In another implementation scenario, the receiver in mono mode is connected to the first acquisition end. If a connection command is received from the second acquisition end and the receiver, the audio processing mode of the second acquisition end is switched to the same mode as that of the first acquisition end. Alternatively, the receiver in non-mono mode is connected to the first acquisition end. If a connection command is received from the second acquisition end and the receiver, the audio processing mode of the second acquisition end is switched to the default mode. This default mode can be the original audio.

[0081] In another implementation scenario, when the receiver's channel mode is switched to mono, and the audio processing methods corresponding to the first and second acquisition ends are original sound and non-original sound, the audio processing method of the acquisition end with the original sound processing method is adjusted so that the audio processing method of the acquisition end is consistent with the audio processing method of the other acquisition end.

[0082] In another implementation scenario, when the receiver switches its channel mode to multi-channel and receives an instruction to adjust the audio processing methods of the first and second acquisition terminals, the audio processing methods of the first and second acquisition terminals are adjusted to the corresponding target audio processing methods, and the target audio processing methods of the first and second acquisition terminals can be different.

[0083] Please see Figure 5 , Figure 5 This is a schematic diagram of an embodiment of the audio synchronization device of this application. The audio, the audio synchronization device is applied to a microphone acquisition system including a first acquisition end and a second acquisition end, the device including an acquisition module 10, a processing module 20 and a delay module 30.

[0084] Specifically, the acquisition module 10 is used to acquire the first frequency domain signal acquired by the first acquisition terminal and the second frequency domain signal acquired by the second acquisition terminal.

[0085] Processing module 20 is used to acquire phase angle information of the first frequency domain signal and the second frequency domain signal at multiple preset frequencies, and to determine the average phase difference and average time difference between the first frequency domain signal and the second frequency domain signal based on the phase angle information.

[0086] The delay module 30 is used to adjust the acquisition delay of the first acquisition end and the second acquisition end based on the average phase difference and the average time difference.

[0087] In one implementation, please refer to... Figure 5The audio synchronization device proposed in this application includes a phase difference average calculation module 21 in its processing module 20. When multiple preset frequencies correspond to multiple first frequencies, the phase difference average calculation module 21 is used to obtain the first phase angle corresponding to each first frequency in the first frequency domain signal and the second phase angle corresponding to each first frequency in the second frequency domain signal; obtain the first difference between the first phase angle and the second phase angle corresponding to each first frequency; and take the average value of the first differences corresponding to all first frequencies as the phase difference average between the first frequency domain signal and the second frequency domain signal.

[0088] In another embodiment, please refer to [link / reference needed]. Figure 5 The processing module 20 in the audio synchronization device proposed in this application further includes a time difference average calculation module 22. When multiple preset frequencies correspond to multiple second frequencies, the time difference average calculation module 22 is used to obtain the third phase angle corresponding to each second frequency in the first frequency domain signal, and to obtain the fourth phase angle corresponding to each second frequency in the second frequency domain signal; to obtain the second difference between the third phase angle and the fourth phase angle corresponding to each second frequency; to obtain the time difference corresponding to each second frequency based on the second frequency and the corresponding second difference; and to obtain the average time difference between the first frequency domain signal and the second frequency domain signal based on the time differences corresponding to all second frequencies.

[0089] In one implementation scenario, the time difference mean calculation module 22 obtains the mean time difference between the first frequency domain signal and the second frequency domain signal based on the time differences corresponding to all second frequencies. This includes: determining a preset weight for each second frequency based on the frequency interval in which the second frequency is located; wherein multiple frequency intervals are matched with their own preset weights, and the preset weights are negatively correlated with the values ​​within the frequency intervals; for each second frequency, obtaining the first product of the second difference and the preset weight; and obtaining the mean time difference between the first frequency domain signal and the second frequency domain signal based on the first product and the preset weights corresponding to all second frequencies.

[0090] In one embodiment, the delay module 30 adjusts the acquisition delay of the first acquisition end and the second acquisition end based on the average phase difference and the average time difference, including: in response to the average phase difference being less than or equal to a preset threshold, determining a first target delay of the first acquisition end and a second target delay of the second acquisition end based on the average time difference; adjusting the audio data acquired by the first acquisition end within a preset number of frames after the current moment based on the first target delay, and adjusting the audio data acquired by the second acquisition end within a preset number of frames after the current moment based on the second target delay.

[0091] In one implementation scenario, the latency module 30 determines a first target latency of the first acquisition end and a second target latency of the second acquisition end based on the average time difference, including: acquiring the first initial latency of the first acquisition end and the second initial latency of the second acquisition end at the current moment; in response to the average time difference being greater than a first value and the second initial latency being greater than the average time difference, acquiring a third difference between the second initial latency and the average time difference, and using the third difference as the second target latency, so that the first target latency is consistent with the first initial latency; in response to the average time difference being greater than a first value and the second initial latency being less than or equal to the average time difference, acquiring the first initial latency and the average time difference... The first sum of the time difference averages is used as the first target delay, making the second target delay consistent with the second initial delay. In response to the time difference average being less than the first value and the first initial delay being greater than the absolute value of the time difference average, the second sum of the first initial delay and the time difference average is obtained, and the second sum is used as the first target delay, making the second target delay consistent with the second initial delay. In response to the time difference average being less than the first value and the first initial delay being less than or equal to the absolute value of the time difference average, the fourth difference between the second initial delay and the time difference average is obtained, and the fourth difference is used as the second target delay, making the first target delay consistent with the first initial delay.

[0092] In yet another implementation, please refer to [link / reference needed]. Figure 5 The audio synchronization device proposed in this application further includes an adjustment module 40 coupled to the delay module 30. In response to the microphone system, a receiver coupled to the first and second acquisition terminals is also included. The receiver's channel mode includes mono and multi-channel, and the first and second acquisition terminals have various audio processing methods. The adjustment module 40 is used to determine the audio processing method of the first and second acquisition terminals based on the channel mode of the receiver; wherein, when the channel mode is mono, the first and second acquisition terminals use the same audio processing method; and when the channel mode is multi-channel, the first and second acquisition terminals use at least one audio processing method.

[0093] Please see Figure 6 , Figure 6This is a schematic diagram of one embodiment of the electronic device of this application. The electronic device includes a memory 50 and a processor 60 coupled to each other. The memory 50 stores program data, and the processor 60 is used to call the program data to implement the audio synchronization method mentioned in any of the above embodiments. Specifically, the electronic device includes, but is not limited to, desktop computers, laptops, tablets, servers, etc., and is not limited thereto. In addition, the processor 60 may also be called a CPU (Center Processing Unit). The processor 60 may be an integrated circuit chip with signal processing capabilities. The processor 60 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor may be a microprocessor or any conventional processor. In addition, the processor 60 may be implemented by integrated circuit chips.

[0094] Please see Figure 7 , Figure 7 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 70 stores program data 80 that can be executed by a processor. When the program data 80 is executed by the processor, it implements the audio synchronization method mentioned in any of the above embodiments.

[0095] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0096] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0097] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0098] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An audio synchronization method, characterized in that, The method is applied to a microphone acquisition system including a first acquisition end and a second acquisition end, and the method includes: Acquire the first frequency domain signal acquired by the first acquisition terminal and the second frequency domain signal acquired by the second acquisition terminal; The phase angle information of the first frequency domain signal and the second frequency domain signal at multiple preset frequencies is obtained, and the average phase difference and average time difference between the first frequency domain signal and the second frequency domain signal are determined based on the phase angle information. Based on the average phase difference and the average time difference, the acquisition delay of the first acquisition end and the second acquisition end is adjusted; In response to the mean phase difference being less than or equal to a preset threshold, the first initial delay of the first acquisition terminal and the second initial delay of the second acquisition terminal at the current moment are obtained. Based on the first initial delay, the second initial delay, and the mean time difference, the first target delay of the first acquisition terminal and the second target delay of the second acquisition terminal are determined. When the mean time difference is greater than a first value, if the second initial delay is greater than the mean time difference, then the third difference between the second initial delay and the mean time difference is taken as the second target delay; otherwise, the first sum of the first initial delay and the mean time difference is taken as the first target delay. When the mean time difference is less than a first value, if the first initial delay is greater than the absolute value of the mean time difference, then the second sum of the first initial delay and the mean time difference is taken as the first target delay; otherwise, the fourth difference between the second initial delay and the mean time difference is taken as the second target delay. In each case, the other target delay is the initial delay of the corresponding acquisition terminal. Based on the first target delay, the audio data within a preset number of frames after the current moment acquired by the first acquisition terminal is adjusted, and based on the second target delay, the audio data within a preset number of frames after the current moment acquired by the second acquisition terminal is adjusted.

2. The method according to claim 1, characterized in that, Multiple preset frequencies correspond to multiple first frequencies. Phase angle information of the first frequency domain signal and the second frequency domain signal at the multiple preset frequencies is obtained. Based on the phase angle information, the average phase difference between the first frequency domain signal and the second frequency domain signal is determined, including: Obtain the first phase angle corresponding to each first frequency in the first frequency domain signal, and obtain the second phase angle corresponding to each first frequency in the second frequency domain signal; Obtain the first difference between the first phase angle and the second phase angle corresponding to each of the first frequencies; The average value of the first differences corresponding to all the first frequencies is taken as the average phase difference between the first frequency domain signal and the second frequency domain signal.

3. The method according to claim 1, characterized in that, Multiple preset frequencies correspond to multiple second frequencies. Phase angle information of the first frequency domain signal and the second frequency domain signal at the multiple preset frequencies is obtained. Based on the phase angle information, the average time difference between the first frequency domain signal and the second frequency domain signal is determined, including: Obtain the third phase angle corresponding to each second frequency in the first frequency domain signal, and obtain the fourth phase angle corresponding to each second frequency in the second frequency domain signal; Obtain the second difference between the third phase angle and the fourth phase angle corresponding to each of the second frequencies; Based on the second frequency and the corresponding second difference, the time difference corresponding to each second frequency is obtained; The mean time difference between the first frequency domain signal and the second frequency domain signal is obtained based on the time difference corresponding to all the second frequencies.

4. The method according to claim 3, characterized in that, The step of obtaining the average time difference between the first frequency domain signal and the second frequency domain signal based on the time differences corresponding to all the second frequencies includes: Based on the frequency range in which the second frequency is located, a preset weight is determined for each of the second frequencies; wherein, multiple frequency ranges are matched with their own preset weights, and the preset weights are negatively correlated with the values ​​within the frequency ranges; For each of the second frequencies, obtain the first product of the second difference and the preset weight; Based on the first product corresponding to all the second frequencies and the preset weight, the mean time difference between the first frequency domain signal and the second frequency domain signal is obtained.

5. The method according to any one of claims 1-4, characterized in that, The microphone acquisition system also includes a receiving end coupled to the first acquisition end and the second acquisition end. The channel mode of the receiving end includes mono and multi-channel. The first acquisition end and the second acquisition end have various audio processing methods. The method further includes: Based on the channel mode of the receiving end, the audio processing methods of the first acquisition end and the second acquisition end are determined; wherein, when the channel mode is mono, the first acquisition end and the second acquisition end use the same audio processing method, and when the channel mode is multi-channel, the first acquisition end and the second acquisition end use at least one audio processing method.

6. An audio synchronization device, characterized in that, The device is applied to a microphone acquisition system including a first acquisition end and a second acquisition end, comprising: The acquisition module is used to acquire the first frequency domain signal acquired by the first acquisition terminal and the second frequency domain signal acquired by the second acquisition terminal; The processing module is used to acquire phase angle information of the first frequency domain signal and the second frequency domain signal at multiple preset frequencies, and to determine the average phase difference and average time difference between the first frequency domain signal and the second frequency domain signal based on the phase angle information. The delay module is used to adjust the acquisition delay of the first acquisition terminal and the second acquisition terminal based on the average phase difference and the average time difference; In response to the mean phase difference being less than or equal to a preset threshold, the first initial delay of the first acquisition terminal and the second initial delay of the second acquisition terminal at the current moment are obtained. Based on the first initial delay, the second initial delay, and the mean time difference, the first target delay of the first acquisition terminal and the second target delay of the second acquisition terminal are determined. When the mean time difference is greater than a first value, if the second initial delay is greater than the mean time difference, then the third difference between the second initial delay and the mean time difference is taken as the second target delay; otherwise, the first sum of the first initial delay and the mean time difference is taken as the first target delay. When the mean time difference is less than a first value, if the first initial delay is greater than the absolute value of the mean time difference, then the second sum of the first initial delay and the mean time difference is taken as the first target delay; otherwise, the fourth difference between the second initial delay and the mean time difference is taken as the second target delay. In each case, the other target delay is the initial delay of the corresponding acquisition terminal. Based on the first target delay, the audio data within a preset number of frames after the current moment acquired by the first acquisition terminal is adjusted, and based on the second target delay, the audio data within a preset number of frames after the current moment acquired by the second acquisition terminal is adjusted.

7. An electronic device, characterized in that, include: A memory and a processor are coupled together, the memory storing program data, and the processor calling the program data to execute the audio synchronization method as described in any one of claims 1-5.

8. A computer-readable storage medium storing program data thereon, characterized in that, When the program data is executed by the processor, it implements the audio synchronization method as described in any one of claims 1-5.

Images