A method and device for synchronizing audio signals

By acquiring the reference signal of the device, determining the sampling time deviation and synchronizing the audio signals of multiple voice interactive devices, the problem of inconsistent sampling time of the device is solved, and the functional accuracy and cost-effectiveness of the voice interactive device are improved.

CN115426067BActive Publication Date: 2024-11-22SHENZHEN LINGZHI TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211064410.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-01
Publication Date
2024-11-22
Estimated Expiration
2042-09-01

AI Technical Summary

Technical Problem

When multiple voice interactive devices collect user voices, the start sampling time cannot be accurately controlled, resulting in the audio signal being out of sync, which in turn affects the accuracy of functions such as sound source positioning.

Method used

By acquiring the reference signals of each device, a sampling time deviation is determined, and the audio signals of each device are synchronized according to the deviation, so that the audio signals are synchronized. The specific method includes obtaining the first audio signal and the first reference signal from the first device, obtaining the second audio signal and the second reference signal from the second device, determining the sampling time deviation using the synchronization of the reference signal, and then adjusting the audio signal to achieve the synchronous.

Benefits of technology

The audio signal synchronization of multiple voice interactive devices is realized, which reduces costs, avoids the needs of wiring and GPS receiving devices, and improves the accuracy of functions such as sound source positioning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115426067B_ABST
    Figure CN115426067B_ABST
Patent Text Reader

Abstract

The present application discloses an audio signal synchronization method and related devices. Taking a first device and a second device as an example, after a target voice is emitted, a first audio signal and a first reference signal for the target voice are obtained from the first device, and a second audio signal and a second reference signal for the target voice are obtained from the second device. According to the first reference signal and the second reference signal, a sampling time deviation is determined, and according to the sampling time deviation, the first audio signal and the second audio signal, the voice delay time of the first device and the second device for the target voice is determined, so that the audio signals of the first device and the second device are synchronized according to the voice delay time. Thus, a third-party signal that arrives at different audio devices and has the same arrival time is used as an alignment reference for the audio signal, so that based on the sampling time deviation between the third-party signals, the voice delay time between the audio signals is determined, thereby achieving synchronization between the audio signals.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of speech signal processing, and in particular to an audio signal synchronization method and related devices. Background Art

[0002] A microphone array is a system composed of a certain number of microphones that samples and filters the spatial characteristics of the sound field. Microphone arrays include centralized microphone arrays and distributed microphone arrays. As a specific form of microphone arrays, distributed microphone arrays can achieve better performance than centralized microphone arrays in many aspects due to their large spatial coverage. At the same time, they can also achieve functions that centralized microphones do not have, such as sound source localization.

[0003] With the popularization of smart devices in recent years, more and more devices are equipped with microphones, such as televisions, air conditioners, refrigerators and other voice interaction devices. The microphones of these voice interaction devices provide the hardware conditions for the formation of distributed microphone arrays, making the implementation conditions of distributed microphone array related technologies more mature.

[0004] However, the time when each voice interaction device that makes up the distributed microphone array starts collecting the user's voice, that is, the start sampling time of the audio signal, will be different. It is impossible to accurately control multiple voice interaction devices to start collecting audio at the same time, which will cause the audio signals collected by multiple voice interaction devices to be out of sync, and then cause errors in functions such as sound source localization. Summary of the invention

[0005] In view of the above problems, the present application provides an audio signal synchronization method and related devices for solving the problem of audio signal synchronization of multiple voice interaction devices.

[0006] Based on this, the embodiments of the present application disclose the following technical solutions:

[0007] On the one hand, an embodiment of the present application provides an audio signal synchronization method, the method comprising:

[0008] Acquire a first audio signal and a first reference signal for a target voice from a first device;

[0009] Acquire a second audio signal and a second reference signal for the target speech from a second device, wherein the time when the first reference signal arrives at the first device is the same as the time when the second reference signal arrives at the second device;

[0010] Determining a sampling time deviation according to the first reference signal and the second reference signal;

[0011] Determining speech delay times of the first device and the second device for the target speech according to the sampling time deviation, the first audio signal, and the second audio signal;

[0012] The audio signal of the first device and the audio signal of the second device are synchronized according to the voice delay time.

[0013] Optionally, determining the speech delay time of the first device and the second device for the target speech according to the sampling time deviation, the first audio signal, and the second audio signal includes:

[0014] determining an audio time offset according to the first audio signal and the second audio signal;

[0015] According to the sampling time deviation and the audio time deviation, the speech delay time of the first device and the second device with respect to the target speech is determined.

[0016] Optionally, determining the speech delay time of the first device and the second device for the target speech according to the sampling time deviation, the first audio signal, and the second audio signal includes:

[0017] Adjust the first audio signal according to the sampling time deviation to obtain a first corrected audio signal;

[0018] According to the first modified audio signal and the second audio signal, speech delay times of the first device and the second device with respect to the target speech are determined.

[0019] Optionally, if the first reference signal and the second reference signal are power signals corresponding to AC power from the same transformer; determining the sampling time deviation according to the first reference signal and the second reference signal includes:

[0020] Filtering out periodic components in the first reference signal to obtain a first preprocessed signal, and filtering out periodic components in the second reference signal to obtain a second preprocessed signal;

[0021] Determine a first zero-crossing point set corresponding to the first preprocessed signal, and determine a second zero-crossing point set corresponding to the second preprocessed signal;

[0022] The sampling time deviation is determined according to the correlation between the first target zero-crossing point and each zero-crossing point in the second zero-crossing point set, and the first target zero-crossing point is any zero-crossing point in the first zero-crossing point set.

[0023] Optionally, the method further includes:

[0024] Acquire a first periodic signal from the first reference signal;

[0025] Determining the number of periods and the number of sampling points of the first periodic signal;

[0026] determining a sampling rate of the first device according to the number of periods and the number of sampling points of the first periodic signal; and / or,

[0027] Acquire a second periodic signal from the second reference signal;

[0028] Determining the number of periods and the number of sampling points of the second periodic signal;

[0029] The sampling rate of the second device is determined according to the number of periods and the number of sampling points of the second periodic signal.

[0030] Optionally, the first reference signal and the second reference signal are power signals corresponding to AC power from the same transformer; or,

[0031] The first reference signal and the second reference signal are both station signals from the same station.

[0032] Optionally, if the first reference signal and the second reference signal are power signals corresponding to AC power from the same transformer, the method further includes:

[0033] Acquire a first power signal, and acquire a second power signal;

[0034] According to the first power signal, obtain a first high-frequency signal through a second-order high-pass filter, and sample the first high-frequency signal to obtain the first reference signal;

[0035] According to the second power signal, a second high-frequency signal is obtained through the second-order high-pass filter, and the second high-frequency signal is sampled to obtain the second reference signal.

[0036] On the other hand, the present application provides an audio signal synchronization device, characterized in that the device includes: a first acquisition unit, a second acquisition unit, a first determination unit, a second determination unit and a synchronization unit;

[0037] The first acquisition unit is used to acquire a first audio signal and a first reference signal for a target voice from a first device;

[0038] The second acquisition unit is used to acquire a second audio signal and a second reference signal for the target speech from a second device, wherein the time when the first reference signal arrives at the first device is the same as the time when the second reference signal arrives at the second device;

[0039] The first determining unit is used to determine a sampling time deviation according to the first reference signal and the second reference signal;

[0040] The second determining unit is used to determine the speech delay time of the first device and the second device for the target speech according to the sampling time deviation, the first audio signal and the second audio signal;

[0041] The synchronization unit is used to synchronize the audio signal of the first device and the audio signal of the second device according to the voice delay time.

[0042] Optionally, the second determining unit is specifically configured to:

[0043] determining an audio time offset according to the first audio signal and the second audio signal;

[0044] According to the sampling time deviation and the audio time deviation, the speech delay time of the first device and the second device with respect to the target speech is determined.

[0045] Optionally, the second determining unit is specifically configured to:

[0046] Adjust the first audio signal according to the sampling time deviation to obtain a first corrected audio signal;

[0047] According to the first modified audio signal and the second audio signal, speech delay times of the first device and the second device with respect to the target speech are determined.

[0048] Optionally, if the first reference signal and the second reference signal are power signals corresponding to AC power from the same transformer; the first determination unit is specifically configured to:

[0049] Filtering out periodic components in the first reference signal to obtain a first preprocessed signal, and filtering out periodic components in the second reference signal to obtain a second preprocessed signal;

[0050] Determine a first zero-crossing point set corresponding to the first preprocessed signal, and determine a second zero-crossing point set corresponding to the second preprocessed signal;

[0051] The sampling time deviation is determined according to the correlation between the first target zero-crossing point and each zero-crossing point in the second zero-crossing point set, and the first target zero-crossing point is any zero-crossing point in the first zero-crossing point set.

[0052] Optionally, the device further includes a third determining unit, configured to:

[0053] Acquire a first periodic signal from the first reference signal;

[0054] Determining the number of periods and the number of sampling points of the first periodic signal;

[0055] determining a sampling rate of the first device according to the number of periods and the number of sampling points of the first periodic signal; and / or,

[0056] Acquire a second periodic signal from the second reference signal;

[0057] Determining the number of periods and the number of sampling points of the second periodic signal;

[0058] The sampling rate of the second device is determined according to the number of periods and the number of sampling points of the second periodic signal.

[0059] Optionally, the first reference signal and the second reference signal are power signals corresponding to AC power from the same transformer; or,

[0060] The first reference signal and the second reference signal are both station signals from the same station.

[0061] Optionally, the device further comprises a preprocessing unit, configured to:

[0062] If the first reference signal and the second reference signal are power signals corresponding to the AC power from the same transformer, obtaining a first power signal and obtaining a second power signal;

[0063] According to the first power signal, obtain a first high-frequency signal through a second-order high-pass filter, and sample the first high-frequency signal to obtain the first reference signal;

[0064] According to the second power signal, a second high-frequency signal is obtained through the second-order high-pass filter, and the second high-frequency signal is sampled to obtain the second reference signal.

[0065] In another aspect, the present application provides a computer device, the device comprising a processor and a memory:

[0066] The memory is used to store program code and transmit the program code to the processor;

[0067] The processor is used to execute the method described in the above aspects according to the instructions in the program code.

[0068] On the other hand, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method described in the above aspects.

[0069] On the other hand, an embodiment of the present application provides a computer program device or a computer program, the computer program device or the computer program includes a computer instruction, the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method described in the above aspects.

[0070] The advantages of the above technical solution of this application are:

[0071] Take two of the multiple audio devices that need to be synchronized, namely the first device and the second device, as an example. After the target voice is emitted, the first audio signal and the first reference signal for the target voice are obtained from the first device, and the second audio signal and the second reference signal for the target voice are obtained from the second device. Since the time when the first reference signal arrives at the first device is the same as the time when the second reference signal arrives at the second device, the first reference signal and the second reference signal can be used as alignment references for the audio signal. Specifically, the sampling time deviation is determined according to the first reference signal and the second reference signal, and the voice delay time of the first device and the second device for the target voice is determined according to the sampling time deviation, the first audio signal and the second audio signal, so as to synchronize the audio signals of the first device and the second device according to the voice delay time. Thus, the third-party signals that arrive at different audio devices respectively and have the same arrival time are used as the alignment reference of the audio signal, so as to determine the voice delay time between the audio signals based on the sampling time deviation between the third-party signals, and then achieve synchronization between the audio signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0073] Figure 1 A flowchart of an audio signal synchronization method provided in an embodiment of the present application;

[0074] Figure 2 A reference signal provided in an embodiment of the present application and a spectrum diagram thereof after filtering out steady-state components;

[0075] Figure 3 A schematic diagram of a reference signal and its corresponding periodic signal provided in an embodiment of the present application;

[0076] Figure 4 A frequency response diagram of a filter provided in an embodiment of the present application;

[0077] Figure 5 A circuit diagram of a filter provided in an embodiment of the present application;

[0078] Figure 6 A schematic diagram of a scenario of an audio signal synchronization method provided in an embodiment of the present application;

[0079] Figure 7 A schematic diagram of a scenario of an audio signal synchronization method provided in an embodiment of the present application;

[0080] Figure 8 A flowchart of an audio signal synchronization method provided in an embodiment of the present application;

[0081] Fig. 9 A structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0083] With the continuous development of artificial intelligence technology, direct voice communication between people and machines has brought great convenience to people's daily work and life, and voice interaction devices have emerged in an endless stream. For example, televisions, air conditioners, speakers and other devices all have voice interaction functions. Users can wake up voice interaction devices through voice keywords and control voice interaction devices through voice commands. However, if there are multiple voice interaction devices with the same wake-up word in a space, there will be a phenomenon of one call and a hundred responses. For example, there are three air conditioners of the same brand in the home. When the user wakes up the air conditioner through "XX (the wake-up keyword of the brand)", which air conditioner will respond to the user? For example, in order to avoid the problem of one call and a hundred responses, the voice interaction device closest to the user can be made to respond to the user.

[0084] In order to determine the voice interaction device closest to the user, it can be determined by the time when the sound is received. However, since the time when each voice interaction device that constitutes the distributed microphone array starts to collect the user's voice, that is, the start sampling time of the audio signal will be different, it is impossible to accurately control multiple voice interaction devices to start collecting audio at the same time, resulting in the audio signals collected by multiple voice interaction devices being out of sync, which in turn causes errors in functions such as sound source positioning, etc. In related technologies, the following two methods are used to synchronize audio signals.

[0085] Method 1: Use a separate wiring method to transmit a synchronous clock for the required voice interaction devices on an additionally arranged line. All voice interaction devices use the synchronous clock for internal timing and correct their own real-time communication (RTC) clock. After each voice interaction device receives the user's voice command, it uploads the voice command with a timestamp to the cloud server. The cloud server can compare the order in which the voice commands are received and the distance between the voice interaction devices and the user. However, this method not only has a high wiring cost, but also has a relatively fixed relationship between the line and the voice interaction device. The installation is not flexible and it is not convenient to add voice interaction devices to the line.

[0086] Method 2: Through the Global Positioning System (GPS) timing, multiple voice interaction devices adjust their own RTC clocks to timestamp the user's voice commands, so that the cloud server can determine the distance between the voice interaction device and the user. However, this method requires that each voice interaction device be equipped with a GPS receiver, which is costly.

[0087] Based on this, an embodiment of the present application provides an audio signal synchronization method. After the target voice is emitted, a first audio signal and a first reference signal for the target voice are obtained from a first device, and a second audio signal and a second reference signal for the target voice are obtained from a second device. Since the time when the first reference signal arrives at the first device is the same as the time when the second reference signal arrives at the second device, the first reference signal and the second reference signal can be used as an alignment reference for the audio signal. Specifically, according to the first reference signal and the second reference signal, a sampling time deviation is determined, and according to the sampling time deviation, the first audio signal and the second audio signal, the voice delay time of the first device and the second device for the target voice is determined, so that the audio signals of the first device and the second device are synchronized according to the voice delay time. Thus, a third-party signal that arrives at different audio devices and has the same arrival time is used as an alignment reference for the audio signal, so that based on the sampling time deviation between the third-party signals, the voice delay time between the audio signals is determined, and then the synchronization between the audio signals is achieved. In addition, there is no need for separate wiring or additional installation of a GPS receiving device, which reduces costs and is more flexible to install.

[0088] Combine the following Figure 1 , an audio signal synchronization method provided by an embodiment of the present application is introduced. Figure 1 , this figure is a flowchart of an audio signal synchronization method provided in an embodiment of the present application, and the method may include S101-S105.

[0089] S101: Acquire a first audio signal and a first reference signal for a target voice from a first device.

[0090] S102: Acquire a second audio signal and a second reference signal for the target speech from a second device.

[0091] The embodiment of the present application does not specifically limit the order in which S101 and S102 are executed. They may be executed one after the other or simultaneously. Those skilled in the art may make arrangements according to actual needs.

[0092] The first device and the second device are devices in the same space that need to synchronize audio signals, and both have devices such as microphones that can collect audio signals. For example, the first device and the second device can be two televisions that both have microphones.

[0093] The target voice is the voice sent to the first device and the second device. For example, the first device and the second device can be devices with the same wake-up word, so that after the user sends the target voice including the wake-up word, the first device and the second device collect the target voice including the wake-up word, and then synchronize the audio signal for the target voice, so that the voice interaction device that responds to the target voice can be determined according to business needs, etc.

[0094] The embodiment of the present application does not specifically limit the length of the target voice. For example, the target voice can be the entire voice uttered by the user, or it can be a portion of the entire voice that only includes key voice commands such as wake-up words. As a possible implementation method, the target voice can be a voice segment with a high signal-to-noise ratio or a large signal amplitude, so that the first device and the second device can collect the target voice in time to improve the user's interactive experience.

[0095] After the target voice is sent to the first device and the second device, the first device collects the first audio signal for the target voice, and the second device collects the second audio signal for the target voice. For example, the target voice is recorded through their respective microphones. Since the first device and the second device start collecting audio signals at different times, in order to synchronize the audio signals later, the first device can collect the first reference signal while collecting the first audio signal. Similarly, the second device can collect the second reference signal while collecting the second audio signal.

[0096] The time when the first reference signal arrives at the first device is the same as the time when the second reference signal arrives at the second device. This application does not make any specific limitation to this, and those skilled in the art can set it according to actual needs.

[0097] For example, the first reference signal and the second reference signal are power signals, and in order for the first reference signal and the second reference signal to reach their respective corresponding devices at the same time, the first reference signal and the second reference signal should be AC ​​power from the same transformer, that is, the first device and the second device are both connected to the power supply network of the same transformer. Since the power signal is transmitted very quickly on the line, it can be considered that the time when the first reference signal arrives at the first device is the same as the time when the second reference signal arrives at the second device.

[0098] For another example, the first reference signal and the second reference signal are radio signals. It should be noted that at this time, both the first device and the second device include a device capable of receiving radio signals, such as a frequency modulation (FM) receiver, other shortwave radios, etc. Because the first device and the second device are connected to the network, it is easy for the host computer or cloud server to issue instructions so that the devices capable of receiving radio signals included in the first device and the second device can simultaneously receive the same radio signal. Since the speed of radio wave transmission is very fast, it can be considered that the time when the first reference signal arrives at the first device is the same as the time when the second reference signal arrives at the second device. It should be noted that the cost of installing a device capable of receiving radio signals in a voice interaction device is lower than the cost of installing a GPS receiving device, so the cost of the equipment can be reduced while synchronizing the audio signal.

[0099] As a possible implementation, in order to ensure that the reference signal is of good quality and not a poor signal or a signal without sound, the host computer or cloud server can send a music or news radio signal with a high signal-to-noise ratio to the first device and the second device.

[0100] S103: Determine a sampling time deviation according to the first reference signal and the second reference signal.

[0101] Since the time when the first reference signal arrives at the first device is the same as the time when the second reference signal arrives at the second device, the first reference signal and the second reference signal can be used as alignment references for the audio signal, thereby determining the speech delay time between the first audio signal and the second audio signal based on the time deviation between the first reference signal and the second reference signal, that is, the sampling time deviation.

[0102] The embodiment of the present application does not specifically limit the method for determining the sampling time deviation. The following is an explanation using an embodiment as an example, see A1-A3.

[0103] A1: Filter out the periodic components in the first reference signal to obtain a first preprocessed signal, and filter out the periodic components in the second reference signal to obtain a second preprocessed signal.

[0104] The periodic components in the reference signal can be filtered out by a linear prediction filter, etc. to obtain the preprocessed signal. Specifically, the periodic components in the first reference signal are filtered out to obtain the first preprocessed signal, and the periodic components in the second reference signal are filtered out to obtain the second preprocessed signal.

[0105] The following description will be made by taking the reference signal as an electric power signal as an example.

[0106] A linear prediction filter can be used to filter out periodic components, which include the power frequency and its harmonics of various orders. The residue of the linear prediction is the non-periodic signal. The linear filter can use a normalized least mean square (NLMS) algorithm or a conventional filter algorithm such as the least squares (RLS) algorithm.

[0107] See also Figure 2 , which is a spectrum diagram of a reference signal provided by an embodiment of the present application and after filtering out the steady-state components. Figure 2 In the figure, the upper signal is the reference signal, and the lower signal is the preprocessed signal. It can be seen that the preprocessed signal includes obvious non-stationary components.

[0108] A2: Filter out the non-periodic components in the first reference signal to obtain a third preprocessed signal, and filter out the non-periodic components in the second reference signal to obtain a fourth preprocessed signal.

[0109] A3: determining a first zero-crossing point set according to the third preprocessing signal, and determining a second zero-crossing point set according to the fourth preprocessing signal;

[0110] The zero-crossing point set includes at least one zero-crossing point.

[0111] The embodiment of the present application does not specifically limit the method of zero crossing, for example, a first periodic signal is obtained from a first reference signal, and a second periodic signal is obtained from a second reference signal. A first zero crossing set is determined according to the first periodic signal, and a second zero crossing set is determined according to the second periodic signal. Wherein, there is a periodic part in the reference signal, and the periodic part can be extracted, and the period of the reference signal is determined by the zero crossing, thereby determining the sampling time deviation between the first reference signal and the second reference signal.

[0112] Taking the power signal as an example, the main components of the power signal are the 50Hz component and its frequency multiple signals, among which the 50Hz component has the strongest energy. The 50Hz component can be extracted from the power signal by conventional narrowband filters, that is, the first periodic signal is obtained from the first reference signal. Similarly, the second periodic signal is obtained from the second reference signal.

[0113] See also Figure 3, which is a schematic diagram of a reference signal and its corresponding periodic signal provided in an embodiment of the present application. Figure 3 In the figure, the upper signal is the reference signal, and the lower signal is the periodic signal. The intersection of the periodic signal and the horizontal axis is the zero crossing point. Figure 3 The periodic signal below includes 13 zero-crossing points, and thus a zero-crossing point set including the 13 zero-crossing points can be determined.

[0114] A4: Align the first preprocessed signal and the second preprocessed signal according to the first zero-crossing point set and the second zero-crossing point set.

[0115] If correlation analysis is performed on two non-periodic signals, one of the signals is slid forward and backward, and the length of time before and after is the possible range of time deviation, such as 500ms. Theoretically, each time a sampling point (sample) is slid, a pair of reference signals is formed, and a correlation can be calculated. By calculating all the correlations, the maximum correlation value can be found. The sliding time corresponding to this correlation value is the time difference of the reference signal. It should be noted that if the first reference signal and the second reference signal are both radio signals from the same radio station, this method can be used to determine the sampling time deviation.

[0116] However, after research, it is found that the above method has a large amount of calculation. Based on this, if the first reference signal and the second reference signal are power signals corresponding to the AC power from the same transformer, the embodiment of the present application makes improvements on this.

[0117] Because the zero crossing points of the alternating current of the two signals must be synchronized, obviously, when the maximum correlation is found, the periodic signals of the two signals are also synchronized, that is, the zero crossing points are aligned. Therefore, there is no need to slide according to such a fine granularity as the sample. First, align the two separated non-periodic signals according to the zero crossing points of the periodic signal, and then slide in steps according to the period of the periodic signal, which can greatly reduce the amount of calculation.

[0118] A5: Determine the sampling time deviation according to the correlation between the first target zero-crossing point and each zero-crossing point in the second zero-crossing point set.

[0119] According to the first preprocessed signal and the second preprocessed signal, the sampling time deviation is determined by the method of maximum cross-correlation, that is, different preprocessed signals are delayed according to the position of the zero-crossing point and then the cross-correlation is calculated, and the delay corresponding to the maximum cross-correlation is taken as the sampling time deviation. Specifically, for the first device and the second device, the delay of the first device is fixed to one of the zero-crossing points, that is, the first target zero-crossing point. Among them, the first target zero-crossing point is any zero-crossing point in the first zero-crossing point set. The positions of the different zero-crossing points of the second preprocessed signal of the second device are delayed, and then the cross-correlation is calculated to take the maximum value. The specific formula is as follows:

[0120]

[0121] Among them, t_m1_p1 is the first target zero crossing point in the first zero crossing point set corresponding to the first device, t_m2_p is each zero crossing point in the second zero crossing point set corresponding to the second device, y_m1 is the first preprocessed signal corresponding to the first device, y_m2 is the second preprocessed signal corresponding to the second device, n is the sampling point sequence number, and T is the total number of sampling points.

[0122] S104: Determine speech delay times of the first device and the second device for the target speech according to the sampling time deviation, the first audio signal and the second audio signal.

[0123] The speech delay time between the first audio signal and the second audio signal includes the time deviation for starting sampling. Therefore, after determining the sampling time deviation between the first reference signal and the second reference signal, the sampling time deviation can be used as a reference for starting the speech delay time. Based on the sampling time deviation, the first audio signal and the second audio signal, the speech delay time of the first device and the second device for the target speech can be determined.

[0124] The embodiments of the present application do not specifically limit the method for determining the voice delay time, and two methods are used as examples for illustration below.

[0125] Method 1: determining an audio time deviation according to the first audio signal and the second audio signal; determining a speech delay time of the first device and the second device for the target speech according to the sampling time deviation and the audio time deviation.

[0126] The audio time deviation between the first audio signal S1 and the second audio signal S2 is dTVn, and the sampling time deviation between the first reference signal V1 and the second reference signal V2 is dTVn. Due to the existence of the start sampling time deviation, the audio time deviation dTVn includes the sampling time deviation dTVn, so the voice delay time dTn can be obtained according to dTn=dTVn-dTSn.

[0127] It should be noted that the deviation between signals can be determined by correlation calculation, and this application does not make any specific limitation on this.

[0128] Method 2: adjusting the first audio signal according to the sampling time deviation to obtain a first corrected audio signal; determining the speech delay time of the first device and the second device for the target speech according to the first corrected audio signal and the second audio signal.

[0129] For example, if it is determined through the first reference signal and the second reference signal that the first device is delayed by 1 second relative to the second device, the first 1 second after the start of the audio signal of the first device can be removed to obtain a first corrected audio signal, and then the correlation between the first corrected audio signal and the second audio signal can be calculated to obtain the voice delay time.

[0130] S105: Synchronize the audio signal of the first device and the audio signal of the second device according to the voice delay time.

[0131] After synchronizing the audio of the first device and the second device, the distance between the first device and the second device and the target voice can be determined, so that the device that responds to the target voice can be determined according to business needs. For example, the device closest to the target voice can be set as the answering device, thereby providing users with better answering response functions based on the distributed microphone array.

[0132] It can be seen from the above technical solution that, taking two of the multiple audio devices that need to be synchronized, namely the first device and the second device as an example, after the target voice is emitted, the first audio signal and the first reference signal for the target voice are obtained from the first device, and the second audio signal and the second reference signal for the target voice are obtained from the second device. Since the time when the first reference signal arrives at the first device is the same as the time when the second reference signal arrives at the second device, the first reference signal and the second reference signal can be used as alignment references for the audio signal. Specifically, according to the first reference signal and the second reference signal, the sampling time deviation is determined, and according to the sampling time deviation, the first audio signal and the second audio signal, the voice delay time of the first device and the second device for the target voice is determined, so as to synchronize the audio signals of the first device and the second device according to the voice delay time. Thus, the third-party signals that arrive at different audio devices respectively and have the same arrival time are used as the alignment reference of the audio signal, so as to determine the voice delay time between the audio signals based on the sampling time deviation between the third-party signals, and then achieve synchronization between the audio signals.

[0133] As a possible implementation, if the first reference signal and the second reference signal are power signals corresponding to the AC power from the same transformer, after the power signals are collected, they can be transformed, filtered and sampled to obtain reference signals.

[0134] Voltage transformation is the process of converting strong electricity into weak electricity signals through transformers, that is, converting the power signal from 220V to 200mV.

[0135] Because the main function of AC power is to provide AC220V\50Hz power to the equipment in the power grid, the main spectrum of the power signal is 50Hz AC. For the convenience of subsequent analysis, noise close to Gaussian characteristics is required. These noises are caused by the different load characteristics of the equipment in the power grid. Therefore, in order to suppress the lower 50Hz power frequency, after collecting the power signal, the high-frequency signal is obtained by filtering. Therefore, filtering can achieve two functions: one is low-pass filtering to ensure that the analog-to-digital converter (ADC) does not produce spectrum aliasing. The second is to attenuate low frequencies to ensure the energy proportion of high frequencies.

[0136] See also Figure 4 , which is a frequency response diagram of a filter provided in an embodiment of the present application. Figure 4 In the figure, F0 is slightly higher than the power frequency (50Hz), F1 is equal to half of the ADC sampling frequency, and A is the response of the power frequency signal. In the actual system, multiple different filter circuits are cascaded to obtain the result. Figure 5 , which is a circuit diagram of a filter provided in an embodiment of the present application. The first end of the first capacitor C1 is used to receive a signal, the second end of the first capacitor C1 is connected to the first end of the first inductor R1, the second end of the first inductor R1 is connected to the first end of the second capacitor C2, the second end of the second capacitor C2 is grounded, and the common end of the first inductor R1 and the second capacitor C2 is used to output a signal. As a possible implementation, the first capacitor C1 is 1 microfarad (uf), the first inductor R1 is 1K ohm (Ω), and the second capacitor C2 is 10uf.

[0137] For example, after collecting the first power signal, a second-order high-pass filter is used to obtain a first high-frequency signal. Among them, the cut-off frequency of the second-order high-pass filter is 500Hz, which can effectively suppress low-frequency power frequency interference. After collecting the second power signal, a second-order high-pass filter is used to obtain a second high-frequency signal. It should be noted that the first power signal and the second power signal can use the same filter or different filters, and this application does not make specific limitations on this.

[0138] The aforementioned power signal and high-frequency signal are both analog voltage signals, and for subsequent analysis, they can also be converted into digital voltage signals by sampling. Specifically, the high-frequency signal is sampled and converted into a digital voltage signal, i.e., a reference signal. For example, the first high-frequency signal is sampled to obtain a first reference signal, and the second high-frequency signal is sampled to obtain a second reference signal.

[0139] In order to make the technical solution provided by the embodiments of the present application clearer, Figure 6 and Figure 7The audio signal synchronization method provided in the embodiments of the present application is described with two examples.

[0140] Example 1: Wired solution.

[0141] See also Figure 6 , which is a scene diagram of an audio signal synchronization method provided in an embodiment of the present application. Figure 6 In the example, n=3 is used to indicate that the user issues voice commands to three air conditioners.

[0142] After the user speaks, the voice interaction devices in the home with voice interaction function can record two audios synchronously, one is the audio signal Sn collected from the air, and the other is the reference signal Vn of AC noise, where n represents different device numbers.

[0143] From the principles of physics, we know that Sn are highly correlated voice signals, but they are not synchronized in time due to the difference in sound path. However, Vn signals are also highly correlated signals because they are from the public power grid, and it can be seen from the above that the signals are synchronized during acquisition. For example, the time when the first reference signal arrives at the first device is the same as the time when the second reference signal arrives at the second device.

[0144] 1. After each voice interaction device is turned on, it enters a standby state waiting for voice signals;

[0145] 2. The user issues a voice command;

[0146] 3. Each voice interaction device collects voice commands and confirms whether it is a valid voice command by waking up and recognizing the engine;

[0147] 4. If it is not a valid voice command, return to state 1 and continue waiting;

[0148] 5. If it is a valid voice command, upload the complete or partial recording of the voice command. The recording includes the audio signal Sn and the synchronously recorded reference signal. In this case, the reference signal is the power signal Vn. The upload object is the cloud server or the host computer.

[0149] 6. Collect the audio signal Sn and power signal Vn uploaded by all voice interaction devices in the processor of the cloud server or host computer;

[0150] 7. First, assume that the recording of a voice interaction device is a reference signal, the audio signal is S1, the power signal is V1, and the Vn and V1 of other devices are correlated, and the time deviation dTVn is found according to the peak value of the correlation. In the same way, Sn and S1 are correlated, and the time deviation dTSn is found according to the peak value of the correlation. For the relevant parts, refer to S101-S105 above.

[0151] 8. At this time, based on simple physical common sense, we can know that the voice delay time of voice interaction device n relative to voice interaction device 1 is: dTn=dTVn+dTSn, where n is the device number [2,∞);

[0152] 9. Based on the difference in dTn, the voice interaction device IoT service can decide the response strategy of the voice interaction device according to business needs, and complete the response function of the distributed microphone array with better subjective experience.

[0153] Example 2: Wireless solution.

[0154] See also Figure 7 , which is a scene diagram of an audio signal synchronization method provided in an embodiment of the present application. Figure 7 In the example, n=3 is used to indicate that the user issues voice commands to three air conditioners.

[0155] 1. Each voice interaction device is initialized, including opening the settings for receiving the same FM channel;

[0156] 2. The user issues a voice command;

[0157] 3. Each voice interaction device collects the command and confirms whether it is a valid voice command by waking up and recognizing the engine;

[0158] 4. If it is not a valid voice command, return to state 1 and continue waiting;

[0159] 5. If it is a valid voice command, upload the complete or partial recording of the voice command. The recording includes the reference signal recorded synchronously with the voice signal Sn. In this case, the reference signal is the radio signal FMn. The upload object is the cloud server or the host computer.

[0160] 6. Collect the audio signals Sn and radio signals FMn uploaded by all voice interaction devices in the processor of the cloud server or host computer;

[0161] 7. First, assume that the recording of a voice interaction device is a reference signal, and its audio signal is S1, and the radio signal is FM1. The correlation between FMn and FM1 of other devices is calculated, and the time deviation dTFMn is found according to the peak value of the correlation. In the same way, Sn and S1 are correlated, and the time deviation dTSn is found according to the peak value of the correlation. For the relevant parts, refer to S101-S105 above.

[0162] 8. At this time, based on simple physical common sense, we can know that the voice delay time of voice interaction device n relative to voice interaction device 1 is: dTn=dTFMn+dTSn, where n is the device number [2,∞);

[0163] 9. Based on the difference in dTn, the voice interaction device IoT service can decide the response strategy of the voice interaction device according to business needs, and complete the response function of the distributed microphone array with better subjective experience.

[0164] In addition to the audio signal synchronization method provided in the embodiment of the present application, an audio signal synchronization device is also provided, such as Figure 8 As shown, it includes: a first acquisition unit 801, a second acquisition unit 802, a first determination unit 803, a second determination unit 804 and a synchronization unit 805;

[0165] The first acquisition unit 801 is used to acquire a first audio signal and a first reference signal for a target voice from a first device;

[0166] The second acquisition unit 802 is configured to acquire a second audio signal and a second reference signal for the target speech from a second device, wherein the time when the first reference signal arrives at the first device is the same as the time when the second reference signal arrives at the second device;

[0167] The first determining unit 803 is used to determine a sampling time deviation according to the first reference signal and the second reference signal;

[0168] The second determining unit 804 is used to determine the speech delay time of the first device and the second device for the target speech according to the sampling time deviation, the first audio signal and the second audio signal;

[0169] The synchronization unit 805 is configured to synchronize the audio signal of the first device and the audio signal of the second device according to the voice delay time.

[0170] As a possible implementation manner, the second determining unit 804 is specifically configured to:

[0171] determining an audio time offset according to the first audio signal and the second audio signal;

[0172] According to the sampling time deviation and the audio time deviation, the speech delay time of the first device and the second device with respect to the target speech is determined.

[0173] As a possible implementation manner, the second determining unit 804 is specifically configured to:

[0174] Adjust the first audio signal according to the sampling time deviation to obtain a first corrected audio signal;

[0175] According to the first modified audio signal and the second audio signal, speech delay times of the first device and the second device with respect to the target speech are determined.

[0176] As a possible implementation manner, the first determining unit 803 is specifically configured to:

[0177] Filtering out periodic components in the first reference signal to obtain a first preprocessed signal, and filtering out periodic components in the second reference signal to obtain a second preprocessed signal;

[0178] Determine a first zero-crossing point set corresponding to the first preprocessed signal, and determine a second zero-crossing point set corresponding to the second preprocessed signal;

[0179] The sampling time deviation is determined according to the correlation between the first target zero-crossing point and each zero-crossing point in the second zero-crossing point set, and the first target zero-crossing point is any zero-crossing point in the first zero-crossing point set.

[0180] As a possible implementation manner, the apparatus further includes a third determining unit, configured to:

[0181] Acquire a first periodic signal from the first reference signal;

[0182] Determining the number of periods and the number of sampling points of the first periodic signal;

[0183] determining a sampling rate of the first device according to the number of periods and the number of sampling points of the first periodic signal; and / or,

[0184] Acquire a second periodic signal from the second reference signal;

[0185] Determining the number of periods and the number of sampling points of the second periodic signal;

[0186] The sampling rate of the second device is determined according to the number of periods and the number of sampling points of the second periodic signal.

[0187] As a possible implementation manner, the first reference signal and the second reference signal are power signals corresponding to AC power from the same transformer; or,

[0188] The first reference signal and the second reference signal are both station signals from the same station.

[0189] As a possible implementation manner, the device further includes a preprocessing unit, which is used to:

[0190] If the first reference signal and the second reference signal are power signals corresponding to the AC power from the same transformer, obtaining a first power signal and obtaining a second power signal;

[0191] According to the first power signal, obtain a first high-frequency signal through a second-order high-pass filter, and sample the first high-frequency signal to obtain the first reference signal;

[0192] According to the second power signal, a second high-frequency signal is obtained through the second-order high-pass filter, and the second high-frequency signal is sampled to obtain the second reference signal.

[0193] It can be seen from the above technical solution that, taking two of the multiple audio devices that need to be synchronized, namely the first device and the second device as an example, after the target voice is emitted, the first audio signal and the first reference signal for the target voice are obtained from the first device, and the second audio signal and the second reference signal for the target voice are obtained from the second device. Since the time when the first reference signal arrives at the first device is the same as the time when the second reference signal arrives at the second device, the first reference signal and the second reference signal can be used as alignment references for the audio signal. Specifically, according to the first reference signal and the second reference signal, the sampling time deviation is determined, and according to the sampling time deviation, the first audio signal and the second audio signal, the voice delay time of the first device and the second device for the target voice is determined, so as to synchronize the audio signals of the first device and the second device according to the voice delay time. Thus, the third-party signals that arrive at different audio devices respectively and have the same arrival time are used as the alignment reference of the audio signal, so as to determine the voice delay time between the audio signals based on the sampling time deviation between the third-party signals, and then achieve synchronization between the audio signals.

[0194] The present application also provides a computer device, see Fig. 9 , which shows a structural diagram of a computer device provided in an embodiment of the present application, such as Fig. 9 As shown, the device includes a processor 910 and a memory 920:

[0195] The memory 910 is used to store program codes and transmit the program codes to the processor;

[0196] The processor 920 is configured to execute any one of the audio signal synchronization methods provided in the above embodiments according to the instructions in the program code.

[0197] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program is used to execute any one of the audio signal synchronization methods provided in the above embodiments.

[0198] The embodiment of the present application also provides a computer program device or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the audio signal synchronization method provided in various optional implementations of the above aspects.

[0199] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0200] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0201] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0202] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0203] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for synchronizing an audio signal, characterized in that: The method comprises: Acquire a first audio signal and a first reference signal for a target voice from a first device; Acquire a second audio signal and a second reference signal for the target voice from a second device, wherein the time when the first reference signal arrives at the first device is the same as the time when the second reference signal arrives at the second device; the target voice is the voice sent by the user to the first device and the second device; the first reference signal and the second reference signal are power signals corresponding to the AC power from the same transformer; or, the first reference signal and the second reference signal are both radio signals from the same radio station; Determining a sampling time deviation according to the first reference signal and the second reference signal includes: filtering out periodic components in the first reference signal to obtain a first preprocessed signal, and filtering out periodic components in the second reference signal to obtain a second preprocessed signal; filtering out non-periodic components in the first reference signal to obtain a third preprocessed signal, and filtering out non-periodic components in the second reference signal to obtain a fourth preprocessed signal; determining a first zero-crossing point set according to the third preprocessed signal, and determining a second zero-crossing point set according to the fourth preprocessed signal; aligning the first preprocessed signal and the second preprocessed signal according to the first zero-crossing point set and the second zero-crossing point set; If the first reference signal and the second reference signal are both radio station signals from the same radio station; the determining the sampling time deviation according to the first reference signal and the second reference signal further includes: performing a correlation analysis on the first reference signal and the second reference signal, calculating a correlation, determining a maximum correlation, and using a sliding time corresponding to the maximum correlation as the sampling time deviation; If the first reference signal and the second reference signal are power signals corresponding to the AC power from the same transformer; the determining of the sampling time deviation according to the first reference signal and the second reference signal further includes: determining the sampling time deviation according to the correlation between the first target zero crossing point and each zero crossing point in the second zero crossing point set, the first target zero crossing point being any zero crossing point in the first zero crossing point set; Determining speech delay times of the first device and the second device for the target speech according to the sampling time deviation, the first audio signal, and the second audio signal; The audio signal of the first device and the audio signal of the second device are synchronized according to the voice delay time.

2. The method according to claim 1, characterized in that The step of determining the speech delay time of the first device and the second device for the target speech according to the sampling time deviation, the first audio signal, and the second audio signal includes: determining an audio time offset according to the first audio signal and the second audio signal; According to the sampling time deviation and the audio time deviation, the speech delay time of the first device and the second device with respect to the target speech is determined.

3. The method according to claim 1, characterized in that: The step of determining the speech delay time of the first device and the second device for the target speech according to the sampling time deviation, the first audio signal, and the second audio signal includes: Adjust the first audio signal according to the sampling time deviation to obtain a first corrected audio signal; According to the first modified audio signal and the second audio signal, speech delay times of the first device and the second device with respect to the target speech are determined.

4. The method according to claim 1, characterized in that: If the first reference signal and the second reference signal are power signals corresponding to AC power from the same transformer, the method further includes: Acquire a first power signal, and acquire a second power signal; According to the first power signal, obtain a first high-frequency signal through a second-order high-pass filter, and sample the first high-frequency signal to obtain the first reference signal; According to the second power signal, a second high-frequency signal is obtained through the second-order high-pass filter, and the second high-frequency signal is sampled to obtain the second reference signal.

5. An audio signal synchronization device, characterized in that: The device comprises: a first acquisition unit, a second acquisition unit, a first determination unit, a second determination unit and a synchronization unit; The first acquisition unit is used to acquire a first audio signal and a first reference signal for a target voice from a first device; The second acquisition unit is used to acquire a second audio signal and a second reference signal for the target voice from a second device, wherein the time when the first reference signal arrives at the first device is the same as the time when the second reference signal arrives at the second device; the target voice is the voice sent by the user to the first device and the second device; the first reference signal and the second reference signal are power signals corresponding to the AC power from the same transformer; or, the first reference signal and the second reference signal are both radio signals from the same radio station; The first determining unit is used to determine a sampling time deviation according to the first reference signal and the second reference signal; The first determining unit is specifically configured to: Filtering out periodic components in the first reference signal to obtain a first preprocessed signal, and filtering out periodic components in the second reference signal to obtain a second preprocessed signal; Determine a first zero-crossing point set corresponding to the first preprocessed signal, and determine a second zero-crossing point set corresponding to the second preprocessed signal; If the first reference signal and the second reference signal are both radio station signals from the same radio station, the first determining unit is further configured to perform a correlation analysis on the first reference signal and the second reference signal, calculate a correlation, determine a maximum correlation, and use a sliding time corresponding to the maximum correlation as the sampling time deviation; If the first reference signal and the second reference signal are power signals corresponding to the AC power from the same transformer; the first determination unit is further used to determine the sampling time deviation according to the correlation between the first target zero crossing point and each zero crossing point in the second zero crossing point set, and the first target zero crossing point is any zero crossing point in the first zero crossing point set; The second determining unit is used to determine the speech delay time of the first device and the second device for the target speech according to the sampling time deviation, the first audio signal and the second audio signal; The synchronization unit is used to synchronize the audio signal of the first device and the audio signal of the second device according to the voice delay time.

6. A computer device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the method according to any one of claims 1 to 4 according to the instructions in the program code.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method according to any one of claims 1 to 4.

8. A computer program product, characterized in that The method comprises a computer program or an instruction; when the computer program or the instruction is executed by a processor, the method according to any one of claims 1 to 4 is executed.