Sound synchronization method and system among multiple devices

Abstract

The invention provides a sound synchronization method and system among multiple devices, and belongs to the technical field of audio synchronization, and the method comprises the following steps: waveform capture: employing an audio collection device to collect audio waveforms output by each device; waveform identification: carrying out feature extraction on the collected waveforms, and carrying out waveform classification based on the extracted features; waveform matching: comparing the target waveform of the to-be-synchronized equipment with the reference waveform, and determining the time offset of the target waveform and the reference waveform; waveform coupling; and dynamically correcting. Synchronization is realized through five steps of waveform acquisition, identification, matching, coupling and dynamic correction: high-sensitivity waveform acquisition, feature extraction and machine learning classification, offset determination through cross-correlation algorithm matching, phase locking adjustment delay establishment, and dynamic correction in combination with environmental parameter monitoring and sound velocity prediction. The system comprises an audio acquisition, processing and control module, can eliminate sound transmission time difference, and ensures that audiences can obtain unified hearing feeling at any position.

Description

Technical Field

[0001] This invention relates to the field of audio synchronization technology, specifically to a method and system for audio synchronization between multiple devices. Background Technology

[0002] Existing technologies optimize the sound synchronization problem of multiple devices by adjusting the delay, but they have obvious limitations: the installation spacing of devices is not uniform in different project scenarios, and the sound propagation speed is affected by environmental factors such as air pressure and temperature, which will cause fluctuations, making it impossible to achieve a uniform sound perception. Listeners in different positions may perceive a time difference in the sound, affecting the listening experience.

[0003] To address this, a method and system for audio synchronization among multiple devices are proposed. Summary of the Invention

[0004] The present invention aims to solve the problems mentioned in the background art by providing a method and system for audio synchronization among multiple devices.

[0005] The specific technical solution is as follows:

[0006] A method for audio synchronization between multiple devices includes the following steps:

[0007] Waveform capture: Audio waveforms output by each device are captured using an audio acquisition device;

[0008] Waveform recognition: Extracting features from the acquired waveforms and classifying the waveforms based on the extracted features;

[0009] Waveform matching: Compare the target waveform of the device to be synchronized with the reference waveform to determine the time offset between the two;

[0010] Waveform coupling: Establish phase-locked relationships between devices based on time offset, adjust the output delay of each device, and enable waveforms to be superimposed at the target point in space;

[0011] Dynamic correction: Real-time monitoring of environmental parameters and audience positions, adjusting delay parameters based on monitoring results to optimize sound synchronization.

[0012] In the above-mentioned multi-device audio synchronization method, in the waveform capture step, a high-sensitivity microphone array is used as the audio acquisition device, and the sampling frequency of the microphone array is ≥96kHz, and the timestamp accuracy during acquisition is accurate to the microsecond level.

[0013] In the above-mentioned multi-device audio synchronization method, the waveform recognition step extracts waveform feature parameters including amplitude envelope, spectral features and zero crossings, and a machine learning algorithm is used in the classification process, and a waveform fingerprint database is established to assist in classification.

[0014] In the above-mentioned multi-device audio synchronization method, in the waveform matching step, the comparison is achieved by calculating the similarity between the reference waveform and the target waveform, and the matching window size can be dynamically adjusted within the range of 10-100ms during the comparison process, and the time offset is determined by a cross-correlation algorithm.

[0015] In the above-mentioned multi-device audio synchronization method, in the waveform coupling step, the output delay of each device is adjusted to an accuracy of ±10μs to ensure that the waveforms are accurately superimposed at the target point in space.

[0016] In the above-mentioned method for sound synchronization between multiple devices, the dynamic correction step involves real-time monitoring of environmental parameters including temperature, humidity, and air pressure. The method predicts changes in sound speed based on these environmental parameters and adjusts the delay parameters according to the changes in sound speed.

[0017] In the above-described method for audio synchronization between multiple devices, the dynamic correction step involves determining the focal area by tracking the listener's position, optimizing the audio output of each device for the focal area, and ensuring audio synchronization within the focal area.

[0018] The above-mentioned method for audio synchronization between multiple devices includes a host device (No. 1) and at least one slave device to be synchronized. When the slave device is in a waiting coupling state, it needs to undergo waveform recognition, waveform coupling and waveform matching before entering the working state to achieve audio synchronization with the host.

[0019] In the aforementioned multi-device audio synchronization method, when there are multiple spatial target points, waveform matching and waveform coupling are performed for different spatial target points respectively, so that the same audio content can reach each spatial target point simultaneously.

[0020] The present invention also provides a multi-device audio synchronization system for implementing the above method, including an audio acquisition module, a processing module, and a control module; the audio acquisition module is used to perform a waveform capture step, acquiring the audio waveforms output by each device; the processing module is used to perform waveform recognition and waveform matching steps, extracting features, classifying and comparing waveforms, and determining time offset; the control module is used to perform waveform coupling and dynamic correction steps, establishing phase locking relationships, adjusting device output delays, and optimizing synchronization effects according to the environment and listener positions.

[0021] In the aforementioned multi-device audio synchronization system, the audio acquisition module is a high-sensitivity microphone array, and the sampling frequency of the microphone array is ≥96kHz, with microsecond-level timestamp recording function.

[0022] In the aforementioned multi-device audio synchronization system, the processing module has a built-in machine learning model and a waveform fingerprint database. The machine learning model is used to classify waveform features, and the waveform fingerprint database is used to store waveform feature parameters to assist in classification and matching.

[0023] In the aforementioned multi-device audio synchronization system, the control module includes an environmental monitoring unit and a position tracking unit. The environmental monitoring unit is used to monitor environmental parameters such as temperature, humidity, and air pressure in real time, and the position tracking unit is used to track the position of the listeners to determine the focus area.

[0024] The present invention has the following beneficial effects:

[0025] 1. Eliminate time difference: Through a five-step processing flow, especially the phase-locked relationship established by waveform coupling and the dynamic correction to adapt to the environment, the time difference caused by the difference in sound propagation speed is effectively eliminated, so that the sound heard by the listener is uniform in any position.

[0026] 2. Adaptable to complex environments: The dynamic correction step monitors environmental parameters and predicts changes in sound speed to cope with the effects of factors such as air pressure and temperature on sound speed, ensuring stable synchronization under different environments.

[0027] 3. Precise multi-device collaboration: Slave devices must undergo strict waveform recognition, coupling, and matching before being integrated into the working state to ensure the collaboration between the host and slave devices and realize the synchronous superposition of sound from multiple devices at different spatial points.

[0028] 4. Optimize the auditory experience: By tracking the audience's location and optimizing the focus area, the sound output is precisely adjusted for the audience's activity area, improving auditory consistency in different locations and ensuring a unified listening experience. Attached Figure Description

[0029] Figure 1 A flowchart of a multi-device audio synchronization method provided in an embodiment of the present invention;

[0030] Figure 2 This is an example flowchart of a multi-device audio synchronization method provided in an embodiment of the present invention;

[0031] Figure 3 The coupling waveform diagram of multiple devices provided in the embodiments of the present invention. Detailed Implementation

[0032] The following is in conjunction with the appendix Figure 1-3 The technical solution of the present invention will be further illustrated through specific embodiments.

[0033] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual images. They should not be construed as limiting the scope of this application. To better illustrate the embodiments of the present invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0034] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "inner," and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present application. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0035] In the description of this invention, unless otherwise explicitly specified and limited, the term "connection" or similar designation indicating a connection between components should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral part; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication between two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0036] Example 1

[0037] Reference Figure 1-3 This embodiment provides a method for audio synchronization between multiple devices, including the following steps:

[0038] S1. Waveform capture: The audio waveforms output by each device are captured using an audio acquisition device;

[0039] S2. Waveform Recognition: Extract features from the acquired waveforms and classify the waveforms based on the extracted features;

[0040] S3. Waveform Matching: Compare the target waveform of the device to be synchronized with the reference waveform to determine the time offset between the two.

[0041] S4. Waveform Coupling: Establish phase-locked relationships between devices based on time offset, adjust the output delay of each device, and enable waveforms to be superimposed at the target point in space.

[0042] S5. Dynamic Correction: Real-time monitoring of environmental parameters and audience positions, adjusting delay parameters based on monitoring results to optimize sound synchronization.

[0043] The multi-device sound synchronization method using the above technical solution achieves sound synchronization between multiple devices through the synergistic effect of five steps: waveform capture, recognition, matching, coupling, and dynamic correction. This eliminates the time difference caused by differences in sound propagation speed, allowing listeners to obtain a uniform sound experience from any location.

[0044] Specifically, in this embodiment, a high-sensitivity microphone array is used as the audio acquisition device in the waveform capture step, and the sampling frequency of the microphone array is ≥96kHz, with the timestamp accuracy during acquisition down to the microsecond level. By using a high-sensitivity microphone array, combined with a high sampling rate and microsecond-level timestamp waveform capture method, the waveform details output by each device can be completely and accurately acquired, providing high-quality raw data for subsequent waveform processing and helping to improve the basic accuracy of multi-device sound synchronization.

[0045] Specifically, in this embodiment, the waveform recognition step extracts waveform feature parameters including amplitude envelope, spectral features, and zero-crossing points. Machine learning algorithms are employed during the classification process, and a waveform fingerprint database is established to assist in classification. By extracting multiple waveform feature parameters and combining them with machine learning algorithms and the waveform fingerprint database for classification, the waveforms output by different devices can be accurately identified. This provides a reliable waveform classification basis for subsequent waveform matching and coupling, ensuring the targeted and accurate nature of the synchronization processing.

[0046] The specific solutions for the machine learning algorithms are as follows:

[0047] A hybrid model combining an improved convolutional neural network (CNN) and a long short-term memory network (LSTM) is used to achieve waveform classification, specifically including:

[0048] 1. Feature preprocessing: The acquired audio waveform is converted into a Mel spectrogram, and features in the time dimension (100-500ms) and frequency dimension (20Hz-20kHz) are extracted. At the same time, time-domain features such as zero-crossing interval and amplitude envelope slope are retained to form a multi-dimensional feature matrix.

[0049] 2. Model Structure:

[0050] The input layer receives a multidimensional feature matrix and extracts local spectral features through three convolutional layers. Each layer is equipped with batch normalization and ReLU activation functions.

[0051] After the convolution result is reduced in dimensionality by the pooling layer, it is input into the bidirectional LSTM layer to capture temporal dependencies and output sequence features.

[0052] Finally, waveform classification (such as device type, audio content category, etc.) is achieved through a fully connected layer and a softmax activation function.

[0053] 3. Training mechanism: The cross-entropy loss function is adopted and the Adam optimizer is used. The training dataset contains more than 100,000 audio waveform samples from different devices and environments. The model parameters are optimized through 5-fold cross-validation to ensure a classification accuracy of ≥98%.

[0054] The specific scheme for the waveform fingerprint database is as follows:

[0055] The database uses a distributed key-value storage structure, and its core design is as follows:

[0056] 1. Storage Content: Each fingerprint record contains three parts:

[0057] Basic features: spectral peak position, amplitude envelope feature vector, zero-crossing timestamp sequence;

[0058] Derived features: hashed feature summary (for fast matching), device identifier (such as model number, factory parameters);

[0059] Metadata: Collection time, environmental tags (temperature, humidity range).

[0060] 2. Organization method: Classified by a secondary index of "Device type - audio scene", supporting multi-dimensional retrieval by feature vector similarity, device identifier and other dimensions.

[0061] 3. Update mechanism:

[0062] Real-time updates: When a new device is connected, its waveform fingerprint is automatically extracted and added to the database;

[0063] Iterative optimization: The fingerprint feature weights are retrained every quarter based on newly added samples (≥10,000) to improve matching robustness.

[0064] 4. Query process: After inputting the waveform features to be matched, candidate fingerprints are first quickly filtered by hash digest (time ≤10ms), and then the exact match is calculated by Euclidean distance, and the Top 5 similarity results are returned.

[0065] The specific scheme of the cross-correlation algorithm is as follows:

[0066] The steps for calculating the time offset between the reference waveform and the target waveform are as follows:

[0067] 1. Preprocessing: The two waveforms are subjected to DC component removal, normalization (amplitude range [-1,1]) and bandpass filtering (retaining the effective frequency band of 20Hz-20kHz) to eliminate environmental noise interference.

[0068] 2. Dynamic window settings:

[0069] The initial window size is set to 50ms. If the waveform change rate (variance of the amplitude difference between adjacent sampling points) is greater than or equal to the threshold (adaptive adjustment), the window will automatically shrink to 10ms to improve accuracy.

[0070] If the rate of change is less than the threshold, the window is expanded to 100ms to reduce the amount of computation.

[0071] 3. Cross-correlation calculation:

[0072] For the reference waveform x(t) and the target waveform y(t) within the window, calculate the cross-correlation function R(τ)=∫x(t)·y(t+τ)dt, where τ is the time offset;

[0073] Find the peak position of R(τ), and the corresponding τ is the time offset between the two waveforms.

[0074] 4. Error correction: The peak region is finely fitted by cubic spline interpolation, and the offset calculation error is controlled within 1 / 10 of the sampling period.

[0075] The above solutions work synergistically: the machine learning algorithm achieves accurate classification based on the waveform fingerprint database, the cross-correlation algorithm is combined with dynamic window optimization to calculate the time offset, and finally provides high-precision data support for multi-device waveform coupling, ensuring the stability and accuracy of sound synchronization.

[0076] Specifically, in this embodiment, the waveform matching step involves comparing the reference waveform and the target waveform by calculating their similarity. During the comparison, the matching window size can be dynamically adjusted within the range of 10-100ms, and a cross-correlation algorithm is used to determine the time offset. By calculating waveform similarity, dynamically adjusting the matching window size, and using a cross-correlation algorithm to determine the time offset, the time difference between the reference waveform and the target waveform can be accurately identified. This provides a precise basis for subsequent adjustments to the device output delay, which is beneficial for improving the accuracy of multi-device audio synchronization.

[0077] Specifically, in this embodiment, during the waveform coupling step, the output delay of each device is adjusted to an accuracy of ±10μs to ensure that the waveforms are accurately superimposed at the target point in space. Adjusting the output delay of each device with high precision during the waveform coupling step effectively establishes a phase-locked relationship between the devices, ensuring that the waveforms are accurately superimposed at the target point in space, enhancing the consistency of sound at the target point, and improving the synchronization effect in the listening experience.

[0078] Specifically, in this embodiment, the dynamic correction step involves real-time monitoring of environmental parameters including temperature, humidity, and air pressure. Sound speed changes are predicted based on these environmental parameters, and the delay parameters are adjusted accordingly. By monitoring environmental parameters and predicting sound speed changes to adjust the delay parameters, the dynamic correction step can address the impact of environmental changes on sound speed, enabling multi-device sound synchronization to adapt to different environmental conditions and maintain stable synchronization.

[0079] Specifically, in this embodiment, the dynamic correction step determines the focus area by tracking the listener's position, and optimizes the sound output of each device for the focus area to ensure sound synchronization within that area. By incorporating listener position tracking and focus area optimization into the dynamic correction, the sound output of each device can be optimized for the listener's focus area, ensuring better sound synchronization within the key areas of listener activity and enhancing the listener's actual listening experience.

[0080] Specifically, in this embodiment, the multiple devices include a host device (Device 1) and at least one slave device to be synchronized. While waiting for coupling, the slave device undergoes waveform recognition, waveform coupling, and waveform matching before entering the working state to achieve audio synchronization with the host. By explicitly requiring the slave device to undergo waveform recognition, coupling, and matching before entering the working state, the collaborative work between the slave device and the host can be guaranteed, ensuring good synchronization among the multiple devices during operation and achieving stable audio output for the overall system.

[0081] Specifically, in this embodiment, when multiple spatial target points exist, waveform matching and coupling are performed separately for each spatial target point, enabling the same audio content to reach each spatial target point simultaneously. Performing waveform matching and coupling separately for multiple spatial target points allows the same audio content to reach different spatial target points simultaneously, meeting the audio synchronization needs of listeners in multiple areas and expanding the effective coverage of audio synchronization.

[0082] Example 2

[0083] This embodiment provides a multi-device audio synchronization system for implementing the multi-device audio synchronization method in the embodiment, including an audio acquisition module, a processing module, and a control module; the audio acquisition module is used to perform a waveform capture step, acquiring the audio waveforms output by each device; the processing module is used to perform waveform recognition and waveform matching steps, extracting features, classifying and comparing waveforms, and determining time offset; the control module is used to perform waveform coupling and dynamic correction steps, establishing phase locking relationships, adjusting device output delays, and optimizing synchronization effects based on the environment and listener positions.

[0084] The multi-device audio synchronization system using the above technical solution executes corresponding steps through an audio acquisition module, a processing module, and a control module. The modules work together to fully realize the method of multi-device audio synchronization, ensuring that the entire system can achieve stable and efficient audio synchronization, allowing listeners to have a unified listening experience.

[0085] Specifically, in this embodiment, the audio acquisition module is a high-sensitivity microphone array with a sampling frequency ≥96kHz and microsecond-level timestamp recording capability. The audio acquisition module employs a high-sensitivity microphone array with specific parameters, enabling efficient and accurate waveform capture. This provides high-quality data for subsequent system processing and control, laying the foundation for high-precision sound synchronization.

[0086] Specifically, in this embodiment, the processing module incorporates a machine learning model and a waveform fingerprint database. The machine learning model is used to classify waveform features, and the waveform fingerprint database stores waveform feature parameters to aid in classification and matching. This configuration allows the processing module to efficiently and accurately perform waveform recognition and matching, providing a precise basis for adjustments to the control module, improving the system's efficiency and accuracy in processing waveforms, and ultimately ensuring effective sound synchronization.

[0087] Specifically, in this embodiment, the control module includes an environmental monitoring unit and a position tracking unit. The environmental monitoring unit monitors environmental parameters such as temperature, humidity, and air pressure in real time, while the position tracking unit tracks the audience's position to determine the focus area. This control module, comprising the environmental monitoring unit and the position tracking unit, can acquire environmental information and audience position in real time, providing comprehensive data support for dynamic correction. This allows the system to flexibly respond to changes in the environment and audience position, continuously optimizing the sound synchronization effect.

[0088] In summary, the multi-device audio synchronization method and system provided in this embodiment achieve multi-device audio synchronization through a five-step collaborative process of "waveform capture - waveform recognition - waveform matching - waveform coupling - dynamic correction".

[0089] 1. Basic Synchronization Logic: For different spatial points (such as points A, B, and C), the output delay of the devices is adjusted to ensure that the same audio content from the corresponding devices arrives at the target point simultaneously. For example, only the audio from device 1 needs to arrive at point A, the audio from devices 1 and 2 needs to arrive at point B simultaneously, and the audio from devices 1, 2, and 3 needs to arrive at point C simultaneously.

[0090] 2. System operation process: After the device is started, it first identifies the audio waveform, establishes a synchronization relationship through waveform coupling and matching, and then enters the working state; for slave devices (such as devices No. 2 and No. 3), it is necessary to wait for the coupling to be completed before working together with the host (device No. 1) to ensure overall synchronization.

[0091] How to use

[0092] 1. Equipment deployment: Set up 1 host (device 1) and at least 1 slave device (such as device 2 and device 3), and determine the installation location of the devices according to the actual space requirements.

[0093] 2. System startup: The host waits for the working command, and the slave devices enter the waiting coupling state; after the system starts, it automatically performs waveform capture and collects the audio waveforms output by each device.

[0094] 3. Waveform processing: Feature extraction and recognition of the acquired waveforms, waveform fingerprinting and classification; time offset between waveforms is determined by matching, and phase locking relationship between devices is established by coupling to adjust the output delay.

[0095] 4. Dynamic optimization: The system monitors environmental parameters (temperature, humidity, air pressure) and audience location in real time, and dynamically adjusts delay parameters to ensure that the sound can reach the target area synchronously under different environments and audience locations.

[0096] The above are merely preferred embodiments of the present invention and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should recognize that any equivalent substitutions and obvious changes made based on the description and illustrations of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for audio synchronization among multiple devices, characterized in that, Includes the following steps: Waveform capture: Audio waveforms output by each device are captured using an audio acquisition device; Waveform recognition: Extracting features from the acquired waveforms and classifying the waveforms based on the extracted features; Waveform matching: Compare the target waveform of the device to be synchronized with the reference waveform to determine the time offset between the two; Waveform coupling: Establish phase-locked relationships between devices based on time offset, adjust the output delay of each device, and enable waveforms to be superimposed at the target point in space; Dynamic correction: Real-time monitoring of environmental parameters and audience positions, adjusting delay parameters based on monitoring results to optimize sound synchronization.

2. The method for audio synchronization between multiple devices according to claim 1, characterized in that, In the waveform capture step, a high-sensitivity microphone array is used as the audio acquisition device, and the sampling frequency of the microphone array is ≥96kHz, with the timestamp accuracy during acquisition being accurate to the microsecond level.

3. The method for audio synchronization among multiple devices according to claim 1, characterized in that, In the waveform recognition step, the extracted waveform feature parameters include amplitude envelope, spectral features and zero crossings. Machine learning algorithms are used in the classification process, and a waveform fingerprint database is established to assist in classification.

4. The method for audio synchronization among multiple devices according to claim 1, characterized in that, In the waveform matching step, the comparison is achieved by calculating the similarity between the reference waveform and the target waveform. During the comparison process, the matching window size can be dynamically adjusted within the range of 10-100ms, and the time offset is determined by a cross-correlation algorithm.

5. The method for audio synchronization between multiple devices according to claim 1, characterized in that, In the waveform coupling step, the output delay of each device is adjusted to an accuracy of ±10μs to ensure that the waveforms are accurately superimposed at the target point in space.

6. The method for audio synchronization between multiple devices according to claim 1, characterized in that, In the dynamic correction step, the environmental parameters monitored in real time include temperature, humidity and air pressure. The change in sound speed is predicted based on the environmental parameters, and the delay parameter is adjusted according to the change in sound speed.

7. The method for audio synchronization among multiple devices according to claim 1, characterized in that, In the dynamic correction step, the focus area is determined by listener position tracking, and the sound output of each device is optimized for the focus area to ensure sound synchronization within the focus area.

8. The method for audio synchronization between multiple devices according to claim 1, characterized in that, The multiple devices include a host device (No. 1) and at least one slave device to be synchronized. When the slave device is in the waiting coupling state, it needs to go through waveform recognition, waveform coupling and waveform matching before it can enter the working state and achieve sound synchronization with the host.

9. The method for audio synchronization among multiple devices according to claim 1, characterized in that, When there are multiple spatial target points, waveform matching and waveform coupling are performed separately for different spatial target points so that the same audio content can reach each spatial target point at the same time.

10. A multi-device audio synchronization system for implementing the multi-device audio synchronization method according to any one of claims 1-9, characterized in that, It includes an audio acquisition module, a processing module, and a control module; the audio acquisition module is used to perform waveform capture steps to acquire the audio waveforms output by each device; the processing module is used to perform waveform recognition and waveform matching steps to extract features, classify and compare waveforms, and determine time offset; the control module is used to perform waveform coupling and dynamic correction steps to establish phase locking relationship, adjust device output delay, and optimize synchronization effect according to environment and listener position.

Images