Audio clock skew compensation method and device based on dynamic decision

Abstract

The invention discloses an audio clock skew compensation method and device based on a dynamic decision, and relates to the field of audio control, and the method comprises the steps: obtaining a plurality of audio clips successively sent by a first audio stream; carrying out deviation compensation processing on the plurality of audio clips according to a first deviation compensation method, and respectively carrying out clock deviation compensation on the plurality of audio clips in the processing process until the plurality of audio clips are processed; the first deviation compensation method comprises the following steps: acquiring first time when a first audio stream is established; obtaining a first frame number and a second time of the first audio clip; calculating a difference value between the second time and the first time to obtain first deviation time; and generating a compensation decision for the first audio stream according to the first deviation time, the first frame number and the first buffer time based on the sending bit sequence of the first audio segment in the plurality of audio segments, so that the first audio stream is subjected to clock deviation compensation according to the compensation decision. According to the invention, the accuracy of clock skew compensation can be improved.

Description

Technical Field

[0001] The present invention relates to the field of audio control, and in particular to a method and device for compensating audio clock deviation based on dynamic decision. Background Art

[0002] In long-playing scenarios such as calls, the headset will be connected to the remote device for a long time to generate an audio stream for outputting audio data. At this time, the slight difference between the clock of the remote device and the clock of the headset will gradually accumulate, causing the headset to continue writing input to a full buffer, resulting in an overflow phenomenon in which data is lost, or an underflow phenomenon in which the buffer is empty after data output is completed and the expected data cannot be obtained, resulting in intermittent audio stream output from the headset, causing a decrease in the user's hearing experience.

[0003] In the prior art, hardware-based chip synchronization solutions are mainly used, which have the following disadvantages: (1) In the chip-based synchronization solution, usually only one local audio clock can be set, that is, there is only one fixed clock frequency locally, so it is difficult to adapt to the mixing synchronization solution in the case of multiple remote devices; (2) In the chip-based synchronization solution, there is a distortion between the estimated value of the clock deviation calculated by evaluating the data in the data buffer and the actual value, resulting in frequent compensation of the audio clock deviation, and the compensation is easy to repeat; (3) In the chip-based synchronization solution, the local audio clock needs to be bound to the system clock, and it is difficult to adjust the local audio clock with high precision, resulting in distortion in the clock deviation compensation decision made by the chip-based synchronization solution based on the audio clock based on the system clock, and the accuracy of the audio clock deviation compensation is low. Summary of the invention

[0004] The present invention provides an audio clock deviation compensation method and device based on dynamic decision making, so as to solve the technical problem of low accuracy of clock deviation compensation in the existing clock deviation compensation method of hardware chip.

[0005] In a first aspect, the present application provides an audio clock deviation compensation method based on dynamic decision making, comprising:

[0006] Acquire multiple audio clips sent successively by a first audio stream; wherein the first audio stream is established when data is transmitted between a local device and a remote device;

[0007] Performing deviation compensation processing on the multiple audio clips according to a preset first deviation compensation method, and performing clock deviation compensation on the multiple audio clips respectively during the processing until the multiple audio clips are processed;

[0008] Wherein, the first deviation compensation method comprises:

[0009] Obtaining a first time of a local clock corresponding to when the first audio stream is established;

[0010] Acquire a first frame number of a first audio segment currently being processed and a second time of a remote clock corresponding to the first audio segment;

[0011] Calculate the difference between the second time and the first time to obtain a first deviation time;

[0012] Based on the sending bit order of the first audio segment in the multiple audio segments, and according to the first deviation time, the first number of frames and the preset first buffer time, a compensation decision for the first audio stream is generated, so that the first audio stream performs clock deviation compensation according to the compensation decision.

[0013] In this way, multiple audio segments sent successively by the first audio stream are first obtained, and then deviation compensation processing is performed on the audio segments. During the processing, the first time of the local clock corresponding to when the first audio stream is established, the first frame number of the first audio segment currently being processed and the second time of the corresponding remote clock are obtained, and the first deviation time is obtained according to the difference between the second time and the first time. In this way, the first deviation time is calculated based on the local clock and the remote clock, so that the deviation value of the local clock and the remote clock can be estimated more accurately, and then based on the sending bit sequence of the first audio segment, a compensation decision is generated according to the first deviation time, the first frame number and the preset first buffer time. In this way, different compensation decisions can be generated according to different sending bit sequences, first deviation times and first frame numbers of the first audio segment, so that the generated compensation decision is more in line with the current local clock deviation compensation requirements, and then clock deviation compensation is performed according to the generated compensation decision, which can improve the accuracy of the deviation compensation of the local clock.

[0014] Further, the generating, based on the sending bit sequence of the first audio segment in the multiple audio segments, a compensation decision for the first audio stream according to the first deviation time, the first number of frames, and a preset first buffer time, so that the first audio stream performs clock deviation compensation according to the compensation decision, specifically includes:

[0015] Calculate the product of the first buffer time and the first number of frames to obtain a second buffer time;

[0016] When the first audio segment is at the first position of the transmission bit sequence of the multiple audio segments, a compensation decision for the first audio stream is generated according to the first deviation time and the second buffering time, and a preset first buffering data is operated according to the compensation decision to perform clock deviation compensation for the first audio stream; wherein the first buffering data corresponds to the first buffering time;

[0017] When the first audio segment is not at the first position in the sending order of the multiple audio segments, a compensation decision for the first audio stream is generated according to the first deviation time and the second buffering time, and an input buffer corresponding to the first audio stream is operated according to the compensation decision to perform clock deviation compensation for the first audio stream.

[0018] In this way, the second buffer time is calculated first, and then based on the different sending bit sequences of the first audio segment, different compensation decisions are generated according to the first deviation time and the second buffer time. According to the different compensation decisions, the first buffer data or the input buffer is selectively operated to perform clock deviation compensation. In this way, the generated compensation decision can be more in line with the requirements of the current local clock for deviation compensation under different situations, thereby improving the accuracy when performing clock deviation compensation.

[0019] Further, when the first audio segment is at the first position of the transmission bit sequence of the multiple audio segments, generating a compensation decision for the first audio stream according to the first deviation time and the second buffering time, and operating the preset first buffering data according to the compensation decision to perform clock deviation compensation for the first audio stream, specifically includes:

[0020] When the first audio segment is at the first position in the sending sequence of the multiple audio segments, calculating the difference between the first deviation time and the second buffer time to obtain a second deviation time;

[0021] When the second deviation time is a positive value, a first compensation decision for the first audio stream is generated, and the first buffered data is discarded according to the first compensation decision, and clock deviation compensation is performed on the first audio stream according to a preset first data consumption method;

[0022] When the second deviation time is a negative value, a second compensation decision for the first audio stream is generated, and the first buffered data is inserted into an output buffer corresponding to the first audio stream according to the second compensation decision to compensate for the clock deviation of the first audio stream.

[0023] In this way, the second deviation time is first calculated based on the first deviation time and the second buffer time, and then different compensation decisions are generated according to the positive and negative conditions of the second deviation time. According to different compensation decisions, the first buffer data is selectively discarded and clock deviation compensation is performed according to the first data consumption method, or the first buffer data is inserted into the output buffer to complete the clock deviation compensation. In this way, the first buffer data and the output buffer are selectively operated, and when the local clock needs to perform deviation compensation, the deviation compensation of the local clock can be achieved through the rapid operation of the preset first buffer data, thereby improving the efficiency of clock deviation compensation.

[0024] Furthermore, the first data consumption method specifically includes:

[0025] The first audio data in the input buffer corresponding to the first audio stream is resampled. During the resampling process, the first type of sampling points are removed once for each frame of the first audio data in chronological order until the sum of the sampling durations of all the removed first type of sampling points reaches a preset first cumulative time, and the resampling is completed. The resampled first audio data is output to complete the clock deviation compensation for the first audio stream; wherein the sampling duration of the first type of sampling points is the preset first sampling time.

[0026] In this way, the first audio data of the input buffer is resampled, and the first type of sampling points are eliminated once for each frame of the first audio data during resampling until the sum of the sampling durations of all the eliminated first type of sampling points reaches a preset value. In this way, the deviation of the first audio data of the input buffer can be evenly compensated, the discontinuity of the first audio data caused by the elimination of sampling points can be reduced, and a better listening experience can be retained.

[0027] Further, when the first audio segment is not at the first position of the transmission bit sequence of the multiple audio segments, generating a compensation decision for the first audio stream according to the first deviation time and the second buffering time, and operating the input buffer corresponding to the first audio stream according to the compensation decision to perform clock deviation compensation on the first audio stream, specifically includes:

[0028] When the first audio segment is not at the first position of the sending position sequence of the multiple audio segments, calculating an absolute value of a difference between the first deviation time and the second buffer time to obtain a third deviation time;

[0029] When the third deviation time is equal to a preset value and the first deviation time is greater than the second buffering time, generating a third compensation decision for the audio stream, and performing clock deviation compensation on the first audio stream based on a preset second data consumption method according to the third compensation decision;

[0030] When the third deviation time is equal to a preset value and the first deviation time is less than the second buffering time, a fourth compensation decision for the audio stream is generated, and according to the fourth compensation decision, clock deviation compensation is performed on the first audio stream based on a preset first data compensation method.

[0031] In this way, the third deviation time is first obtained according to the absolute value of the difference between the first deviation time and the second buffer time, and then different compensation decisions are generated according to the relationship between the third deviation time and the first deviation time and the second buffer time. According to different compensation decisions, the second data consumption method is selectively used for clock deviation compensation or the first data compensation method is used for clock deviation compensation. In this way, different compensation decisions can be generated according to different situations of the local clock, and then a suitable method can be selected for clock deviation compensation, thereby improving the accuracy of clock deviation compensation.

[0032] In a second aspect, the present application provides an audio clock deviation compensation device based on dynamic decision making, including an audio acquisition module and an audio processing module;

[0033] The audio acquisition module is used to acquire multiple audio clips sent successively by a first audio stream; wherein the first audio stream is established when data is transmitted between a local device and a remote device;

[0034] The audio processing module is used to perform deviation compensation processing on the multiple audio clips according to a preset first deviation compensation method, and perform clock deviation compensation on the multiple audio clips respectively during the processing until the multiple audio clips are processed;

[0035] Wherein, the first deviation compensation method comprises:

[0036] Obtaining a first time of a local clock corresponding to when the first audio stream is established;

[0037] Acquire a first frame number of a first audio segment currently being processed and a second time of a remote clock corresponding to the first audio segment;

[0038] Calculate the difference between the second time and the first time to obtain a first deviation time;

[0039] Based on the sending bit order of the first audio segment in the multiple audio segments, and according to the first deviation time, the first number of frames and the preset first buffer time, a compensation decision for the first audio stream is generated, so that the first audio stream performs clock deviation compensation according to the compensation decision.

[0040] Further, the audio processing module includes a decision compensation submodule; the decision compensation submodule includes a second buffer calculation unit, a first order compensation unit and a second order compensation unit;

[0041] The second buffer calculation unit is used to calculate the product of the first buffer time and the first frame number to obtain a second buffer time;

[0042] The first order compensation unit is configured to generate a compensation decision for the first audio stream according to the first deviation time and the second buffering time when the first audio segment is at the first position of the transmission bit sequence of the multiple audio segments, and operate the preset first buffering data according to the compensation decision to perform clock deviation compensation on the first audio stream; wherein the first buffering data corresponds to the first buffering time;

[0043] The second order compensation unit is used to generate a compensation decision for the first audio stream according to the first deviation time and the second buffering time when the first audio segment is not at the first position in the sending position order of the multiple audio segments, and operate the input buffer corresponding to the first audio stream according to the compensation decision to compensate the clock deviation of the first audio stream.

[0044] Further, the first order compensation unit includes a second deviation calculation subunit, a first compensation subunit and a second compensation subunit;

[0045] The second deviation calculation subunit is configured to calculate a difference between the first deviation time and the second buffer time to obtain a second deviation time when the first audio segment is at the first position in the sending sequence of the multiple audio segments;

[0046] The first compensation subunit is used for generating a first compensation decision for the first audio stream when the second deviation time is a positive value, discarding the first buffered data according to the first compensation decision, and performing clock deviation compensation for the first audio stream according to a preset first data consumption method;

[0047] The second compensation subunit is used to generate a second compensation decision for the first audio stream when the second deviation time is a negative value, and insert the first buffered data into an output buffer corresponding to the first audio stream according to the second compensation decision to compensate for the clock deviation of the first audio stream.

[0048] Furthermore, the first data consumption method specifically includes:

[0049] The first audio data in the input buffer corresponding to the first audio stream is resampled. During the resampling process, the first type of sampling points are removed once for each frame of the first audio data in chronological order until the sum of the sampling durations of all the removed first type of sampling points reaches a preset first cumulative time, and the resampling is completed. The resampled first audio data is output to complete the clock deviation compensation for the first audio stream; wherein the sampling duration of the first type of sampling points is the preset first sampling time.

[0050] Further, the second order compensation unit includes a third deviation calculation subunit, a third compensation subunit and a fourth compensation subunit;

[0051] The third deviation calculation subunit is configured to calculate an absolute value of a difference between the first deviation time and the second buffer time to obtain a third deviation time when the first audio segment is not at the first position in the sending sequence of the multiple audio segments;

[0052] The third compensation subunit is configured to generate a third compensation decision for the audio stream when the third deviation time is equal to a preset value and the first deviation time is greater than the second buffering time, and perform clock deviation compensation on the first audio stream based on the third compensation decision and based on a preset second data consumption method;

[0053] The fourth compensation subunit is used to generate a fourth compensation decision for the audio stream when the third deviation time is equal to a preset value and the first deviation time is less than the second buffering time, and according to the fourth compensation decision, perform clock deviation compensation on the first audio stream based on the preset first data compensation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 : A flowchart of an embodiment of an audio clock deviation compensation method based on dynamic decision-making provided by the present invention;

[0055] Figure 2 : A module structure diagram of an embodiment of an audio clock deviation compensation device based on dynamic decision-making provided by the present invention;

[0056] Figure 3 : A processing scenario for connecting multiple remote devices provided by the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, unless otherwise clearly and specifically defined, "multiple" and "several" mean two or more.

[0059] In playback scenarios such as long calls, long music playback or mixed playback, the headset will be connected to one or more remote devices for a long time to generate an audio stream for transmitting audio data. In these playback scenarios, the clock of the remote device and the clock of the local device need to be synchronized or corrected to avoid discontinuity and reduce the user's hearing experience. However, the hardware chip synchronization solution used in the prior art can often only set one local clock, which makes it impossible to adapt to the situation where multiple remote devices are connected. At the same time, the evaluation of the data in the data buffer of the audio stream is distorted, resulting in frequent compensation caused by clock deviation, and the compensation is easy to repeat. In the chip-based solution, the local clock needs to be bound to the system clock, which makes it difficult to adjust the local clock with high precision, resulting in low accuracy of the deviation compensation of the local clock. Based on the above practical problems, the present invention provides an audio clock deviation compensation method and device based on dynamic decision-making, and provides the following partial embodiments.

[0060] Embodiment 1

[0061] See also Figure 1 , an audio clock deviation compensation method based on dynamic decision-making provided by an embodiment of the present invention, includes steps S101 to S102, each step is specifically as follows:

[0062] Step S101: Acquire multiple audio clips sent successively by a first audio stream; wherein the first audio stream is established when data is transmitted between a local device and a remote device.

[0063] In an optional embodiment, the local device and different remote devices may respectively establish at least one audio stream, and any one of them may be selected as the first audio stream of this embodiment.

[0064] Specifically, when the local and remote devices need to establish an audio stream, the input buffer and output buffer of the currently established audio stream are set according to the frame interval time defined in the communication protocol used for communication between the local and remote devices, and the frame interval time of writing the input buffer is recorded as I i , the frame interval time consumed by the output buffer is recorded as I o ; Wherein, the communication protocols include USB, Bluetooth and DECT protocols; the frame gap time includes 1ms, 7.5ms and 10ms.

[0065] Step S102: performing deviation compensation processing on the multiple audio clips according to a preset first deviation compensation method, and performing clock deviation compensation on the multiple audio clips respectively during the processing until the processing of the multiple audio clips is completed;

[0066] Wherein, the first deviation compensation method comprises:

[0067] Obtaining a first time of a local clock corresponding to when the first audio stream is established;

[0068] Acquire a first frame number of a first audio segment currently being processed and a second time of a remote clock corresponding to the first audio segment;

[0069] Calculate the difference between the second time and the first time to obtain a first deviation time;

[0070] Based on the sending bit order of the first audio segment in the multiple audio segments, and according to the first deviation time, the first number of frames and the preset first buffer time, a compensation decision for the first audio stream is generated, so that the first audio stream performs clock deviation compensation according to the compensation decision.

[0071] In an optional embodiment, the first time is recorded as T 0 , is the current time of the local clock corresponding to the establishment of the first audio stream. Specifically, it is the current SYSTICK timer accumulated value of the local clock.

[0072] In an optional embodiment, the second time is recorded as T n , is the current time of the remote clock corresponding to the first audio clip. Specifically, it is the clock time triggered by external hardware when the local device communicates with the corresponding remote device. More specifically, it is the accumulated time of data interruption during the communication process; wherein the data interruption includes USB data interruption and Bluetooth data interruption.

[0073] In an optional embodiment, the first frame number is recorded as n, which is the frame number of the first audio segment currently being processed.

[0074] In an optional embodiment, the first deviation time is recorded as T 1 , is the difference between the second time and the first time, specifically T 1 =T n -T 0 .

[0075] In an optional embodiment, the first buffer time is recorded as I, then I=min(I i , I o ), and according to the first buffering time, pre-filling first buffering data with a duration of the first buffering time in the input buffer and output buffer of the first audio stream.

[0076] Further, the generating, based on the sending bit sequence of the first audio segment in the multiple audio segments, a compensation decision for the first audio stream according to the first deviation time, the first number of frames, and a preset first buffer time, so that the first audio stream performs clock deviation compensation according to the compensation decision, specifically includes:

[0077] Calculate the product of the first buffer time and the first number of frames to obtain a second buffer time;

[0078] When the first audio segment is at the first position of the transmission bit sequence of the multiple audio segments, a compensation decision for the first audio stream is generated according to the first deviation time and the second buffering time, and a preset first buffering data is operated according to the compensation decision to perform clock deviation compensation for the first audio stream; wherein the first buffering data corresponds to the first buffering time;

[0079] When the first audio segment is not at the first position in the sending order of the multiple audio segments, a compensation decision for the first audio stream is generated according to the first deviation time and the second buffering time, and an input buffer corresponding to the first audio stream is operated according to the compensation decision to perform clock deviation compensation for the first audio stream.

[0080] In an optional embodiment, the second buffer time is recorded as I′, then I′=I*n.

[0081] In this way, the second buffer time is calculated first, and then based on the different sending bit sequences of the first audio segment, different compensation decisions are generated according to the first deviation time and the second buffer time. According to the different compensation decisions, the first buffer data or the input buffer is selectively operated to perform clock deviation compensation. In this way, the generated compensation decision can be more in line with the requirements of the current local clock for deviation compensation under different situations, thereby improving the accuracy when performing clock deviation compensation.

[0082] Further, when the first audio segment is at the first position of the transmission bit sequence of the multiple audio segments, generating a compensation decision for the first audio stream according to the first deviation time and the second buffering time, and operating the preset first buffering data according to the compensation decision to perform clock deviation compensation for the first audio stream, specifically includes:

[0083] When the first audio segment is at the first position in the sending sequence of the multiple audio segments, calculating the difference between the first deviation time and the second buffer time to obtain a second deviation time;

[0084] When the second deviation time is a positive value, a first compensation decision for the first audio stream is generated, and the first buffered data is discarded according to the first compensation decision, and clock deviation compensation is performed on the first audio stream according to a preset first data consumption method;

[0085] When the second deviation time is a negative value, a second compensation decision for the first audio stream is generated, and the first buffered data is inserted into an output buffer corresponding to the first audio stream according to the second compensation decision to compensate for the clock deviation of the first audio stream.

[0086] In an optional embodiment, the second deviation time is recorded as T 2 , then T 2 =T 1 -I'.

[0087] Considering that when the first audio segment is at the first place in the sending order, the second deviation time is calculated and the first compensation decision is generated according to the second deviation time. This is because when the first audio segment is at the first place in the sending order, only the first audio segment exists in the input buffer and output buffer corresponding to the first audio stream. At this time, if the calculated second deviation time is a positive value, it means that the clock of the remote device is faster than the local device and the local data consumption is slower. In this case, there is no need to retain the first buffer data, and the data consumption is accelerated according to the first data consumption method so that the local clock can keep up with the remote clock in time; if the calculated second deviation time is a negative value, it means that the clock of the remote device is slower than the local device and the remote data is sent slowly. At this time, the pre-filled first buffer data needs to be inserted into the output buffer corresponding to the first audio stream to extend the time of the current output audio segment to wait for the arrival of the remote device data.

[0088] In this way, the second deviation time is first calculated based on the first deviation time and the second buffer time, and then different compensation decisions are generated according to the positive and negative conditions of the second deviation time. According to different compensation decisions, the first buffer data is selectively discarded and clock deviation compensation is performed according to the first data consumption method, or the first buffer data is inserted into the output buffer to complete the clock deviation compensation. In this way, the first buffer data and the output buffer are selectively operated, and when the local clock needs to perform deviation compensation, the deviation compensation of the local clock can be achieved through the rapid operation of the preset first buffer data, thereby improving the efficiency of clock deviation compensation.

[0089] Furthermore, the first data consumption method specifically includes:

[0090] The first audio data in the input buffer corresponding to the first audio stream is resampled. During the resampling process, the first type of sampling points are removed once for each frame of the first audio data in chronological order until the sum of the sampling durations of all the removed first type of sampling points reaches a preset first cumulative time, and the resampling is completed. The resampled first audio data is output to complete the clock deviation compensation for the first audio stream; wherein the sampling duration of the first type of sampling points is the preset first sampling time.

[0091] In an optional embodiment, the first accumulated time is preferably: I .

[0092] In this way, the first audio data of the input buffer is resampled, and the first type of sampling points are eliminated once for each frame of the first audio data during resampling until the sum of the sampling durations of all the eliminated first type of sampling points reaches a preset value. In this way, the deviation of the first audio data of the input buffer can be evenly compensated, the discontinuity of the first audio data caused by the elimination of sampling points can be reduced, and a better listening experience can be retained.

[0093] Further, when the first audio segment is not at the first position of the transmission bit sequence of the multiple audio segments, generating a compensation decision for the first audio stream according to the first deviation time and the second buffering time, and operating the input buffer corresponding to the first audio stream according to the compensation decision to perform clock deviation compensation on the first audio stream, specifically includes:

[0094] When the first audio segment is not at the first position of the sending position sequence of the multiple audio segments, calculating an absolute value of a difference between the first deviation time and the second buffer time to obtain a third deviation time;

[0095] When the third deviation time is equal to a preset value and the first deviation time is greater than the second buffering time, generating a third compensation decision for the audio stream, and performing clock deviation compensation on the first audio stream based on a preset second data consumption method according to the third compensation decision;

[0096] When the third deviation time is equal to a preset value and the first deviation time is less than the second buffering time, a fourth compensation decision for the audio stream is generated, and according to the fourth compensation decision, clock deviation compensation is performed on the first audio stream based on a preset first data compensation method.

[0097] In an optional embodiment, the third deviation time is recorded as T 3 , then T 3 =|T 1 -I'|.

[0098] In an optional embodiment, the preset value is preferably I.

[0099] Considering that when the first audio segment is not at the first position in the sending sequence, the third deviation time is calculated and a compensation decision is generated according to the third deviation time. This is because when the first audio segment is not at the first position in the sending sequence, it means that the input buffer and the output buffer corresponding to the first audio stream are not empty after removing the first audio segment. At this time, if the calculated third deviation time is not equal to the preset value, it means that the accumulated deviation of the local clock has not reached the compensation standard, and there is no need to execute the clock compensation method; if the calculated third deviation time is equal to the preset value and is a positive value (the first deviation time is greater than the second buffer time), it means that the local clock has generated a delay equal to the preset value, and it is necessary to accelerate the local data consumption so that the local clock can keep up with the remote clock in time; if the calculated third deviation time is equal to the preset value and is a negative value (the first deviation time is less than the second buffer time), it means that the local data is consumed too quickly so that a delay equal to the preset value is generated, and it is necessary to slow down the local data consumption and compensate for the local data so that the local clock is synchronized with the remote clock. Based on this, when audio is transmitted between the local and remote devices, the clock deviation can be gradually accumulated, and a compensation operation will be performed only after the clock deviation accumulates to a certain value for a long time, and the clock deviation is accumulated again after compensation to ensure that compensation is not repeated.

[0100] In this way, the third deviation time is first obtained according to the absolute value of the difference between the first deviation time and the second buffer time, and then different compensation decisions are generated according to the relationship between the third deviation time and the first deviation time and the second buffer time. According to different compensation decisions, the second data consumption method is selectively used for clock deviation compensation or the first data compensation method is used for clock deviation compensation. In this way, different compensation decisions can be generated according to different situations of the local clock, and then a suitable method can be selected for clock deviation compensation, thereby improving the accuracy of clock deviation compensation.

[0101] Furthermore, the second data consumption method specifically includes:

[0102] Resampling the first audio data in the input buffer corresponding to the first audio stream, removing the second type of sampling points from each frame of the first audio data in time sequence during the resampling process, until the sum of the sampling durations of all the removed second type of sampling points reaches a preset second cumulative time, and outputting the resampled first audio data to complete the clock deviation compensation for the first audio stream; wherein the sampling duration of the second type of sampling points is the preset second sampling time;

[0103] The sum of the first time and the second accumulated time is used as the initial first time when the next processing starts.

[0104] In an optional embodiment, the second accumulated time is preferably: I .

[0105] The reason why the sum of the first time and the second accumulated time is considered as the initial first time at the beginning of the next processing is that after the current first audio data is resampled through the second data consumption method, the current local clock has been accelerated by a time equal to the second accumulated time, so the first time needs to be increased accordingly.

[0106] In this way, the first audio data in the input buffer is resampled, and the second-category sampling points are eliminated for each frame of the first audio data during the resampling until the sum of the sampling durations of all the eliminated second-category sampling points reaches the preset second cumulative time. In this way, the deviation of the first audio data in the input buffer can be evenly compensated, and the discontinuity of the first audio data due to the elimination of the sampling points can be reduced. The sum of the first time and the second cumulative time is used as the initial first time at the beginning of the next processing. In this way, sufficient deviation can be accumulated between two clock deviation compensations, reducing the frequent clock deviation compensation behavior, ensuring that the clock deviation compensation does not recur, and thus improving the accuracy of the clock deviation compensation.

[0107] Furthermore, the first data compensation method specifically includes:

[0108] Resampling the first audio data in the input buffer corresponding to the first audio stream, inserting a third type of sampling point into each frame of the first audio data in time sequence during the resampling process, until the sum of the sampling durations of all the inserted third type of sampling points reaches a preset third cumulative time, and then the resampling is completed, and the resampled first audio data is output to complete the clock deviation compensation for the first audio stream; wherein the sampling duration of the third type of sampling point is the preset third sampling time;

[0109] The difference between the first time and the third accumulated time is used as the initial first time when the next processing starts.

[0110] In an optional embodiment, the third accumulated time is preferably: I .

[0111] The reason why the difference between the first time and the third accumulated time is considered as the initial first time at the beginning of the next processing is that after the current first audio data is resampled by the first data compensation method, the data missing caused by the local clock being too fast has been filled, that is, the time of the local clock has been slowed down accordingly, and therefore the first time needs to be reversed (reduced) accordingly.

[0112] In this way, the first audio data in the input buffer is resampled, and during the resampling, a third-type sampling point is inserted into each frame of the first audio data until the sum of the sampling durations of all the inserted third-type sampling points reaches the third cumulative time. In this way, the first audio data in the input buffer can be evenly compensated for the deviation, the data missing caused by the local clock is filled, and the difference between the first time and the third cumulative time is used as the initial first time at the beginning of the next processing. In this way, sufficient deviation can be accumulated between two clock deviation compensations, reducing the frequent clock deviation compensation behavior, ensuring that the clock deviation compensation does not recur, and thus improving the accuracy of the clock deviation compensation.

[0113] Considering that the present invention evaluates time based on the local clock and the remote clock and generates corresponding compensation decisions, and that the present invention is a software method, which is different from the hardware chip solution of the prior art, by executing the method of the embodiment of the present invention on the audio stream established with different remote devices, it is possible to support the adaptation of the clocks of multiple remote devices without interfering with each other; at the same time, the method of the present invention is not bound to the system clock, and is easier to adjust the local clock than the hardware chip solution, thereby improving the accuracy of the local clock deviation compensation.

[0114] See also Figure 3 , which is a possible scenario of the embodiment of the present invention when multiple remote devices are connected. In this scenario, 4 remote devices are locally connected, 2 of which are received locally and 2 are sent locally; wherein, receive 1, receive 2, send 1 and send 2 are 4 established audio streams; receive 1 buffer and receive 2 buffer are input buffers; send 1 buffer and send 2 buffer are output buffers; pre-filled data areas 1 to 4 are first buffer data pre-filled according to the first buffer time. By locally executing the method of the embodiment of the present invention on the audio streams established by the 4 remote devices respectively, the local clocks corresponding to the 4 remote devices can be compensated for deviations respectively, the clocks of multiple remote devices can be adapted, and the accuracy of the local clock deviation compensation can be improved.

[0115] In this way, multiple audio segments sent successively by the first audio stream are first obtained, and then deviation compensation processing is performed on the audio segments. During the processing, the first time of the local clock corresponding to when the first audio stream is established, the first frame number of the first audio segment currently being processed and the second time of the corresponding remote clock are obtained, and the first deviation time is obtained according to the difference between the second time and the first time. In this way, the first deviation time is calculated based on the local clock and the remote clock, so that the deviation value of the local clock and the remote clock can be estimated more accurately, and then based on the sending bit sequence of the first audio segment, a compensation decision is generated according to the first deviation time, the first frame number and the preset first buffer time. In this way, different compensation decisions can be generated according to different sending bit sequences, first deviation times and first frame numbers of the first audio segment, so that the generated compensation decision is more in line with the current local clock deviation compensation requirements, and then clock deviation compensation is performed according to the generated compensation decision, which can improve the accuracy of the deviation compensation of the local clock.

[0116] Embodiment 2

[0117] See also Figure 2 , an audio clock deviation compensation device based on dynamic decision-making provided by an embodiment of the present invention, comprising an audio acquisition module 210 and an audio processing module 220;

[0118] The audio acquisition module 210 is used to acquire multiple audio segments sent successively by a first audio stream; wherein the first audio stream is established when data is transmitted between a local device and a remote device;

[0119] The audio processing module 220 is used to perform deviation compensation processing on the multiple audio clips according to a preset first deviation compensation method, and perform clock deviation compensation on the multiple audio clips respectively during the processing until the multiple audio clips are processed;

[0120] Wherein, the first deviation compensation method comprises:

[0121] Obtaining a first time of a local clock corresponding to when the first audio stream is established;

[0122] Acquire a first frame number of a first audio segment currently being processed and a second time of a remote clock corresponding to the first audio segment;

[0123] Calculate the difference between the second time and the first time to obtain a first deviation time;

[0124] Based on the sending bit order of the first audio segment in the multiple audio segments, and according to the first deviation time, the first number of frames and the preset first buffer time, a compensation decision for the first audio stream is generated, so that the first audio stream performs clock deviation compensation according to the compensation decision.

[0125] Further, the audio processing module 220 includes a decision compensation submodule 221; the decision compensation submodule 221 includes a second buffer calculation unit 2211, a first order compensation unit 2212 and a second order compensation unit 2213;

[0126] The second buffer calculation unit 2211 is used to calculate the product of the first buffer time and the first frame number to obtain a second buffer time;

[0127] The first order compensation unit 2212 is configured to generate a compensation decision for the first audio stream according to the first deviation time and the second buffering time when the first audio segment is at the first position of the transmission bit sequence of the multiple audio segments, and operate the preset first buffering data according to the compensation decision to perform clock deviation compensation on the first audio stream; wherein the first buffering data corresponds to the first buffering time;

[0128] The second order compensation unit 2213 is used to generate a compensation decision for the first audio stream according to the first deviation time and the second buffering time when the first audio segment is not at the first position in the sending position order of the multiple audio segments, and operate the input buffer corresponding to the first audio stream according to the compensation decision to perform clock deviation compensation for the first audio stream.

[0129] Further, the first order compensation unit includes a second deviation calculation subunit, a first compensation subunit and a second compensation subunit;

[0130] The second deviation calculation subunit is configured to calculate a difference between the first deviation time and the second buffer time to obtain a second deviation time when the first audio segment is at the first position in the sending sequence of the multiple audio segments;

[0131] The first compensation subunit is used for generating a first compensation decision for the first audio stream when the second deviation time is a positive value, discarding the first buffered data according to the first compensation decision, and performing clock deviation compensation for the first audio stream according to a preset first data consumption method;

[0132] The second compensation subunit is used to generate a second compensation decision for the first audio stream when the second deviation time is a negative value, and insert the first buffered data into an output buffer corresponding to the first audio stream according to the second compensation decision to compensate for the clock deviation of the first audio stream.

[0133] Furthermore, the first data consumption method specifically includes:

[0134] The first audio data in the input buffer corresponding to the first audio stream is resampled. During the resampling process, the first type of sampling points are removed once for each frame of the first audio data in chronological order until the sum of the sampling durations of all the removed first type of sampling points reaches a preset first cumulative time, and the resampling is completed. The resampled first audio data is output to complete the clock deviation compensation for the first audio stream; wherein the sampling duration of the first type of sampling points is the preset first sampling time.

[0135] Further, the second order compensation unit includes a third deviation calculation subunit, a third compensation subunit and a fourth compensation subunit;

[0136] The third deviation calculation subunit is configured to calculate an absolute value of a difference between the first deviation time and the second buffer time to obtain a third deviation time when the first audio segment is not at the first position in the sending sequence of the multiple audio segments;

[0137] The third compensation subunit is configured to generate a third compensation decision for the audio stream when the third deviation time is equal to a preset value and the first deviation time is greater than the second buffering time, and perform clock deviation compensation on the first audio stream based on the third compensation decision and based on a preset second data consumption method;

[0138] The fourth compensation subunit is used to generate a fourth compensation decision for the audio stream when the third deviation time is equal to a preset value and the first deviation time is less than the second buffering time, and according to the fourth compensation decision, perform clock deviation compensation on the first audio stream based on the preset first data compensation method.

[0139] Furthermore, the second data consumption method specifically includes:

[0140] Resampling the first audio data in the input buffer corresponding to the first audio stream, removing the second type of sampling points from each frame of the first audio data in time sequence during the resampling process, until the sum of the sampling durations of all the removed second type of sampling points reaches a preset second cumulative time, and outputting the resampled first audio data to complete the clock deviation compensation for the first audio stream; wherein the sampling duration of the second type of sampling points is the preset second sampling time;

[0141] The sum of the first time and the second accumulated time is used as the initial first time when the next processing starts.

[0142] Furthermore, the first data compensation method specifically includes:

[0143] Resampling the first audio data in the input buffer corresponding to the first audio stream, inserting a third type of sampling point into each frame of the first audio data in time sequence during the resampling process, until the sum of the sampling durations of all the inserted third type of sampling points reaches a preset third cumulative time, and then the resampling is completed, and the resampled first audio data is output to complete the clock deviation compensation for the first audio stream; wherein the sampling duration of the third type of sampling point is the preset third sampling time;

[0144] The difference between the first time and the third accumulated time is used as the initial first time when the next processing starts.

[0145] In this way, multiple audio segments sent successively by the first audio stream are first obtained, and then deviation compensation processing is performed on the audio segments. During the processing, the first time of the local clock corresponding to when the first audio stream is established, the first frame number of the first audio segment currently being processed and the second time of the corresponding remote clock are obtained, and the first deviation time is obtained according to the difference between the second time and the first time. In this way, the first deviation time is calculated based on the local clock and the remote clock, so that the deviation value of the local clock and the remote clock can be estimated more accurately, and then based on the sending bit sequence of the first audio segment, a compensation decision is generated according to the first deviation time, the first frame number and the preset first buffer time. In this way, different compensation decisions can be generated according to different sending bit sequences, first deviation times and first frame numbers of the first audio segment, so that the generated compensation decision is more in line with the current local clock deviation compensation requirements, and then clock deviation compensation is performed according to the generated compensation decision, which can improve the accuracy of the deviation compensation of the local clock.

[0146] Accordingly, the embodiments of the present invention also adaptively provide a computer device and a computer-readable storage medium.

[0147] The computer device comprises: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor;

[0148] Wherein, when the processor executes the computer program, the audio clock deviation compensation method based on dynamic decision as described above is implemented.

[0149] The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the audio clock deviation compensation method as described above.

[0150] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. An audio clock deviation compensation method based on dynamic decision making, characterized in that: include: Acquire multiple audio clips sent successively by a first audio stream; wherein the first audio stream is established when data is transmitted between a local device and a remote device; Performing deviation compensation processing on the multiple audio clips according to a preset first deviation compensation method, and performing clock deviation compensation on the multiple audio clips respectively during the processing until the multiple audio clips are processed; Wherein, the first deviation compensation method comprises: Obtaining a first time of a local clock corresponding to when the first audio stream is established; Acquire a first frame number of a first audio segment currently being processed and a second time of a remote clock corresponding to the first audio segment; Calculate the difference between the second time and the first time to obtain a first deviation time; Based on the sending bit order of the first audio segment in the multiple audio segments, and according to the first deviation time, the first number of frames and the preset first buffer time, a compensation decision for the first audio stream is generated, so that the first audio stream performs clock deviation compensation according to the compensation decision.

2. The method for compensating audio clock deviation based on dynamic decision making according to claim 1, characterized in that: The step of generating a compensation decision for the first audio stream based on the sending bit sequence of the first audio segment in the multiple audio segments and according to the first deviation time, the first number of frames, and a preset first buffer time, so that the first audio stream performs clock deviation compensation according to the compensation decision, specifically includes: Calculate the product of the first buffer time and the first number of frames to obtain a second buffer time; When the first audio segment is at the first position of the transmission bit sequence of the multiple audio segments, a compensation decision for the first audio stream is generated according to the first deviation time and the second buffering time, and a preset first buffering data is operated according to the compensation decision to perform clock deviation compensation for the first audio stream; wherein the first buffering data corresponds to the first buffering time; When the first audio segment is not at the first position in the sending order of the multiple audio segments, a compensation decision for the first audio stream is generated according to the first deviation time and the second buffering time, and an input buffer corresponding to the first audio stream is operated according to the compensation decision to perform clock deviation compensation for the first audio stream.

3. The method for compensating audio clock deviation based on dynamic decision making according to claim 2, characterized in that: The method further comprises: generating a compensation decision for the first audio stream according to the first deviation time and the second buffer time when the first audio segment is at the first position of the transmission bit sequence of the multiple audio segments, and operating the preset first buffer data according to the compensation decision to perform clock deviation compensation on the first audio stream, specifically comprising: When the first audio segment is at the first position in the sending sequence of the multiple audio segments, calculating the difference between the first deviation time and the second buffer time to obtain a second deviation time; When the second deviation time is a positive value, a first compensation decision for the first audio stream is generated, and the first buffered data is discarded according to the first compensation decision, and clock deviation compensation is performed on the first audio stream according to a preset first data consumption method; When the second deviation time is a negative value, a second compensation decision for the first audio stream is generated, and the first buffered data is inserted into an output buffer corresponding to the first audio stream according to the second compensation decision to compensate for the clock deviation of the first audio stream.

4. The method for compensating audio clock deviation based on dynamic decision making according to claim 3, characterized in that: The first data consumption method specifically includes: The first audio data in the input buffer corresponding to the first audio stream is resampled. During the resampling process, the first type of sampling points are removed once for each frame of the first audio data in chronological order until the sum of the sampling durations of all the removed first type of sampling points reaches a preset first cumulative time, and the resampling is completed. The resampled first audio data is output to complete the clock deviation compensation for the first audio stream; wherein the sampling duration of the first type of sampling points is the preset first sampling time.

5. The method for compensating audio clock deviation based on dynamic decision making according to claim 2, characterized in that: The step of generating a compensation decision for the first audio stream according to the first deviation time and the second buffering time when the first audio segment is not at the first position of the transmission bit sequence of the multiple audio segments, and operating an input buffer corresponding to the first audio stream according to the compensation decision to perform clock deviation compensation on the first audio stream specifically includes: When the first audio segment is not at the first position of the sending position sequence of the multiple audio segments, calculating an absolute value of a difference between the first deviation time and the second buffer time to obtain a third deviation time; When the third deviation time is equal to a preset value and the first deviation time is greater than the second buffering time, generating a third compensation decision for the audio stream, and performing clock deviation compensation on the first audio stream based on a preset second data consumption method according to the third compensation decision; When the third deviation time is equal to a preset value and the first deviation time is less than the second buffering time, a fourth compensation decision for the audio stream is generated, and according to the fourth compensation decision, clock deviation compensation is performed on the first audio stream based on a preset first data compensation method.

6. An audio clock deviation compensation device based on dynamic decision making, characterized in that: It includes an audio acquisition module and an audio processing module; The audio acquisition module is used to acquire multiple audio clips sent successively by a first audio stream; wherein the first audio stream is established when data is transmitted between a local device and a remote device; The audio processing module is used to perform deviation compensation processing on the multiple audio clips according to a preset first deviation compensation method, and perform clock deviation compensation on the multiple audio clips respectively during the processing until the multiple audio clips are processed; Wherein, the first deviation compensation method comprises: Obtaining a first time of a local clock corresponding to when the first audio stream is established; Acquire a first frame number of a first audio segment currently being processed and a second time of a remote clock corresponding to the first audio segment; Calculate the difference between the second time and the first time to obtain a first deviation time; Based on the sending bit order of the first audio segment in the multiple audio segments, and according to the first deviation time, the first number of frames and the preset first buffer time, a compensation decision for the first audio stream is generated, so that the first audio stream performs clock deviation compensation according to the compensation decision.

7. The audio clock deviation compensation device based on dynamic decision making according to claim 6, characterized in that: The audio processing module includes a decision compensation submodule; the decision compensation submodule includes a second buffer calculation unit, a first order compensation unit and a second order compensation unit; The second buffer calculation unit is used to calculate the product of the first buffer time and the first frame number to obtain a second buffer time; The first order compensation unit is configured to generate a compensation decision for the first audio stream according to the first deviation time and the second buffering time when the first audio segment is at the first position of the transmission bit sequence of the multiple audio segments, and operate the preset first buffering data according to the compensation decision to perform clock deviation compensation on the first audio stream; wherein the first buffering data corresponds to the first buffering time; The second order compensation unit is used to generate a compensation decision for the first audio stream according to the first deviation time and the second buffering time when the first audio segment is not at the first position in the sending position order of the multiple audio segments, and operate the input buffer corresponding to the first audio stream according to the compensation decision to compensate the clock deviation of the first audio stream.

8. The audio clock deviation compensation device based on dynamic decision making according to claim 7, characterized in that: The first order compensation unit includes a second deviation calculation subunit, a first compensation subunit and a second compensation subunit; The second deviation calculation subunit is configured to calculate a difference between the first deviation time and the second buffer time to obtain a second deviation time when the first audio segment is at the first position in the sending sequence of the multiple audio segments; The first compensation subunit is used for generating a first compensation decision for the first audio stream when the second deviation time is a positive value, discarding the first buffered data according to the first compensation decision, and performing clock deviation compensation for the first audio stream according to a preset first data consumption method; The second compensation subunit is used to generate a second compensation decision for the first audio stream when the second deviation time is a negative value, and insert the first buffered data into an output buffer corresponding to the first audio stream according to the second compensation decision to compensate for the clock deviation of the first audio stream.

9. The audio clock deviation compensation device based on dynamic decision making according to claim 8, characterized in that: The first data consumption method specifically includes: The first audio data in the input buffer corresponding to the first audio stream is resampled. During the resampling process, the first type of sampling points are removed once for each frame of the first audio data in chronological order until the sum of the sampling durations of all the removed first type of sampling points reaches a preset first cumulative time, and the resampling is completed. The resampled first audio data is output to complete the clock deviation compensation for the first audio stream; wherein the sampling duration of the first type of sampling points is the preset first sampling time.

10. The audio clock deviation compensation device based on dynamic decision according to claim 7, characterized in that: The second order compensation unit includes a third deviation calculation subunit, a third compensation subunit and a fourth compensation subunit; The third deviation calculation subunit is configured to calculate an absolute value of a difference between the first deviation time and the second buffer time to obtain a third deviation time when the first audio segment is not at the first position in the sending sequence of the multiple audio segments; The third compensation subunit is configured to generate a third compensation decision for the audio stream when the third deviation time is equal to a preset value and the first deviation time is greater than the second buffering time, and perform clock deviation compensation on the first audio stream based on the third compensation decision and based on a preset second data consumption method; The fourth compensation subunit is used to generate a fourth compensation decision for the audio stream when the third deviation time is equal to a preset value and the first deviation time is less than the second buffering time, and according to the fourth compensation decision, perform clock deviation compensation on the first audio stream based on the preset first data compensation method.

Images