Speaker playback method and system based on audio processing

Abstract

The present invention relates to the field of speech processing technology, and in particular to a speaker playback method and system based on audio processing, the method comprising: obtaining a first audio signal and a second audio signal; determining a first duration of transmission of a sound signal in the first audio signal from a microphone to a speaker, a first distance between the microphone and the speaker, and a second duration; determining an initial echo delay according to the first duration, the second duration, the first distance, and the speed of sound; recording a first time period corresponding to the initial echo delay as a pure segment, and recording a second time period adjacent to the first time period as a superposition segment; optimizing the initial echo delay based on a first sound frequency of the pure segment and a second sound frequency of the superposition segment to obtain a final echo delay; and eliminating the echo signal based on the final echo delay. In this way, the present invention improves the accuracy of the determined echo delay of the speaker, thereby improving the effect of echo cancellation.

Description

Technical Field

[0001] The present invention relates to the technical field of speech processing, and in particular to a speaker playing method and system based on audio processing. Background Art

[0002] A speaker is a device in an audio system that converts electrical signals into sound waves, allowing people to hear sounds. In a conference room, when a speaker speaks through a microphone, the sound is played through the speaker. If the sound played by the speaker is received by the microphone again, an echo will be generated. This echo is mixed with the original sound, resulting in sound overlap, making the speaker's voice unclear and difficult to understand. The echo also increases background noise and reduces the overall quality of the audio. Acoustic Echo Cancellation (ACE) is a technology used to eliminate acoustic echoes and is widely used in video conferencing, telephone conferencing, and other real-time communication systems. When the ACE algorithm eliminates echoes, it first establishes an acoustic path model between the speaker and the microphone, then uses an adaptive filter to estimate the echo signal, and then subtracts the estimated echo signal from the total signal received by the microphone to eliminate the echo.

[0003] In a scenario where multiple people are speaking, the distance between each speaker and the speaker is different, resulting in different echo delays received by each person's microphone. Due to the different distances between each speaker and the speaker, the echo delays are different, and because each speaker affects each other when speaking, the accuracy of the determined speaker's echo delay is low, which in turn affects the effect of echo cancellation and reduces the user experience. Summary of the invention

[0004] In order to solve the technical problem that the accuracy of determining the echo delay is low, resulting in poor echo cancellation effect, the purpose of the present invention is to provide a speaker playback method and system based on audio processing, and the technical solution adopted is as follows:

[0005] In a first aspect, an embodiment of the present invention provides a speaker playback method based on audio processing, comprising: obtaining a first audio signal and a second audio signal, the first audio signal comprising a sound signal and an echo signal of a speaking user collected by a microphone, and the second audio signal comprising an audio after the first audio signal is amplified by the speaker; determining a first duration for the sound signal in the first audio signal to be transmitted from the microphone to the speaker, a first distance between the microphone and the speaker, and a second duration for the echo signal to be transmitted to an echo processing module of the microphone; determining an initial echo delay of the speaking user according to the first duration, the second duration, the first distance, and the speed of sound; recording a first time period corresponding to the initial echo delay as a pure segment, and recording a second time period adjacent to the first time period as a superimposed segment, the pure segment comprising a sound signal, and the superimposed segment comprising the first audio signal; optimizing the initial echo delay based on a first sound frequency of the pure segment and a second sound frequency of the superimposed segment to obtain a final echo delay; and eliminating the echo signal based on the final echo delay.

[0006] Optionally, determining the initial echo delay of the speaking user according to the first duration, the second duration, the first distance and the speed of sound includes: calculating a first product of the speed of sound and the first distance; and determining the sum of the first duration, the first product and the second duration as the initial echo delay.

[0007] Optionally, the initial echo delay is optimized based on the first sound frequency of the pure segment and the second sound frequency of the superimposed segment to obtain the final echo delay, which includes: recording a first ratio of the first sound frequency of each maximum point in the pure segment to the second sound frequency of the corresponding extreme point in the superimposed segment as a scaling ratio; stretching the third audio signal of the superimposed segment at the scaling ratio to obtain a fourth audio signal of the superimposed segment; decomposing the first audio signal of the pure segment and the fourth audio signal of the superimposed segment to obtain a first IMF component of the pure segment and a second IMF component of the superimposed segment; recording the first IMF component closest to the sound frequency in the superimposed segment as the superimposed segment; The method comprises the following steps: determining the IMF component of the second IMF component of the superimposed segment in the pure segment; determining the similarity between the second IMF component and the first IMF component of the second IMF component of the superimposed segment in the pure segment according to the first IMF component and the second IMF component; recording the average of the similarities between all the second IMF components of the superimposed segment under each scaling ratio and the first IMF component of the corresponding pure segment as the matching degree of each scaling ratio; recording the superimposed segment under the scaling ratio with the highest matching degree as the optimal superimposed segment of the initial echo delay; optimizing the initial echo delay by using the optimal superimposed segment and the pure segment of the initial echo delay to obtain the final echo delay.

[0008] Optionally, determining the similarity between the second IMF component and the first IMF component corresponding to the second IMF component of the superimposed segment in the pure segment based on the first IMF component and the second IMF component includes: determining the number of first zero crossing points of the second IMF component of the superimposed segment, the number of second zero crossing points of the first IMF component corresponding to the second IMF component of the superimposed segment in the pure segment, the first sound frequency of the second IMF component at any time in the superimposed segment, and the second sound frequency of the first IMF component corresponding to the second IMF component of the superimposed segment in the pure segment at any time in the pure segment; calculating the absolute value of the first difference between the first number of zero crossing points and the second number of zero crossing points; calculating the absolute value of the second difference between the first sound frequency and the second sound frequency, and superimposing the absolute values ​​of the second difference corresponding to each time to obtain the superimposed frequency difference; determining the second product between the absolute value of the first difference and the superimposed frequency difference, and inversely normalizing the second product to obtain the similarity between the second IMF component and the first IMF component corresponding to the second IMF component of the superimposed segment in the pure segment.

[0009] Optionally, optimizing the initial echo delay by using the optimal superposition segment and the pure segment of the initial echo delay to obtain the final echo delay includes: determining a first maximum sound frequency and a first minimum sound frequency in the optimal superposition segment of the initial echo delay, and determining a second maximum sound frequency and a second minimum sound frequency in the pure segment of the initial echo delay; determining a first average distance between adjacent extreme value points in the optimal superposition segment of the initial echo delay, and a second average distance between adjacent extreme value points in the pure segment of the initial echo delay; and determining a first average distance between adjacent extreme value points in the optimal superposition segment of the initial echo delay according to the first maximum sound frequency, the first minimum sound frequency, the second maximum sound frequency, the second minimum sound frequency, the first average distance, and a second average distance, calculating a first oscillation difference of an initial echo delay; determining a third product of the first oscillation difference and the degree of matching corresponding to an optimal superposition segment as a delay confidence evaluation of the initial echo delay; determining an echo delay range according to the initial echo delay, the echo delay range including a plurality of preset echo delays; respectively determining a second oscillation difference of each preset echo delay; determining a fourth product of the second oscillation difference and the degree of matching corresponding to an optimal superposition segment of the preset echo delay as a delay confidence evaluation of each preset echo delay; selecting an echo delay corresponding to a maximum value from the delay confidence evaluation of the initial echo delay and the delay confidence evaluations of each preset echo delay as the final echo delay.

[0010] Optionally, calculating the first oscillation difference of the initial echo delay based on the first maximum sound frequency, the first minimum sound frequency, the second maximum sound frequency, the second minimum sound frequency, the first average distance and the second average distance includes: calculating the second ratio between the first maximum sound frequency and the first minimum sound frequency, and the third ratio between the second maximum sound frequency and the second minimum sound frequency; calculating the absolute value of the third difference between the second ratio and the third ratio, and the absolute value of the fourth difference between the first average distance and the second average distance; and normalizing the fifth product between the absolute value of the third difference and the absolute value of the fourth difference to obtain the first oscillation difference of the initial echo delay.

[0011] Optionally, eliminating the echo signal based on the final echo delay includes: determining an initial weight coefficient of the filter; processing the sound signal using the filter to obtain an analog echo signal; calculating the error between the analog echo signal and the sound signal; determining an iterative equation of the weight coefficient of the filter using the final echo delay, the initial weight coefficient of the filter, the error and the sound signal; iterating the iterative equation to update the initial weight coefficient of the filter until convergence to obtain an updated filter; and eliminating the echo signal using the updated filter.

[0012] Optionally, the iterative equation for determining the weight coefficient of the filter through the final echo delay, the initial weight coefficient of the filter, the error and the sound signal includes: determining the step factor of the filter according to the initial step factor of the filter and the final echo delay; determining the sum of the sixth product of twice the step factor, the error and the sound signal and the initial weight coefficient as the iterative equation.

[0013] Optionally, determining the step factor of the filter according to the initial step factor of the filter and the final echo delay includes: calculating the sum of a preset value and the final echo delay; and determining a fourth ratio of the initial step factor to the sum as the step factor of the filter.

[0014] In a second aspect, an embodiment of the present invention provides a speaker playback system based on audio processing, comprising: a processor and a memory; wherein the memory is used to store a computer program that can be run on the processor; and the processor is used to execute the program stored in the memory to implement the steps of the speaker playback method based on audio processing as mentioned in the first aspect.

[0015] The present invention has the following beneficial effects: firstly, a first audio signal and a second audio signal are obtained, wherein the first audio signal includes a sound signal and an echo signal of a speaking user collected by a microphone, and the second audio signal includes an audio after the first audio signal is amplified by a speaker; then, a first duration of transmission of the sound signal in the first audio signal from the microphone to the speaker, a first distance between the microphone and the speaker, and a second duration of transmission of the echo signal to the microphone are determined; then, an initial echo delay of the speaking user is determined according to the first duration, the second duration, the first distance, and the speed of sound; secondly, a first time period corresponding to the initial echo delay is recorded as a pure section, and a second time period adjacent to the first time period is recorded as a superposition section, wherein the pure section includes the sound signal, and the superposition section includes the first audio signal; and the initial echo delay is optimized based on the first sound frequency of the pure section and the second sound frequency of the superposition section to obtain a final echo delay; finally, the echo signal is eliminated based on the final echo delay.

[0016] In this way, the embodiment of the present invention can obtain the audio data collected by the microphone and the audio data played by the speaker, and then preliminarily estimate the initial echo delay of the current speaker according to information such as the distance between the microphone and the speaker. Furthermore, the initial echo delay is optimized according to the audio data in the pure segment and the superimposed segment in the initial echo delay, so as to obtain the final echo delay. In this way, the accuracy of the determined echo delay of the speaker is improved, thereby improving the effect of echo cancellation and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A flowchart of a speaker playing method based on audio processing provided by one embodiment of the present invention;

[0019] Figure 2 A schematic diagram of the module composition of a speaker playback system based on audio processing provided by one embodiment of the present invention;

[0020] Figure 3 A schematic structural diagram of a speaker playback system based on audio processing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation, structure, features and effects of a speaker playback method and system based on audio processing proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0022] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0023] The specific scheme of a speaker playing method and system based on audio processing provided by the present invention is described in detail below with reference to the accompanying drawings.

[0024] Embodiment 1:

[0025] See also Figure 1 , which shows a flow chart of a speaker playing method based on audio processing provided by an embodiment of the present invention, including:

[0026] S101: Acquire a first audio signal and a second audio signal.

[0027] The first audio signal includes a voice signal and an echo signal of a speaking user collected by a microphone, and the second audio signal includes audio obtained by amplifying the first audio signal through a speaker.

[0028] Specifically, the audio of all microphones is collected in real time, and the sampling frequency can be set to 44.1kHz to obtain the audio signal corresponding to each microphone at each moment, wherein the audio signal collected by the microphone includes the real-time speech of the speaking user, and also includes the echo after the speaker is amplified. The sound collected by the microphone is amplified by the speaker and then output. The audio amplified by the speaker is then obtained for subsequent echo modeling. In addition, the distance from each microphone to the speaker can be determined according to the number of speakers. If there are multiple speakers, the average value of the distance from the microphone to all speakers is determined as the distance from the microphone to the speaker.

[0029] Furthermore, the specific process of echo generation is: the speaker speaks to the microphone, the speech content is transmitted by the speaker, and the sound transmitted by the speaker is transmitted to the microphone again to form an echo. Since the intensity of the sound will gradually weaken during the propagation process, the embodiment of the present invention only considers one echo. If the echo is not processed, the echo and the speech content are mixed together to form a composite voice signal, which affects the audience's recognition of the subsequent speech content and affects the quality of the meeting.

[0030] S102, determining a first duration for a sound signal in a first audio signal to be transmitted from a microphone to a speaker, a first distance between the microphone and the speaker, and a second duration for an echo signal to be transmitted to an echo processing module of the microphone.

[0031] Specifically, when the speaking user speaks to the microphone, since the distance between the speaking user and the microphone is very close, the time spent on this process is very short and can be ignored. After the microphone receives the speaker's voice signal, it converts the voice signal into an electrical signal, and then amplifies and processes the electrical signal and transmits it to the speaker. The speaker converts the electrical signal into a voice signal and then sends it out. This process usually takes tens of milliseconds. The specific duration depends on the device performance and the complexity of the calculation. In the embodiment of the present invention, the first duration of 20ms is taken as an example. The time spent on the sound transmitted from the speaker to the microphone again depends on the speed of sound and the distance from the speaker to the microphone. In the embodiment of the present invention, the speed of sound is 340m / s under 15 degrees Celsius and 1 standard atmospheric pressure. The echo delay from the sound transmitted from the speaker to the microphone at this stage = the speed of sound × the distance from the speaker to the microphone. In addition to the above processes, before processing the echo signal, it also takes a certain amount of time for the echo signal to reach the microphone and then be transmitted from the microphone to the echo processing module. The specific transmission speed depends on the device performance. In the embodiment of the present invention, the second duration of 5ms is taken as an example.

[0032] S103: Determine an initial echo delay of the speaking user according to the first duration, the second duration, the first distance, and the speed of sound.

[0033] Specifically, when determining the initial echo delay of the speaking user, as an optional embodiment of the present invention, firstly, the first product of the speed of sound and the first distance is calculated; then the sum of the first duration, the first product and the second duration is determined as the initial echo delay.

[0034] The embodiment of the present invention uses the following formula to calculate the initial echo delay of the speaking user:

[0035]

[0036] In the above formula, Indicates the initial echo delay. Indicates the first duration of the process from the speaker's voice being collected by the microphone to the speaker's output, which is 20ms here. represents the speed of sound, Indicates the distance between the microphone and the speaker of the current speaker. Indicates the second duration of the echo signal from the microphone to the echo processing module, which is 5ms here.

[0037] S104: Record a first time period corresponding to the initial echo delay as a pure time period, and record a second time period adjacent to the first time period as a superposition time period.

[0038] The pure segment includes a sound signal, and the superimposed segment includes a first audio signal.

[0039] Specifically, the echo delay refers to the time required from the speaker starting to speak to the generation of the echo. Before this time, the audio data collected by the microphone is not affected by the echo. After the speaker starts to speak and the length of the echo delay has passed, the audio data collected by the microphone will have echo interference, that is, the subsequent part of the speaker's speech content overlaps with the previous part. The above steps of the embodiment of the present invention preliminarily estimate the initial echo delay. The size of the initial echo delay is hour, The number of all moments in the time period is recorded as T, and the audio signals collected by the microphone from the 0th to the Tth moment are recorded as the pure segment. The audio signals from the Tth to the 2Tth moment are recorded as the superposition segment. For example, the speech content of the user who spoke from the 0th to the Tth moment is "ABC", and the echo appears at the Tth moment. The speech content of the user who spoke from the Tth to the 2Tth moment is "DEF". Then the speech content from the Tth to the 2Tth moment affected by the echo is the superposition of "DEF" and "ABC" after the propagation loss.

[0040] S105, optimizing the initial echo delay based on the first sound frequency of the pure segment and the second sound frequency of the superimposed segment to obtain a final echo delay.

[0041] Specifically, when determining the final echo delay, the embodiment of the present invention may use an empirical mode decomposition (EMD) algorithm to decompose the audio signal, and determine the final echo delay based on a number of IMF components from high frequency to low frequency obtained by the decomposition. Since the echo that appears in the time period from T to 2T is after propagation loss, when the EMD algorithm is used for decomposition, in order to obtain more accurate results in the subsequent analysis of the similarity of the decomposition results of the pure segment and the superposition segment, the decomposition result of the superposition segment is scaled to obtain a result in which the echo part of the decomposition result is closer to the state before the loss.

[0042] As an optional embodiment of the present invention, a first ratio of a first sound frequency of each maximum point in a pure segment to a second sound frequency of a corresponding extreme point in a superimposed segment is recorded as a scaling ratio; a third audio signal of the superimposed segment is stretched at the scaling ratio to obtain a fourth audio signal of the superimposed segment; the first audio signal of the pure segment and the fourth audio signal of the superimposed segment are decomposed to obtain a first IMF component of the pure segment and a second IMF component of the superimposed segment; a first IMF component of the superimposed segment closest to the sound frequency in the pure segment is recorded as an IMF component corresponding to the second IMF component of the superimposed segment in the pure segment; similarity between the second IMF component and the first IMF component corresponding to the second IMF component of the superimposed segment in the pure segment is determined according to the first IMF component and the second IMF component; an average of similarities between all second IMF components of the superimposed segment under each scaling ratio and the first IMF component of the corresponding pure segment is recorded as a matching degree of each scaling ratio; the superimposed segment under the scaling ratio with the highest matching degree is recorded as an optimal superimposed segment of an initial echo delay; the initial echo delay is optimized using the optimal superimposed segment and the pure segment of the initial echo delay to obtain a final echo delay.

[0043] Specifically, the ratio of the sound frequency of each maximum point in the pure segment to the sound frequency of the corresponding maximum point in the superimposed segment is obtained as the scaling ratio, and the audio signal of the superimposed segment is stretched at each scaling ratio. The pure segment and each stretched superimposed segment are decomposed using the EMD algorithm to obtain several high-frequency to low-frequency IMF components of the pure segment and each superimposed segment, and the IMF component with the closest frequency between the superimposed segment and the pure segment is recorded as the IMF component corresponding to the IMF component of the superimposed segment in the pure segment, and the similarity of each IMF component in the superimposed segment under each scaling ratio and its corresponding IMF component is compared.

[0044] Further, when determining the similarity between the second IMF component and the first IMF component corresponding to the second IMF component of the superimposed segment in the pure segment, as an optional embodiment of the present invention, first determine the number of first zero crossing points of the second IMF component of the superimposed segment, the number of second zero crossing points of the first IMF component corresponding to the second IMF component of the superimposed segment in the pure segment, the first sound frequency of the second IMF component at any time in the superimposed segment, and the second sound frequency of the first IMF component corresponding to the second IMF component of the superimposed segment in the pure segment at any time in the pure segment; then calculate the absolute value of the first difference between the first number of zero crossing points and the second number of zero crossing points; then calculate the absolute value of the second difference between the first sound frequency and the second sound frequency, and superimpose the absolute values ​​of the second difference corresponding to each time to obtain the superimposed frequency difference; finally determine the second product between the absolute value of the first difference and the superimposed frequency difference, and inversely normalize the second product to obtain the similarity between the second IMF component and the first IMF component corresponding to the second IMF component of the superimposed segment in the pure segment.

[0045] Specifically, the zero crossing point refers to a data point where the value of the curve is zero and the values ​​of the data points on the left and right ends are one positive and one negative. The embodiment of the present invention uses the following formula to calculate the similarity between the second IMF component and the first IMF component corresponding to the second IMF component of the superposition segment in the pure segment:

[0046]

[0047] In the above formula, It represents the similarity between the i-th second IMF component in the superimposed segment at the k-th scaling ratio and the first IMF component corresponding to the i-th second IMF component in the pure segment at the k-th scaling ratio. Indicates the number of first zero crossings of the i-th second IMF component in the overlay segment at the k-th scaling factor. It represents the number of second zero crossing points of the first IMF component corresponding to the i-th second IMF component in the superimposed segment under the k-th scaling ratio in the pure segment. It represents the number of all moments in a pure segment or a superimposed segment, wherein in the embodiment of the present invention, the number of moments in the pure segment and the superimposed segment is the same. It represents the first sound frequency of the i-th second IMF component at the i-th moment in the superposition segment under the k-th scaling ratio. It represents the second sound frequency of the ith second IMF component in the superimposed segment under the kth scaling ratio at the ith moment of the first IMF component corresponding to the pure segment. Indicates taking the absolute value, Represents an exponential function with a natural constant as the base, used for inverse normalization.

[0048] Furthermore, It indicates the difference in the number of zero crossing points between the i-th second IMF component of the superimposed segment and its corresponding first IMF component in the pure segment under the k-th scaling ratio. The larger the value, the greater the difference in the number of zero crossing points between the i-th second IMF component of the superimposed segment and its corresponding first IMF component in the pure segment. The smaller. It reflects the difference in sound frequency between the second IMF component of the i-th superimposed segment at the k-th scaling ratio and the first IMF component corresponding to it in the pure segment at each moment. The larger the value, the greater the frequency difference, and the similarity The smaller.

[0049] Similarly, obtain the similarity between each second IMF component of the superimposed segment at each scaling ratio and the first IMF component corresponding to the pure segment. The average of the similarities between all the second IMF components of the superimposed segment at each scaling ratio and the corresponding first IMF components is recorded as the matching degree at the current scaling ratio. The higher the matching degree, the better the scaling ratio can restore the sound state before the loss, and the superimposed segment at the corresponding scaling ratio is recorded as the optimal superimposed segment.

[0050] Furthermore, since the phase changes after the sound is superimposed, the oscillation frequency and amplitude of the pure segment and the superimposed segment will be different, that is, the amplitude and density of the extreme points will be different. The more accurate the echo delay is, the more accurate the actual pure segment and the superimposed segment are divided. At this time, the difference in oscillation frequency and amplitude reaches the maximum. Therefore, when the initial echo delay is optimized using the optimal superimposed segment and the pure segment, as an optional embodiment of the present invention, the first maximum sound frequency and the first minimum sound frequency in the optimal superimposed segment of the initial echo delay are determined, and the second maximum sound frequency and the second minimum sound frequency in the pure segment of the initial echo delay are determined; the first average distance between adjacent extreme points in the optimal superimposed segment of the initial echo delay and the second average distance between adjacent extreme points in the pure segment of the initial echo delay are determined; according to the first maximum sound frequency, the first minimum sound frequency, the second maximum sound frequency, the second minimum sound frequency, the first average distance and the first minimum sound frequency, the first average distance and the second minimum sound frequency are determined. second average distance, calculate the first oscillation difference of the initial echo delay; determine the third product of the first oscillation difference and the matching degree corresponding to the optimal superposition segment as the delay confidence evaluation of the initial echo delay; determine the echo delay range according to the initial echo delay, and the echo delay range includes multiple preset echo delays; respectively determine the second oscillation difference of each preset echo delay; determine the fourth product of the second oscillation difference and the matching degree corresponding to the optimal superposition segment of the preset echo delay as the delay confidence evaluation of each preset echo delay; select the echo delay corresponding to the maximum value from the delay confidence evaluation of the initial echo delay and the delay confidence evaluation of each preset echo delay as the final echo delay.

[0051] Specifically, when calculating the first oscillation difference of the initial echo delay, as an optional embodiment of the present invention, first, the second ratio between the first maximum sound frequency and the first minimum sound frequency, and the third ratio between the second maximum sound frequency and the second minimum sound frequency are calculated; then, the absolute value of the third difference between the second ratio and the third ratio, and the absolute value of the fourth difference between the first average distance and the second average distance are calculated; secondly, the fifth product between the absolute value of the third difference and the absolute value of the fourth difference is normalized to obtain the first oscillation difference of the initial echo delay.

[0052] The embodiment of the present invention uses the following formula to calculate the first oscillation difference of the initial echo delay:

[0053]

[0054] In the above formula, Indicates the initial echo delay The first shock difference. Indicates that the echo delay is The first maximum sound frequency of the optimal superposition segment. Indicates that the echo delay is The first minimum sound frequency of the optimal superposition segment and is not 0. Indicates that the echo delay is The second highest sound frequency in the pure segment. Indicates that the echo delay is The second minimum sound frequency of the pure segment is not 0. Indicates that the echo delay is The first average distance between adjacent extreme points of the optimal superposition segment is Indicates that the echo delay is The second average distance between adjacent extreme points of the pure segment. Indicates normalization of positive proportion.

[0055] in, The larger the value of, the greater the difference in oscillation amplitude. The larger the value, the greater the difference in oscillation frequency.

[0056] Furthermore, when determining the echo delay range, the preset step size is 1ms and the left boundary is set to , the right boundary is set to The left and right boundaries are determined as the echo delay range, Starting from the echo delay range, several possible preset echo delays are selected. After determining the preset echo delay, the second oscillation difference of each preset echo delay is determined according to the method for determining the oscillation difference of the initial echo delay in the above embodiment of the present invention. When the preset echo delay changes, the pure segment and the optimal superposition segment will also change, and the corresponding matching degree corresponding to the optimal superposition segment of each preset echo delay will also change. When the oscillation difference is larger and the matching degree is larger, the echo delay is more likely to be a true echo delay. The product of the oscillation difference and the matching degree is recorded as the delay confidence evaluation under the current echo delay. The echo delay corresponding to the maximum value of the delay confidence evaluation is selected as the final echo delay. Obtaining an accurate echo delay can improve the performance of echo cancellation to a certain extent, reduce the overhead of the filter in the echo cancellation algorithm during signal alignment, and improve system performance.

[0057] S106: Eliminate the echo signal based on the final echo delay.

[0058] Specifically, after determining the final echo delay, echo cancellation is performed using an acoustic echo cancellation (AEC) algorithm. In the embodiment of the present invention, the second audio signal played by the speaker is recorded as the initial input signal , which is the voice signal of the speaking user. The first audio signal actually collected by the microphone is recorded as the actual signal Since the actual signal is the superposition of the initial input signal and the echo signal, the actual signal can be expressed as: .in, It represents the echo signal.

[0059] Furthermore, when eliminating the echo signal, as an optional embodiment of the present invention, first determine the initial weight coefficient of the filter; then use the filter to process the sound signal to obtain an analog echo signal; secondly calculate the error between the analog echo signal and the sound signal; then determine the iterative equation of the weight coefficient of the filter through the final echo delay, the initial weight coefficient of the filter, the error and the sound signal; and iterate the iterative equation to update the initial weight coefficient of the filter until convergence to obtain an updated filter; finally, use the updated filter to eliminate the echo signal.

[0060] Specifically, the filter in the embodiment of the present invention can adopt an adaptive filter (Normalized Least Mean Squares, NLMS), and use the NLMS filter to model the echo path of the audio signal played by the speaker. Specifically, the NLMS filter is first used to filter the initial input signal Processing, initialize the weights of the NLMS filter , the initial input signal Input to NLMS filter to obtain analog echo signal , simulated echo signal Reflects the echo signal obtained by the NLMS filter Then calculate the simulated echo signal and the initial input signal The error between , expressed as , Represents a signal from which the echo signal has been removed.

[0061] Furthermore, after determining the error, when determining the iterative equation of the weight coefficient of the filter, as an optional embodiment of the present invention, the step factor of the filter is determined based on the initial step factor of the filter and the final echo delay; the sum of the sixth product of twice the step factor, the error and the sound signal and the initial weight coefficient is determined as the iterative equation.

[0062] Specifically, the embodiment of the present invention uses the following formula to represent the iterative equation:

[0063]

[0064] In the above formula, represents the weight parameter of the filter for the n+1th iteration, represents the weight parameter of the filter at the nth iteration, represents the step size factor, Indicates the error, Represents the initial input signal.

[0065] Furthermore, the weight parameters of the filter are updated by minimizing the mean square error until convergence.

[0066] Further, when determining the step factor of the filter, as an optional embodiment of the present invention, the sum of the preset value and the final echo delay is calculated; and a fourth ratio of the initial step factor and the sum is determined as the step factor of the filter.

[0067] Specifically, the preset value can be determined according to the actual situation, and the value in the embodiment of the present invention is 1. When setting the initial step size factor, a value between 0.001 and 0.1 is usually selected. When the echo signal is relatively stable, a larger step size factor can be selected to increase the echo modeling speed. When the echo is unstable, a smaller step size factor is selected to ensure modeling stability. When the echo delay is larger, the period of the microphone collecting the echo signal is longer, and the process is more susceptible to noise interference, that is, the echo is more unstable. Therefore, when the echo delay is larger, a smaller step size factor value should be selected to ensure the stability of the filter iteration process. In the embodiment of the present invention, the initial step size factor is preset to .

[0068] Furthermore, in the embodiment of the present invention, the step size factor may be calculated using the following formula:

[0069]

[0070] In the above formula, represents the step size factor of the filter, represents the initial step size factor, Indicates the final echo delay.

[0071] Through the above-mentioned records of the embodiments of the present invention, the echo of the speaking user is eliminated, and the echo delay and related parameters are updated after changing different speakers, so as to realize adaptive echo cancellation. Because there will be obvious intervals between the speeches of different speakers in the meeting, it is possible to distinguish whether to switch speakers by setting a pause threshold. For example, the pause threshold is set to 1.5s. When the audio data collected by the microphone pauses for more than 1.5s, it is considered that the speaker has been changed, and the echo delay and step factor and other related parameters need to be updated again. It is worth noting that the principle of echo cancellation through the AEC algorithm can also be found in the prior art, and the embodiments of the present invention will not be repeated here.

[0072] The embodiment of the present invention can obtain the audio data collected by the microphone and the audio data played by the speaker, and then preliminarily estimate the initial echo delay of the current speaker according to information such as the distance between the microphone and the speaker. Furthermore, the initial echo delay is optimized according to the audio data in the pure segment and the superimposed segment in the initial echo delay, so as to obtain the final echo delay. In this way, the accuracy of the determined echo delay of the speaker is improved, thereby improving the effect of echo cancellation and improving the user experience.

[0073] Embodiment 2:

[0074] Corresponding to the speaker playing method based on audio processing provided in the above embodiment, based on the same technical concept, an embodiment of the present invention further provides a speaker playing system based on audio processing, and the speaker playing system based on audio processing is used to execute the above speaker playing method based on audio processing. Figure 2 A schematic diagram of the module composition of a speaker playback system based on audio processing provided by an embodiment of the present invention is shown in FIG. Figure 2As shown. The audio processing-based speaker playback system 200 includes: an acquisition module 201, which is used to acquire a first audio signal and a second audio signal, wherein the first audio signal includes a sound signal and an echo signal of a speaking user collected by a microphone, and the second audio signal includes an audio after the first audio signal is amplified by the speaker; a determination module 202, which is used to determine a first duration of the sound signal in the first audio signal being transmitted from the microphone to the speaker, a first distance between the microphone and the speaker, and a second duration of the echo signal being transmitted to the echo processing module of the microphone; the determination module 202, which is also used to determine the initial echo delay of the speaking user according to the first duration, the second duration, the first distance, and the speed of sound; the determination module 202, which is also used to record a first time period corresponding to the initial echo delay as a pure section, and record a second time period adjacent to the first time period as a superposition section, wherein the pure section includes a sound signal, and the superposition section includes the first audio signal; an optimization module 203, which is used to optimize the initial echo delay based on a first sound frequency of the pure section and a second sound frequency of the superposition section to obtain a final echo delay; and an elimination module 204, which is used to eliminate the echo signal based on the final echo delay.

[0075] The embodiment of the present invention can obtain the audio data collected by the microphone and the audio data played by the speaker, and then preliminarily estimate the initial echo delay of the current speaker according to information such as the distance between the microphone and the speaker. Furthermore, the initial echo delay is optimized according to the audio data in the pure segment and the superimposed segment in the initial echo delay, so as to obtain the final echo delay. In this way, the accuracy of the determined echo delay of the speaker is improved, thereby improving the effect of echo cancellation and improving the user experience.

[0076] Optionally, the determination module 202 is further configured to calculate a first product of the speed of sound and the first distance; and determine the sum of the first duration, the first product, and the second duration as the initial echo delay.

[0077] Optionally, the optimization module 203 is further used to record the first ratio of the first sound frequency of each maximum point in the pure segment to the second sound frequency of the corresponding extreme point in the superimposed segment as a scaling ratio; stretch the third audio signal of the superimposed segment at the scaling ratio to obtain the fourth audio signal of the superimposed segment; decompose the first audio signal of the pure segment and the fourth audio signal of the superimposed segment to obtain the first IMF component of the pure segment and the second IMF component of the superimposed segment; record the first IMF component of the superimposed segment closest to the sound frequency in the pure segment as the IMF component corresponding to the second IMF component of the superimposed segment in the pure segment; determine the similarity between the second IMF component and the first IMF component corresponding to the second IMF component of the superimposed segment in the pure segment according to the first IMF component and the second IMF component; record the average of the similarities between all the second IMF components of the superimposed segment under each scaling ratio and the first IMF component of the corresponding pure segment as the matching degree of each scaling ratio; record the superimposed segment under the scaling ratio with the highest matching degree as the optimal superimposed segment of the initial echo delay; optimize the initial echo delay using the optimal superimposed segment and the pure segment of the initial echo delay to obtain the final echo delay.

[0078] Optionally, the determination module 202 is also used to determine the number of first zero-crossing points of the second IMF component of the superimposed segment, the number of second zero-crossing points of the first IMF component corresponding to the second IMF component of the superimposed segment in the pure segment, the first sound frequency of the second IMF component at any time in the superimposed segment, and the second sound frequency of the first IMF component corresponding to the second IMF component of the superimposed segment in the pure segment at any time in the pure segment; calculate the absolute value of the first difference between the first number of zero-crossing points and the second number of zero-crossing points; calculate the absolute value of the second difference between the first sound frequency and the second sound frequency, and superimpose the absolute values ​​of the second difference corresponding to each moment to obtain the superimposed frequency difference; determine the second product between the absolute value of the first difference and the superimposed frequency difference, and inversely normalize the second product to obtain the similarity between the second IMF component and the first IMF component corresponding to the second IMF component of the superimposed segment in the pure segment.

[0079] Optionally, the optimization module 203 is further used to determine a first maximum sound frequency and a first minimum sound frequency in an optimal superposition segment of the initial echo delay, and to determine a second maximum sound frequency and a second minimum sound frequency in a pure segment of the initial echo delay; determine a first average distance between adjacent extreme value points in the optimal superposition segment of the initial echo delay, and a second average distance between adjacent extreme value points in the pure segment of the initial echo delay; calculate the initial echo delay according to the first maximum sound frequency, the first minimum sound frequency, the second maximum sound frequency, the second minimum sound frequency, the first average distance, and the second average distance. The invention relates to a method for determining a first oscillation difference of a first echo delay and a first oscillation difference of a first echo delay; determining a third product of the matching degree of the first echo delay and the optimal superposition segment as a delay confidence evaluation of the initial echo delay; determining an echo delay range according to the initial echo delay, wherein the echo delay range includes a plurality of preset echo delays; respectively determining a second oscillation difference of each preset echo delay; determining a fourth product of the matching degree of the second echo delay and the optimal superposition segment of the preset echo delay as a delay confidence evaluation of each preset echo delay; selecting an echo delay corresponding to a maximum value from the delay confidence evaluation of the initial echo delay and the delay confidence evaluation of each preset echo delay as the final echo delay.

[0080] Optionally, the optimization module 203 is also used to calculate a second ratio between the first maximum sound frequency and the first minimum sound frequency, and a third ratio between the second maximum sound frequency and the second minimum sound frequency; calculate the absolute value of a third difference between the second ratio and the third ratio, and the absolute value of a fourth difference between the first average distance and the second average distance; and normalize the fifth product between the absolute value of the third difference and the absolute value of the fourth difference to obtain a first oscillation difference of the initial echo delay.

[0081] Optionally, the elimination module 204 is also used to determine the initial weight coefficient of the filter; use the filter to process the sound signal to obtain an analog echo signal; calculate the error between the analog echo signal and the sound signal; determine the iterative equation of the weight coefficient of the filter through the final echo delay, the initial weight coefficient of the filter, the error and the sound signal; iterate the iterative equation to update the initial weight coefficient of the filter until convergence to obtain an updated filter; and use the updated filter to eliminate the echo signal.

[0082] Optionally, the determination module 202 is further used to determine the step factor of the filter according to the initial step factor of the filter and the final echo delay; determine the sum of the sixth product of the double of the step factor, the error and the sound signal and the initial weight coefficient as an iterative equation.

[0083] Optionally, the determination module 202 is further configured to calculate a sum of a preset value and a final echo delay; and determine a fourth ratio of the initial step factor to the sum as a step factor of the filter.

[0084] Embodiment three:

[0085] Corresponding to the speaker playing method based on audio processing provided in the above embodiment, based on the same technical concept, an embodiment of the present invention further provides a speaker playing system based on audio processing, and the speaker playing system based on audio processing is used to execute the above speaker playing method based on audio processing. Figure 3 A schematic diagram of a speaker playback system based on audio processing provided by an embodiment of the present invention is shown in FIG. Figure 3 The speaker playback system based on audio processing may have relatively large differences due to different configurations or performances, and may include one or more processors 301 and memory 302, the memory 302 is used to store computer programs that can be run on the processor 301, and the processor 301 is used to execute the program stored in the memory 302 to achieve the above Figure 1 The memory 302 may be a temporary storage or a permanent storage. The application stored in the memory 302 may include one or more modules (not shown in the figure), each of which may include a series of computer executable instructions in the speaker playback system based on audio processing.

[0086] Furthermore, the processor 301 can be configured to communicate with the memory 302 to execute a series of computer executable instructions in the memory 302 on the audio processing-based speaker playback system. The audio processing-based speaker playback system can also include one or more power supplies 303, one or more wired or wireless network interfaces 304, one or more input and output interfaces 305, and one or more keyboards 306.

[0087] Specifically in this embodiment, the audio processing-based speaker playback system includes a processor, a communication interface, a memory and a communication bus; wherein the processor, the communication interface and the memory communicate with each other through the bus; the memory is used to store computer programs; the processor is used to execute the programs stored in the memory to achieve the above Figure 1 The various steps in the method embodiment are similar to those in the method embodiment, and have the beneficial effects of the above method embodiments. To avoid repetition, the embodiments of the present invention will not be described again here.

[0088] It should be noted that the speaker playback system based on audio processing provided in an embodiment of the present invention and the speaker playback method based on audio processing provided in an embodiment of the present invention are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned speaker playback method based on audio processing, and has the same or similar beneficial effects, and the repeated parts will not be repeated.

[0089] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0090] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A speaker playing method based on audio processing, characterized in that: The speaker playing method based on audio processing comprises: Acquire a first audio signal and a second audio signal, wherein the first audio signal includes a voice signal and an echo signal of a speaking user collected by a microphone, and the second audio signal includes audio after the first audio signal is amplified by a speaker; Determine a first duration for a sound signal in the first audio signal to be transmitted from the microphone to the speaker, a first distance between the microphone and the speaker, and a second duration for the echo signal to be transmitted to an echo processing module of the microphone; Determine the initial echo delay of the speaking user according to the first duration, the second duration, the first distance, and the speed of sound; Recording a first time period corresponding to the initial echo delay as a pure time period, and recording a second time period adjacent to the first time period as a superposition time period, wherein the pure time period includes the sound signal, and the superposition time period includes the first audio signal; Optimizing the initial echo delay based on the first sound frequency of the pure segment and the second sound frequency of the superimposed segment to obtain a final echo delay; The echo signal is cancelled based on the final echo delay.

2. The speaker playing method based on audio processing according to claim 1, characterized in that: Determining the initial echo delay of the speaking user according to the first duration, the second duration, the first distance, and the speed of sound includes: Calculating a first product of the speed of sound and the first distance; The sum of the first duration, the first product and the second duration is determined as the initial echo delay.

3. The speaker playing method based on audio processing according to claim 1, characterized in that: The optimizing the initial echo delay based on the first sound frequency of the pure segment and the second sound frequency of the superimposed segment to obtain the final echo delay comprises: Recording a first ratio of a first sound frequency of each maximum point in the pure segment to a second sound frequency of a corresponding maximum point in the superposition segment as a scaling ratio; stretching the third audio signal of the superimposed segment at the scaling ratio to obtain a fourth audio signal of the superimposed segment; Decomposing the first audio signal of the pure segment and the fourth audio signal of the superimposed segment to obtain a first IMF component of the pure segment and a second IMF component of the superimposed segment; The first IMF component in the superimposed segment and the pure segment that is closest in sound frequency is recorded as the IMF component corresponding to the second IMF component of the superimposed segment in the pure segment; Determine, according to the first IMF component and the second IMF component, a similarity between the second IMF component and a first IMF component corresponding to the second IMF component of the superposition segment in a pure segment; The average of the similarities between all the second IMF components of the superimposed segment under each scaling ratio and the first IMF components of the corresponding pure segment is recorded as the matching degree of each scaling ratio; Recording the superposition segment at the scaling ratio with the highest matching degree as the optimal superposition segment of the initial echo delay; The initial echo delay is optimized by using the optimal superposition segment of the initial echo delay and the pure segment to obtain a final echo delay.

4. The speaker playing method based on audio processing according to claim 3 is characterized in that: The determining, according to the first IMF component and the second IMF component, the similarity between the second IMF component and the first IMF component corresponding to the pure segment of the second IMF component of the superposition segment comprises: Determine the number of first zero crossing points of the second IMF component of the superimposed segment, the number of second zero crossing points of the first IMF component corresponding to the second IMF component of the superimposed segment in the pure segment, the first sound frequency of the second IMF component at any time in the superimposed segment, and the second sound frequency of the first IMF component corresponding to the second IMF component of the superimposed segment in the pure segment at any time in the pure segment; calculating an absolute value of a first difference between the first number of zero-crossing points and the second number of zero-crossing points; calculating an absolute value of a second difference between the first sound frequency and the second sound frequency, and superimposing the absolute values ​​of the second difference corresponding to each moment to obtain a superimposed frequency difference; A second product between the absolute value of the first difference and the superimposed frequency difference is determined, and the second product is inversely normalized to obtain a similarity between the second IMF component and the second IMF component of the superimposed segment in the first IMF component corresponding to the pure segment.

5. The speaker playing method based on audio processing according to claim 3 is characterized in that: The optimizing the initial echo delay by using the optimal superposition segment of the initial echo delay and the pure segment to obtain the final echo delay comprises: Determining a first maximum sound frequency and a first minimum sound frequency in an optimal superposition section of the initial echo delay, and determining a second maximum sound frequency and a second minimum sound frequency in a pure section of the initial echo delay; Determine a first average distance between adjacent extreme value points in the optimal superposition segment of the initial echo delay, and a second average distance between adjacent extreme value points in the pure segment of the initial echo delay; Calculating a first oscillation difference of the initial echo delay according to the first maximum sound frequency, the first minimum sound frequency, the second maximum sound frequency, the second minimum sound frequency, the first average distance, and the second average distance; Determining a third product of the matching degree of the first oscillation difference and the optimal superposition segment as a delay confidence evaluation of the initial echo delay; Determining an echo delay range according to the initial echo delay, wherein the echo delay range includes a plurality of preset echo delays; respectively determining a second oscillation difference of each of the preset echo delays; Determining a fourth product of the matching degree corresponding to the optimal superposition section of the second oscillation difference and the preset echo delay as the delay confidence evaluation of each of the preset echo delays; An echo delay corresponding to a maximum value is selected from the delay confidence evaluation of the initial echo delay and the delay confidence evaluations of each of the preset echo delays as the final echo delay.

6. The speaker playing method based on audio processing according to claim 5, characterized in that: The calculating the first oscillation difference of the initial echo delay according to the first maximum sound frequency, the first minimum sound frequency, the second maximum sound frequency, the second minimum sound frequency, the first average distance, and the second average distance comprises: calculating a second ratio between the first maximum sound frequency and the first minimum sound frequency, and a third ratio between the second maximum sound frequency and the second minimum sound frequency; calculating an absolute value of a third difference between the second ratio and the third ratio, and an absolute value of a fourth difference between the first average distance and the second average distance; A fifth product of the absolute value of the third difference and the absolute value of the fourth difference is normalized to obtain a first oscillation difference of the initial echo delay.

7. The speaker playing method based on audio processing according to claim 1, characterized in that: The eliminating the echo signal based on the final echo delay comprises: Determine the initial weight coefficients of the filter; Processing the sound signal using the filter to obtain a simulated echo signal; Calculating an error between the simulated echo signal and the sound signal; An iterative equation for determining the weight coefficient of the filter by using the final echo delay, the initial weight coefficient of the filter, the error and the sound signal; Iterating the iterative equation to update the initial weight coefficient of the filter until convergence, to obtain an updated filter; The echo signal is eliminated using the updated filter.

8. The speaker playing method based on audio processing according to claim 7 is characterized in that: The iterative equation for determining the weight coefficient of the filter through the final echo delay, the initial weight coefficient of the filter, the error and the sound signal includes: Determining a step size factor of the filter according to an initial step size factor of the filter and the final echo delay; The sum of the sixth product of twice the step factor, the error and the sound signal and the initial weight coefficient is determined as the iterative equation.

9. The speaker playing method based on audio processing according to claim 8, characterized in that: The step of determining the step factor of the filter according to the initial step factor of the filter and the final echo delay comprises: Calculating a sum of a preset value and the final echo delay; A fourth ratio of the initial step factor to the sum value is determined as the step factor of the filter.

10. A speaker playing system based on audio processing, characterized in that: The audio processing-based speaker playback system includes: a processor and a memory; wherein the memory is used to store a computer program that can be run on the processor; the processor is used to execute the program stored in the memory to implement the steps of the audio processing-based speaker playback method as described in any one of claims 1-9.

Images