Echo cancellation method, system and electronic device based on microphone array

Abstract

The present application provides an echo cancellation method, system and electronic device based on a microphone array, wherein the method comprises: collecting multiple echo signals through a microphone array; the microphone array is located at a first fixed position, and the loudspeaker is located at a second fixed position; according to the position information of the loudspeaker relative to the microphone array, designing a fixed beamforming coefficient for each echo signal, preprocessing the corresponding echo signal, and obtaining multiple preprocessed first signals; filtering the multiple first signals through adaptive filters corresponding to the multiple first signals one by one, and obtaining multiple error signals; adaptively adjusting the beam parameters for each error signal according to the sound source position of the near-end speaker, and merging the multiple error signals into a single signal based on the multiple beam parameters; and eliminating residual nonlinear echoes from the single signal by using a deep learning model.

Description

Technical Field

[0001] The present application relates to the technical field of echo cancellation, and in particular to an echo cancellation method, system and electronic device based on a microphone array. Background Art

[0002] With the popularization of smart devices and the development of communication technology, people's demand for voice communication and audio processing is growing. In various voice interaction scenarios, such as remote meetings, voice assistants, smart homes, etc., the clarity and accuracy of sound quality are crucial. However, echo, as a common audio interference, seriously affects the quality of voice signals and reduces the efficiency and experience of communication.

[0003] On devices with integrated speakers and microphones, since the distance between the speakers and microphones is usually close (for example, 5 cm), the sound attenuation from the speakers to the microphones is small, which increases the difficulty of echo cancellation, thus affecting the comprehensive effect of single-talk and double-talk. How to effectively eliminate echoes has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present application provide an echo cancellation method, system and electronic device based on a microphone array, which are used to effectively cancel echoes.

[0005] In a first aspect, an embodiment of the present application provides an echo cancellation method based on a microphone array, comprising:

[0006] Multiple echo signals are collected by the microphone array; each echo signal is a signal collected by a corresponding microphone in the microphone array after being reflected from a playback signal of a loudspeaker; the microphone array is located at a first fixed position, and the loudspeaker is located at a second fixed position;

[0007] According to the position information of the loudspeaker relative to the microphone array, a fixed beamforming coefficient for each echo signal is designed, and the corresponding echo signal is preprocessed to obtain multiple preprocessed first signals;

[0008] Performing filtering processing on the multiple first signals through adaptive filters corresponding to the multiple first signals one by one, to obtain multiple error signals;

[0009] Adaptively adjusting beam parameters for each error signal according to the sound source position of the near-end speaker, and combining the multiple error signals into a single signal based on the multiple beam parameters;

[0010] The residual nonlinear echo is eliminated from the single-channel signal using a deep learning model; the deep learning model is a pre-built model for learning and identifying nonlinear echo and near-end speech.

[0011] In one possible implementation, designing a fixed beamforming coefficient for each echo signal according to position information of the speaker relative to the microphone array, preprocessing the corresponding echo signal, and obtaining multiple preprocessed first signals includes:

[0012] Determining position information of the speaker relative to the microphone array according to the first fixed position and the second fixed position;

[0013] Determining, according to the position information, a direction of the speaker relative to the microphone array;

[0014] Based on the directions, designing fixed beamforming coefficients for each echo signal;

[0015] By preprocessing the corresponding echo signal with respect to the fixed beamforming coefficient of each echo signal, a plurality of preprocessed first signals are obtained, and the plurality of first signals are suppressed in the direction.

[0016] In one possible implementation manner, filtering the multiple paths of first signals by using adaptive filters corresponding to the multiple paths of first signals one by one to obtain multiple paths of error signals includes:

[0017] The playback signal is used as a reference signal, and after being processed by an adaptive filter corresponding to the multiple first signals one by one, an estimated echo signal is obtained; the estimated echo signal is specifically an estimate of the echo signal on the propagation path of the playback signal from the speaker to the microphone array;

[0018] The estimated echo signal is subtracted from each first signal to obtain a multi-path error signal.

[0019] In one possible implementation, the step of adaptively adjusting the beam parameters for each error signal according to the sound source position of the near-end speaker, and combining the multiple error signals into a single signal based on the multiple beam parameters includes:

[0020] Using sound source localization technology to determine the sound source position of the near-end speaker relative to the microphone array;

[0021] Determining the direction of the near-end speaker according to the sound source position;

[0022] Adaptively adjusting beam parameters for each error signal according to the direction of the near-end speaker to obtain adjusted beam parameters for the corresponding error signal;

[0023] Based on the adjusted beam parameters of each path, the error signal of the corresponding path is processed to obtain a plurality of processed second signals, wherein the plurality of second signals are enhanced in the direction of the near-end speaker;

[0024] Adaptively adjusting a weight coefficient for processing each second signal according to the direction of the near-end speaker;

[0025] Based on the weight coefficient for each second signal, weighted processing is performed on the multiple second signals to obtain a processed single signal.

[0026] In a second aspect, an embodiment of the present application provides an echo cancellation system based on a microphone array, comprising:

[0027] A loudspeaker, used to transmit the playback signal;

[0028] A microphone array, used for collecting multiple echo signals; each echo signal is a signal collected by a corresponding microphone in the microphone array after the playback signal is reflected; the microphone array is located at a first fixed position, and the speaker is located at a second fixed position;

[0029] A fixed beamforming module, used to design a fixed beamforming coefficient for each echo signal according to the position information of the loudspeaker relative to the microphone array, preprocess the corresponding echo signal, and obtain multiple preprocessed first signals;

[0030] An adaptive filtering module, used for filtering the multiple first signals through adaptive filters corresponding to the multiple first signals one by one, to obtain multiple error signals;

[0031] An adaptive beamforming module, used for adaptively adjusting beam parameters for each error signal according to the sound source position of the near-end speaker, and combining the multiple error signals into a single signal based on the multiple beam parameters;

[0032] The echo suppression module is used to eliminate the residual nonlinear echo from the single-channel signal; the deep learning model is a pre-built model for learning and identifying nonlinear echo and near-end speech.

[0033] In one possible implementation manner, the fixed beamforming module is specifically used to:

[0034] Determining position information of the speaker relative to the microphone array according to the first fixed position and the second fixed position;

[0035] Determining, according to the position information, a direction of the speaker relative to the microphone array;

[0036] Based on the directions, designing fixed beamforming coefficients for each echo signal;

[0037] By preprocessing the corresponding echo signal with respect to the fixed beamforming coefficient of each echo signal, a plurality of preprocessed first signals are obtained, and the plurality of first signals are suppressed in the direction.

[0038] In one possible implementation manner, the adaptive filtering module is specifically used to:

[0039] The playback signal is used as a reference signal, and after being processed by an adaptive filter corresponding to the multiple first signals one by one, an estimated echo signal is obtained; the estimated echo signal is specifically an estimate of the echo signal on the propagation path of the playback signal from the speaker to the microphone array;

[0040] The estimated echo signal is subtracted from each first signal to obtain a multi-path error signal.

[0041] In one possible implementation manner, the adaptive beamforming module is specifically used to:

[0042] Using sound source localization technology to determine the sound source position of the near-end speaker relative to the microphone array;

[0043] Determining the direction of the near-end speaker according to the sound source position;

[0044] Adaptively adjusting beam parameters for each error signal according to the direction of the near-end speaker to obtain adjusted beam parameters for the corresponding error signal;

[0045] Based on the adjusted beam parameters of each path, the error signal of the corresponding path is processed to obtain a plurality of processed second signals, wherein the plurality of second signals are enhanced in the direction of the near-end speaker;

[0046] Adaptively adjusting a weight coefficient for processing each second signal according to the direction of the near-end speaker;

[0047] Based on the weight coefficient for each second signal, weighted processing is performed on the multiple second signals to obtain a processed single signal.

[0048] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0049] Memory, used to store computer programs;

[0050] The processor is used to implement the method steps described in any one of the above items when executing the computer program stored in the memory.

[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute the method steps described in any one of the above items.

[0052] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes: a computer program code, and when the computer program code is run on a computer, the computer executes any of the methods described above.

[0053] The beneficial effects of the embodiments of the present application are as follows:

[0054] The embodiment of the present application provides an echo cancellation method, system and electronic device based on a microphone array. First, multiple echo information is collected through the microphone array, wherein each echo signal is a signal collected by a corresponding microphone in the microphone array after the playback signal from the speaker is reflected; the microphone array is located at a first fixed position, and the speaker is located at a second fixed position; accordingly, the microphone array and the speaker are at a fixed distance; then, according to the position information of the speaker relative to the microphone array, a fixed beamforming coefficient for each echo signal is designed, and the corresponding echo signal is preprocessed to obtain multiple preprocessed first signals; since the fixed beamforming coefficient can be designed according to the position information of the speaker relative to the microphone array, the sound in the direction of the speaker can be effectively suppressed while the sound collection in the direction of the near-end speaker is not affected, thereby significantly improving the signal-to-echo ratio, creating favorable conditions for subsequent echo cancellation, and enhancing the overall echo cancellation capability and accuracy; in addition, the preprocessing of the echo signal by the fixed beamforming coefficient is a linear processing method, which will not have a negative impact on the linear echo cancellation of the subsequent adaptive filter.

[0055] Then, the multiple first signals are filtered by adaptive filters corresponding to the multiple first signals one by one to obtain multiple error signals; in this way, the linear parts in the multiple first signals can be effectively eliminated. In addition, the beam parameters for each error signal can be adaptively adjusted according to the sound source position of the near-end speaker, and the multiple error signals can be merged into a single signal based on the multi-channel beam parameters. In this way, on the one hand, the sound signal in the direction of the near-end speaker can be improved; on the other hand, the algorithm complexity of the subsequent echo suppression of the single signal after the adaptive beam forming is reduced. In addition, the pre-built deep learning model for learning and identifying nonlinear echoes and near-end speech can be used to eliminate the residual nonlinear echo from the single signal, thereby effectively eliminating the nonlinear part of the echo. In this way, single-talk and double-talk echoes can be better processed, and the overall effect of echo cancellation is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A flowchart of one of the methods of echo cancellation based on a microphone array provided in an embodiment of the present application;

[0057] Figure 2 for Figure 1 A method flow chart of step S102;

[0058] Figure 3 for Figure 1 A method flow chart of step S103;

[0059] Figure 4 for Figure 1 A method flow chart of step S104;

[0060] Figure 5 A schematic diagram of a structure of an echo cancellation system based on a microphone array provided in an embodiment of the present application;

[0061] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0063] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0064] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0065] In related technologies, with the widespread popularity of smart devices and the rapid development of communication technology, the application of voice communication and audio processing in many fields has become increasingly critical. In voice interaction scenarios such as remote conferencing, voice assistants, and smart homes, clear and accurate sound quality plays a decisive role in effective communication and good user experience. However, echo, as a common and thorny audio interference problem, seriously damages the quality of voice signals, greatly reducing communication efficiency and user satisfaction.

[0066] Echoes are caused by many factors, including the reflections experienced by the sound when it propagates from the speaker to the microphone, and the sound played by the speaker is captured again by the microphone. Although traditional echo cancellation techniques, such as adaptive filters in the field of digital signal processing, can mitigate the negative impact of echoes to a certain extent, they often seem to be powerless when faced with complex and changing acoustic environments. In particular, the performance of these traditional methods will be significantly reduced in the presence of multi-source noise, nonlinear acoustic paths, or dynamically changing room environments.

[0067] At the same time, as electronic devices continue to develop towards being thinner and more portable, the physical distance between microphones and speakers is greatly limited, which makes the echo cancellation problem more serious. The emergence of microphone array technology has brought new opportunities to solve this problem. By rationally arranging multiple microphones to capture sound, and using the time difference and intensity difference of sound reaching different microphones, it is possible to accurately locate the sound source, and then effectively distinguish between direct sound and reflected sound. This technical means can not only improve the accuracy of echo cancellation, but also suppress background noise to a certain extent, significantly improving the clarity of voice signals.

[0068] In recent years, the booming development of deep learning technology has made echo cancellation algorithms based on machine learning a research hotspot. These algorithms can deeply grasp the characteristics of echoes and clean speech by learning from a large amount of training data, so as to deal with complex acoustic scenes more intelligently. However, how to cleverly combine the physical characteristics of microphone arrays with the advantages of deep learning algorithms to develop more efficient and accurate echo cancellation technology is still a key challenge that needs to be overcome in the current field.

[0069] In view of this, if Figure 1 As shown, the present application provides an echo cancellation method based on a microphone array, comprising:

[0070] S101: collecting multiple echo signals through the microphone array; each echo signal is a signal collected by a corresponding microphone in the microphone array after being reflected by a playback signal from a loudspeaker; the microphone array is located at a first fixed position, and the loudspeaker is located at a second fixed position;

[0071] In the specific implementation process, multiple echo signals are collected through the microphone array; each echo signal is set in one-to-one correspondence with each microphone in the microphone array. That is to say, the echo signal of the corresponding path is collected by each microphone in the microphone array. Among them, each echo signal is a signal collected by the corresponding microphone in the microphone array after the playback signal from the speaker is reflected. Exemplarily, in scenes such as video conferencing, audio conferencing, Voice over Internet Protocol (VOIP) calls, game interactions, education and training, emergency command, and telemedicine, the playback signal may be received by the built-in audio playback component in related devices such as conference whiteboards, desktop speakerphones, and conference meeting bars. The playback signal may also be a signal generated by locally stored audio materials such as explanation audio played in meetings and teaching audio in training courses after decoding and other processing, and then played through the speaker.

[0072] In addition, the microphone array is located at a first fixed position, and the speaker is located at a second fixed position, that is, the position of the microphone array is fixed, and the position of the speaker is also fixed. Accordingly, there is a relatively fixed positional relationship between the microphone array and the speaker. In this way, in the subsequent echo cancellation related processing, the fixed positional relationship between the microphone array and the speaker can be used to quickly determine the direction of the echo, which provides a favorable guarantee for echo cancellation.

[0073] S102: designing a fixed beamforming coefficient for each echo signal according to the position information of the loudspeaker relative to the microphone array, and preprocessing the corresponding echo signal to obtain multiple preprocessed first signals;

[0074] In the specific implementation process, since the positions of the loudspeaker and the microphone array are fixed, the design of fixed beamforming can effectively utilize the position information. Specifically, according to the positional relationship of the loudspeaker relative to the microphone array, a fixed beamforming coefficient is designed for each echo signal, and the corresponding echo signal is preprocessed, so as to suppress the sound in the direction of the loudspeaker while keeping the sound collected in the direction of the near-end speaker unchanged. In this way, the signal-to-echo ratio of the obtained multi-channel first signal after preprocessing is effectively improved, thereby providing a favorable guarantee for the subsequent echo cancellation and enhancing the ability and accuracy of the overall echo cancellation; in addition, the preprocessing of the echo signal by the fixed beamforming coefficient is a linear processing method, which will not have a negative impact on the linear echo cancellation of the subsequent adaptive filter. Moreover, this can effectively reduce the intensity of the echo signal and reduce the possibility of echo leakage during single-talk; even in the double-talk scenario, the sound in the direction of the loudspeaker can still be suppressed, the near-end signal echo ratio can be improved, and the echo interference can be reduced, while keeping the sound collection of the two speakers relatively stable, laying the foundation for subsequent processing.

[0075] S103: filtering the multiple first signals through adaptive filters corresponding to the multiple first signals one by one to obtain multiple error signals;

[0076] In the specific implementation process, after obtaining multiple pre-processed first signals, the multiple first signals are filtered through adaptive filters corresponding to the multiple first signals one by one to obtain multiple error signals, so that the linear parts of the multiple first signals can be effectively eliminated, further reducing the influence of the echo signal, ensuring that the echo is effectively suppressed during single speaking, making the voice signal purer, and achieving the effect of single speaking without echo; even in a double speaking scenario, it can still ensure that the voice signals of the two speakers are clearer and more distinguishable, reducing the influence of double speaking voice communication during echo.

[0077] S104: adaptively adjusting beam parameters for each error signal according to the sound source position of the near-end speaker, and combining the multiple error signals into a single signal based on the multiple beam parameters;

[0078] In the specific implementation process, after obtaining the multi-path error signals, the beam parameters for each error signal can be adaptively adjusted according to the sound source position of the near-end speaker, which can enhance the strength and quality of the near-end speaker's voice and further suppress the echo signal; even in the dual-talk scenario, the beam can still be adaptively formed according to the angle of the two speakers, pointing to the direction of the speaker, improving the strength and quality of the speaker's voice, and further suppressing the echo signal, so that in the dual-talk process, even if two speakers occur at the same time, the voice signal and the echo signal can be effectively distinguished, reducing the interference of the echo on the dual-talk communication, and avoiding the problem of poor dual-talk suppression effect. In addition, the multi-path error signals can be merged into a single-path signal based on the multi-path beam parameters, reducing the algorithm complexity of the subsequent echo suppression of the single-path signal after the adaptive beam forming.

[0079] S105: Eliminate residual nonlinear echo from the single-channel signal using a deep learning model; the deep learning model is a pre-built model for learning and identifying nonlinear echo and near-end speech.

[0080] In the specific implementation process, after obtaining a single-channel signal, a deep learning model can be used to eliminate the residual nonlinear echo from the single-channel signal. For example, a neural network can be used to learn the relevant features of the residual echo, accurately distinguish the single-speaking situation, and accurately suppress the residual echo, thereby ensuring that there is almost no echo residual in the single-speaking scenario, providing users with a clear and natural voice experience; in addition, even in the dual-speaking scenario, it can still avoid misjudgment and excessive suppression of normal voice signals, accurately suppress residual echoes, ensure the fluency and intelligibility of voice during dual-speaking, achieve a good effect of dual-speaking suppression, and improve the quality of voice interaction during dual-speaking.

[0081] In the embodiments of the present application, Figure 2 As shown, step S102: designing a fixed beamforming coefficient for each echo signal according to the position information of the loudspeaker relative to the microphone array, preprocessing the corresponding echo signal, and obtaining multiple preprocessed first signals, including:

[0082] S201: Determine position information of the speaker relative to the microphone array according to the first fixed position and the second fixed position;

[0083] S202: Determine, according to the position information, a direction of the speaker relative to the microphone array;

[0084] S203: Designing fixed beamforming coefficients for each echo signal based on the direction;

[0085] S204: Preprocessing the corresponding echo signal by using a fixed beamforming coefficient for each echo signal to obtain a plurality of preprocessed first signals, wherein the plurality of first signals are suppressed in the direction.

[0086] In the specific implementation process, the specific implementation process of step S201 to step S204 is as follows:

[0087] First, according to the first fixed position of the microphone array and the second fixed position of the speaker, the position information of the speaker relative to the microphone array is determined; then, according to the position information, the direction of the speaker relative to the microphone array is determined. Exemplarily, the relevant position information can be geometric parameters such as the distance and angle of the speaker relative to the microphone array. These position information can be used as a key basis for the subsequent algorithm parameter design. Exemplarily, when determining the beam direction and shape, it will be optimized according to the layout of the microphone array and the speaker to better realize the processing of the echo signal. Specifically, according to the direction of the speaker relative to the microphone array, a fixed beamforming coefficient for each echo signal can be designed; then, the corresponding echo signal is preprocessed by the fixed beamforming coefficient for each echo signal to obtain multiple preprocessed first signals, wherein the multiple first signals are suppressed in the direction of the speaker relative to the microphone array, thereby improving the signal-to-echo ratio, which provides a favorable guarantee for the subsequent echo cancellation. Exemplarily, the signal-to-echo ratio of the multiple first signals will be improved by 6db to 10db.

[0088] Exemplarily, a variety of fixed beamforming algorithms can be used to pre-process the corresponding echo signals according to actual application needs. For example, filter-and-sum beamforming, superdirective beamforming, differential beamforming, etc. In one exemplary embodiment, a differential beamforming algorithm can be used to pre-process the multi-channel echo signals. The specific principle is to form beams based on the difference between the signals received by multiple microphones in the microphone array. For the differential beamforming algorithm, it will perform differential operations on the signals received by different microphones in the microphone array, thereby enhancing the signals in the desired direction (usually the speaker direction) and suppressing the signals in other directions (especially the speaker direction); after processing, the original multi-channel echo signals are converted into multi-channel pre-processed first signals. On the basis of maintaining the sound characteristics of the speaker direction, the echo components of these first signals are initially consistent and optimized, providing a higher quality input signal for subsequent adaptive filtering, thereby improving the performance and effect of the entire echo cancellation.

[0089] In the embodiments of the present application, Figure 3 As shown, step S103: filtering the multiple first signals through adaptive filters corresponding to the multiple first signals one by one to obtain multiple error signals, including:

[0090] S301: taking the playback signal as a reference signal and processing it through an adaptive filter corresponding to the multiple first signals one by one, thereby obtaining an estimated echo signal; the estimated echo signal is specifically an estimation of the echo signal on the propagation path of the playback signal from the loudspeaker to the microphone array;

[0091] S302: Subtract the estimated echo signal from each first signal to obtain a multi-path error signal.

[0092] In the specific implementation process, the specific implementation process of step S301 to step S302 is as follows:

[0093] After obtaining multiple first signals, since each signal may have different characteristics, accurate processing of each channel echo signal can be achieved by configuring an independent adaptive filter for each channel. In practical applications, a variety of adaptive filtering algorithms can be used as needed, such as the Normalized Least Mean Square (NLMS) algorithm, the Recursive Least Squares (RLS) algorithm, the Kalman filter algorithm, etc. In one exemplary embodiment, the Kalman filter algorithm can be used to efficiently estimate and filter the multiple first signals; the Kalman filter algorithm is an optimal estimation filter algorithm based on a state space model, which can efficiently estimate and filter the signal in the presence of noise and uncertainty by predicting the system state and updating the measured value.

[0094] In practical applications, the playback signal can be used as a reference signal, and after being processed by an adaptive filter corresponding to the multi-channel first signal, an estimated echo signal can be obtained; the estimated echo signal is specifically an estimate of the echo signal on the propagation path of the playback signal from the speaker to the microphone array; in this way, the estimated echo signal is subtracted from each first signal to obtain a multi-channel error signal. In this way, the linear part of the echo can be gradually eliminated from the multi-channel first signal, and the multi-channel error signal after linear echo elimination is obtained.

[0095] Taking the Kalman filter based on the Kalman filter algorithm as an example, before starting to process the echo signal, the parameters of the Kalman filter need to be initialized, including the initial value of the state vector, the covariance matrix, etc.; the initial setting of these parameters will affect the performance and convergence speed of the Kalman filter; during operation, the Kalman filter will continuously adjust its own parameters according to the input echo signal (i.e., the first signal in this article) and the pre-set criteria to adapt to the changing characteristics of the echo signal, such as the change of the echo path, the fluctuation of the noise, etc. For each echo signal, the Kalman filter continuously updates the coefficients of the Kalman filter iteratively according to the statistical characteristics of the signal and the preset objective function through real-time analysis and processing of the signal, so that the output of the Kalman filter is as close to the linear part of the echo signal as possible; in this way, the linear part of the echo is gradually eliminated from the multi-channel first signal, and the multi-channel error signal after linear echo elimination is output, thereby effectively reducing the interference of the echo on the voice signal, improving the quality and clarity of the voice signal, and providing a guarantee for achieving a more accurate echo cancellation effect.

[0096] In the embodiments of the present application, Figure 4 As shown, step S104: adaptively adjusting the beam parameters for each error signal according to the sound source position of the near-end speaker, and combining the multiple error signals into a single signal based on the multiple beam parameters, including:

[0097] S401: using sound source localization technology to determine the sound source position of the near-end speaker relative to the microphone array;

[0098] S402: Determine the direction of the near-end speaker according to the sound source position;

[0099] S403: Adaptively adjusting beam parameters for each error signal according to the direction of the near-end speaker to obtain adjusted beam parameters for the corresponding error signal;

[0100] S404: Based on the adjusted beam parameters of each path, the error signal of the corresponding path is processed to obtain a plurality of processed second signals, wherein the plurality of second signals are enhanced in the direction of the near-end speaker;

[0101] S405: adaptively adjusting a weight coefficient for processing each second signal according to the direction of the near-end speaker;

[0102] S406: Based on the weight coefficient for each second signal, perform weighted processing on the multiple second signals to obtain a processed single signal.

[0103] In the specific implementation process, the specific implementation process of step S401 to step S406 is as follows:

[0104] After obtaining the multi-path error signal, if the current near-end speaker is speaking (i.e., there is a near-end voice signal), the sound source localization technology can be used to determine the sound source position of the near-end speaker relative to the microphone array, thereby determining the direction of the near-end speaker. Exemplarily, the multi-path error signal is analyzed, and the arrival time difference, phase difference, amplitude difference and other characteristics of the signal are used, combined with the geometric layout of the microphone array, and injection-based time delay estimation (TDE), generalized cross correlation (GCC) and other methods are used to determine the approximate direction and angle range of the speaker. Exemplarily, according to the time sequence in which different microphones in the microphone array receive the speaker's voice signal, the arrival direction of the sound is estimated by calculating the time delay, combining the known microphone spacing and sound propagation speed, thereby preliminarily determining the direction of the speaker.

[0105] Then, according to the direction of the near-end speaker, the beam parameters for each error signal are adaptively adjusted to obtain the adjusted beam parameters for the corresponding path. Exemplarily, a variety of adaptive beamforming algorithms can be selected as needed, such as the minimum variance distortionless response (MVDR) algorithm, the generalized sidelobe cancellation (GSC) algorithm, the neural network-based beamforming algorithm, etc. In one exemplary embodiment, the MVDR algorithm can be used to minimize the signal power in other directions while ensuring that the signal in the direction of the speaker passes without distortion, thereby forming a beam pointing to the speaker. It should be noted that when applying the MVDR algorithm, it is necessary to calculate and configure relevant covariance matrix and other parameters based on the characteristics of the microphone array (such as the number of array elements, spacing, arrangement, etc.) and the previously detected speaker direction to optimize the shape and directivity of the beam.

[0106] Then, based on the adjusted beam parameters, the corresponding path error signals are processed to obtain processed multi-path second signals, which are enhanced in the direction of the near-end speaker; then, according to the direction of the near-end speaker, the weight coefficients for processing each path of the second signal are adaptively adjusted; based on the weight coefficients for each path of the second signal, the multi-path second signals are weightedly processed to obtain a processed single-path signal.

[0107] In practical applications, based on the determined speaker direction and the configured beam parameters, the input multi-path error signals are weighted and summed; for the signal received by each microphone, it is adjusted according to its weight in beamforming, so that the signal from the speaker direction is enhanced after summing, while the signal from other directions (such as the speaker direction) is suppressed. Exemplarily, by setting the weights reasonably, the multi-path error signals are added in a certain proportion, so that the signal in the speaker direction is superimposed and enhanced in phase, while the signals in other directions cancel each other out or weaken each other due to phase difference and other reasons, and finally a signal (i.e., a single-path signal) processed by beamforming is output, which highlights the speaker's voice component and further suppresses echoes and other interference signals, providing a higher quality input signal for subsequent echo cancellation, which helps to improve the performance and effect of the entire echo cancellation system in complex acoustic environments, and improves the clarity and intelligibility of voice communication, especially when multiple people are talking at the same time or there is strong background noise and echo interference, which can better focus on the voice of the target speaker and improve the user's voice interaction experience.

[0108] It should be noted that after obtaining a single-channel signal, there may still be residual nonlinear echo in the single-channel signal. Through the pre-built deep learning model for learning and identifying nonlinear echo and near-end speech, the residual nonlinear echo can be further eliminated, thereby improving the performance and effect of the entire echo cancellation system in a complex acoustic environment. In practical applications, it is necessary to pre-select a suitable neural network architecture to build a deep learning model. For example, structures such as deep neural network (DNN), convolutional neural network (CNN), long short-term memory network (LSTM) and their variants are used. Then, a large number of annotated speech data samples are collected to obtain training data; these samples cover various single-speaking, two-speaking scenarios and different acoustic environments, and the feature information of the residual nonlinear echo is clearly annotated; for example, for each sample, the specific performance of the residual nonlinear echo in the time domain and frequency domain, the relative relationship with the speech signal and other key features are annotated so that the neural network can learn these patterns during the training process. Then, the prepared training data is input into the constructed deep learning model. Based on the set loss function (such as mean square error loss function, etc.) and optimization algorithm (such as stochastic gradient descent, Adam, etc.), the model continuously adjusts its weight parameters so that the difference between the predicted results output by the model and the labeled real results is continuously reduced. During multiple iterative training processes, the model gradually learns the characteristic laws of residual nonlinear echoes and how to suppress them, so as to achieve the ability to accurately distinguish speech signals from residual nonlinear echoes and effectively suppress residual nonlinear echoes.

[0109] When processing a single-channel signal, the trained deep learning model will automatically extract the time domain and frequency domain features of the input signal containing residual nonlinear echoes, and use the learned pattern recognition capabilities to determine which parts belong to residual nonlinear echoes and which are normal voice signals. Then, according to the learned suppression strategy, the parts determined to be residual nonlinear echoes are processed in a targeted manner, such as adjusting the corresponding weights, attenuating the amplitude of the residual nonlinear echoes, or modifying their spectral features, thereby suppressing the residual echoes and minimizing their impact on the voice signal. Finally, a processed signal is output, so that no echo residue can be achieved in a single-talk scenario, and the double-talk suppression can be small in a double-talk scenario, thereby improving the voice signal quality and clarity during the entire voice interaction process, and providing users with a clearer and more natural voice experience.

[0110] Based on the same inventive concept, Figure 5 As shown, the embodiment of the present application provides an echo cancellation system based on a microphone array, comprising:

[0111] A loudspeaker 10, used for transmitting a playback signal;

[0112] The microphone array 20 is used to collect multiple echo signals; each echo signal is a signal collected by a corresponding microphone in the microphone array 20 after the playback signal is reflected; the microphone array 20 is located at a first fixed position, and the speaker is located at a second fixed position;

[0113] The fixed beamforming module 30 is used to design a fixed beamforming coefficient for each echo signal according to the position information of the loudspeaker 10 relative to the microphone array 20, and preprocess the corresponding echo signal to obtain multiple preprocessed first signals;

[0114] An adaptive filtering module 40, configured to filter the multiple first signals through adaptive filters corresponding to the multiple first signals one by one, to obtain multiple error signals;

[0115] An adaptive beamforming module 50, configured to adaptively adjust beam parameters for each error signal according to the sound source position of the near-end speaker, and combine the multiple error signals into a single signal based on the multiple beam parameters;

[0116] The echo suppression module 60 is used to eliminate the residual nonlinear echo from the single-channel signal; the deep learning model is a pre-built model for learning and identifying nonlinear echo and near-end speech.

[0117] In the embodiment of the present application, the fixed beamforming module 30 is specifically used for:

[0118] Determining position information of the speaker 10 relative to the microphone array 20 according to the first fixed position and the second fixed position;

[0119] Determine the direction of the speaker 10 relative to the microphone array 20 according to the position information;

[0120] Based on the directions, designing fixed beamforming coefficients for each echo signal;

[0121] By preprocessing the corresponding echo signal with respect to the fixed beamforming coefficient of each echo signal, a plurality of preprocessed first signals are obtained, and the plurality of first signals are suppressed in the direction.

[0122] In the embodiment of the present application, the adaptive filtering module 40 is specifically used for:

[0123] The playback signal is used as a reference signal, and after being processed by an adaptive filter corresponding to the multiple first signals one by one, an estimated echo signal is obtained; the estimated echo signal is specifically an estimation of the echo signal on the propagation path of the playback signal from the speaker 10 to the microphone array 20;

[0124] The estimated echo signal is subtracted from each first signal to obtain a multi-path error signal.

[0125] In the embodiment of the present application, the adaptive beamforming module 50 is specifically used for:

[0126] Using sound source localization technology to determine the sound source position of the near-end speaker relative to the microphone array 20;

[0127] Determining the direction of the near-end speaker according to the sound source position;

[0128] Adaptively adjusting beam parameters for each error signal according to the direction of the near-end speaker to obtain adjusted beam parameters for the corresponding error signal;

[0129] Based on the adjusted beam parameters of each path, the error signal of the corresponding path is processed to obtain a plurality of processed second signals, wherein the plurality of second signals are enhanced in the direction of the near-end speaker;

[0130] Adaptively adjusting a weight coefficient for processing each second signal according to the direction of the near-end speaker;

[0131] Based on the weight coefficient for each second signal, weighted processing is performed on the multiple second signals to obtain a processed single signal.

[0132] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the electronic device can implement the functions of the echo cancellation method based on the microphone array, referring to Figure 6 , the electronic device comprises:

[0133] Memory 100, used for storing computer programs;

[0134] The processor 200 is configured to implement the steps of the microphone array-based echo cancellation method as described in any of the above items when executing the computer program stored in the memory.

[0135] The specific connection medium between the processor 200 and the memory 100 is not limited in the embodiment of the present application. Figure 6 In the example, the processor 200 and the memory 100 are connected via the bus 300. The bus 300 is connected to the memory 100 via the bus 300. Figure 6 The bus 300 is represented by a bold line, and the connection between other components is only for schematic illustration and is not intended to be limiting. The bus 300 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the figure, but it does not mean that there is only one bus 300 or one type of bus 300. Alternatively, the processor 200 may also be called a controller, and there is no limitation on the name.

[0136] In the embodiment of the present application, the memory 100 stores instructions that can be executed by the processor 200. When the processor 200 executes the computer program stored in the memory 100, the echo cancellation method based on the microphone array discussed above can be executed. The processor 200 can implement Figure 5 The functions of each module are shown in the figure.

[0137] Among them, the processor 200 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing the computer program stored in the memory 100 and calling the data stored in the memory 100, the various functions of the device and process data, the device can be monitored as a whole.

[0138] In one possible design, the processor 200 may include one or more processing units, and the processor 200 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. In some embodiments, the processor 200 and the memory 100 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0139] The memory 100, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 100 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. Alternatively, the memory 100 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 100 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0140] By designing and programming the processor, the code corresponding to the echo cancellation method based on the microphone array introduced in the above embodiment can be fixed into the chip, so that the chip can execute the steps of the echo cancellation method based on the microphone array discussed above when running. How to design and program the processor is a technology well known to those skilled in the art and will not be described in detail here.

[0141] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is used to enable a computer to execute the steps of the microphone array-based echo cancellation method discussed above.

[0142] In some possible implementations, various aspects of the microphone array-based echo cancellation method provided in the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on an apparatus, the program code is used to enable the control device to execute the steps of the microphone array-based echo cancellation method according to various exemplary implementations of the present application described above in this specification.

[0143] The embodiment of the present application provides an echo cancellation method, system and electronic device based on a microphone array. First, multiple echo information is collected through the microphone array, wherein each echo signal is a signal collected by a corresponding microphone in the microphone array after the playback signal from the speaker is reflected; the microphone array is located at a first fixed position, and the speaker is located at a second fixed position; accordingly, the microphone array and the speaker are at a fixed distance; then, according to the position information of the speaker relative to the microphone array, a fixed beamforming coefficient for each echo signal is designed, and the corresponding echo signal is preprocessed to obtain multiple preprocessed first signals; since the fixed beamforming coefficient can be designed according to the position information of the speaker relative to the microphone array, the sound in the direction of the speaker can be effectively suppressed while the sound collection in the direction of the near-end speaker is not affected, thereby significantly improving the signal-to-echo ratio, creating favorable conditions for subsequent echo cancellation, and enhancing the overall echo cancellation capability and accuracy; in addition, the preprocessing of the echo signal by the fixed beamforming coefficient is a linear processing method, which will not have a negative impact on the linear echo cancellation of the subsequent adaptive filter.

[0144] Then, the multiple first signals are filtered by adaptive filters corresponding to the multiple first signals one by one to obtain multiple error signals; in this way, the linear parts in the multiple first signals can be effectively eliminated. In addition, the beam parameters for each error signal can be adaptively adjusted according to the sound source position of the near-end speaker, and the multiple error signals can be merged into a single signal based on the multi-channel beam parameters. In this way, on the one hand, the sound signal in the direction of the near-end speaker can be improved; on the other hand, the algorithm complexity of the subsequent echo suppression of the single signal after the adaptive beam forming is reduced. In addition, the pre-built deep learning model for learning and identifying nonlinear echoes and near-end speech can be used to eliminate the residual nonlinear echo from the single signal, thereby effectively eliminating the nonlinear part of the echo. In this way, single-talk and double-talk echoes can be better processed, and the overall effect of echo cancellation is improved.

[0145] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0146] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0149] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An echo cancellation method based on a microphone array, characterized in that: include: Collecting multiple echo signals through the microphone array; Each echo signal is a signal collected by a corresponding microphone in the microphone array after the playback signal from the loudspeaker is reflected; The microphone array is located at a first fixed position, and the speaker is located at a second fixed position; According to the position information of the loudspeaker relative to the microphone array, a fixed beamforming coefficient for each echo signal is designed, and the corresponding echo signal is preprocessed to obtain multiple preprocessed first signals; Performing filtering processing on the multiple first signals through adaptive filters corresponding to the multiple first signals one by one, to obtain multiple error signals; Adaptively adjusting beam parameters for each error signal according to the sound source position of the near-end speaker, and combining the multiple error signals into a single signal based on the multiple beam parameters; Eliminating residual nonlinear echo from the single-channel signal using a deep learning model; The deep learning model is a pre-built model for learning and recognizing nonlinear echo and near-end speech.

2. The method according to claim 1, characterized in that The method of designing a fixed beamforming coefficient for each echo signal according to the position information of the loudspeaker relative to the microphone array, preprocessing the corresponding echo signal, and obtaining multiple preprocessed first signals includes: Determining position information of the speaker relative to the microphone array according to the first fixed position and the second fixed position; Determining, according to the position information, a direction of the speaker relative to the microphone array; Based on the directions, designing fixed beamforming coefficients for each echo signal; By preprocessing the corresponding echo signal with respect to the fixed beamforming coefficient of each echo signal, a plurality of preprocessed first signals are obtained, and the plurality of first signals are suppressed in the direction.

3. The method according to claim 1 or 2, characterized in that The filtering process of the multiple first signals by using adaptive filters corresponding to the multiple first signals one by one to obtain multiple error signals includes: The playback signal is used as a reference signal, and after being processed by an adaptive filter corresponding to the multiple first signals one by one, an estimated echo signal is obtained; the estimated echo signal is specifically an estimate of the echo signal on the propagation path of the playback signal from the speaker to the microphone array; The estimated echo signal is subtracted from each first signal to obtain a multi-path error signal.

4. The method according to claim 1 or 2, characterized in that: The method adaptively adjusting the beam parameters for each error signal according to the sound source position of the near-end speaker, and combining the multiple error signals into a single signal based on the multiple beam parameters, includes: Using sound source localization technology to determine the sound source position of the near-end speaker relative to the microphone array; Determining the direction of the near-end speaker according to the sound source position; Adaptively adjusting beam parameters for each error signal according to the direction of the near-end speaker to obtain adjusted beam parameters for the corresponding error signal; Based on the adjusted beam parameters of each path, the error signal of the corresponding path is processed to obtain a plurality of processed second signals, wherein the plurality of second signals are enhanced in the direction of the near-end speaker; Adaptively adjusting a weight coefficient for processing each second signal according to the direction of the near-end speaker; Based on the weight coefficient for each second signal, weighted processing is performed on the multiple second signals to obtain a processed single signal.

5. An echo cancellation system based on a microphone array, characterized in that: include: A loudspeaker, used to transmit the playback signal; Microphone array, used to collect multi-channel echo signals; Each echo signal is a signal collected by a corresponding microphone in the microphone array after the playback signal is reflected; The microphone array is located at a first fixed position, and the speaker is located at a second fixed position; A fixed beamforming module, used to design a fixed beamforming coefficient for each echo signal according to the position information of the loudspeaker relative to the microphone array, preprocess the corresponding echo signal, and obtain multiple preprocessed first signals; An adaptive filtering module, used for filtering the multiple first signals through adaptive filters corresponding to the multiple first signals one by one, to obtain multiple error signals; An adaptive beamforming module, used for adaptively adjusting beam parameters for each error signal according to the sound source position of the near-end speaker, and combining the multiple error signals into a single signal based on the multiple beam parameters; An echo suppression module, used to eliminate residual nonlinear echo from the single-channel signal; The deep learning model is a pre-built model for learning and recognizing nonlinear echo and near-end speech.

6. The system according to claim 5, characterized in that The fixed beamforming module is specifically used for: Determining position information of the speaker relative to the microphone array according to the first fixed position and the second fixed position; Determining, according to the position information, a direction of the speaker relative to the microphone array; Based on the directions, designing fixed beamforming coefficients for each echo signal; By preprocessing the corresponding echo signal with respect to the fixed beamforming coefficient of each echo signal, a plurality of preprocessed first signals are obtained, and the plurality of first signals are suppressed in the direction.

7. The system according to claim 5 or 6, characterized in that The adaptive filtering module is specifically used for: The playback signal is used as a reference signal, and after being processed by an adaptive filter corresponding to the multiple first signals one by one, an estimated echo signal is obtained; the estimated echo signal is specifically an estimate of the echo signal on the propagation path of the playback signal from the speaker to the microphone array; The estimated echo signal is subtracted from each first signal to obtain a multi-path error signal.

8. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to implement the method steps as described in any one of claims 1 to 4 when executing the computer program stored in the memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute the method steps according to any one of claims 1 to 4.

10. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 4.

Images