A method and device for processing audio signals
By performing multi-level processing of the mixed audio signals collected by the microphone, including echo cancellation, feedback cancellation and noise reduction dereverberation, combined with howling suppression processing, the problem of poor howling control effect in the existing technology is solved, and a better acoustic system amplification effect is achieved.
Patent Information
- Application Number
- CN202510003154.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-02
- Publication Date
- 2025-04-04
AI Technical Summary
The existing audio signal processing methods have limitations in controlling howling, resulting in poor amplification effect of the acoustic system.
By obtaining the mixed audio signals collected by each microphone, echo cancellation processing, feedback cancellation processing and noise reduction dereverberation processing are performed to obtain the target audio signal, and the howling suppression processing is performed, and finally play it through the speaker.
It effectively improves the control effect of howling, and thus improves the amplification effect of the acoustic system, avoiding the generation of howling.
Smart Images
Figure CN119767212A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of signal processing technology, and in particular to an audio signal processing method and related devices. Background Art
[0002] As people's living standards continue to improve, in more and more scenarios (such as smart classroom scenarios, car scenarios, smart conference room scenarios, etc.), people have an increasingly strong demand for the sound amplification of acoustic systems (i.e., audio signal amplification), and have put forward higher requirements for the sound amplification effect (such as gain, hearing, recognition rate, etc.). In the acoustic system, since the microphone and the speaker are coupled, acoustic feedback is easy to occur (i.e., the sound from the speaker to the microphone is picked up by the microphone and then played by the speaker). If the acoustic feedback phenomenon is serious, howling will occur, affecting the listening experience. Therefore, how to provide an audio signal processing method that effectively controls howling and improves the sound amplification effect is a core issue in the acoustic system.
[0003] Generally speaking, upgrading the equipment in the acoustic system can effectively control howling and improve the sound amplification effect. However, in these scenarios, upgrading the equipment in the acoustic system is a relatively long process, and users may lack sufficient motivation to upgrade the equipment. Therefore, it is of great significance to provide an audio signal processing method from the software level based on the existing equipment to effectively control howling and improve the sound amplification effect of the acoustic system.
[0004] However, various solutions adopted by existing audio signal processing methods for controlling howling all have certain limitations, resulting in an inability to effectively control howling, and causing poor sound amplification effects of the acoustic system.
[0005] Therefore, how to provide an audio signal processing method that can improve the control effect of howling and further improve the sound amplification effect of the acoustic system has become an urgent problem to be solved by those skilled in the art. Summary of the invention
[0006] In view of the above problems, the present application provides an audio signal processing method and related devices to achieve the purpose of improving the control effect of howling, thereby improving the sound amplification effect of the acoustic system. The specific scheme is as follows:
[0007] The first aspect of the present application provides an audio signal processing method, comprising:
[0008] Obtain mixed audio signals collected by each microphone;
[0009] For the mixed audio signal collected by each microphone, the mixed audio signal is subjected to echo cancellation processing, feedback cancellation processing, noise reduction and reverberation removal processing to obtain the target audio sub-signal corresponding to the microphone;
[0010] Combining and processing the target audio sub-signals corresponding to the microphones to obtain a target audio signal;
[0011] Performing a howling suppression process on the target audio signal to obtain a target audio signal after the howling suppression process;
[0012] The target audio signal after the howling suppression processing is amplified and then played through a loudspeaker.
[0013] In a possible implementation, performing echo cancellation processing, feedback cancellation processing, and noise reduction and reverberation removal processing on the mixed audio signal to obtain a target audio sub-signal corresponding to the microphone includes:
[0014] Performing echo cancellation processing on the mixed audio signal to obtain an audio signal after the echo cancellation processing;
[0015] Inputting the audio signal after the echo cancellation processing into a feedback cancellation model to obtain the audio signal after the feedback cancellation processing;
[0016] The audio signal after the feedback elimination process is input into a noise reduction and dereverberation model to obtain the audio signal after the noise reduction and dereverberation process as the target audio sub-signal.
[0017] In a possible implementation, the training method of the feedback elimination model and the denoising and dereverberation model includes:
[0018] obtaining a pre-recorded acoustic near-end path and an acoustic feedback path;
[0019] Acquire a preset speech data set, where the speech data set includes clean speech data and noise data;
[0020] Performing data simulation using the acoustic proximal path and the acoustic feedback path, as well as the speech data set, to generate feedback elimination training data and noise reduction and dereverberation training data, wherein the feedback elimination training data and the noise reduction and dereverberation training data are both signals containing multiple continuous frames;
[0021] Using the feedback elimination training data to train a feedback elimination model to obtain the trained feedback elimination model;
[0022] The denoising and dereverberation training data is used to train a denoising and dereverberation model to obtain the trained denoising and dereverberation model.
[0023] In a possible implementation, the recording method of the acoustic proximal path and the acoustic feedback path includes:
[0024] Calibrate microphones and speakers in real-world scenarios;
[0025] Select the target microphone and target speaker;
[0026] The acoustic near-end path and the acoustic feedback path are recorded using the target microphone and the target speaker.
[0027] In a possible implementation, the using the acoustic proximal path and the acoustic feedback path, and the speech data set to perform data simulation to generate feedback elimination training data and noise reduction and reverberation removal training data includes:
[0028] Obtain a pre-built data simulation system, the data simulation system comprising a simulated microphone group, a feedback elimination simulation model, a noise reduction and dereverberation simulation model, a signal combination module, a sound effect adding module and a simulated speaker; wherein the feedback elimination simulation model and the noise reduction and dereverberation simulation model correspond to preset parameters;
[0029] Obtaining data distribution of preset parameters corresponding to the feedback elimination simulation model and the noise reduction and dereverberation simulation model;
[0030] The acoustic proximal path, the acoustic feedback path, the feedback elimination simulation model, and the noise reduction and dereverberation simulation model have data distributions of corresponding preset parameters and the data simulation system to perform data simulation on the speech data set to generate the feedback elimination training data and the noise reduction and dereverberation training data.
[0031] In a possible implementation, the method of using the acoustic proximal path, the acoustic feedback path, the data distribution of the preset parameters corresponding to the feedback elimination simulation model and the noise reduction and dereverberation simulation model and the data simulation system to perform data simulation on the speech data set to generate the feedback elimination training data and the noise reduction and dereverberation training data includes:
[0032] For each group of simulation data in the speech data set, sampling is performed to obtain preset parameters corresponding to the simulation data in the data distribution of the preset parameters corresponding to the feedback elimination simulation model and the noise reduction and dereverberation simulation model;
[0033] Determining a simulated mixed audio signal corresponding to the simulation data based on the acoustic near-end path and the acoustic feedback path;
[0034] The simulated mixed audio signal is input into the data simulation system, and it is iterated frame by frame to obtain the input of the feedback elimination simulation model and the input of the noise reduction and dereverberation simulation model, wherein the input of the feedback elimination simulation model is the feedback elimination training data, and the input of the noise reduction and dereverberation simulation model is the noise reduction and dereverberation training data.
[0035] In a possible implementation, the using the feedback cancellation training data to train the feedback cancellation model to obtain the trained feedback cancellation model includes:
[0036] Acquire a first audio signal label corresponding to the feedback elimination training data;
[0037] The feedback cancellation training data is input into the feedback cancellation model, and the feedback cancellation model is trained with the output of the feedback cancellation model approaching the first audio signal label as a training goal to obtain the trained feedback cancellation model.
[0038] In a possible implementation, the training of the denoising and dereverberation model using the denoising and dereverberation training data to obtain the trained denoising and dereverberation model includes:
[0039] Obtaining a second audio signal label corresponding to the denoising and dereverberation detaching training data;
[0040] The denoising and dereverberation training data is input into the denoising and dereverberation model, and the denoising and dereverberation model is trained with the output of the denoising and dereverberation model approaching the second audio signal label as a training goal to obtain the trained denoising and dereverberation model.
[0041] A second aspect of the present application provides an audio signal processing device, comprising:
[0042] An acquisition unit, used to acquire mixed audio signals collected by various microphones;
[0043] A first processing unit is used to perform echo cancellation processing, feedback cancellation processing, noise reduction and reverberation removal processing on the mixed audio signal collected by each microphone to obtain a target audio sub-signal corresponding to the microphone;
[0044] A signal combination processing unit, used for combining and processing the target audio sub-signals corresponding to the microphones to obtain a target audio signal;
[0045] a howling suppression processing unit, configured to perform a howling suppression process on the target audio signal to obtain a target audio signal after the howling suppression process;
[0046] The amplifying unit is used to amplify the target audio signal after the howling suppression processing and play it through a loudspeaker.
[0047] In a possible implementation, the first processing unit includes:
[0048] An echo cancellation processing unit, configured to perform echo cancellation processing on the mixed audio signal to obtain an audio signal after the echo cancellation processing;
[0049] A feedback elimination processing unit, used for inputting the audio signal after the echo elimination processing into a feedback elimination model to obtain the audio signal after the feedback elimination processing;
[0050] The denoising and dereverberation processing unit is used to input the audio signal after the feedback elimination processing into the denoising and dereverberation model to obtain the audio signal after the denoising and dereverberation processing as the target audio sub-signal.
[0051] In a possible implementation, the device further includes: a model training unit;
[0052] The model training unit comprises:
[0053] An acoustic path acquisition unit, used for acquiring a pre-recorded acoustic near-end path and an acoustic feedback path;
[0054] A speech data set acquisition unit, used to acquire a preset speech data set, wherein the speech data set includes clean speech data and noise data;
[0055] A training data generating unit, configured to perform data simulation using the acoustic proximal path and the acoustic feedback path, as well as the speech data set, to generate feedback elimination training data and noise reduction and dereverberation training data, wherein the feedback elimination training data and the noise reduction and dereverberation training data are both signals containing multiple continuous frames;
[0056] A feedback elimination model training unit, used to train the feedback elimination model using the feedback elimination training data to obtain the trained feedback elimination model;
[0057] The denoising and dereverberation model training unit is used to train the denoising and dereverberation model using the denoising and dereverberation training data to obtain the trained denoising and dereverberation model.
[0058] In a possible implementation, the device further includes: an acoustic path recording unit;
[0059] The acoustic path recording unit is specifically used for:
[0060] Calibrate microphones and speakers in real-world scenarios;
[0061] Select the target microphone and target speaker;
[0062] The acoustic near-end path and the acoustic feedback path are recorded using the target microphone and the target speaker.
[0063] In a possible implementation, the training data generating unit includes:
[0064] A data simulation system acquisition unit, used to acquire a pre-built data simulation system, wherein the data simulation system comprises a simulated microphone group, a feedback elimination simulation model, a noise reduction and dereverberation simulation model, a signal combination module, a sound effect adding module, and a simulated speaker; wherein the feedback elimination simulation model and the noise reduction and dereverberation simulation model correspond to preset parameters;
[0065] A parameter data distribution acquisition unit, used to acquire data distribution of preset parameters corresponding to the feedback elimination simulation model and the noise reduction and dereverberation simulation model;
[0066] A data simulation unit is used to utilize the data distribution of preset parameters corresponding to the acoustic proximal path, the acoustic feedback path, the feedback elimination simulation model, and the noise reduction and dereverberation simulation model and the data simulation system to perform data simulation on the speech data set to generate the feedback elimination training data and the noise reduction and dereverberation training data.
[0067] In a possible implementation, the data simulation unit is specifically used to:
[0068] For each group of simulation data in the speech data set, sampling is performed to obtain preset parameters corresponding to the simulation data in the data distribution of the preset parameters corresponding to the feedback elimination simulation model and the noise reduction and dereverberation simulation model;
[0069] Determining a simulated mixed audio signal corresponding to the simulation data based on the acoustic near-end path and the acoustic feedback path;
[0070] The simulated mixed audio signal is input into the data simulation system, and it is iterated frame by frame to obtain the input of the feedback elimination simulation model and the input of the noise reduction and dereverberation simulation model, wherein the input of the feedback elimination simulation model is the feedback elimination training data, and the input of the noise reduction and dereverberation simulation model is the noise reduction and dereverberation training data.
[0071] In a possible implementation, the feedback elimination model training unit is specifically used to:
[0072] Acquire a first audio signal label corresponding to the feedback elimination training data;
[0073] The feedback cancellation training data is input into the feedback cancellation model, and the feedback cancellation model is trained with the output of the feedback cancellation model approaching the first audio signal label as a training goal to obtain the trained feedback cancellation model.
[0074] In a possible implementation, the denoising and dereverberation model training unit is specifically used to:
[0075] Obtaining a second audio signal label corresponding to the denoising and dereverberation detaching training data;
[0076] The denoising and dereverberation training data is input into the denoising and dereverberation model, and the denoising and dereverberation model is trained with the output of the denoising and dereverberation model approaching the second audio signal label as a training goal to obtain the trained denoising and dereverberation model.
[0077] A third aspect of the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the audio signal processing method of the first aspect or any implementation manner of the first aspect.
[0078] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:
[0079] The memory is used to store computer programs;
[0080] The processor is used to execute the computer program so that the electronic device can implement the audio signal processing method of the first aspect or any implementation manner of the first aspect.
[0081] A fifth aspect of the present application provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the audio signal processing method of the first aspect or any implementation of the first aspect.
[0082] By means of the above technical scheme, the present application provides an audio signal processing method and related devices. After obtaining the mixed audio signals collected by each microphone, the target audio sub-signal corresponding to the microphone is obtained by first performing echo elimination processing, feedback elimination processing and noise reduction and reverberation processing on each mixed audio signal, and then the target audio sub-signals corresponding to each microphone are combined and processed to obtain the target audio signal, and then the target audio signal is further subjected to howling suppression processing to obtain the target audio signal after the howling suppression processing, and finally, the target audio signal after the howling suppression processing is amplified and played through a loudspeaker. This scheme can improve the howling control effect through multi-level howling control, so that the target audio signal collected by the microphone can be played using the same loudspeaker without affecting the original playback content of the loudspeaker, thereby achieving a better amplification effect and avoiding howling. BRIEF DESCRIPTION OF THE DRAWINGS
[0083] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.
[0084] Figure 1 A flowchart of an audio signal processing method provided in an embodiment of the present application;
[0085] Figure 2 A flowchart of a method for performing echo cancellation, feedback cancellation, and noise reduction and reverberation removal processing on a mixed audio signal to obtain a target audio sub-signal corresponding to a microphone, provided in an embodiment of the present application;
[0086] Figure 3 A schematic diagram of the working principle of a feedback elimination model provided in an embodiment of the present application;
[0087] Figure 4 A flow chart of a training method for a feedback elimination model and a noise reduction and dereverberation removal model provided in an embodiment of the present application;
[0088] Figure 5 A flow chart of a method for performing data simulation using an acoustic proximal path, an acoustic feedback path, and a speech data set to generate feedback elimination training data and noise reduction and reverberation removal training data is provided in an embodiment of the present application;
[0089] Figure 6 A schematic diagram of a data simulation process of a data simulation system provided in an embodiment of the present application;
[0090] Figure 7 A schematic diagram of the structure of an audio signal processing device provided in an embodiment of the present application;
[0091] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0092] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.
[0093] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0094] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0095] As people's living standards continue to improve, in more and more scenarios (such as smart classroom scenarios, car scenarios, smart conference room scenarios, etc.), people have an increasingly strong demand for the sound amplification of acoustic systems (i.e., audio signal amplification), and have put forward higher requirements for the sound amplification effect (such as gain, hearing, recognition rate, etc.). In the acoustic system, since the microphone and the speaker are coupled, acoustic feedback is easy to occur (i.e., the sound from the speaker to the microphone is picked up by the microphone and then played by the speaker). If the acoustic feedback phenomenon is serious, howling will occur, affecting the hearing experience. Therefore, how to provide an audio signal processing method that can effectively control acoustic feedback while ensuring the stable operation of the acoustic system, and then control howling to improve the sound amplification effect is a core issue in the acoustic system.
[0096] Generally speaking, upgrading the equipment of the acoustic system can effectively control the acoustic feedback while ensuring the stable operation of the acoustic system, so as to control the howling and improve the sound amplification effect. However, in these scenarios, upgrading the equipment in the acoustic system is a relatively long process, and users may lack sufficient motivation to upgrade the equipment. Therefore, it is of great significance to provide an audio signal processing method from the software level based on the existing equipment to effectively control the acoustic feedback while ensuring the stable operation of the acoustic system, so as to control the howling and improve the sound amplification effect of the acoustic system.
[0097] Taking the in-car scenario as an example, with the rapid development of science and technology, people have higher and higher requirements for the riding experience of cars. Among them, intelligent in-car entertainment is an important factor affecting the riding experience. At present, intelligent in-car entertainment includes "microphone-free" karaoke, which means that karaoke is not performed in the form of a traditional handheld microphone, but the voice microphone embedded in the car is used for karaoke, so as to provide a more flexible and convenient way of karaoke. However, abandoning the handheld microphone means abandoning the advantage of strong directional sound pickup. In a noisy environment such as karaoke, the in-car microphone is more likely to pick up noise and cause howling problems. Therefore, howling suppression is required to ensure the user experience of "microphone-free" karaoke.
[0098] However, cars usually only have omnidirectional microphones for voice interaction and speaker groups for playing streaming media. To use omnidirectional microphones to achieve seamless integration of "microphone-free" karaoke and voice interaction, it is necessary to provide a pure software solution for the acoustic feedback problem. Only by achieving sufficient acoustic feedback suppression while ensuring low latency of the algorithm can the processed audio be suitable for voice recognition and try to ensure that "microphone-free" karaoke does not produce howling.
[0099] Existing audio signal processing methods often use the following solutions to control howling:
[0100] Phase modulation is used to avoid howling conditions and control howling. The advantage is that the computing power cost is very low, but the disadvantage is that the effect is limited and generally cannot achieve sufficient amplification gain. In addition, some phase modulation has a significant impact on the sense of hearing. Since general acoustic feedback cannot be eliminated, the sense of hearing is not good and it is not conducive to recognition.
[0101] The howling detection and suppression system uses some signal statistics to estimate the time and frequency of howling, and uses an equalizer or notch filter to control howling. Howling is more effective than phase modulation in controlling howling, and can achieve higher sound amplification gain, but the relevant parameters of howling detection require complex adjustments by relevant experts in different acoustic scenarios, and it is also unable to eliminate general acoustic feedback.
[0102] Acoustic feedback suppression based on signal processing adaptive filter algorithm uses adaptive filter to estimate the acoustic feedback received by microphone using the audio broadcast by the speaker, and removes the feedback by subtraction. Compared with the above two solutions, the effect is very significant and can achieve higher sound amplification gain, but the adjustment of related parameters is more complicated.
[0103] In recent years, the use of neural networks for acoustic feedback suppression has promoted the development of related fields. However, most solutions rely too much on neural networks to handle a variety of scenarios, resulting in the network being too large and difficult to implement in general embedded devices or vehicle systems. In addition, there are problems such as overfitting, mismatch with actual scenarios, and difficulty in subsequent modifications.
[0104] It can be seen that the various solutions adopted by the existing audio signal processing methods for controlling howling have certain limitations, which makes it impossible to effectively control howling while ensuring the stable operation of the acoustic system, resulting in poor sound amplification effect of the acoustic system.
[0105] Therefore, how to provide an audio signal processing method that can improve the effect of controlling howling while ensuring the stable operation of the acoustic system, and further improve the sound amplification effect of the acoustic system, has become an urgent problem to be solved by those skilled in the art.
[0106] In order to solve the above problems, an embodiment of the present application provides an audio signal processing method. The audio signal processing method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.
[0107] Reference Figure 1 , Figure 1 A flowchart of an audio signal processing method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, an audio signal processing method provided in an embodiment of the present application may include the following steps, which are described in detail below.
[0108] S101: Acquire mixed audio signals collected by various microphones;
[0109] In actual scenarios (i.e., scenarios that require sound amplification), microphone groups and speaker groups are often included. The microphone group often includes multiple microphones, and each microphone will collect a mixed audio signal. In the present application, the mixed audio signal collected by each microphone includes the target audio signal and other audio signals that actually need to be collected. The target audio signal is the audio signal collected by the microphone through the acoustic near-end path after the user emits the audio. The other audio signals include playback audio signals (i.e., the audio signal collected by the microphone through the acoustic feedback path after the speaker broadcasts the audio), background audio signals (such as music, voice broadcasts, and other media sounds), and noise audio signals, etc. Therefore, it is necessary to separate (remove) other audio signals in the mixed audio signal to obtain the target audio signal, thereby achieving howling suppression at the source.
[0110] S102: performing echo cancellation, feedback cancellation, noise reduction and reverberation removal processing on the mixed audio signal collected by each microphone to obtain a target audio sub-signal corresponding to the microphone;
[0111] In the present application, for the mixed audio signal collected by each microphone, the mixed audio signal can be sequentially subjected to echo cancellation processing, feedback cancellation processing, and noise reduction and dereverberation processing to obtain a target audio sub-signal corresponding to the microphone.
[0112] S103: Combining and processing the target audio sub-signals corresponding to the microphones to obtain a target audio signal;
[0113] In the present application, the existing mixing method can be used to combine and process the target audio sub-signals corresponding to the microphones to obtain the target audio signal, which will not be described in detail in the present application.
[0114] S104: performing howling suppression processing on the target audio signal to obtain a target audio signal after the howling suppression processing;
[0115] In a possible implementation, the target audio signal may be subjected to howling suppression by using an existing howling suppression algorithm (such as a howling detection control equalizer notch filter) to obtain a target audio signal after howling suppression processing.
[0116] S105: amplifying the target audio signal after the howling suppression processing, and playing it through a loudspeaker.
[0117] In the present application, amplifying the target audio signal after the howling suppression processing includes adding sound effects to the target audio signal after the howling suppression processing. In one achievable manner, the target audio signal after the howling suppression processing can be played through a loudspeaker after adding sound effects through an existing sound effect adding algorithm (equalizer, reverberation, etc.). In addition, the loudspeaker also obtains the background audio signal to be played (such as music, voice broadcast and other media sounds) for playback.
[0118] The audio signal processing method disclosed in this embodiment, after obtaining the mixed audio signals collected by each microphone, first performs echo elimination processing, feedback elimination processing, and noise reduction and reverberation processing on each mixed audio signal to obtain the target audio sub-signal corresponding to the microphone, and then combines the target audio sub-signals corresponding to each microphone to obtain the target audio signal, and then further performs howling suppression processing on the target audio signal to obtain the target audio signal after the howling suppression processing, and finally, after amplifying the target audio signal after the howling suppression processing, it is played through a loudspeaker. This scheme can improve the howling control effect through multi-level howling control, so that the target audio signal collected by the microphone can be played using the same loudspeaker without affecting the original playback content of the loudspeaker, thereby achieving a better amplification effect and avoiding howling.
[0119] In another embodiment of the present application, a specific implementation method of performing echo cancellation processing, feedback cancellation processing, and noise reduction and dereverberation processing on the mixed audio signal to obtain a target audio sub-signal corresponding to the microphone is described in detail.
[0120] Reference Figure 2 , Figure 2 A flow chart of a method for performing echo cancellation, feedback cancellation, and noise reduction and reverberation removal processing on a mixed audio signal to obtain a target audio sub-signal corresponding to a microphone is provided in an embodiment of the present application. The method may include the following steps:
[0121] S201: performing echo cancellation processing on the mixed audio signal to obtain an audio signal after the echo cancellation processing;
[0122] In this application, the background audio signal (such as music, voice broadcast and other media sounds) is not the target audio signal, but the content that the speaker is supposed to broadcast. Therefore, in this application, the signal stream of the background audio signal can be obtained, and based on this, the echo cancellation algorithm is used to perform echo cancellation processing on the mixed audio signal to eliminate the background audio signal (such as music, voice broadcast and other media sounds) in the mixed audio signal. Among them, the echo cancellation algorithm is an existing mature algorithm and will not be described in detail in this application.
[0123] S202: Inputting the audio signal after the echo cancellation processing into a feedback cancellation model to obtain an audio signal after the feedback cancellation processing;
[0124] In the present application, the existence of a playback audio signal (i.e., a target audio signal that is amplified and collected by a microphone after being broadcast by a speaker) may cause howling. Therefore, in the present application, the playback audio signal (i.e., a target audio signal that is amplified and collected by a microphone after being broadcast by a speaker) in the audio signal after echo cancellation processing can be eliminated based on feedback cancellation processing. In a possible implementation, a feedback cancellation model can be pre-trained, and the audio signal after echo cancellation processing and the playback signal of the speaker broadcast signal are input into the feedback cancellation model to obtain the audio signal after feedback cancellation processing.
[0125] In the present application, the feedback elimination model estimates the linear filter coefficients for linear feedback suppression, thereby obtaining a feedback suppression effect that is significantly better than the signal processing algorithm. In order to minimize the amount of computation and memory occupied by the feedback elimination model, in the present application, the feedback elimination model can estimate a suitable update step size for its output based on the update amount of the traditional NLMS (Normalized Least Mean Squares) algorithm, thereby improving the effect of feedback suppression and minimizing the requirements for the scale of the feedback elimination model. Based on this, the feedback elimination model takes the feedback reference signal (i.e., the echo signal of the simulated loudspeaker broadcast signal), feedback error, and update amount as inputs, shares the same set of model parameters for all frequency points, and outputs the update step size for each frequency point. Based on the update step size of each frequency point, the linear filter coefficients for linear feedback suppression can be estimated to obtain the output of the feedback elimination model.
[0126] For ease of understanding, refer to Figure 3 , Figure 3 A schematic diagram of the working principle of a feedback elimination model provided in an embodiment of the present application.
[0127] In a possible implementation, the structure of the feedback elimination model may use only a single-layer GRU and a linear layer.
[0128] S203: Input the audio signal after the feedback elimination process into a noise reduction and dereverberation model to obtain the audio signal after the noise reduction and dereverberation process as the target audio sub-signal.
[0129] In this application, the audio signal after noise reduction and demixing processing has high quality and can be used for back-end processing (for example, voice wake-up, voice recognition, etc.).
[0130] In the present application, the noise reduction and dereverberation model improves the voice quality while reducing noise, removing residual echo and residual reverberation as much as possible by restoring the amplitude and phase to ensure the listening experience. In the present application, the structure of the noise reduction and dereverberation model can adopt the deepfilter two-stage model. In order to reduce the amount of calculation and memory occupied by the model as much as possible, this case has greatly compressed the original deepfilter model, mainly for the GRU part of the model, reducing the GRU hidden layer dimension, and grouping the GRU dimension for calculation, thereby greatly reducing the amount of calculation and the number of parameters. In order to reduce the algorithm delay, this algorithm uses asymmetric window technology, so that the algorithm can process very short frame lengths.
[0131] In another embodiment of the present application, the training method of the feedback elimination model and the denoising and dereverberation model is described.
[0132] Reference Figure 4 , Figure 4 A flow chart of a training method for a feedback elimination model and a noise reduction and dereverberation model provided in an embodiment of the present application, the flow may include the following steps:
[0133] S301: Acquire a pre-recorded acoustic near-end path and an acoustic feedback path;
[0134] In this application, the recording methods of the acoustic proximal path and the acoustic feedback path include:
[0135] S3011: Calibrate microphones and speakers in actual scenes;
[0136] In order to make the training data obtained by simulation match the actual scene, it is necessary to calibrate the microphone and speaker in the actual scene. Since the microphone and speaker are fixed in the actual scene, in this application, the process of calibrating the microphone and speaker in the actual scene only involves adjusting the gain of the microphone and speaker.
[0137] S3012: Select a target microphone and a target speaker;
[0138] For a directional microphone, you can select a microphone that points toward the speaker and is not in front of the speaker as the target microphone, and select a speaker that is not in front of the microphone as the target speaker. For an omnidirectional microphone, you can select a microphone that has a large angle with the speaker as the target microphone.
[0139] S3013: Recording the acoustic near-end path and the acoustic feedback path by using the target microphone and the target speaker;
[0140] In the present application, the acoustic feedback path can be recorded by using the target microphone and the target speaker in a manner where a single speaker broadcasts and all microphones receive. In one feasible manner, the target speaker can play an MLS (Maximum Length Sequence, pseudo-random number sequence) signal, and the acoustic feedback path is recorded based on the reception of the target microphone. It should be noted that in order to ensure the diversity of the acoustic feedback path, different types of occlusions can be placed to record more acoustic feedback paths for expanding the training data. In the case where the speaker or microphone moves, more acoustic feedback paths also need to be recorded. In the present application, a typical sound source position can be selected to record the acoustic proximal path.
[0141] S302: Acquire a preset speech data set, where the speech data set includes clean speech data and noise data;
[0142] S303: Performing data simulation using the acoustic proximal path and the acoustic feedback path, as well as the speech data set, to generate feedback elimination training data and noise reduction and dereverberation training data, wherein the feedback elimination training data and the noise reduction and dereverberation training data are both signals containing multiple continuous frames;
[0143] In the present application, in the traditional scheme of using neural networks for acoustic feedback suppression, model training is performed in a closed-loop system by frame-by-frame calculation. Practice has shown that this training method will seriously slow down the training speed on the one hand, and on the other hand, it cannot control the distribution of training data, because the performance of the model is uncertain during the training process, which will either cause howling and lead to data anomalies, or cause abnormal distortion of the model output. Although howling may not necessarily occur at this time, it will cause the data to be too biased and the robustness of the model cannot be improved. Therefore, in the present application, the acoustic proximal path and the acoustic feedback path, as well as the speech data set, are used for data simulation to generate feedback elimination training data and noise reduction and dereverberation training data. The signals containing multiple consecutive frames can be pre-calculated frame by frame in a closed-loop manner to obtain signals. By using these signals containing multiple consecutive frames, the closed-loop training of the feedback elimination model and the noise reduction and dereverberation model can be avoided. Moreover, the process of using the acoustic proximal path and the acoustic feedback path, as well as the speech data set to perform data simulation to generate feedback elimination training data and noise reduction and dereverberation training data can be run in multiple threads on the CPU, and the efficiency is much higher than that of closed-loop training. By using these signals containing multiple consecutive frames to train the model, the GPU optimization of the RNN network can be fully utilized to quickly train a large amount of data, thereby optimizing the generalization ability and stability of the model, and thus improving the training efficiency and effect.
[0144] S304: training a feedback elimination model using the feedback elimination training data to obtain the trained feedback elimination model;
[0145] In a possible implementation, the training of the feedback cancellation model using the feedback cancellation training data to obtain the trained feedback cancellation model includes: acquiring a first audio signal label corresponding to the feedback cancellation training data;
[0146] The feedback cancellation training data is input into the feedback cancellation model, and the feedback cancellation model is trained with the output of the feedback cancellation model approaching the first audio signal label as a training goal to obtain the trained feedback cancellation model.
[0147] It should be noted that, in the present application, the method of training the feedback cancellation model using the feedback cancellation training data can refer to the existing related training methods, the difference being that the training data is a signal sequence containing multiple frames.
[0148] S305: Training a denoising and dereverberation model using the denoising and dereverberation training data to obtain the trained denoising and dereverberation model.
[0149] In a possible implementation, the using the denoising and dereverberation training data to train the denoising and dereverberation model to obtain the trained denoising and dereverberation model includes: obtaining a second audio signal label corresponding to the denoising and dereverberation training data; inputting the denoising and dereverberation training data into the denoising and dereverberation model, training the denoising and dereverberation model with the output of the denoising and dereverberation model approaching the second audio signal label as a training goal, and obtaining the trained denoising and dereverberation model.
[0150] It should be noted that, in the present application, the method of training the denoising and dereverberation model using the denoising and dereverberation training data can refer to the existing related training methods, the difference being that the training data is a signal sequence containing multiple frames.
[0151] In another embodiment of the present application, a specific implementation method of performing data simulation using the acoustic proximal path and the acoustic feedback path, as well as the speech data set to generate feedback elimination training data and noise reduction and dereverberation training data is described.
[0152] Reference Figure 5 , Figure 5 A flow chart of using an acoustic proximal path and an acoustic feedback path, as well as a speech data set to perform data simulation and generate feedback elimination training data and noise reduction and reverberation removal training data is provided in an embodiment of the present application. The flow chart may include the following steps:
[0153] S401: Acquire a pre-built data simulation system, wherein the data simulation system comprises a simulated microphone group, a feedback elimination simulation model, a noise reduction and dereverberation simulation model, a signal combination module, a sound effect adding module, and a simulated speaker; wherein the feedback elimination simulation model and the noise reduction and dereverberation simulation model correspond to preset parameters;
[0154] In the present application, the feedback elimination simulation model is not an actual feedback elimination model, and the noise reduction and reverberation simulation model is not an actual noise reduction and reverberation model, but a fake model. The fake model here is not an actual neural network, but in the simulation data, because the target of the model output is known, some feedback, noise, reverberation, etc. can be retained on the basis of this target to simulate the output of the actual model. Specifically, the feedback elimination simulation model and the noise reduction and reverberation simulation model can be abstracted into some parameters to reflect different model capabilities. In an achievable manner, the feedback elimination simulation model is abstracted into a misalignment (system misalignment, indicating the error of the acoustic feedback path estimated by the model relative to the actual path) parameter, and the noise reduction and reverberation simulation model is abstracted into SNR (signal-to-noise ratio, here indicating the amount of improvement in the signal-to-noise ratio of the input signal by the model), SRR (source to reverberation ratio, here indicating the amount of improvement in the source reverberation ratio of the input signal by the model), and vad (the model has the ability to directly set non-speech segments to 0).
[0155] S402: Obtain data distribution of preset parameters corresponding to the feedback elimination simulation model and the noise reduction and dereverberation simulation model;
[0156] S403: Utilizing the acoustic proximal path, the acoustic feedback path, the data distribution of the preset parameters corresponding to the feedback elimination simulation model and the noise reduction and dereverberation simulation model and the data simulation system, perform data simulation on the speech data set to generate the feedback elimination training data and the noise reduction and dereverberation training data.
[0157] In a possible implementation, the method of using the acoustic proximal path, the acoustic feedback path, the data distribution of the preset parameters corresponding to the feedback elimination simulation model and the noise reduction and dereverberation simulation model and the data simulation system to perform data simulation on the speech data set to generate the feedback elimination training data and the noise reduction and dereverberation training data includes:
[0158] S4031: for each group of simulation data in the speech data set, sampling and obtaining preset parameters corresponding to the simulation data in data distributions of preset parameters corresponding to the feedback elimination simulation model and the noise reduction and dereverberation simulation model;
[0159] Each group of simulation data in the speech data set includes one clean speech data and one noise data.
[0160] S4032: Determine a simulated mixed audio signal corresponding to the simulation data based on the acoustic near-end path and the acoustic feedback path;
[0161] In the present application, the acoustic near-end path and the acoustic feedback path can be used to mix a clean speech data and a noise data in the simulation data into a simulated mixed audio signal collected by a simulated microphone group.
[0162] S4033: Input the simulated mixed audio signal into the data simulation system, iterate frame by frame, and obtain the input of the feedback elimination simulation model and the input of the noise reduction and dereverberation simulation model, wherein the input of the feedback elimination simulation model is the feedback elimination training data, and the input of the noise reduction and dereverberation simulation model is the noise reduction and dereverberation training data.
[0163] It should be noted that in the process of iterating frame by frame, if it is the first frame, due to system delay, the simulated speaker has no content to play, so there is no feedback, and the feedback elimination simulation model does not work. The noise reduction and dereverberation simulation model reduces the ratio of reverberation and noise in the simulated mixed audio signal according to the sampled parameters, and then mixes and adds sound effects through the signal combination module and the sound effect addition module to obtain the future broadcast signal of the simulated speaker. In the subsequent frames, the simulated speaker starts to play, and the feedback elimination simulation model also starts to work. It reduces the feedback ratio in the simulated mixed audio signal according to the misalignment, and outputs it to the noise reduction and dereverberation simulation model to reduce the ratio of reverberation and noise, and then mixes and adds sound effects through the signal combination module and the sound effect addition module to obtain the future broadcast signal of the simulated speaker. For ease of understanding, refer to Figure 6 , Figure 6 A schematic diagram of a data simulation process of a data simulation system provided in an embodiment of the present application.
[0164] An audio signal processing method provided in an embodiment of the present application is introduced above, and a device for executing the above audio signal processing method will be introduced below.
[0165] See also Figure 7 , Figure 7 Schematic diagram of the structure of an audio signal processing device provided in an embodiment of the present application. Figure 7 As shown, the audio signal processing device comprises:
[0166] An acquisition unit 11 is used to acquire mixed audio signals collected by various microphones;
[0167] The first processing unit 12 is used to perform echo cancellation processing, feedback cancellation processing, noise reduction and reverberation removal processing on the mixed audio signal collected by each microphone to obtain a target audio sub-signal corresponding to the microphone;
[0168] A signal combination processing unit 13, used for combining and processing the target audio sub-signals corresponding to the microphones to obtain a target audio signal;
[0169] A howling suppression processing unit 14 is used to perform a howling suppression process on the target audio signal to obtain a target audio signal after the howling suppression process;
[0170] The amplifying unit 15 is used to amplify the target audio signal after the howling suppression processing and play it through a speaker.
[0171] In a possible implementation, the first processing unit includes:
[0172] An echo cancellation processing unit, configured to perform echo cancellation processing on the mixed audio signal to obtain an audio signal after the echo cancellation processing;
[0173] A feedback elimination processing unit, used for inputting the audio signal after the echo elimination processing into a feedback elimination model to obtain the audio signal after the feedback elimination processing;
[0174] The denoising and dereverberation processing unit is used to input the audio signal after the feedback elimination processing into the denoising and dereverberation model to obtain the audio signal after the denoising and dereverberation processing as the target audio sub-signal.
[0175] In a possible implementation, the device further includes: a model training unit;
[0176] The model training unit comprises:
[0177] An acoustic path acquisition unit, used for acquiring a pre-recorded acoustic near-end path and an acoustic feedback path;
[0178] A speech data set acquisition unit, used to acquire a preset speech data set, wherein the speech data set includes clean speech data and noise data;
[0179] A training data generating unit, configured to perform data simulation using the acoustic proximal path and the acoustic feedback path, as well as the speech data set, to generate feedback elimination training data and noise reduction and dereverberation training data, wherein the feedback elimination training data and the noise reduction and dereverberation training data are both signals containing multiple continuous frames;
[0180] A feedback elimination model training unit, used to train the feedback elimination model using the feedback elimination training data to obtain the trained feedback elimination model;
[0181] The denoising and dereverberation model training unit is used to train the denoising and dereverberation model using the denoising and dereverberation training data to obtain the trained denoising and dereverberation model.
[0182] In a possible implementation, the device further includes: an acoustic path recording unit;
[0183] The acoustic path recording unit is specifically used for:
[0184] Calibrate microphones and speakers in real-world scenarios;
[0185] Select the target microphone and target speaker;
[0186] The acoustic near-end path and the acoustic feedback path are recorded using the target microphone and the target speaker.
[0187] In a possible implementation, the training data generating unit includes:
[0188] A data simulation system acquisition unit, used to acquire a pre-built data simulation system, wherein the data simulation system comprises a simulated microphone group, a feedback elimination simulation model, a noise reduction and dereverberation simulation model, a signal combination module, a sound effect adding module, and a simulated speaker; wherein the feedback elimination simulation model and the noise reduction and dereverberation simulation model correspond to preset parameters;
[0189] A parameter data distribution acquisition unit, used to acquire data distribution of preset parameters corresponding to the feedback elimination simulation model and the noise reduction and dereverberation simulation model;
[0190] A data simulation unit is used to utilize the data distribution of preset parameters corresponding to the acoustic proximal path, the acoustic feedback path, the feedback elimination simulation model, and the noise reduction and dereverberation simulation model and the data simulation system to perform data simulation on the speech data set to generate the feedback elimination training data and the noise reduction and dereverberation training data.
[0191] In a possible implementation, the data simulation unit is specifically used to:
[0192] For each group of simulation data in the speech data set, sampling is performed to obtain preset parameters corresponding to the simulation data in the data distribution of the preset parameters corresponding to the feedback elimination simulation model and the noise reduction and dereverberation simulation model;
[0193] Determining a simulated mixed audio signal corresponding to the simulation data based on the acoustic near-end path and the acoustic feedback path;
[0194] The simulated mixed audio signal is input into the data simulation system, and it is iterated frame by frame to obtain the input of the feedback elimination simulation model and the input of the noise reduction and dereverberation simulation model, wherein the input of the feedback elimination simulation model is the feedback elimination training data, and the input of the noise reduction and dereverberation simulation model is the noise reduction and dereverberation training data.
[0195] In a possible implementation, the feedback elimination model training unit is specifically used to:
[0196] Acquire a first audio signal label corresponding to the feedback elimination training data;
[0197] The feedback cancellation training data is input into the feedback cancellation model, and the feedback cancellation model is trained with the output of the feedback cancellation model approaching the first audio signal label as a training goal to obtain the trained feedback cancellation model.
[0198] In a possible implementation, the denoising and dereverberation model training unit is specifically used to:
[0199] Obtaining a second audio signal label corresponding to the denoising and dereverberation detaching training data;
[0200] The denoising and dereverberation training data is input into the denoising and dereverberation model, and the denoising and dereverberation model is trained with the output of the denoising and dereverberation model approaching the second audio signal label as a training goal to obtain the trained denoising and dereverberation model.
[0201] The present application also provides an electronic device in an embodiment. Figure 8 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0202] like Figure 8 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0203] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0204] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any audio signal processing method provided in the embodiment of the present application.
[0205] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any audio signal processing method provided in the embodiment of the present application.
[0206] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.
[0207] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0208] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0209] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.
Claims
1. An audio signal processing method, characterized in that: include: Obtain mixed audio signals collected by each microphone; For the mixed audio signal collected by each microphone, the mixed audio signal is subjected to echo cancellation processing, feedback cancellation processing, noise reduction and reverberation removal processing to obtain the target audio sub-signal corresponding to the microphone; Combining and processing the target audio sub-signals corresponding to the microphones to obtain a target audio signal; Performing a howling suppression process on the target audio signal to obtain a target audio signal after the howling suppression process; The target audio signal after the howling suppression processing is amplified and then played through a loudspeaker.
2. The method according to claim 1, characterized in that The performing echo cancellation processing, feedback cancellation processing, and noise reduction and reverberation removal processing on the mixed audio signal to obtain a target audio sub-signal corresponding to the microphone includes: Performing echo cancellation processing on the mixed audio signal to obtain an audio signal after the echo cancellation processing; Inputting the audio signal after the echo cancellation processing into a feedback cancellation model to obtain the audio signal after the feedback cancellation processing; The audio signal after the feedback elimination process is input into a noise reduction and dereverberation model to obtain the audio signal after the noise reduction and dereverberation process as the target audio sub-signal.
3. The method according to claim 2, characterized in that The training method of the feedback elimination model and the denoising and dereverberation model includes: obtaining a pre-recorded acoustic near-end path and an acoustic feedback path; Acquire a preset speech data set, wherein the speech data set includes clean speech data and noise data; Performing data simulation using the acoustic proximal path and the acoustic feedback path, as well as the speech data set, to generate feedback elimination training data and noise reduction and dereverberation training data, wherein the feedback elimination training data and the noise reduction and dereverberation training data are both signals containing multiple continuous frames; Using the feedback elimination training data to train a feedback elimination model to obtain the trained feedback elimination model; The denoising and dereverberation training data is used to train a denoising and dereverberation model to obtain the trained denoising and dereverberation model.
4. The method according to claim 3, characterized in that The recording method of the acoustic proximal path and the acoustic feedback path includes: Calibrate microphones and speakers in real-world scenarios; Select the target microphone and target speaker; The acoustic near-end path and the acoustic feedback path are recorded using the target microphone and the target speaker.
5. The method according to claim 3, characterized in that: The method of using the acoustic proximal path and the acoustic feedback path, as well as the speech data set to perform data simulation to generate feedback elimination training data and noise reduction and reverberation removal training data, includes: Obtain a pre-built data simulation system, the data simulation system comprising a simulated microphone group, a feedback elimination simulation model, a noise reduction and dereverberation simulation model, a signal combination module, a sound effect adding module and a simulated speaker; wherein the feedback elimination simulation model and the noise reduction and dereverberation simulation model correspond to preset parameters; Obtaining data distribution of preset parameters corresponding to the feedback elimination simulation model and the noise reduction and dereverberation simulation model; The acoustic proximal path, the acoustic feedback path, the feedback elimination simulation model, and the noise reduction and dereverberation simulation model have data distributions of corresponding preset parameters and the data simulation system to perform data simulation on the speech data set to generate the feedback elimination training data and the noise reduction and dereverberation training data.
6. The method according to claim 5, characterized in that The method utilizes the acoustic proximal path, the acoustic feedback path, the feedback elimination simulation model, and the noise reduction and dereverberation simulation model to perform data simulation on the speech data set to generate the feedback elimination training data and the noise reduction and dereverberation training data, and the data simulation system, including: For each group of simulation data in the speech data set, sampling is performed to obtain preset parameters corresponding to the simulation data in the data distribution of the preset parameters corresponding to the feedback elimination simulation model and the noise reduction and dereverberation simulation model; Determining a simulated mixed audio signal corresponding to the simulated data based on the acoustic near-end path and the acoustic feedback path; The simulated mixed audio signal is input into the data simulation system, and it is iterated frame by frame to obtain the input of the feedback elimination simulation model and the input of the noise reduction and dereverberation simulation model, wherein the input of the feedback elimination simulation model is the feedback elimination training data, and the input of the noise reduction and dereverberation simulation model is the noise reduction and dereverberation training data.
7. The method according to claim 3, characterized in that The using the feedback elimination training data to train the feedback elimination model to obtain the trained feedback elimination model includes: Acquire a first audio signal label corresponding to the feedback elimination training data; The feedback cancellation training data is input into the feedback cancellation model, and the feedback cancellation model is trained with the output of the feedback cancellation model approaching the first audio signal label as a training goal to obtain the trained feedback cancellation model.
8. The method according to claim 3, characterized in that The step of training the denoising and dereverberation model using the denoising and dereverberation training data to obtain the trained denoising and dereverberation model comprises: Obtaining a second audio signal label corresponding to the denoising and dereverberation detaching training data; The denoising and dereverberation training data is input into the denoising and dereverberation model, and the denoising and dereverberation model is trained with the output of the denoising and dereverberation model approaching the second audio signal label as a training goal to obtain the trained denoising and dereverberation model.
9. An audio signal processing device, characterized in that: include: An acquisition unit, used to acquire mixed audio signals collected by each microphone; A first processing unit is used to perform echo cancellation processing, feedback cancellation processing, noise reduction and reverberation removal processing on the mixed audio signal collected by each microphone to obtain a target audio sub-signal corresponding to the microphone; A signal combination processing unit, used for combining and processing the target audio sub-signals corresponding to the microphones to obtain a target audio signal; a howling suppression processing unit, configured to perform a howling suppression process on the target audio signal to obtain a target audio signal after the howling suppression process; The amplifying unit is used to amplify the target audio signal after the howling suppression processing and play it through a loudspeaker.
10. A computer program product, characterized in that The method comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the audio signal processing method according to any one of claims 1 to 8.
11. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the audio signal processing method according to any one of claims 1 to 8.
12. A computer-readable storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the audio signal processing method as described in any one of claims 1 to 8.