Signal processing method, array microphone device, computer device, storage medium and program product
Through the single-input channel and dual-output channel design, local sound reinforcement and remote signals in remote meetings are independently processed, solving noise and echo problems, improving audio clarity and call experience.
Patent Information
- Application Number
- CN202411482876.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-23
- Publication Date
- 2024-12-10
AI Technical Summary
In remote meetings, speakers are often accompanied by noise and echoes that are not related to the content of the speech, making it difficult for remote and local participants to obtain accurate conference voice information.
Using a single input channel and dual output channel design, the speaker reference signal is received through a reference input channel, and the local sound reinforcement signal and remote signal are independently processed to achieve echo cancellation and noise reduction. The system hardware design is simple and has good compatibility. sex.
It realizes independent processing of local sound reinforcement and remote signals, ensuring that the sound quality and local sound reinforcement effects of remote calls do not interfere with each other in complex scenarios, significantly improving audio clarity and call experience.
Smart Images
Figure CN119110189A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of microphone array technology, and in particular to a signal processing method, an array microphone device, a computer device, a storage medium, and a program product. Background Art
[0002] With the increasing frequency of business activities, multi-person remote conferences are becoming more and more widely used. Through terminals with voice call functions (such as array microphone devices), different people in different locations can hold online conferences based on the network. All participants do not need to gather together, and only need to transmit audio information over long distances. When speakers speak, they are often accompanied by noise and echoes from the surrounding environment that are irrelevant to the content of the speech, which is not conducive to remote and local participants obtaining accurate conference voice information. Therefore, there is an urgent need for an array microphone device that can effectively perform echo cancellation and noise reduction. Summary of the invention
[0003] The embodiments of the present application provide a signal processing method, an array microphone device, a computer device, a storage medium and a program product, which are used to solve at least one of the above-mentioned technical problems.
[0004] In a first aspect, an embodiment of the present application provides a signal processing method for an array microphone device, wherein the array microphone device includes a plurality of microphones and a processor, wherein the processor includes a first input terminal, a first output terminal, and a second output terminal, wherein the first output terminal outputs a signal to a local speaker, and the second output terminal outputs a signal to a remote end, and the plurality of microphones are used to collect multi-channel microphone signals, and the method includes: The processor collects a speaker reference signal through a communication path between the first output terminal and the first input terminal; The processor performs a first preset processing on the multi-channel microphone signal at least according to the speaker reference signal to obtain a first output signal, and outputs the first output signal to the local speaker through the first output terminal; The processor performs a second preset processing on the multi-channel microphone signals at least according to the loudspeaker reference signal to obtain a second output signal, and outputs the second output signal to the remote end through the second output terminal.
[0005] In some embodiments, the processor performs a first preset processing on the multi-channel microphone signal at least according to the speaker reference signal to obtain a first output signal, including: The processor performs a first echo cancellation process on the multi-channel microphone signals at least according to the speaker reference signal to obtain a first echo cancellation result; The processor performs sound source localization at least according to the multi-channel microphone signals; The processor performs beamforming processing on the first echo cancellation result at least according to the sound source localization result, and performs second echo cancellation processing on the beamformed signal to obtain a first output signal.
[0006] In some embodiments, the processor performs a second preset processing on the multi-channel microphone signal at least according to the speaker reference signal to obtain a second output signal, including: The processor performs a first echo cancellation process on the multi-channel microphone signals at least according to the speaker reference signal to obtain a first echo cancellation result; The processor performs sound source localization at least according to the multi-channel microphone signals; The processor performs beamforming processing on the first echo cancellation result at least according to the sound source localization result, and performs echo suppression processing on the beamformed signal to obtain a second output signal.
[0007] In some embodiments, the processor further includes a second input terminal, and the processor obtains a downlink audio signal from the remote end through the second input terminal; the processor performs sound source localization at least according to the multi-channel microphone signal, including: The processor performs sound source localization according to the multi-channel microphone signals and the downlink audio signal.
[0008] In some embodiments, the processor performs beamforming processing on the first echo cancellation result at least according to the sound source localization result, and performs echo suppression processing on the beamformed signal, including: When the downlink audio signal indicates that there is far-end voice activity, performing a first echo suppression process on the beamformed signal; When the downlink audio signal indicates that there is no far-end voice activity, a second echo suppression process is performed on the beamformed signal; wherein the strength of the first echo suppression is greater than the strength of the second echo suppression.
[0009] In some embodiments, the processor further includes a third output terminal, and the method further includes: the processor performs sound source localization at least according to the multi-channel microphone signal to determine the coordinate information of the sound source and outputs it to an external device through the third output terminal.
[0010] In some embodiments, it also includes: Before outputting the first output signal, the first output signal is subjected to local sound reinforcement preprocessing, wherein the local sound reinforcement preprocessing includes at least one of the following: noise suppression, neural network reverberation suppression, automatic gain control and frequency equalization processing; and / or, Before outputting the second output signal, the second output signal is subjected to remote signal preprocessing, wherein the remote signal preprocessing includes at least one of the following: noise suppression, neural network dynamic rebound suppression, automatic gain control, and frequency equalization processing.
[0011] In some embodiments, after performing local sound reinforcement preprocessing on the first output signal, a local sound reinforcement signal is obtained, and the preprocessed first output signal (i.e., the local sound reinforcement signal) is further mixed with the downlink audio signal to obtain a final output signal, which is output through the first output terminal.
[0012] In this embodiment, the downlink audio signal is input into the array microphone device through the second input terminal In2, and the downlink audio signal is mixed with the local sound reinforcement signal by the mixing module inside the array microphone, and finally the output of the local sound reinforcement channel is obtained and played by the external speaker.
[0013] In a second aspect, the present application further provides an array microphone device, wherein the array microphone device comprises a plurality of microphones and a processor, the processor comprises a first input terminal, a first output terminal, and a second output terminal, wherein the first output terminal outputs a signal to a local speaker, the second output terminal outputs a signal to a remote end, and the plurality of microphones are used to collect multi-channel microphone signals; The processor is configured as follows: The processor collects a speaker reference signal through a communication path between the first output terminal and the first input terminal; The processor performs a first preset processing on the multi-channel microphone signal at least according to the speaker reference signal to obtain a first output signal, and outputs the first output signal to the local speaker through the first output terminal; The processor performs a second preset processing on the multi-channel microphone signals at least according to the loudspeaker reference signal to obtain a second output signal, and outputs the second output signal to the remote end through the second output terminal.
[0014] In some embodiments, performing a first preset processing on the multi-channel microphone signal at least according to the speaker reference signal to obtain a first output signal includes: Performing a first echo cancellation process on the multi-channel microphone signals at least according to the loudspeaker reference signal to obtain a first echo cancellation result; Performing sound source localization at least according to the multi-channel microphone signals; The first echo cancellation result is subjected to beamforming processing at least according to the sound source localization result, and a second echo cancellation processing is performed on the beamformed signal to obtain a first output signal.
[0015] In some embodiments, performing a second preset processing on the multi-channel microphone signal at least according to the speaker reference signal to obtain a second output signal includes: Performing a first echo cancellation process on the multi-channel microphone signals at least according to the loudspeaker reference signal to obtain a first echo cancellation result; Performing sound source localization at least according to the multi-channel microphone signals; At least according to the sound source localization result, beamforming processing is performed on the first echo cancellation result, and echo suppression processing is performed on the beamformed signal to obtain a second output signal.
[0016] In some embodiments, the processor further includes a second input terminal, and the processor obtains a downlink audio signal from the remote end through the second input terminal; and performing sound source localization at least according to the multi-channel microphone signal includes: Sound source localization is performed according to the multi-channel microphone signals and the downlink audio signal.
[0017] In some embodiments, performing beamforming processing on the first echo cancellation result at least according to the sound source localization result, and performing echo suppression processing on the beamformed signal, includes: When the downlink audio signal indicates that there is far-end voice activity, performing a first echo suppression process on the beamformed signal; When the downlink audio signal indicates that there is no far-end voice activity, a second echo suppression process is performed on the beamformed signal; wherein the strength of the first echo suppression is greater than the strength of the second echo suppression.
[0018] In some embodiments, the processor further includes a third output terminal, and the processor is further configured to: perform sound source localization at least according to the multi-channel microphone signal to determine the coordinate information of the sound source and output it to an external device through the third output terminal.
[0019] In some embodiments, the processor is further configured to: Before outputting the first output signal, the first output signal is subjected to local sound reinforcement preprocessing, wherein the local sound reinforcement preprocessing includes at least one of the following: noise suppression, neural network reverberation suppression, automatic gain control and frequency equalization processing; and / or, Before outputting the second output signal, the second output signal is subjected to remote signal preprocessing, wherein the remote signal preprocessing includes at least one of the following: noise suppression, neural network dynamic rebound suppression, automatic gain control, and frequency equalization processing.
[0020] In a third aspect, the present application further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in any embodiment of the present application.
[0021] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program / instruction stored thereon, wherein the computer program / instruction, when executed by a processor, implements the steps of the method of any embodiment of the present application.
[0022] In a fifth aspect, the present application also provides a computer program product, including a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the steps of the method described in any embodiment of the present application are implemented.
[0023] The embodiment of the present application adopts a single input channel and dual output channel design, and only uses a reference input channel formed by a first input end to receive a loudspeaker reference signal to achieve independent processing of a local sound reinforcement signal (first output signal) and a remote signal (second output signal), and can independently achieve noise reduction and echo cancellation processing of the two signals. The system hardware design is simple and has good compatibility. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0025] Figure 1 A schematic diagram of conducting a remote conference through a conference device in the related art; Figure 2 A flowchart of an embodiment of a signal processing method of the present application; Figure 3 A schematic diagram of the principle of an embodiment of an array microphone device of the present application; Figure 4 A schematic diagram of another embodiment of the array microphone device of the present application; Figure 5 A principle block diagram of another embodiment of the array microphone device of the present application; Figure 6 A principle block diagram of another embodiment of the array microphone device of the present application; Figure 7 A principle block diagram of another embodiment of the array microphone device of the present application; Figure 8A principle block diagram of another embodiment of the array microphone device of the present application; Fig. 9 A flowchart of another embodiment of the signal processing method of the present application; Fig.10 It is a structural schematic diagram of an embodiment of a computer device of the present application. DETAILED DESCRIPTION
[0026] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.
[0027] It should also be noted that, in this article, the terms "include" and "comprise" include not only those elements, but also other elements not explicitly listed, or elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the sentence "include..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.
[0028] like Figure 1 The figure shows a schematic diagram of conducting a remote conference through a conference device in the related art. It includes a local conference device, a remote conference device, a local speaker and a remote speaker. The local conference device can be set at the first venue, and the remote conference device can be set at the second venue. The local conference device collects the audio signal of the first venue through a microphone, and sends the audio signal to the remote conference device through an uplink; the remote conference device collects the audio signal of the second venue through a microphone, and sends the audio signal to the local conference device through a downlink. In addition, the local conference device plays the locally collected audio signal and the audio signal received from the remote conference device through the local speaker; the remote conference device plays the remotely collected audio signal and the audio signal received from the local conference device through the remote speaker.
[0029] In the process of implementing this application, the inventor has designed a variety of different solutions, which are exemplified as follows: Solution 1: Single channel input solution This solution uses only one reference input channel to receive the speaker reference signal. The far-end voice and the speaker echo signal are processed through the same channel, and the system relies on more complex algorithms to distinguish the sources of these signals. However, the inventors found that the mixed signal processing of this solution is complex, which can easily lead to inaccurate signal source distinction, affecting echo cancellation (AEC) and speech clarity. In addition, the system has difficulty in achieving effective isolation between local sound reinforcement and far-end calls, resulting in reduced sound quality.
[0030] Solution 2: External DSP processor solution An external DSP processor is used to separate and process the remote downlink signal and the speaker reference signal. The microphone array is only responsible for picking up the sound, and all signal processing functions are completed on the external DSP. However, the inventor found that the system cost of this solution has increased significantly, and additional hardware support is required, which increases the complexity. In addition, the integration is difficult, and the communication delay between the external processor and the microphone array may affect the real-time performance.
[0031] In order to solve at least one technical problem in the related art, the present application proposes a signal processing method that can be used for an array microphone device, wherein the array microphone device can be a ceiling array microphone, a desktop array microphone, etc. The present application does not limit the specific product form. Figure 2 As shown, an embodiment of the present application provides a signal processing method for an array microphone device. Figure 3 The schematic diagram of the principle of an embodiment of an array microphone device in the present application is shown, wherein the array microphone device includes multiple microphones and a processor, the processor includes a first input terminal In1, a first output terminal Out1 and a second output terminal Out2, wherein the first output terminal outputs a signal to a local speaker, and the second output terminal outputs a signal to a remote end, and the multiple microphones are used to collect multi-channel microphone signals. Exemplarily, the processor is a signal processor, and the array microphone device may also include signal acquisition and peripheral circuits that cooperate with the first input terminal, the first output terminal and the second output terminal, etc. The present application does not limit the specific circuit structure. The signal processing method for an array microphone device provided in the present application includes the following steps: S10, the processor collects a speaker reference signal through a communication path between the first output terminal and the first input terminal; S20, the processor performs a first preset processing on the multi-channel microphone signal at least according to the speaker reference signal to obtain a first output signal, and outputs the first output signal to the local speaker through the first output terminal; S30: The processor performs a second preset processing on the multi-channel microphone signals at least according to the speaker reference signal to obtain a second output signal, and outputs the second output signal to the remote end through the second output terminal.
[0032] This embodiment adopts a single input channel and dual output channel design, and only uses a reference input channel formed by a first input end to receive a loudspeaker reference signal, thereby realizing independent processing of a local sound reinforcement signal (first output signal) and a remote signal (second output signal), and can independently realize noise reduction and echo cancellation processing of the two signals. The system hardware design is simple and has good compatibility.
[0033] like Figure 4 FIG. 1 is a schematic diagram showing another embodiment of an array microphone device of the present application. Figure 3 The difference is that in this embodiment, the array microphone device further includes a second input terminal. Exemplarily, when the array microphone device includes a processor, the processor is provided with a second input terminal, and the processor can obtain the downlink audio signal from the remote end through the second input terminal.
[0034] Further, the processor performs a first preset processing on the multi-channel microphone signals at least according to the speaker reference signal to obtain a first output signal, including: the processor performs a first preset processing on the multi-channel microphone signals at least according to the speaker reference signal, the downlink audio signal, and obtains the first output signal; the processor performs a second preset processing on the multi-channel microphone signals at least according to the speaker reference signal to obtain a second output signal, including: the processor performs a second preset processing on the multi-channel microphone signals at least according to the speaker reference signal, the downlink audio signal, and obtains the second output signal.
[0035] In this embodiment, a dual-channel input and dual-channel output design is implemented. Among them, through this embodiment, a single microphone array system has two independent reference input channels (In1 and In2), which receive the speaker reference signal and the far-end downlink audio signal respectively, thereby effectively distinguishing the far-end voice and local sound reinforcement. In addition, this design also provides an independent signal processing path for mixed scenarios, without relying on an external DSP, and realizes efficient signal processing. Furthermore, through two independent output channels, the far-end call and local sound reinforcement signals are processed separately, ensuring that in complex scenarios, the sound quality of the far-end call and the local sound reinforcement effect do not interfere with each other.
[0036] like Figure 5 FIG. 1 is a schematic block diagram of another embodiment of an array microphone device of the present application. Figure 5 The part in the dotted box corresponds to Figure 3 and Figure 4 In addition, Figure 5 The illustrated embodiment may correspond to Figure 1The local conference device in receives the audio signal (ie, the downlink audio signal) of the remote conference device through the downlink of the second input terminal In2, and sends the audio signal to the remote conference device through the uplink of the second output port Out2. Figure 5 The illustrated embodiment also includes an AEC module, a BF module, a NN_AFC module, a DOA module and an AES module.
[0037] Among them, the AEC module (Acoustic Echo Cancellation) is used to eliminate echoes in multi-channel microphone signals. When the sound emitted by the local speaker is picked up by the local microphone and sent back to the remote end, an echo may be generated. The AEC module dynamically estimates the echo path through an adaptive filter and eliminates the echo emitted by the speaker from the local microphone signal. The AEC module ensures that the sound of the local speaker does not interfere with the remote call, providing a better call experience.
[0038] The DOA (Direction of Arrival) module is used to calculate the angle of arrival of the sound signal. For example, in a local sound reinforcement channel, the DOA module calculates the azimuth of the local sound source (such as the speaker) to provide input for the beamforming (BF) algorithm. The DOA module determines the direction of the sound source by analyzing the time difference and phase difference between different channels of the microphone array.
[0039] For example, by using the signal of the sampling channel 2 (corresponding to the downlink), the remote signal and the local sound source can be accurately distinguished, avoiding misjudgment of the loudspeaker playback signal as the local speaker signal, thereby more accurately tracking the real local speaker. Figure 5 As shown in FIG. 1 , the DOA module performs sound source localization based on the multi-channel microphone signals and the downlink audio signals, rather than directly performing sound source localization based on the multi-channel microphone signals, thereby avoiding the influence of the far-end audio-downlink audio signals that may be contained in the multi-channel microphone signals on the accuracy of sound source localization, thereby ensuring the accuracy of BF.
[0040] In some embodiments, the second input terminal In2 can accurately distinguish the real speaker sound source and the speaker echo signal in a mixed conference scenario. Because in a local sound reinforcement environment, the sound played by the speaker may be picked up by the microphone array, thereby generating a false sound source direction. If it is not processed, the DOA module may mistakenly locate the direction of the echo signal. In order to solve this problem, a reference signal (i.e., a downlink audio signal) is provided through the second input terminal In2 to indicate that the speaker plays the far-end voice, helping the microphone array to determine whether the sound currently received by the microphone array contains echo interference from the speaker. It has at least the following functions: •Prevent DOA from misjudging echo signals: By introducing the In2 signal, the microphone array can effectively distinguish the real sound source from the speaker echo, avoiding the DOA module's incorrect positioning due to echo, thereby improving the accuracy of sound source positioning.
[0041] • Improve positioning accuracy: When In2 indicates the presence of a far-end downlink signal, the microphone array can automatically eliminate interference from the speaker, ensuring that the DOA algorithm always focuses on the actual local sound source, effectively improving the accuracy of beamforming (BF) and the reliability of subsequent signal processing.
[0042] The BF (Beamforming) module is a beamforming algorithm that performs weighted combination of microphone array signals based on the azimuth estimated by DOA to form a "beam" to focus on the sound source in a specific direction. This process can enhance the signal of the target sound source while suppressing noise from other directions, thereby improving the signal quality of local sound reinforcement.
[0043] The NN_AFC (Neural Network Adaptive Feedback Cancellation) module is a feedback cancellation algorithm that combines deep learning technology. Compared with the traditional AFC algorithm, NN_AFC can better capture the complex characteristics of the feedback path through the neural network model and adapt to different environments and device configurations. The model can automatically learn the characteristics of feedback and make adaptive adjustments, thereby providing more efficient feedback cancellation effects in complex scenarios.
[0044] The AES (Acoustic Echo Suppression) module further processes the residual echo from the AEC. Although the AEC has removed most of the echo, there may still be some residual echo. AES further suppresses these residual echoes through advanced filtering and processing. AES ensures that there is almost no echo interference in the far-end call, providing a more natural call experience.
[0045] In some embodiments, the processor performs a first preset processing on the multi-channel microphone signal at least according to the speaker reference signal to obtain a first output signal, including: The processor performs a first echo cancellation process on the multi-channel microphone signal at least according to the loudspeaker reference signal to obtain a first echo cancellation result; wherein the first echo cancellation process can be implemented by using an AEC module; The processor performs sound source localization at least according to the multi-channel microphone signals; The processor performs beamforming processing on the first echo cancellation result at least according to the sound source localization result, and performs second echo cancellation processing on the beamformed signal to obtain a first output signal. The second echo cancellation processing is (NN_AFC, neural network adaptive feedback cancellation) which is a feedback cancellation algorithm combined with deep learning technology. Compared with the traditional AFC algorithm, NN_AFC can better capture the complex characteristics of the feedback path through the neural network model and adapt to different environments and device configurations.
[0046] In some embodiments, the processor performs a second preset processing on the multi-channel microphone signal at least according to the speaker reference signal to obtain a second output signal, including: The processor performs a first echo cancellation process on the multi-channel microphone signal at least according to the loudspeaker reference signal to obtain a first echo cancellation result; wherein the first echo cancellation process can be implemented by using an AEC module; The processor performs sound source localization at least according to the multi-channel microphone signals; The processor performs beamforming processing on the first echo cancellation result at least according to the sound source localization result, and performs echo suppression processing on the beamformed signal to obtain a second output signal.
[0047] In some embodiments, the processor performs beamforming processing on the first echo cancellation result at least according to the sound source localization result, and performs echo suppression processing on the beamformed signal, including: When the downlink audio signal indicates that there is far-end voice activity, performing a first echo suppression process on the beamformed signal; When the downlink audio signal indicates that there is no far-end voice activity, a second echo suppression process is performed on the beamformed signal; wherein the strength of the first echo suppression is greater than the strength of the second echo suppression.
[0048] For example, when there is a voice signal in the far-end downlink (i.e., the downlink audio signal indicates that there is far-end voice activity), the suppression strength of the first echo suppression processing (AES, Acoustic Echo Suppression) is about 30-40dB to ensure that there is no echo leakage. In the sound reinforcement mode, if there is no far-end human voice speaking (i.e., the downlink audio signal indicates that there is no far-end voice activity), the suppression strength of the second echo suppression should be lowered to a maximum suppression strength of no more than 5dB, or completely turned off, so as to ensure the voice quality of the near-end human voice in the uplink channel.
[0049] In this embodiment, through the dual-recall signal input design, the AES module can distinguish the far-end voice played by the speaker from the local sound reinforcement signal in real time. When the far-end voice activity is detected in the recall channel 2 (corresponding to the downlink), the AES module will increase the echo suppression strength and reduce the residual echo; when the recall channel 2 is inactive but the recall channel 1 is active, the system recognizes it as a local sound reinforcement signal, and the AES module reduces the suppression strength to ensure the clarity of the local sound reinforcement voice.
[0050] like Figure 6 FIG. 1 is a schematic diagram of another embodiment of an array microphone device of the present application. Figure 5 The difference between the embodiment in the embodiment is that the array microphone device of the present application is provided with a third output port (Out_DOA_info) for outputting the coordinate information of the sound source. Exemplarily, when the array microphone device includes a processor, the processor is configured with the third output port. Moreover, the processor performs sound source localization at least according to the multi-channel microphone signal to determine the coordinate information of the sound source and outputs it to the external device through the third output port.
[0051] By adding a DOA coordinate output channel in this embodiment, the coordinate information of the current speaker can be output in real time to assist external devices (such as cameras) to automatically and quickly track the speaker in the current meeting, thereby improving the accuracy and efficiency of meeting recording.
[0052] like Figure 7 The schematic diagram of another embodiment of the array microphone device of the present application is shown. In this embodiment, it also includes: before outputting the first output signal, the first output signal is subjected to local sound reinforcement preprocessing, and the local sound reinforcement preprocessing includes at least one of the following: noise suppression (NR, NN_NR), neural network reverberation suppression (NN_DER), automatic gain control (AGC) and frequency equalization processing (EQ); and / or, before outputting the second output signal, the second output signal is subjected to remote signal preprocessing, and the remote signal preprocessing includes at least one of the following: noise suppression (NR, NN_NR), neural network dynamic rebound suppression (NN_DER), automatic gain control (AGC) and frequency equalization processing (EQ).
[0053] This embodiment uses full-link audio signal processing to achieve all-round optimization from noise suppression to automatic gain control. For example, this application integrates a complete audio processing chain, including modules such as noise suppression (NR), dereverberation (NN_DER), equalizer (EQ), and automatic gain control (AGC). This full-link processing system ensures that the audio signal can be optimized at multiple levels throughout the entire process from input to output.
[0054] Effects: Significantly improved audio clarity: Through NR and dereverberation processing, the system can effectively improve audio clarity, especially in noisy or reverberant environments; Smooth volume adjustment: The AGC module automatically adjusts the gain to ensure that the audio output is not too high or too low, providing a more natural listening experience.
[0055] like Figure 8 FIG. 1 is a block diagram of another embodiment of an array microphone device of the present application. In this embodiment, independent signal processing modules are configured for the local sound reinforcement link and the uplink of the remote signal. Figure 7 Compared with the embodiment, Figure 8 Will Figure 7 The AEC in the system is split into the AFC for the local sound reinforcement link and the AEC for the uplink of the remote signal, the BF is split into two independent BFs, and the DOA is split into two independent DOAs.
[0056] It should be noted that Figure 5-Figure 8 The system block diagram of the array microphone device shown shows an array microphone unit with integrated complete mixed scene audio processing capabilities. The overall unit in the dotted box not only supports audio acquisition of multi-channel microphones, but also includes dual-channel reference audio acquisition, DSP signal processing, sound reinforcement channel output, and uplink call link signal output. The entire system block diagram has a high degree of integration and can independently handle complex audio scenarios of remote calls and local sound reinforcement, reducing dependence on external DSP processors.
[0057] The array microphone device system of the present application supports multiple signal transmission formats. External signal transmission can be transmitted using either analog audio signals or digital audio protocols (such as Dante, AES67). If analog audio signals are used, the array microphone unit also needs to add corresponding ADC and DAC modules to complete analog-to-digital conversion to ensure accurate signal processing.
[0058] like Fig. 9 FIG. 4 is a flow chart of another embodiment of the signal processing method of the present application.
[0059] Step 1: Audio Collection Array microphone devices collect local audio signals through microphone arrays. These microphone arrays are usually composed of multiple microphones, and the layout is designed to capture sound source information from different directions. Each microphone will collect different audio signals, which will be transmitted to the subsequent signal processing module. With the help of the microphone array, the system can realize spatial audio perception for subsequent beamforming (BF) and azimuth angle estimation (DOA).
[0060] Step 1.2: Execute the AFC algorithm The Adaptive Feedback Cancellation (AFC) algorithm is used to detect and eliminate feedback in local sound reinforcement systems. This feedback is usually caused by the microphone picking up the sound from the speaker, causing the sound to circulate between the microphone and the speaker, forming a howling sound.
[0061] AFC dynamically adjusts the signal through an adaptive filter to reduce or eliminate the feedback signal so that the system can operate normally under high gain conditions.
[0062] Step 1.3: Calculate the local sound reinforcement channel DOA1 The DOA (Direction of Arrival) algorithm is used to calculate the angle of arrival of sound signals. In the local sound reinforcement channel, the DOA1 module calculates the azimuth of the local sound source (such as the speaker) to provide input for the beamforming (BF) algorithm.
[0063] DOA1 determines the direction of the sound source by analyzing the time and phase differences between different channels of the microphone array.
[0064] Step 1.4: Execute the beamforming (BF) algorithm The beamforming (BF) algorithm performs weighted combination of microphone array signals based on the azimuth angle estimated by DOA1 to form a "beam" to focus on the sound source in a specific direction.
[0065] This process enhances the signal from the target sound source while suppressing noise from other directions, thereby improving the signal quality of local sound reinforcement.
[0066] Step 1.5: Execute NN_AFC algorithm NN_AFC (Neural Network Adaptive Feedback Cancellation) is a feedback cancellation algorithm that combines deep learning technology. Compared with the traditional AFC algorithm, NN_AFC can better capture the complex characteristics of the feedback path through the neural network model and adapt to different environments and device configurations.
[0067] The model can automatically learn the characteristics of feedback and make adaptive adjustments to provide more efficient feedback elimination effects in complex scenarios.
[0068] Step 1.6: Execute the local sound reinforcement processing chain (NR, NN_NR, NN_DER, AGC, EQ) After beamforming, the local sound reinforcement signal needs to go through a series of processing links to ensure that the output audio signal is of good quality and suitable for use by local sound reinforcement equipment.
[0069] NR (Noise Reduction): Suppresses the noise of the signal collected by the microphone to reduce the interference of environmental noise.
[0070] NN_NR (Neural Network Noise Suppression): A noise suppression algorithm based on deep learning that can better separate speech signals from complex noise.
[0071] NN_DER (Neural Network Reverberation Suppression): Dynamically suppresses reverberation tails through a neural network model to ensure clear sound reinforcement signals.
[0072] AGC (Automatic Gain Control): Automatically adjusts the gain of the audio signal to ensure volume balance and prevent the signal from being too loud or too soft.
[0073] EQ (Equalization): Frequency equalization of audio signals, enhancing certain frequency bands or weakening other frequency bands to ensure the sound quality of the sound reinforcement signal.
[0074] Step 1.7: Local sound reinforcement output The local sound reinforcement signal that has passed through the entire processing chain is finally transmitted to the local sound reinforcement loudspeaker for playback.
[0075] The signal should have high-quality sound clarity, low noise and no howling, ensuring that users can hear clear and natural sound in the local environment.
[0076] Step 2.2: Execute the AEC algorithm The AEC (Acoustic Echo Cancellation) algorithm is used to eliminate echoes in far-end calls. When the sound from the local speaker is picked up by the local microphone and sent back to the far end, an echo may be generated. AEC dynamically estimates the echo path through an adaptive filter and eliminates the echo from the speaker from the local microphone signal.
[0077] AEC ensures that the sound from the local speaker does not interfere with the far-end call, providing a better call experience.
[0078] Step 2.3: Calculate the DOA2 of the remote call channel DOA2 (Far-end Azimuth Estimation) is used to calculate the azimuth of the far-end call source. By analyzing the angle of arrival of the far-end signal, the system can help the beamforming algorithm to enhance the sound in the direction of the far-end call.
[0079] DOA2 provides important input for beamforming of far-end calls, ensuring that the far-end voice is transmitted clearly.
[0080] Step 2.4: Execute the beamforming (BF) algorithm Based on the far-end azimuth estimated by DOA2, the beamforming (BF) algorithm performs direction-enhanced far-end call signals. By focusing on the far-end sound source, the system can suppress other interfering signals and ensure that the far-end voice signal is clearly audible.
[0081] This beamforming technology can significantly improve the signal-to-noise ratio (SNR) of far-end calls and ensure the quality of voice transmission.
[0082] Step 2.5: Execute AES algorithm and post-process residual echo The AES (Acoustic Echo Suppression) algorithm further processes the residual echo of the AEC. Although the AEC has removed most of the echo, there may still be some residual echo. AES further suppresses these residual echoes through advanced filtering and processing.
[0083] AES ensures that there is virtually no echo interference on the far end, providing a more natural calling experience.
[0084] Step 2.6: Execute remote call processing link (NR, NN_NR, NN_DER, AGC, EQ) After beamforming and AES processing, the far-end call signal still needs to pass through multiple processing modules to ensure the voice quality of the far-end output.
[0085] NR (Noise Reduction): Reduces background noise in the far-end call signal.
[0086] NN_NR (Neural Network Noise Suppression): Uses deep learning algorithms to separate speech signals from complex noise, further improving speech clarity.
[0087] NN_DER (Neural Network Dynamic Echo Suppression): Dynamically suppresses reverberation tails through a neural network model to ensure that the call signal sent to the far end is clear and has no reverberation tails.
[0088] AGC (Automatic Gain Control): Automatically adjusts the gain of the far-end call signal to ensure balanced voice volume.
[0089] EQ (Equalization): Performs frequency equalization on the remote call signal to ensure that the sound quality of the output signal is suitable for the remote device.
[0090] Step 2.7: Remote signal output The far-end signal after the processing link is finally sent to the far-end call device (such as a phone, video conferencing system). The signal has been processed by echo cancellation, beamforming, noise suppression, etc. to ensure that the voice signal is clear without echo and noise interference. The far-end user will receive high-quality call audio, bringing a better call experience.
[0091] In some embodiments, the embodiments of the present application adopt a dual-channel input and dual-return design, which can accurately distinguish the far-end signal from the local sound reinforcement signal and improve the signal processing accuracy. Specifically, the present application significantly improves the flexibility and accuracy of audio signal processing through a dual-channel input design (In1 and In2). The In1 channel is specifically used to receive the reference signal (SPK_ref) of the speaker, thereby providing accurate basic data for echo cancellation (AEC). The In2 channel is used to receive the far-end downlink signal to ensure that the system can clearly distinguish the local sound reinforcement signal from the far-end call signal.
[0092] In addition, the audio processing effect is further optimized by designing dual-recall signal inputs. Recall channel 1 (corresponding to the link between the first output terminal and the first input terminal, and the corresponding signal includes local sound reinforcement and far-end downlink signals) provides the system with a complete audio reference of the actual playback of the speakers, while recall channel 2 (including only far-end downlink signals) assists the system in more accurately distinguishing far-end signals from local sound reinforcement signals.
[0093] The effect: More accurate echo cancellation: The dual echo design ensures that the system can better identify speaker echo, significantly improving the AEC effect, especially in complex far-end call scenarios.
[0094] Independent processing of calls and sound reinforcement: The local sound reinforcement and remote call signals are processed separately without interfering with each other, effectively improving the sound quality and voice clarity, especially in mixed scenarios.
[0095] Optimization of local sound reinforcement quality: Through In2 and the collected signal, the AES module can effectively distinguish the remote signal from the local sound reinforcement signal and optimize the local sound reinforcement effect.
[0096] In some embodiments, the embodiments of the present application adopt a dual signal processing link design to independently process local sound reinforcement and remote calls, thereby improving system integration and audio quality. Specifically, the present application designs two independent signal processing links to process the uplink signal (Out2) of the remote call and the local sound reinforcement signal (Out1) respectively. This independent signal processing architecture allows the system to process remote calls and local sound reinforcement at the same time, avoiding the limitation of traditional technologies that can only process a single signal.
[0097] The effect: Higher system integration: No additional external audio processor is required to support hybrid scenarios, which greatly simplifies device design, reduces costs, and improves system integration.
[0098] Independent audio signal optimization: local sound reinforcement and remote calls do not interfere with each other, ensuring clear reception of remote voices, while local sound reinforcement signals can maintain high gain and good sound quality.
[0099] In some embodiments, the embodiments of the present application use a DOA (direction of arrival) algorithm to accurately locate the sound source and improve the speech processing effect. Specifically, the DOA (Direction of Arrival) algorithm of the present application uses an array microphone and beamforming (BF) technology to efficiently and accurately detect the direction of the sound source and locate the speaker. Compared with the inaccurate sound source positioning in traditional technologies, the system achieves higher-precision sound source positioning by combining a multi-microphone array and advanced algorithms.
[0100] The effect: Improved beamforming efficiency: After the direction of the sound source is accurately located, beamforming technology can more effectively focus on the direction of the target sound and reduce interference from ambient noise and non-target sounds.
[0101] Optimized call quality: In noisy environments, the system can capture the speaker's voice signal more clearly and improve the quality of far-end calls.
[0102] In some embodiments, the embodiments of the present application use the AES module to perform intelligent echo suppression based on dual-recall input, dynamically adjust the echo suppression, and improve the remote call and local sound reinforcement experience. Specifically, through the dual-recall signal input design, the AES module can distinguish between the remote voice and the local sound reinforcement signal played by the speaker in real time. When the remote voice activity is detected in the recall channel 2, the AES module will enhance the echo suppression strength and reduce the residual echo; when the recall channel 2 is inactive but the recall channel 1 is active, the system recognizes it as a local sound reinforcement signal, and the AES module reduces the suppression strength to ensure the clarity of the local sound reinforcement voice.
[0103] The effect: Intelligent echo suppression: The AES module can dynamically adjust the intensity of echo suppression according to the presence or absence of remote signals to eliminate echo interference to the greatest extent.
[0104] Optimized balance between sound reinforcement and calls: The system can intelligently adjust between echo suppression and local sound reinforcement to ensure clear transmission of far-end voice without affecting local sound reinforcement.
[0105] In some embodiments, the embodiments of the present application use NN_AFC and NN_NR modules to achieve adaptive echo cancellation and noise suppression. Specifically, the NN_AFC and NN_NR modules of the present application are based on neural networks (NN) and can adaptively process complex echoes and noise. These modules can recognize and learn dynamically changing noise and echo patterns through deep learning technology, and have stronger processing capabilities than traditional algorithms.
[0106] The effect: Stronger noise adaptability: The NN_NR module can suppress complex environmental noise, such as wind noise, traffic noise, etc. It is more adaptable and suitable for a variety of environments.
[0107] Intelligent echo cancellation: The NN_AFC module can learn the acoustic environment of the room and adaptively adjust the echo cancellation parameters to reduce residual echo and improve voice quality.
[0108] In a second aspect, the present application further provides an array microphone device, wherein the array microphone device comprises a plurality of microphones and a processor, the processor comprises a first input terminal, a first output terminal, and a second output terminal, wherein the first output terminal outputs a signal to a local speaker, the second output terminal outputs a signal to a remote end, and the plurality of microphones are used to collect multi-channel microphone signals; The processor is configured as follows: The processor collects a speaker reference signal through a communication path between the first output terminal and the first input terminal; The processor performs a first preset processing on the multi-channel microphone signal at least according to the speaker reference signal to obtain a first output signal, and outputs the first output signal to the local speaker through the first output terminal; The processor performs a second preset processing on the multi-channel microphone signals at least according to the loudspeaker reference signal to obtain a second output signal, and outputs the second output signal to the remote end through the second output terminal.
[0109] In some embodiments, performing a first preset processing on the multi-channel microphone signal at least according to the speaker reference signal to obtain a first output signal includes: Performing a first echo cancellation process on the multi-channel microphone signals at least according to the loudspeaker reference signal to obtain a first echo cancellation result; Performing sound source localization at least according to the multi-channel microphone signals; The first echo cancellation result is subjected to beamforming processing at least according to the sound source localization result, and a second echo cancellation processing is performed on the beamformed signal to obtain a first output signal.
[0110] In some embodiments, performing a second preset processing on the multi-channel microphone signal at least according to the speaker reference signal to obtain a second output signal includes: Performing a first echo cancellation process on the multi-channel microphone signals at least according to the loudspeaker reference signal to obtain a first echo cancellation result; Performing sound source localization at least according to the multi-channel microphone signals; At least according to the sound source localization result, beamforming processing is performed on the first echo cancellation result, and echo suppression processing is performed on the beamformed signal to obtain a second output signal.
[0111] In some embodiments, the processor further includes a second input terminal, and the processor obtains a downlink audio signal from the remote end through the second input terminal; and performing sound source localization at least according to the multi-channel microphone signal includes: Sound source localization is performed according to the multi-channel microphone signals and the downlink audio signal.
[0112] In some embodiments, performing beamforming processing on the first echo cancellation result at least according to the sound source localization result, and performing echo suppression processing on the beamformed signal, includes: When the downlink audio signal indicates that there is far-end voice activity, performing a first echo suppression process on the beamformed signal; When the downlink audio signal indicates that there is no far-end voice activity, a second echo suppression process is performed on the beamformed signal; wherein the strength of the first echo suppression is greater than the strength of the second echo suppression.
[0113] In some embodiments, the processor further includes a third output terminal, and the processor is further configured to: perform sound source localization at least according to the multi-channel microphone signal to determine the coordinate information of the sound source and output it to an external device through the third output terminal.
[0114] In some embodiments, the processor is further configured to: Before outputting the first output signal, the first output signal is subjected to local sound reinforcement preprocessing, wherein the local sound reinforcement preprocessing includes at least one of the following: noise suppression, neural network reverberation suppression, automatic gain control and frequency equalization processing; and / or, Before outputting the second output signal, the second output signal is subjected to remote signal preprocessing, wherein the remote signal preprocessing includes at least one of the following: noise suppression, neural network dynamic rebound suppression, automatic gain control, and frequency equalization processing.
[0115] In a third aspect, the present application further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in any embodiment of the present application.
[0116] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program / instruction stored thereon, wherein the computer program / instruction, when executed by a processor, implements the steps of the method of any embodiment of the present application.
[0117] In a fifth aspect, the present application also provides a computer program product, including a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the steps of the method described in any embodiment of the present application are implemented.
[0118] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of actions combined, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0119] Fig.10 is a schematic diagram of the hardware structure of a computer device for executing a signal processing method provided by another embodiment of the present application, such as Fig.10 As shown, the device includes: One or more processors 1010 and memory 1020, Fig.10 A processor 1010 is taken as an example.
[0120] The apparatus for executing the signal processing method may further include: an input device 1030 and an output device 1040 .
[0121] The processor 1010, the memory 1020, the input device 1030 and the output device 1040 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.
[0122] The memory 1020 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the signal processing method in the embodiment of the present application. The processor 1010 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 1020, that is, the signal processing method in the above method embodiment is implemented.
[0123] The memory 1020 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of the signal processing device, etc. In addition, the memory 1020 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1020 may optionally include a memory remotely arranged relative to the processor 1010, and these remote memories may be connected to the signal processing device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0124] The input device 1030 may receive input digital or character information and generate signals related to user settings and function control of the signal processing device. The output device 1040 may include a display device such as a display screen.
[0125] The one or more modules are stored in the memory 1020, and when executed by the one or more processors 1010, perform the signal processing method in any of the above method embodiments.
[0126] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.
[0127] The computer device of the embodiment of the present application exists in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication functions and their main purpose is to provide voice and data communications. These terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.
[0128] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, such as iPad.
[0129] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players (such as iPods), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.
[0130] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.
[0131] (5) Other electronic devices with data interaction functions.
[0132] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0133] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.
[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A signal processing method for an array microphone device, characterized in that: The array microphone device comprises a plurality of microphones and a processor, wherein the processor comprises a first input terminal, a first output terminal, and a second output terminal, wherein the first output terminal outputs a signal to a local speaker, and the second output terminal outputs a signal to a remote end, and the plurality of microphones are used to collect multi-channel microphone signals. The method comprises: The processor collects a speaker reference signal through a communication path between the first output terminal and the first input terminal; The processor performs a first preset processing on the multi-channel microphone signal at least according to the speaker reference signal to obtain a first output signal, and outputs the first output signal to the local speaker through the first output terminal; The processor performs a second preset processing on the multi-channel microphone signals at least according to the loudspeaker reference signal to obtain a second output signal, and outputs the second output signal to the remote end through the second output terminal.
2. The method according to claim 1, characterized in that The processor performs a first preset processing on the multi-channel microphone signal at least according to the loudspeaker reference signal to obtain a first output signal, including: The processor performs a first echo cancellation process on the multi-channel microphone signals at least according to the speaker reference signal to obtain a first echo cancellation result; The processor performs sound source localization at least according to the multi-channel microphone signals; The processor performs beamforming processing on the first echo cancellation result at least according to the sound source localization result, and performs second echo cancellation processing on the beamformed signal to obtain a first output signal.
3. The method according to claim 1, characterized in that The processor performs a second preset processing on the multi-channel microphone signal at least according to the loudspeaker reference signal to obtain a second output signal, including: The processor performs a first echo cancellation process on the multi-channel microphone signals at least according to the speaker reference signal to obtain a first echo cancellation result; The processor performs sound source localization at least according to the multi-channel microphone signals; The processor performs beamforming processing on the first echo cancellation result at least according to the sound source localization result, and performs echo suppression processing on the beamformed signal to obtain a second output signal.
4. The method according to claim 2 or 3, characterized in that: The processor further includes a second input terminal, and the processor obtains a downlink audio signal from the remote end through the second input terminal; The processor performs sound source localization at least according to the multi-channel microphone signals, including: The processor performs sound source localization according to the multi-channel microphone signals and the downlink audio signal.
5. The method according to claim 4, characterized in that The processor performs beamforming processing on the first echo cancellation result at least according to the sound source localization result, and performs echo suppression processing on the beamformed signal, including: When the downlink audio signal indicates that there is far-end voice activity, performing a first echo suppression process on the beamformed signal; When the downlink audio signal indicates that there is no far-end voice activity, a second echo suppression process is performed on the beamformed signal; wherein the strength of the first echo suppression is greater than the strength of the second echo suppression.
6. The method according to any one of claims 1 to 3, characterized in that The processor further includes a third output terminal, and the method further includes: the processor performs sound source localization at least according to the multi-channel microphone signals to determine the coordinate information of the sound source and outputs the coordinate information to an external device through the third output terminal.
7. The method according to any one of claims 1 to 3, characterized in that Also includes: Before outputting the first output signal, performing local sound reinforcement preprocessing on the first output signal, wherein the local sound reinforcement preprocessing includes at least one of the following: noise suppression, neural network reverberation suppression, automatic gain control, and frequency equalization processing; and / or, Before outputting the second output signal, the second output signal is subjected to remote signal preprocessing, wherein the remote signal preprocessing includes at least one of the following: noise suppression, neural network dynamic rebound suppression, automatic gain control, and frequency equalization processing.
8. An array microphone device, characterized in that: The array microphone device comprises a plurality of microphones and a processor, wherein the processor comprises a first input terminal, a first output terminal and a second output terminal, wherein the first output terminal outputs a signal to a local speaker, and the second output terminal outputs a signal to a remote end, and the plurality of microphones are used to collect multi-channel microphone signals; The processor is configured as follows: The processor collects a speaker reference signal through a communication path between the first output terminal and the first input terminal; The processor performs a first preset processing on the multi-channel microphone signal at least according to the speaker reference signal to obtain a first output signal, and outputs the first output signal to the local speaker through the first output terminal; The processor performs a second preset processing on the multi-channel microphone signals at least according to the loudspeaker reference signal to obtain a second output signal, and outputs the second output signal to the remote end through the second output terminal.
9. The array microphone device according to claim 8, characterized in that: Performing a first preset processing on the multi-channel microphone signals at least according to the loudspeaker reference signal to obtain a first output signal includes: Performing a first echo cancellation process on the multi-channel microphone signals at least according to the loudspeaker reference signal to obtain a first echo cancellation result; Performing sound source localization at least according to the multi-channel microphone signals; The first echo cancellation result is subjected to beamforming processing at least according to the sound source localization result, and a second echo cancellation processing is performed on the beamformed signal to obtain a first output signal.
10. The array microphone device according to claim 8, characterized in that: Performing a second preset processing on the multi-channel microphone signal at least according to the loudspeaker reference signal to obtain a second output signal includes: Performing a first echo cancellation process on the multi-channel microphone signals at least according to the loudspeaker reference signal to obtain a first echo cancellation result; Performing sound source localization at least according to the multi-channel microphone signals; At least according to the sound source localization result, beamforming processing is performed on the first echo cancellation result, and echo suppression processing is performed on the beamformed signal to obtain a second output signal.
11. The array microphone device according to claim 9 or 10, characterized in that: The processor further includes a second input terminal, and the processor obtains a downlink audio signal from the remote end through the second input terminal; and performing sound source localization at least according to the multi-channel microphone signal includes: Sound source localization is performed according to the multi-channel microphone signals and the downlink audio signal.
12. The array microphone device according to claim 11, characterized in that: The step of performing beamforming processing on the first echo cancellation result at least according to the sound source localization result, and performing echo suppression processing on the beamformed signal, includes: When the downlink audio signal indicates that there is far-end voice activity, performing a first echo suppression process on the beamformed signal; When the downlink audio signal indicates that there is no far-end voice activity, a second echo suppression process is performed on the beamformed signal; wherein the strength of the first echo suppression is greater than the strength of the second echo suppression.
13. The array microphone device according to any one of claims 8 to 10, characterized in that: The processor further includes a third output terminal, and the processor is further configured to: perform sound source localization at least according to the multi-channel microphone signals to determine the coordinate information of the sound source and output it to an external device through the third output terminal.
14. The array microphone device according to any one of claims 8 to 10, characterized in that: The processor is further configured to: Before outputting the first output signal, performing local sound reinforcement preprocessing on the first output signal, wherein the local sound reinforcement preprocessing includes at least one of the following: noise suppression, neural network reverberation suppression, automatic gain control, and frequency equalization processing; and / or, Before outputting the second output signal, the second output signal is subjected to remote signal preprocessing, wherein the remote signal preprocessing includes at least one of the following: noise suppression, neural network dynamic rebound suppression, automatic gain control, and frequency equalization processing.
15. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.
16. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
17. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.