Stereo echo cancellation method and terminal equipment

Abstract

The embodiment of the invention discloses a stereo echo cancellation method and terminal equipment. The method comprises the following steps: acquiring a plurality of first microphone signals and a plurality of loudspeaker signals; respectively carrying out time-frequency conversion, feature extraction, sub-band decomposition and feature fusion on the plurality of first microphone signals and the plurality of loudspeaker signals to obtain low sub-band combined features and high sub-band combined features; inputting the low sub-band combined features and the high sub-band combined features into a neural network model, and outputting a plurality of amplitude spectrum gain factors; a plurality of second microphone signals is generated based on the plurality of amplitude spectrum gain factors and the plurality of first microphone signals. According to the method and the device, the high-low sub-band combined features are obtained through feature fusion of the microphone signals and the loudspeaker signals, the echo-removed microphone signals are generated through the gain factors output by the neural network model, processing such as inter-channel decorrelation is not needed, the complexity of an algorithm is reduced, and the tone quality is improved.

Description

Technical Field

[0001] This application relates to the field of echo cancellation technology, and in particular to a stereo echo cancellation method and terminal equipment. Background Technology

[0002] With the rapid development of communication systems such as video telephony, the number of speakers and microphones used in these systems has increased in order to provide users with an immersive audio experience. As the number of speakers and microphones increases, traditional mono echo cancellation algorithms are no longer sufficient to meet the demands.

[0003] Compared to single-channel echo cancellation algorithms, stereo echo cancellation suffers from strong correlations between different channels, resulting in a singular correlation matrix and thus no unique solution. Furthermore, stereo echo cancellation significantly increases the probability of echo path changes. If a traditional adaptive filter is used, it must track not only abrupt changes in the near-end acoustic path but also variations in the far-end acoustic path, leading to slow convergence. Even with adaptive filtering, a cascaded nonlinear suppression module is still needed to suppress residual echoes, resulting in a degraded speech quality. Summary of the Invention

[0004] Some embodiments of this application provide a stereo echo cancellation method and terminal device, which fuses multiple microphone signals and speaker signals through feature fusion to obtain high and low subband merging features, and generates multiple echo-removed microphone signals, i.e., stereo, through the gain factor output by the neural network model. This eliminates the need for inter-channel decorrelation processing, reduces algorithm complexity, and improves sound quality.

[0005] Firstly, some embodiments of this application provide a stereo echo cancellation method, including: When playing audio through multiple speakers, if multiple microphones are detected to be activated, multiple first microphone signals and multiple speaker signals are acquired. The first microphone signal is the audio signal collected by the corresponding microphone, and the speaker signal is the original audio signal played by the corresponding speaker. Multiple first microphone signals are converted into multiple first spectra by time-frequency conversion, and multiple speaker signals are converted into multiple second spectra by time-frequency conversion. Multiple first spectra are subjected to feature extraction to obtain multiple first amplitude spectra, and multiple second spectra are subjected to feature extraction to obtain multiple second amplitude spectra; wherein, the first amplitude spectra are used to characterize the energy distribution of the corresponding first microphone signal, and the second amplitude spectra are used to characterize the energy distribution of the corresponding loudspeaker signal; Subband decomposition is performed on multiple first amplitude spectra to obtain multiple low-frequency subbands and multiple high-frequency subbands corresponding to each first amplitude spectrum; and subband decomposition is performed on multiple second amplitude spectra to obtain multiple low-frequency subbands and multiple high-frequency subbands corresponding to each second amplitude spectrum. Feature fusion is performed on all low-frequency subbands to obtain low-subband merging features, and feature fusion is performed on all high-frequency subbands to obtain high-subband merging features; The low subband merging features and high subband merging features are input into the neural network model to output multiple amplitude spectrum gain factors through the neural network model. Different amplitude spectrum gain factors correspond to different first microphone signals. Multiple amplitude spectrum gain factors are multiplied by their corresponding first spectra to obtain multiple gain spectra; Multiple gain spectra are converted from frequency to time to obtain multiple second microphone signals, which are microphone signals with echo and noise removed.

[0006] The above technical solution has the following advantages or beneficial effects: it fuses multiple microphone signals and speaker signals through feature fusion to obtain high and low subband merging features, and generates multiple echo-removed microphone signals, i.e. stereo, through the gain factor output by the neural network model. It does not require processing such as inter-channel decorrelation, reducing the complexity of the algorithm and improving the sound quality.

[0007] In some embodiments, the controller performs time-frequency conversion on multiple first microphone signals to obtain multiple first spectra, and performs time-frequency conversion on multiple speaker signals to obtain multiple second spectra, and is further configured to: Multiple first microphone signals are subjected to frame-by-frame windowing processing to obtain multiple first windowed data, and multiple speaker signals are subjected to frame-by-frame windowing processing to obtain second windowed data; Multiple first-windowed data are subjected to Fast Fourier Transform (FFT) to obtain multiple first spectra, and multiple second-windowed data are subjected to FFT to obtain multiple second spectra.

[0008] The above technical solution has the following advantages or beneficial effects: framing and windowing can reduce spectral leakage and edge effects caused by framing, and converting to the spectrum is beneficial for echo cancellation processing.

[0009] In some embodiments, the controller performs feature fusion on all low-frequency subbands to obtain low-subband merging features, which is further configured to: All low-frequency subbands of the same low-frequency band are input into the first convolution module to output the first time-frequency joint feature. The first time-frequency joint feature is used to characterize the energy distribution of the audio signal in the joint domain of time and frequency when the number of channels is expanded to a first number. The first time-frequency joint feature is input into the second convolution module to output the first frequency compression feature through the second convolution module. The first frequency compression feature is the energy distribution of the audio signal in the joint domain of time and frequency after frequency compression. The first frequency compression feature is input into the third convolution module to output the first long-time and cross-frequency features through the third convolution module. The first long-time and cross-frequency features are used to characterize the energy distribution of the audio signal in the joint time and frequency domain when the number of channels is expanded to a second number, where the second number is greater than the first number. The first long-term and cross-frequency features are input into the fourth convolution module to output the first channel interaction features through the fourth convolution module. The first channel interaction features are used to characterize the energy distribution of the audio signal in the joint time and frequency domain after the inter-channel interaction features. The first channel interaction feature is input into the fifth convolution module to output the first final joint feature. The first final joint feature is used to characterize the energy distribution of the audio signal in the joint domain of time and frequency when the number of channels is compressed to the first number. The first time-frequency joint feature, the first long-time and cross-frequency feature, and the first final joint feature are fused to obtain the first fused feature corresponding to the low-frequency band. The first fusion features of multiple low-frequency bands are merged to obtain the low-subband merging features.

[0010] The above technical solution has the following advantages or beneficial effects: as the number of convolutional layers increases, the resolution becomes lower and lower, but at the same time the receptive field becomes larger and larger. By fusing the outputs of multiple convolutional modules, features with different resolutions and receptive fields can be fused, taking into account both resolution and receptive field to obtain more and more accurate features.

[0011] In some embodiments, the controller performs feature fusion on all high-frequency subbands to obtain high-frequency subband merging features, which is further configured to: All high-frequency subbands of the same high-frequency band are input into the first convolution module to output the second time-frequency joint feature through the first convolution module. The second time-frequency joint feature is used to characterize the energy distribution of the audio signal in the joint domain of time and frequency when the number of channels is expanded to the first number. The second time-frequency joint feature is input into the second convolution module to output the second frequency compression feature. The second frequency compression feature is the energy distribution of the audio signal in the joint domain of time and frequency after frequency compression. The second frequency compression feature is input into the third convolution module to output the second long-time and cross-frequency features through the third convolution module. The second long-time and cross-frequency features are used to characterize the energy distribution of the audio signal in the joint time and frequency domain when the number of channels is expanded to a second number. The second long-time and cross-frequency features are input into the fourth convolution module to output the second channel interaction features. The second channel interaction features are used to characterize the energy distribution of the audio signal in the joint time and frequency domain after the inter-channel interaction features. The second channel interaction feature is input into the fifth convolution module to output the second final joint feature through the fifth convolution module. The second final joint feature is used to characterize the energy distribution of the audio signal in the joint domain of time and frequency when the number of channels is compressed to the first number. The second time-frequency joint feature, the second long-time and cross-frequency feature, and the second final joint feature are fused to obtain the second fused feature corresponding to the high-frequency band. By merging the second fusion features of multiple high-frequency bands, the high sub-band merging feature is obtained.

[0012] The above technical solution has the following advantages or beneficial effects: as the number of convolutional layers increases, the resolution becomes lower and lower, but at the same time the receptive field becomes larger and larger. By fusing the outputs of multiple convolutional modules, features with different resolutions and receptive fields can be fused, taking into account both resolution and receptive field to obtain more and more accurate features.

[0013] In some embodiments, the neural network model includes a high-low subband fusion module, a fully connected network, and an activation function layer. The controller executes inputting low-subband merging features and high-subband merging features into the neural network model to output multiple amplitude spectrum gain factors through the neural network model, and is further configured to: The low subband merging feature and the high subband merging feature are input into the high-low subband fusion module to output the third fusion feature. The third fusion feature is input into a fully connected network to output a frequency domain mask for multiple channels. The frequency domain mask is used to characterize the degree of preservation or suppression of signal energy at different frequencies in the frequency domain. By activating the frequency domain mask using an activation function layer, multiple amplitude spectrum gain factors are obtained.

[0014] The above technical solution has the following advantages or beneficial effects: the features that combine low subband merging features and high subband merging features can have both local details and global context, which can significantly improve the accuracy and robustness of mask estimation.

[0015] In some embodiments, the neural network model further includes a first time-frequency network and a dimensionality reduction network. The controller executes inputting low-subband merging features and high-subband merging features into a high-low subband fusion module to output a third fused feature through the high-low subband fusion module, and is further configured to: The low-subband merging features and high-subband merging features are respectively input into the first time-frequency network to enhance the low-subband merging features and high-subband merging features through the first time-frequency network; The enhanced low-subband merging features and high-subband merging features are input into the dimensionality reduction network, so that the dimensionality reduction network outputs low-subband merging features and high-subband merging features with reduced dimensions. The reduced-dimensional low-subband merging features and high-subband merging features are input into the high-low subband fusion module to output a third fused feature.

[0016] The above technical solutions have the following advantages or beneficial effects: Time-frequency networks can further enhance features by modeling long-term temporal dependencies and local features of speech, thereby further improving model stability and generalization ability. Dimensionality reduction networks mainly reduce model complexity.

[0017] In some embodiments, the neural network model further includes a second time-frequency network, and the controller executes inputting a third fused feature into a fully connected network to output a frequency domain mask with multiple channels through the fully connected network, which is further configured to: The third fusion feature is input into the second time-frequency network to enhance the third fusion feature through the second time-frequency network; The enhanced third fusion feature is input into a fully connected network to output a frequency domain mask for multiple channels.

[0018] The above technical solution has the following advantages or beneficial effects: time-frequency networks can further enhance features and improve model stability and generalization ability.

[0019] In some embodiments, the controller performs the acquisition of multiple first microphone signals and multiple speaker signals, and is further configured to: The input audio signal is read frame by frame from the buffer. The input audio signal is the signal stored in the buffer as an interleaved combination of the first microphone signal and the speaker signal. The input audio signal is deinterleaved to obtain multiple first microphone signals and multiple speaker signals.

[0020] The above technical solution has the following advantages or beneficial effects: interleaving storage can reduce memory fragmentation and copying frequency, improve input / output efficiency, and reduce system latency. The multi-channel signals obtained after deinterleaving have precise time synchronization, which facilitates accurate echo cancellation.

[0021] In some embodiments, after frequency-time conversion of multiple gain spectra to obtain multiple second microphone signals, the controller is further configured to: Multiple second microphone signals and multiple speaker signals are interleaved to obtain an output signal, which is used for speech recognition.

[0022] The above technical solution has the following advantages or beneficial effects: It arranges audio samples from multiple independent channels in a time-interleaved manner, preparing multi-channel interleaved output data conforming to a standard format for downstream modules, such as speech recognition, audio drivers, encoders, or transport layers. Using the speaker signal as a reference can prevent the device's own playback from falsely waking up the device's voice service.

[0023] Secondly, in some embodiments of this application, a method is provided where, when playing audio through multiple speakers, if multiple microphones are detected to be activated, multiple first microphone signals and multiple speaker signals are acquired, wherein the first microphone signals are audio signals collected by the corresponding microphones, and the speaker signals are the original audio signals played by the corresponding speakers. Multiple first microphone signals are converted into multiple first spectra by time-frequency conversion, and multiple speaker signals are converted into multiple second spectra by time-frequency conversion. Multiple first spectra are subjected to feature extraction to obtain multiple first amplitude spectra, and multiple second spectra are subjected to feature extraction to obtain multiple second amplitude spectra; wherein, the first amplitude spectra are used to characterize the energy distribution of the corresponding first microphone signal, and the second amplitude spectra are used to characterize the energy distribution of the corresponding loudspeaker signal; Subband decomposition is performed on multiple first amplitude spectra to obtain multiple low-frequency subbands and multiple high-frequency subbands corresponding to each first amplitude spectrum; and subband decomposition is performed on multiple second amplitude spectra to obtain multiple low-frequency subbands and multiple high-frequency subbands corresponding to each second amplitude spectrum. Feature fusion is performed on all low-frequency subbands to obtain low-subband merging features, and feature fusion is performed on all high-frequency subbands to obtain high-subband merging features; The low subband merging features and high subband merging features are input into the neural network model to output multiple amplitude spectrum gain factors through the neural network model. Different amplitude spectrum gain factors correspond to different first microphone signals. Multiple amplitude spectrum gain factors are multiplied by their corresponding first spectra to obtain multiple gain spectra; Multiple gain spectra are converted from frequency to time to obtain multiple second microphone signals, which are microphone signals with echo and noise removed.

[0024] The above technical solution has the following advantages or beneficial effects: it fuses multiple microphone signals and speaker signals through feature fusion to obtain high and low subband merging features, and generates multiple echo-removed microphone signals, i.e. stereo, through the gain factor output by the neural network model. It does not require processing such as inter-channel decorrelation, reducing the complexity of the algorithm and improving the sound quality.

[0025] In this embodiment of the application, when playing audio through multiple speakers, if multiple microphones are detected to be activated, multiple first microphone signals and multiple speaker signals are acquired. The first microphone signal is the audio signal collected by the corresponding microphone, and the speaker signal is the original audio signal played by the corresponding speaker. Multiple first microphone signals are converted to time-frequency signals to obtain multiple first spectra, and multiple speaker signals are converted to time-frequency signals to obtain multiple second spectra. Feature extraction is performed on the multiple first spectra to obtain multiple first amplitude spectra, and feature extraction is performed on the multiple second spectra to obtain multiple second amplitude spectra. Subband decomposition is performed on the multiple first amplitude spectra to obtain multiple low-frequency subbands and multiple high-frequency subbands corresponding to each first amplitude spectrum, and subband decomposition is performed on the multiple second amplitude spectra to obtain multiple low-frequency subbands and multiple high-frequency subbands corresponding to each second amplitude spectrum. Feature fusion is performed on all low-frequency subbands to obtain low-subband merging features, and feature fusion is performed on all high-frequency subbands to obtain high-subband merging features. The low-subband merging features and high-subband merging features are input into a neural network model to output multiple amplitude spectrum gain factors. The multiple amplitude spectrum gain factors are multiplied by their corresponding first spectra to obtain multiple gain spectra. Frequency-time conversion is performed on the multiple gain spectra to obtain multiple second microphone signals, where the second microphone signals are microphone signals with echo and noise removed. This application embodiment can fuse multiple microphone signals and speaker signals through feature fusion to obtain high and low subband merging features, and generate multiple echo-removed microphone signals, i.e. stereo, through the gain factor output by the neural network model. This eliminates the need for inter-channel decorrelation processing, reduces algorithm complexity, and improves sound quality. Attached Figure Description

[0026] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application; Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application; Figure 3 This is a schematic diagram of the software configuration of a display device provided in some embodiments of this application; Figure 4 A flowchart illustrating a stereo echo cancellation method provided in some embodiments of this application; Figure 5 A schematic diagram of a sub-band feature fusion module structure provided in some embodiments of this application; Figure 6 A schematic diagram illustrating a sub-band feature fusion method provided in some embodiments of this application; Figure 7 A schematic diagram of a neural network model provided for some embodiments of this application; Figure 8 A schematic diagram of a time-frequency network structure provided in some embodiments of this application; Figure 9 A flowchart illustrating the algorithm of a stereo echo cancellation method provided in some embodiments of this application; Figure 10 Timing diagram of a stereo echo cancellation method provided for some embodiments of this application. Detailed Implementation

[0027] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0028] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0029] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0030] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0031] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0032] In this embodiment, the terminal device includes devices with sound acquisition and audio playback functions, such as display devices 200, smart speakers, and wearable devices. Display devices 200 generally refer to devices with image display and data processing capabilities. For example, display devices 200 include, but are not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, and "best friend machines." A "best friend machine" is a smart terminal device integrating a high-definition camera, microphone, speaker, and a movable / rotatable large screen.

[0033] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1 As shown, a user can operate the display device 200 via touch operation, a mobile terminal 300, and a control device 100. The control device 100 receives user input commands and converts them into control commands that the display device 200 can recognize and respond to. For example, the control device 100 can be a remote control, a stylus, a gamepad, etc.

[0034] The mobile terminal 300 can function as a control device for human-computer interaction between the user and the display device 200. It can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can have software applications installed on it and communicate with the display device 200 via network communication protocols to achieve one-to-one control and data communication. Furthermore, it can transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronized display.

[0035] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application that controls the display device 200.

[0036] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0037] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0038] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0039] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface 280.

[0040] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0041] In some embodiments, the display 260 includes display function components for presenting images and driving components for driving image display. The display 260 is used to receive and display image signals output from the controller 250. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces, etc.

[0042] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communication device 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communication device 220 with Bluetooth functionality.

[0043] The communication device 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0044] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0045] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0046] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface 280 receives the user input commands through the graphical user interface (GUI).

[0047] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0048] In some embodiments, the user input interface 280 can be used to receive instructions from user input.

[0049] In some embodiments, to enable user interaction, the display device 200 may run an operating system. An operating system is a computer program that manages and controls the hardware and software resources of the display device 200. The operating system can control the display device to provide a user interface; for example, the operating system can directly control the display device to provide a user interface, or it can provide a user interface by running applications. The operating system also allows users to interact with the display device 200.

[0050] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices.

[0051] An operating system can be divided into different modules or levels based on the functions it implements, for example... Figure 3 As shown, in some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Library layer, and the Kernel layer.

[0052] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with the user based on the applications. The application layer may contain at least one application, which may be a built-in Windows program, system settings program, or clock program of the operating system; or it may be an application developed by a third-party developer. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0053] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0054] like Figure 3 As shown, the application framework layer in this embodiment includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0055] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling changes to the display window, such as shrinking the display window, shaking the display, or distorting the display.

[0056] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0057] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, ... Figure 3As shown, hardware drivers can be configured in the kernel layer. The kernel layer can contain at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0058] It should be noted that the above examples are merely a simple division of operating system functions and do not limit the specific form of the operating system of the display device 200 in this application embodiment. Depending on the function of the display device, the type of operating system, and other factors, the number of levels and the specific level type of the operating system may be expressed in other forms.

[0059] With the rapid development of communication systems such as video telephony, the number of speakers and microphones used in these systems has increased to provide users with an immersive audio experience. As the number of speakers and microphones increases, traditional monophonic echo cancellation algorithms are no longer sufficient, necessitating a new echo cancellation algorithm to eliminate echoes. Compared to single-channel echo cancellation algorithms, stereo echo cancellation suffers from strong correlations between different channels, resulting in a singular correlation matrix and thus no unique solution. Furthermore, the probability of echo path changes is significantly increased in stereo echo cancellation. If a traditional adaptive filter is used, it must track not only abrupt changes in the near-end acoustic path but also changes in the far-end acoustic path, leading to slow convergence of the adaptive filter. Even with adaptive filtering, a cascaded nonlinear suppression module is still needed to suppress residual echoes, which can degrade speech quality.

[0060] There are generally four methods for stereo echo cancellation algorithms. The first method is to estimate the echo directly through the speaker signal and then use Wiener filtering to suppress the echo. Although this method is simple, the voice quality will drop sharply as the signal-to-echo ratio decreases, and it can only be applied to scenarios with a high signal-to-echo ratio.

[0061] The second method involves decorrelation processing of the speaker signal to reduce the correlation between channels and improve the convergence speed of linear filtering. To reduce channel correlation, this requires adding a component to the original signal or transforming the original signal. The drawback of this method is its algorithm complexity; the stronger the decorrelation, the greater the damage to sound quality.

[0062] The third method combines traditional techniques with artificial intelligence (AI). Similar to most current single-channel echo cancellation algorithms, it first performs adaptive filtering, then passes the signal through a neural network model to achieve echo cancellation. The drawback of this method is that because of the preceding adaptive filtering, it inevitably requires decorrelation between channels, which can lead to some loss of sound quality.

[0063] The fourth approach combines traditional methods with AI, but here the AI ​​is directly applied to adaptive filtering to address the slow convergence of current adaptive filtering. Then, a traditional nonlinear processing module is added to suppress some residual echoes.

[0064] To reduce algorithm complexity while ensuring sound quality, such as Figure 4 As shown in the figure, this application provides a stereo echo cancellation method, applied to a terminal device, which can perform the following steps: Step S401: When playing audio through multiple speakers, if multiple microphones are detected to be activated, acquire multiple first microphone signals and multiple speaker signals.

[0065] Upon receiving a user's instruction to play audio, the terminal device can play audio through multiple built-in or external speakers. The microphone can be continuously activated so that the terminal device can continuously acquire audio signals and identify whether the audio signal includes a far-field wake-up word to initiate the voice control service. After receiving a user's instruction to activate the voice service by pressing the voice button on the control device 100, the terminal device's built-in or external microphones are activated, allowing multiple microphones to acquire audio signals.

[0066] The first microphone signal is the audio signal captured by the corresponding microphone, which includes the user's voice, noise, and echoes caused by the audio played by the speaker of the display device 200. The speaker signal is the raw audio signal played by the corresponding speaker, which can be provided by an audio source.

[0067] In some embodiments, the microphone stores the acquired first microphone signal in a buffer, and the audio source also stores the original audio signal, i.e., the speaker signal, in the same buffer. The first microphone signal and the speaker signal are stored in an interleaved manner. Taking two first microphone signals and two speaker signals, for a total of four channels, as an example, the first two channels are the first microphone signals, and the last two channels are the speaker signals. Interleaved storage can also store the data of the left and right channels together sequentially. For example, for sampling points L1 (the first sampling point data of the left channel) and R1 (the first sampling point data of the right channel), during interleaved storage, L1 is stored first, followed by R1, and then L2 and R2.

[0068] One implementation of acquiring multiple first microphone signals and multiple speaker signals may include: reading input audio signals frame by frame from a buffer, wherein the input audio signals are signals of first microphone signals and speaker signals interleaved and stored in the buffer, and then performing deinterleaving processing on the input audio signals to obtain multiple first microphone signals and multiple speaker signals.

[0069] Deinterleaving involves separating the interleaved input audio signal into individual channels, such as the first microphone signal and the speaker signal. T represents the interleaved signal, where... , Indicates the first in the interleaved sequence The sampled signal at the nth position, after deinterleaving, the nth position... The first channel Each sampling point is represented as: .

[0070] Where N represents the number of channels interleaved. Taking two microphone signals and two speaker signals, i.e., 4 channels, as an example, N is currently set to 4. After deinterleaving, the signals are restored to four channels: microphone 1, microphone 2, speaker 1, and speaker 2.

[0071] In some embodiments, the microphone stores the acquired first microphone signal in a microphone buffer, and the audio source also stores the original audio signal, i.e., the speaker signal, in a speaker buffer. One implementation of acquiring multiple first microphone signals and multiple speaker signals may include: acquiring multiple first microphone signals from the microphone buffer and acquiring multiple speaker signals from the speaker buffer.

[0072] It should be noted that the number of microphone signals depends on the number of microphones installed. For example, if one microphone is installed on each side of the display device 200, there will be two microphone signals. The number of speaker signals depends on the number of speakers installed. For example, if one speaker is installed on each side of the display device 200, there will be two speaker signals. The speakers can be the built-in speakers of the display device 200 or external audio devices connected to the display device 200.

[0073] Step S402: Perform time-frequency conversion on multiple first microphone signals to obtain multiple first spectra, and perform time-frequency conversion on multiple speaker signals to obtain multiple second spectra.

[0074] In some embodiments, one implementation of performing time-frequency conversion on multiple first microphone signals to obtain multiple first spectra, and performing time-frequency conversion on multiple speaker signals to obtain multiple second spectra, may include: performing frame-by-frame windowing processing on multiple first microphone signals to obtain multiple first windowed data, and performing frame-by-frame windowing processing on multiple speaker signals to obtain second windowed data; then performing Fast Fourier Transform (FFT) on multiple first windowed data to obtain multiple first spectra, and performing FFT on multiple second windowed data to obtain multiple second spectra. The windowed data is a set of short-time signal segments that have undergone time localization and smoothing weighting, which can provide a quasi-stationary input with good spectral characteristics for subsequent time-frequency analysis.

[0075] Time-frequency conversion refers to converting the first microphone signal and speaker signal from the time domain to the frequency domain. This mainly involves frame segmentation and windowing, as well as Fast Fourier Transform (FFT). Frame segmentation and windowing treats the first microphone signal and speaker signal as a minimum processing unit of 10ms and then adds a Hanning window. The purpose of windowing is mainly to reduce spectral leakage and edge effects caused by frame segmentation.

[0076] The Hanning window is a commonly used window function widely applied in digital signal processing, particularly in spectrum analysis, filter design, and audio processing. Its main function is to smoothly truncate signals of finite length, reducing spectral leakage caused by abrupt signal changes during Fourier transforms.

[0077] The formula for the window function is as follows: ; .

[0078] in This represents the data after windowing. This represents the original input data after framing. This represents the Hanning window. The range of values ​​for n is... . Indicates the length of the window function.

[0079] The Fast Fourier Transform (FFT) is an efficient algorithm for calculating the Discrete Fourier Transform (DFT) and its inverse, which can greatly improve the efficiency of signal conversion between the time and frequency domains. The DFT converts a finite-length time-domain signal into a frequency-domain representation, thereby revealing the frequency components contained in the signal.

[0080] The spectrum is a complex spectrum. A complex spectrum is a complex number corresponding to each time frame and each frequency point, containing amplitude and phase. Amplitude is used to represent the energy level of the frequency component, and phase is used to represent the starting angle or timing information of the frequency component. The spectrum obtained after time-frequency conversion is a complex matrix that represents the amplitude and phase information of the signal at different time frames and frequency points; it is a mapping of the time-domain signal to the time-frequency domain.

[0081] Step S403: Perform feature extraction on multiple first spectra to obtain multiple first amplitude spectra, and perform feature extraction on multiple second spectra to obtain multiple second amplitude spectra.

[0082] The first amplitude spectrum is used to characterize the energy distribution of the corresponding first microphone signal, and the second amplitude spectrum is used to characterize the energy distribution of the corresponding loudspeaker signal.

[0083] Feature extraction is primarily performed in the frequency domain, currently mainly using amplitude spectrum. The amplitude spectrum is calculated separately for each channel: a first amplitude spectrum is calculated based on the first spectra corresponding to multiple first microphone signals, and a second amplitude spectrum is calculated based on the second spectra corresponding to multiple speaker signals. Typically, amplitude spectrum compression is used to improve the model's generalization ability. Amplitude spectrum compression refers to nonlinearly compressing the amplitude spectrum of a signal (i.e., the amplitude of each frequency component in the frequency domain) to improve dynamic range, enhance weak components, suppress strong components, or make it more consistent with human auditory perception characteristics.

[0084] The formula for calculating the compressed amplitude spectrum is: .

[0085] in Indicates the compressed amplitude spectrum. The base is The logarithm of and Indicate the real and imaginary parts. This represents the square root function. Represents the amplitude spectrum. This represents a very small constant, currently set to 1e-8, mainly to avoid abnormal situations caused by the amplitude spectrum being 0.

[0086] Step S404: Perform sub-band decomposition on multiple first amplitude spectra to obtain multiple low-frequency sub-bands and multiple high-frequency sub-bands corresponding to each first amplitude spectrum; and perform sub-band decomposition on multiple second amplitude spectra to obtain multiple low-frequency sub-bands and multiple high-frequency sub-bands corresponding to each second amplitude spectrum.

[0087] Subband decomposition is a signal processing technique that divides a broadband signal into multiple narrowband sub-bands (sub-bands) according to frequency. Its core idea is that different frequency components may have different statistical characteristics or processing requirements. By processing them independently in sub-bands, algorithm performance can be improved, computational complexity can be reduced, and frequency domain behavior can be controlled more precisely.

[0088] In some embodiments, the subbands are specifically divided into 0-500Hz, 500-1000Hz, 1000-2000Hz, 2000-3000Hz, 3000-4000Hz, 4000-6000Hz, and 6000-8000Hz. A total of 7 subbands are divided, with the first 5 being low-frequency subbands and the last 2 being high-frequency subbands. This application does not limit the method of subband division.

[0089] Step S405: Perform feature fusion on all low-frequency subbands to obtain low-subband merging features, and perform feature fusion on all high-frequency subbands to obtain high-subband merging features.

[0090] In some embodiments, inputting low-frequency sub-bands and high-frequency sub-bands of the same frequency band into the sub-band feature fusion module can obtain the fusion features of each low-frequency band, i.e., the first fusion feature, and the fusion features of each high-frequency band, i.e., the second fusion feature. For example... Figure 5 As shown, the subband feature fusion module includes a first convolutional module, a second convolutional module, a third convolutional module, a fourth convolutional module, a fifth convolutional module, a fusion module, and a dimensionality reduction network. The first, second, third, fourth, and fifth convolutional modules respectively include two-dimensional convolution (conv), batch normalization (batch normalization), and an activation function (reLU). Two-dimensional convolution and batch normalization mainly accelerate convergence and improve model stability. The expression for the reLU activation function is: .

[0091] The fusion module primarily adjusts and compresses dimensions, then merges multiple extracted features. Dimensionality reduction networks mainly reduce the dimensionality of features, thus reducing model complexity. This embodiment can employ fully connected networks for dimensionality reduction, which is suitable when computational resources are limited. When computational resources are sufficient, dimensionality reduction networks are unnecessary.

[0092] In some embodiments, one implementation of feature fusion of all low-frequency subbands to obtain low-frequency subband merging features may include: 1) Input all low-frequency subbands of the same low-frequency band into the first convolution module, so as to output the first time-frequency joint feature through the first convolution module. The first time-frequency joint feature is used to characterize the energy distribution of the audio signal in the joint domain of time and frequency when the number of channels is expanded to a first number.

[0093] For example, taking two microphone signals and two speaker signals, i.e., four channels, the low-frequency sub-bands of the same frequency band (0-500Hz, 500-1000Hz, 1000-2000Hz, 2000-3000Hz, or 3000-4000Hz) of the four channels are input into the first convolutional module to output the first time-frequency joint feature corresponding to the low-frequency band. The original number of channels is 4, and the first convolutional module can expand the number of channels to 8.

[0094] For example, the kernel size of the first convolutional module can be 3×3, the stride is 2, and the padding is (1,1). The purpose of the first convolutional module is to expand the channels, initially extract the joint time-frequency features, downsample, and reduce complexity. The input is data in BCTF format, where B (Batch size) is the number of samples in the batch, C (Channels) is the number of channels of the signal, T (Time) is the number of time frames in the time-frequency representation, and F (Frequency) is the frequency, i.e., the number of frequency points per frame. The input of the first convolutional module is 1×4×10¹×16, and the output is 1×8×10¹×8.

[0095] 2) Input the first time-frequency joint feature into the second convolution module to output the first frequency compression feature through the second convolution module. The first frequency compression feature is the energy distribution of the audio signal in the joint domain of time and frequency after frequency compression.

[0096] For example, the second convolutional module has a kernel size of 1×3, a stride of 2, and padding of (0, 1). The purpose of the second convolutional module is to refine the frequency dimension features and downsample the feature vectors to reduce the complexity of the model. The input of the second convolutional module is 1×8×10¹×8, and the output is 1×8×10¹×4. The second convolutional module can compress the frequency from 8 to 4.

[0097] 3) Input the first frequency compression feature into the third convolution module to output the first long-time and cross-frequency features through the third convolution module. The first long-time and cross-frequency features are used to characterize the energy distribution of the audio signal in the joint time and frequency domain when the number of channels is expanded to a second number, and the second number is greater than the first number.

[0098] For example, the third convolutional module has a kernel size of 3×3, a stride of 2, and padding of (1,1). The purpose of the third convolutional module is to expand the channels and extract long-term and cross-frequency features. Simultaneously, it further downsamples the feature vectors, further reducing model complexity. The input to the third convolutional module is 1×8×10¹×4, and the output is 1×16×10¹×2. The third convolutional module can expand the number of channels from 8 to 16.

[0099] 4) Input the first long-time and cross-frequency features into the fourth convolution module to output the first channel interaction features through the fourth convolution module. The first channel interaction features are used to characterize the energy distribution of the audio signal in the joint time and frequency domain after the inter-channel interaction features.

[0100] For example, the fourth convolutional module has a kernel size of 1×1, a stride of 1, and padding of (0, 0). The purpose of the fourth convolutional module is to enhance the feature interaction between channels without increasing computational cost, providing more integrated features for subsequent convolutional layers and avoiding channel feature redundancy. The input of the fourth convolutional module is 1×16×10¹×², and the output is 1×16×10¹×².

[0101] 5) Input the first channel interaction feature into the fifth convolution module to output the first final joint feature through the fifth convolution module. The first final joint feature is used to characterize the energy distribution of the audio signal in the joint domain of time and frequency when the number of channels is compressed to the first number.

[0102] For example, the fifth convolutional module has a 3×3 kernel size, a stride of 1, and padding of (1,1). The purpose of this fifth convolutional module is to compress channels, reduce feature redundancy, and improve computational efficiency, while simultaneously refining the final joint features using a 3×3 kernel. The input to the fifth convolutional module is 1×16×10¹×², and the output is 1×8×10¹×². This fifth convolutional module can compress the number of channels from 16 to 8.

[0103] 6) The first time-frequency joint feature, the first long-time and cross-frequency feature and the first final joint feature are fused to obtain the first fused feature corresponding to the low-frequency band.

[0104] The fusion module primarily fuses features from different resolutions, balancing resolution and receptive field. It extracts detailed features while also incorporating global features, further improving system stability. The fusion process combines the outputs of the first, third, and fifth convolutions. First, the channel and frequency dimensions are merged, and then the three convolutions are merged again.

[0105] For example, the first convolution outputs a first time-frequency joint feature of 1×8×10¹×8, the third convolution outputs a first long-time and cross-frequency feature of 1×16×10¹×2, and the fifth convolution outputs a first final joint feature of 1×8×10¹×2. First, the channel and frequency dimensions are merged, so the first convolution output is 1×10¹×64, the third convolution output is 1×10¹×32, and the fifth convolution output is 1×10¹×16. After feature fusion, these three convolution outputs become 1×10¹×112, which is the first fused feature.

[0106] 7) The first fusion features of multiple low-frequency bands are merged to obtain the low sub-band fusion features.

[0107] The first fusion features corresponding to the low-frequency bands 0-500Hz, 500-1000Hz, 1000-2000Hz, 2000-3000Hz and 3000-4000Hz are merged to obtain the low-subband merging features.

[0108] In some embodiments, after obtaining the first fused feature, the first fused feature can be passed through a dimensionality reduction network to obtain a reduced-dimensional first fused feature. Then, the reduced-dimensional first fused features of multiple low-frequency subbands are merged to obtain the low-frequency subband merged feature. The dimensionality reduction network mainly reduces the complexity of the model. The feature vector output by the fusion module becomes larger relative to the input. To keep the overall network complexity from becoming too high, a dimensionality reduction network is needed. A fully connected network can be used for the dimensionality reduction. The input is 1×10¹×1¹², which becomes 1×10¹×6⁴ after passing through the dimensionality reduction network.

[0109] For example, the first fusion features corresponding to the low-frequency bands 0-500Hz, 500-1000Hz, 1000-2000Hz, 2000-3000Hz, and 3000-4000Hz are 1×101×64, 1×101×64, 1×101×32, and 1×101×32, respectively. The low-subband fusion feature vector after merging the first fusion features of the seven low-frequency bands becomes 1×101×256.

[0110] In some embodiments, one implementation of feature fusion of all high-frequency subbands to obtain high-frequency subband merging features may include: 1) Input all high-frequency subbands of the same high-frequency band into the first convolution module to output the second time-frequency joint feature through the first convolution module. The second time-frequency joint feature is used to characterize the energy distribution of the audio signal in the joint domain of time and frequency when the number of channels is expanded to the first number.

[0111] 2) Input the second time-frequency joint feature into the second convolution module to output the second frequency compression feature through the second convolution module. The second frequency compression feature is the energy distribution of the audio signal in the joint domain of time and frequency after frequency compression.

[0112] 3) Input the second frequency compression feature into the third convolution module to output the second long-time and cross-frequency features through the third convolution module. The second long-time and cross-frequency features are used to characterize the energy distribution of the audio signal in the joint time and frequency domain when the number of channels is expanded to the second number. 4) Input the second long-time and cross-frequency features into the fourth convolution module to output the second channel interaction features through the fourth convolution module. The second channel interaction features are used to characterize the energy distribution of the audio signal in the joint time and frequency domain after the inter-channel interaction features. 5) Input the second channel interaction feature into the fifth convolution module to output the second final joint feature through the fifth convolution module. The second final joint feature is used to characterize the energy distribution of the audio signal in the joint domain of time and frequency when the number of channels is compressed to the first number.

[0113] 6) The second time-frequency joint feature, the second long-time and cross-frequency feature, and the second final joint feature are fused to obtain the second fused feature corresponding to the high-frequency band.

[0114] It should be noted that the method for obtaining the second fusion feature is the same as that for obtaining the first fusion feature, and the convolutional module used is the same, so it will not be repeated here.

[0115] 7) The second fusion features of multiple high-frequency bands are merged to obtain the high sub-band merging features.

[0116] The second fusion features corresponding to the high-frequency bands 4000-6000Hz and 6000-8000Hz are merged to obtain the high subband merging features.

[0117] For example, the second fusion features corresponding to the high-frequency bands 4000-6000Hz and 6000-8000Hz are 1×101×64 and 1×101×32, respectively. The high-frequency sub-band fusion feature vector after merging the second fusion features of the high-frequency sub-band becomes 1×101×96.

[0118] For speech, the low-frequency components are far more important than the high-frequency components. This application's embodiments extract features from sub-bands. The sub-band division method primarily allocates more parameters to the low-frequency components and fewer parameters to the high-frequency components. Simultaneously, features with different resolutions and receptive fields are fused, balancing both resolution and receptive field. As the number of convolutional layers increases, the resolution decreases, but the receptive field increases. This application's embodiments achieve this balance by extracting features from different convolutional layer numbers and fusing them.

[0119] Subband feature fusion methods such as Figure 6As shown, the input is the compressed amplitude spectrum for each channel. The compressed amplitude spectrum is divided into sub-bands 1-7. Then, sub-bands of the same frequency band are sequentially input into the sub-band feature fusion module to obtain the fusion features corresponding to that frequency band. Then, the fusion features of multiple low-frequency bands are merged to obtain the low sub-band merged features, and the fusion features of multiple high-frequency bands are merged to obtain the high sub-band merged features. In this embodiment, as the number of convolutional layers increases, the receptive field becomes larger, but the resolution becomes lower, thus simultaneously balancing frequency resolution and receptive field.

[0120] Step S406: Input the low subband merging features and high subband merging features into the neural network model to output multiple amplitude spectrum gain factors. Different amplitude spectrum gain factors correspond to different first microphone signals.

[0121] In some embodiments, the neural network model may include a high-low subband fusion module, a fully connected network, and an activation function layer. One implementation that inputs low-subband merging features and high-subband merging features into the neural network model to output multiple amplitude spectrum gain factors may include: inputting the low-subband merging features and high-subband merging features into the high-low subband fusion module to output a third fusion feature; then inputting the third fusion feature into the fully connected network to output a frequency domain mask for multiple channels, the frequency domain mask representing the degree of preservation or suppression of signal energy at different frequencies in the frequency domain; and finally activating the frequency domain mask using an activation function layer to obtain multiple amplitude spectrum gain factors.

[0122] In some embodiments, such as Figure 7 As shown, the neural network model may include a high-low subband fusion module, a fully connected network, a sigmoid activation function layer, a first time-frequency network, a second time-frequency network, and a dimensionality reduction network. One implementation method that inputs low-subband merging features and high-subband merging features into the neural network model to output multiple amplitude spectrum gain factors through the neural network model may include: 1) Input the low subband merging features and high subband merging features into the first time-frequency network respectively, so as to enhance the low subband merging features and high subband merging features through the first time-frequency network.

[0123] The main role of the time-frequency network is to further enhance features by modeling long-term temporal dependencies and local features in speech, thereby improving model stability and generalization ability. This step is not essential and can be omitted in scenarios with limited computing power.

[0124] 2) The enhanced low-subband merging features and high-subband merging features are input into the dimensionality reduction network, respectively, to output low-subband merging features and high-subband merging features with reduced dimensionality. The dimensionality reduction network mainly reduces the model complexity.

[0125] 3) Input the reduced-dimensional low-subband merging feature and high-subband merging feature into the high-low subband fusion module, so as to output the third fusion feature through the high-low subband fusion module.

[0126] For example, after passing through the time-frequency network, the feature vector is consistent with the input. After passing through the dimensionality reduction network, the feature vector becomes 1×10¹×1¹² and 1×10¹×4⁸, respectively. After passing through the high-low subband fusion module, the feature vector becomes 1×10¹×16⁰.

[0127] 4) The third fused feature is input into the second time-frequency network to enhance the third fused feature. The main function of the time-frequency network is to make the feature explicit. After passing through the second time-frequency network, the feature vector remains the same as the input, 1×101×160.

[0128] 5) Input the enhanced third fusion feature into the fully connected network to output a frequency domain mask for multiple channels through the fully connected network.

[0129] 6) Activate the frequency domain mask using an activation function layer to obtain multiple amplitude spectrum gain factors.

[0130] For example, the amplitude spectrum gain factors mask1 and mask2 of the left and right channels are obtained by passing through a time-frequency network, a fully connected network, and a sigmoid activation function, respectively.

[0131] The main function of the fully connected network and the sigmoid activation function is to generate the mask. The fully connected network outputs the frequency domain masks for the left and right channels. Then, the sigmoid activation function is used to activate this mask, and the output is a coefficient between 0 and 1, where 0 represents filtering and 1 represents retention.

[0132] It should be added that time-frequency networks can all adopt one of four types: Temporal Convolutional Network (TCN), Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), or Conformer network structure, with little overall difference. Time-frequency networks can be constructed by stacking them.

[0133] For example, such as Figure 8 As shown, the time-frequency network uses TCN, and the number of stacked layers N can be adjusted according to the device's computing power. Conv1 represents a convolution with a kernel of 1, and Conv3 represents a convolution with a kernel of 3. BatchNorm represents batch normalization, and PReLU is the activation function, whose formula is as follows: , where 'a' is obtained through training.

[0134] Step S407: Multiply the multiple amplitude spectrum gain factors by the corresponding first spectrum to obtain multiple gain spectra.

[0135] The gain spectrum is obtained by multiplying the microphone channel signal after FFT with the corresponding amplitude spectrum gain factor. That is, the gain spectrum of the left and right channel microphone signals is obtained by multiplying the first spectrum of the left and right channel microphone signals with mask1 and mask2 respectively.

[0136] Step S408: Perform frequency-time conversion on multiple gain spectra to obtain multiple second microphone signals.

[0137] By performing an inverse FFT transform on the gain spectrum, the time-domain signal after removing echo and noise can be obtained, which is the second microphone signal.

[0138] In some embodiments, after converting the gain spectrum into a second microphone signal, multiple second microphone signals and multiple speaker signals are interleaved to obtain an output signal, and then the output signal is output.

[0139] The processed channel signals are restored to the same interleaved state as the input. The microphone signal is the time-domain signal obtained by inverse FFT in the previous step, i.e., the second microphone signal, while the speaker signal is directly the deinterleaved speaker signal, such as the signals of speaker 1 and speaker 2 channels.

[0140] The formula for calculating interleaved sequences is as follows: .

[0141] in, This represents the sampled signal at the m-th position in the interleaved sequence. This represents the k-th sampling point of the i-th channel, and N represents the number of interleaved channels. Taking two microphone signals and two speaker signals, i.e., 4 channels, as an example, N is currently set to 4.

[0142] The upstream provides data in an interleaved format, and the echo cancellation process outputs the data in the same interleaved format. Downstream components (such as speech recognition, encoders, and network transmission) do not require format conversion, thus reducing latency and complexity.

[0143] For example, display device 200 has two microphones (left / right) for stereo sound pickup, and speakers play stereo music (L / R). The audio signal is transmitted in an interleaved format. After stereo echo cancellation processing, it is still in an interleaved format. Subsequent voice call encoders can directly package this interleaved audio signal and send it to the other party, maintaining the stereo effect without echo.

[0144] In some embodiments, the algorithm flowchart of the stereo echo cancellation method can be as follows: Figure 9 As shown, the input signals, containing two microphone signals and two speaker signals, are deinterleaved to obtain microphone signal 1, microphone signal 2, speaker signal 1, and speaker signal 2. The signals from these four channels are then subjected to time-frequency conversion to obtain four spectra. Feature extraction and fusion are performed on the four spectra to obtain low-subband merging features and high-subband merging features. These features are input into a neural network model to obtain the amplitude spectrum gain factors mask1 and mask2 for the left and right channels. Multiplying microphone signal 1 with mask1 and performing an inverse FFT transform yields the echo- and noise-removed left channel microphone signal. Multiplying microphone signal 2 with mask2 and performing an inverse FFT transform yields the echo- and noise-removed right channel microphone signal. The left channel microphone signal, right channel microphone signal, speaker signal 1, and speaker signal 2 are interleaved to obtain the output signal.

[0145] In some embodiments, the method for recording training data may include: playing different music and voice messages in different rooms using the built-in player of the display device 200 and third-party music playback software respectively, and saving the signals collected by the microphone and the speaker signals at the system level to the device. Record 300 hours of voice and 300 hours of music scenes respectively. The recording rooms cover different room sizes and different reverberation times.

[0146] Using recorded training data Figure 9 The algorithm model was trained using PyTorch. The training process included: training with an initial learning rate (lr) of 0.001, using the Adaptive Moments Estimator (Adam) as the optimizer, and a maximum training epoch of 300.

[0147] In some embodiments, the display device 200 incorporates an echo cancellation module to enable stereo echo cancellation functionality. The echo cancellation module may include a deinterleaving module, a time-frequency conversion module, a feature extraction module, a neural network model, a frequency-time conversion module, and an interleaving module. The timing diagram of the stereo echo cancellation method can be as follows: Figure 10 As shown. The deinterleaving module can acquire the input audio signal from the buffer, then deinterleave the input audio signal to obtain multiple first microphone signals and multiple speaker signals. These signals are then sent to the time-frequency conversion module, and the speaker signals are also sent to the interleaving module. The first microphone signals are audio signals acquired by the microphones, and the speaker signals are the original audio signals played by the speakers.

[0148] The time-frequency conversion module performs time-frequency conversion on the first microphone signal and the speaker signal respectively, obtaining a first spectrum corresponding to the first microphone signal and a second spectrum corresponding to the speaker signal, and sends the first and second spectra to the feature extraction module. The feature extraction module extracts features from the first and second spectra respectively, obtaining a first amplitude spectrum and a second amplitude spectrum. Then, it performs sub-band decomposition on the first and second amplitude spectra respectively, obtaining multiple low-frequency sub-bands and multiple high-frequency sub-bands corresponding to each amplitude spectrum. Then, it performs feature fusion on the multiple low-frequency sub-bands and multiple high-frequency sub-bands respectively, obtaining low-sub-band merging features and high-sub-band merging features. Finally, it sends the low-sub-band merging features and high-sub-band merging features to the neural network model.

[0149] The neural network model takes low-subband merging features and high-subband merging features as input, outputs multiple amplitude spectrum gain factors, and sends these multiple amplitude spectrum gain factors to the frequency-time conversion module. The frequency-time conversion module multiplies each of the multiple amplitude spectrum gain factors by its corresponding first spectrum to obtain multiple gain spectra, then performs frequency-time conversion on these gain spectra to obtain multiple second microphone signals, which are then sent to the interleaving module. The interleaving module interleaves the multiple second microphone signals and multiple speaker signals to obtain the output signal.

[0150] Compared to current stereo echo cancellation algorithms, this embodiment maintains two distinct output channels, achieving true stereo output. Secondly, the entire algorithm is end-to-end, eliminating the need for traditional algorithms and significantly improving overall sound quality. Thirdly, the backbone networks for both microphones are unified, eliminating the need for separate processing of each channel and reducing algorithm complexity. Finally, a sub-band feature fusion approach is used to extract features, balancing receptive field and resolution. This approach allocates more parameters to lower sub-bands and fewer to higher sub-bands, resulting in a significant performance improvement with only a slight increase in the number of parameters.

[0151] Some embodiments of this application also provide a computer-readable storage medium that can store a program. When the computer storage medium is configured in a display device or server, the program, when executed, can include the program steps involved in the stereo echo cancellation method in the above embodiments. The computer storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0152] This application provides an electronic device, which includes a processor and a memory for storing processor-executable instructions. The processor is configured to read executable instructions from the memory and execute the instructions to implement the stereo echo cancellation method described in the above embodiments.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A terminal device, characterized in that, include: Multiple microphones; Multiple speakers; The controller is configured as follows: When playing audio through the multiple speakers, if the activation of the multiple microphones is detected, multiple first microphone signals and multiple speaker signals are acquired, wherein the first microphone signals are the audio signals collected by the corresponding microphones, and the speaker signals are the original audio signals played by the corresponding speakers. The multiple first microphone signals are subjected to time-frequency conversion to obtain multiple first spectra, and the multiple speaker signals are subjected to time-frequency conversion to obtain multiple second spectra; Feature extraction is performed on the plurality of first spectra to obtain a plurality of first amplitude spectra, and feature extraction is performed on the plurality of second spectra to obtain a plurality of second amplitude spectra, wherein the first amplitude spectra are used to characterize the energy distribution of the corresponding first microphone signal, and the second amplitude spectra are used to characterize the energy distribution of the corresponding loudspeaker signal; Subband decomposition is performed on the plurality of first amplitude spectra to obtain a plurality of low-frequency subbands and a plurality of high-frequency subbands corresponding to each first amplitude spectrum; and subband decomposition is performed on the plurality of second amplitude spectra to obtain a plurality of low-frequency subbands and a plurality of high-frequency subbands corresponding to each second amplitude spectrum. Feature fusion is performed on all the low-frequency subbands to obtain low subband merging features, and feature fusion is performed on all the high-frequency subbands to obtain high subband merging features; The low subband merging feature and the high subband merging feature are input into a neural network model to output multiple amplitude spectrum gain factors through the neural network model. Different amplitude spectrum gain factors correspond to different first microphone signals. The plurality of amplitude spectrum gain factors are multiplied by the corresponding first spectrum to obtain a plurality of gain spectra; The multiple gain spectra are frequency-time converted to obtain multiple second microphone signals, which are microphone signals with echo and noise removed.

2. The terminal device according to claim 1, characterized in that, The controller performs time-frequency conversion on the plurality of first microphone signals to obtain a plurality of first spectra, and performs time-frequency conversion on the plurality of speaker signals to obtain a plurality of second spectra, and is further configured to: The plurality of first microphone signals are subjected to frame-by-frame windowing processing to obtain a plurality of first windowed data, and the plurality of speaker signals are subjected to frame-by-frame windowing processing to obtain second windowed data; The plurality of first windowed data are subjected to Fast Fourier Transform (FFT) to obtain the plurality of first spectra, and the plurality of second windowed data are subjected to FFT to obtain the plurality of second spectra.

3. The terminal device according to claim 1, characterized in that, The controller performs feature fusion on all the low-frequency sub-bands to obtain low-sub-band merging features, which are further configured as follows: All low-frequency subbands of the same low-frequency band are input into the first convolution module to output a first time-frequency joint feature. The first time-frequency joint feature is used to characterize the energy distribution of the audio signal in the joint domain of time and frequency when the number of channels is expanded to a first number. The first time-frequency joint feature is input into the second convolution module to output the first frequency compression feature through the second convolution module. The first frequency compression feature is the energy distribution of the audio signal in the joint domain of time and frequency after frequency compression. The first frequency compression feature is input into the third convolution module to output the first long-time and cross-frequency features through the third convolution module. The first long-time and cross-frequency features are used to characterize the energy distribution of the audio signal in the joint time and frequency domain when the number of channels is expanded to a second number, where the second number is greater than the first number. The first long-duration and cross-frequency features are input into the fourth convolution module to output the first channel interaction features through the fourth convolution module. The first channel interaction features are used to characterize the energy distribution of the audio signal in the joint time and frequency domain after the inter-channel interaction features. The first channel interaction feature is input into the fifth convolution module to output the first final joint feature through the fifth convolution module. The first final joint feature is used to characterize the energy distribution of the audio signal in the joint domain of time and frequency when the number of channels is compressed to the first number. The first time-frequency joint feature, the first long-time and cross-frequency feature, and the first final joint feature are fused to obtain the first fused feature of the low-frequency band. The first fusion features of multiple low-frequency bands are merged to obtain the low-subband merging features.

4. The terminal device according to claim 3, characterized in that, The controller performs feature fusion on all the high-frequency subbands to obtain high-frequency subband merging features, which are further configured as follows: All high-frequency subbands of the same high-frequency band are input into the first convolution module to output a second time-frequency joint feature through the first convolution module. The second time-frequency joint feature is used to characterize the energy distribution of the audio signal in the joint domain of time and frequency when the number of channels is expanded to a first number. The second time-frequency joint feature is input into the second convolution module to output the second frequency compression feature through the second convolution module. The second frequency compression feature is the energy distribution of the audio signal in the joint domain of time and frequency after frequency compression. The second frequency compression feature is input into the third convolution module to output the second long-time and cross-frequency features through the third convolution module. The second long-time and cross-frequency features are used to characterize the energy distribution of the audio signal in the joint time and frequency domain when the number of channels is expanded to a second number. The second long-duration and cross-frequency features are input into the fourth convolution module to output the second channel interaction features through the fourth convolution module. The second channel interaction features are used to characterize the energy distribution of the audio signal in the joint time and frequency domain after the inter-channel interaction features. The second channel interaction feature is input into the fifth convolution module to output the second final joint feature through the fifth convolution module. The second final joint feature is used to characterize the energy distribution of the audio signal in the joint domain of time and frequency when the number of channels is compressed to the first number. The second time-frequency joint feature, the second long-time and cross-frequency feature, and the second final joint feature are fused to obtain the second fused feature corresponding to the high-frequency band; By merging the second fusion features of multiple high-frequency bands, a high sub-band merging feature is obtained.

5. The terminal device according to claim 1, characterized in that, The neural network model includes a high-low subband fusion module, a fully connected network, and activation function layers. The controller executes the input of the low-subband merging features and the high-subband merging features into the neural network model to output multiple amplitude spectrum gain factors, which are further configured as follows: The low subband merging feature and the high subband merging feature are input into the high-low subband fusion module to output a third fusion feature through the high-low subband fusion module; The third fusion feature is input into the fully connected network to output a frequency domain mask for multiple channels through the fully connected network. The frequency domain mask is used to characterize the degree of preservation or suppression of signal energy at different frequencies in the frequency domain. The frequency domain mask is activated using the activation function layer to obtain multiple amplitude spectrum gain factors.

6. The terminal device according to claim 5, characterized in that, The neural network model further includes a first time-frequency network and a dimensionality reduction network. The controller executes the input of the low subband merging feature and the high subband merging feature into the high-low subband fusion module, so as to output a third fused feature through the high-low subband fusion module, and is further configured to: The low-subband merging feature and the high-subband merging feature are respectively input into the first time-frequency network to enhance the low-subband merging feature and the high-subband merging feature through the first time-frequency network; The enhanced low-subband merging features and high-subband merging features are respectively input into the dimensionality reduction network, so that the dimensionality reduction network outputs the low-subband merging features and high-subband merging features with reduced dimensions. The reduced-dimensional low-subband merging feature and the high-subband merging feature are input into the high-low subband fusion module to output a third fusion feature.

7. The terminal device according to claim 6, characterized in that, The neural network model further includes a second time-frequency network, and the controller executes the input of the third fused feature into the fully connected network to output a frequency domain mask with multiple channels through the fully connected network, which is further configured as follows: The third fusion feature is input into the second time-frequency network to enhance the third fusion feature through the second time-frequency network; The enhanced third fusion feature is input into the fully connected network to output a frequency domain mask for multiple channels through the fully connected network.

8. The terminal device according to claim 1, characterized in that, The controller is configured to acquire multiple first microphone signals and multiple speaker signals, and is further configured to: The input audio signal is read frame by frame from the buffer, wherein the input audio signal is a signal that is interleaved and stored in the buffer from the first microphone signal and the speaker signal; The input audio signal is deinterleaved to obtain multiple first microphone signals and multiple speaker signals.

9. The terminal device according to claim 8, characterized in that, After performing frequency-time conversion on the multiple gain spectra to obtain multiple second microphone signals, the controller is further configured to: Multiple signals from the second microphone and multiple signals from the speaker are interleaved to obtain an output signal, which is used for speech recognition.

10. A stereo echo cancellation method, characterized in that, include: When playing audio through multiple speakers, if multiple microphones are detected to be activated, multiple first microphone signals and multiple speaker signals are acquired. The first microphone signal is the audio signal collected by the corresponding microphone, and the speaker signal is the original audio signal played by the corresponding speaker. The multiple first microphone signals are subjected to time-frequency conversion to obtain multiple first spectra, and the multiple speaker signals are subjected to time-frequency conversion to obtain multiple second spectra; Feature extraction is performed on the plurality of first spectra to obtain a plurality of first amplitude spectra, and feature extraction is performed on the plurality of second spectra to obtain a plurality of second amplitude spectra; wherein, the first amplitude spectra are used to characterize the energy distribution of the corresponding first microphone signal, and the second amplitude spectra are used to characterize the energy distribution of the corresponding loudspeaker signal; Subband decomposition is performed on the plurality of first amplitude spectra to obtain a plurality of low-frequency subbands and a plurality of high-frequency subbands corresponding to each first amplitude spectrum; and subband decomposition is performed on the plurality of second amplitude spectra to obtain a plurality of low-frequency subbands and a plurality of high-frequency subbands corresponding to each second amplitude spectrum. Feature fusion is performed on all the low-frequency subbands to obtain low subband merging features, and feature fusion is performed on all the high-frequency subbands to obtain high subband merging features; The low subband merging feature and the high subband merging feature are input into a neural network model to output multiple amplitude spectrum gain factors through the neural network model. Different amplitude spectrum gain factors correspond to different first microphone signals. The plurality of amplitude spectrum gain factors are multiplied by the corresponding first spectrum to obtain a plurality of gain spectra; The multiple gain spectra are frequency-time converted to obtain multiple second microphone signals, which are microphone signals with echo and noise removed.

Images