System and method for mask-based neural beamforming for multi-channel speech enhancement
Through a mask-based multi-channel voice enhancement system, using mask estimation model and beamforming technology, the problem of poor performance of voice enhancement in noisy environments in the prior art is solved, and effective on resource-constrained devices is achieved Denoising and noise suppression.
Patent Information
- Application Number
- CN202480001753.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-08-29
- Filing Date
- 2024-03-13
- Publication Date
- 2024-12-03
AI Technical Summary
Existing voice enhancement technologies are difficult to effectively denoise in noisy environments, especially on resource-constrained devices, existing methods such as deep learning and hybrid solutions fail to satisfactorily solve the speech denoising problem, and existing masking methods Lack of direct connection to multi-channel characteristics, poor performance.
Using a mask-based multi-channel speech enhancement system, the mask estimation model is used to predict the mask of clean speech by generating noisy time-frequency representation, and the beamforming filter weight is determined, and the noisy time-frequency representation is applied to the noisy time-frequency representation. Isolate clean speech, use snapshot matching mask and amplitude-limited complex mask constraints, train the mask estimation network to improve denoising performance.
Effective voice denoising is achieved on resource-constrained devices, improving noise suppression performance, suitable for edge device deployment, and significantly improving voice quality.
Smart Images

Figure CN119072745A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates generally to machine learning systems and more particularly to systems and methods for mask-based neural beamforming for multi-channel speech enhancement. Background Art
[0002] Communication between users of electronic devices (such as mobile devices, wearable devices, and smart home appliances) or interaction with intelligent assistants using electronic devices has become increasingly common. Typically, these communications or interactions involve the use of various voice user interfaces driven by systems such as automatic speech recognition, keyword spotting, etc. However, ambient noise and interference in real environments pose problems for properly understanding voice input. For example, voice quality drops sharply in noisy environments, causing key information to be lost on the receiver side (human ears or intelligent assistants). Summary of the invention
[0003] Technical Solutions
[0004] The present disclosure relates to systems and methods for mask-based neural beamforming for multi-channel speech enhancement.
[0005] In a first embodiment, a method includes receiving a set of noisy audio signals from a plurality of audio input devices during a first time window. The method also includes generating a noisy time-frequency representation based on the set of noisy audio signals. The method also includes providing the noisy time-frequency representation as an input to a mask estimation model, which is trained to output a mask for predicting a clean time-frequency representation of clean speech audio from the noisy time-frequency representation. The method also includes determining beamforming filter weights based on the mask. The method also includes applying the beamforming filter weights to the noisy time-frequency representation to isolate the clean speech audio from the set of noisy audio signals. In addition, the method includes outputting clean speech audio.
[0006] In a second embodiment, an electronic device includes at least one processing device, the at least one processing device being configured to receive a group of noisy audio signals from a plurality of audio input devices during a first time window. The at least one processing device is also configured to generate a noisy time-frequency representation based on the group of noisy audio signals. The at least one processing device is also configured to provide the noisy time-frequency representation as an input to a mask estimation model, the mask estimation model being trained to output a mask for predicting a clean time-frequency representation of clean speech audio from the noisy time-frequency representation. The at least one processing device is also configured to determine beamforming filter weights based on the mask. The at least one processing device is also configured to apply beamforming filter weights to the noisy time-frequency representation to isolate clean speech audio from the group of noisy audio signals. In addition, the at least one processing device is configured to output clean speech audio.
[0007] In a third embodiment, a non-transitory machine-readable medium includes instructions that, when executed, cause at least one processor of an electronic device to receive a set of noisy audio signals from a plurality of audio input devices during a first time window. The non-transitory machine-readable medium also includes instructions that, when executed, cause the at least one processor to generate a noisy time-frequency representation based on the set of noisy audio signals. The non-transitory machine-readable medium also includes instructions that, when executed, cause the at least one processor to provide the noisy time-frequency representation as input to a mask estimation model that is trained to output a mask for predicting a clean time-frequency representation of clean speech audio from the noisy time-frequency representation. The non-transitory machine-readable medium also includes instructions that, when executed, cause the at least one processor to determine beamforming filter weights based on the mask. The non-transitory machine-readable medium also includes instructions that, when executed, cause the at least one processor to apply the beamforming filter weights to the noisy time-frequency representation to isolate the clean speech audio from the set of noisy audio signals. Additionally, the non-transitory machine-readable medium further includes instructions that, when executed, cause the at least one processor to output clean speech audio.
[0008] Other technical features may be clear to those skilled in the art from the following drawings, descriptions and claims.
[0009] Before proceeding to the following detailed description, it may be advantageous to set forth the definitions of certain words and phrases used throughout this patent document. The terms "send," "receive," and "communicate," and their derivatives, include direct and indirect communications. The terms "include," and "comprise," and their derivatives, are meant to include, but are not limited to, this. The term "or" is inclusive, meaning and / or. The phrase "associated with," and its derivatives, means to include, be included within, be interconnected with, include, be contained within, be connected to or connected with, be coupled to or coupled with, be communicable with, collaborate with, be intertwined, be juxtaposed, be close to, be bound to or bound with, have, have the nature of, have a relationship with, etc.
[0010] In addition, the various functions described below can be implemented or supported by one or more computer programs, each of which is formed by a computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data or a portion thereof that are suitable for implementation in a suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a compact disk (CD), a digital video disk (DVD) or any other type of memory. "Non-transitory" computer-readable media does not include wired, wireless, optical or other communication links that transmit temporary electrical signals or other signals. Non-transitory computer-readable media include media that can permanently store data and media that can store data and rewrite data later, such as rewritable optical disks or erasable memory devices.
[0011] As used herein, terms and phrases such as "having", "may have", "include" or "may include" a feature (such as a number, function, operation or a component such as a part) indicate the presence of the feature and do not exclude the presence of other features. In addition, as used herein, the phrases "A or B", "at least one of A and / or B" or "one or more of A and / or B" may include all possible combinations of A and B. For example, "A or B", "at least one of A and B" and "at least one of A or B" may indicate all of the following: (1) including at least one A, (2) including at least one B or (3) including at least one A and at least one B. In addition, as used herein, the terms "first" and "second" may modify various components regardless of their importance and do not limit the components. These terms are only used to distinguish one component from another. For example, a first user device and a second user device may indicate user devices that are different from each other, regardless of the order or importance of the devices. Without departing from the scope of the present disclosure, a first component may be represented as a second component, and vice versa.
[0012] It should be understood that when an element (such as a first element) is referred to as being "coupled" or "connected" (operably or communicatively) to another element (such as a second element), or when an element (such as a first element) is referred to as being "coupled to" or "connected to" (operably or communicatively) to another element (such as a second element), it may be coupled or connected to the other element directly or via a third element, or may be coupled or connected to the other element directly or via a third element. Conversely, it should be understood that when an element (such as a first element) is referred to as being "directly coupled" or "directly coupled to" another element (such as a second element), or being "directly connected" or "directly connected to" another element (such as a second element), there are no other elements (such as a third element) between the element and the other element.
[0013] As used herein, the phrase "configured (or set) to" may be used interchangeably with the phrases "suitable for," "capable of," "designed to," "suitable for," "manufactured to," or "capable of," depending on the circumstances. The phrase "configured (or set) to" does not inherently mean "specially designed in hardware to." On the contrary, the phrase "configured to" may indicate that a device may perform an operation together with another device or component. For example, the phrase "a processor configured (or set) to perform A, B, and C" may represent a general-purpose processor (such as a CPU or application processor) that can perform operations by executing one or more software programs stored in a memory device, or a special-purpose processor (such as an embedded processor) for performing operations.
[0014] The terms and phrases used herein are only used to describe some embodiments of the present disclosure, rather than to limit the scope of other embodiments of the present disclosure. It should be understood that the singular forms "one", "an" and "the" include plural references unless the context clearly states otherwise. All terms and phrases used herein, including technical and scientific terms and phrases, have the same meanings as those commonly understood by those of ordinary skill in the art to which the embodiments of the present disclosure belong. It will be further understood that terms and phrases, such as those defined in common dictionaries, should be interpreted as having meanings consistent with their meanings in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense, unless explicitly defined herein. In some cases, the terms and phrases defined herein may be interpreted as excluding embodiments of the present disclosure.
[0015] Examples of "electronic devices" according to embodiments of the present disclosure may include at least one of the following: a smart phone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (such as smart glasses, a head mounted device (HMD), electronic clothing, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, or a smart watch). Other examples of electronic devices include smart home appliances. Examples of smart home appliances may include at least one of the following: a television, a digital video disc (DVD) player, an audio player, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave, a washing machine, a dryer, an air purifier, a set-top box, a home automation control panel, a security control panel, a TV box (such as Samsung HOMESYNC, APPLETV, or GOOGLE TV), a smart speaker or a speaker with an integrated digital assistant (such as Samsung GALAXY HOME, APPLE HOMEPOD, or AMAZON ECHO), a game console (such as XBOX, PLAYSTATION, or NINTENDO), an electronic dictionary, an electronic key, a portable camera, or an electronic photo frame. Other examples of electronic devices include at least one of the following: various medical devices (such as various portable medical measuring devices (such as blood sugar measuring devices, heart rate measuring devices, or body temperature measuring devices), magnetic resource angiography (MRA) devices, magnetic resource imaging (MRI) devices, computed tomography (CT) devices, imaging devices, or ultrasound devices), navigation devices, global positioning system (GPS) receivers, event data recorders (EDRs), flight data recorders (FDRs), automotive infotainment devices, navigation electronic devices (such as navigation navigation devices or gyrocompasses), avionics equipment, security equipment, vehicle head units, industrial or household robots, automated teller machines (ATM) point of sale (POS) devices, or Internet of Things (IoT) devices (such as light bulbs, various sensors, electricity or gas meters, sprinklers, fire alarms, thermostats, street lights, toasters, fitness equipment, hot water tanks, heaters, or boilers). Other examples of electronic devices include at least a portion of the following: a piece of furniture or a building / structure, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (such as devices for measuring water, electricity, gas, or electromagnetic waves). Note that according to various embodiments of the present disclosure, the electronic device may be one or a combination of the devices listed above. According to some embodiments of the present disclosure, the electronic device may be a flexible electronic device. The electronic device disclosed herein is not limited to the devices listed above, and may include new electronic devices according to the development of technology.
[0016] In the following description, an electronic device is described with reference to the accompanying drawings according to various embodiments of the present disclosure. As used herein, the term "user" may refer to a person using an electronic device or another device (such as an artificial intelligence electronic device).
[0017] Definitions for certain other words and phrases may be provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.
[0018] Nothing in the description of this application should be read as implying that any particular element, step, or function is an essential element that must be included within the scope of the claims. The scope of the patented subject matter is limited solely by the claims. In addition, none of the claims are intended to invoke 35 U.S.C. §112(f) unless the exact words "means for..." are followed by a participle. The use of any other terms, including but not limited to "mechanism," "module," "device," "unit," "component," "element," "member," "device," "machine," "system," "processor," or "controller" used within the claims is understood by the applicant to refer to structures known to persons skilled in the relevant art and are not intended to invoke 35 U.S.C. §112(f). BRIEF DESCRIPTION OF THE DRAWINGS
[0019] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals represent like parts:
[0020] Figure 1 An example network configuration including electronic devices according to the present disclosure is shown;
[0021] Figure 2 An example speech enhancement system according to the present disclosure is shown;
[0022] Figure 3 Another example speech enhancement system according to the present disclosure is shown;
[0023] Figure 4 An example multi-channel speech enhancement architecture according to the present disclosure is shown;
[0024] Figure 5 An example snapshot matching mask estimation process according to the present disclosure is shown;
[0025] Figure 6 An example mask estimation model training process according to the present disclosure is shown;
[0026] Figure 7 An example audio signal denoising process according to the present disclosure is shown;
[0027] Figure 8 An example method for training a mask estimation model according to the present disclosure is shown; and
[0028] Fig. 9 An example method for denoising an audio signal according to the present disclosure is shown. DETAILED DESCRIPTION
[0029] The following discussion is described with reference to the accompanying drawings. Figures 1 to 9 As well as various embodiments of the present disclosure. However, it should be understood that the present disclosure is not limited to these embodiments, and all changes and / or equivalents or replacements thereof also belong to the scope of the present disclosure. Throughout the specification and the drawings, the same or similar reference numerals may be used to refer to the same or similar elements.
[0030] As mentioned above, communication between users using electronic devices (such as mobile devices, wearable devices, and smart home appliances) or interaction with intelligent assistants using electronic devices has become increasingly common. Typically, these communications or interactions involve the use of various voice user interfaces driven by systems such as automatic speech recognition, keyword recognition, etc. However, the surrounding noise and interference in real environments pose problems for properly understanding voice input. For example, voice quality drops sharply in a noisy environment, causing key information to be lost on the receiver side (human ears or intelligent assistants).
[0031] Speech enhancement (SE) systems have attempted to alleviate this problem by combining multiple microphone sensor signals to suppress background noise via spatial filtering or beamforming. These speech enhancement systems are usually front-end modules that pre-process the incoming audio stream, and these systems typically need to run in very low power computing and limited memory devices and platforms, where minimal resource consumption is desired.
[0032] Deep learning-based algorithms have been developed to improve beamforming performance, but existing methods have not yet satisfactorily solved the problem of speech denoising, especially in limited resource environments. Existing techniques based on deep learning and pure data-driven solutions, such as filter-and-Sum Network (FaSNet) technology, train speech denoising models in an end-to-end manner, which typically requires a large modeling capacity for properly learning the mapping of noise to clean signals from training data. However, such networks are not suitable for deployment on memory and power-constrained devices because they will require a large amount of (substantial) GPU / CPU use. In addition, direct model size reduction is not feasible because small networks will have difficulty learning such complex noise to clean (noisy-clean) signal mapping without special design. Existing techniques that use model-based signal processing and deep learning (data-driven) hybrid solutions use deep neural networks to assist signal processing-based beamforming algorithms, where signal and noise properties are mathematically modeled to approach solutions. These hybrid methods use prior knowledge of signal statistics to achieve smaller model sizes than pure deep learning methods. However, these methods train DNNs to estimate predefined ideal masks that are not based on multi-channel characteristics, but are based on monophonic speech enhancement solutions, such as estimating ideal binary masks (IBMs) or ideal ratio masks (IRMs). These predefined ideal masks lack a direct connection to the multi-channel power spectral density (PSD) estimates used to derive beamforming weights. That is, these predefined masks do not directly target minimization of the estimated signal PSD matrix to the true PSD, so their performance is not optimal.
[0033] The present disclosure provides a system and method for enhanced mask-based multi-channel speech enhancement performance, which uses an improved mask strategy, a learning protocol, and a PSD estimation scheme to more effectively utilize multi-channel signal characteristics for improved beamforming. In addition, the present disclosure provides an effective machine learning-based multi-channel speech enhancement system that can be applied to resource-constrained devices and can utilize a DNN method to assist a model-based statistical beamformer in pursuit of better denoising performance. The system and method of the present disclosure overcome the previously described problems and facilitate deployment on edge devices while obtaining satisfactory noise suppression performance by using a novel time-frequency (TF) mask framework of a mask estimation network or model based on an estimated snapshot matching mask, a training process for training the mask estimation network or model using a snapshot matching loss, and a complex-valued mask constraint that bounds the mask magnitude within the search space.
[0034] The speech enhancement system and method of the present disclosure allow for receiving noisy audio signals from multiple audio input devices. The noisy audio signal can be transformed into a noisy time-frequency representation using time-frequency operations. The noisy time-frequency representation is provided to a mask estimation model that is trained to output a mask for predicting a clean time-frequency representation of clean speech audio from the group of noisy audio signals. The mask output by the mask estimation model is used to determine beamforming filter weights for denoising the audio signal. The determined beamforming filter weights are applied to the noisy time-frequency representation to isolate the clean speech audio from the group of noisy audio signals, and an enhanced or clean speech audio signal can be output for further processing.
[0035] Note that while some of the embodiments discussed below are described in the context of use in a consumer electronic device (such as a smart phone), this is merely an example. It should be understood that the principles of the present disclosure can be implemented in any number of other suitable contexts and any suitable device or devices can be used. It should also be noted that while some of the embodiments discussed below are described based on the assumption that one device (such as a server) performs training of a machine learning model deployed to one or more other devices (such as one or more consumer electronic devices), this is merely an example. It should be understood that the principles of the present disclosure can be implemented using any number of devices, including a single device that both trains and uses a machine learning model. In general, the present disclosure is not limited to use with any particular type of device.
[0036] Figure 1 An example network configuration 100 including electronic devices according to the present disclosure is shown. Figure 1The embodiment of the network configuration 100 shown is for illustration only. Other embodiments of the network configuration 100 may be used without departing from the scope of the present disclosure.
[0037] According to an embodiment of the present disclosure, an electronic device 101 is included in a network configuration 100. The electronic device 101 may include at least one of the following: a bus 110, a processor 120, a memory 130, an input / output (I / O) interface 150, a display 160, a communication interface 170, or a sensor 180. In some embodiments, the electronic device 101 may exclude at least one of these components, or may add at least one other component. The bus 110 includes circuits for connecting components 120-180 to each other and for transmitting communications (such as control messages and / or data) between components.
[0038] The processor 120 includes one or more processing devices, such as one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs). In some embodiments, the processor 120 includes one or more of the following: a central processing unit (CPU), an application processor (AP), a communication processor (CP), or a graphics processor unit (GPU). The processor 120 is capable of controlling at least one other component of the electronic device 101 and / or performing operations or data processing related to communication or other functions. As described in more detail below, the processor 120 may perform various operations related to mask-based neural beamforming for multi-channel speech enhancement. For example, as described below, the processor 120 may receive and process inputs (such as audio inputs or data received from one or more audio input devices (such as one or more microphones)), and perform speech enhancement tasks using one or more machine learning models and inputs. The processor 120 may also instruct one or more other devices to perform certain operations (such as outputting audio using one or more audio output devices (such as one or more speakers)) or display content on one or more displays 160. Processor 120 may also receive inputs, such as data samples for training one or more machine learning models, and manage such training by inputting samples into the machine learning models, receiving outputs from the machine learning models, and performing learning functions, such as a loss function, to improve the machine learning models.
[0039] The memory 130 may include volatile and / or non-volatile memory. For example, the memory 130 may store commands or data related to at least one other component of the electronic device 101. According to an embodiment of the present disclosure, the memory 130 may store software and / or programs 140. The programs 140 include, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and / or an application program (or "application") 147. At least a portion of the kernel 141, the middleware 143, or the API 145 may be represented as an operating system (OS).
[0040] The kernel 141 may control or manage system resources (such as the bus 110, processor 120, or memory 130) for performing operations or functions implemented in other programs (such as the middleware 143, API 145, or application 147). The kernel 141 provides an interface that allows the middleware 143, API 145, or application 147 to access various components of the electronic device 101 to control or manage system resources. Application 147 may support various functions related to mask-based neural beamforming for multi-channel speech enhancement. These functions may be performed by a single application or multiple applications, each of which performs one or more of these functions. For example, the middleware 143 may be used as a repeater to allow the API 145 or application 147 to communicate data with the kernel 141. Multiple applications 147 may be provided. The middleware 143 is capable of controlling work requests received from the application 147, such as by assigning a priority to the use of the system resources (such as the bus 110, processor 120, or memory 130) of the electronic device 101 to at least one of the multiple applications 147. The API 145 is an interface that allows the application 147 to control functions provided from the kernel 141 or the middleware 143. For example, the API 145 includes at least one interface or function (such as a command) for filing control, window control, image processing, or text control.
[0041] The I / O interface 150 serves as an interface that can, for example, transmit commands or data input from a user or other external devices to other components of the electronic device 101. The I / O interface 150 can also output commands or data received from other components of the electronic device 101 to the user or other external devices.
[0042] The display 160 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. The display 160 may also be a depth perception display, such as a multi-focal display. The display 160 is capable of displaying, for example, various contents (such as text, images, videos, icons, or symbols) to the user. The display 160 may include a touch screen and may receive, for example, a touch, gesture, proximity, or hover input using an electronic pen or a body part of the user.
[0043] For example, the communication interface 170 can establish communication between the electronic device 101 and an external electronic device (such as the first electronic device 102, the second electronic device 104, or the server 106). For example, the communication interface 170 can be connected to the network 162 or 164 through wireless or wired communication to communicate with the external electronic device. The communication interface 170 can be a wired or wireless transceiver or any other component for sending and receiving signals.
[0044] Wireless communication can use, for example, WiFi, long term evolution (LTE), long term evolution-advanced (LTE-A), fifth generation wireless system (5G), millimeter wave or 60 GHz wireless communication, wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro) or global system for mobile communications (GSM) as a communication protocol. Wired connection can include, for example, at least one of universal serial bus (USB), high-definition multimedia interface (HDMI), recommended standard 232 (RS-232) or plain old telephone service (POTS). Network 162 or 164 includes at least one communication network, such as a computer network (such as a local area network (LAN) or a wide area network (WAN)), the Internet or a telephone network.
[0045] The electronic device 101 also includes one or more sensors 180, which can measure physical quantities or detect the activation state of the electronic device 101, and convert the measured or detected information into electrical signals. For example, one or more sensors 180 may include one or more microphones or other audio sensors for capturing audio data, or one or more cameras or other imaging sensors for capturing images of scenes. The sensor 180 may also include one or more buttons for touch input, gesture sensors, gyroscopes or gyroscope sensors, air pressure sensors, magnetic sensors or magnetometers, acceleration sensors or accelerometers, grip sensors, proximity sensors, color sensors (such as red, green, and blue (RGB) sensors), biophysical sensors, temperature sensors, humidity sensors, illumination sensors, ultraviolet (UV) sensors, electromyography (EMG) sensors, electroencephalography (EEG) sensors, electrocardiography (ECG) sensors, infrared (IR) sensors, ultrasonic sensors, iris sensors, or fingerprint sensors. The sensor 180 may also include an inertial measurement unit, which may include one or more accelerometers, gyroscopes, and other components. In addition, the sensor 180 may include a control circuit for controlling at least one sensor included here. Any of these sensors 180 may be located within the electronic device 101 .
[0046] In some embodiments, the first external electronic device 102 or the second external electronic device 104 may be a wearable device or a wearable device (such as an HMD) in which the electronic device can be installed. When the electronic device 101 is installed in the electronic device 102 (such as an HMD), the electronic device 101 may communicate with the electronic device 102 through the communication interface 170. The electronic device 101 may be directly connected to the electronic device 102 to communicate with the electronic device 102 without involving a separate network. The electronic device 101 may also be an augmented reality wearable device including one or more imaging sensors, such as glasses.
[0047] The first external electronic device 102 and the second external electronic device 104 and the server 106 can each be a device of the same or different type as the electronic device 101. According to certain embodiments of the present disclosure, the server 106 includes a group of one or more servers. In addition, according to certain embodiments of the present disclosure, all or some of the operations performed on the electronic device 101 can be performed on another one or more other electronic devices (such as the electronic devices 102 and 104 or the server 106). In addition, according to certain embodiments of the present disclosure, when the electronic device 101 should automatically or upon request perform some functions or services, the electronic device 101 can request another device (such as the electronic devices 102 and 104 or the server 106) to perform at least some functions associated with it, rather than performing the functions or services by itself or in addition. Other electronic devices (such as the electronic devices 102 and 104 or the server 106) are capable of performing the requested functions or additional functions and transmitting the results of the execution to the electronic device 101. The electronic device 101 can provide the requested functions or services by processing the received results as is or in addition. To this end, for example, cloud computing, distributed computing, or client-server computing technology can be used. Although Figure 1 The electronic device 101 is shown to include a communication interface 170 to communicate with an external electronic device 104 or a server 106 via a network 162 or 164 , but according to some embodiments of the present disclosure, the electronic device 101 may operate independently without a separate communication function.
[0048] The server 106 may include components 110-180 (or a suitable subset thereof) that are the same or similar to the electronic device 101. The server 106 may support driving the electronic device 101 by performing at least one operation (or function) implemented on the electronic device 101. For example, the server 106 may include a processing module or processor that may support the processor 120 implemented in the electronic device 101. As described in more detail below, the server 106 may perform various operations related to mask-based neural beamforming for multi-channel speech enhancement. For example, as described below, the server 106 may receive and process inputs (such as audio inputs or data received from one or more audio input devices (such as one or more microphones)), and perform speech enhancement tasks using one or more machine learning models and inputs. The server 106 may also instruct one or more other devices to perform certain operations (such as outputting audio using one or more audio output devices (such as one or more speakers)), or display content on one or more displays 160. Server 106 may also receive inputs, such as data samples to be used to train one or more machine learning models, and manage such training by inputting samples into the machine learning models, receiving outputs from the machine learning models, and performing learning functions, such as a loss function, to improve the machine learning models.
[0049] although Figure 1 An example of a network configuration 100 including an electronic device 101 is shown, but the Figure 1 Various changes may be made. For example, network configuration 100 may include any number of each component in any suitable arrangement. In general, computing and communication systems have a wide variety of configurations, and Figure 1 The scope of the present disclosure is not limited to any particular configuration. Figure 1 One operating environment is shown in which the various features disclosed in this patent document may be used, but these features may be used in any other suitable system.
[0050] Figure 2 An example speech enhancement system 200 according to the present disclosure is shown. For ease of explanation, the system 200 is described as involving Figure 1 The electronic device 101 is used in the network configuration 100 of FIG. However, the system 200 may be used with any other suitable electronic device (such as the server 106), and in any other suitable system.
[0051] like Figure 2As shown, the speech enhancement system 200 can be deployed on and executed by an electronic device such as the electronic device 101. The electronic device 101 includes a multi-channel speech enhancement system 202, a receiver 204, and a plurality of audio input devices 206 (such as microphones). The audio input is captured by the audio input device 206 and provided as an input to the multi-channel speech enhancement system 202. The multi-channel speech enhancement system 202 is configured to receive an audio signal including speech, which may have a certain level of noise or interference caused by the noisy environment in which the electronic device 101 operates. As described in more detail below, the multi-channel speech enhancement system 202 performs various operations on noisy multi-channel audio data, such as by performing time-frequency analysis on the audio data, estimating a snapshot matching mask, determining beamforming weights using the snapshot matching mask, applying the beamforming weights to the noisy audio input so as to denoise the audio signal and create an enhanced audio signal, and converting the enhanced audio signal back to the time domain.
[0052] The multi-channel speech enhancement system 202 outputs the enhanced audio signal to a receiver 204. The receiver 204 may be, for example, a receiving audio output device (such as a speaker) that outputs the enhanced audio signal to a user, or an intelligent assistant that receives the enhanced audio signal for further processing, such as performing wake-up word detection, automatic speech recognition (ASR) tasks, and / or other associated operations of the electronic device. It should be understood that the enhanced audio signal created using the multi-channel speech enhancement system 202 and the processes described in the present disclosure may be used for a variety of other purposes.
[0053] although Figure 2 One example of a speech enhancement system 200 is shown, but may be used for Figure 2 Various changes may be made. For example, the receiver 204 and the audio input device 206 may be connected to a processor (such as the processor 120) within the electronic device 101, such as via a wired connection or circuit. In some embodiments, the receiver 204 and the audio input device 206 may be external to the electronic device 101 and connected via a wired or wireless connection. Moreover, in some embodiments, the multi-channel speech enhancement system 202 may be stored away from the electronic device 101, such as on a server 106. Here, the electronic device 101 may transmit a request including an input (such as captured audio data) to the server 106 for processing the input using the multi-channel speech enhancement system 202, and the result may be sent back to the electronic device 101. In other embodiments, the electronic device 101 may be a server 106 that receives audio input from a client device and transmits instructions back to the client device to perform functions associated with instructions included in an utterance.
[0054] Figure 3Another example speech enhancement system 300 according to the present disclosure is shown. For ease of explanation, the system 300 is described as involving Figure 1 The electronic device 101 is used in the network configuration 100 of FIG. However, the system 300 may be used with any other suitable electronic device (such as the server 106), and in any other suitable system.
[0055] like Figure 3 As shown, the system 300 includes an electronic device 101, which includes a processor 120. The processor 120 is operatively coupled to or otherwise configured to use one or more machine learning models, such as a mask estimation model 302. The mask estimation model 302 can be trained to predict a snapshot matching mask (SMM) from a noisy audio signal received via a multi-channel audio input device 306 (such as a microphone). The SMM is used to reduce or minimize the distance between the predicted signal snapshot and the true signal snapshot during training, resulting in a systematic method of estimating speech and noise power spectral density (PSD) matrices, which are used to derive beamformer weights during inferencing.
[0056] The system 300 also includes a statistical beamformer 304 that includes the PSD matrix updated during the SMM-based derivation. The processor 120 can apply these updated beamformer weights to the noisy audio signal to suppress noise within the noisy audio signal and provide an enhanced audio signal for further use by the electronic device 101 or other device. In some cases, the mask estimation model 302 and the statistical beamformer 304 can represent the above description of Figure 2 At least a portion of the multi-channel speech enhancement system 202 described herein.
[0057] The processor 120 may also be operatively coupled to or otherwise configured to use one or more other models 305, such as one or more models associated with a voice assistant or other process, such as one or more ASR models and / or one or more natural language understanding (NLU) models. It should be understood that the machine learning models 302-305 may be stored in a memory of the electronic device 101 (such as memory 130) and accessed by the processor 120 to perform multi-channel speech enhancement and / or other tasks. However, the machine learning models 302-305 may be stored in any other suitable manner.
[0058] The system 300 also includes an audio output device 308 (such as a speaker or headphones) and a display 310 (such as a screen or monitor, such as the display 160). The processor 120 provides the output enhanced audio signal provided by the statistical beamformer 304 using the predicted SMM to a receiver (such as the audio output device 308 or one or more other models 305). For example, the enhanced or clean audio signal can be output via the audio output device 308 to provide a clearer audio output to the user. As another example, the clean audio signal can be provided to the intelligent assistant for further audio / language processing, such as by using one or more other models 305 to perform one or more tasks, such as wake-up word detection, ASR, NLU, etc., and / or other associated operations of the electronic device 101.
[0059] As a specific example, based on the clean audio signal, the processor 120 may indicate one or more further actions corresponding to one or more instructions or requests provided in the speech included in the clean audio signal. For example, assume that a speech (such as "hey BIXBY, call mom") is received from a user via an audio input device 306 as a noisy audio signal. Here, the trained mask estimation model 302 and the statistical beamformer 304 are used to denoise the audio signal and provide an enhanced audio signal, which allows the voice assistant to use other models 305 to improve voice detection accuracy. Based on the speech, the processor 120 can instruct the audio output device 308 to output "calling Mom", and the processor 120 can cause a phone application or other communication application to start a communication session with a "mom" contact stored on the electronic device 101 or otherwise associated with the user of the electronic device 101. As another example, assume that the speech "hey BIXBY, start a timer" is received. Here, the mask estimation model 302 and the statistical beamformer 304 are used to denoise the audio signal so that the voice assistant using the other model 305 can perform accurate voice detection, and the processor 120 can instruct the execution of the timer application and the display of the timer on the display 310 of the electronic device 101.
[0060] although Figure 3 Another example of a speech enhancement system 300 is shown, but may be used for Figure 3Various changes may be made. For example, the audio input device 306, the audio output device 308, and the display 310 may be connected to the processor 120 within the electronic device 101, such as via a wired connection or circuit. In other embodiments, the audio input device 306, the audio output device 308, and the display 310 may be external to the electronic device 101 and connected via a wired or wireless connection. In addition, in some embodiments, one or more models including the mask estimation model 302, the statistical beamformer 304, and the other models 305 may be stored away from the electronic device 101, such as on a server 106. Here, the electronic device 101 may send a request including an input (such as captured audio data) to the server 106 for processing the input using the mask estimation model 302 and the statistical beamformer 304, and the result may be sent back to the electronic device 101. In other embodiments, the electronic device 101 may be a server 106 that receives audio input from a client device and sends instructions back to the client device to perform a function associated with the instructions included in the utterance. Additionally, in some cases, the mask estimation model 302 and the statistical beamformer 304 and one or more other machine learning models 305 may be stored as separate models that are called by the processor 120 to perform certain tasks, or may be included in and form part of one or more larger machine learning models.
[0061] Figure 4 An example multi-channel speech enhancement architecture 400 according to the present disclosure is shown. For ease of explanation, Figure 4 The architecture 400 shown in FIG. 4 is described as Figure 1 100 is implemented on or supported by the server 106. However, Figure 4 The architecture 400 shown in FIG. 4 may be used with any other suitable device and in any other suitable system, such as when at least a portion of the architecture 400 is implemented on or supported by the electronic device 101 .
[0062] like Figure 4 As shown, architecture 400 includes a mask estimation network 402, which in some cases can represent the above Figure 3The mask estimation model 302 described above. The mask estimation network 402 receives as input a noisy time-frequency representation generated by performing a time-frequency operation on a noisy audio signal received via a plurality of audio input devices, such as a short-time Fourier transform, a Mel Spectrogram, a Mel Frequency Cepstral Coefficient (MFCC), and / or other methods. As a result, the noisy audio signal received as a time domain waveform via the audio input device is transformed into a complex x-valued spectrogram. In various embodiments, an additive noise model may be used. For example, let and denote the frequency index and the time frame index (the sum of the bin and the frame), respectively, where the noise signal TF of the microphone array is transformed x i (f, t) can be expressed as follows.
[0063] X i (f, t) = S i (f, t)-V i (f, t), (1)
[0064] here Where S i (f, t) and V i (f, t) are the speech and noise components received by the i-th microphone, respectively.
[0065] exist Figure 4 In the example shown, the noisy signal is converted to the time-frequency domain using a short-time Fourier transform, and the mask estimation network 402 receives a plurality of noisy signal short-time Fourier transforms (STFTs) 401 produced by the transform. In some cases, the noisy signal STFTs 401 may be provided as input in the form of at least one vector of microphone signals, such as a noisy signal snapshot 403 (x(f, t)). As a specific example, the noisy signal snapshot 403 may be represented as follows.
[0066]
[0067] In some embodiments, the mask estimation network 402 can be a deep neural network (DNN) model. The mask estimation network 402 may include multiple complex convolutional layers for performing multiple convolutions to downsample and upsample input data. In addition, in some embodiments, a complex leaky rectified linear unit (ReLU) operation may be performed as an activation function in the DNN model. The mask estimation network 402, using the noisy input signal STFT401, estimates the TF mask 405 (M), referred to as a snapshot matching mask (SMM) in the present disclosure. When the mask estimation network 402 is implemented using a DNN model or includes a DNN model, any suitable DNN model may be used, such as a complex U-NET model. As a specific example, the various techniques described in the present disclosure may be used in conjunction with the techniques described in Lee et al., “Improved Mask-Based Neural Beamforming for Multichannel Speech Enhancement by Snapshot Matching Masking,” IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), June 2023 (available at https: / / ieeexplore.ieee.org / document / 10096213), which is incorporated herein by reference in its entirety.
[0068] like Figure 4 As shown, the search space can be constrained, such as by using Tanh nonlinearity, to limit the magnitude of the output of the final layer of the mask estimation network 402. Figure 5 , which illustrates an example snapshot matching mask estimation process 500 according to the present disclosure, showing a more detailed view of the mask estimation network 402. Figure 5 As shown, the mask estimation network 402 includes a main DNN model 502 and an amplitude Tanh limiting operation 504. The main DNN model 502 may correspond to Figure 4 The structure of the mask estimation network 402 is shown. In this embodiment, the SMM 405 is configured to estimate the likelihood that the input audio includes speech, where a value closer to one indicates that the audio may include speech, and a value closer to zero indicates that the audio may not include speech. This can be expressed as follows.
[0069]
[0070] In various embodiments, the amplitude Tanh limit operation 504 is a nonlinear module imposed on the DNN output O to limit its amplitude to within the unit circle on the complex plane, such as to ensure that its amplitude is less than or equal to 1. As described above, the amplitude of the mask can therefore be intuitively interpreted as a speech presence probability (SPP) to indicate the likelihood that a speech component is present in each TF bin, where the probability ranges from zero to one and can be reflected by an amplitude less than or equal to one. The amplitude Tanh limit operation 504 applies a Tanh nonlinearity to the amplitude of the network output while keeping the phase unchanged. Incorporating a complex-valued masking constraint for the amplitude limit of the output SMM 405 (M) to explore complex domain representations provides increased modeling flexibility while also alleviating optimization difficulty by limiting the masked amplitude to a suitable search space. This is because incorporating the amplitude Tanh limit operation 504 avoids the challenge of optimizing from an infinite unbounded search space, which is particularly helpful when processing multi-channel data (because the data is more complex than mono data). Compared to existing masking methods such as ideal binary mask (IBM) and ideal ratio mask (IRM), SMM uses complex-valued representation. Therefore, compared to IBM and IRM that use real-valued masks, SMM can also manipulate the phase component of the signal snapshot to better exploit the spatial characteristics of multi-channel signals.
[0071] although Figure 5 One example of a snapshot matching mask estimation process 500 is shown, but may be used for Figure 5 Make various changes. For example, Figure 5 The various components and functions in the embodiment can be combined, further subdivided, rearranged, duplicated or omitted according to specific needs. In addition, if necessary or desired, one or more additional components and functions can be included.
[0072] Reference again Figure 4 During training, SMM 405 is used to reduce or minimize the distance between the estimated signal snapshot and the clean signal snapshot 409 (s(f, t)), providing a systematic method for estimating the speech and noise power spectral density (PSD) matrices used to derive beamformer weights during derivation. In some cases, the clean signal snapshot 409 can be represented as follows.
[0073]
[0074] In various embodiments, the clean signal snapshots 409 and the associated clean speech STFTs 411 are included in the training data samples of the training data set. The training data set also includes multiple sets of sample noisy signal STFTs 401 to be used as inputs to the mask estimation network 402, where the sample noisy signal STFTs 401 are associated with the corresponding clean speech STFTs. During training, the SMM 405 output by the mask estimation network 402 is combined with the noisy signal snapshots 403, such as via an element-wise multiplication applied to all channels, and the resulting output is provided to the snapshot matching loss operation 407. Figure 6 , which shows an example mask estimation model training process 600 according to the present disclosure, the output generated by combining the SMM 405 with the noisy signal snapshot 403 includes an estimated signal snapshot 602.
[0075] like Figure 4 and Figure 6 As shown, the snapshot matching loss operation 407 receives both the clean speech snapshot 409 and the estimated signal snapshot 602, and the snapshot matching loss operation 407 reduces or minimizes the distance between the true signal snapshot 409 (given by the clean speech STFT 409) and the estimated signal snapshot 602 (given by multiplying the SMM 405 with the noisy signal snapshot 403). Again, f and t represent the frequency index and the time frame index, respectively. The minimization of the distance between the clean signal snapshot 409 and the estimated signal snapshot 602 of the SMM 405 can be expressed as follows.
[0076]
[0077] here, Where x(f, t) is a noisy signal snapshot 402, s(f, t) is a clean signal snapshot 409, M s (f, t) is the estimated SMM 405, and is the clean speech snapshot 409 (s(f, t)) and the estimated signal snapshot 602 is a measure of the difference between By matching snapshots, the multi-channel speech enhancement system can make the estimated signal snapshot 602 close to the clean signal snapshot 409. To estimate speech PSD more accurately
[0078] The snapshot matching loss operation 407 determines an error or loss using a loss function and modifies the mask estimation network 402 based on the error or loss. That is, the snapshot matching loss operation 407 uses a loss function to calculate the error or loss associated with the estimated SMM 405 provided by the mask estimation network 402. For example, when the estimated signal snapshot created using the predicted SMM is different from the ground truth, the difference can be used to calculate the loss defined by the loss function. In various embodiments, the loss function can use any suitable measure of the loss associated with the output generated by the mask estimation network 402, such as a cross entropy loss or a mean square error. In some embodiments, the loss function can be based on a combined power-law compressed mean square error (MSE) criterion applied to the SMM 405 and the clean speech snapshot 409, where the combined power-law compressed MSE criterion can be expressed as follows.
[0079]
[0080] Therefore, in various embodiments, the loss function can be expressed as follows.
[0081]
[0082] here, in And therefore, For i=1,...,N.
[0083] It has been found that the above loss function based on the combined power-law compression MSE can perform better than other loss functions (such as standard mean square error and mean absolute error loss). Among other reasons, this may be due to the fact that the combined power-law compression MSE loss function takes into account the speech time-frequency characteristics by compressing the amplitude part, which is suitable for the dynamic range of the speech intensity distribution for multi-channel signal (snapshot) estimation purposes. However, any other suitable loss function can be used here.
[0084] When the loss calculated by the loss function is greater than expected, the parameters of the mask estimation network 402 can be adjusted. Once adjusted, the same or additional training data can be provided to the mask estimation network 402, and additional outputs (estimated signal snapshots 602 created from the combination of the output SMM 405 and the noisy signal snapshot 403) can be compared with the ground truth (clean signal snapshot 409) so that the loss function can be used to determine the additional loss. Over time, the mask estimation network produces more accurate outputs that more closely match the ground truth, and the measured loss becomes less. Once the measured loss drops below a specified threshold, the training of the mask estimation network 402 can be completed. Unlike existing methods that use only the information of a single selected reference channel as a target during training by matching the estimated speech mask with an ideal mask calculated from clean and noisy speech signals from a single channel (the basic principle is that the remaining channels are considered redundant), the training of the mask estimation network 402 of the present disclosure can consider all audio channels to be equally important for training the network to generate SMM 405, resulting in more accurate and significantly increased noise reduction.
[0085] although Figure 6 An example of a mask estimation model training process 600 is shown, but may be used for Figure 6 Make various changes. For example, Figure 6 The various components and functions in the can be combined, further subdivided, rearranged, duplicated or omitted according to specific needs. In addition, if necessary or desired, one or more additional components and functions can be included.
[0086] Reference again Figure 4 During derivation, the trained mask estimation network 402 outputs an estimated speech TF mask (SMM) 405, which is provided to the statistical beamformer 404 to update the signal PSD matrix of the statistical beamformer 404. The updated signal PSD matrix is used to obtain the filter weights to be applied to the noisy input signal in order to denoise the noisy input signal and obtain an enhanced or clean speech signal. Figure 7 , which shows an example audio signal denoising process 700 according to the present disclosure, a noise signal snapshot 403 and an estimated SMM 405 output by a mask estimation network 402 are provided to a statistical beamformer 404. The present disclosure provides a PSD update scheme based on SMM for obtaining beamforming filter weights to be applied to the noise signal to suppress noise. In some cases, the beamforming filter weights may be expressed as follows.
[0087]
[0088] Here, θ is the closed-form formula for calculating the beamforming filter coefficients, and Φ s(f, t) and Φ v (f, t) are the speech and noise PSD estimates. The SMM 405 provided by the mask estimation network 402 is used to perform SMM-based speech PSD matrix updates and SMM-based noise PSD matrix updates.
[0089] In some cases, the SMM-based speech PSD matrix update can be expressed as follows.
[0090]
[0091] Furthermore, in some cases, the SMM-based noise PSD matrix update can be expressed as follows.
[0092] Φ v (f, t) = λ v Φ v (f, t-1)-(1-|M z (f, t)| 2 )x(f,t)| 2 )x(f,t)x H (f,t) (10)
[0093] Here, λ s ,λ v ∈〔0,1] is the forgetting factor that controls the smoothness of speech and noise PSD estimates respectively.
[0094] As shown above, Φ can be estimated based on the speech PSD s (f, t) and noise PSD estimate Φ v (f, t) to determine the beamforming filter weights w(f, t), where both speech and noise PSD estimates use the SMM 405 (M) provided by the mask estimation network 402. s Once the filter weights w(f, t) are determined, the filter weights w(f, t) are applied to the noisy signal provided in the noisy signal snapshot 403 to obtain the enhanced speech signal 702. In some cases, applying the filter weights to the noisy signal to obtain the enhanced speech signal 702 can be represented as follows.
[0095]
[0096] Once the enhanced speech signal 702 is obtained, the enhanced speech signal 702 can be converted back to the time domain and provided as a system output for use in other downstream devices and / or application processes, such as sending the enhanced speech signal 702 to an audio output device or providing the enhanced speech signal 702 to other applications (such as voice assistants and their related processes).
[0097] although Figure 7 An example of an audio signal denoising process 700 is shown, but the Figure 7 Make various changes. For example, Figure 7 The various components and functions in the can be combined, further subdivided, rearranged, duplicated, or omitted according to specific needs. In addition, one or more additional components and functions can be included if needed or desired. Although Figure 4 An example of a multi-channel speech enhancement architecture 400 is shown, but may be used for Figure 4 Make various changes. For example, Figure 4 The various components and functions in the embodiment can be combined, further subdivided, rearranged, duplicated or omitted according to specific needs. In addition, if necessary or desired, one or more additional components and functions can be included.
[0098] Although the mask estimation network 402 is described above as having a DNN model architecture, the SMM framework and training process of the present disclosure can adopt various architectures for TF mask estimation operations, such as various convolutional neural network (CNN) architectures, recurrent neural network (RNN) architectures, U-NET architectures, RESNET architectures, transformer architectures, etc., to allow adaptation to different device specifications. In addition, the SMM-based PSD update process of the present disclosure can be utilized by any statistical beamforming algorithm operating in the TF domain, such as a multi-channel Wiener filter (MWF) beamformer, a minimum variance distortionless response (MVDR) beamformer, etc.
[0099] It should be noted that Figures 4 to 7 The functions shown or described above may be implemented in any suitable manner in the electronic device 101, 102, 104, the server 106, or other devices. For example, in some embodiments, one or more software applications or other software instructions executed by the processor 120 of the electronic device 101, 102, 104, the server 106, or other devices may be used to implement or support Figures 4 to 7 In other embodiments, dedicated hardware components may be used to implement or support at least some of the functions shown or described above. Figures 4 to 7 At least some of the functions shown in or described above. In general, any suitable hardware or any suitable combination of hardware and software / firmware instructions may be used to implement Figures 4 to 7 In addition, Figures 4 to 7 The functions shown or described above may be performed by a single device or by multiple devices. For example, server 106 may be used to train mask estimation network 402, such as based on Figures 4 to 6The server 106 can deploy the trained mask estimation network 402 to one or more other devices (such as the electronic device 101) for use, such as performing SMM prediction based on the received audio signal, performing PSD matrix updates to obtain beamforming filter weights, and using the obtained beamforming filter weights to denoise the audio signal, as described with respect to Figure 4 , 5 and 7 as shown and described.
[0100] Figure 8 An example method 800 for training a mask estimation model according to the present disclosure is shown. For ease of explanation, Figure 8 The method 800 shown in FIG. 8 is described as using Figure 1 The method 800 may be performed by the server 106 in the network configuration 100. As a specific example, the method 800 may be performed by the server 106 in the network configuration 100. Figure 1 The method 800 is executed on the server 106 in the network configuration 100, and one or more trained machine learning models (such as the trained mask estimation model 302 and / or the trained mask estimation network 402) can be deployed to the client electronic device 101 for use. However, the method 800 can be used with any other suitable device (such as the electronic device 101) and can be used in any other suitable system.
[0101] like Figure 8 As shown, at box 802, noisy and clean audio data samples are obtained for supervised learning purposes to train a mask estimation model. The data samples can be obtained in any suitable manner, such as by recording real-world audio using a target device, synthesizing data using some room acoustic simulation software, or using other methods. At box 804, time-frequency analysis is performed on the data samples to transform the data samples from the time domain to a spectral-temporal representation. This can include, for example, a processor 120 of the server 106, performing time-frequency operations on the data samples, such as STFT, spectrogram, MFCC, and / or other time-frequency operations.
[0102] At block 806, mask estimation model training using the training data samples begins. As described in the present invention, the mask estimation model is trained to accurately predict a snapshot matching mask (SMM), which is used to update a beamforming power spectral density (PSD) matrix during deployment to obtain beamforming weights for enhancing or cleaning noisy audio signals. At block 808, a snapshot matching mask is obtained from the mask estimation model using the noisy snapshot as input. This may include, for example, the processor 120 of the server 106 executing an untrained mask estimation model to predict the SMM from the noisy training audio samples, as described above with respect to Figures 4 to 6 An example thereof is described.
[0103] At block 810, the SMM output by the mask estimation model is combined with the noisy snapshot audio signal used to create the SMM to obtain an estimated signal snapshot. This may include, for example, the processor 120 of the server 106 performing an element-wise complex multiplication applied to all channels using the SMM and the noise signal, as described above with respect to Figures 4 to 6 An example of which is described in detail in FIG. 8A . At block 812, a snapshot matching loss operation is performed using the clean signal snapshot from the training data sample and the estimated signal snapshot obtained at block 810. This may include, for example, the processor 120 of the server 106 executing a matching loss function, such as the matching loss function 407, using a suitable loss measure associated with the output generated by the mask estimation network 402. The matching loss function 407 may use any suitable loss, such as a cross entropy loss, a mean squared error, or a combined power law compression MSE criterion, examples of which are described above with respect to Figures 4 to 6 Description. The snapshot matching loss operation reduces or minimizes the distance between the true signal snapshot and the estimated signal snapshot.
[0104] At decision block 814, a determination is made as to whether training of the mask estimation model is complete, such as based on whether the measured loss is below a threshold. If not, the method 800 moves to block 816. At block 816, the mask estimation model is updated based on the result of the snapshot matching loss function. This may include, for example, the processor 120 of the server 106, modifying the mask estimation model based on a calculated error or loss associated with the estimated SMM provided by the mask estimation model, as described above with respect to Figures 4 to 6 An example thereof is described. For example, when the estimated signal snapshot created using the predicted SMM is different from the ground truth, the difference can be used to calculate the loss defined by the loss function. When the loss calculated by the loss function is greater than the expectation, the parameters of the mask estimation model can be adjusted. Once adjusted, method 800 moves back to box 808 to continue training the mask estimation model using the training data sample. The same or additional training data can be provided to the mask estimation model for further training, and the additional output (the estimated signal snapshot created from the combination of the output SMM and the noisy signal snapshot) can be compared with the ground truth (the clean signal snapshot) so that the loss function can be used to determine the additional loss. Method 800 can include any number of training sessions, so that the additional loss can be determined using the loss function, and the mask estimation model can be further adjusted.
[0105] Over time, the mask estimation model produces a more accurate output that more closely matches the ground truth, and the measured loss becomes less. Once the measured loss drops below a specified threshold, the training of the mask estimation model can be completed, and the process ends at box 818. Once it is determined that the training of the mask estimation model is complete, the mask estimation model can be deployed to, for example, an electronic device 101. Unlike existing methods that use only the information of a single selected reference channel as a target during training by matching the estimated speech mask with an ideal mask calculated from clean and noisy speech signals from a single channel, the training of the mask estimation model here can consider all microphone channels to be equally important for training the network to generate SMMs, thereby producing more accurate and significantly increased noise reduction during live derivation.
[0106] although Figure 8 An example of a method 800 for training a mask estimation model is shown, but may be used for Figure 8 For example, although shown as a series of steps, Figure 8 The various steps in may overlap, occur in parallel, occur in a different order, or occur any number of times (including zero).
[0107] Fig. 9 An example method 900 for denoising an audio signal according to the present disclosure is shown. For ease of explanation, Fig. 9 The method 900 shown in FIG. 1 is described as using Figure 1 The method 900 may be performed by the electronic device 101 in the network configuration 100. However, the method 900 may be performed using any other suitable device (such as the server 106) and in any other suitable system.
[0108] At block 902, noisy multi-channel audio data is obtained from a set of audio input devices. This may include, for example, a processor 120 of the electronic device 101, receiving a noisy audio waveform recorded by the audio input device, which may be received during a first time window. At block 904, a time-frequency analysis is performed on the noisy audio data. This may include, for example, the processor 120 of the electronic device 101, performing a time-frequency operation, such as an STFT, a spectrogram, MFCC, and / or other time-frequency operation, on the original recorded audio data to generate a noisy time-frequency representation of the audio data, as described above with respect to Figure 4 , Figure 5 and Figure 7 An example of which is described. In various embodiments, the particular time-frequency analysis performed at block 904 matches the time-frequency analysis used during training of the mask estimation model.
[0109] At block 906, a snapshot matching mask (SMM) is estimated using at least one trained mask estimation model, such as the mask estimation model 302 and / or the mask estimation network 402. This may include, for example, the processor 120 of the electronic device 101 providing the noisy time-frequency representation as an input to the mask estimation model, wherein the mask estimation model is trained to output the SMM. The SMM is used to predict a clean time-frequency representation of the clean speech audio from the noisy time-frequency representation, as described above with respect to Figure 4 , 5 and 7 describe examples thereof.
[0110] At block 908, beamforming weights are determined based on a multi-channel power spectral density (PSD) matrix, wherein the PSD matrix is updated using the SMM output by the mask estimation model. This may include, for example, the processor 120 of the electronic device 101, updating the PSD matrix using the SMM output by the mask estimation model and obtaining beamforming filter weights from the matrix, as described above with respect to Figure 4 , Figure 5 and Figure 7 An example thereof is described. The PSD matrix may include a first PSD matrix corresponding to speech audio and a second PSD matrix corresponding to noise audio. In various embodiments, the speech audio corresponding to the first PSD matrix and the noise audio corresponding to the second PSD matrix may be from a second time window prior to the first time window in which the noisy audio input is received.
[0111] At block 910, the determined beamforming filter weights are applied to the noisy multi-channel audio data in the TF domain to denoise the audio signal and create an enhanced audio signal. This may include, for example, the processor 120 of the electronic device 101 using the determined beamforming weights on the noisy time-frequency representation to isolate the clean speech audio from the set of noisy audio signals, as described above with respect to Figure 4 , 5 Examples of this are described in and 7. At block 912, the enhanced audio signal obtained at block 910 is converted from the TF domain back to the time domain. This may include, for example, the processor 120 of the electronic device 101 reversing the time-frequency operation performed at block 904.
[0112] At box 914, the resulting clean speech audio signal in the time domain is output by the speech enhancement system and can be used by one or more other devices, processes or applications. For example, the clean speech audio can be output by an audio output device, such as a local device speaker, or a remote speaker of a device that receives a transmission including clean speech audio during a voice call or message. As another example, the clean speech audio signal can be used by other machine learning processes, such as one or more voice assistance processes and / or ASR / NLU processes. For example, the clean speech audio signal can be used to determine whether the speech includes a device command, such as a command to call a contact, play music, change settings in an IoT device, etc. It should be understood that the speech enhancement system and method of the present disclosure can be used in various embodiments to perform speech and audio denoising on edge devices and mobile platforms with multiple microphones, such as refrigerators, mobile phones, vacuum cleaners, smart watches, AR / VR glasses, earplugs, smart TVs, etc. The speech enhancement system and method of the present disclosure can also be used in various embodiments as a pre-processing unit for voice control, wake-up word detection, automatic speech recognition, audio anomaly detection, acoustic scene classification, and assisted listening to improve the human auditory experience in noisy environments. Therefore, the speech enhancement system and method of the present disclosure can benefit various intelligent applications. It has been found that the speech enhancement system and method of the present disclosure provide improved speech quality over different numbers of microphones compared to existing methods (including other mask-based methods). Method 900 ends at block 916.
[0113] although Fig. 9 An example of a method 900 for denoising an audio signal is shown, but the Fig. 9 For example, although shown as a series of steps, Fig. 9 The various steps in may overlap, occur in parallel, occur in a different order, or occur any number of times (including zero).
[0114] Although the present disclosure has been described with reference to various exemplary embodiments, various changes and modifications may be suggested to one skilled in the art. The present disclosure is intended to encompass such changes and modifications as fall within the scope of the appended claims.
Claims
1. A method, include: receiving a set of noisy audio signals from a plurality of audio input devices during a first time window; generating a noisy time-frequency representation based on the set of noisy frequency signals; providing the noisy time-frequency representation as input to a mask estimation model, the mask estimation model being trained to output a mask for predicting a clean time-frequency representation of clean speech audio from the noisy time-frequency representation; determining beamforming filter weights based on the mask; applying beamforming filter weights to the noisy time-frequency representation to isolate clean speech audio from the set of noisy frequency signals; and Output clean voice audio.
2. The method according to claim 1, in, The beamforming filter weights include a first power spectral density (PSD) matrix corresponding to speech audio and a second PSD matrix corresponding to noise audio.
3. The method according to claim 2, in, The speech audio corresponding to the first PSD matrix and the noise audio corresponding to the second PSD matrix are from a second time window preceding the first time window.
4. The method according to claim 2, further comprising: include: The first PSD matrix and the second PSD matrix are updated using the mask.
5. The method according to claim 1, in: Generating a noisy time-frequency representation includes performing a time-frequency analysis on the set of noisy frequency signals in the time domain; and The method also includes converting the clean speech audio back to the time domain before outputting the clean speech audio.
6. The method according to claim 1, in: The mask estimation model is trained using a training data set including multiple sets of sample noisy frequency signals; Each set of noisy audio signals is associated with a clean time-frequency representation of speech audio in the sample noisy audio signals and a noisy time-frequency representation of the sample noisy audio signals; The difference in amplitude and phase between the clean time-frequency representation and the predicted clean time-frequency representation is minimized; as well as The clean time-frequency representation and the predicted clean time-frequency representation are determined via applying the mask output by the mask estimation model to the noisy time-frequency representation.
7. The method according to claim 1, in, The mask estimation model is trained to output a mask with a magnitude within a unit circle on the complex plane.
8. An electronic device, include: At least one processing device configured to: receiving a set of noisy audio signals from a plurality of audio input devices during a first time window; generating a noisy time-frequency representation based on the set of noisy frequency signals; providing the noisy time-frequency representation as input to a mask estimation model, the mask estimation model being trained to output a mask for predicting a clean time-frequency representation of clean speech audio from the noisy time-frequency representation; determining beamforming filter weights based on the mask; applying beamforming filter weights to the noisy time-frequency representation to isolate clean speech audio from the set of noisy frequency signals; and Output clean voice audio.
9. The electronic device according to claim 8, in, The beamforming filter weights include a first power spectral density (PSD) matrix corresponding to speech audio and a second PSD matrix corresponding to noise audio.
10. The electronic device according to claim 9, in, The speech audio corresponding to the first PSD matrix and the noise audio corresponding to the second PSD matrix are from a second time window preceding the first time window.
11. The electronic device according to claim 9, in, The at least one processing device is further configured to update the first PSD matrix and the second PSD matrix using the mask.
12. The electronic device according to claim 8, in: To generate the noisy time-frequency representation, the at least one processing device is configured to perform a time-frequency analysis on the set of noisy frequency signals in the time domain; and The at least one processing device is further configured to convert the clean speech audio back to the time domain before outputting the clean speech audio.
13. The electronic device according to claim 8, in: The mask estimation model is trained using a training data set including multiple sets of sample noisy frequency signals; Each set of sample noisy audio signals is associated with a clean time-frequency representation of speech audio in the sample noisy audio signals and a noisy time-frequency representation of the sample noisy audio signals; The difference in amplitude and phase between the clean time-frequency representation and the predicted clean time-frequency representation is minimized; as well as The clean time-frequency representation and the predicted clean time-frequency representation are determined via applying the mask output by the mask estimation model to the noisy time-frequency representation.
14. The electronic device according to claim 8, in, The mask estimation model is trained to output a mask with a magnitude within a unit circle on the complex plane.
15. A non-transitory machine-readable medium comprising instructions that, when executed, cause at least one processor of an electronic device to: receiving a set of noisy audio signals from a plurality of audio input devices during a first time window; generating a noisy time-frequency representation based on the set of noisy frequency signals; providing the noisy time-frequency representation as input to a mask estimation model, the mask estimation model being trained to output a mask for predicting a clean time-frequency representation of clean speech audio from the noisy time-frequency representation; determining beamforming filter weights based on the mask; applying beamforming filter weights to the noisy time-frequency representation to isolate clean speech audio from the set of noisy frequency signals; and Output clean voice audio.
Citation Information
Cited By
Voice detection method based on millimeter wave radar
CN119780912A
