Method, apparatus, and computer readable medium for generating a clear voice signal using a voice and noise modeling based voice

Abstract

A method performed by an electronic device includes obtaining noisy speech signals, generating a predicted speech mask based on a first subset of the noisy speech signals using a speech mask prediction model, and generating a predicted noise mask based on a second subset of the noisy speech signals using a noise mask prediction model, a predicted filter mask is generated based on the predicted speech mask and the predicted noise mask using the filter mask prediction model, and a clean speech signal is generated using the predicted filter mask.

Description

Technical Field

[0001] The present disclosure relates generally to machine learning systems and more particularly to speech denoising networks using speech and noise modeling. Background Art

[0002] Edge devices and mobile platforms typically have relatively low computing resources due to low power requirements and very limited memory capacity. Therefore, hardware constraints such as power consumption (battery), inference time (latency), etc. are limiting factors for deploying deep learning models, which are usually too large for such devices and platforms. Many audio denoising units need to have less footprint and less computing power to perform proper noise reduction. For example, some digital signal processing (DSP) chips only allow up to one megabyte of runtime memory for processing. Denoising techniques based on deep learning typically require large neural networks to learn complex mappings between noisy and clean signals, which may prohibit them from being deployed to such devices. When the model size is limited to be small enough for deployment purposes, the network usually cannot achieve satisfactory performance. Summary of the invention

[0003] Technical Solution

[0004] The present disclosure relates to a speech denoising network using speech and noise modeling.

[0005] In an embodiment of the present disclosure, a method may include obtaining a noisy speech signal. In an embodiment of the present disclosure, the method may include generating a predicted speech mask based on a first subset of the noisy speech signal using a speech mask prediction model, and generating a predicted noise mask based on a second subset of the noisy speech signal using a noise mask prediction model. In an embodiment of the present disclosure, the method may include generating a predicted filter mask based on the predicted speech mask and the predicted noise mask using a filter mask prediction model. In an embodiment of the present disclosure, the method may include generating a clean speech signal using the predicted filter mask.

[0006] In an embodiment of the present invention, an apparatus may include at least one processor configured to obtain a noisy speech signal. In an embodiment of the present disclosure, at least one processor may be configured to generate a predicted speech mask based on a first subset of the noisy speech signal using a speech mask prediction model, and to generate a predicted noise mask based on a second subset of the noisy speech signal using a noise mask prediction model. In an embodiment of the present disclosure, at least one processor may be configured to generate a predicted filter mask based on the predicted speech mask and the predicted noise mask using a filter mask prediction model. In an embodiment of the present disclosure, at least one processor may be configured to generate a clean speech signal using the predicted filter mask.

[0007] In an embodiment of the present disclosure, a computer-readable medium may include instructions that, when executed, cause at least one processor to obtain a noisy speech signal. In an embodiment of the present disclosure, a computer-readable medium may include instructions that, when executed, cause at least one processor to generate a predicted speech mask based on a first subset of the noisy speech signal using a speech mask prediction model, and to generate a predicted noise mask based on a second subset of the noisy speech signal using a noise mask prediction model. In an embodiment of the present disclosure, a computer-readable medium may include instructions that, when executed, cause at least one processor to generate a predicted filter mask based on a predicted speech mask and a predicted noise mask using a filter mask prediction model. In an embodiment of the present disclosure, a computer-readable medium may include instructions that, when executed, cause at least one processor to generate a clean speech signal using the predicted filter mask.

[0008] Other technical features may be clear to those skilled in the art from the following drawings, descriptions and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals represent like parts:

[0010] Figure 1 An example network configuration including electronic devices according to an embodiment of the present disclosure is shown;

[0011] Figure 2 An example audio denoising system according to an embodiment of the present disclosure is shown;

[0012] Figure 3 An example audio denoising process according to an embodiment of the present disclosure is shown;

[0013] Figure 4 An example masking estimation and audio denoising process according to an embodiment of the present disclosure is shown;

[0014] Figure 5A and Figure 5B An example two-stage denoising training process according to an embodiment of the present disclosure is shown;

[0015] Figure 6 An example multi-channel masking estimation and audio denoising process according to an embodiment of the present disclosure is shown;

[0016] Figure 7 An example audio denoising and post-filtering process according to an embodiment of the present disclosure is shown;

[0017] FIG. 8A to FIG. 8Cshows an example audio signal waveform and accompanying spectrogram according to an embodiment of the present disclosure;

[0018] Fig.9A and Fig. 9B An example method for training and using one or more denoising machine learning models according to an embodiment of the present disclosure is shown;

[0019] Fig. 10A and Fig. 10B An example method for two-stage denoising model training according to an embodiment of the present disclosure is shown; and

[0020] Fig.11 An example method for audio signal denoising according to an embodiment of the present disclosure is shown.

[0021] Fig.12 A method executed by an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0022] Before proceeding to the following detailed description, it may be advantageous to set forth the definitions of certain words and phrases used throughout this patent document. The terms "transmit", "receive" and "communicate" and their derivatives cover both direct and indirect communications. The terms "include" and "comprise" and their derivatives mean including but not limited to. The term "or" is inclusive, meaning and / or. The phrase "associated with..." and its derivatives mean including, included within, interconnected with, including, contained within, connected to or connected with, coupled to or coupled with, communicable with, collaborate with, interlace, juxtapose, close to, bound to or bound with, have, have the property of, have to or with the relationship of, etc.

[0023] In addition, the various functions described below can be implemented or supported by one or more computer programs, each of which is formed by a computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data or a portion thereof suitable for implementation in a suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a compact disk (CD), a digital video disk (DVD) or any other type of memory. "Non-transitory" computer-readable media does not include wired, wireless, optical or other communication links that transmit temporary electrical signals or other signals. Non-transitory computer-readable media include media in which data can be permanently stored and media in which data can be stored and rewritten later, such as rewritable optical disks or erasable memory devices.

[0024] As used herein, terms and phrases such as "having", "may have", "include", or "may include" a feature (such as a number, function, operation, or component such as a part) indicate the presence of the feature without excluding the presence of other features. In addition, as used herein, the phrases "A or B", "at least one of A and / or B", or "one or more of A and / or B" may include all possible combinations of A and B. For example, "A or B", "at least one of A and B", and "at least one of A or B" may indicate (1) including at least one A, (2) including at least one B, or (3) including all of at least one A and at least one B. Throughout this disclosure, the expression "at least one of a, b, or c" indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0025] In addition, as used herein, the terms "first" and "second" can modify various components regardless of importance and do not limit the components. These terms are only used to distinguish one component from another component. For example, a first user device and a second user device can indicate user devices that are different from each other, regardless of the order or importance of the devices. Without departing from the scope of the present disclosure, a first component can be represented as a second component, and vice versa.

[0026] It should be understood that when an element (such as a first element) is referred to as being "coupled" / "coupled to" another element (such as a second element) or "connected" / "connected to" another element (operably or communicatively), it can be coupled to or connected to the other element directly or via a third element. Conversely, it should be understood that when an element (such as a first element) is referred to as being "directly coupled" / "directly coupled to" another element (such as a second element) or "directly connected" or "directly connected to" another element, no other element (such as a third element) is interposed between the element and the other elements.

[0027] As used herein, the phrase "configured (or set) to" may be used interchangeably with the phrases "suitable for," "capable of," "designed to," "adapted to," "manufactured to," or "capable of," depending on the circumstances. The phrase "configured (or set) to" does not substantially mean "designed specifically in hardware." On the contrary, the phrase "configured to" may mean that a device may perform an operation together with another device or part. For example, the phrase "a processor configured (or set) to perform A, B, and C" may mean a general-purpose processor (such as a CPU or an application processor) that can perform operations by executing one or more software programs stored in a memory device, or a dedicated processor (such as an embedded processor) for performing operations.

[0028] The terms and phrases used herein are only used to describe some embodiments of the present disclosure, rather than to limit the scope of other embodiments of the present disclosure. It should be understood that the singular forms "a", "an" and "the" include plural references unless the context clearly indicates otherwise. All terms and phrases used herein (including technical and scientific terms and phrases) have the same meaning as the meanings commonly understood by those of ordinary skill in the art to which the embodiments of the present disclosure belong. It will be further understood that terms and phrases (such as those defined in common dictionaries) should be interpreted as having meanings consistent with their meanings in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense, unless explicitly defined herein. In some cases, the terms and phrases defined herein may be interpreted as excluding embodiments of the present disclosure.

[0029] Examples of "electronic devices" according to embodiments of the present disclosure may include at least one of a smart phone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (such as smart glasses, a head mounted device (HMD), electronic clothing, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, a smart mirror, or a smart watch). Other examples of electronic devices include smart home appliances. Examples of smart home appliances may include a television, a digital video disc (DVD) player, an audio player, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave, a washing machine, a dryer, an air purifier, a set-top box, a home automation control panel, a security control panel, a TV box (such as SAMSUNG HOMESYNC, APPLE TV, or GOOGLE TV), a smart speaker or speaker with an integrated digital assistant (such as SAMSUNG GALAXY HOME, APPLE HOMEPOD, or AMAZON ECHO), a game console (such as XBOX, PLAYSTATION, or NINTENDO), an electronic dictionary, an electronic key, a camera, or at least one of an electronic photo frame. Other examples of electronic devices include various medical devices (such as various portable medical measurement devices (such as blood sugar measurement devices, heart rate measurement devices, or body temperature measurement devices), magnetic resource angiography (MRA) devices, magnetic resource imaging (MRI) devices, computer tomography (CT) devices, imaging devices, or ultrasound devices), navigation devices, global positioning system (GPS) receivers, event data recorders (EDR), flight data recorders (FDR), car infotainment devices, navigation electronic devices (such as navigation navigation devices or gyrocompasses), avionics equipment, security equipment, vehicle-mounted head units, industrial or domestic robots, automatic teller machines (ATMs), point of sale (POS) devices, or Internet of Things (IoT) devices (such as light bulbs, various sensors, electricity or gas meters, sprinklers, fire alarms, thermostats, street lights, ovens, fitness equipment, hot water tanks, heaters, or boilers). Other examples of electronic devices include at least a part of a piece of furniture or a building / structure, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (such as devices for measuring water, electricity, gas, or electromagnetic waves). Note that according to one or more embodiments of the present disclosure, the electronic device may be one or a combination of the devices listed above. According to some embodiments of the present disclosure, the electronic device may be a flexible electronic device. The electronic device disclosed herein is not limited to the devices listed above, and may include any other electronic device now known or later developed.

[0030] In the following description, according to one or more embodiments of the present disclosure, an electronic device is described with reference to the accompanying drawings. As used herein, the term "user" may refer to a person using an electronic device or another device (such as an artificial intelligence electronic device).

[0031] In an embodiment of the present disclosure, "generating A based on B using the C model" may refer to obtaining, generating, or identifying A by inputting at least one of B or data associated with B into the C model and outputting at least one of A or data associated with A through the C model. In an example, the data associated with B may include at least one of data generated by performing any processing (e.g., pre-processing) or any operation on B, data extracted or calculated from B, data calculated, generated, or determined based on B, or data including at least a portion of B. In an example, the data associated with A may include at least one of data generating A by performing any processing (e.g., post-processing) or any operation, data extracting or calculating A from it, data serving as a basis for calculating, generating, determining A, or data including at least a portion of A.

[0032] Definitions for certain other words and phrases may be provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

[0033] Nothing in the description in this application should be read as implying that any particular element, step, or function is an essential element that must be included within the scope of the claims. The scope of the patented subject matter is limited solely by the claims. In addition, no claim is intended to invoke 35 U.S.C. §112(f) unless the exact words "means for..." are followed by a participle. The use of any other terms within the claims, including but not limited to "mechanism," "module," "device," "unit," "component," "element," "member," "device," "machine," "system," "processor," or "controller," are understood by applicant to refer to structures known to those skilled in the relevant art and are not intended to invoke 35 U.S.C. §112(f).

[0034] The following discussion is described with reference to the accompanying drawings. Figures 1 to 11 As well as one or more embodiments of the present disclosure. However, it should be understood that the present disclosure is not limited to these embodiments, and all changes and / or equivalents or replacements thereto also belong to the scope of the present disclosure. Throughout the specification and the drawings, the same or similar reference numerals may be used to refer to the same or similar elements.

[0035] As mentioned above, edge devices and mobile platforms typically have relatively low computing resources due to low power requirements and very limited memory capacity. Therefore, hardware constraints such as power consumption (battery), inference time (latency), etc. are limiting factors for deploying deep learning models, which are usually too large for such devices and platforms. Many audio denoising units need to have less footprint and less computing power to perform proper noise reduction. For example, some digital signal processing (DSP) chips only allow up to one megabyte of runtime memory for processing. Denoising techniques based on deep learning typically require large neural networks to learn complex mappings between noisy signals and clean signals, which may prohibit them from being deployed to such devices. When the model size is limited to be small enough for deployment purposes, the network usually cannot achieve satisfactory performance.

[0036] Currently, there are two main existing pre-processing solutions that are widely used. However, these two methods still do not completely solve the denoising problem in noisy environments under limited resource constraints. First, some signal processing / mixed signal processing-based solutions use beamforming techniques with signal processing-based noise suppression algorithms, where the signal and noise properties are mathematically modeled to approach the solution. Although this type of noise suppression method is suitable for use with resource-constrained computing devices such as DSP chipsets, it usually leads to suboptimal performance because it cannot handle complex real-world audio properties. In addition, signal processing-based methods usually require some manual tuning of some or all of their hyperparameters, which in turn makes the solution difficult to scale for various applications and use cases, and leads to lengthy parameter tuning for different settings and scenarios. In addition, some data-driven solutions based on deep learning use techniques for training speech denoising models in an end-to-end manner, which usually requires large modeling capabilities of network models for appropriately learning noisy to clean signal mappings from data. However, such networks are not suitable for deployment on memory / power-constrained devices because they will require a large number of central processing units (CPUs) or graphics processing units (GPUs) to use. Furthermore, direct model size reduction is not possible because small networks have difficulty learning such complex noisy-to-clean signal mappings without special design. Therefore, although existing signal processing and mixed signal processing solutions can be applied on resource-constrained devices, they are usually not as powerful as deep learning-based methods. In addition, although deep learning-based methods can have better denoising capabilities, they are difficult to deploy to resource-constrained devices.

[0037] Cloud-based artificial intelligence (AI) methods can also be used to perform deep learning preprocessing by streaming raw audio data to the cloud. However, this may lead to user privacy issues because all private audio data is streamed to the cloud for further processing. In addition, this method may introduce high latency and high power consumption, and may require high bandwidth due to heavy reliance on Internet communications for processing. These shortcomings may make it impossible for the system to achieve always-on services for continuous processing tasks (such as denoising processes).

[0038] One or more embodiments of the present disclosure solve these or other problems by efficiently designing model architecture and learning strategies to provide the benefit of using a small network to perform accurate denoising using prior knowledge of signal and noise characteristics. The speech denoising and / or speech enhancement model of the present disclosure can form a key component to realize various audio applications. It is disadvantageous that the system that does not include speech denoising and / or speech enhancement model is because the received audio input will be provided to a speech processing system with polluted and noisy audio, such as automatic speech recognition (ASR) and wake-up service, thereby reducing the accuracy of the speech processing system. Therefore, an embodiment of the present disclosure includes one or more speech denoising and / or speech enhancement models that act as an audio preprocessing front end to reduce or remove various environmental noises that pollute audio, so that subsequent speech processing systems (such as ASR and wake-up services) can still work normally, and performance degradation is minimal or there is no performance degradation. In some cases, the preprocessing model can use an always-online listening attribute to process all incoming audio streams, and can be executed in very low-power computing and limited memory devices and platforms that expect minimal resource consumption.

[0039] One or more embodiments of the present disclosure include a tiny machine learning ("tinyML") model as a preprocessing model / architecture. The model is applied to resource-constrained devices and can provide better denoising performance using a data-driven neural network method based on deep learning. The tinyML model of an embodiment of the present disclosure includes a dual-branch architecture having a speech prediction model and a noise prediction model, which are configured to learn intermediate variables for characterizing speech and noise properties, respectively. In addition, the tinyML model uses a speech filter model to improve the learning of the final signal-to-noise ratio (SNR) related denoising filter mask, and the speech filter model predicts a clean speech signal for subsequent use by one or more speech processing systems (such as ASR and wake-up systems). In one or more embodiments of the present disclosure, the tinyML model can use a multi-channel or multi-microphone approach to model speech and noise separately by utilizing two subsets of microphones. In addition, the tinyML model can be trained progressively, wherein the speech and noise prediction models are trained in a first training phase, and the final stages of the model (such as speech filtering and clean speech prediction) are trained in a second training phase.

[0040] The audio preprocessing model of the present disclosure allows an always-on, small footprint machine learning preprocessing service to be run on an edge device (such as a smart phone, smart glasses, augmented reality glasses, smart watches, smart refrigerators, headphones or earbuds, robot vacuum cleaners, or other electronic devices described in the present disclosure). Among other things, this provides advantages such as improved privacy, low latency, and low bandwidth because in one or more embodiments of the present disclosure, the preprocessing denoising system can be run on the device and the raw audio data is not sent to a cloud server. Lower power consumption can also be achieved because the small footprint micro-machine process consumes very little power, enabling the process to run for always-on tasks without requiring high device power consumption.

[0041] Figure 1 An example network configuration 100 including electronic devices according to an embodiment of the present disclosure is shown. Figure 1 The embodiment of the network configuration 100 shown is for illustration only. Other embodiments of the network configuration 100 may be used without departing from the scope of the present disclosure.

[0042] According to one or more embodiments of the present disclosure, the electronic device 101 may be included in the network configuration 100. The electronic device 101 may include at least one of a bus 110, a processor 120, a memory 130, an input / output (I / O) interface 150, a display 160, a communication interface 170, or a sensor 180. In one or more embodiments of the present disclosure, the electronic device 101 may exclude at least one of these components, or may add at least one other component. The bus 110 may include circuits for connecting the components 120-180 to each other and for transmitting communications (such as control messages and / or data) between the components.

[0043] The processor 120 may include one or more processing devices, such as one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). In one or more embodiments, the processor 120 may include one or more of a central processing unit (CPU), an application processor (AP), a communication processor (CP), or a graphics processor unit (GPU). The processor 120 may be capable of controlling at least one of the other components of the electronic device 101 and / or performing operations or data processing related to communication or other functions. As described below, the processor 120 may receive and process inputs (such as audio inputs or data received from an audio input device such as a microphone), perform audio denoising, output a clean / denoised audio signal, and perform one or more ASR tasks or other tasks using the clean / denoised audio signal. The processor 120 may also instruct one or more other devices to perform one or more operations (such as outputting audio using an audio output device such as a speaker) or display content on one or more displays 160. The processor 120 may also receive inputs, such as data samples to be used in training a machine learning model, and manage such training to improve the machine learning model by inputting the samples into the machine learning model, receiving outputs from the machine learning model, and executing a learning function, such as a loss function.

[0044] The memory 130 may include volatile and / or non-volatile memory. For example, the memory 130 may store commands or data related to at least one other component of the electronic device 101. According to one or more embodiments of the present disclosure, the memory 130 may store software and / or programs 140. The programs 140 may include, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and / or an application program (or "application") 147. At least a portion of the kernel 141, the middleware 143, or the API 145 may be represented as an operating system (OS).

[0045] The kernel 141 may control or manage system resources (such as the bus 110, the processor 120, or the memory 130) for performing operations or functions implemented in other programs (such as the middleware 143, the API 145, or the application 147). The kernel 141 may provide an interface that allows the middleware 143, the API 145, or the application 147 to access the various components of the electronic device 101 to control or manage system resources. The application 147 may include one or more applications that support the reception of audio data or signals; use the audio data or signals to predict voice masking, noise masking, and filter masking to clean / denoise the audio data or signals; identify the wake-up words in the speech included in the audio data or signals; perform speech recognition on the speech; and perform tasks related to the content of the speech. These functions may be performed by a single application or multiple applications, each of which performs one or more of these functions. For example, the middleware 143 may be used as a repeater to allow the API 145 or the application 147 to communicate data with the kernel 141. Multiple applications 147 may be provided. The middleware 143 can control work requests received from the applications 147, such as by assigning a priority of using system resources (such as the bus 110, the processor 120, or the memory 130) of the electronic device 101 to at least one of the plurality of applications 147. The API 145 may be an interface that allows the application 147 to control functions provided from the kernel 141 or the middleware 143.

[0046] The I / O interface 150 may serve as an interface that may, for example, transmit commands or data input from a user or other external devices to other components of the electronic device 101. The I / O interface 150 may also output commands or data received from other components of the electronic device 101 to the user or other external devices.

[0047] The display 160 may include, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. The display 160 may also be a depth perception display, such as a multi-focal display. The display 160 may be capable of displaying, for example, various contents (such as text, images, videos, icons, or symbols) to the user. The display 160 may include a touch screen and may receive, for example, a touch, gesture, proximity, or hover input using an electronic pen or a body part of the user.

[0048] For example, the communication interface 170 may be able to establish communication between the electronic device 101 and an external electronic device (such as the first electronic device 102, the second electronic device 104, or the server 106). For example, the communication interface 170 may be connected to the network 162 or 164 via wireless or wired communication to communicate with the external electronic device. The communication interface 170 may be a wired or wireless transceiver or any other component for sending and receiving signals (such as images).

[0049] The electronic device 101 may include one or more sensors 180, which can measure physical quantities or detect the activation state of the electronic device 101, and convert the measured or detected information into electrical signals. The sensor 180 may also include one or more buttons for touch input, one or more microphones, gesture sensors, gyroscopes or gyroscope sensors, air pressure sensors, magnetic sensors or magnetometers, acceleration sensors or accelerometers, grip sensors, proximity sensors, color sensors (such as RGB sensors), biophysical sensors, temperature sensors, humidity sensors, illumination sensors, ultraviolet (UV) sensors, electromyography (EMG) sensors, electroencephalogram (EEG) sensors, electrocardiogram (ECG) sensors, infrared (IR) sensors, ultrasonic sensors, iris sensors or fingerprint sensors. The sensor 180 may also include an inertial measurement unit, which may include one or more accelerometers, gyroscopes and other components. In an example, the sensor 180 may include a control circuit for controlling at least one of the sensors included here. Any of these sensors 180 may be located within the electronic device 101.

[0050] The first external electronic device 102 or the second external electronic device 104 may be a wearable device or a wearable device (such as an HMD) in which the electronic device can be installed. When the electronic device 101 can be installed in the electronic device 102 (such as an HMD), the electronic device 101 can communicate with the electronic device 102 through the communication interface 170. The electronic device 101 can be directly connected to the electronic device 102 to communicate with the electronic device 102 without involving a separate network. The electronic device 101 can also be an augmented reality wearable device including one or more cameras, such as glasses.

[0051] Wireless communication may be able to use, for example, long term evolution (LTE), advanced long term evolution (LTE-A), fifth generation wireless system (5G), millimeter wave or 60GHz wireless communication, wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro) or global system for mobile communications (GSM) as at least one of the cellular communication protocol. Wired connection may include, for example, at least one of universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232) or plain old telephone service (POTS). Network 162 may include at least one communication network, such as a computer network (such as a local area network (LAN) or a wide area network (WAN)), the Internet or a telephone network.

[0052] The first external electronic device 102 and the second external electronic device 104 and the server 106 may each be a device of the same or different type as the electronic device 101. According to one or more embodiments of the present disclosure, the server 106 may include a group of one or more servers. In addition, according to one or more embodiments of the present disclosure, all or some operations performed on the electronic device 101 may be performed on another or more other electronic devices (such as the electronic devices 102 and 104 or the server 106). In addition, according to one or more embodiments of the present disclosure, when the electronic device 101 should automatically or upon request perform some functions or services, the electronic device 101 may request another device (such as the electronic devices 102 and 104 or the server 106) to perform at least some functions associated with it, rather than performing the functions or services separately or additionally. Other electronic devices (such as the electronic devices 102 and 104 or the server 106) may be able to perform the requested functions or additional functions and transmit the results of the execution to the electronic device 101. The electronic device 101 may provide the requested functions or services by processing the received results as is or additionally. To this end, for example, cloud computing, distributed computing, or client-server computing technologies may be used. Although Figure 1 The electronic device 101 is shown to include a communication interface 170 that communicates with the external electronic device 104 or the server 106 via the network 162 , but according to one or more embodiments of the present disclosure, the electronic device 101 may operate independently without a separate communication function.

[0053] The server 106 may include components that are the same or similar to the electronic device 101 (or a suitable subset thereof). The server 106 may support driving the electronic device 101 by performing at least one of the operations (or functions) implemented on the electronic device 101. For example, the server 106 may include a processing module or processor that may support the processor 120 implemented in the electronic device 101. As described below, the server 106 may receive and process inputs (such as audio inputs or data received from an audio input device such as a microphone), perform audio denoising, output a clean / denoised audio signal, and use the clean / denoised audio signal to perform one or more ASR tasks or other tasks. The server 106 may also instruct one or more other devices to perform one or more operations (such as outputting audio using an audio output device such as a speaker) or display content on one or more displays 160. The server 106 may also receive inputs (such as data samples to be used in training a machine learning model) and manage such training to improve the machine learning model by inputting samples to the machine learning model, receiving outputs from the machine learning model, and executing a learning function (such as a loss function).

[0054] although Figure 1 An example of a network configuration 100 including an electronic device 101 is shown, but the Figure 1 Various changes may be made. For example, network configuration 100 may include any suitable number of each component in any suitable arrangement. In general, computing and communication systems have a wide variety of configurations, and Figure 1 The scope of the present disclosure is not limited to any particular configuration. Figure 1 One operating environment is shown in which the various features disclosed in this patent document may be used, but these features may be used in any other suitable system.

[0055] Figure 2 An example audio denoising system 200 according to an embodiment of the present disclosure is shown. For ease of explanation, the system 200 may be described as involving Figure 1 The electronic device 101 is used in the network configuration 100 of FIG. However, the system 200 may be used with any other suitable electronic device (such as the server 106) and may be used in any other suitable system.

[0056] like Figure 2As shown, the system 200 may include an electronic device 101, which includes a processor 120. The processor 120 may be operably coupled to or otherwise configured to use one or more machine learning models, such as one or more audio preprocessing models 202 to process audio data or signals received from at least one audio input device 204 (such as a microphone). As described in the present disclosure, the audio preprocessing model 202 may include a micro machine learning ("tinyML") model architecture, which can be applied to resource-constrained devices and can provide better denoising performance using a data-driven neural network method based on deep learning. The audio preprocessing model 202 of one or more embodiments of the present disclosure may include a dual-branch architecture, which includes a speech prediction model and a noise prediction model configured to learn intermediate variables for characterizing speech and noise attributes, respectively. The audio preprocessing model 202 may also include a speech filter model that receives speech and noise characteristics from the speech and noise prediction models and estimates a filter mask for predicting a clean speech signal for subsequent use by one or more speech processing systems (such as ASR and wake-up systems). In one or more embodiments of the present disclosure, the audio pre-processing model 202 may use a multi-channel or multi-microphone approach to model speech and noise separately by utilizing two subsets of the audio input devices 204. In an embodiment of the present disclosure, the audio pre-processing model 202 may be trained progressively, such as where speech and noise prediction models are trained in a first training phase, and the final stages of the model (such as speech filtering and clean speech prediction) are trained in a second training phase.

[0057] The system 200 may also include one or more speech recognition models 206 that receive the denoised audio data or signal from the preprocessing model 202, such as one or more wake-up word detector models trained to recognize one or more wake-up words or phrases in the denoised audio data or signal, which can trigger the execution of further automated speech or other processes and actions of the electronic device 101. The speech recognition model 206 may also include one or more ASR models and / or one or more natural language understanding (NLU) models that use the denoised audio data or signal to determine a command or action in the speech included in the denoised audio data or signal, which causes the processor 120 to instruct one or more actions of the electronic device 101 or one or more other electronic devices. It should be understood that the machine learning models 202, 206 can be stored in a memory (such as memory 130) of the electronic device 101 and accessed by the processor 120 to perform audio denoising tasks, automated speech recognition tasks, or other tasks. However, the machine learning models 202, 206 can be stored in any other suitable manner.

[0058] In some cases, the system 200 may include at least one audio output device 208 (such as a speaker or headphone interface) and at least one display 210 (such as a screen or monitor, such as display 160). In one or more embodiments of the present disclosure, the processor 120 may receive audio input from one or more audio input devices 204 and provide the audio input to one or more trained audio pre-processing models 202. The trained audio pre-processing model 202 may clean or denoise the audio data or signal, and the processor 120 may provide the denoised audio data or signal to the speech recognition model 206 to perform one or more other tasks. Other tasks may include one or more speech recognition tasks (such as wake-up word detection or wake-up process) or one or more ASR tasks (such as detecting user intent and commands in the denoised audio data or signal for the processor 120 to use to instruct one or more further actions of the electronic device 101 or one or more other electronic devices).

[0059] As an example, assume that an audio input including an utterance is received from a user via an audio input device 204, wherein the utterance includes a wake-up word or phrase (such as "HEY, BIXBY, CALL MOM"). Here, the trained audio pre-processing model 202 may denoise / clean the audio input, and the processor 120 may provide the denoised / cleaned audio to a wake-up word detector model, which detects the presence of the wake-up word "BIXBY" or the wake-up phrase "HEY, BIXBY". One or more other ASR / NLU models may be triggered to determine an action from the audio, and the processor 120 may instruct the audio output device 208 to output "Call Mom". In this example, the processor 120 may cause a phone application or other communication application to start a communication session with a "Mom" contact stored on the electronic device 101 or otherwise associated with the user of the electronic device 101. As an example, assume that the utterance "HEY, BIXBY, start the timer" is received. The trained audio preprocessing model 202 can denoise / clean the audio input, and the processor 120 can provide the denoised / clean audio to the wake-up word detector model, which can detect the presence of the wake-up word "BIXBY" or the wake-up phrase "Hey, BIXBY". One or more ASR / NLU models can be triggered to determine an action from the audio, and the processor 120 can instruct the execution of a timer application and display a timer on the display 210 of the electronic device 101.

[0060] The audio preprocessing model 202 (including speech denoising and / or speech enhancement model) of the present disclosure can be a key component for implementing various audio applications. Systems that do not include speech denoising and / or speech enhancement models are disadvantageous because the received audio input will be provided to a speech processing system with contaminated and noisy audio, such as ASR and wake-up services, thereby reducing the accuracy of the speech processing system. Therefore, one or more embodiments of the present disclosure may include one or more audio preprocessing models 202 used as an audio preprocessing front end to reduce or remove various environmental noises that contaminate the audio, so that subsequent speech processing systems such as ASR and wake-up services can still operate normally with minimal or no performance degradation.

[0061] In one or more embodiments of the present disclosure, the audio pre-processing model 202 can be configured for always-on listening to process all incoming audio streams, and can be executed in very low-power computing and limited memory devices and platforms where minimal resource consumption is desired. The audio pre-processing model 202 of the present disclosure can allow an always-on, small-footprint machine learning pre-processing service to be run on edge devices, which provides advantages such as privacy, low latency, low bandwidth, and low power consumption, as described above.

[0062] although Figure 2 One example of an audio denoising system 200 is shown, but may be used for Figure 2 Various changes are made. For example, the audio input device 204, the audio output device 208, and the display 210 may be connected to the processor 120 within the electronic device 101, such as via a wired connection or circuit. In one or more embodiments of the present disclosure, the audio input device 204, the audio output device 208, and the display 210 may be connected outside the electronic device 101 and via a wired or wireless connection. In addition, the separate models described in the present disclosure and included in the audio preprocessing model 202 may be stored as separate models called by the processor 120 to perform certain tasks, or may be included in one or more larger machine learning models and form a part thereof. In addition, in one or more embodiments of the present disclosure, one or more of the machine learning models (including the audio preprocessing model 202 and the speech recognition model 206) may be stored away from the electronic device 101, such as stored on the server 106. Here, the electronic device 101 may send a request including an input (such as captured audio data) to the server 106 to process the input using the machine learning model, and the result may be sent back to the electronic device 101, such as to indicate that the electronic device 101 performs at least one subsequent action based on the audio input.

[0063] Figure 3 An example audio denoising process 300 according to an embodiment of the present disclosure is shown. For ease of explanation, the process 300 is described as involving Figure 1 The electronic device 101 is used in the network configuration 100 of FIG. However, the process 300 can be used with any other suitable electronic device (such as the server 106) or a combination of devices (such as the electronic device 101 and the server 106), and can be used in any other suitable system.

[0064] like Figure 3 As shown in FIG. 3 , process 300 may include one or more machine learning models 302 receiving one or more noisy frequency signals 301. In one or more embodiments of the present disclosure, machine learning model 302 may represent the above-mentioned Figure 2 The audio preprocessing model 202 described. The machine learning model 302 may include one or more neural networks, such as one or more fully connected networks, convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc. The machine learning model 302 may be configured to predict a denoising (filtering) mask based on the capabilities of the pre-learned machine learning model 302, which in one or more embodiments of the present disclosure is an SNR-related variable to estimate signal and noise intermediate variables to improve the final audio quality.

[0065] During example operation, the processor 120 may extract any acoustic features (such as amplitude spectrum distribution information) from the original noisy audio signal 301, and the extracted acoustic features may be input into the machine learning model 302. For denoising purposes, the speech and noise components in the spectrum may be estimated from the audio input by the machine learning model 302, and the estimates of the speech and noise components may be converted into a sequence of intermediate variables. The estimated intermediate variables of speech and noise may reflect the SNR of each frequency bin (the spacing between samples in the frequency domain), and may be constructed into a filter mask via the estimated SNR of each frequency component. The predicted SNR-related filter mask may be compared with a reference microphone audio signal 304 (such as a Figure 3 The noise signal #1 is combined (such as multiplied) with the noise signal 301 shown in the figure to output the denoised audio signal 306. In this way, instead of directly learning clean speech as a target, the speech and noise components of each frequency bin can be estimated as learnable intermediate parameters to construct an SNR-dependent filtering mask. This method can produce a miniature machine learning model that can be deployed on resource-constrained electronic devices or other devices.

[0066] although Figure 3 An example of an audio denoising process 300 is shown, but may be used for Figure 3Various changes are made. For example, a separate model described in the present disclosure and included in the machine learning model 302 can be stored as a separate model called by the processor 120 to perform certain tasks, or can be included in one or more larger machine learning models and form a part thereof. In addition, process 300 can be performed using a distributed architecture. For example, the machine learning model 302 can be executed on a client electronic device (such as electronic device 101) or at a server (such as server 106). When executed by the server, the server can provide the client electronic device with a determined action to be performed by the client electronic device. In one or more embodiments of the present disclosure, the client electronic device can provide audio data received via an audio input device of the client electronic device to the server, audio denoising can be performed by the server, and one or more ASR / NLU models can be performed by the client electronic device or the server based on the denoised audio data.

[0067] Figure 4 An example masking estimation and audio denoising process 400 according to an embodiment of the present disclosure is shown. For ease of explanation, the process 400 may be described as involving Figure 1 The electronic device 101 is used in the network configuration 100 of FIG. However, the process 400 can be used with any other suitable electronic device (such as the server 106) or a combination of devices (such as the electronic device 101 and the server 106), and can be used in any other suitable system.

[0068] like Figure 4 As shown, process 400 may include using a speech prediction model 402, a noise prediction model 404, and a speech filter model 406. Speech prediction model 402, noise prediction model 404, and speech filter model 406 may be included in the above Figure 2 and Figure 3 The described audio preprocessing model 202 and / or machine learning model 302 may include one or more neural networks, such as one or more fully connected networks, CNNs, RNNs, etc.

[0069] During example operation, the processor 120 may extract acoustic features (such as amplitude spectrum distribution information) from a noisy audio signal 401 received from an audio input device (such as a microphone), and input the extracted acoustic features to a speech prediction model 402 and a noise prediction model 404. The speech prediction model 402 and the noise prediction model 404 may be trained to process the acoustic features and estimate speech and noise intermediate variables to improve final audio quality. For example, based on the acoustic features of the audio input 401, the speech prediction model 402 may be trained to estimate speech masking (M S ), and the noise prediction model can be trained to estimate the noise mask (M V). These estimated speech masking and noise masking can reflect the SNR of each frequency bin (the spacing between samples in the frequency domain).

[0070] In one or more embodiments of the present disclosure, the process 400 can use the speech prediction model 402 and the noise prediction model 404 to model speech and noise respectively by utilizing two subsets of audio devices (such as microphones). For example, the process 400 can utilize multi-channel / multi-microphone raw data input and can utilize the above-mentioned dual-branch model architecture to apply a separate subset of microphone data to the speech prediction model 402 and the noise prediction model 404, such as by providing the speech prediction model 402 with noisy audio signal #1 data and providing the noise prediction model 404 with noisy audio signal #N data. Some microphones can be closer to the noise source, while other microphones can be closer to the speech source. Therefore, some microphones can have more information about noise characteristics, and other microphones can have more information about speech characteristics. In order to take advantage of these situations, the process 400 can (i) use a first subset of acoustic features from a first subset of microphones to use the speech prediction model 402 to estimate speech characteristics, and (ii) use a second subset of acoustic features from a second subset of microphones to use the noise prediction model 404 to estimate noise characteristics. However, in one or more embodiments of the present disclosure, a single microphone may be used, wherein an audio signal from the single microphone is extracted and provided to both the speech prediction model 402 and the noise prediction model 404 .

[0071] Process 400 may use the output estimated speech masking (M S ) and the estimated noise mask (M V ) to estimate the filter mask (W). For example, after creating the speech mask and the noise mask from the acoustic feature subset, the estimated speech mask can be compared with the first reference microphone signal 408 (such as Figure 4 The estimated noise mask may be combined (such as multiplied) with the noisy audio signal #1 in FIG. 4 and the second reference microphone signal 410 (such as Figure 4 The noisy audio signal #N in the subset is combined (such as multiplied) to predict the speech and noise components for use by the speech filter model 406. By filtering the speech and noise from some reference microphones in the subset, the speech and noise information can be fused together for further processing by the speech filter model 406. Fusing the speech and noise components can involve cascading of information, or other information fusion techniques such as partial processing and layer-by-layer multiplication can be used.

[0072] The speech filter model 406 can receive the fused speech and noise components and output a filter mask (W). The filter mask can be SNR-related and can be estimated using the estimated SNR of each frequency component based on the speech and noise components. In this way, instead of directly learning clean speech as a target, the speech and noise components of each frequency bin can be estimated as learnable intermediate parameters to construct an SNR-related filter mask. The estimation of the final SNR-related filter mask can be improved using the predicted speech and noise components. In other words, there is no need to directly learn clean speech, but the process 400 can be used to learn the intermediate mask / filter mask. S 、M V and W. In one or more embodiments of the present disclosure, and In one or more embodiments of the present disclosure, the SNR filtering mask may be a Wiener-like ideal ratio mask.

[0073] In one or more embodiments of the present disclosure, the filtered mask (W) can be combined (such as multiplied) with a third reference microphone signal 412 (such as one of the noisy audio signals 301) to extract a denoised / clean audio signal 414 from the reference microphone signal 412. The denoised / clean audio signal can be used to perform at least one more accurate speech recognition process, such as wake-up word detection and wake-up process and / or ASR / NLU process. Process 400 can produce a miniature machine learning model that can be deployed on a resource-constrained electronic device or other device. For example, the mask estimation performed by process 400 can provide a smaller machine learning model, which allows deployment on devices with lower memory and lower computing resources.

[0074] although Figure 4 One example of a masking estimation and audio denoising process 400 is shown, but may be used for Figure 4Various changes are made. For example, the individual models described in the present disclosure (such as speech prediction model 402, noise prediction model 404, and speech filter model 406) can be stored as individual models called by processor 120 to perform certain steps of process 400, or can be included in one or more larger machine learning models and form a part thereof. In addition, process 400 can be performed using a distributed architecture. For example, speech prediction model 402, noise prediction model 404, and speech filter model 406 can be performed on a client electronic device (such as electronic device 101) or at a server (such as server 106). When executed by a server, the server can provide a determined action to be performed by the client electronic device to the client electronic device. In one or more embodiments of the present disclosure, the client electronic device can provide audio data received via an audio input device of the client electronic device to the server; audio denoising can be performed by the server using speech prediction model 402, noise prediction model 404, and speech filter model 406; and one or more ASR / NLU models can be performed by the client electronic device or the server based on denoised audio data.

[0075] Figure 5A and Figure 5B An example two-stage denoising training process 500 according to an embodiment of the present disclosure is shown. For ease of explanation, the process 500 can be described as involving Figure 1 The server 106 is used in the network configuration 100. As an example, the process 500 can be performed in Figure 1 The process 500 is executed on the server 106 in the network configuration 100, and one or more trained machine learning models (such as a trained speech prediction model, a trained noise prediction model, and a trained speech filter model) can be deployed to the client electronic device 101 for use. However, the process 500 can be used with any other suitable device (such as the electronic device 101) and can be used in any other suitable system.

[0076] Given that the denoising system of the present disclosure can be constrained to a smaller number of parameters than existing deep learning models, in some cases it may be advantageous to train each model of the system with inputs that most closely resemble ground truth in order to teach the model what speech and noise look like. To achieve this, training can be divided into multiple stages. For example, process 500 can progressively train a machine learning model using two training stages. Figure 5A As shown, process 500 may include a first training phase, in which a speech model 502 (such as Figure 4 The speech prediction model 402 in FIG. 4 and the noise model 504 (such as Figure 4The first stage here may not include executing the filtering model 506 (such as Figure 4 Any training of the speech filter model 406 in.

[0077] The first stage training of the speech model 502 and the noise model 504 to predict the speech mask and the noise mask, respectively, may include using training data to train the speech model 502 and the noise model 504, the training data including the input audio 501 with the corresponding reference truth speech mask and the noise mask. In one or more embodiments of the present disclosure, the input audio 501 may include one or more of (i) a noisy speech signal, (ii) a clean speech signal, and (iii) a noise signal. During the first stage training, the speech model 502 and the noise model 504 may be updated, such as by using a loss function, to reduce or minimize the output speech mask (M), respectively. S ) and the difference or loss between the reference truth speech mask and the reduction or minimization of the output noise mask (M V ) and the noise mask of the ground truth. In one or more cases, this can be expressed as The loss function may calculate the error or loss associated with the predictions of the speech model 502 and the noise model 504.

[0078] As an example, when the output of the speech model 502 and / or the noise model 504 differs from the ground truth, the difference can be used to calculate the loss defined by the loss function. The loss function can use any suitable measure of the loss associated with the output generated by the speech model 502 and the noise model 504, such as cross entropy loss or mean squared error. Figure 5A During the first stage of training shown, multiple training iterations are performed until a determination is reached that the first stage of training of the speech model 502 and the noise model 504 is complete, such as when it is determined that the models 502, 504 provide predictions at an acceptable level of accuracy. When the loss calculated by the loss function is greater than the expectation, the parameters of the models 502, 504 can be adjusted. Once adjusted, the same or additional training data can be provided to the adjusted models 502, 504, and additional outputs from the models 502, 504 can be generated and compared with the reference truth value, so that the additional loss can be determined using the loss function. Ideally, over time, the models 502, 504 can produce more accurate outputs that more closely match the reference truth value, and the measured loss becomes less. The amount of training data used can vary depending on the number of training cycles. At a certain moment, the measured loss can drop below a specified threshold, and the first training stage can be completed. The first training stage can allow the speech model 502 and the noise model 504 to learn to accurately predict masking before training the filter model 506.

[0079] After the first training phase, Figure 5B As shown, the final stage of the system can be trained during the second training phase. In the second training phase, the speech model 502, the noise model 504, and the filter model 506 can be jointly trained to predict the final output mask (filter mask (W)) using one or more training samples including the input audio 503. Each training sample can be associated with a reference truth filter mask. The speech model 502, the noise model 504, and the filter model 506 can be updated during the second stage training, such as by using a loss function, to reduce or minimize the difference between the predicted filter mask and the reference truth filter mask. In one or more cases, this can be expressed as The loss function may calculate the error or loss associated with the prediction of the final filtered mask output by the denoising system.

[0080] As an example, when a filter mask output by a filter model is received, the difference between the filter mask and the ground truth can be used to calculate a loss defined by a loss function. The loss function can use any suitable measure of the loss associated with the filter mask output, such as cross entropy loss or mean squared error. Figure 5B During the second stage of training shown, multiple training iterations are performed until a determination is reached that the second stage of joint training of the speech model 502, the noise model 504, and the filter model 506 is completed, such as when it is determined that the models 502, 504, 506 provide filter mask predictions at an acceptable level of accuracy. When the loss calculated by the loss function is greater than the expectation, the parameters of one or more of the models 502, 504, 506 can be adjusted. Once adjusted, the same or additional training data can be provided to the adjusted models 502, 504, 506, and additional filter mask outputs can be generated and compared with the reference truth value, so that the loss function can be used to determine the additional loss. Ideally, over time, the models 502, 504, 506 can produce more accurate outputs that more closely match the reference truth value, and the measured loss becomes less. The amount of training data used can vary depending on the number of training cycles. At a certain moment, the measured loss can drop below a specified threshold, and the second stage of training can be completed. In this way, the system can be progressively trained starting from the initial module and then moving downstream. Since the initial speech model 502 and noise model 504 have already been trained, the audio input to the downstream model is very similar to the ground truth, greatly improving the learning and accuracy of the entire denoising system.

[0081] although Figure 5A and Figure 5B An example of a two-stage denoising training process 500 is shown, but may be used for Figure 5Aand Figure 5B Various changes may be made. For example, the individual models described in the present disclosure (such as speech model 502, noise model 504, and filter model 506) may be stored as individual models that are called by processor 120 to perform certain steps of training process 500, or may be included in and form part of one or more larger machine learning models. In addition, in one or more embodiments of the present disclosure, process 500 may be performed using electronic device 101 instead of server 106.

[0082] Figure 6 An example multi-channel masking estimation and audio denoising process 600 according to an embodiment of the present disclosure is shown. For ease of explanation, the process 600 may be described as involving Figure 1 The electronic device 101 is used in the network configuration 100 of FIG. However, the process 600 can be used with any other suitable electronic device (such as the server 106) or a combination of devices (such as the electronic device 101 and the server 106), and can be used in any other suitable system.

[0083] As described above, some microphones may be closer to the noise source and some microphones may be closer to the speech source. Thus, some microphones may have more information about the noise characteristics and other microphones may have more information about the speech characteristics. Figure 6 As shown, to take advantage of those situations, process 600 can estimate speech characteristics using acoustic features from a first subset of multiple microphones 601, and estimate noise characteristics using acoustic features from a second subset of microphones of multiple microphones 601. Process 600 can begin with multiple microphones 601 receiving a noisy audio signal. Acoustic features can be extracted from each audio channel, such as by transforming the audio waveform into a spectrum-time representation using time-frequency analysis. For example, at step 602, a first subset of acoustic features (such as amplitude spectrum distribution information) can be extracted from a first subset of audio channels associated with one or more microphones in multiple microphones 601. At step 604, a speech prediction model (such as the one described above with respect to Figure 4 The speech prediction model 402 described above estimates the multiplicative speech masking from the first subset of acoustic features. As described in the present disclosure, the speech prediction model can be trained using the reference truth clean speech data. At step 606, the estimated speech masking can be used to estimate the speech features. For example, as described above with respect to Figure 4 As described, after extracting the speech mask from the first subset of acoustic features, the estimated speech mask can be combined (such as multiplied) with the first reference microphone signal to estimate the speech features of the audio data. Thus, the estimated speech mask and the noisy speech acoustic features from the first subset can be used to calculate an estimate of the clean speech acoustic features.

[0084] At step 608, a second subset of acoustic features (such as amplitude spectrum distribution information) may be extracted from a second subset of audio channels associated with one or more microphones in the plurality of microphones 601. At step 610, a noise prediction model (such as described above with respect to Figure 4 The noise prediction model 404) estimates the multiplicative noise mask from the second subset of acoustic features. As described in the present disclosure, the noise prediction model can be trained using the reference truth noise data. In step 612, the estimated noise mask can be used to estimate the noise features. For example, as described above with respect to Figure 4 As described, after extracting the noise mask from the second subset of acoustic features, the estimated noise mask can be combined (such as multiplied) with the second reference microphone signal to estimate the noise features of the audio data. The estimated noise mask and the noisy speech acoustic features from the second subset can be used to calculate an estimate of the noisy acoustic features.

[0085] By filtering the speech and noise from a reference microphone audio in the subset, the speech and noise information can be fused together for further processing at step 614. The fused estimated speech and noise features can involve a cascade of information, or other information fusion techniques such as partial processing and layer-by-layer multiplication can be used. At step 416, the fused speech and noise features can be provided to a speech filtering model, such as described above with respect to Figure 4 The speech filter model 406 described. At step 416, a multiplicative filter mask can be estimated from the speech and noise features. In one or more embodiments of the present disclosure, the estimated filter mask can be SNR-related and can be estimated using the estimated SNR of each frequency component based on the speech and noise features. In this way, instead of directly learning clean speech as a target, the speech and noise components of each frequency bin can be estimated as learnable intermediate parameters to construct an SNR-related filter mask. The estimation of the final SNR-related filter mask can be improved using the predicted speech and noise components. In other words, there is no need to directly learn clean speech, but the process 400 can be used to learn intermediate masking / filtering, which provides a smaller machine learning model and allows deployment on devices with lower memory and lower computing resources.

[0086] At step 616, speech enhancement may be performed using the filter mask to create a clean / denoised audio signal, such as for use by one or more other machine learning models such as a wake-up word detection model or an ASR / NLU model. At step 616, the filter mask may be combined (such as multiplied) with a third reference microphone signal (such as one of the noisy audio signals received from microphone 601) to extract a denoised / clean audio signal from the reference microphone signal.

[0087] although Figure 6 One example of a multi-channel masking estimation and audio denoising process 600 is shown, but may be used for Figure 6 Various changes are made. For example, steps 602-606 and steps 608-612 can be performed sequentially or substantially simultaneously. In addition, the individual models described in the present disclosure (such as speech prediction model 402, noise prediction model 404 and speech filter model 406) can be stored as individual models called by processor 120 to perform certain steps of process 600, or can be included in one or more larger machine learning models and form a part thereof. In addition, a distributed architecture can be used to perform process 600. For example, speech prediction model, noise prediction model and speech filter model can be performed on a client electronic device (such as electronic device 101) or at a server (such as server 106). When executed by a server, the server can provide a determined action to be performed by the client electronic device to the client electronic device. In one or more embodiments of the present disclosure, the client electronic device can provide the audio data received via the audio input device of the client electronic device to the server, and audio denoising can be performed by the server using speech prediction model, noise prediction model and speech filter model, and one or more ASR / NLU models can be performed by the client electronic device or server based on denoised audio data.

[0088] Figure 7 An example audio denoising and post-filtering process 700 according to an embodiment of the present disclosure is shown. For ease of explanation, the process 700 may be described as involving Figure 1 The electronic device 101 is used in the network configuration 100 of FIG. However, the process 700 can be used with any other suitable electronic device (such as server 106) or combination of devices (such as electronic device 101 and server 106), and can be used in any other suitable system.

[0089] In one or more embodiments of the present disclosure, one or more machine learning models, such as Figure 2 Machine learning model 202, Figure 3 Machine learning model 302 or Figure 4 The models 402-406 may be accompanied by at least one other small footprint post-filtering model. Figure 7As shown, process 700 may include one or more machine learning denoising models 702, which may receive a noisy audio stream 701 and denoise / clean the noisy audio signal, as described in the present disclosure. Process 700 may include a post-filtering model 704 that receives the denoised audio signal from the machine learning denoising model 702. At step 706, the post-filtering model 704 may enhance the denoised audio signal and output an enhanced audio signal. The post-filtering model 704 may apply various techniques to further enhance the audio signal, such as by applying a Wiener filter, spectral and / or temporal smoothing, beamforming, algorithms based on dictionary learning (such as non-negative matrix decomposition), etc. The post-filtering operation performed by the post-filtering model 704 may remove residual noise from the clean audio signal output by the machine learning model 704 for even further improved noise reduction.

[0090] As an example, FIG. 8A to FIG. 8C An example audio signal waveform and accompanying spectrogram according to an embodiment of the present disclosure are shown. Specifically, Fig. 8A An example audio signal waveform 801 and an accompanying spectrogram 802 generated without using a denoising model are shown. Figure 8B shows an example audio signal waveform 806 and accompanying spectrogram 807 generated using a denoising model according to the present disclosure, and Figure 8C An example audio signal waveform 811 and accompanying spectrogram 812 generated using a denoising model and a post-filtering model according to the present disclosure are shown. It should be understood that the audio signal waveforms 801, 806, 811 and the accompanying spectrograms 802, 807 and 812 are merely examples and do not limit the scope of the present disclosure.

[0091] FIG. 8A to FIG. 8C The audio signal results shown in may correspond to a two-microphone system receiving noisy human speech and processing the output in real time. Fig. 8A As shown, the positions of speech 803 and 805 and the position of noise 804 may be difficult to locate due to the overall noisy nature of the audio, as shown by the irregular shape of waveform 801. Figure 8B As shown in waveform 806 of FIG. 1 , the use of a denoising model as described in the present disclosure can produce a clear distinction between locations 808 and 810 corresponding to human speech and location 809 corresponding to silence between speech. Noise in both speech and silence locations can be removed, while speech is also emphasized, as can be seen by comparing spectrogram 807 with spectrogram 802. Figure 8C As shown, further noise reduction can be achieved by using a post-filtering model (such as post-filtering model 704). Figure 8C As shown, noise can be further reduced, and the difference between the speech positions 813 and 815 and the silence position 814 can be further emphasized and enhanced.

[0092] although Figure 7 One example of an audio denoising and post-filtering process 700 is shown, but may be used for Figure 7 Various changes can be made. For example, a distributed architecture can also be used to perform process 700. For example, the individual models described in the present disclosure (such as those included in the machine learning model 702 and the post-filtering model 704) can be stored as individual models called by the processor 120 to perform certain tasks, or can be included in one or more larger machine learning models and form part of one or more larger machine learning models. In addition, a distributed architecture can be used to perform process 700. For example, the machine learning model 702 and the post-filtering model 704 can be executed on a client electronic device (such as the electronic device 101) or at a server (such as the server 106). When executed by the server, the server can provide the client electronic device with a determined action to be performed by the client electronic device. In one or more embodiments of the present disclosure, the client electronic device can provide audio data received via an audio input device of the client electronic device to the server, audio denoising and post-filtering can be performed by the server, and one or more ASR / NLU models can be executed by the client electronic device or the server based on the denoised audio data. Although FIG. 8A to FIG. 8C An example of an audio signal waveform and accompanying spectrogram is shown, but FIG. 8A to FIG. 8C For example, the audio data may vary widely depending on the situation, and FIG. 8A to FIG. 8C The present disclosure is not limited to any particular type of audio data or to any particular type of results obtained using embodiments of the present disclosure.

[0093] Fig.9A and Fig. 9B An example method 900 for training and using one or more denoising machine learning models according to an embodiment of the present disclosure is shown. For ease of explanation, Fig.9A and Fig. 9B The method 900 shown in FIG. 1 can be described as using Figure 1 The method 900 may be performed by a combination of a server 106 and an electronic device 101 in the network configuration 100. As an example, a portion of the method 900 related to model training may be performed on the server 106, and a portion of the method 900 related to inference may be performed on the electronic device 101 after the trained machine learning model is deployed to the electronic device 101. However, the method 900 may be used with any other suitable device (such as only one of the electronic device 101 or the server 106), and may be used in any other suitable system.

[0094] At box 902, a processor of a training device may collect multi-channel noisy and clean audio data samples for supervised training of one or more denoising models (such as the speech model 502, noise model 504, and filter model 506 described above with respect to FIG. 5). In one or more cases, this may be accomplished by recording real-world audio using a target electronic device or synthesizing data using a room acoustic simulation process. At box 904, a processor of a training device may perform a time-frequency analysis to transform the waveform of the audio data sample into a spectrum-time representation, such as by performing a short-time Fourier transform, a wavelet transform, a least squares spectral analysis, or another transform function on the waveform of the audio data.

[0095] Considering that the denoising system of the present disclosure can be constrained to a smaller number of parameters than existing deep learning models, in some cases, it may be advantageous to train each model of the system with inputs that are most closely similar to the reference truth in order to teach the model what speech and noise look like. To achieve this, training can be divided into multiple stages. For example, the processor can use two training stages to progressively train the machine learning model. At box 906, the processor of the training device can train the speech model and the noise model in the time-frequency domain to predict speech masking and noise masking, respectively. In the second training stage, the processor of the training device can jointly train the entire denoising network to estimate the final SNR-related filter masking.

[0096] Once the denoising model is trained, the denoising model can be deployed to another electronic device and can be executed by the processor of the electronic device. For example, at box 910, the processor of the inference device can use one or more audio input devices to record a noisy audio waveform. At box 912, the processor of the inference device can transform the recorded raw audio data into a time-frequency representation, such as by performing a short-time Fourier transform, a wavelet transform, a least squares spectral analysis, or another transform function on the waveform of the recorded raw audio data. At box 914, the processor of the inference device can use the trained denoising model in the transform domain to estimate the filter mask by taking the acoustic features extracted from the noisy data as input. At box 916, the processor of the inference device can apply the estimated filter mask to the reference microphone audio input to perform denoising, thereby creating an enhanced and denoised audio signal. At box 918, the processor of the inference device can convert the enhanced audio signal back to a time domain audio waveform. The denoised audio can be used to perform at least one more accurate speech recognition process, such as the wake-up word detection and wake-up process and / or ASR / NLU process described in the present disclosure.

[0097] although Fig.9A and Fig. 9BOne example of a method 900 for training and using one or more denoising machine learning models is shown, but may be used for Fig.9A and Fig. 9B For example, although shown as a series of steps, Fig.9A and Fig. 9B The various steps in may overlap, occur in parallel, occur in a different order, or occur any number of times. As a specific example, blocks 906 and 908 may be performed any number of times, or include additional sub-steps, such as using a loss function to perform two-stage training of the model. In addition, block 918 may include an additional step of performing post-filtering on the enhanced audio signal, as described in embodiments of the present disclosure.

[0098] Fig. 10A and Fig. 10B An example method 1000 for two-stage denoising model training according to an embodiment of the present disclosure is shown. For ease of explanation, Fig. 10A and Fig. 10B The method 1000 shown in FIG. 1 can be described as using Figure 1 The method 1000 may be performed by the server 106 in the network configuration 100. As an example, the method 1000 may be performed by the server 106 in the network configuration 100. Figure 1 The method 1000 is executed on a server 106 in a network configuration 100, and one or more trained machine learning models (such as trained speech prediction, noise prediction, and speech filtering models) can be deployed to a client electronic device 101 for use. However, the method 1000 can be used with any other suitable device (such as the electronic device 101) and can be used in any other suitable system.

[0099] As described above, considering that the denoising system of the present disclosure can be constrained to a smaller number of parameters than the existing deep learning model, in some cases, it may be advantageous to train each model of the system with the input that is most closely similar to the reference truth value so as to teach the model what speech and noise look like. To achieve this, method 1000 can divide the training into multiple stages to progressively train the machine learning model. At box 1002, the processor can start the first training stage, wherein a speech masking prediction model (such as the speech model 502 in Figure 5) and a noise masking prediction model (such as the noise model 504 in Figure 5) are trained to predict speech masking and noise masking respectively. This can be done without performing training of a filter masking prediction model (such as the filter model 506 in Figure 5). As part of box 1002, the processor can access one or more training samples, which include one or more of (i) noisy speech signals, (ii) clean speech signals, and (iii) noise signals. Each training sample can be associated with a reference truth voice masking and / or a reference truth noise masking.

[0100] At block 1004, the processor may input one or more training samples into a speech masking prediction model and a noise masking prediction model, and receive output speech masking and noise masking from the models, respectively. During the first phase of training, the processor may update the speech masking prediction model and the noise masking prediction model, respectively, at blocks 1006 and 1008, such as by using a loss function to reduce or minimize the output speech masking (M S ) and the difference or loss between the reference truth speech mask and reduce or minimize the output noise mask (M V ) and the noise mask of the ground truth. In one or more cases, this can be expressed as For example, when the output of the speech masking prediction model and / or the noise masking prediction model differs from the ground truth, the difference can be used to calculate the loss defined by the loss function. The loss function can use any suitable measure of the loss associated with the output generated by the speech masking prediction model and the noise masking prediction model, such as cross entropy loss or mean squared error.

[0101] At decision box 1010, the processor can determine whether the first phase training is completed. If not, the method 1000 can loop back to box 1002. For example, multiple training iterations can be performed during the first phase training until the determination that the first phase training of the speech masking prediction model and the noise masking prediction model is completed is reached, such as when determining that the model provides predictions with an acceptable level of accuracy. When the loss calculated by the loss function is greater than the expectation, the parameters of the model can be adjusted. Once adjusted, the same or additional training data can be provided to the adjusted model, and the additional output from the model can be compared with the reference true value so that the loss function can be used to determine the additional loss. At a certain moment, at decision box 1010, the processor can determine that the measured loss drops below the specified threshold, and the first phase training can be completed. The first training phase can allow the speech masking prediction model and the noise masking prediction model to learn to accurately predict their associated masking before training the filter masking prediction model.

[0102] After the first training phase, Fig. 10BAs shown, the final stage of the system can be trained during the second training phase. At box 1012, the processor can access one or more training samples associated with the baseline truth filter mask. At box 1014, the processor can input one or more training samples into the speech masking prediction model and the noise masking prediction model, and receive the output speech mask and the output noise mask. At box 1016, the processor can input the predicted speech features and noise features into the filter masking prediction model using the predicted speech mask and noise mask. At box 1018, the processor can jointly train the speech masking prediction model, the noise masking prediction model, and the filter masking prediction model to predict the filter mask (W) using one or more training samples associated with the baseline truth filter mask. The speech model 502, the noise model 504, and the filter model 506 can be updated during the second stage training, such as by using a loss function, to reduce or minimize the difference between the predicted filter mask and the baseline truth filter mask. In one or more cases, this can be expressed as The processor may use a loss function to calculate the error or loss associated with the prediction of the final filter mask output by the denoising system. For example, when a filter mask output by a filter mask prediction model is received, the difference between the filter mask and the ground truth may be used to calculate the loss defined by the loss function. The loss function may use any suitable measure of the loss associated with the filter mask output, such as a cross entropy loss or a mean squared error.

[0103] At decision box 1020, the processor can determine whether the second stage training is completed. If not, the method 1000 can loop back to box 1012. For example, multiple training iterations can be performed during the second stage training until the determination of the completion of the second stage joint training of the speech masking prediction model, the noise masking prediction model and the filter masking prediction model is reached, such as when the determination model provides the filter masking prediction with an acceptable level of accuracy. When the loss calculated by the loss function is greater than the expectation, one or more parameters in the model can be adjusted. Once adjusted, the same or additional training data can be provided to the adjusted model, and the additional filter masking output can be compared with the reference true value so that the loss function can be used to determine the additional loss. At a certain moment, at decision box 1020, the processor can determine that the measured loss drops below the specified threshold, and the second stage training can be completed. In this way, the method 1000 can involve training the system progressively from the initial model and then moving downstream. Since the initial speech masking prediction model and the noise masking prediction model have been trained, the audio input of the downstream model is very similar to the reference truth, which greatly improves the learning and accuracy of the entire denoising system. At block 1022, the trained denoising model can be deployed to, for example, electronic device 101.

[0104] although Fig. 10A and Fig. 10B An example of method 1000 for two-stage denoising model training is shown, but it can be used for Fig. 10A and Fig. 10B For example, although shown as a series of steps, Fig. 10A and Fig. 10B The steps in can overlap, occur in parallel, occur in a different order, or occur any number of times.

[0105] Fig.11 An example method 1100 for audio signal denoising according to an embodiment of the present disclosure is shown. For ease of explanation, Fig.11 The method 1100 shown can be described as using Figure 1 The method 1100 may be performed by the electronic device 101 in the network configuration 100. However, the method 1100 may be performed using any other suitable device (such as the server 106) and in any other suitable system.

[0106] At block 1102, the processor may obtain a noisy speech signal from one or more audio input devices. At block 1104, the processor may extract acoustic features from the noisy speech signal. The processor may provide the extracted acoustic features to a speech masking prediction model (such as speech prediction model 402) and a noise masking prediction model (such as noise prediction model 404). As described in the present disclosure, the speech masking prediction model and the noise masking prediction model may be trained to process the acoustic features and estimate speech and noise intermediate variables to improve the final audio quality. For example, based on the acoustic features of the audio input, the speech prediction model may be trained to estimate the speech masking (M S ), and a noise prediction model can be trained to estimate the noise masking (M V ). These estimated speech masking and noise masking can reflect the SNR of each frequency bin (the spacing between samples in the frequency domain).

[0107] At box 1106, the processor can receive predicted speech masking from a speech masking prediction model based on a first subset of acoustic features. At box 1108, the processor can receive predicted noise masking from a noise masking prediction model based on a second subset of acoustic features. As described above, in one or more embodiments of the present disclosure, method 1100 can use a speech masking prediction model and a noise masking prediction model to model speech and noise respectively by utilizing two subsets (such as microphones) of an audio device. Therefore, method 1100 can utilize multi-channel / multi-microphone raw data input, and can utilize the above-mentioned dual-branch speech and noise model architecture to apply a separate microphone data subset to a speech prediction model and a noise prediction model. However, in an embodiment of the present disclosure, a single microphone can be used.

[0108] At block 1110, the processor may determine predicted speech features using the predicted speech masking, and may determine predicted noise features using the predicted noise masking. For example, in one or more embodiments of the present disclosure, the predicted speech masking may be used to extract speech amplitude distribution information from a reference audio signal, and the predicted noise masking may be used to extract noise amplitude distribution information from another reference audio signal. For example, in determining the speech masking (M) from the acoustic feature subset, the processor may determine predicted speech features using the predicted speech masking (M) from the acoustic feature subset. S ) and noise masking (M V ) can then be used to mask the estimated speech (M S ) is combined (such as multiplied) with the first reference audio signal, and the estimated noise mask (M V ) is combined (such as multiplied) with a second reference audio signal to determine predicted speech and noise features. In one or more embodiments of the present disclosure, at box 1112, the processor may fuse the predicted speech features and the predicted noise features, and provide the fused features to the filtering masking prediction model. By filtering speech and noise from a reference audio in the subset, the speech and noise information may be fused together for further processing by the speech filtering model. In one or more embodiments of the present disclosure, fusing speech and noise features may involve a cascade of information, or other information fusion techniques may be used, such as partial processing and layer-by-layer multiplication.

[0109] At box 1114, the processor can receive the predicted filter mask output by the filter mask prediction model. For example, the speech filter model can receive the fused speech and noise features and output filter mask (W). Filter mask (W) can be SNR related, and can be estimated based on the estimated SNR of each frequency component using the speech and noise features. In this way, instead of directly learning clean speech as a target, the speech and noise features of each frequency bin can be estimated as learnable intermediate parameters to construct SNR related filter mask. The estimation of final SNR related filter mask (W) can be improved by using the predicted speech and noise components. At box 1116, the processor can use the predicted filter mask to generate a clean / de-noised speech signal, such as by applying the predicted filter mask (via multiplication or other techniques) to another reference noisy speech signal received using an audio input device.

[0110] At decision block 1118, the processor may determine whether to perform post-filtering on the clean speech signal. If not, the method 1100 may end at block 1122. If yes, then at block 1120, the processor may perform post-filtering on the clean speech signal to remove residual noise from the clean speech signal, such as described above with respect to Figure 7Described. Method 1100 can then end at box 1122. The denoised / clean audio signal can be used to perform at least one more accurate speech recognition process, such as wake-up word detection and wake-up process and / or ASR / NLU process. In one or more embodiments of the present disclosure, the masking estimation performed by method 1100 can provide a smaller machine learning model, which allows deployment on devices with lower memory and lower computing resources while still achieving improved accuracy over existing models.

[0111] although Fig.11 An example of a method 1100 for denoising an audio signal is shown, but the Fig.11 For example, although shown as a series of steps, Fig.11 The various steps in can overlap, occur in parallel, occur in different orders, or occur any number of times. In addition, a distributed architecture can be used to perform method 1100. For example, a speech prediction model, a noise prediction model, and a speech filter model can be executed on a client electronic device (such as electronic device 101) or at a server (such as server 106). When executed by a server, the server may provide a determined action to be performed by the client electronic device to the client electronic device. In one or more embodiments of the present disclosure, the client electronic device may provide audio data received via an audio input device of the client electronic device to the server, audio denoising may be performed by the server using a speech prediction model, a noise prediction model, and a speech filter model, and one or more ASR / NLU models may be performed by the client electronic device or the server based on the denoised audio data.

[0112] Fig.12 1 shows a method 1200 executed by an electronic device according to an embodiment of the present disclosure. Fig.12 According to an embodiment of the present disclosure, a method 1200 performed by an electronic device (eg, at least one processor of a device) may include operations 1210 to 1240. The method 1200 is not limited to Fig.12 The method shown in , and may also include Fig.12 Operations not shown in FIG.

[0113] In operation 1210, the electronic device may obtain a noisy speech signal. In an embodiment of the present invention, the electronic device may obtain the noisy speech signal using a microphone. In an embodiment of the present invention, the electronic device may receive the noisy speech signal from another device or a server.

[0114] In operation 1220, the electronic device may generate a predicted voice masking based on a first subset of noisy speech signals using a voice masking prediction model, and generate a predicted noise masking based on a second subset of noisy speech signals using a noise masking prediction model. In an embodiment of the present disclosure, the electronic device may generate the predicted voice masking and the predicted noise masking simultaneously, in parallel, or sequentially. In an embodiment of the present disclosure, the first subset of noisy speech signals is obtained using a first set of microphones, and the second subset of noisy speech signals is obtained using a second set of microphones.

[0115] In operation 1230, the electronic device may generate a predicted filter mask based on the predicted speech mask and the predicted noise mask using a filter mask prediction model.

[0116] In operation 1240, the electronic device may generate a clean speech signal using the predicted filter mask. In an embodiment of the present disclosure, the electronic device may extract the predicted speech feature using the predicted speech mask, and extract the predicted noise feature using the predicted noise mask. In an embodiment of the present disclosure, the electronic device may generate the predicted filter mask based on the predicted speech feature and the predicted noise feature.

[0117] The computer-readable storage medium may be provided in the form of a non-transitory storage medium. In this regard, the term "non-transitory" simply means that the storage medium does not include a signal and is a tangible device, and the term does not distinguish between where data is semi-permanently stored in the storage medium and where data is temporarily stored in the storage medium. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0118] In addition, the program according to the embodiments disclosed in this specification may be included in a computer program product when provided. The computer program product may be traded between a seller and a buyer as a product. For example, the computer program product may be distributed in the form of a computer-readable storage medium (e.g., a compact disc ROM (CD-ROM)), or via an application store (e.g., Google Play Store TM ) online distribution (e.g., downloading or uploading) or directly between two user devices (e.g., smart phones). For online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be at least temporarily stored or temporarily created on a computer-readable storage medium (such as a memory of a manufacturer's server, an application store's server, or a relay server).

[0119] In an embodiment of the present invention, a method performed by an electronic device may include obtaining a noisy speech signal. In an embodiment of the present disclosure, the method may include generating a predicted speech mask based on a first subset of the noisy speech signal using a speech mask prediction model, and generating a predicted noise mask based on a second subset of the noisy speech signal using a noise mask prediction model. In an embodiment of the present disclosure, the method may include generating a predicted filter mask based on the predicted speech mask and the predicted noise mask using a filter mask prediction model. In an embodiment of the present disclosure, the method may include generating a clean speech signal using the predicted filter mask.

[0120] In an embodiment of the present disclosure, the generation of the clean speech signal may include applying a predicted filter mask to a reference noisy speech signal.

[0121] In an embodiment of the present disclosure, a first subset of acoustic features may be extracted from a first subset of noisy speech signals obtained using a first set of microphones. In an embodiment of the present disclosure, a second subset of acoustic features may be extracted from a second subset of noisy speech signals obtained using a second set of microphones.

[0122] In an embodiment of the present disclosure, the generation of predicted speech masking and predicted noise masking may include extracting acoustic features from a noisy speech signal. In an embodiment of the present disclosure, the generation of predicted speech masking and predicted noise masking may include generating predicted speech masking based on a first subset of acoustic features extracted from a first subset of noisy speech signals using a speech masking prediction model. In an embodiment of the present disclosure, the generation of predicted speech masking and predicted noise masking may include generating predicted noise masking based on a second subset of acoustic features extracted from a second subset of noisy speech signals using a noise masking prediction model. In an embodiment of the present disclosure, a first set of microphones may be used to obtain a first subset of noisy speech signals. In an embodiment of the present disclosure, a second set of microphones may be used to obtain a second subset of noisy speech signals.

[0123] In an embodiment of the present disclosure, the generation of a predicted filter mask may include extracting predicted speech features using a predicted speech mask and extracting predicted noise features using a predicted noise mask. In an embodiment of the present disclosure, the generation of a predicted filter mask may include generating a predicted filter mask based on predicted speech features and predicted noise features. In an embodiment of the present disclosure, the generation of a predicted filter mask may include fusing predicted speech features and predicted noise features.

[0124] In an embodiment of the present disclosure, the method may include training a speech masking prediction model and a noise masking prediction model in a first training stage. In an embodiment of the present disclosure, the training of the speech masking prediction model and the noise masking prediction model may include identifying one or more training samples including one or more of (i) noisy speech signals, (ii) clean speech signals, and (iii) noise signals. In an embodiment of the present disclosure, one or more training samples may be associated with a reference truth speech masking and a reference truth noise masking. In an embodiment of the present disclosure, the training of the speech masking prediction model and the noise masking prediction model may include updating the speech masking prediction model to reduce the difference between the output speech masking and the reference truth speech masking. In an embodiment of the present disclosure, the training of the speech masking prediction model and the noise masking prediction model may include updating the noise masking prediction model to reduce the difference between the output noise masking and the reference truth noise masking.

[0125] In an embodiment of the present disclosure, the method may include training a speech masking prediction model, a noise masking prediction model, and a filter masking prediction model in a second training stage after the first training stage. In an embodiment of the present disclosure, the training of the speech masking prediction model, the noise masking prediction model, and the filter masking prediction model may include identifying one or more training samples. In an embodiment of the present disclosure, one or more training samples may be further associated with a reference truth filter masking. In an embodiment of the present disclosure, the training of the speech masking prediction model, the noise masking prediction model, and the filter masking prediction model may include updating the speech masking prediction model, the noise masking prediction model, and the filter masking prediction model to reduce the difference between the predicted filter masking and the reference truth filter masking.

[0126] In an embodiment of the present disclosure, the predicted speech masking can be used to extract speech amplitude distribution information. In an embodiment of the present disclosure, the predicted noise masking can be used to extract noise amplitude distribution information.

[0127] In an embodiment of the present disclosure, the method may include performing post-filtering on the clean speech signal to remove residual noise from the clean speech signal.

[0128] In an embodiment of the present invention, a device may include at least one processor configured to obtain a noisy speech signal. In an embodiment of the present disclosure, at least one processor may be configured to extract acoustic features from the noisy speech signal. In an embodiment of the present disclosure, at least one processor may be configured to generate a predicted speech masking based on a first subset of the noisy speech signal using a speech masking prediction model, and to generate a predicted noise masking based on a second subset of the noisy speech signal using a noise masking prediction model. In an embodiment of the present disclosure, at least one processor may be configured to generate a predicted filter masking based on the predicted speech masking and the predicted noise masking using a filter masking prediction model. In an embodiment of the present disclosure, at least one processor may be configured to generate a clean speech signal using the predicted filter masking.

[0129] In an embodiment of the present disclosure, at least one processor may be configured to apply a predicted filter mask to a reference noisy speech signal.

[0130] In an embodiment of the present disclosure, at least one processor may be configured to extract acoustic features from a noisy speech signal. In an embodiment of the present disclosure, at least one processor may be configured to generate a predicted speech masking based on a first subset of acoustic features extracted from a first subset of noisy speech signals using a speech masking prediction model. In an embodiment of the present disclosure, at least one processor may be configured to generate a predicted noise masking based on a second subset of acoustic features extracted from a second subset of noisy speech signals using a noise masking prediction model. In an embodiment of the present disclosure, a first subset of noisy speech signals may be obtained using a first set of microphones. In an embodiment of the present disclosure, a second subset of noisy speech signals may be obtained using a second set of microphones.

[0131] In an embodiment of the present disclosure, a first subset of acoustic features may be extracted from a first subset of noisy speech signals obtained using a first set of microphones. In an embodiment of the present disclosure, a second subset of acoustic features may be extracted from a second subset of noisy speech signals obtained using a second set of microphones.

[0132] In an embodiment of the present disclosure, at least one processor may be configured to extract predicted speech features using predicted speech masking, and to extract predicted noise features using predicted noise masking. In an embodiment of the present disclosure, at least one processor may be configured to generate a predicted filter mask based on the predicted speech features and the predicted noise features.

[0133] In an embodiment of the present disclosure, at least one processor may be configured to train a speech masking prediction model and a noise masking prediction model in a first training phase. In an embodiment of the present disclosure, in order to train the speech masking prediction model and the noise masking prediction model, at least one processor may be configured to identify one or more training samples including one or more of (i) noisy speech signals, (ii) clean speech signals, and (iii) noise signals. In an embodiment of the present disclosure, one or more training samples may be associated with a reference truth voice masking and a reference truth noise masking. In an embodiment of the present disclosure, at least one processor may be configured to update the speech masking prediction model to reduce the difference between the output speech masking and the reference truth voice masking. In an embodiment of the present disclosure, at least one processor may be configured to update the noise masking prediction model to reduce the difference between the output noise masking and the reference truth noise masking.

[0134] In an embodiment of the present disclosure, at least one processor may be configured to train a speech masking prediction model, a noise masking prediction model, and a filter masking prediction model in a second training phase after the first training phase. In an embodiment of the present disclosure, in order to train the speech masking prediction model, the noise masking prediction model, and the filter masking prediction model, at least one processor may be configured to identify one or more training samples, wherein the one or more training samples are also associated with a reference truth filter mask. In an embodiment of the present disclosure, at least one processor may be configured to update the speech masking prediction model, the noise masking prediction model, and the filter masking prediction model to reduce the difference between the predicted filter mask and the reference truth filter mask.

[0135] In an embodiment of the present disclosure, at least one processor may be configured to extract speech amplitude distribution information using predicted speech masking. In an embodiment of the present disclosure, at least one processor may be configured to extract noise amplitude distribution information using predicted noise masking.

[0136] In an embodiment of the present disclosure, at least one processor may be configured to perform post-filtering on the clean speech signal to remove residual noise from the clean speech signal.

[0137] In an embodiment of the present disclosure, a computer-readable medium may include instructions that, when executed, cause at least one processor to obtain a noisy speech signal. In an embodiment of the present disclosure, a computer-readable medium may include instructions that, when executed, cause at least one processor to generate a predicted speech mask based on a first subset of the noisy speech signal using a speech mask prediction model, and to generate a predicted noise mask based on a second subset of the noisy speech signal using a noise mask prediction model. In an embodiment of the present disclosure, a computer-readable medium may include instructions that, when executed, cause at least one processor to generate a predicted filter mask based on a predicted speech mask and a predicted noise mask using a filter mask prediction model. In an embodiment of the present disclosure, a computer-readable medium may include instructions that, when executed, cause at least one processor to generate a clean speech signal using the predicted filter mask.

[0138] Although the present disclosure has been described with exemplary embodiments, various changes and modifications may be suggested to one skilled in the art. The present disclosure is intended to encompass such changes and modifications as fall within the scope of the appended claims.

Claims

1. A method performed by an electronic device (101), comprising: Obtaining a noisy speech signal (301); generating a predicted speech mask based on a first subset of the noisy speech signals using a speech mask prediction model, and generating a predicted noise mask based on a second subset of the noisy speech signals using a noise mask prediction model; generating a predicted filter mask based on the predicted speech mask and the predicted noise mask using a filter mask prediction model; and A clean speech signal is generated using the predicted filter mask (306).

2. The method according to claim 1, wherein: The generation of the clean speech signal includes applying the predicted filter mask to the reference noisy speech signal (304).

3. The method according to claim 1 or 2, in, The generation of the predicted speech masker and the predicted noise masker includes: Extract acoustic features from noisy speech signals; generating a predicted speech masker based on a first subset of acoustic features extracted from the first subset of noisy speech signals using a speech masker prediction model; and generating a predicted noise mask based on a second subset of acoustic features extracted from a second subset of noisy speech signals using the noise mask prediction model, and The first subset of noisy speech signals is obtained using a first set of microphones, and the second subset of noisy speech signals is obtained using a second set of microphones.

4. The method according to any one of claims 1 to 3, in, The generation of the predicted filter mask includes: extracting predicted speech features using the predicted speech masker, and extracting predicted noise features using the predicted noise masker, and A predicted filter mask is generated based on the predicted speech characteristics and the predicted noise characteristics.

5. The method according to any one of claims 1 to 4, further comprising: In the first training stage, the speech masking prediction model and the noise masking prediction model are trained, wherein the training of the speech masking prediction model and the noise masking prediction model includes: identifying one or more training samples comprising one or more of (i) a noisy speech signal, (ii) a clean speech signal, and (iii) a noise signal, wherein the one or more training samples are associated with a ground truth speech mask and a ground truth noise mask; updating the speech masking prediction model to reduce the difference between the output speech masking and the reference truth speech masking; and The noise mask prediction model is updated to reduce the difference between the output noise mask and the reference true noise mask.

6. The method according to claim 5, further comprising: In a second training phase after the first training phase, a speech masking prediction model, a noise masking prediction model and a filter masking prediction model are trained, wherein the training of the speech masking prediction model, the noise masking prediction model and the filter masking prediction model includes: identifying one or more training samples, wherein the one or more training samples are also associated with a ground truth filter mask; and The speech masking prediction model, the noise masking prediction model and the filter masking prediction model are updated to reduce the difference between the predicted filter masking and the reference true value filter masking.

7. The method according to any one of claims 1 to 6, further comprising: Post filtering is performed on the clean speech signal to remove residual noise from the clean speech signal.

8. An apparatus comprising: At least one processor (120) is configured to: Obtaining a noisy speech signal (301); generating a predicted speech mask based on a first subset of the noisy speech signals using a speech mask prediction model, and generating a predicted noise mask based on a second subset of the noisy speech signals using a noise mask prediction model; generating a predicted filter mask based on the predicted speech mask and the predicted noise mask using a filter mask prediction model; and A clean speech signal is generated using the predicted filter mask (306).

9. The device according to claim 8, wherein: The at least one processor (120) is configured to apply a predicted filter mask to a reference noisy speech signal (304).

10. The device according to claim 8 or 9, in, The at least one processing device is further configured to: Extract acoustic features from noisy speech signals; generating a predicted speech masker based on a first subset of acoustic features extracted from a first subset of noisy speech signals using a speech masker prediction model; as well as generating a predicted noise mask based on a second subset of acoustic features extracted from a second subset of noisy speech signals using the noise mask prediction model, and The first subset of noisy speech signals is obtained using a first set of microphones, and the second subset of noisy speech signals is obtained using a second set of microphones.

11. The device according to any one of claims 8 to 10, in, The at least one processing device is further configured to: extracting predicted speech features using the predicted speech masker, and extracting predicted noise features using the predicted noise masker, and A predicted filter mask is generated based on the predicted speech characteristics and the predicted noise characteristics.

12. The device according to any one of claims 8 to 11, wherein: The at least one processor (120) is further configured to train a speech masking prediction model and a noise masking prediction model in a first training phase; and In order to train the speech masking prediction model and the noise masking prediction model, the at least one processor (120) is configured to: identifying one or more training samples comprising one or more of (i) a noisy speech signal, (ii) a clean speech signal, and (iii) a noise signal, wherein the one or more training samples are associated with a ground truth speech mask and a ground truth noise mask; updating the speech masking prediction model to reduce the difference between the output speech masking and the reference truth speech masking; and The noise mask prediction model is updated to reduce the difference between the output noise mask and the reference true noise mask.

13. The device according to claim 12, wherein: The at least one processor (120) is further configured to train the speech masking prediction model, the noise masking prediction model and the filter masking prediction model in a second training phase after the first training phase; and In order to train the speech masking prediction model, the noise masking prediction model and the filter masking prediction model, the at least one processor (120) is configured to: identifying one or more training samples, wherein the one or more training samples are also associated with a ground truth filter mask; and The speech masking prediction model, the noise masking prediction model and the filter masking prediction model are updated to reduce the difference between the predicted filter masking and the reference true value filter masking.

14. The device according to any one of claims 8 to 13, wherein: The at least one processor (120) device is also configured to perform post filtering on the clean speech signal to remove residual noise from the clean speech signal.

15. A computer readable medium comprising instructions which, when executed, cause at least one processor to: Obtaining a noisy speech signal; generating a predicted speech mask based on a first subset of the noisy speech signals using a speech mask prediction model, and generating a predicted noise mask based on a second subset of the noisy speech signals using a noise mask prediction model; generating a predicted filter mask based on the predicted speech mask and the predicted noise mask using a filter mask prediction model; and A clean speech signal is generated using the predicted filter mask (306).

Images