Adaptive energy limiting for transient noise suppression

Abstract

The present disclosure describes aspects of adaptive energy limiting for transient noise suppression. In some aspects, an adaptive energy limiter sets a limiter upper limit of an audio signal to full scale and receives a portion of the audio signal. For the portion of the audio signal, the adaptive energy limiter determines a maximum amplitude and evaluates the portion with a neural network to provide a speech likelihood estimate. Based on the maximum amplitude and the speech likelihood estimate, the adaptive energy limiter determines that the portion of the audio signal includes noise. In response to determining that the portion of the audio signal includes noise, the adaptive energy limiter reduces the limiter upper limit and provides the limiter upper limit to a limiter module to effectively limit the amount of energy of the audio signal. This can effectively prevent the audio signal from bringing full energy transient noise into the conference audio.

Description

Background Art

[0001] An audio conference or video conference typically includes many participants, one or several of whom are actively speaking at any given time. When not speaking, other participants typically generate noise that may be picked up by their microphones and fed into the audio of the conference for all participants to hear. Examples of noise generated by conference participants may include typing on a keyboard, placing a coffee cup on a table, flipping papers, moving a chair, closing a door, and the like. Some of these noises have transient characteristics, i.e., unlike static or recurring noises, these noises with transient characteristics cannot be suppressed by conventional noise reduction techniques. In addition, the audio energy of transient noises is typically as high or higher than the energy level associated with the voices of conference participants. Therefore, these transient noises are typically fed into the conference audio as unsuppressed energy, resulting in noise that may disturb speakers and listeners, suppress speakers' voices, trigger residual echo suppression, erroneously trigger audio or video switching schemes, and the like. Summary of the invention

[0002] The present disclosure describes an apparatus and technology for adaptive energy limitation of transient noise suppression. In some aspects, a method for adaptive energy limitation includes setting a limiter upper limit of an audio signal to full scale and receiving a portion of the audio signal. The method then determines the maximum amplitude of the portion of the audio signal and evaluates the portion of the audio signal with a neural network to provide a speech likelihood estimate for the portion of the audio signal. Based on the maximum amplitude and the speech likelihood estimate, the method determines that the portion of the audio signal includes noise. In response to determining that the portion of the audio signal includes noise, the method reduces the limiter upper limit. The limiter upper limit is then provided to a limiter module that the audio signal is to pass through to limit the amount of energy of the audio signal. By doing so, the audio signal can be prevented from carrying full energy transient noise into a conference audio or a subsequent audio process (e.g., speaker selection for a video conference).

[0003] In other aspects, a device includes a network interface and a limiter module, the network interface is used to receive or send an audio signal through a data network, and the limiter module is used to limit the energy of the audio signal. The device also includes a hardware-based processor and a storage medium, the hardware-based processor is associated with the data interface, and the storage medium stores processor executable instructions for an adaptive energy limiter. The adaptive energy limiter is implemented to set the limiter upper limit of the audio signal to full scale, and from the audio signal, provide a frame of audio corresponding to the audio duration from the audio signal. Then, the adaptive energy limiter determines the maximum amplitude of the audio signal for the frame of the audio, and evaluates the frame of the audio with a neural network to provide a speech likelihood estimate for the frame of the audio. Based on the maximum amplitude and the speech likelihood estimate, the adaptive energy limiter determines that the frame of the audio includes noise. The adaptive energy limiter then reduces the limiter upper limit in response to the determination that the frame of the audio includes noise, and provides the limiter upper limit to the limiter module to reduce the energy of the audio signal.

[0004] In other aspects, a system includes a hardware-based processor and a storage medium, the hardware-based processor being operably associated with an audio interface or a data interface, the storage medium storing processor executable instructions for an adaptive energy limiter, wherein an audio signal is received via the audio interface or the data interface. The adaptive energy limiter is implemented to set the limiter upper limit of the audio signal to full scale, and to generate a frame of audio corresponding to the audio duration from the audio signal based on the audio signal. The adaptive energy limiter then determines the maximum amplitude of the audio signal for the frame of the audio, and evaluates the frame of the audio with a neural network to provide a speech likelihood estimate for the frame of the audio. Based on the maximum amplitude and the speech likelihood estimate, the adaptive energy limiter determines that the frame of the audio includes noise. The adaptive energy limiter then lowers the limiter upper limit in response to the determination that the frame of the audio includes noise, and provides the limiter upper limit to the limiter module to reduce the energy of the audio signal.

[0005] The details of one or more embodiments of adaptive energy limiting for transient noise suppression are set forth in the accompanying drawings and the following description. Other features and advantages will be apparent from the specification and drawings, and from the claims. This summary is provided to introduce the subject matter that is further described in the detailed description and the drawings. Therefore, this summary should not be considered to describe essential features, nor should it be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] This specification describes apparatus and techniques for adaptive energy limitation for transient noise suppression with reference to the following drawings. In all drawings, the same reference numerals are used to refer to the same features and components:

[0007] Figure 1An example conferencing environment is shown in which various aspects of adaptive energy limiting for transient noise suppression may be implemented.

[0008] Figure 2 An example device diagram of user devices and conferencing devices including respective instances of an adaptive energy limiter according to one or more aspects is shown.

[0009] Figure 3 Example configurations of components capable of implementing various aspects of adaptive energy limiting are shown.

[0010] Figure 4 Example methods for adaptively limiting the energy of an audio signal in accordance with one or more aspects are shown.

[0011] Figure 5A and Figure 5B An example method of scaling an audio signal based on instantaneous speech likelihood provided by a neural network-enabled voice activity detector is shown.

[0012] Figure 6 An example graph is shown for limiting the energy of an audio signal according to one or more aspects.

[0013] Figure 7 A system diagram of components for implementing adaptive energy limiting for transient noise suppression according to one or more aspects is shown. DETAILED DESCRIPTION

[0014] Overview

[0015] An audio conference or video conference typically includes many participants, one or several of whom are actively speaking at any given time. When not speaking, other participants typically generate noise that may be picked up by their microphones and fed into the audio of the conference for all participants to hear. Examples of noise generated by conference participants may include typing on a keyboard, placing a coffee cup on a table, flipping papers, moving a chair, closing a door, and the like. Some of these noises have transient characteristics, i.e., unlike static or recurring noises, these noises with transient characteristics cannot be suppressed by conventional noise reduction techniques. In addition, the audio energy of transient noises is typically as high or higher than the energy level associated with the voices of conference participants. Therefore, these transient noises are typically fed into the conference audio as raw, unsuppressed energy, resulting in noise that may disturb speakers and listeners, suppress speakers' voices, trigger residual echo suppression, erroneously trigger audio or video switching schemes, and the like.

[0016] Because conventional noise reduction techniques cannot mitigate transient noise, there are multiple negative consequences that affect conference call participants. Typically, unsuppressed noise is transmitted to the other end of the call, disrupting the speaker and other listeners. When this unsuppressed noise is transmitted to the current speaker in the call, it may also trigger residual echo suppression, which dampens the speaker's voice or affects back-end speaker selection schemes such as top-3 filtering (e.g., passing the corresponding audio of the three call participants with the greatest energy). In addition, the conferencing system may erroneously prioritize noisy participants over actively speaking participants, or interrupt video switching schemes by switching the speaker's video feed to the participant generating the noise.

[0017] Some conventional techniques involve having participants who are not currently speaking manually mute their corresponding microphones. However, mute solutions are unwelcome and inconvenient because they result in unnatural conversational flows and often cause problems when participants forget to unmute their microphones before speaking. Manually muting microphones is particularly frustrating in large conference rooms where many participants take turns speaking because muting can occur very frequently. For example, every time someone wants to speak to another participant, the person needs to reach for a button on a remote control or device to unmute the microphone, and then remember to mute it again afterwards. Therefore, manual muting, which relies on timely manual interaction from all participants, is inconvenient and is generally ineffective in suppressing transient noise.

[0018] Other conventional techniques also typically fail to prevent transient noise from entering the conference audio, or do so at the expense of other impairments to call flow or quality. For example, some phones include noise gates that automatically mute unless there is strong energy in the audio stream. However, these noise gates can result in unstable audio quality and often allow high-energy noise to enter the conference audio. Other noise reduction techniques only work for stationary or slightly non-stationary noise (e.g., fans, traffic, background noise), and not for sudden, non-constant, and high-energy transient noise. In other cases, keyboard suppression predicts when keyboard sounds will occur and selectively suppresses them. This suppression is limited to typing on the same laptop computer that is hosting the meeting, and only applies to keyboard noise. Therefore, conventional noise suppression techniques for conference calls cannot suppress or limit transient noise, which often interferes with call flow and quality.

[0019] This document describes an apparatus and technology for adaptive energy limitation of transient noise suppression. As described above, participants in a conference call may generate transient noise, which, when allowed to enter the conference audio, will usually disturb the speaker and other participants. Transient noise may also interfere with or degrade the conference service process for audio and video features, such as the audio stream or video stream selection presented to other participants (e.g., active speakers). Generally, aspects of adaptive energy limitation manage or control the maximum energy level that a participant is allowed to contribute based on the history of the participant's recent noise or voice generation. In various aspects, an adaptive energy limiter of a user device or conference system sets the limiter upper limit of an audio signal to full scale and receives a portion of the audio signal. For this portion of the audio signal, the adaptive energy limiter determines the maximum amplitude and evaluates the portion with a neural network to provide a speech likelihood estimate. Based on the maximum amplitude and the speech likelihood estimate, the adaptive energy limiter determines that this portion of the audio signal includes noise. In response to determining that this portion of the audio signal includes noise, the adaptive energy limiter reduces the limiter upper limit and provides the limiter upper limit to a limiter module to effectively limit the amount of energy of the audio signal. By doing so, the adaptive energy limiter can prevent the audio signal from carrying full energy transient noise into the conference audio or subsequent audio processes (eg, speaker selection for a video conference).

[0020] For example, if a participant makes noise, the adaptive energy limiter will gradually reduce the upper limit of energy allowed to pass. Typically, this will cause sudden noises generated by participants in the future to become less intrusive and more easily ignored by other conference service algorithms, such as speaker selection for video conferencing. In some aspects, after about 10 to 15 seconds of medium or high energy noise, the upper limit of audio energy is reduced to a minimum level, after which the audio energy (e.g., noise energy) from the participant will be very limited. When the participant does start speaking, the adaptive energy limiter can reset the upper limit of audio energy to a maximum level (e.g., voice level or full scale) to allow the participant's voice audio to pass to other conference participants. The adaptive energy limiter does this very quickly so that the transient noise suppression provided by the adaptive energy limiter has almost no harmful effect on the voice audio of the conference call. Alternatively or additionally, if the participant is silent, quiet or emits low-energy background noise, the adaptive energy limiter can maintain or keep the upper limit of audio energy high so as not to affect the voice audio when the participant starts speaking.

[0021] In general, aspects of adaptive energy limiting for transient noise suppression limit the energy of transient noise without compromising the voice audio quality of a conference call or voice call. For example, by using the long-term statistical properties of noise and / or voice in the context of an audio or video conferencing scenario, an adaptive energy limiter can significantly reduce the amount or impact of transient noise while minimally affecting voice. In other words, the adaptive noise limiter does not attempt to remove noise from concurrent noise and voice, which is a typical problem with conventional noise reduction techniques when attempting to remove noise, especially noise that may be confused with voice.

[0022] In various aspects of adaptive energy limiting, the amplitude of the audio signal is measured over a period of time, and along with other described exploits of statistical properties, an upper limiter on the audio energy is configured to prevent or suppress transient noise from entering the conference call. In some cases, a neural network is implemented to provide statistical properties about the audio signal. According to various aspects, a small neural network has sufficient accuracy for such a task so that no special acceleration hardware is required and the voice quality is not affected by the accuracy limitations of the neural network or associated voice activity detector (VAD). Alternatively or additionally, an adaptive energy manager can be implemented to adjust or manage the gain or subband gain of the audio signal based on the audio signal evaluation described herein.

[0023] In this way, various aspects of energy limitation (or energy management) can be implemented to limit or reduce the amount of energy that an audio signal can bring into a conference service or out to a conference call participant through a conference call. In other words, for each participant, an adaptive energy limiter can track the noise debt that accumulates as the participant continues to make noise. As the noise debt accumulates (or the reduction of energy limitation), the adaptive energy limiter prevents or does not allow the participant to send a large amount of energy to the call until the participant proves that they are sending voice (e.g., by sending a statistically large amount of voice audio). The adaptive energy limiter can also effectively suppress transient noise by using (e.g., via a neural network) the statistical energy difference of transient noise (e.g., high energy), vowels (e.g., medium energy) and consonants (e.g., low energy) to allow voice (e.g., consonants) to pass through unaffected in perception even when transient noise is reduced by 20dB or more. The aspects of adaptive energy limitation can achieve this effect by using a limiter upper limit of the audio signal energy and / or by managing the subband gain used to process the audio signal of the participant.

[0024] Although any number of different environments, systems, devices, and / or various configurations may implement the features and concepts of the described techniques and apparatus for adaptive energy limiting for transient noise suppression, various aspects of adaptive energy limiting for transient noise suppression are described in the context of the following example environments, devices, configurations, methods, and systems.

[0025] Example Environment

[0026] Figure 1 An example environment 100 is shown in which various aspects of adaptive energy limiting for transient noise suppression can be implemented. In the example environment 100, a user device 102 can transmit audio and / or video through a conference system 104, where access to the system is provided by a conference service 106 (e.g., a cloud-based conference service or conference service). The user devices 102 in this example include a smartphone 102-1, a laptop computer 1022, a tablet computer 102-3, a smart watch 102-4, a phone 102-5, a conference bridge 102-6, and a video conference display 102-7. Although illustrated as a device, the user device may be implemented as any suitable computing or electronic device, such as a mobile communication device, a computing device, a client device, an entertainment device, a gaming device, a mobile gaming console, a personal media device, a media playback device, a charging station, an advanced driver assistance system (ADAS), a point-of-sale (POS) transaction system, a health monitoring device, a drone, a camera, a wearable smart device, a navigation device, a mobile Internet device (MID), an Internet home appliance with wireless Internet access and browsing capabilities, an Internet of Things (IoT) device, a fifth generation new radio (5G NR) user equipment, and / or other types of user devices.

[0027] Typically, the corresponding user of the user device 102 can interact with other users through the audio and / or video data exchanged by the data or voice connection of the conference service 106. In some aspects, each user device 102 participating in the conference call instance facilitated by the conference service 106 provides an audio signal 108 and / or a video signal through a corresponding connection with the conference service. For example, any or all user devices 102 can provide a channel of an audio signal 108 (or audio data) corresponding to the audio captured by the microphone of the device. During the conference call, participants usually take turns to speak, while other inactive or non-speaking participants listen or watch. However, some of these participants may choose to move their chairs, write emails or take notes on their computers. This movement and typing activity may generate transient noise, which may include sounds or sound waves with short pulse-shaped signal characteristics. Other potential sources of transient noise may include click noises from a computer mouse, moving items on a table or work surface, closing doors, phone keys or phone rings, etc. For example, if two participants (each at a respective endpoint of a conference or voice call) are located near each other in an open office, one of the participants is using smartphone 102-1 and the other is using laptop 102-2, when the participant using laptop 102-2 begins typing, potential transient noise may be generated at both endpoints.

[0028] In terms of adaptive energy limiting for transient noise suppression, the conference service 106 includes an instance of an adaptive energy limiter 110 (adaptive limiter 110), which can limit or manage the energy of an audio signal to suppress various forms of transient noise. Although described with reference to the conference service 106, any or all user devices 102 may also include an instance of the adaptive energy limiter 110. Therefore, the adaptive energy limiter 110 can limit or manage the energy of an audio signal sent to the conference service 106, processed by the conference service 106, or sent by the conference service to other user devices 102. The adaptive energy limiter 110 is associated with or has access to a neural network 112, which can be implemented as a recursive neural network (RNN). In this example, the neural network 112 includes a voice activity detector 114 (VAD 114), which can be configured to provide an indication of the speech likelihood of an audio signal or a frame of audio. For example, the adaptive energy limiter 110 can use the voice activity detector 114 to obtain an indication of the speech likelihood of a frame of audio. Such an indication may be useful for determining whether an audio signal or a frame of audio is more likely to be speech or noise.Alternatively or additionally, the voice activity detector 114 may be implemented as a neural network enabled voice activity detector that uses a neural network to determine or provide a speech likelihood measure for a sample of an audio signal or an audio frame.

[0029] Figure 2 At 200, an example device diagram of a user device 102 and a conference device 202 that can provide a conference service 106 is shown. Although each device is shown with an instance of an adaptive energy limiter, aspects of adaptive energy limiting can be implemented on one device, on both devices, or in coordination between devices. For example, the adaptive energy limiter 110 of the user device 102 can interact with the adaptive energy limiter 110 or the neural network 112 of the conference device 202 to set the limiter upper limit value at the user device 102. In the exemplary configuration shown, the user device 102 or the conference device 202 can also include an adaptive energy limiter that is not provided for clarity or visual simplicity. Figure 2 Alternatively or additionally, any corresponding component of the user device 102 or the conference device 202 may be implemented in whole or in part as hardware logic or circuitry that is integrated or separate from other components.

[0030] In this example, the user device 102 includes a network interface 204 for exchanging data, such as audio signals or video streams, through various types of networks or communication protocols. Typically, the network interface 204 can be implemented as any one or more of a serial and / or parallel interface, a wireless interface, a wired interface, or a modem for sending or receiving data or signals. In some cases, the network interface 204 provides a connection and / or a communication link between the user device 102 and the communication network, through which other user devices 102 and conference devices 202 transmit audio signals 108, video data, etc. for conference media communications. The user device 102 also includes at least one microphone 206 for capturing audio (e.g., voice, sound, or noise) from the environment of the user device 102, and at least one speaker 208 for generating audio or sound based on the audio data of the user device 102. In some aspects, the microphone captures audio generated by the user, such as voice, and provides the audio signal to the audio circuit (not shown) of the user device 102 for encoding or other signal processing.

[0031] The user device 102 also includes a processor 210 and a computer readable storage medium 212 (CRM 212). The processor 210 can be a single-core processor or a multi-core processor composed of a variety of materials such as silicon, polysilicon, high-k dielectrics, etc. The computer readable storage medium 212 is configured as a memory and therefore does not include temporary signals or carrier waves. The CRM 212 can include any suitable memory or storage device, such as a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a non-volatile random access memory (NVRAM), a read-only memory (ROM), or a flash memory that can be used to store device data 214 of the user device 102.

[0032] Device data 214 may include user data, multimedia data (e.g., audio data or video data), applications 216 (e.g., media conference client applications), user interfaces 218, and / or operating systems of user devices 102, which can be accessed or executed by processor 210 to enable audio or video conferencing and / or other user interactions with user devices 102. User interface 218 may be configured to receive input from a user of user device 102, such as receiving input from a user that may define and / or facilitate one or more aspects of adaptive energy limitation for transient noise suppression. User interface 218 may include a graphical user interface (GUI) that receives input information via touch input. In other cases, user interface 218 includes an intelligent assistant that receives input information via audible input. Alternatively or additionally, the operating system of user device 102 may be maintained as firmware or an application on CRM 212 and executed by processor 210.

[0033] The CRM 212 also includes an adaptive energy limiter 110, a neural network 112, and a voice activity detector 114. In various aspects, the adaptive energy limiter 110 uses the neural network 112 and / or the voice activity detector 114 (VAD 114) to determine whether the audio signal includes speech or noise. Based on this determination, the adaptive energy limiter 110 can lower the upper limiter limiter to limit the energy of the noise that would otherwise disrupt the conference call or voice call if allowed to pass at full energy. The implementation and use of the adaptive energy limiter 110, the neural network 112, and / or the voice activity detector 114 are varied and described throughout the disclosure.

[0034] Aspects and functions of the user device 102 may be managed via operating system controls presented through at least one application programming interface 220 (API 220). In some aspects, the adaptive energy limiter 110 or an application of the user device 102 accesses the API 220 or API service of the user device 102 to control aspects and functions of an audio or video conferencing application. For example, the adaptive energy limiter 110 may access low-level audio processor settings of the user device 102 to implement aspects of adaptive energy limiting, such as setting a minimum limiter cap level, adjusting audio gain settings, managing respective signal levels of incoming and outgoing audio signals, etc. The CRM 212 of the user device 102 may also include a user device manager 222, which may be implemented in whole or in part as hardware logic or circuitry integrated or separate from other components of the user device 102. In at least some aspects, the user device manager 222 configures the microphone 206 and other audio circuits of the user device 102 to implement the transient noise suppression techniques described herein.

[0035] User device 102 also includes a display 224 for displaying and / or providing information or a video feed to a user. For example, through display 224, user device 102 can provide a user with a video feed from a video conference enabled by conferencing service 106. Alternatively or additionally, user device 102 can also include a camera (not shown) to enable generation of a video feed for a multimedia conference from user device 102.

[0036] The conference device 202 may be implemented as a computing device, server, cloud-based hardware, or other resource used to provide the conference service 106 to the user device 102. In general, the conference device 202 may act as a collector and / or arbitrator of multimedia data or streams for a conference call instance. In this way, the conference device 202 may implement aspects of adaptive energy limiting with respect to inbound audio data received from the user device 102, internal multimedia processing operations, or outbound audio data sent to the user device 102 as part of a conference or voice call.

[0037] In this example, the conference device 202 includes a network interface 226 for exchanging data, such as audio signals or video streams, through various types of networks or communication protocols. Typically, the network interface 226 can be implemented as any one or more of a serial and / or parallel interface, a wireless interface, a wired interface, or a modem for sending or receiving data or signals. In some cases, the network interface 226 provides a connection and / or communication link between the conference device 202 and a communication network through which the user device 102 transmits audio signals 108, video data, etc. for conference media communications.

[0038] In this example, the conference device 202 also includes a processor 228 or computing resource, and a computer readable storage medium 230 (CRM 230). The computer readable storage medium 230 is configured as a memory and therefore does not include a transient signal or carrier wave. The CRM 230 may include any suitable memory or storage device that can be used to store multimedia data 232 of the conference device 202, such as RAM, SRAM, DRAM, NVRAM, ROM, or flash memory.

[0039] The multimedia data 232 of the conference device 202 may include audio data, audio signals, or video data that facilitates a conference call through an instance of the conference service 106. The multimedia data 232 and the conference service 106 and other applications (e.g., a media conference server application) and / or an operating system of the conference device 202 can be accessed or executed by the processor 228 to enable audio or video conferencing of multiple user devices 102.

[0040] In this example, the CRM 230 also includes instances of the adaptive energy limiter 110, the neural network 112, and the voice activity detector 114. As mentioned above, aspects of adaptive energy limiting can be implemented by the user device 102, the conference device 202, or a combination of the two devices. In various aspects, the adaptive energy limiter 110 uses the neural network 112 and / or the voice activity detector 114 to determine whether one or more audio signals include speech or noise. Based on this determination, the adaptive energy limiter 110 of the conference device 202 can reduce the limiter upper limit of the corresponding audio signal or audio feed to limit the noise energy that would otherwise disturb the conference call or voice call if the noise was allowed to pass at full energy. The implementation and use of the adaptive energy limiter 110, the neural network 112, and / or the voice activity detector 114 are varied and are described throughout the disclosure.

[0041] Aspects and functionality of the conferencing device 202 may be managed via system controls presented through at least one application programming interface (API) of the API library 234. In some aspects, the adaptive energy limiter 110 or an application of the conferencing device 202 accesses an API or library of the API library 234 to implement aspects of transient noise limiting. For example, the adaptive energy limiter 110 may be implemented as part of or in conjunction with a network-based real-time communications library.

[0042] Figure 3 An example configuration of components capable of implementing various aspects of adaptive energy limiting is shown at 300. In general, Figure 3 The components may be embodied on the user device 102, the conference device 202, or a combination thereof. In some aspects, the components shown at 300 are implemented as integrated components of a device (e.g., a system on a chip) and / or combined with a memory storing processor executable instructions to provide the corresponding functions of one or more components. Therefore, Figure 3 The configuration of components shown in is non-limiting and may be implemented in any suitable device, combination of devices, and / or as hardware (e.g., logic circuitry) in combination with firmware or software to provide the described functionality.

[0043] In some aspects, the audio signal 108 is segmented or divided into audio frames 302 corresponding to respective portions of the audio signal. For example, each of the audio frames 302 may correspond to a portion, segment, or duration of audio (e.g., speech and / or noise) of the audio signal 108. In some cases, the audio frames 302 or frames of audio correspond to a range of approximately 5 milliseconds to 50 milliseconds of audio (e.g., 10 milliseconds of audio). Alternatively or additionally, the audio frames 302 may be converted from the time domain to the frequency domain, e.g., to enable spectral analysis or other frequency domain-based processing.

[0044] like Figure 3 As shown, the example components include an amplitude detector 304 and a neural network 112, the neural network 112 including or providing a voice activity detector 114 for processing an audio frame 302. In general, the amplitude detector 304 measures or determines the amplitude of the audio signal 108 corresponding to the audio frame. For example, the amplitude detector 304 can generate or provide an indication of a maximum amplitude 306 of an audio frame or portion of an audio signal. In some aspects, the adaptive energy limiter 110 determines or updates an average amplitude 308 (e.g., a moving average) of the audio signal 108 or audio frame 302 based on a plurality of maximum amplitudes 306.

[0045] The neural network 112 may be implemented as a network operating on a processor of the user device 102 to provide a speech likelihood estimate for the audio frame 302. Alternatively or additionally, the neural network 112 may be implemented as a recurrent neural network (RNN) or a machine learning model with a memory (e.g., RNNoise). In some aspects, the voice activity detector 114 provides an instantaneous voice likelihood 310 (IVL 310) for one or more of the audio frames. Although described as a neural network-enabled voice activity detector, other types of voice activity detection or voice classification may also be used.

[0046] For example, the neural network 112 and / or the voice activity detector 114 may be implemented as a neural network (e.g., a deep neural network (DNN)) that includes an input layer, an output layer, and one or more hidden intermediate layers located between the input layer and the output layer of the neural network. Any or all nodes of the neural network may then be fully connected or partially connected between the layers of the neural network. The voice activity detector 114 may be implemented with or by any type of neural network, such as a convolutional neural network including GoogleNet or a similar convolutional network. Alternatively or additionally, the voice activity detector 114 or machine-learned voice activity detection model may include any suitable recursive neural network (RNN) or any variant thereof. In general, the neural network 112 and / or the voice activity detector 114 employed by the adaptive energy limiter may also include any other supervised learning, unsupervised learning, reinforcement learning algorithms, etc.

[0047] In various aspects, the neural network 112 and / or the voice activity detector 114 associated with the adaptive energy limiter 110 can be implemented as a recurrent neural network (RNN) having connections between nodes, where the nodes form a loop to retain information from a previous portion of the input data sequence for a later portion of the input data sequence (e.g., a previous frame of audio of speech or noise generated by a participant). In other cases, the neural network 112 is implemented as a feedforward neural network having connections between nodes that do not form a loop between input data sequences. Alternatively, the neural network 112 can be implemented as a convolutional neural network (CNN) with a multi-layer perceptron, where each neuron in a particular layer is connected to all neurons in adjacent layers. In various aspects of adaptive energy limiting, the neural network 112 and / or the voice activity detector 114 can use a previous determination of noise or speech by a participant to predict or determine whether a subsequent frame of the audio signal includes speech or noise that can be suppressed.

[0048] In general, the neural network 112 can enable the determination of speech likelihood estimates that converge quickly to high statistical confidence, especially in the presence of vowels. Recall that transient noise typically has stronger full-band energy than vowels, and even more so than consonants in speech. Therefore, while utilizing the statistical confidence provided by the neural network 112, the adaptive energy limiter can use the historical noise or speech patterns of the participants to distinguish between noise, vowels, and consonants of speech. In other words, speech and noise tend to be bursty, that is, a participant who recently spoke is more likely to continue speaking in the near future (e.g., less than one second). Alternatively, a participant who recently generated noise is more likely to generate additional noise in the future. In some cases, any lag introduced by the adaptive energy limiter is imperceptible to the conference call participants, yet the neural network 112 can determine in retrospect (e.g., several 100 milliseconds) more accurately than in real time whether the audio of a frame or signal is noise or speech.

[0049] Based on one or more of the instantaneous speech likelihoods 310, the adaptive energy limiter 110 may determine an aggregate speech likelihood estimate 312 (ASLE 312) for the audio signal 108 or the audio 302. The aggregate speech likelihood estimate 312 may be configured or updated based on a current aggregate speech likelihood estimate 312 and / or a threshold for detecting speech or noise. For example, in some cases, the adaptive energy limiter 110 increases the aggregate speech likelihood estimate 312 in response to the instantaneous speech likelihood 310 exceeding the current aggregate speech likelihood estimate 312 and exceeding a threshold for speech detection. In other cases, the adaptive energy limiter 110 may decrease the aggregate speech likelihood estimate 312 in response to the instantaneous speech likelihood 310 not exceeding the current aggregate speech likelihood estimate 312 or not exceeding a threshold for speech detection.

[0050] The adaptive energy limiter 110 also includes or provides an upper limiter limiter 314 by which the energy of the audio signal 108 can be limited, for example, to suppress the energy of transient noise. Typically, the upper limiter limiter 314 is provided to an audio signal limiter module 316 through which the audio signal 108 passes before being transmitted to other audio components or processes. The audio signal limiter module 316 can pass the audio signal at a full scale (e.g., unreduced or unlimited) or a reduced scale or reduced amplitude as specified by the upper limiter limiter 314 set by the adaptive energy limiter 110. Figure 3 In the context of , based on the limiter cap 314 provided by the adaptive energy limiter 110, the audio signal limiter module 316 limits or reduces the energy of the audio signal 108 to provide or generate an energy-limited audio signal 318. In various aspects, the adaptive energy limiter 110 limits the energy of the audio signal determined to be noise or including noise in order to suppress the noise and possible future noise. The energy-limited audio signal 318 can then be sent to the audio-based processing 320 for subsequent processing or for other features (e.g., speaker selection) before being included in the conference audio 322 shared with other participants of the audio or video conference call.

[0051] Example Method

[0052] According to one or more aspects of adaptive energy limiting for transient noise suppression, reference Figure 4 , Figure 5A and Figure 5BExample methods 400 and 500 are described. In general, methods 400 and 500 illustrate a set of operations (or actions) performed in the order or combination of operations shown here, but are not necessarily limited to this. In addition, any one or more of these operations may be repeated, combined, reorganized, skipped, or chained to provide a wide range of additional and / or alternative methods. In the following discussion, reference may be made to Figure 1 Example conference environment 100, Figure 2 Example devices Figure 4 Example components of Figure 7 Example systems and / or Figure 1 Entities detailed in the description, reference to these is for example only. The techniques and apparatus described in this disclosure are not limited to embodiments or capabilities of one or more entities operating on one device.

[0053] Method 400 is a method performed by user device 102 or conference device 202. Method 400 limits the amount of energy of an audio signal to mitigate the effects associated with transient noise in a conference environment or other audio process (e.g., speaker selection for a video conference). In some aspects, the operations of method 400 are implemented by or using adaptive limiter 110, neural network 112, and / or voice activity detector 114 of user device 102 or conference device 202.

[0054] At 402, an upper limiter of an audio signal is set to full scale. In some cases, the upper limiter or limit value is set to full scale upon initialization of the adaptive energy limiter or in response to speech of a participant for whom the audio signal is being processed to suppress noise.

[0055] At 404, a portion of an audio signal is received. The portion of the audio signal may include a frame of audio, an audio frame, an audio signal segment, etc. In some cases, the audio signal is received and divided into frames of audio for analysis by the adaptive energy limiter. For example, a frame of audio may correspond to a range of approximately 5 milliseconds to 50 milliseconds of audio. Alternatively or additionally, the frames of audio may be converted from the time domain to the frequency domain to enable spectral analysis or other frequency domain-based processing.

[0056] At 406, a maximum amplitude of the portion of the audio signal is determined. The maximum amplitude may be determined for the portion of the audio signal corresponding to a frame of audio or an audio duration (e.g., 10 milliseconds). In some cases, the maximum amplitude of the audio signal is compared to a threshold to determine whether the participant is silent, quiet, or otherwise not making noise. Alternatively, if the audio signal is quiet or silent, the method 400 may return from operation 406 to operation 404. Thus, if and when a silent participant begins to speak, the voice energy of the silent participant will not decrease.

[0057] At 408, the portion of the audio signal is evaluated with a neural network to provide a speech likelihood estimate. In some aspects, the portion of the audio signal or the frame of audio is evaluated with a neural network or a neural network-enabled voice activity detector to provide an instantaneous speech likelihood for the portion of the audio signal or the audio frame. Generally, the instantaneous speech likelihood can indicate whether the audio stream is more likely to be speech or more likely to be noise, and the adaptive energy limiter will suppress the noise.

[0058] At 410, based on the maximum amplitude and speech likelihood estimates, it is determined whether the portion of the audio signal includes speech or noise. For example, if the maximum amplitude of the portion of the audio signal exceeds the moving average of the maximum amplitude (e.g., the maximum average plus a small correction value) and the instantaneous speech likelihood is less than 0.5 or 50% (indicating noise), it can be determined that the portion of the audio includes noise or is noise. Alternatively, if the maximum amplitude of the portion of the audio signal does not exceed the moving average of the maximum amplitude (e.g., the maximum average plus a small correction value) or the instantaneous speech likelihood is greater than 0.5 or 50%, it can be determined that the portion of the audio is not noise or speech (e.g., exceeds the maximum average and IVL is greater than 50%). Optionally, if it is determined that the portion of the audio signal is the voice of the participant or includes the voice of the participant, the method 400 can return from operation 410 to operation 402.

[0059] At 412, in response to determining that the portion of the audio signal includes noise, an upper limiter limit for the audio signal is reduced. In some aspects, the upper limiter limit is reduced by a specific rate or amount based on the aggregate speech likelihood estimate. For example, if the aggregate speech likelihood estimate is high, the upper limit is reduced by a small amount or slowly reduced toward a minimum upper limiter limit value. In other cases, when the aggregate speech likelihood estimate is low, the upper limit can be reduced by a large amount, or quickly reduced toward a minimum upper limiter limit value. Alternatively or additionally, the minimum upper limiter limit can be configured based on the aggregate speech likelihood estimate, an average of corresponding amplitudes of multiple portions of the audio signal, or an average of corresponding maximum amplitudes of multiple portions of the audio signal, e.g., representing a portion of the current energy estimated to be speech.

[0060] At 414, the limiter cap is provided to a limiter module through which the audio signal passes. The limiter module limits the amount of energy of the audio signal based on the limiter cap. By limiting the energy that the audio signal is allowed to send or carry into the conference audio environment, the adaptive energy limiting aspect can prevent full energy transient noise from entering the conference audio and disturbing participants and / or other audio-based processes.

[0061] Figure 5A and Figure 5B The method 500 is a method performed by the user device 102 or the conference device 202. The method 500 scales the audio signal to not exceed the limiter upper limit, which can effectively prevent the audio signal from carrying full energy transient noise into the conference audio environment. In some aspects, the operation of the method 500 is implemented by or using the adaptive limiter 110, the neural network 112, and / or the voice activity detector 114 of the user device 102 or the conference device 202.

[0062] At 502, an upper limiter of the audio signal is set to full scale (e.g., 1.0 or 100%). The upper limiter or limit value may be set to full scale upon initialization of the adaptive energy limiter, or reset to full scale in response to the speech of a participant for whom the audio signal is being processed to suppress noise.

[0063] At 504, a frame of audio corresponding to a portion of an audio signal is generated. In some cases, the audio signal is received and / or separated, segmented or otherwise divided into frames of audio for analysis by a voice activity detector and / or an adaptive energy limiter. In other cases, the audio frame may be received from an audio codec or other entity configured to provide a frame from an audio signal. For example, a frame of audio may correspond to a range of approximately 5 milliseconds to 50 milliseconds (e.g., 10 milliseconds) of audio. Alternatively or additionally, the frame of audio may be converted from the time domain to the frequency domain to enable spectral analysis or other frequency domain-based processing.

[0064] At 506, the frame of audio is evaluated with a neural network enabled voice activity detector to provide an instantaneous voice likelihood (IVL). In some aspects, the portion of the audio signal or a frame of audio is evaluated with a neural network or a neural network enabled voice activity detector to provide an instantaneous voice likelihood for the portion of the audio signal or the frame of audio. Generally, the instantaneous voice likelihood can indicate whether the audio stream is more likely to be speech or more likely to be noise, and the adaptive energy limiter will suppress the noise.

[0065] At 508, the maximum amplitude of the audio signal is recorded from the frame of the audio. The maximum amplitude can be determined or recorded for the audio signal duration or audio duration corresponding to one frame of the audio (e.g., 10 milliseconds). In some cases, the maximum amplitude of the audio signal is compared to a threshold value to determine whether the participant is silent, quiet, or otherwise not making noise. In this case, if the audio signal is quiet or silent, the method 500 can return to operation 504.

[0066] At 510, a moving average of the maximum amplitude of the audio signal is updated based on the maximum amplitude of the frames of the recorded audio. The moving average of the maximum amplitude can correspond to any suitable number of audio frames or audio duration, such as a range of approximately 100 milliseconds to 500 milliseconds.

[0067] like Figure 5B As shown at 512 in the operation 514, an aggregate voice likelihood estimate (ASLE) is determined based on the instantaneous voice likelihood (IVL) of the frame of the audio. The aggregate voice likelihood estimate can be determined or configured based on the current aggregate voice likelihood estimate and / or a threshold for detecting speech (or noise). In some cases, in response to the instantaneous voice likelihood exceeding the current aggregate voice likelihood estimate and the speech detection threshold, the aggregate voice likelihood estimate is increased. In other cases, in response to the instantaneous voice likelihood not exceeding the current aggregate voice likelihood estimate or the speech detection threshold, the aggregate voice likelihood estimate is reduced.

[0068] At 516, it is determined whether the maximum amplitude exceeds the moving average and whether the instantaneous speech likelihood indicates that the frame of audio is noise. For example, if the maximum amplitude of the portion of the audio signal exceeds the moving average of the maximum amplitude (e.g., the maximum average plus a small correction value), and the instantaneous speech likelihood is less than 0.5 or 50% (indicating noise), then the audio frame may include noise or be noise. Alternatively, if the maximum amplitude of the portion of the audio signal does not exceed the moving average of the maximum amplitude (e.g., the maximum average plus a small correction value), or the instantaneous speech likelihood is greater than 0.5 or 50%, then the audio frame may not include noise or may not be primarily noise.

[0069] Optionally, at 518, in response to the maximum amplitude not exceeding the moving average and / or the instantaneous speech likelihood not indicating that the frame of the audio is noise, the upper limit of the limiter is not reduced. Optionally, at 520, the upper limit of the limiter is reduced based on the aggregated voice likelihood estimate (ASLE). The upper limit of the limiter is reduced in response to the maximum amplitude exceeding the moving average and the IVL indicating that the frame of the audio is noise. Typically, the amount or rate at which the upper limit of the limiter is reduced is determined based on the aggregated voice likelihood estimate.

[0070] At 522, the current value of the limiter upper limit is provided to the limiter module, so that the audio signal is scaled to no more than the current value. The limiter module scales the quantity of the energy of the audio signal through the limiter module based on the limiter upper limit. By scaling or limiting the energy that the audio signal is allowed to send or carry in the conference audio environment, the aspect of adaptive energy limitation can prevent full energy transient noise from entering the conference audio and disturbing participants and / or other audio-based processes. From operation 522, method 500 can return to operation 504, to perform another iteration of method 500, thereby further limiting the energy of the audio signal, resetting the limiter upper limit or maintaining the limiter upper limit. In some aspects, the method 500 or process for adaptive energy limitation is iterated or repeated approximately every 5 milliseconds to 50 milliseconds (e.g., 10 milliseconds) to provide the response suppression of transient noise.

[0071] For example, consider Figure 6 , where a graph 600 illustrates aspects of adaptive energy limiting. In the context of a limiter module, the energy of an audio signal is delivered at full scale 602, or is limited to a minimum value 604 of a limiter upper limit. In this example, assume that an audio signal 606 is received from a participant that continuously produces a medium to high level of noise (no speech). Here, the adaptive energy limiter 110 can quickly limit the energy of the audio signal delivered to the conference audio environment to prevent the noise of the audio signal 606 from disturbing other participants of the conference call.

[0072] As another example, consider graph 608, which includes an audio signal 610 of another participant of a conference call. Here, assume that the participant is not speaking, but is not making too much noise either. The adaptive energy limiter 110 gradually limits the audio signal 610 until the participant begins speaking at 612. In response to detecting speech, the adaptive energy limiter 110 resets the upper limiter limiter to full scale 602 at 614, and does not begin limiting the energy of the audio signal 610 until the participant stops speaking at 616.

[0073] system

[0074] Figure 7 Various components of an example system 700 are shown, which may be implemented as a reference Figures 1 to 6Any type of user device 102 or conference device 202 described herein can be used to implement adaptive energy limiting for transient noise suppression. In some aspects, system 700 is implemented as a component of a user equipment device or a base station or is embodied on a user equipment device or a base station. For example, system 700 can be implemented as a system of hardware-based components, such as, but not limited to, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a system on a chip (SoC), a system-level package, a complex programmable logic device (CPLD), an audio codec, an audio processor, a coprocessor, a context hub, a communication coprocessor, a sensor coprocessor, etc.

[0075] System 700 includes a communication device 702 that implements wired and / or wireless communication of system data 704 (e.g., encoded audio data or audio signals). System data 704 or other system content may include configuration settings of the system, media content stored on the device, and / or information associated with a device user. The media content stored on the system 700 may include any type of audio, video, and / or image data. System 700 includes one or more data inputs 706, via which any type of data, media content, and / or input may be received, such as human speech, voice, interaction with a radar field, user-selectable input (explicit or implicit), messages, music, television media content, recorded video content, and any other type of audio, video, and / or image data received from any content and / or data source.

[0076] The system 700 also includes a communication interface 708, which can be implemented as any one or more of a serial and / or parallel interface, a wireless interface, a network interface, a modem, and any other type of communication interface. The communication interface 708 provides a connection and / or communication link between the system 700 and a communication network through which other electronic, computing, and communication devices communicate data with the system 700.

[0077] The system 700 includes one or more processors 710 (e.g., any of a microprocessor, controller, etc.) that process various computer executable instructions to control the operation of the system 700 and enable the techniques of adaptive energy limiting for transient noise suppression to be enabled or embodied therein. Alternatively or additionally, the system 700 may be implemented in any one or combination of hardware, firmware, or fixed logic circuitry in conjunction with processing and control circuitry, which is generally identified at 712. Although not shown, the system 700 may include a system bus or data transmission system that couples various components within the device. The system bus may include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any one of a variety of bus architectures.

[0078] The system 700 also includes a computer readable medium 714 (CRM 714), such as one or more storage devices that enable persistent and / or non-temporary data storage and therefore do not include temporary signals or carrier waves. Examples of CRM 714 include random access memory (RAM), non-volatile memory (e.g., any one or more of read-only memory (ROM), flash memory, EPROM, EEPROM, etc.), or a disk storage device. The disk storage device can be implemented as a magnetic or optical storage device, such as a hard drive, a recordable and / or rewritable compact disk (CD), any type of digital versatile disk (DVD), etc. The system 700 can also include a mass storage medium device (storage medium) 716 or a mass storage device interface. In this example, the system 700 also includes or can be implemented as an audio codec 722 to support the encoding or decoding of audio signals or audio data, such as encoding audio from a microphone to provide audio signals or audio data for conference services or voice calls.

[0079] The computer readable medium 714 provides a data storage mechanism to store device data, as well as various system applications 718 and any other type of information and / or data related to the operational aspects of the system 700. For example, the operating system 720 can be maintained as a computer application with the computer readable medium 714 executing on the processor 710. The system applications 718 can include system managers, such as any form of control applications, software applications, signal processing and control modules, code inherent to a specific device, abstract modules or gesture modules, etc. The system applications 718 also include system components and utilities that implement adaptive energy limiting for transient noise suppression, such as the adaptive limiter 110, the neural network 112, and the voice activity detector 114. Although not shown, one or more elements of the adaptive limiter 110, the neural network 112, or the voice activity detector 114 can be implemented in whole or in part by hardware or firmware.

[0080] According to an example, the present disclosure describes aspects of adaptive energy limiting for transient noise suppression. In some aspects, an adaptive energy limiter sets a limiter upper limit of an audio signal to full scale and receives a portion of an audio signal. For the portion of the audio signal, the adaptive energy limiter determines a maximum amplitude and evaluates the portion with a neural network to provide a speech likelihood estimate. Based on the maximum amplitude and the speech likelihood estimate, the adaptive energy limiter determines that the portion of the audio signal includes noise. In response to determining that the portion of the audio signal includes noise, the adaptive energy limiter reduces the limiter upper limit and provides the limiter upper limit to a limiter module that effectively limits the amount of energy of the audio signal. This can effectively prevent the audio signal from bringing full energy transient noise into the conference audio.

[0081] Although the above-described devices, systems, and methods are described in the context of adaptive energy limiting for transient noise suppression in an audio / video conferencing environment, the described devices, systems, or methods are non-limiting and may be applied to other contexts, user device deployments, or audio-based communication environments.

[0082] In addition to the above description, controls may be provided to the user, allowing the user to choose whether and when the systems, programs, and / or features described herein can collect user information (e.g., audio, sound, voice, speech, user preferences, user current location) and whether to send content and / or communications from a server to the user. In addition, certain data may be processed in one or more ways before being stored or used to remove personally identifiable information. For example, the identity of the user may be processed so that the user's personally identifiable information cannot be determined. For example, where location information is obtained, the user's geographic location (e.g., city, zip code, or state / province level) may be generalized so that the user's specific location cannot be determined. Therefore, the user can control what information (e.g., audio) is collected about the user, how the information is used, and what information is provided to the user.

[0083] Here are a few examples:

[0084] Example 1: A method comprising: setting an upper limiter limit of an audio signal to full scale; receiving a portion of the audio signal; determining a maximum amplitude of the portion of the audio signal; evaluating the portion of the audio signal using a neural network to provide a speech likelihood estimate for the portion of the audio signal; determining that the portion of the audio signal includes noise based on the maximum amplitude and the speech likelihood estimate; in response to determining that the portion of the audio signal includes noise, lowering the upper limiter limit; and providing the upper limiter limit to a limiter module through which the audio signal is to pass to limit the amount of energy of the audio signal.

[0085] Example 2: The method of Example 1, wherein the portion of the audio signal is a frame of audio corresponding to the portion of the audio signal, and the method further comprises: converting the frame of audio from the time domain to the frequency domain before evaluating the frame of audio.

[0086] Example 3: The method of Example 2, wherein the frames of the audio are frames of a first audio, and the method further comprises: receiving frames of a second audio corresponding to a second portion of the audio signal; evaluating the frames of the second audio with the neural network to provide corresponding speech likelihood estimates for the frames of the second audio; determining that the frames of the second audio include speech based on the corresponding speech likelihood estimates; and resetting the limiter upper limit to the full scale.

[0087] Example 4: A method according to Example 2, wherein the frames of the audio are frames of a first audio, and the method further includes: receiving frames of a second audio corresponding to a second portion of the audio signal; determining a corresponding maximum amplitude of the frames of the second audio; comparing the corresponding maximum amplitude of the frames of the second audio with a threshold, the threshold corresponding to an average of corresponding maximum amplitudes of multiple frames of audio, the multiple frames of audio corresponding to multiple corresponding portions of the audio signal; and maintaining a current limiter upper limit in response to the corresponding amplitude of the frames of the second audio not exceeding the threshold.

[0088] Example 5: The method of any of Examples 2 to 4, wherein the frames of the audio correspond to an audio duration ranging from approximately 10 milliseconds of the audio to approximately 50 milliseconds of the audio.

[0089] Example 6: A method according to any of Examples 1 to 5, wherein evaluating the portion of the audio signal with the neural network to provide the speech likelihood estimate includes analyzing the portion of the audio signal with a neural network-enabled voice activity detector (VAD) to provide an instantaneous speech likelihood (IVL) of the portion of the audio signal.

[0090] Example 7: A method according to Example 6, wherein the limiter upper limit is reduced by a predefined amount, and the method further includes: determining an aggregate speech likelihood estimate (ASLE) based on multiple IVLs provided by the neural network-enabled VAD; updating the aggregate speech likelihood estimate (ASLE) based on the IVL by: increasing the ASLE in response to the IVL exceeding the ASLE and exceeding a speech detection threshold; or reducing the ASLE in response to the IVL not exceeding the ASLE or not exceeding the speech detection threshold; and setting the predefined amount by which the limiter upper limit is reduced based on the ASLE.

[0091] Example 8: The method of Example 7, wherein the upper limiter limit has a minimum value, and the method further comprises configuring the minimum value of the upper limiter limit based on the ASLE.

[0092] Example 9: The method according to Example 8 also includes configuring a minimum value of the limiter upper limit based on the ASLE and one of the following: an average value of corresponding amplitudes of multiple parts of the audio signal; or an average value of corresponding maximum amplitudes of multiple parts of the audio signal.

[0093] Example 10: An apparatus comprising: a network interface for receiving or sending an audio signal over a data network; a limiter module for limiting the energy of the audio signal; a hardware-based processor associated with the data interface; and a storage medium storing processor-executable instructions, the processor-executable instructions, in response to being executed by the hardware-based processor, implementing an adaptive energy limiter to: set an upper limiter limiter of the audio signal to full scale; provide a frame of audio corresponding to an audio duration from the audio signal from the audio signal; determine a maximum amplitude of the audio signal for the frame of the audio; evaluate the frame of the audio with a neural network to provide a speech likelihood estimate for the frame of the audio; determine, based on the maximum amplitude and the speech likelihood estimate, that the frame of the audio includes noise; in response to determining that the frame of the audio includes noise, reduce the upper limiter limit; and provide the upper limiter limit to the limiter module to reduce the energy of the audio signal.

[0094] Example 11: The apparatus of Example 10, wherein the adaptive energy limiter is further implemented to: capture frames of audio as part of the audio signal; and convert the frames of audio from the time domain to the frequency domain for evaluation by the neural network.

[0095] Example 12: An apparatus according to Example 11, wherein the frames of the audio are frames of the first audio, and the adaptive energy limiter is further implemented to: capture frames of second audio corresponding to a second portion of the audio signal; convert the frames of the second audio from the time domain to the frequency domain; evaluate the frames of the second audio of the audio signal with the neural network to provide corresponding speech likelihood estimates for the frames of the second audio; determine that the frames of the second audio include speech based on the corresponding speech likelihood estimates; and reset the limiter upper limit to the full scale.

[0096] Example 13: An apparatus according to Example 11, wherein the frames of the audio are frames of the first audio, and the adaptive energy limiter is further implemented as: capturing frames of second audio corresponding to a second portion of the audio signal; determining a corresponding maximum amplitude of the frames of the second audio; comparing the corresponding maximum amplitude of the frames of the second audio with a threshold, the threshold corresponding to an average of corresponding maximum amplitudes of multiple frames of audio, the multiple frames of audio corresponding to multiple corresponding portions of the audio signal; and maintaining the limiter upper limit at a current level in response to the corresponding amplitude of the frames of the second audio not exceeding the threshold.

[0097] Example 14: An apparatus according to any one of Examples 11 to 13, wherein the frame of the audio corresponds to a duration of audio information from the audio signal, and the duration ranges from approximately 5 milliseconds of audio information to approximately 50 milliseconds of audio information.

[0098] Example 15: An apparatus according to any of Examples 10 to 14, wherein the neural network includes a voice activity detector (VAD) and the adaptive energy limiter is further implemented to use the VAD of the neural network to provide the speech likelihood estimate as an instantaneous speech likelihood (IVL) of the portion of the audio signal.

[0099] Example 16: An apparatus according to Example 15, wherein the adaptive energy limiter reduces the limiter upper limit by a predefined amount, and the adaptive energy limiter is further implemented as: determining an aggregate speech likelihood estimate (ASLE) based on multiple IVLs provided by the VAD of the neural network; updating the aggregate speech likelihood estimate (ASLE) based on the IVL in the following manner: in response to the IVL exceeding the ASLE and exceeding a speech detection threshold, increasing the ASLE; or in response to the IVL not exceeding the ASLE or not exceeding the speech detection threshold, reducing the ASLE; and setting the predefined amount by which the limiter upper limit is reduced based on the ASLE.

[0100] Example 17: The apparatus of Example 16, wherein the upper limiter limit has a minimum value, and the adaptive energy limiter is further implemented to configure the minimum value of the upper limiter limit based on the ASLE.

[0101] Example 18: An apparatus according to Example 17, wherein the adaptive energy limiter is further implemented to configure a minimum value of the limiter upper limit based on the ASLE and one of: an average value of corresponding amplitudes of multiple parts of the audio signal; or an average value of corresponding maximum amplitudes of multiple parts of the audio signal.

[0102] Example 19: A system comprising: a hardware-based processor, the hardware-based processor being operably associated with an audio interface or a data interface, receiving an audio signal through the audio interface or the data interface; and a storage medium, the storage medium storing processor-executable instructions, the processor-executable instructions, in response to execution by the hardware-based processor, implementing an adaptive energy limiter to: set an upper limiter limiter of the audio signal to full scale; generate a frame of audio corresponding to an audio duration from the audio signal based on the audio signal; determine a maximum amplitude of the audio signal for the frame of the audio; evaluate the frame of the audio with a neural network to provide a speech likelihood estimate for the frame of the audio; determine, based on the maximum amplitude and the speech likelihood estimate, that the frame of the audio includes noise; in response to determining that the frame of the audio includes noise, lower the upper limiter limit; and provide the upper limiter limit to a limiter module to reduce the energy of the audio signal.

[0103] Example 20: The system of Example 19, wherein the system is implemented as one of: an audio conferencing system, a video conferencing system, an application specific integrated circuit, an application specific standard product, a system on a chip, a system in a package, a complex programmable logic device, an audio codec, or an audio processor.

Claims

1. A method for adaptive energy limitation, comprising: Set the upper limit level of the limiter module through which the audio signal is to pass to the full scale level; receiving a portion of the audio signal; determining a maximum amplitude of the portion of the audio signal; evaluating the portion of the audio signal with a neural network to provide a speech likelihood estimate for the portion of the audio signal; determining, based on the maximum amplitude and the speech likelihood estimate, that the portion of the audio signal comprises noise; responsive to determining that the portion of the audio signal includes noise, reducing the upper limit level of the limiter module from the full-scale level to a reduced upper limit level; as well as The lowered ceiling level is provided to the limiter module through which the audio signal passes to limit the amount of energy of the audio signal.

2. The method according to claim 1, wherein: The portion of the audio signal is a frame of audio corresponding to the portion of the audio signal, and the method further comprises: Prior to evaluating the frames of audio, the frames of audio are converted from the time domain to the frequency domain.

3. The method according to claim 2, wherein: The frames of the audio are frames of the first audio, and the method further comprises: receiving a frame of second audio corresponding to a second portion of the audio signal; evaluating frames of the second audio with the neural network to provide corresponding speech likelihood estimates for the frames of the second audio; determining, based on the corresponding speech likelihood estimate, that a frame of the second audio includes speech; and The upper limit level of the limiter module is reset to the full scale level.

4. The method according to claim 2, wherein: The frames of the audio are frames of the first audio, and the method further comprises: receiving a frame of second audio corresponding to a second portion of the audio signal; determining corresponding maximum amplitudes of frames of the second audio; comparing the corresponding maximum amplitude of the frame of the second audio to a threshold value, the threshold value corresponding to an average of the corresponding maximum amplitudes of a plurality of frames of audio, the plurality of frames of audio corresponding to a plurality of corresponding portions of the audio signal; and In response to the corresponding amplitude of the frame of the second audio not exceeding the threshold, maintaining the current upper limit level of the limiter module.

5. The method according to claim 2, wherein: A frame of the audio corresponds to an audio duration ranging from 10 milliseconds of the audio to 50 milliseconds of the audio.

6. The method according to any one of claims 1 to 5, wherein: Evaluating the portion of the audio signal with the neural network to provide the speech likelihood estimate includes analyzing the portion of the audio signal with a neural network enabled voice activity detector (VAD) to provide an instantaneous speech likelihood (IVL) of the portion of the audio signal.

7. The method according to claim 6, wherein: The upper limit level of the limiter module is reduced by a predefined amount to provide the reduced upper limit level, and the method further comprises: determining an aggregated speech likelihood estimate ASLE based on a plurality of IVLs provided by the neural network-enabled VAD; The aggregated speech likelihood estimate ASLE is updated based on the IVL in the following manner: In response to the IVL exceeding the ASLE and exceeding a speech detection threshold, increasing the ASLE; or In response to the IVL not exceeding the ASLE or not exceeding the speech detection threshold, decreasing the ASLE; and The predefined amount by which the upper level of the limiter module is reduced is set based on the ASLE.

8. The method according to claim 7, wherein: The upper limit level of the limiter module has a minimum value, and the method further includes configuring the minimum value of the upper limit level based on the ASLE.

9. The method of claim 8, further comprising configuring a minimum value of the upper limit level of the limiter module based on the ASLE and one of: an average of respective amplitudes of a plurality of parts of the audio signal; or The average of the respective maximum amplitudes of the plurality of portions of the audio signal.

10. An apparatus for adaptive energy limitation, comprising: A network interface, the network interface being used to receive or send an audio signal via a data network; A limiter module, the limiter module is used to limit the energy of the audio signal; a hardware-based processor associated with the data interface; as well as a storage medium storing processor-executable instructions that, in response to being executed by the hardware-based processor, implement an adaptive energy limiter for: setting an upper limit level of the limiter module of the audio signal to a full scale level; providing, from the audio signal, a frame of audio corresponding to a duration of audio from the audio signal; determining a maximum amplitude of the audio signal for a frame of the audio; evaluating frames of the audio with a neural network to provide speech likelihood estimates for the frames of the audio; determining, based on the maximum amplitude and the speech likelihood estimate, that a frame of the audio includes noise; responsive to determining that the frame of the audio includes noise, reducing the upper limit level of the limiter module from the full-scale level to a reduced upper limit level; as well as The lowered ceiling level is provided to the limiter module to reduce energy of the audio signal.

11. The device according to claim 10, wherein: The adaptive energy limiter is further implemented to: capturing frames of the audio corresponding to the duration of the audio; and Frames of the audio are converted from the time domain to the frequency domain for evaluation by the neural network.

12. The device according to claim 11, wherein The frames of the audio are frames of a first audio, and the adaptive energy limiter is further implemented to: capturing a frame of second audio corresponding to a second duration of audio of the audio signal; converting frames of the second audio from the time domain to the frequency domain; evaluating frames of the second audio of the audio signal with the neural network to provide corresponding speech likelihood estimates for the frames of the second audio; determining, based on the corresponding speech likelihood estimate, that a frame of the second audio includes speech; and The upper limit level of the limiter module is reset to the full scale level.

13. The device according to claim 11, wherein: The frames of the audio are frames of a first audio, and the adaptive energy limiter is further implemented to: capturing a frame of second audio corresponding to a second duration of audio of the audio signal; determining corresponding maximum amplitudes of frames of the second audio; comparing the corresponding maximum amplitude of the frame of the second audio to a threshold value, the threshold value corresponding to an average of the corresponding maximum amplitudes of a plurality of frames of audio, the plurality of frames of audio corresponding to a plurality of corresponding portions of the audio signal; as well as In response to the corresponding amplitude of the frame of the second audio not exceeding the threshold, maintaining the upper limit level of the limiter module at a current upper limit level.

14. The device according to any one of claims 11 to 13, wherein: The frames of audio correspond to a duration of audio information from the audio signal, the duration ranging from 5 milliseconds of audio information to 50 milliseconds of audio information.

15. The device according to any one of claims 10 to 13, wherein: The neural network comprises a voice activity detector, VAD, and the adaptive energy limiter is further implemented for providing the speech likelihood estimate as an instantaneous speech likelihood, IVL, of a frame of the audio using the VAD of the neural network.

16. The device according to claim 15, wherein: The adaptive energy limiter reduces the upper limit level of the limiter module by a predefined amount, and the adaptive energy limiter is further implemented to: determining an aggregated voice likelihood estimate ASLE based on a plurality of IVLs provided by the VAD of the neural network; The aggregated speech likelihood estimate ASLE is updated based on the IVL in the following manner: In response to the IVL exceeding the ASLE and exceeding a speech detection threshold, increasing the ASLE; or In response to the IVL not exceeding the ASLE or not exceeding the speech detection threshold, decreasing the ASLE; as well as The predefined amount by which the upper level of the limiter module is reduced is set based on the ASLE.

17. The device according to claim 16, wherein: The upper limit level of the limiter module has a minimum value, and the adaptive energy limiter is further implemented to configure the minimum value of the upper limit level based on the ASLE.

18. The device according to claim 17, wherein: The adaptive energy limiter is further implemented to configure a minimum value of the upper limit level based on the ASLE and one of: an average of respective amplitudes of a plurality of parts of the audio signal; or The average of the respective maximum amplitudes of the plurality of portions of the audio signal.

19. A system for adaptive energy limitation, comprising: a hardware-based processor operably associated with an audio interface or a data interface through which an audio signal is received; as well as a storage medium storing processor-executable instructions that, in response to execution by the hardware-based processor, implement an adaptive energy limiter for: setting the upper limit level of the limiter module through which the audio signal is to pass to a full-scale level; generating, based on the audio signal, a frame of audio corresponding to a duration of audio from the audio signal; determining a maximum amplitude of the audio signal for a frame of the audio; evaluating frames of the audio with a neural network to provide speech likelihood estimates for the frames of the audio; determining, based on the maximum amplitude and the speech likelihood estimate, that a frame of the audio includes noise; responsive to determining that the frame of the audio includes noise, reducing the upper limit level of the limiter module from the full-scale level to a reduced upper limit level; as well as The lowered ceiling level is provided to the limiter module through which the audio signal passes, so as to reduce the energy of the audio signal.

20. The system of claim 19, wherein: The system is implemented as one of the following: an audio conferencing system, a video conferencing system, an application specific integrated circuit, an application specific standard product, a system on a chip, a system in a package, a complex programmable logic device, an audio codec, or an audio processor.

Images