Source separation for multi-channel beamforming based on Personal Voice Section Detection (VAD).

Abstract

This disclosure provides methods, apparatus, and systems for speech enhancement. More specifically, these embodiments relate to using a personal speech section detector (VAD) to suppress sounds from interfering sources without distorting sounds from target sources. In some embodiments, the speech enhancement system may receive a multi-channel audio signal via a microphone array and further generate inferences, based on a neural network, about whether the current frame of the audio signal contains speech from known sources. For example, the neural network may be a personal VAD trained to detect speech IDs associated with one or more target sources. In some embodiments, the speech enhancement system may, based on at least part of these inferences, selectively direct a beam associated with a multi-channel beamformer in the direction of arrival (DOA) of the current audio frame.

Description

[Technical Field]

[0001] (Cross-reference to related applications) This patent application claims priority to U.S. Patent Application No. 18 / 310,955, filed on 2 May 2023, entitled "Source Isolation for Multichannel Beamforming Based on Personal Voice Section Detection (VAD)," which has been assigned to the assignee and is incorporated in its entirety by reference.

[0002] This embodiment generally relates to signal processing, and more specifically to source separation for multi-channel beamforming based on personal voice section detection (VAD). [Background technology]

[0003] Beamforming is a signal processing technique that can concentrate the energy of transmitted and received signals in the spatial direction. For example, a beamformer can improve the quality of speech detected by a microphone array through signal synthesis at the microphone output. More specifically, a beamformer applies weighting to the audio signals output from each microphone in the microphone array and amplifies the signal intensity in the direction of speech (or suppresses it in the direction of noise) when synthesizing the audio signals. Adaptive beamformers can dynamically adjust the weighting of the microphone outputs to optimize the quality of the synthesized audio signal, in other words, the signal-to-noise ratio (SNR). Therefore, adaptive beamformers can adapt to changes in the environment. Examples of adaptive beamformers include least mean squares error (MMSE) beamforming, minimum variance-free response (MVDR) beamforming, and generalized eigenvalue (GEV) beamforming.

[0004] In far-field applications, adaptive beamformers may not be able to distinguish between speech originating from a target source (e.g., the user of the microphone array) and speech originating from a disruptive source (e.g., someone speaking in the background). As a result, when the target source and the disruptive source speak simultaneously, the adaptive beamformer may not be able to suppress the disruptive speech as background noise. Therefore, there is a need to improve the separation of target speech and disruptive speech by adaptive beamformers in far-field applications. [Overview of the project]

[0005] This summary provides a simplified overview of selected concepts, which will be further explained in more detail later. This summary is not intended to identify the main or essential features of the subject matter of the claims, nor is it intended to limit the scope of the subject matter of the claims.

[0006] One innovative aspect of the subject matter of this disclosure can be implemented in a method for processing an audio signal. This method includes receiving an audio signal through a plurality of microphones, generating an inference, based on a neural network, about whether a first frame of the received audio signal contains speech associated with a known sound source, and selectively directing a beam associated with a multichannel beamformer in the direction of arrival (DOA) of the first frame, based on at least a portion of the inference about whether the first frame contains speech associated with a known sound source.

[0007] Other innovative aspects of the subject matter of this disclosure can be implemented in an audio enhancement system that includes a processing system and a memory. The memory stores instructions that, when executed by the processing system, cause the audio enhancement system to receive an audio signal via a plurality of microphones, generate an inference as to whether a first frame of the received audio signal includes speech associated with a known sound source based on a neural network, and selectively direct a beam associated with a multi-channel beamformer to the DOA of the first frame based on at least a portion of the inference as to whether the first frame includes speech associated with a known sound source. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] This embodiment is shown by way of example and is not intended to be limited to the form of the accompanying drawings.

[0009] [Figure 1] FIG. 1 depicts an example of an environment in which audio enhancement may be implemented.

[0010] [Figure 2] FIG. 2 depicts an example of an audio receiver that supports multi-channel beamforming.

[0011] [Figure 3] FIG. 3 depicts a block diagram of an example of an audio enhancement system according to some embodiments.

[0012] [Figure 4] FIG. 4 depicts a block diagram of an example of a target activity detector (TAD) according to some embodiments.

[0013] [Figure 5] FIG. 5 depicts a block diagram of an example of a single speaker detection system according to some embodiments.

[0014] [Figure 6]Figure 6 shows another block diagram of an example of TAD, based on several embodiments.

[0015] [Figure 7] Figure 7 shows another block diagram of an example of a speech enhancement system, based on several embodiments.

[0016] [Figure 8] Figure 8 shows an exemplary flowchart illustrating an example of the operation for processing an audio signal in several embodiments. [Modes for carrying out the invention]

[0017] The following description includes many specific details, such as examples of specific components, circuits, and processes, to fully understand the disclosure. As used herein, the term “connected” means directly connected or connected via components or circuits between one or more components. The terms “electronic system” and “electronic device” may be used interchangeably to refer to any system capable of electronically processing information. Furthermore, certain nomenclature is provided for the purposes of the following description and to fully understand the aspects of the disclosure. However, it will be apparent to those skilled in the art that these specific details are not necessarily required to carry out the examples of embodiments. In other examples, well-known circuits and devices are shown in block diagram form, so as not to obscure the disclosure. Some of the detailed descriptions that follow use procedures, logical blocks, processes, and other symbolic representations to describe operations on data bits in computer memory.

[0018] These descriptions and expressions are means used by experts in data processing technology to most effectively communicate the essence of their work to other experts in the technology. In this disclosure, procedures, logical blocks, processes, etc., are conceived as a consistent sequence of steps or instructions that produce a desired result. These steps require the physical manipulation of physical quantities. While not always the case, these quantities typically take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, or otherwise manipulated within a computer system. However, it should be noted that these and similar terms should all be associated with the appropriate physical quantities and are merely convenient labels assigned to those quantities.

[0019] Unless otherwise specified, or unless evident from the following description, throughout this application, any description using terms such as “access,” “receive,” “transmit,” “use,” “select,” “determine,” “normalize,” “multiply,” “average,” “monitor,” “compare,” “apply,” “update,” “measure,” and “derive” describes the operation and processes of a computer system or similar electronic computing device, manipulating and converting data represented as physical (electronic) quantities in the registers and memory of the computer system to other data similarly represented as physical quantities in the memory and registers of the computer system, or other information storage devices, transmission and display devices.

[0020] In diagrams, a single block may be described as performing one or more functions; however, in practice, the one or more functions performed by that block may be performed in a single component, across multiple components, and / or using hardware, software, or a combination of hardware and software. To clearly demonstrate this hardware-software compatibility, various exemplary components, blocks, modules, circuits, and steps are described below in general terms with respect to their functionality. Whether such functionality is implemented as hardware or executed as software depends on the design constraints imposed on the particular application and the overall system. A person skilled in the art may implement the described functionality in various ways depending on each particular application, but such implementation decisions should not be construed as departing from the scope of this disclosure. Furthermore, the exemplary input devices may include components other than those illustrated, such as well-known components such as processors and memory.

[0021] The technologies described herein may be implemented in hardware, software, firmware, or any combination thereof, unless expressly stated otherwise. Any functionality described as a module or component may be implemented together in an integrated logical device, or individually as separate but interoperable logical devices. When implemented as software, the technology may be implemented, at least in part, as a non-temporary processor-readable storage medium containing instructions that perform one or more of the aforementioned methods at runtime. This non-temporary processor-readable data storage medium may form part of a computer program product, which may include packaging materials.

[0022] Non-temporary processor-readable storage media may include random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, and other known storage media. In addition, or instead, this technique may be implemented, at least in part, by a processor-readable communication medium that transmits or transmits code in the form of instructions or data structures, which a computer or other processor can access, read, and / or execute.

[0023] Various exemplary logic blocks, modules, circuits, and instructions described in connection with the embodiments disclosed herein may be executed by one or more processors (or processing systems). As used herein, the term “processor” may refer to any general-purpose processor, dedicated processor, conventional processor, controller, microcontroller, and / or state machine capable of executing scripts or instructions of one or more software programs stored in memory.

[0024] As described above, beamformers can improve the quality of speech detected by a microphone array through signal synthesis at the microphone output. For example, a beamformer may amplify the signal intensity in the speech direction (or suppress it in the noise direction) when synthesizing the audio signals by applying weights to the audio signals output from each microphone in the microphone array. Adaptive beamformers can dynamically adjust the weights of the microphone outputs to optimize the quality of the combined audio signal, in other words, the signal-to-noise ratio (SNR). Examples of adaptive beamforming techniques include least mean squares error (MMSE) beamforming, minimum variance-free response (MVDR) beamforming, and generalized eigenvalue (GEV) beamforming.

[0025] In far-field applications, adaptive beamformers may be unable to distinguish between speech originating from a target source (e.g., the user of a microphone array) and speech originating from a disruptive source (e.g., someone speaking in the background). As a result, when the target and disruptive sources speak simultaneously, the adaptive beamformer may be unable to suppress the disruptive speech as background noise. Aspects of this disclosure recognize that each speaker's voice possesses unique biometric characteristics (also known as “voice ID”) that can be used to distinguish between target and disruptive speech. For example, a neural network may be trained or otherwise configured to determine whether an audio signal contains a voice ID associated with a known source (such as the target source). Such a neural network is commonly referred to as a “Personal Voice Activity Detector” or “Personal VAD”.

[0026] Various embodiments generally relate to speech enhancement, and more specifically, to using a personal VAD to suppress sounds from interfering sources without distorting sounds from target sources. In some embodiments, the speech enhancement system receives a multi-channel audio signal via a microphone array and may further generate an inference, based on a neural network, whether the current frame of the audio signal contains speech from a known source. For example, the neural network may be a personal VAD trained to detect speech IDs associated with one or more target sources. In some embodiments, the speech enhancement system may, based on at least part of this inference, selectively direct a beam associated with a multi-channel beamformer towards the direction of arrival (DOA) of the current audio frame. More specifically, the speech enhancement system may direct the beam towards the DOA of the current audio frame when the audio frame contains speech from a known source, and de-direct the beam towards the DOA of the current audio frame when the audio frame does not contain speech from a known source.

[0027] Certain embodiments of the subject matter described herein may be implemented to achieve one or more of the following potential benefits: By using a personal VAD to determine whether each frame of the audio signal contains speech from the target source, embodiments of the disclosure may improve the quality of speech detected from far-field sources. More specifically, the speech enhancement system of this embodiment may use inferences generated by the personal VAD to verify or validate the accuracy of the beam direction adopted by the adaptive beamformer. For example, the speech enhancement system may direct the beam associated with the multi-channel beamformer towards the adopted beam direction when the adopted beam direction aligns with a known source (or target source), and suppress directing the beam towards the adopted beam direction when the adopted beam direction does not align with a known source. Unlike existing speech enhancement techniques that rely solely on adaptive beamforming, embodiments of the disclosure can separate the target sound from interfering sounds, even in far-field applications.

[0028] Figure 1 shows an example of an environment 100 in which voice enhancement may be implemented. The illustrated environment 100 includes a communication device 110, a user 120 of the communication device 110 (also called the “target sound source” or “target source”), and a background speaker 130 (also called the “interference sound source” or “interference source”). In some embodiments, the communication device 110 may include multiple microphones 112 (also called a “microphone array”). In the example in Figure 1, the communication device 110 is shown to include two microphones 112. However, in actual implementations, the communication device 110 may include additional microphones (not shown for simplification).

[0029] Microphone 112 is positioned, or otherwise configured, to detect acoustic waves (including target speech 122 and interfering speech 132) propagating through the environment 100. For example, target speech 122 may include any sound produced by user 120. In contrast, interfering speech 132 may include any sound produced by a speaker behind 130, as well as any other background noise sources (not shown for simplicity). Microphone 112 may convert the detected acoustic waves into electrical signals (also called “audio signals”) that represent the acoustic waveforms. Thus, each audio signal may include a speech component (representing target speech 122) and a noise component (representing interfering speech 132). Due to differences in spatial position, a delay may occur between sounds detected by one microphone 112 and sounds detected by the other microphone. In other words, microphone 112 may produce audio signals with different phase offsets.

[0030] In some embodiments, the communication device 110 may include a multi-channel beamformer that weights and synthesizes the audio signals generated by each microphone 112 in order to amplify the voice component or suppress the noise component. More specifically, the weighting applied to the audio signal may improve the signal strength or SNR in the direction of the target voice 122. Such signal processing techniques are generally called “beamforming.” In some embodiments, the adaptive beamformer estimates (or predicts) a set of weights to apply to the audio signal (also called a “beamforming filter”) and directs the beam in the direction of the target voice 122. The voice quality of the resulting signal depends on the accuracy of the beamforming filter. For example, when the beam direction is aligned with the direction of the user 120, the voice component may be amplified. On the other hand, when the beam direction is aligned with the direction of the speaker 130 behind (or any direction away from the user 120), the voice component may be distorted or suppressed.

[0031] In near-field applications (for example, when user 120 is very close to microphone 122 and speaker 130 in the background is considerably farther away), the signal-to-noise ratio (SNR) of the target voice 122 may be substantially higher than that of the interfering voice 132. Therefore, a voice section detector (VAD) can be used to distinguish between voices originating from the target source and voices originating from the interfering source. In far-field applications (for example, when user 120 and speaker 130 in the background are relatively far from microphone 122), the SNR of the target voice 122 may be similar to that of the interfering voice 132. Therefore, existing VADs may not be able to distinguish between voices originating from the target source and voices originating from the interfering source. In other words, adaptive beamformers may not be able to distinguish whether user 120 or speaker 130 in the background is the target source. As a result, adaptive beamformers may employ beam directions that amplify both the target voice and the interfering voice.

[0032] In aspects of this disclosure, it is acknowledged that user 120 may have a unique voice ID (based on the biometric characteristics of the user's voice) that can be used to distinguish the target voice 122 from the interfering voice 132. For example, a neural network may be trained to detect user 120's voice ID in an audio signal received via microphone 112, or otherwise configured. Such a neural network is commonly referred to as a “personal voice section detector” or “personal VAD”. In some embodiments, communication device 110 may determine, based on inferences generated by the personal VAD, whether the beam direction adopted by the adaptive beamformer aligns with the direction of user 120. For example, this inference may indicate whether the received audio signal contains a voice ID associated with user 120 in the beam direction adopted from this audio signal. In other words, communication device 110 may use inferences generated by the personal VAD to verify that user 120 is speaking before directing the beam associated with the multichannel beamformer in the direction of the detected voice.

[0033] Figure 2 shows an example of an audio receiver 200 that supports multi-channel beamforming. The audio receiver 200 includes a plurality of microphones 210(1) to 210(M) arranged in a microphone array and a beamforming filter 220. In some embodiments, the audio receiver 200 may be an example of the communication device 110 in Figure 1. For example, with respect to Figure 1, each microphone 210(1) to 210(M) may be an example of one of the microphones 112.

[0034] Microphones 210(1) to 210(M) transmit a series of sound waves 201 (also called "acoustic waves") to audio signals X1(l,k) to X M It is configured to convert to (l,k), where l is the frame index and k is the frequency index associated with the time-frequency domain. As shown in Figure 2, sound wave 201 is incident on microphones 210(1)~210(M) at an angle (θ). The angle θ is the audio signal X1(l,k)~X M Also called the "direction of arrival" (DOA) of (l,k). In some embodiments, the sound wave 201 may contain the target sound (e.g., target sound 122 in Figure 1) mixed with interfering sound (e.g., interfering sound 132 in Figure 1). The target sound and interfering sound are each audio signal X1(l,k)~X M In (l,k), these represent the speech component (S(l,k)) and the noise component (N(l,k)), respectively.

[0035] Depending on the spatial position of microphones 210(1) to 210(M), each audio signal X1(l,k) to X M (l,k) may represent a delayed version of the same audio signal. For example, if the first audio signal X1(l,k) is used as the reference audio signal, then the remaining audio signals X2(l,k) ~ X M (l,k) can be considered as phase-delayed versions of the initial audio signal X1(l,k). Therefore, audio signal X1(l,k) ~ X M(l,k) can be modeled as the following vector (X(l,k)). TIFF2026518848000002.tif31170 Here, X(l,k) = [X1(l,k), …, X M (l,k)] T is a multi-channel audio signal, and a(θ,k) is a steering vector representing a set of phase delays for the sound wave 201 incident on microphones 210(1) to 210(M).

[0036] The beamforming filter 220 applies a weight vector w(l,k) = [w1(l,k), …, w M (l,k)] T (where w1 to w M are called filter coefficients) to generate the following enhanced audio signal (Y(l,k)). TIFF2026518848000003.tif40165 The weight vector w(l,k) determines the direction of the "beam" associated with the beamforming filter 220. Therefore, by adjusting the filter coefficients w1 to w M the beam can be "directed" in various directions.

[0037] In some aspects, an adaptive beamformer (not shown for simplicity) may determine a weight vector w(l,k) that optimizes the enhanced audio signal Y(l,k) with respect to one or more conditions. For example, an MVDR beamformer is configured to determine a weight vector w(l,k) that reduces or minimizes the variance of the noise component of the enhanced audio signal Y(l,k) without distorting the speech component of the enhanced audio signal Y(l,k). In other words, the weight vector w(l,k) may satisfy the following condition. TIFF2026518848000004.tif31168 Here, Φ NN (l,k) is the covariance of the noise component N(l,k) of the received audio signal X(l,k). The resulting weight vector w(l,k) is the MVDR beamforming filter (w MVDR(k)) and can be expressed as follows: TIFF2026518848000005.tif33165

[0038] As shown in equation (3), some MVDR beamformers may depend on geometric properties (such as the steering vector a(θ,k)) to determine the weight vector w(l,k). Therefore, the MVDR beamforming filter w MVDR The accuracy of (l,k) depends on the estimation accuracy of the steering vector a(θ,k), which may make it difficult to adapt to different users. A part of this disclosure is an MVDR beamforming filter w MVDR (l,k) is the covariance (Φ SS We acknowledge that it can also be expressed as a function of (l,k). TIFF2026518848000006.tif56149 Here, u(l,k) is a one-hot vector representing the reference microphone channel, W norm (l,k) is the normalization coefficient associated with W(l,k). Two examples of appropriate normalization coefficients are listed below. TIFF2026518848000007.tif45146

[0039] In some embodiments, the noise covariance Φ NN (l,k) and speech covariance Φ SS (l,k) may be estimated or updated over time through supervised learning. For example, when speech is present in the received audio signal X(l,k), the speech covariance Φ SS (l,k) can be estimated, and the noise covariance Φ occurs when there is no sound in the received audio signal X(l,k). NN(l,k) can be estimated. In some embodiments, a deep neural network (DNN) may be used to determine the presence or absence of speech in the audio signal X(l,k). For example, the DNN may be trained to infer the likelihood or probability of speech in each frame of the audio signal X(l,k). As mentioned with reference to Figure 1, conventional VADs may not be able to separate target speech from interfering speech in far-field applications. Therefore, in some embodiments, the adaptive beamformer relies on the inference generated by the personal VAD to determine the covariance Φ SS (l,k) and Φ NN In some cases, (l,k) may need to be determined.

[0040] Figure 3 shows a block diagram illustrating an example of the audio enhancement system 300 in several embodiments. The audio enhancement system 300 is configured to generate an enhanced audio signal Y(l,k) based on a multi-channel audio signal X(l,k) received via a microphone array. For example, in Figure 2, the multi-channel audio signal X(l,k) is an audio signal X1(l,k) to X received via microphones 210(1) to 210(M). M This may be an example of (l,k).

[0041] The speech enhancement system 300 includes a DNN 310, a targeted interval detector (TAD) 320, and a multi-channel beamformer 330. The DNN 310 is configured to infer speech probabilities p(l,k) in each frame l of an audio signal X(l,k) based on a neural network model, where 0 ≤ p(l,k) ≤ 1. For example, during the training phase, the DNN 310 may be provided with a large number of audio signals containing speech mixed with background noise. Alternatively, the DNN 310 may be provided with clean audio signals representing only the speech components of the audio signal (without background noise). The DNN 310 compares the audio signals and the clean audio signals to determine a set of features that can be used to classify speech. During the inference phase, based on the classification results, the DNN 310 infers speech probabilities at each frequency index k in each frame l of the audio signal X(l,k). Suitable examples of DNNs include convolutional neural networks (CNNs) and recurrent neural networks (RNNs).

[0042] The TAD320 is configured to determine or predict whether each frame l of an audio signal X(l,k) originates from a target sound source. An audio frame is said to "originate" from a particular sound source only if it contains speech associated with that source and does not contain speech associated with any other sound source. In some embodiments, the TAD320 may determine whether an audio signal X(l,k) contains speech associated with a target sound source based on a personal VAD. For example, the personal VAD may be a neural network trained to detect speech IDs associated with a target sound source.

[0043] In aspects of this disclosure, it is acknowledged that some audio signals may include speech associated with multiple sound sources (e.g., a target sound source and a disruptive sound source). Therefore, even if an audio frame includes speech associated with a target sound source, some audio frames may not originate from that target sound source. For this reason, in some embodiments, the TAD320 may further determine whether each frame l of the audio signal X(l,k) includes speech associated with multiple sound sources. For example, the TAD320 may include a neural network model trained to detect the presence of two or more voices within each audio frame.

[0044] The TAD320 may further output a target interval value (T(l)) based on whether the audio frame contains audio associated with a target sound source or audio associated with multiple sound sources. In some embodiments, the target interval value T(l) may indicate that the audio frame originates from the target sound source if the audio frame contains audio associated with a target sound source but does not contain audio associated with multiple sound sources. In some other embodiments, the target interval value T(l) indicates that the audio frame does not originate from the target sound source if the audio frame does not contain audio associated with a target sound source, or if the audio frame contains audio associated with multiple sound sources.

[0045] The multichannel beamformer 330 is configured to generate an enhanced audio signal Y(l,k) by applying a weight vector w(l,k) to the audio signal X(l,k) (for example, according to equation (2)). In some embodiments, the multichannel beamformer 330 may also be an adaptive beamformer that determines the weight vector w(l,k) to apply to each frame l of the audio signal X(l,k) based on at least a portion of the speech probability p(l,k) associated with each audio frame and the target interval value T(l). As shown in equations (4) and (5), the MVDR beamforming filter w MVDR(l,k) is the noise covariance Φ in the audio signal X(l,k). NN (l,k) and speech covariance Φ SS Based on (l,k), it can be determined. In some embodiments, the multichannel beamformer 330 determines the speech covariance Φ based on the speech probability p(l,k) and the target interval value T(l) associated with each audio frame. SS (l,k) and noise covariance Φ NN (l,k) may be updated dynamically.

[0046] In some embodiments, the multichannel beamformer 330 calculates the audio covariance Φ based on the audio existence probability p(l,k) when the target interval value T(l) indicates that the current audio frame originates from the target sound source (e.g., T(l)=1). SS (l,k) may be updated. TIFF2026518848000008.tif42144

[0047] In some other implementations, the multichannel beamformer 330 calculates the noise covariance Φ based on the speech probability p(l,k) when the target interval value T(l) indicates that the current audio frame does not originate from the target sound source (e.g., T(l)=0). NN (l,k) may be updated. TIFF2026518848000009.tif40158

[0048] In aspects of this disclosure, it is acknowledged that an adaptive beamformer may sometimes adopt a beam direction that aligns with an interfering sound source. In some embodiments, a multichannel beamformer may use a target interval value T(l) to determine or verify whether the adopted beam direction aligns with a target sound source. In other words, the multichannel beamformer 330 may selectively orient its beam in the adopted beam direction based on the target interval value T(l). In some embodiments, the multichannel beamformer 330 may orient its beam in the adopted beam direction when the target interval value T(l) indicates that the current frame of the audio signal X(l,k) originates from a target sound source.

[0049] In some other embodiments, the multichannel beamformer 330 may suppress directing its beam in the adopted beam direction when the target interval value T(l) indicates that the current frame of the audio signal X(l,k) does not originate from the target source. In such embodiments, the multichannel beamformer 330 is bypassed (and therefore no beamforming is performed on the current audio frame) when the target interval value T(l) indicates that the current frame of the audio signal X(l,k) does not originate from the target source. Alternatively, the multichannel beamformer 330 may implement a beamforming filter w(l,k) known to align with the target source. For example, the multichannel beamformer 330 may store beam directions associated with known target sources to support faster beam adaptation.

[0050] Figure 4 shows an exemplary block diagram of TAD400 in several embodiments. In some embodiments, TAD400 may be an example of TAD320 in Figure 3. More specifically, TAD400 may be configured to determine or predict whether each frame l of a multichannel audio signal X(l,k) originates from a target sound source, based on a speech probability p(l,k) associated with the audio frame. For example, the speech probability p(l,k) may be inferred by a DNN (e.g., DNN310 in Figure 3) trained to detect speech in the audio signal X(l,k).

[0051] The TAD400 includes a personal VAD410, a solo speaker (SS) detection component 420, and a target interval estimation component 430. The personal VAD410 is configured to generate an inference q(l) for each frame l of the audio signal X(l,k) based on whether the audio frame contains speech associated with a known sound source. For example, the personal VAD410 may include a neural network trained to detect speech IDs of one or more known sound sources in each audio frame. Therefore, the inference q(l) may indicate that the audio frame contains speech associated with a known sound source if the personal VAD410 detects speech IDs within the audio frame.

[0052] In some embodiments, the TAD400 may be configured to operate in single-user mode. In such embodiments, the personal VAD410 may look up a specific voice ID (also called the “target voice ID”) associated with a single sound source within the audio signal X(l,k). In other words, the inference q(l) may indicate that the current frame of the audio signal X(l,k) contains voice associated only with known sound sources, only if the personal VAD410 has detected the target voice ID within the current audio frame.

[0053] In some other embodiments, the TAD400 may be configured to operate in conference mode. In such embodiments, the personal VAD410 may look up audio IDs (also called "conference audio IDs") associated with multiple sound sources within the audio signal X(l,k). In other words, inference q(l) may indicate that if the personal VAD410 detects any conference audio ID within the current audio frame, the current frame of the audio signal X(l,k) contains audio associated with a known sound source.

[0054] A part of this disclosure acknowledges that the accuracy of the inference q(l) may depend on the number of acoustic features extracted by the neural network. For example, when speech is first detected in an audio signal X(l,k), the availability of acoustic features for classifying the speech is limited, so the personal VAD410 may have low certainty as to whether the speech matches the speech ID of a known sound source. However, after analyzing acoustic features over a certain number of audio frames, the personal VAD410 may significantly increase its certainty in making that decision.

[0055] In some embodiments, the personal VAD410 may be learned to classify each frame l of the audio signal X(l,k) into one of three classes: (1) voice ID detected, (2) voice ID not detected, or (3) undecidable. In other words, the inference q(l) may also be one of three values ​​indicating (1) that the audio frame contains voice associated with a known source (q(l)=1), (2) that the audio frame does not contain voice associated with any known source (q(l)=0), or (3) that the personal VAD410 cannot decide whether the audio frame contains voice associated with a known source (q(l)=-1).

[0056] In some embodiments, the personal VAD410 may be trained using a cost function that supports multiple or single detections (MOOD). For example, the MOOD cost function has tunable hyperparameters that can be used to compel the speech ID classifier to produce only a single detection (or up to any number of detections) within a target region (ROT), where ROT refers to the region of the audio signal used as the true value for training. In other words, the MOOD cost function does not have to penalize the neural network if it fails to classify a particular audio frame as "speech ID detected" or "speech ID not detected." Rather, the MOOD cost function may only penalize the neural network if, after processing a certain number of audio frames (corresponding to the ROT), it fails to classify at least one audio frame as "speech ID detected" or "speech ID not detected."

[0057] The SS detection component 420 is configured to determine whether each frame l of the audio signal X(l,k) contains speech associated with exactly one sound source. In some embodiments, the SS detection component 420 may include a neural network trained to detect the presence of two or more voices within each audio frame. More specifically, the neural network may generate inferences indicating whether the audio frame contains speech associated with multiple sound sources. In some embodiments, the SS detection component 420 may generate a detection signal (d(l)) based on at least part of whether the audio frame contains speech associated with multiple sound sources.

[0058] In some embodiments, the SS detection component 420 may be configured to classify each frame l of the audio signal X(l,k) into one of three classes: (1) single-speaker detection, (2) zero or multiple-speaker detection, or (3) undecidable. In other words, the SS detection signal d(l) may also be one of three values ​​indicating (1) that the audio frame contains speech associated with exactly one sound source (d(l)=1), (2) that the audio frame does not contain speech associated with exactly one sound source (d(l)=0), or (3) that the SS detection component 420 cannot decide whether the audio frame contains speech associated with exactly one sound source (d(l)=-1).

[0059] The target interval estimation component 430 is configured to generate a corresponding target interval value T(l) for each frame l of the audio signal X(l,k) based on at least a portion of the inference q(l) and the detected signal d(l). More specifically, the target interval estimation component 430 may estimate whether the beam direction adopted by the adaptive beamformer (e.g., the multi-channel beamformer 330 in Figure 3) matches the direction of a known sound source (or target sound source). In some embodiments, the target interval value T(l) may be one of three values, as follows: TIFF2026518848000010.tif57161

[0060] In some embodiments, the adaptive beamformer may direct its beam in the adopted beam direction when T(l)=1 and suppress directing its beam in the adopted beam direction when T(l)=0 or -1. As described with reference to Figure 3, the multichannel beamformer 330 directs its beam in the audio covariance φ when T(l)=1. SS Update (l, k), and when T(l)=0, the noise covariance φ NN (l,k) may be updated. In some embodiments, the multichannel beamformer 330 has an audio covariance φ when T(l)=-1. SS (l,k) is also the noise covariance φ NN (l,k) does not need to be updated either.

[0061] Figure 5 shows a block diagram illustrating an example of a solo speaker (SS) detection system 500 according to several embodiments. In some embodiments, the system 500 may be an example of the SS detection component 420 in Figure 4. More specifically, the SS detection system 500 may be configured to determine whether each frame l of the audio signal X(l,k) contains speech associated with just one sound source, based on at least a portion of the speech probability p(l,k) associated with the audio frame. In some embodiments, the speech probability p(l,k) may be inferred by a DNN (e.g., DNN310 in Figure 3).

[0062] The SS detection system 500 includes a broadband conversion component 510, a broadband VAD 520, a DNN 530, and a speaker estimation component 540. The broadband conversion component 510 is configured to normalize the speech probability p(l,k) across all frequency subbands k. In some embodiments, the broadband conversion component 510 normalizes the broadband speech probability p(l,k) as a function of the speech probability p(l,k) and the audio signal X(l,k). total (l) may be generated. TIFF2026518848000011.tif48151

[0063] The broadband VAD520 is a broadband speech probability p total (l) is a VAD value that indicates whether audio was detected in the current frame of the audio signal X(l,k) It is configured to convert to TIFF2026518848000012.tif22154). In some embodiments, the broadband VAD520 is configured to convert to broadband speech probability P total If (l) is greater than or equal to the first threshold probability (γ0), it is determined that there is speech in the current audio frame, and the broadband speech probability P total If (l) is less than or equal to the second threshold probability (γ1), it may be determined that there is no sound in the current audio frame. In other words, the VAD value TIFF2026518848000013.tif22154 can have three possible values, as follows: TIFF2026518848000014.tif55155

[0064] The DNN530 is trained, or otherwise configured, to detect multiple voices in the current frame of an audio signal X(l,k). More specifically, the DNN530 may generate an inference r(l) indicating whether multiple voices were detected in the current audio frame. In some embodiments, the inference r(l) may be two values, indicating either two or more voices were detected in the current audio frame (r(l)=1) or one or fewer voices were detected (r(l)=0).

[0065] The speaker estimation component 540 is based on the inference r(l) and VAD value. Based on TIFF2026518848000015.tif21155, the system is configured to determine whether the current frame of the audio signal X(l,k) contains speech from exactly one source. More specifically, the speaker estimation component 540 may generate a detection signal d(l) indicating whether the current audio frame contains speech from exactly one source. In some embodiments, the detection signal d(l) may have three values, as follows: TIFF2026518848000016.tif32155

[0066] Figure 6 shows other block diagrams of exemplary TAD600 in several embodiments. In some embodiments, TAD600 may be an example of TAD320 in Figure 3. More specifically, TAD600 may be configured to determine or predict whether each frame l of a multichannel audio signal X(l,k) originates from a target sound source, based on a speech probability p(l,k) associated with the audio frame. For example, the speech probability p(l,k) may be inferred by a DNN (e.g., DNN310 in Figure 3) trained to detect speech in the audio signal X(l,k).

[0067] The TAD600 includes a personal VAD610, a solo speaker (SS) detection component620, a direction of arrival (DOA) estimation component630, and a target interval estimation component640. The personal VAD610 is configured to generate an inference q(l) for each frame l of the audio signal X(l,k) based on whether the audio frame contains speech associated with a known sound source. In some embodiments, the personal VAD610 may be an example of the personal VAD410 shown in Figure 4. For example, the personal VAD610 may include a neural network trained to detect one or more speech IDs. Therefore, the inference q(l) may indicate that the audio frame contains speech associated with a known sound source if the personal VAD610 detects a speech ID within the audio frame.

[0068] In some embodiments, the TAD600 may be configured to operate in single-user mode. In such embodiments, inference q(l) may indicate that the current frame of audio signal X(l,k) contains audio associated with a known source only if the personal VAD610 detects a target audio ID within the current audio frame (as described with reference to Figure 4). In some other embodiments, the TAD600 may be configured to operate in conference mode. In such embodiments, inference q(l) may indicate that the current frame of audio signal X(l,k) contains audio associated with a known source, as long as the personal VAD610 detects any conference audio ID within the current audio frame (as described with reference to Figure 4).

[0069] In some embodiments, the personal VAD610 may be trained to classify each frame l of the audio signal X(l,k) into one of three classes: (1) voice ID detected, (2) voice ID not detected, or (3) undecidable. In other words, the inference q(l) may also be one of three values ​​indicating (1) that the audio frame contains voice associated with a known source (q(l)=1), (2) that the audio frame does not contain voice associated with any known source (q(l)=0), or (3) that the personal VAD610 cannot decide whether the audio frame contains voice associated with any known source (q(l)=-1). In some embodiments, the personal VAD610 may be trained using a MOOD cost function (as described with reference to Figure 4).

[0070] The SS detection component 620 is configured to determine whether each frame l of the audio signal X(l,k) contains speech associated with exactly one sound source. In some embodiments, the SS detection component 620 may be an example of the SS detection component 420 in Figure 4 or the SS detection system 500 in Figure 5. For example, the SS detection component 420 may include a neural network trained to detect the presence of two or more voices within each audio frame. In some embodiments, the SS detection component 620 may generate a detection signal d(l) based on at least a portion of whether the audio frame contains speech associated with multiple sound sources (as described with reference to Figure 5).

[0071] In some embodiments, the SS detection component 620 may be configured to classify each frame l of the audio signal X(l,k) into one of three classes: (1) single-speaker detection, (2) zero or multiple-speaker detection, or (3) undecidable. In other words, the SS detection signal d(l) may also be one of three values ​​indicating (1) that the audio frame contains speech associated with exactly one sound source (d(l)=1), (2) that the audio frame does not contain speech associated with exactly one sound source (d(l)=0), or (3) that the SS detection component 620 cannot decide whether the audio frame contains speech associated with exactly one sound source (d(l)=-1).

[0072] The DOA estimation component 630 is configured to estimate the DOA(θ~(l)) for each frame l of the audio signal X(l,k). More specifically, the DOA estimation component 630 may estimate the beam direction to be adopted by the adaptive beamformer (e.g., the multi-channel beamformer 330 in Figure 3) based on the received audio signal X(l,k). For example, in Figure 2, the DOA θ~(l) represents the angle (θ) at which the sound wave 201 is incident on the microphones 210(1)~210(M).

[0073] In some embodiments, the DOA estimation component 630 uses the audio signal X1(l,k)~X received through each microphone of the microphone array. M The DOA θ~(l) may be estimated based on the delay between (l,k). For example, in Figure 2, the audio signals X1(l,k) and X2(l,k) received via microphones 210(1) and 210(2), respectively, can be represented as time-domain signals x1(t) and x2(t). TIFF2026518848000017.tif49148 Here, s(t) represents the speech component in each audio signal x1(t) and x2(t). n1(t) and n2(t) represent the noise component in audio signals x1(t) and x2(t), respectively. α is the attenuation rate associated with the second audio signal x2(t). D is the delay between the first audio signal x1(t) and the second audio signal x2(t).

[0074] In aspects of this disclosure, the delay D is the cross-correlation of audio signals x1(t) and x2(t) ( It is acknowledged that this can be determined by calculating TIFF2026518848000018.tif18144). TIFF2026518848000019.tif31151 Here, E[·] is the expected value, The value of τ that maximizes TIFF2026518848000020.tif18144 provides an estimate of the delay D (and therefore DOA θ~(l)).

[0075] The target interval estimation component 640 is configured to generate each target interval value T(l) for each frame l of the audio signal X(l,k) based on the inference q(l), the detected signal d(l), and at least a portion of the DOA θ~(l). More specifically, the target interval estimation component 640 may estimate whether the beam direction adopted by the adaptive beamformer (e.g., the multi-channel beamformer 330 in Figure 3) matches the direction of a known sound source (or target sound source).

[0076] In some embodiments, the target interval estimation component 640 may generate an intermediate interval value T0(l) based on the inference q(l) and the detection signal d(l). In some embodiments, the intermediate interval value T0(l) may be one of three values, as follows: TIFF2026518848000021.tif51161

[0077] In some embodiments, the target interval value T(l) may indicate, when T0(l)=1, that the current frame of the audio signal X(l,k) originated from the target sound source. In some other embodiments, the target interval value T(l) may indicate, when T0(l)=0, that the current frame of the audio signal X(l,k) does not originate from the target sound source. Furthermore, in some embodiments, the target interval estimation component 640 may use DOA θ~(l) to resolve indecision by the personal VAD 610 or SS detection component 620 (e.g., when T0(l)=-1).

[0078] In some embodiments, the target interval estimation component 640 may compare DOA θ~(l) with a set of target DOA(D) known to match the target sound source. t If it matches the target interval value T(l), then when T0(l)=-1, the current frame of the audio signal X(l,k) may indicate that it originated from the target sound source. For example, if DOA θ~(l) is the target DOA θ t A "match" may be detected if it falls within the threshold range (Δ).

[0079] However, if DOA θ~(l) does not match any of the target DOAs in set D, DOA θ~(l) may not be used to determine indecision by the personal VAD610 or SS detection component 620. Therefore, in some embodiments, the target interval value T(l) may be one of the following three values ​​based on the intermediate interval value T0(l) and DOA θ~(l). TIFF2026518848000022.tif61168

[0080] In some embodiments, the target interval estimation component 640 may dynamically update the set D of target DOA based on at least a portion of the intermediate interval values ​​T0(l). In some embodiments, when T0(l)=1, the target interval estimation component 640 sets DOA θ~(l) to target DOA θt It may be added to set D as follows. In some other embodiments, the target interval estimation component 640 matches the target DOA θ~(l) in set D when T0(l)=1. t The following may be updated. Furthermore, in some embodiments, the target interval estimation component 640 matches the target DOA θ~(l) when T0(l)=0. t It may be necessary to remove it from set D.

[0081] In some embodiments, the adaptive beamformer may direct its beam in the adopted beam direction when T(l)=1, and suppress directing its beam in the adopted beam direction when T(l)=0 or -1. As illustrated with reference to Figure 3, the multichannel beamformer 330 directs its beam in the audio covariance Φ when T(l)=1. SS Update (l,k) and when T(l)=0, the noise covariance Φ NN (l,k) may be updated. In some embodiments, the multichannel beamformer 330 has an audio covariance Φ when T(l)=-1. SS (l,k) is also the noise covariance Φ NN (l,k) does not need to be updated either.

[0082] Figure 7 shows another block diagram of an example of the voice enhancement system 700 according to several embodiments. More specifically, the voice enhancement system 700 may be configured to receive a multi-channel audio signal and generate an amplified audio signal by filtering or suppressing noise in the received audio signal based on at least a portion of the voice IDs associated with known sound sources. In some embodiments, the voice enhancement system 700 may be an example of the audio receiver 200 in Figure 2 or the voice enhancement system 300 in Figure 3.

[0083] The voice enhancement system 700 includes a device interface 710, a processing system 720, and a memory 730. The device interface 710 is configured to communicate with various components of an audio receiver. In some embodiments, the device interface 710 may include a microphone interface (I / F) 712 configured to receive audio signals through a plurality of microphones. For example, the microphone I / F 712 may sample or receive individual frames of the audio signal in frame hops associated with the voice enhancement system 700.

[0084] The memory 730 may include an audio data storage device 731 configured to store one or more frames of an audio signal. The memory 730 also includes a non-temporary computer-readable medium (including, but not limited to, one or more non-volatile memory elements such as EPROM, EEPROM, flash memory, and hard drives) and may store at least the following software (SW) modules. • Personal VAD SW module 732, based on a neural network, generates inferences about whether the first frame of a received audio signal contains speech associated with a known sound source. Based on at least part of the reasoning about whether the first frame contains audio associated with a known sound source, a beamforming SW module 733 selectively directs the beam associated with the multichannel beamformer to the DOA of the first frame. Each software module, when executed by the processing system 720, includes instructions that cause the speech enhancement system 700 to perform the corresponding function.

[0085] The processing system 720 may include one or more suitable processors capable of executing scripts or instructions of one or more software programs stored in the speech enhancement system 700 (for example, in memory 730). For example, the processing system 720 may run a personal VAD SW module 732 to generate inferences, based on a neural network, about whether the first frame of the received audio signal contains speech associated with a known sound source. The processing system 720 may also run a beamforming SW module 733 to selectively direct the beam associated with a multichannel beamformer to the DOA of the first frame, based on at least part of the inferences about whether the first frame contains speech associated with a known sound source.

[0086] Figure 8 shows an exemplary flowchart illustrating an example of operation 800 for processing an audio signal in several embodiments. In some embodiments, the exemplary operation 800 may be performed by a voice enhancement system (e.g., the audio receiver 200 in Figure 2, or either the voice enhancement system 300 or 700 in Figures 3 and 7).

[0087] The speech enhancement system receives audio signals through multiple microphones (810). Based on a neural network, the speech enhancement system generates an inference about whether the first frame of the received audio signal contains speech associated with a known source (820). In some embodiments, this inference may be one of three values ​​indicating that the first frame contains speech associated with a known source, that the first frame does not contain speech associated with a known source, or that the neural network cannot determine whether the first frame contains speech associated with a known source.

[0088] Furthermore, the speech enhancement system selectively directs the beam associated with the multichannel beamformer to the DOA of the first frame based on at least part of its inference about whether the first frame contains speech associated with a known source (830). In some embodiments, the speech enhancement system may suppress directing the beam to the DOA of the first frame if the three values ​​indicate that the first frame does not contain speech associated with a known source, or if the three values ​​indicate that the neural network cannot determine whether the first frame contains speech associated with a known source.

[0089] In some embodiments, the speech enhancement system may further determine whether the first frame of the received audio signal contains speech associated with multiple sound sources. In such embodiments, selective beam direction may further depend on whether the first frame contains speech associated with multiple sound sources. In some embodiments, if the first frame contains speech associated with multiple sound sources, the speech enhancement system may suppress beam direction to the DOA of the first frame.

[0090] In some embodiments, the speech enhancement system may further determine the speech probability in the first frame of the received audio signal. In such embodiments, selective beam direction may be further based on the speech probability in the first frame. In some embodiments, the speech enhancement system may suppress beam direction to the DOA of the first frame if the speech probability in the first frame is below a threshold probability. In some other embodiments, the speech enhancement system may beam to the DOA of the first frame if the speech probability in the first frame is greater than or equal to a threshold probability, the first frame does not contain speech associated with multiple sound sources, and the three values ​​indicate that the first frame contains speech associated with a known sound source.

[0091] In some embodiments, the multichannel beamformer may be an MVDR beamformer that reduces the power of the noise component of the audio signal without distorting the speech component of the audio signal. In some embodiments, the speech enhancement system may further calculate a filter associated with the MVDR beamformer based on the covariance of the noise component of the audio signal and the covariance of the speech component of the audio signal. In some embodiments, the speech enhancement system may refrain from determining the covariance of either the speech or noise component of the audio signal when the speech probability in the first frame is greater than a first threshold probability and less than a second threshold probability, or when the three values ​​indicate that the neural network cannot determine whether the first frame contains speech associated with a known sound source.

[0092] In some other embodiments, the speech enhancement system may determine the covariance of the speech component of the audio signal when the speech probability in the first frame is greater than or equal to a threshold probability, the first frame does not contain speech associated with multiple sound sources, and the three values ​​indicate that the first frame contains speech associated with a known sound source. Furthermore, in some other embodiments, the speech enhancement system may determine the covariance of the noise component of the audio signal when the speech probability in the first frame is less than a threshold probability, the first frame contains speech associated with multiple sound sources, or the three values ​​indicate that the first frame does not contain speech associated with a known sound source.

[0093] In some embodiments, the audio enhancement system may further store the DOA of the first frame as a response to directing the beam to the DOA of the first frame, determine the DOA of the second frame of the received audio signal, determine whether the DOA of the second frame falls within a threshold range of the stored DOAs, and selectively direct the beam to the DOA of the second frame based at least part on whether the DOA of the second frame falls within a threshold range of the stored DOAs.

[0094] Those skilled in the art will understand that information and signals can be represented using any variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description can be represented by voltage, electric current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

[0095] Furthermore, those skilled in the art will understand that various exemplary logic blocks, modules, circuits, and algorithmic steps described in relation to the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly demonstrate this hardware- and software compatibility, various exemplary components, blocks, modules, circuits, and steps are generally described above with respect to their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. A skilled technician may implement the described functionality in various ways for each specific application, but such a decision on implementation should not be construed as a deviation from the scope of the disclosure.

[0096] Methods, sequences, or algorithms relating to embodiments disclosed herein may be embodied in hardware directly, in software modules executed by a processor, or in a combination of the two. The software modules may be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is connected to the processor so that the processor can read information from and write information to the storage medium. Alternatively, the storage medium may be integrated with the processor.

[0097] Embodiments have been described in this specification with reference to specific examples. However, it is clear that various changes and modifications are possible without departing from the broad scope of the disclosure set forth in the appended claims. Therefore, this specification and the drawings should be understood in an illustrative rather than restrictive sense.

Claims

1. Receiving audio signals through multiple microphones, Based on a neural network, generate an inference about whether the first frame of the received audio signal contains speech associated with a known sound source. Based on at least part of the reasoning regarding whether the first frame contains sound associated with a known sound source, selectively direct the beam associated with the multichannel beamformer in the direction of arrival (DOA) of the first frame, A method for processing audio signals that include [specific components / signals].

2. The method according to claim 1, wherein the inference includes three values ​​indicating that the neural network cannot determine whether the first frame contains audio associated with a known sound source, does not contain audio associated with a known sound source, or contains audio associated with a known sound source.

3. Selectively directing the aforementioned beam is If the three values ​​indicate that the first frame does not contain audio associated with a known sound source, or if the three values ​​indicate that the neural network cannot determine whether the first frame contains audio associated with a known sound source, then the beam to the DOA of the first frame is suppressed. The method according to claim 2, including the method described in claim 2.

4. Determining whether the first frame of the received audio signal contains audio associated with multiple sound sources, and further determining whether the beam selectively directs the first frame contains audio associated with multiple sound sources. The method according to claim 2, further comprising:

5. Selectively directing the aforementioned beam is If the first frame includes audio related to multiple sound sources, the beam is suppressed from being directed to the DOA of the first frame. The method according to claim 4, including the method described in claim 4.

6. The audio probability in the first frame of the received audio signal is determined, and the selective direction of the beam is further based on the audio probability in the first frame. The method according to claim 4, further comprising:

7. Selectively directing the aforementioned beam is If the audio probability in the first frame is less than the threshold probability, the beam is suppressed from being directed at the DOA in the first frame. The method according to claim 6, including the method described in claim 6.

8. Selectively directing the aforementioned beam is If the audio probability in the first frame is greater than or equal to the threshold probability, the first frame does not contain audio associated with multiple sound sources, and the three values ​​indicate that the first frame contains audio associated with a known sound source, then direct the beam towards the DOA of the first frame. The method according to claim 6, including the method described in claim 6.

9. The method according to claim 6, wherein the multi-channel beamformer includes a minimum dispersion-free response (MVDR) beamformer that reduces the power of the noise component of the audio signal without distorting the audio component of the audio signal.

10. The filter associated with the MVDR beamformer is calculated based on the covariance of the noise component of the audio signal and the covariance of the speech component of the audio signal. The method according to claim 6, further comprising:

11. The covariance of the audio component of the audio signal is determined when the audio probability in the first frame is greater than or equal to the threshold probability, the first frame does not contain audio associated with multiple sound sources, and the three values ​​indicate that the first frame contains audio associated with a known sound source. The method according to claim 10, further comprising:

12. When the audio probability in the first frame is less than the threshold probability, and the first frame includes audio associated with multiple sound sources, or when the three values ​​indicate that the first frame does not include audio associated with a known sound source, the covariance of the noise component of the audio signal is determined. The method according to claim 10, further comprising:

13. When the speech probability in the first frame is greater than the first threshold probability and less than the second threshold probability, or when the three values ​​indicate that the neural network cannot determine whether the first frame contains speech associated with a known sound source, the covariance of either the speech component or the noise component of the audio signal is not determined. The method according to claim 10, further comprising:

14. The DOA of the first frame is stored as a response to directing the beam towards the DOA of the first frame, To determine the DOA of the second frame of the received audio signal, To determine whether the DOA of the second frame is within the threshold range of the stored DOA, Based on at least part of whether the DOA of the second frame is within the threshold range of the stored DOA, the beam is selectively directed to the DOA of the second frame. The method according to claim 1, further comprising:

15. It is a speech enhancement system, Processing system and, Memory for storing instructions, Equipped with, When the aforementioned instruction is executed by the processing system, it is transmitted to the speech enhancement system. Receiving audio signals through multiple microphones, Based on a neural network, generate an inference about whether the first frame of the received audio signal contains speech associated with a known sound source. Based on at least part of the reasoning regarding whether the first frame contains sound associated with a known sound source, the beam associated with the multichannel beamformer is selectively directed in the direction of arrival (DOA) of the first frame, A voice enhancement system that enables this process.

16. The speech enhancement system according to claim 15, wherein the inference includes three values ​​indicating that the first frame contains speech associated with a known sound source, the first frame does not contain speech associated with a known sound source, or the neural network cannot determine whether the first frame contains speech associated with a known sound source.

17. The execution of the aforementioned command is further transmitted to the voice enhancement system. Determining whether the first frame of the received audio signal contains audio associated with multiple sound sources, and further determining whether the beam is selectively directed based on whether the first frame contains audio associated with multiple sound sources. The voice enhancement system according to claim 16, which causes to perform the following:

18. The execution of the aforementioned command is performed by the voice enhancement system. The audio probability in the first frame of the received audio signal is determined, and the selective direction of the beam is further based on the audio probability in the first frame. The voice enhancement system according to claim 17, further comprising the following steps.

19. Selectively directing the aforementioned beam is The beam is directed to the DOA of the first frame only if the audio probability of the first frame is greater than or equal to the threshold probability, the first frame does not contain audio associated with multiple sound sources, and the three values ​​indicate that the first frame contains audio associated with a known sound source. The voice enhancement system according to claim 18, including the following:

20. The execution of the aforementioned command is performed by the voice enhancement system. The DOA of the first frame is stored as a response to directing the beam towards the DOA of the first frame, To determine the DOA of the second frame of the received audio signal, To determine whether the DOA of the second frame falls within the threshold range of the stored DOA, Based on at least part of whether the DOA of the second frame is within the threshold range of the stored DOA, the beam is selectively directed to the DOA of the second frame. The voice enhancement system according to claim 15, further comprising the following steps.

Images