System and method for multi-channel speech detection

Abstract

A system and method for multi-channel speech detection is provided. The method includes receiving a multi-channel audio input comprising a set of audio signals; processing the multi-channel audio input using a neural network classifier to generate a series of classification results in a series of time windows of the multi-channel audio input, wherein the neural network classifier comprises a causal temporal convolution network (TCN) configured to determine a classification result for each time window based on portions of the multi-channel audio input in the corresponding time window and one or more time windows preceding the corresponding time window; and determining whether the multi-channel audio input includes one or more speech segments in the series of time windows based on the series of classification results. The speech detection method can be implemented in real-time or near real-time.

Description

Technical Field

[0001] This invention relates to the field of speech detection, and more specifically to multi-channel speech detection using machine learning techniques. Background Technology

[0002] Audio signals captured in an environment may include speech segments, non-speech segments, or both. For example, an audio signal may include speech data for a first duration and non-speech data (e.g., noise data without speech) for a second duration. In low-noise environments without reverberation, a simple energy detector can be applied to detect speech data from the audio signal. However, in real-world environments where audio signals are corrupted by noise or reverberation, an energy detector may fail to distinguish between speech and non-speech data in the audio signal. Therefore, machine learning techniques can be used to detect speech data in audio signals.

[0003] Recently, recurrent neural networks (RNNs) have been widely used to model sequential data. For example, RNNs with long short-term memory (LSTM) structures and RNNs with gated recurrent units (GRUs) have been successfully applied to sequence-to-label classification problems of text and audio data. However, due to their complex network structures and numerous hyperparameters, RNNs may require large amounts of memory and long training times. Furthermore, RNNs suffer from the vanishing gradient problem. For instance, in each iteration of the training process, each weight of an RNN receives an update proportional to the partial derivative of the error function with respect to the current weight. However, in some cases, the gradient may be very small, preventing the corresponding weight from changing its value. In the worst case, the vanishing gradient problem can completely prevent further training of the RNN. Summary of the Invention

[0004] Therefore, the present invention provides a speech detection method that can be implemented in real time or near real time.

[0005] To achieve the above objectives, the present invention provides a speech detection method, comprising: receiving a multi-channel audio input including a set of audio signals from a set of audio channels in an audio detection array; processing the multi-channel audio input using a neural network classifier to generate a series of classification results within a series of time windows of the multi-channel audio input, wherein the neural network classifier includes a causal temporal convolutional network (TCN) configured to determine the classification result for each time window based on portions of the multi-channel audio input within a corresponding time window and one or more time windows preceding the corresponding time window; and determining, based on the series of classification results, whether the multi-channel audio input includes one or more speech segments within the series of time windows.

[0006] Optionally, the causal TCN includes one or more serially connected residual blocks, each containing an expanded one-dimensional (1D) convolution.

[0007] Optionally, the causal TCN may further include a dense layer connected to the one or more serially connected residual blocks, the dense layer being configured to adjust the size of the causal TCN's output.

[0008] Optionally, each residual block also includes: a modified linear unit (ReLU) activation function applied in series with the one-dimensional convolution of the expansion; and a discard layer configured for regularization and applied after the ReLU activation function.

[0009] Optionally, processing the multichannel audio input using the neural network classifier to generate a series of classification results within a series of time windows of the multichannel audio input includes: preprocessing the set of audio signals to output a combined audio signal; dividing the combined audio signal into a series of frames within the series of time windows; and for each time window: obtaining a frame corresponding to the corresponding time window from the series of frames; applying the frame to the neural network classifier to generate an estimate of the corresponding time window; and determining the classification result of the corresponding time window based on the estimate.

[0010] Optionally, applying the frame to the neural network classifier to generate the estimation result of the corresponding time window includes: generating a feature sequence of the frame, the feature sequence including the features of the frame and T-1 features of T-1 previous frames, where T is a positive integer; inputting the feature sequence into the neural network classifier to generate a value sequence, wherein the value sequence includes a value estimate of the frame and a value estimate of T-1 previous frames; updating the value buffer based on the value sequence; and generating the estimation result of the corresponding time window based on the value buffer.

[0011] Optionally, generating the feature sequence of the frame includes: processing the frame using a high-pass filter; calculating the Mel spectrum of the frame; and concatenating the Mel spectrum of the frame with the T-1 Mel spectra of the previous T-1 frames to generate the feature sequence.

[0012] Optionally, the value buffer includes T value estimates for the corresponding time window, and the estimation result for the corresponding time window is generated based on the T value estimates.

[0013] Optionally, processing the multichannel audio input using the neural network classifier to generate a series of classification results within a series of time windows of the multichannel audio input includes: processing each audio signal using the neural network classifier to perform speech estimation based on the corresponding audio signal, thereby generating a set of speech estimates for the set of audio signals, wherein the set of speech estimates includes a corresponding set of estimates for the set of audio signals for each time window; and fusing the set of speech estimates of the set of audio signals to generate the series of classification results within the series of time windows of the multichannel audio input.

[0014] Optionally, processing each audio signal using the neural network classifier further includes: dividing each audio signal into a series of frames within the series of time windows; for each time window: obtaining the frame corresponding to the corresponding time window from the series of frames; and applying the frame to the neural network classifier to generate an estimate of the corresponding time window.

[0015] Optionally, fusing the group speech estimates of the group audio signals to generate the series of classification results in the series of time windows of the multi-channel audio input includes: for each time window, fusing the corresponding group estimation results to generate the classification result for the corresponding time window.

[0016] Optionally, determining whether the multichannel audio input includes one or more speech segments in the series of time windows based on the series of classification results includes: for each time window, determining whether there is a speech segment in the multichannel audio input during the corresponding time window according to the classification result of the corresponding time window.

[0017] The present invention also provides a speech detection system, comprising: a memory configured to store a multi-channel audio input acquired by an audio detection array, wherein the multi-channel audio input includes a set of audio signals from a set of audio channels in the audio detection array; and a processor configured to: process the multi-channel audio input using a neural network classifier to generate a series of classification results within a series of time windows of the multi-channel audio input, wherein the neural network classifier includes a causal temporal convolutional network (TCN) configured to determine a classification result for each time window based on portions of the multi-channel audio input within a corresponding time window and one or more time windows preceding the corresponding time window; and to determine, based on the series of classification results, whether the multi-channel audio input includes one or more speech segments within the series of time windows.

[0018] The present invention also provides a non-transitory computer-readable storage medium configured to store instructions responsive to execution of a processor, causing the processor to perform a process comprising: receiving a multi-channel audio input comprising a set of audio signals from a set of audio channels in an audio detection array; processing the multi-channel audio input using a neural network classifier to generate a series of classification results within a series of time windows of the multi-channel audio input, wherein the neural network classifier comprises a causal temporal convolutional network (TCN) configured to determine a classification result for each time window based on portions of the multi-channel audio input within the corresponding time window and one or more time windows preceding the corresponding time window; and determining, based on the series of classification results, whether the multi-channel audio input includes one or more speech segments within the series of time windows.

[0019] The technical solution of this invention has the following advantages: The speech detection method and system use a neural network classifier including a causal TCN. Causal TCN can add memory capabilities to the network using one or more dilated convolutions. The receptive field of causal TCN can be controlled by the number of dilated convolutions and the kernel size. Compared with other neural networks such as RNNs, causal TCN has lower complexity and can be trained faster and with lower memory requirements. The speech detection method can be implemented in real-time or near real-time. Attached Figure Description

[0020] The accompanying drawings, which are incorporated herein and form a part of this specification, illustrate embodiments of the present disclosure and, together with the description, further serve to explain the principles of the disclosure and enable those skilled in the art to make and use it. To more clearly illustrate the technical solutions in specific embodiments of the present invention or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0021] Figure 1 A block diagram of an exemplary system for speech detection according to an embodiment of the present invention is shown.

[0022] Figure 2A A block diagram of an exemplary structure of a speech detector coupled to an audio detection array according to an embodiment of the present invention is shown.

[0023] Figure 2B A block diagram of another exemplary structure of a speech detector coupled to an audio detection array according to an embodiment of the present invention is shown.

[0024] Figure 3 This is a flowchart of an exemplary method for speech detection according to an embodiment of the present invention.

[0025] Figure 4 This is a flowchart of another exemplary method for speech detection according to an embodiment of the present invention.

[0026] Figure 5 This is a flowchart of yet another exemplary method for speech detection according to an embodiment of the present invention.

[0027] Figure 6 This is a flowchart of yet another exemplary method for speech detection according to an embodiment of the present invention.

[0028] Figure 7A This is a flowchart of an exemplary method for calculating the Mel spectrum according to an embodiment of the present invention.

[0029] Figure 7B This is a graphical representation of an exemplary Mel-scale filter bank for calculating the Mel spectrum according to an embodiment of the present invention.

[0030] Figure 8A This is a schematic diagram illustrating an exemplary structure of a causal TCN according to an embodiment of the present invention.

[0031] Figure 8B This is a schematic diagram illustrating an exemplary structure of a residual block in a causal TCN according to an embodiment of the present invention.

[0032] Figure 9 This is a schematic diagram illustrating an exemplary value cache area according to an embodiment of the present invention.

[0033] Figure 10 This is a flowchart of an exemplary method for training a neural network classifier for speech detection according to an embodiment of the present invention.

[0034] Figure 11 This is a flowchart of an exemplary method for generating a dataset for training a neural network classifier for speech detection, according to an embodiment of the present invention.

[0035] Figure 12 This is a flowchart of an exemplary method for generating audio signals for training a neural network classifier for speech detection, according to an embodiment of the present invention.

[0036] Figures 13A-13E This is a graphical representation illustrating the exemplary speech detection performance of a causal TCN according to an embodiment of the present invention. Detailed Implementation

[0037] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Reference will now be made in detail to exemplary embodiments, examples of which are shown in the accompanying drawings. Where possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts.

[0039] Although specific configurations and arrangements have been discussed, it should be understood that this is for illustrative purposes only. Those skilled in the art will recognize that other configurations and arrangements can be used without departing from the spirit and scope of this disclosure. It will be apparent to those skilled in the art that this disclosure can also be used in a variety of other applications.

[0040] It should be noted that references to "one embodiment," "example embodiment," "some embodiments," "certain embodiments," etc., in the specification indicate that the described embodiments may include specific features, structures, or characteristics, but each embodiment may not necessarily include those specific features, structures, or characteristics. Furthermore, these phrases do not necessarily refer to the same embodiment. Additionally, when a specific feature, structure, or characteristic is described in connection with an embodiment, implementing such a feature, structure, or characteristic in relation to other embodiments is within the knowledge of those skilled in the art, whether explicitly described or not.

[0041] Generally, terms can be understood, at least in part, from their use in context. For example, depending at least in part on the context, the term “one or more” as used herein can be used to describe any feature, structure, or characteristic in a singular sense, or in a plural sense, to describe a combination of features, structures, or characteristics. Similarly, depending at least in part on the context, terms such as “a” or “the” can also be understood to convey either a singular or a plural usage. Furthermore, the term “based on” can be understood to not necessarily be intended to convey an exclusive set of factors, and can conversely allow for the presence of additional factors that are not necessarily explicitly described, again depending at least in part on the context.

[0042] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0043] Generally, audio signals containing speech data, such as speech signals, can be considered a semi-stationary stochastic process. Audio signals can be considered static for short periods (e.g., 20-40 milliseconds). Audio signals can be segmented into a series of frames using overlapping sliding time windows. This series of frames can be viewed as a temporal sequence of data, where contextual information from one or more previous frames can be used to analyze the current frame. By transforming the frames from the time domain to the frequency domain, audio signal analysis can be performed in the frequency domain. For example, audio signal classification can be performed in the frequency domain to determine whether the audio signal includes speech data within a series of time windows.

[0044] This disclosure provides speech detection techniques, such as systems and methods for improving speech detection, by applying robust neural network classifiers to multi-channel audio input for real-time or near-real-time speech or non-speech classification. Multi-channel audio input can be obtained in noisy and reverberant environments via an audio detection array. The neural network classifier can include a causal TCN. The causal TCN can add memory capabilities to the network using one or more dilated convolutions. The receptive field of the causal TCN can be controlled by the number of dilated convolutions and the kernel size. Compared to other neural networks such as RNNs, causal TCNs have lower complexity and can be trained faster and with lower memory requirements. As a result, the speech detection techniques described herein can be implemented in real-time or near real-time.

[0045] Consistent with publicly available information, the term "near real-time" refers to data processing that responds quickly to events with a slight delay. This slight delay can be in milliseconds (ms), seconds, or minutes, depending on various factors such as computing power, available memory, and the signal sampling rate. For example, the speech detection technique described herein can be performed near real-time with a delay of milliseconds.

[0046] Figure 1A block diagram 100 of an exemplary system 101 for speech detection according to an embodiment of the present invention is shown. In some embodiments, system 101 may be embodied in a device with which a user 112 may interact. For example, the device may be a robot (e.g., a humanoid robot) or any other interactive machine with which the user 112 may interact. The user 112 may be in a noisy and reverberant environment and interact with the device using voice, gestures, or any other method. The user 112 may be a person who can “speak” and send audio signals or another robot or device (e.g., a speaker).

[0047] In some embodiments, system 101 and user 112 may be located at the same site. For example, system 101 may be implemented in a local robot, and user 112 may interact with the local robot face-to-face. Alternatively, user 112 may also interact with the local robot via a local network (not shown).

[0048] In some other embodiments, system 101 and user 112 may be located at different locations, and user 112 may interact with system 101 via a network (not shown). For example, system 101 may be implemented in a remote robot at a remote site, while an audio detection array may be located at the same location as user 112 to monitor and record acoustic data from the environment in which user 112 is located. For example, the audio detection array may be a microphone array installed in a user device operated by user 112. The audio detection array may provide the recorded acoustic data to system 101 via a network so that user 112 can interact with the remote robot.

[0049] In some embodiments, system 101 may include at least one processor (e.g., processor 102), at least one memory (e.g., memory 103), and at least one memory (e.g., memory 104). System 101 may also include at least one audio detection array, such as audio detection array 106. In some embodiments, system 101 may also include one or more databases (e.g., training database 107, validation database 108, etc.). Training database 107 and validation database 108 can be used to train a neural network classifier. System 101 may also include one or more caches for data storage (e.g., ...). Figure 2A-2B (As shown in feature cache 208 and value cache 209). It should be understood that system 101 may also include any other suitable components for performing the functions described herein.

[0050] In some embodiments, system 101 may have different modules in a single device (e.g., an integrated circuit (IC) chip) or separate devices with dedicated functions. For example, the IC may be implemented as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). In some embodiments, one or more components of system 101 may be located in a cloud computing environment, or alternatively in a single location (e.g., inside a robot) or distributed locations. Components of system 101 may be located in an integrated device or distributed in different locations, but may communicate with each other via a network. For example, audio detection array 106 may or may not be located in the same location as system 101. If audio detection array 106 is located in a different location than system 101, audio detection array 106 may be coupled to system 101 via a network.

[0051] Processor 102 may include any suitable type of general-purpose or special-purpose microprocessor, digital signal processor, graphics processing unit (GPU), or microcontroller. Processor 102 may include one or more hardware units (e.g., part of an integrated circuit) designed to be used with other components or to execute part of a program. This program may be stored on a computer-readable medium and, when executed by processor 102, may perform one or more functions. Processor 102 may be configured as a separate processor module dedicated to speech detection. Alternatively, processor 102 may be configured as a shared processor module for performing other functions unrelated to speech detection.

[0052] Processor 102 may include several modules, such as training module 110 and speech detector 105. In some embodiments, training module 110 may include training data generator 109. Although Figure 1 The training module 110 and speech detector 105 are shown within a single processor 102, but they are more likely to be implemented on different processors located close to or far from each other. For example, the training module 110 may be implemented by a processor dedicated to offline training (e.g., one or more GPUs), while the speech detector 105 may be implemented by another processor for near real-time detection.

[0053] The training module 110 and the speech detector 105 (and any corresponding submodules or subunits) may be hardware units (e.g., portions of an integrated circuit) of the processor 102, designed to be used in conjunction with other components or software units implemented by the processor 102 by executing at least a portion of a program. This program may be stored on a computer-readable medium, such as memory 103 or memory 104, and when executed by the processor 102, it may perform one or more functions.

[0054] In some embodiments, the speech detector 105 may be configured to receive multichannel audio input from the audio detection array 106. The speech detector 105 may process the multichannel audio input using a neural network classifier to determine whether the multichannel audio input includes one or more speech segments. For example, the speech detector 105 may use a neural network classifier to process the multichannel audio input to generate a series of classification results within a series of time windows of the multichannel audio input. The speech detector 105 may determine, based on this series of classification results, whether the multichannel audio input includes one or more speech segments within that series of time windows. References are made below. Figure 2A-9 The speech detector 105 is described in more detail.

[0055] In some embodiments, the neural network classifier may include a causal TCN, configured to determine the classification result for each time window based on portions of the multi-channel audio input within a corresponding time window and one or more time windows preceding that corresponding time window. An exemplary structural reference for a causal TCN is provided. Figures 8A-8B To explain.

[0056] In some embodiments, training module 110 may be configured to train a neural network classifier. For example, training data generator 109 may be configured to generate training data such that training module 110 can use the training data to train the neural network classifier before speech detector 105 applies it to speech detection. References below Figure 10-12 The training module 110 and the training data generator 109 are described in more detail.

[0057] Memory 103 and storage 104 may include any suitable type of mass storage provided to store any type of information that the processor 102 may need to operate. For example, memory 103 and storage 104 may be volatile or non-volatile, magnetic, semiconductor-based, magnetic tape-based, optical, removable, non-removable, or other types of storage devices or tangible (i.e., non-transitory) computer-readable media, including but not limited to ROM, flash memory, dynamic RAM, and static RAM. Memory 103 and / or storage 104 may be configured to store one or more computer programs that can be executed by the processor 102 to perform the functions disclosed herein. For example, memory 103 and / or storage 104 may be configured to store a program that can be executed by the processor 102 to perform speech detection on an audio signal. Memory 103 and / or storage 104 may be further configured to store information and data used by the processor 102.

[0058] The audio detection array 106 may include a set of audio detection devices configured to record acoustic signals in the presence of the user 112. This set of audio detection devices may include one or more audio detection devices, such as one or more microphones. The set of audio detection devices may be aligned on the same line or arranged in a rectangular array, a circular array, or any other suitable form. For example, the set of audio detection devices may include four microphones forming a circular microphone array assembled on the head of a humanoid robot.

[0059] In some embodiments, the set of audio detection devices can each generate a set of audio signals from a set of channels. This set of audio signals can form a multi-channel audio input for the speech detector 105. For example, each audio detection device can record acoustic signals in the environment in one channel and generate an audio signal based on the recorded acoustic signals. If user 112 speaks in the environment, the audio signal may include speech data. In some cases, the audio signal may be corrupted by one or more noises, echoes, or reverberation in the environment. Due to various factors, such as the different orientations of the audio detection devices and different noise levels at the audio detection devices, the audio detection devices may generate different audio signals for the same environment.

[0060] Training database 107 can be configured to store training data for training a neural network classifier. For example, training database 107 can store one or more training datasets. In another example, training database 107 can store data from... Figure 12 The audio signal generated by method 1200 is shown below. (Refer to the following text.) Figure 10-12 The training database 107 is described in more detail.

[0061] The validation database 108 can be configured to store validation data to validate the training of a neural network classifier. For example, the validation database 108 can store one or more validation datasets.

[0062] In some embodiments, the training module 110, the training database 107, and the validation database 108 can be implemented in another system including a GPU, which can be separate from system 101. This other system including the GPU can then be dedicated to offline training of the neural network classifier, and system 101, including the speech detector 105, can be used for speech detection.

[0063] Figure 2AA block diagram 200 illustrates an exemplary structure of a speech detector 105 coupled to an audio detection array 106 according to an embodiment of the present invention. The audio detection array 106 may include a group of audio detection devices 202A, 202B, ..., 202N (also individually or collectively referred to as audio detection device 202). The speech detector 105 may include a preprocessing unit 204, a speech activity detector 206, and a decision module 207. In some embodiments, the preprocessing unit 204 may be a component of the audio detection array 106 and integrated into the audio detection array 106. The speech activity detector 206 may be coupled to one or more storage devices of the system 101, such as a feature cache 208 and a value cache 209.

[0064] In some embodiments, each audio detection device 202 can capture acoustic signals in the environment where user 112 is present. Each audio detection device 202 can generate an audio signal based on the captured acoustic signals and provide the audio signal to preprocessing unit 204. As a result, preprocessing unit 204 can receive a set of audio signals from the group of audio detection devices 202. Preprocessing unit 204 can be configured to process the set of audio signals to output a combined audio signal. The combined audio signal can include a combination of acoustic information captured by the group of audio detection devices 202. For example, preprocessing unit 204 can include a spatial filter or beamformer configured to filter the set of audio signals to output the combined audio signal. The spatial filter or beamformer can be directed towards the estimated direction of arrival of the acoustic signals.

[0065] In some embodiments, the voice activity detector 206 may be configured to process the combined audio signals using a neural network classifier to generate a series of classification results within a series of time windows of the combined audio signals, as described in more detail below. Each time window may have a predetermined duration and may or may not overlap with adjacent time windows. For example, each time window may have a length of 16 milliseconds and an overlap of 8 milliseconds with adjacent time windows (e.g., 4 milliseconds overlap with the previous time window and 4 milliseconds overlap with the next time window).

[0066] Specifically, the speech activity detector 206 can divide the combined audio signal into a series of frames within a series of time windows. For example, the speech activity detector 206 can generate a corresponding frame for each time window from the combined audio signal, thereby generating a series of frames for the series of time windows. Here, the series of frames from the combined audio signal can correspond one-to-one with the series of time windows. Each frame can include a segment of the combined audio signal within the corresponding time window and can overlap with adjacent frames using a raised-cosine window. For example, each frame can include a segment of the combined audio signal with a length of 16 milliseconds and can have an 8-millisecond overlap with adjacent frames using a raised-cosine window. In some embodiments, the sampling rate of the audio signal (or the combined audio signal) can be 16 kHz. Therefore, the total number of samples in each 16-millisecond frame can be 16 ms × 16 kHz = 256.

[0067] Next, for each time window, the speech activity detector 206 can obtain a frame corresponding to the corresponding time window from the series of frames and generate a feature sequence for that frame. The feature sequence may include the features of that frame and T-1 features from T-1 previous frames, where T is a positive integer. In some embodiments, the features of the frame may include the Mel-frequency spectrum of the frame.

[0068] For example, the speech activity detector 206 can process the frame using a high-pass filter to remove DC offset and low-frequency noise. The speech activity detector 206 can calculate the Mel spectrum of the frame. The speech activity detector 206 can also calculate the Mel spectra of T-1 previous frames. Alternatively, the speech activity detector 206 can retrieve the Mel spectra of T-1 previous frames from the feature buffer 208. Then, the speech activity detector 206 can concatenate the Mel spectrum of the frame with the Mel spectra of the T-1 previous frames to generate the feature sequence. See below for reference. Figure 7A A method for calculating the Mel spectrum is described. The speech activity detector 206 can store the feature sequence of a frame in a feature buffer 208. In some embodiments, the feature buffer 208 can be configured to store a series of features from a series of frames, thereby allowing the stored features to be used to form a feature sequence for each frame.

[0069] For each time window, the speech activity detector 206 can input the feature sequence of the corresponding frame into a neural network classifier to generate a value sequence for that frame. This value sequence may include a value estimate for the corresponding frame and T-1 value estimates for T-1 previous frames. The speech activity detector 206 can update the value buffer 209 based on this value sequence.

[0070] Value buffer 209 can be configured to store value estimates for each frame. In some embodiments, value buffer 209 can store a sequence of values ​​for each frame. The value estimate for a frame can include a value representing, for example, the probability that the frame includes speech data. In some embodiments, multiple value estimates can be generated and stored for each frame, such as... Figure 9 As shown, the following text will describe it in more detail. Figure 9 .

[0071] For each time window, the speech activity detector 206 can also generate an estimate for the corresponding time window based on the value buffer 209. For example, the speech activity detector 206 can generate an estimate for the corresponding time window based on one or more value estimates of frames stored in the value buffer 209. In another example, the estimate can be the average, maximum, minimum, or median of the values ​​in one or more value estimates. See below for reference. Figure 9 This describes an exemplary method for generating estimation results from one or more value estimates.

[0072] For each time window, decision module 207 can be configured to determine the classification result for that time window based on the estimation result. For example, decision module 207 can compare the estimation result with a predetermined threshold (e.g., 0.5 or any other suitable value), and if the estimation result is not less than the threshold (“speech classification result”), the signal in the corresponding time window is determined to be speech. If the estimation result is less than the threshold, decision module 207 determines the signal in the corresponding time window to be non-speech (“non-speech classification result”). The classification result can be either a speech classification result (e.g., a “1” value indicating the presence of speech data in the corresponding time window) or a non-speech classification result (e.g., a “0” value indicating the absence of speech data in the corresponding time window).

[0073] Therefore, a series of classification results can be generated for a series of time windows from the multi-channel audio input, where each time window has a classification result (equivalently, each frame from the combined audio signal has a classification result). The decision module 207 can also be configured to determine, based on the series of classification results, whether the multi-channel audio input includes one or more speech segments within the series of time windows. Specifically, for each time window, the decision module 207 can determine whether a speech segment exists in the corresponding time window based on the classification result of that time window. For example, if the classification result is a speech classification result (e.g., "1"), the decision module 207 determines that the multi-channel audio input includes speech data in the corresponding time window. If the classification result is a non-speech classification result (e.g., "0"), the decision module 207 determines that the multi-channel audio input does not include speech data in the corresponding time window.

[0074] Figure 2B A block diagram 250 illustrates another exemplary structure of a speech detector 105 coupled to an audio detection array 106 according to an embodiment of the present invention. The speech detector 105 may include a decision module 207 and a set of speech activity detectors 206A, 206B, ..., 206N (also collectively or individually referred to as speech activity detectors 206). Each speech activity detector 206 may be coupled to a feature buffer 208 and a value buffer 209.

[0075] In some embodiments, each audio detection device 202 can capture acoustic signals in the environment where user 112 is present, and can generate audio signals based on the captured acoustic signals. As a result, the group of audio detection devices 202 can provide a set of audio signals to speech detector 105. Each speech activity detector 206 in speech detector 105 can receive a corresponding audio signal and process the corresponding audio signal using a neural network classifier to perform speech estimation on the corresponding audio signal, as described in more detail below.

[0076] Specifically, the speech activity detector 206 can segment the corresponding audio signal into a series of frames within a series of time windows. For example, the speech activity detector 206 can generate one frame for each time window from the corresponding audio signal, thereby generating a series of frames for the series of time windows. Each frame can include a segment of the audio signal during the corresponding time window and can overlap with adjacent frames using raised cosine windows.

[0077] Next, for each time window, the speech activity detector 206 can obtain a frame corresponding to the corresponding time window from the series of frames of the corresponding audio signal and generate a feature sequence for that frame. The speech activity detector 206 can store this feature sequence in the feature buffer 208. The speech activity detector 206 can input the feature sequence of the frame into a neural network classifier to generate a value sequence and update the value buffer 209 based on the value sequence. The speech activity detector 206 can generate an estimation result for the corresponding time window based on the value buffer 209. As a result, the speech estimation generated by the speech activity detector 206 can include, for example, a series of estimation results for the series of time windows, wherein one estimation result is used for one time window.

[0078] By performing operations similar to those described above, the group of speech activity detectors 206 can generate a set of speech estimates for each group of audio signals. For each time window, the set of speech estimates can include the corresponding group estimate results for that group of audio signals. Alternatively, the group of audio signals can be processed one by one by applying a single speech activity detector 206 multiple times to generate the set of speech estimates.

[0079] For example, suppose there are K speech activity detectors 206 for processing K audio signals, where K is a positive integer. This series of time windows includes W time windows TW(1), TW(2), ..., TW(W), where W is a positive integer. For an audio signal S(k), 1 ≤ k ≤ K, the speech activity detectors 206k generate a speech estimate E(k) comprising W estimation results across the W time windows. For example, the W estimation results for the audio signal S(k) are ERk(1), ERk(2), ..., ERk(W), where the first estimation result ERk(1) is used for the first time window TW(1), the second estimation result ERk(2) is used for the second time window TW(2), and so on. Table 1 below illustrates the estimation results based on the W time windows of the K audio signals.

[0080] Table 1

[0081]

[0082] As can be seen from each row of Table 1, each speech estimate E(k) includes a series of estimation results (ERk(1), ERk(2), ..., and ERk(W)) for the series of time windows (TW(1), TW(2), ..., and TW(W)). As can also be seen from each column of Table 1, for each time window TW(w), 1≤w≤W, the set of speech estimates (E(1), E(2), ..., and E(K))) can include a corresponding set of estimation results (e.g., K estimation results, including ER1(w), ER2(w), ..., and ERK(w)) for the time window TW(w).

[0083] The decision module 207 can be configured to fuse the set of speech estimates from the set of audio signals to generate a series of classification results within a series of time windows for multi-channel audio input. Specifically, for each time window, the decision module 207 can fuse the corresponding set of estimates to generate the classification result for that time window, as described in more detail below.

[0084] In some embodiments, for each time window, the decision module 207 can determine a fusion estimation result from a corresponding set of estimation results. The fusion estimation result can be, for example, the maximum, minimum, average, or other statistical arrangement of the corresponding set of estimation results. For example, the fusion estimation result for time window TW(w) can be the maximum, minimum, or average of the K estimation results (ER1(w), ER2(w), ..., ERK(w)) for time window TW(w). The decision module 207 can then determine a classification result based on the fusion estimation result. For example, for each time window, the decision module 207 can compare the fusion estimation result with a predetermined threshold (e.g., 0.5), and if the fusion estimation result is not less than the predetermined threshold, then the speech classification result for the corresponding time window is determined (e.g., "1"). If the fusion estimation result is less than the predetermined threshold, then the decision module 207 determines a non-speech classification result for the corresponding time window (e.g., "0").

[0085] In some embodiments, for each time window, the decision module 207 can determine a set of preliminary classification results based on a corresponding set of estimation results. Then, the decision module 207 can determine the classification result for the corresponding time window based on the set of preliminary classification results.

[0086] For example, for each time window TW(w), decision module 207 can compare each estimate in the corresponding set of estimates (ER1(w), ER2(w), ..., and ERK(w)) with a predetermined threshold to determine the preliminary classification result for the corresponding time window. If the estimate is not less than the predetermined threshold, a speech classification result (“1”) is generated as the preliminary classification result; otherwise, a non-speech classification result (e.g., “0”) is generated as the preliminary classification result. As a result, a set of preliminary classification results is generated based on the corresponding set of estimates (ER1(w), ER2(w), ..., and ERK(w)). Next, if any preliminary classification result is a speech classification result (“1”), decision module 207 can determine the speech classification result (“1”) for the corresponding time window. If none of the preliminary classification results are speech classification results (“1”), decision module 207 can determine the non-speech classification result (“0”) for the corresponding time window.

[0087] By performing similar operations, decision module 207 can generate a series of classification results for a series of time windows, with each time window having one classification result. Decision module 207 can be configured to determine, based on this series of classification results, whether the multi-channel audio input from audio detection array 106 includes one or more speech segments within that series of time windows. For example, for each time window, if the classification result for the corresponding window is a speech classification result (e.g., "1"), then decision module 207 determines that the multi-channel audio input includes speech data in the corresponding time window. If the classification result is a non-speech classification result (e.g., "0"), then decision module 207 determines that the multi-channel audio input does not include speech data in the corresponding time window.

[0088] and Figure 2A-2B Consistently, the audio detection array 106 can be configured to detect a set of audio signals as different representations of the same underlying acoustic signal. This set of audio signals can form a multi-channel audio input for the speech detection technique described herein. The multi-channel audio input can capture information from acoustic signals from different directions and is more resistant to noise and reverberation compared to capturing acoustic signals from a single direction. As a result, the speech detection accuracy of the speech detection technique described herein can be improved based on the multi-channel audio input.

[0089] Figure 3 This is a flowchart of an exemplary speech detection method 300 according to an embodiment of the present invention. Method 300 can be implemented by system 101, particularly speech detector 105, and may include steps 302-306 as described below. Some steps may be optional to perform the disclosure provided herein. Furthermore, some steps may be performed simultaneously or in a different manner. Figure 3 Execute in the order shown.

[0090] In step 302, the speech detector 105 receives a multi-channel audio input, which includes a set of audio signals from a set of audio channels in the audio detection array 106.

[0091] In step 304, the speech detector 105 processes the multi-channel audio input using a neural network classifier to generate a series of classification results within a series of time windows for the multi-channel audio input. In some embodiments, the neural network classifier may include a causal TCN configured to determine the classification result for each time window based on portions of the multi-channel audio input within the corresponding time window and one or more time windows preceding the corresponding time window.

[0092] In some embodiments, the voice detector 105 may perform Figure 4Steps 404-409 are used to generate classification results for each time window. See below for reference. Figure 4 Describe steps 404-409. In some embodiments, the speech detector 105 may perform... Figure 5 Steps 504-506 are used to generate classification results for each time window. See below for reference. Figure 5 Describe steps 504-506. In some embodiments, the speech detector 105 may perform... Figure 6 Steps 604-618 are used to generate classification results for each time window. See below for reference. Figure 6 Describe steps 604-618.

[0093] In step 306, the speech detector 105 determines, based on the series of classification results, whether the multi-channel audio input includes one or more speech segments within the series of time windows. For example, for each time window, the speech detector 105 determines, based on the classification results for the corresponding time window, whether a speech segment exists in the multi-channel audio input during the corresponding time window.

[0094] Figure 4 This is a flowchart of another exemplary method 400 for speech detection according to an embodiment of the present invention. Method 400 can be provided by system 101, particularly speech detector 105 (e.g., Figure 2A This is implemented using the preprocessing unit 204, the voice activity detector 206, and the decision module 207 of the voice detector 105 shown, and may include steps 402-410 as described below. In some embodiments, steps 404-409 may be performed to achieve... Figure 3 Step 304 in the document. Some steps may be optional to perform the disclosure provided herein. Furthermore, some steps may be performed concurrently or in conjunction with... Figure 4 The different execution orders shown.

[0095] In step 402, the preprocessing unit 204 receives a multi-channel audio input, which includes a set of audio signals from a set of audio channels in the audio detection array 106.

[0096] In step 404, the preprocessing unit 204 preprocesses the set of audio signals to output a combined audio signal. For example, the preprocessing unit 204 may include a spatial filter or a beamformer to filter the set of audio signals and generate a combined audio signal as output.

[0097] In step 406, the speech activity detector 206 segments the combined audio signal into a series of frames within a series of time windows. For example, the speech activity detector 206 can generate one frame for each time window from the combined audio signal, thereby generating a series of frames for the series of time windows. Each frame may include a segment of the combined audio signal within the corresponding time window and overlaps with adjacent frames using a raised cosine window.

[0098] In step 408, the speech activity detector 206 applies each frame in the series of frames to a neural network classifier to generate an estimate of the corresponding time window for that frame.

[0099] Specifically, the speech activity detector 206 can generate a feature sequence for the frame. For example, the speech activity detector 206 can perform... Figure 6 Steps 606-610 are performed to generate a feature sequence for the frame. Next, the speech activity detector 206 can input the feature sequence into a neural network classifier to produce a value sequence. This value sequence may include a value estimate for the frame and (T-1) value estimates for (T-1) previous frames. The value buffer 209 can be updated based on the value sequence. The speech activity detector 206 can generate an estimate for a corresponding time window based on the value buffer 209. For example, the speech activity detector 206 can generate the estimate for the corresponding time window as the average, maximum, minimum, or median of the values ​​among multiple value estimates of the frame stored in the value buffer 209. In another example, the speech activity detector 206 can select one value from multiple value estimates as the estimate for the corresponding time window.

[0100] In step 409, decision module 207 determines the classification result for the corresponding time window based on the estimation result. For example, decision module 207 can compare the estimation result with a predetermined threshold, and if the estimation result is not less than the predetermined threshold, then the speech classification result for the corresponding time window is determined (e.g., "1"). If the estimation result is less than the predetermined threshold, then decision module 207 determines the non-speech classification result for the corresponding time window (e.g., "0").

[0101] In step 410, decision module 207 determines whether a speech segment exists in the multi-channel audio input during the corresponding time window based on the classification result of the corresponding time window. For example, if the classification result is a speech classification result (e.g., "1"), then decision module 207 determines that the multi-channel audio input includes speech data in the corresponding time window. If the classification result is a non-speech classification result (e.g., "0"), then decision module 207 determines that the multi-channel audio input does not include speech data in the corresponding time window.

[0102] Figure 5This is a flowchart of yet another exemplary method 500 for speech detection according to an embodiment of the present invention. Method 500 can be provided by system 101, particularly speech detector 105 (e.g., Figure 2B This is implemented using a set of voice activity detectors 206 and a decision module 207 (as shown in the speech detector 105), and may include steps 502-508 as described below. In some embodiments, steps 504-506 may be performed to achieve... Figure 3 Step 304 in the document. Some steps may be optional to perform the disclosure provided herein. Furthermore, some steps may be performed concurrently or in a different manner. Figure 5 Execute in the order shown.

[0103] In step 502, the speech detector 105 receives a multi-channel audio input comprising a set of audio signals from a set of audio channels in the audio detection array 106. For example, each speech activity detector 206 in the speech detector 105 may receive a corresponding audio signal from the audio detection array 106.

[0104] In step 504, speech detector 105 processes each audio signal using a neural network classifier to perform speech estimation based on the corresponding audio signal, thereby generating a set of speech estimates for the set of audio signals. For each time window, the set of speech estimates may include a corresponding set of estimation results for the set of audio signals. For example, each speech activity detector 206 may use a neural network classifier to process the corresponding audio signal to perform speech estimation based on the corresponding audio signal. As a result, by performing similar operations, the set of speech activity detectors 206 can generate a set of speech estimates for the set of audio signals respectively.

[0105] In step 506, the speech detector 105 fuses the set of speech estimates from the set of audio signals to generate a series of classification results within a series of time windows for the multi-channel audio input. For example, for each time window, the decision module 207 in the speech detector 105 can fuse a corresponding set of estimates included in the set of speech estimates to generate a classification result for the corresponding time window. As a result, the decision module 207 can generate a series of classification results for a series of time windows, where each time window has one classification result.

[0106] In step 508, for each time window, the speech detector 105 determines whether a speech segment exists in the multichannel audio input during the corresponding time window based on the classification result of the corresponding time window. For example, for each time window, the decision module 207 determines whether the multichannel audio input includes speech data in the corresponding time window based on the classification result of the corresponding time window.

[0107] Figure 6This is a flowchart of yet another exemplary method 600 for speech detection according to an embodiment of the present invention. Method 600 can be implemented by system 101, specifically by speech detector 105 (e.g., Figure 2A-2B The speech detector 105 shown includes a speech activity detector 206 and a decision module 207, and may include steps 602-620 as described below. In some embodiments, steps 604-618 may be performed to achieve... Figure 3 Step 304 in the document. Some steps may be optional to perform the disclosure provided herein. Furthermore, some steps may be performed concurrently or in conjunction with... Figure 6 The different execution orders shown.

[0108] In step 602, the voice activity detector 206 receives the audio signal and stores it in a buffer. For example, the audio signal could be generated by... Figure 2A The preprocessing unit 204 outputs a combined audio signal. Alternatively, the audio signal can be any audio signal from a set of audio signals output by the audio detection array 106.

[0109] In step 604, the voice activity detector 206 generates a series of frames based on the audio signal within a series of time windows.

[0110] In step 606, the speech activity detector 206 processes each frame using a high-pass filter. For example, the speech activity detector 206 can use a high-pass filter to remove DC offset and low-frequency noise from the corresponding frame.

[0111] In step 608, the speech activity detector 206 calculates the Mel spectrum for each frame in the frame sequence. The speech activity detector 206 can store the Mel spectrum of each frame in the feature buffer 208.

[0112] In step 610, for each frame, the speech activity detector 206 concatenates the Mel spectrum of the corresponding frame with the Mel spectra of T-1 previous frames to generate a feature sequence for that frame. For example, the speech activity detector 206 can read the Mel spectra of T-1 previous frames from the feature buffer 208 and can use the Mel spectra of T-1 previous frames and the Mel spectrum of the corresponding frame to form a feature sequence.

[0113] In step 612, the speech activity detector 206 inputs the feature sequence into a neural network classifier to generate a value sequence for the corresponding frame. The value sequence may include a value estimate for the corresponding frame and T-1 value estimates for T-1 previous frames. The value sequence may have the same length as the feature sequence.

[0114] In step 614, the voice activity detector 206 updates the value buffer 209 using the value sequence. For example, the voice activity detector 206 stores the value sequence in the value buffer 209.

[0115] In step 616, the speech activity detector 206 generates an estimation result for each frame in the corresponding time window based on the value buffer 209. For example, the value buffer 209 may include one or more value estimates for each frame. The speech activity detector 206 may generate an estimation result for each frame as the average, maximum, or minimum value of one or more values ​​included in the one or more value estimates for the corresponding frame.

[0116] In step 618, decision module 207 determines the classification result for the corresponding time window based on the estimation result. For example, if the estimation result is not less than a predetermined threshold, decision module 207 can compare the estimation result with the predetermined threshold and determine the speech classification result for the corresponding time window (e.g., "1"). If the estimation result is less than the predetermined threshold, decision module 207 determines the non-speech classification result for the corresponding time window (e.g., "0").

[0117] In step 620, the decision module 207 determines whether a speech segment exists in each frame of the corresponding time window based on the classification result within that time window. For example, if the classification result is a speech classification result (e.g., "1"), the decision module 207 determines that the multi-channel audio input includes speech data in the corresponding time window. If the classification result is a non-speech classification result (e.g., "0"), the decision module 207 determines that the multi-channel audio input does not include speech data in the corresponding time window.

[0118] Figure 7A This is a flowchart of an exemplary method 700 for calculating the Mel spectrum of a frame according to an embodiment of the present invention. Method 700 can be implemented by system 101, particularly speech detector 105 (e.g., Figure 2A-2B The voice activity detector 206 shown may include steps 702-710 as described below. In some embodiments, steps 704-710 may be performed to achieve... Figure 6 Step 608 in the document. Some steps may be optional to perform the disclosure provided herein. Furthermore, some steps may be performed concurrently or in a different manner. Figure 7A Execute in the order shown.

[0119] In step 702, the voice activity detector 206 receives one frame from a series of frames. This series of frames can be obtained, for example, from an audio signal (or a combination of audio signals). Each frame may include one or more samples of the audio signal (or the combination of audio signals) within a corresponding time window.

[0120] In step 704, the speech activity detector 206 calculates the Fast Fourier Transform (FFT) of the frame to obtain the FFT power spectrum of the frame. For example, the FFT of the frame is calculated using an FFT size equal to the total number of samples included in the frame.

[0121] In step 706, the speech activity detector 206 converts the FFT power spectrum into a Mel spectrum by multiplying the FFT power spectrum by a Mel-scale filter bank. For example, the ear is more discriminative at lower frequencies and less discriminative at higher frequencies. The Mel scale is designed to simulate the non-linear perception of sound by the human ear. A Mel filter bank can produce higher resolution at low frequencies and lower resolution at high frequencies. In some embodiments, the Mel-scale filter bank may include multiple filters (e.g., 24 triangular filters). Figure 7B The image shows an exemplary Mel-scale filter bank with a triangular filter.

[0122] In step 708, the speech activity detector 206 calculates the logarithm of the Mel spectrum. For example, the speech activity detector 206 can calculate the logarithm of the Mel spectrum to base 10.

[0123] In step 710, the speech activity detector 206 moves and scales the logarithm of the Mel spectrum. For example, the logarithm of the Mel spectrum can be moved and scaled to remain within the same range as the logarithms of the Mel spectra in other frames.

[0124] Figure 8A This is a schematic diagram 800 illustrating an exemplary structure of a causal TCN according to an embodiment of the present invention. In a "causal" TCN, there is no leakage of information from the future to the past. For example, the output of a causal TCN at time step t is determined based on the input of the TCN at time step t and one or more previous inputs from one or more previous time steps preceding time step t. In some embodiments, a causal TCN uses one or more dilated convolutions to add memory capacity to the network. The application of dilated convolutions causes the receptive field of the causal TCN to grow exponentially.

[0125] In some embodiments, a causal TCN may include one or more cascaded residual blocks 802. For example, a causal TCN may include N cascaded residual blocks, with a total depth of N, where N is a positive integer. Figure 8BAs shown, each residual block 802 may include an expanded one-dimensional (1D) convolution. In some embodiments, the expansion factor d of the expanded 1D convolution in each residual block 802 may increase exponentially with the depth of the residual block 802. For example, for the nth residual block (1≤n≤N), the expansion factor d of the corresponding expanded 1D convolution is d = 2. n-1 .

[0126] In some embodiments, the causal TCN may further include one or more of a time-distributed fully connected layer 804 with a hyperbolic tangent (tanh) activation function, a dropout layer 806, and a dense layer 808 with a tanh activation function. The time-distributed fully connected layer 804, dropout layer 806, and dense layer 808 may be applied in series with the residual block 802. The time-distributed fully connected layer 804 may be a time-distributed dense layer with D units and configured to expand the output of the residual block 802, where D is a positive integer. The dropout layer 806 may be configured for regularization. The dense layer 808 may be a time-distributed dense layer with D units (e.g., D = 1 or any other suitable value) and configured to adjust the dimension of the causal TCN's output. D may be the number of neurons in the dense layer 808. For example, the dense layer 808 may be configured to adjust the output of the causal TCN to a size of T × 1.

[0127] In some embodiments, the feature sequence may include features of the current frame concatenated with T-1 features from T-1 previous frames, where each feature is a vector of length B, and T and B are positive integers. Therefore, the feature sequence may have a size of T×B (e.g., length T, width B). Here, T represents the number of frames processed by the causal TCN, which is also equal to the number of time windows (or time steps) processed by the causal TCN. B represents the number of samples in each feature, such as the number of feature samples in each Mel spectrum (e.g., the number of Mel bands). F represents... Figure 8B The number of filters in the dilated one-dimensional convolution is shown. If the features of a frame are represented by the Mel spectrum of that frame, then B also represents the number of samples in the Mel spectrum of that frame.

[0128] This describes an example of the operation of a causal TCN. Initially, the feature sequence can be input into the causal TCN and processed by residual block 802, generating an output of size T×F. Here, F represents the number of filters in each 1D dilated convolution of residual block 802. For example, the first residual block of N residual blocks can receive the feature sequence as input and generate a first intermediate result of size T×F. The first intermediate result can be input into the second residual block, so that the second residual block can generate a second intermediate result of size T×F. Similarly, the (n-1)th intermediate result can be input into the nth residual block, so that the nth residual block can generate an nth intermediate result of size T×F. The output of residual block 802 can be the Nth intermediate result generated by the Nth residual block of size T×F.

[0129] Next, the output of residual block 802 can be fed into the temporally distributed fully connected layer 804 and discard layer 806 to generate an output of size T×D. This T×D output can be processed by dense layer 808 to produce an output of size T×1, which is a sequence of values. Dense layer 808 can then transform the T×D output of discard layer 806 into an output of size T×1. This value sequence may include value estimates for the current frame concatenated with T-1 value estimates from T-1 previous frames (equivalently, this value sequence may include value estimates for the current time window concatenated with T-1 value estimates from T-1 previous time windows). For example, each value estimate may include values ​​in the range [0, 1].

[0130] Figure 8B This is a schematic diagram 850 illustrating an exemplary structure of a residual block (e.g., residual block 802) in a causal TCN according to an embodiment of the present invention. In some embodiments, the causal TCN includes N serially connected residual blocks. Figure 8B The residual block in the diagram can represent the nth residual block (1 ≤ n ≤ N) among N serially connected residual blocks. The residual block can include F filters, a kernel size L, and a spread factor d = 2. (n-1) (For example, [L]Conv1D(×F), where d=2) (n-1) The extended 1D convolution 852. The residual block may also include a modified linear unit (ReLU) activation function applied in cascade with the extended 1D convolution 852. Consistent with this disclosure, [L]Conv1D(×F) can represent a 1D convolution with F filters and a kernel size L, where F represents the dimension of the output space (e.g., the number of output filters in the convolution) and the kernel size L specifies the length of the 1D convolution window.

[0131] In some embodiments, the residual block may further include a dropout layer 856 applied in series with the dilated one-dimensional convolution 852. The dropout layer 856 may be configured for regularization and applied after the ReLU activation function. For example, the dropout layer 856 may be spatial dropout added after the dilated 1D convolution 852 for regularization to avoid overfitting problems.

[0132] In some embodiments, the residual block may further include an adder 860 configured to add the output of the discard layer 856 to the input 851 of the residual block. For example, the adder 860 may be configured to perform element-wise addition on the output of the discard layer 856 and the input 851.

[0133] In some embodiments, the input 851 of the residual block (e.g., the nth residual block, 1 ≤ n ≤ N) has a size of T × Z, where Z is a positive integer. For example, if the residual block is the first residual block in a series of N residual blocks, then the input 851 will be a feature sequence of size T × B (in other words, Z = B in this case). Otherwise, the residual block (e.g., the nth residual block, n > 1) receives the (n-1)th intermediate result from the (n-1)th residual block as input 851. In this case, the input 851 has a size of T × F (in other words, Z = F in this case).

[0134] This describes an example of the residual block's operation. Initially, input 851 is processed by a dilated 1D convolution 852 and a ReLU activation function to generate a first output. The first output is then processed by a discard layer 856 to generate a second output of size T×F. Next, adder 860 can be applied to the second output from the discard layer 856 and input 851 (or a variant of input 851) to generate output 861, which is referenced above. Figure 8A The nth intermediate result described (1≤n≤N).

[0135] For example, if the residual block is the first residual block where Z = B and F ≠ B, the residual block may also include residual connections (e.g., residual 1D convolutions 858) applied in parallel with the dilated 1D convolutions 852. The remaining 1D convolutions 858 may be configured to account for the difference between the input width (e.g., B) and the output width (e.g., F) to ensure that the adder 860 (e.g., an adder for performing element-wise addition) receives tensors of the same shape. The remaining 1D convolutions 858 may have F filters and a kernel size of 1 (e.g., [1]Conv1D(×F)). Linear activation functions may be applied in concatenation with the remaining 1D convolutions 858. The remaining 1D convolutions 858 may process an input 851 of size T×B to generate a third output of size T×F. The third output may be considered a variant of the input 851. Adder 860 can be applied to the second output from the discarded layer 856 and the third output from the remaining 1D convolution 858 to generate output 861, which is the first intermediate result.

[0136] In another example, if the residual block is the first residual block and Z = F = B, then no residual connections are needed in the residual block. For example, because input 851 and the second output from discarded layer 856 have the same size, the remaining 1D convolution 858 is not included in the residual block. Adder 860 can be applied to the second output from discarded layer 856 and input 851 to produce output 861, which is the first intermediate result.

[0137] In yet another example, if the residual block is the nth residual block (n > 1, in which case Z = F), then no residual connection is needed in the residual block. Adder 860 can be applied to the second output and input 851 from the discard layer 856 to generate output 861, which is the nth intermediate result (n > 1).

[0138] and Figures 8A-8B Consistent with this paper, the causal TCN described can be a simplified version of a typical TCN. For example, a typical TCN for speech detection and classification typically uses an encoder-TCN-decoder architecture, with each residual block of a standard TCN including at least two dilated convolutions. However, the causal TCN described in this paper eliminates the encoder and decoder and can instead include only dilated causal convolutions in each residual block (hence the "simplification"). As a result, the number of hyperparameters in the causal TCN described in this paper is reduced by at least 10 orders of magnitude compared to a standard TCN, which significantly reduces the computational cost of speech detection and classification. Consequently, the speech detection technique with causal TCN described in this paper can be implemented in real-time or near real-time.

[0139] Figure 9This is a schematic diagram 900 illustrating an exemplary value cache (e.g., value cache 209) according to an embodiment of the present invention. Figure 9 The example illustrates multiple time windows M-2, M-1, M, M+1, and M+2 corresponding to frames M-2, M-1, M, M+1, and M+2, where M is a positive integer greater than 2. A value buffer stores the value estimate for each frame.

[0140] Assume T = 3 (e.g., a feature sequence includes 3 features from 3 frames) and B = 2 (e.g., 2 samples in each frame). For a time window M corresponding to frame M, a feature sequence for frame M can be generated by concatenating the features of frame M with the features of frame M-1 and frame M-2. The feature sequence of frame M can be processed by a neural network classifier to generate a value sequence 902 for frame M. The value sequence 902 includes a value estimate of frame M (including value V). M，1 ), Value estimation of frame M-1 (including value V) M-1，2 ), and value estimates for frame M-2 (including value V) M-2，3 Next, for time window M+1 corresponding to frame M+1, a value sequence for frame M+1 can be generated to include the value estimate for frame M+1 (including value V). M+1，1 ), value estimation of frame M (including value V) M，2 ) and value estimates for frame M-1 (including value V) M-1，3 Similarly, for a time window M+2 corresponding to frame M+2, a value sequence for frame M+2 can be generated to include the value estimate for frame M+2 (including value V). M+2，1 ), value estimation of frame M+1 (including value V) M+1，2 ) and value estimation of frame M (including value V) M，3 As a result, for time window M, T value estimates 904 (T=3) are generated for frame M corresponding to time window M, including V. M，t , 1≤T≤T (e.g., V M，1 V M2 , and V M，3 ).

[0141] An estimate of time window M can be generated based on the value estimate 904 of frame M. For example, the estimate could be the average, maximum, minimum, or median of the values ​​in value estimate 904. In another example, the estimate could be any value in value estimate 904.

[0142] In some embodiments, the estimation result of time window M can be output to decision module 207 with a delay of T frames (or equivalently, T time windows). Therefore, the classification result of time window M generated by decision module 207 can have a delay of T frames. As a result, the speech detection technique described herein can have a delay of T frames.

[0143] Figure 10 This is a flowchart of an exemplary method 1000 for training a neural network classifier according to an embodiment of the present invention. Method 1000 can be implemented by system 101, specifically training module 110, and may include steps 1002-1024 as described below. Some steps may be optional to perform the disclosure provided herein. Furthermore, some steps may be performed simultaneously or in conjunction with... Figure 10 The different execution orders shown.

[0144] For example, method 1000 can be applied to training. Figures 8A-8B The example shown is a causal TCN. A causal TCN can be trained over a predetermined number of epochs (one epoch means training the model once using all the data in the training set). For each training iteration (each epoch), a subset or all of the samples in the training or validation dataset are evaluated in batches. For example, the stochastic gradient descent (SGD) method can be used for batch evaluation of samples.

[0145] In step 1002, the speech activity detector 206 creates a neural network classifier. For example, the neural network classifier is created to include... Figures 8A-8B The causal TCN is shown.

[0146] In step 1004, training module 110 initializes the weights in the neural network classifier. For example, all weights can be set to be equal from the beginning. As another example, the weights can be set using the weights of a previously trained neural network classifier.

[0147] In step 1006, training module 110 determines whether a predetermined number of training epochs has been reached. If the predetermined number of training epochs has been reached, method 1000 proceeds to step 1024. Otherwise, method 1000 proceeds to step 1008.

[0148] In step 1008, training data generator 109 generates a training dataset. The training dataset may include one or more batches of feature data and one or more batches of target value data. In some embodiments, step 1008 is performed to provide the training dataset in real-time or near real-time to avoid loading all the datasets into memory before the training process begins. However, alternatively, step 1008 may be performed before the training process begins.

[0149] In some embodiments, each batch of feature data may have a size of Q×T×B, where Q is a positive integer. For example, each batch of feature data includes Q feature sequences, each of size T×B. Each batch of target value data may have a size of Q×T×1. For example, each batch of target value data may include Q target value sequences, each of size T×1 and corresponding to one feature sequence in the batch of feature data. In some embodiments, each target value sequence includes a list of target value estimates for a feature list included in the corresponding feature sequence. For example, each target value sequence may include T target value estimates corresponding to T features included in the feature sequence. References below... Figure 11-12 The generation of the training dataset and the target value estimate is described in more detail.

[0150] In step 1010, training module 110 applies a neural network classifier to the training dataset to detect speech data therein. For example, training module 110 inputs each batch of feature data into the neural network classifier to generate a corresponding batch of value data. This batch of value data can be Q×T×1 in size and can include Q value sequences, each of size T×1 corresponding to its respective feature sequence in the batch of feature data. Training module 110 can compare the Q value sequences output by the neural network classifier with the Q target value sequences contained in the training dataset to generate a training result. For example, the training result can be a value representing the difference between the Q value sequences output by the neural network classifier and the Q target value sequences contained in the training dataset. The smaller the difference, the closer the weights are to their optimal values.

[0151] In step 1012, training module 110 updates the weights of the neural network classifier. For example, training module 110 may update the weights based on the training results. When the training results show large differences, the weights may change significantly, while when the training results show small differences, the weights may be slightly adjusted.

[0152] In step 1014, the training data generator 109 generates a validation dataset. The validation dataset may include one or more batches of feature data and one or more batches of target value data. The validation dataset may include data similar to the training dataset. Similar descriptions are not repeated here. The validation dataset can be used to validate the performance of the neural network classifier. To achieve this, the validation dataset should be independent of and distinct from the training dataset used to train the neural network classifier. In some embodiments, step 1014 is performed to provide the validation dataset in real-time or near real-time to avoid loading all datasets into memory before the training process. However, alternatively, step 1014 may be considered to be performed before the start of the training process.

[0153] In step 1016, training module 110 applies the neural network classifier to the validation dataset. In step 1018, training module 110 calculates the loss based on the output of the neural network classifier when applied to the validation dataset (“validation output”).

[0154] For example, in conjunction with steps 1016 and 1018, training module 110 inputs each batch of feature data from the validation dataset into the neural network classifier to generate a corresponding batch of value data. This batch of value data may include one or more value sequences, each corresponding to a respective feature sequence in the batch of feature data. Training module 110 can compare one or more value sequences output by the neural network classifier with one or more corresponding target value sequences contained in the validation dataset to evaluate a loss function, and can use this loss function to generate the loss of the neural network classifier. This loss estimate measures the difference between the value sequences output by the neural network classifier and the corresponding target value sequences contained in the validation dataset, thus indicating the performance of the neural network classifier with the weights in the current iteration.

[0155] In step 1020, training module 110 determines whether the loss of the current iteration is an improvement over the previous iteration. For example, training module 110 determines whether the loss is reduced relative to a previous loss calculated in a previous iteration. If the loss is improved (e.g., the loss is reduced relative to the previous loss), method 1000 proceeds to step 1022. Otherwise, method 1000 returns to step 1006.

[0156] In step 1022, training module 110 updates the neural network classifier. For example, training module 110 may adjust the architecture of the neural network classifier (e.g., the number of hidden units in the neural network) based on its performance on the validation dataset. The structure, weights, and other parameters of the neural network classifier may be stored in memory 104 or training database 107. Method 1000 may return to step 1006 to determine whether a predetermined number of training epochs has been reached.

[0157] In step 1024, training module 110 provides a trained neural network classifier for speech detection. For example, if a predetermined number of training epochs have been reached, training module 110 can stop training the neural network classifier and provide the trained neural network classifier for application to speech detection.

[0158] Figure 11This is a flowchart of an exemplary method 1100 for generating a dataset according to an embodiment of the present invention. In some embodiments, the dataset may be a training dataset or a validation dataset. Method 1100 may be implemented by system 101, particularly training data generator 109, and may include steps 1102-1124 as described below. In some embodiments, method 1100 may be performed to achieve... Figure 10 Each of steps 1008 and 1014 herein. Some steps may be optional to perform the disclosure provided herein. Furthermore, some steps may be performed concurrently or in a different manner. Figure 11 Execute in the order shown.

[0159] In step 1102, the training data generator 109 determines the total number of batches in the dataset for one epoch and the batch size of each batch in the dataset. For example, the dataset to be generated may include P batches of data (e.g., P batches of feature data and P batches of target value data), where P is a positive integer.

[0160] In step 1106, the training data generator 109 determines whether a total number of batches has been generated for the dataset. If a total number of batches has been generated for the dataset, method 1100 proceeds to step 1107. Otherwise, method 1100 proceeds to step 1108.

[0161] In step 1107, the training data generator 109 outputs the dataset for that epoch.

[0162] In step 1108, the training data generator 109 determines whether a batch of data with the specified batch size has already been generated. If the batch of data has been generated, method 1100 proceeds to step 1109. Otherwise, method 1100 proceeds to step 1110. For example, the training data generator 109 determines whether a batch of feature data and a batch of target value data corresponding to the batch of feature data have been generated. If the batch of feature data and the batch of target value data have been generated, method 1100 proceeds to step 1109. Otherwise, method 1100 proceeds to step 1110.

[0163] In step 1109, the training data generator 109 includes this batch of data in the dataset. For example, the training data generator 109 includes the batch of feature data and the batch of target value data in the dataset.

[0164] In step 1110, to generate the batch of data, the training data generator 109 retrieves audio signals from the training database 107. The training data generator 109 can retrieve audio signals from an audio database configured to store various audio signals. In some embodiments, the training data generator 109 can do so by performing the following reference... Figure 12 The described operation generates audio signals in real time or near real time.

[0165] In step 1112, the training data generator 109 normalizes the amplitude of the audio signal.

[0166] In step 1114, the training data generator 109 segments the audio signal into a series of frames within a series of time windows. For example, the training data generator 109 performs operations similar to those described above with reference to the speech activity detector 206 to segment the audio signal into a series of frames within a series of time windows. Similar descriptions will not be repeated here.

[0167] In step 1116, the training data generator 109 processes each frame in the series of frames using a high-pass filter. For example, the training data generator 109 processes each frame using a high-pass filter to remove DC offset and low-frequency noise.

[0168] In step 1118, the training data generator 109 calculates the Mel spectrum for each frame. For example, the training data generator 109 performs operations similar to those described above with reference to the speech activity detector 206 to calculate the Mel spectrum for each frame. Similar descriptions will not be repeated here.

[0169] In step 1120, for each frame, the training data generator 109 generates a feature sequence for the corresponding frame. For example, the feature sequence for the corresponding frame may include features of the corresponding frame and T-1 features from T-1 previous frames. In some embodiments, the training data generator 109 performs operations similar to those described above with reference to the speech activity detector 206 to generate the feature sequence for each frame. Similar descriptions will not be repeated here.

[0170] In step 1122, for each frame, the training data generator 109 obtains the target value estimate associated with the corresponding frame from the training database 107, thereby generating a target value sequence for the feature sequence. For example, the training data generator 109 can read the target value estimate of the corresponding frame and T-1 target value estimates of T-1 previous frames from the training database 107, and can concatenate the target value estimate of the corresponding frame and the T-1 target value estimates of T-1 previous frames to form a target value sequence.

[0171] In step 1124, for each frame, the training data generator 109 stacks the feature sequence and the target value sequence in the batch of data. For example, the training data generator 109 stacks the feature sequence in the batch of feature data and the target value sequence in the batch of target value data. Method 1100 may return to step 1108 to determine whether all data in the batch of data with the batch size has been generated. If all data in the batch of data has been generated, method 1100 proceeds to step 1109. Otherwise, method 1100 may repeat steps 1110-1124 to continue generating feature sequences and target value sequences for the batch of data.

[0172] Figure 12 This is a flowchart of an exemplary method 1200 for generating audio signals for a training process according to an embodiment of the present invention. Method 1200 can be implemented by system 101, particularly training data generator 109, and may include steps 1202-1226 as described below. In some embodiments, method 1200 can be performed to provide audio signals to... Figure 11 Step 1110 in the document. Some steps may be optional to perform the disclosure provided herein. Furthermore, some steps may be performed concurrently or in conjunction with... Figure 12 The different execution orders shown.

[0173] In step 1202, the training data generator 109 retrieves clean speech signals from the speech database. Consistent with this disclosure, a "clean" speech signal is an audio signal that primarily contains speech data and is substantially free of noise. The speech database can be configured to store various clean speech signals recorded by different audio detection devices. The speech database can be included in system 101.

[0174] In step 1204, the training data generator 109 normalizes the clean speech signal. For example, the training data generator 109 normalizes the amplitude of the clean speech signal.

[0175] In step 1206, the training data generator 109 randomly reads impulse responses from the impulse response database. In some embodiments, the impulse response database may be configured to store various impulse responses under different acoustic conditions. Impulse responses can be randomly selected from the various impulse responses. The impulse response database may be included in system 101.

[0176] In step 1208, the training data generator 109 applies a random gain to the impulse response. For example, the random gain could be a randomly selected gain to extend the acoustic conditions of the impulse response.

[0177] In step 1210, the training data generator 109 performs convolution on the clean speech signal and the impulse response to generate a clean audio signal. For example, the clean speech signal is convolved with the impulse response to generate a clean audio signal.

[0178] In step 1212, the training data generator 109 randomly reads noise signals from a noise database. In some embodiments, the noise database may be configured to store various noise signals recorded or simulated by a computer in different noise environments. Noise signals may be randomly selected as one of various noise signals. The noise database may be included in system 101.

[0179] In step 1214, the training data generator 109 normalizes the noise signal. For example, the amplitude of the noise signal is normalized.

[0180] In step 1216, the training data generator 109 applies a gain to the noise signal based on a signal-to-noise ratio (SNR) value. For example, the training data generator 109 randomly selects an SNR value. The training data generator 109 calculates the signal power level of the clean audio signal and determines the noise power level of the noise signal as: (noise power level) = (signal power level) ÷ (SNR). The gain applied to the noise signal can be calculated as: (gain) = (noise power level) ÷ (power level of the normalized noise signal). As a result, after applying the gain, the power level of the noise signal reaches the target noise power level.

[0181] In step 1218, the training data generator 109 adds noise to the clean audio signal to generate a noisy audio signal. For example, the audio signal may be a mixture of a clean speech signal, an impulse response, and noise.

[0182] In step 1220, the training data generator 109 normalizes the audio signal. For example, the amplitude of the audio signal is normalized.

[0183] In step 1222, the training data generator 109 stores the audio signal in a training database. In some embodiments, the training data generator 109 may store the audio signal in an audio database. The audio database may be included in system 101.

[0184] In step 1224, the training data generator 109 processes the clean speech signal from step 1204 to determine a corresponding target value estimate for each time window in the clean speech signal. For example, since the clean speech signal is essentially free of noise, a simple energy detector can be applied to detect speech data in the speech signal. Specifically, for each time window, if a speech segment is present in the speech signal, a target value estimate with a value of "1" can be generated for that time window. Otherwise, a target value estimate with a value of "0" can be generated for the time window, indicating that no speech data is present in that time window. In some embodiments, the training data generator 109 may use a neural network classifier or any other available classifier to process the clean speech signal to determine a corresponding target value estimate for each time window in the clean speech signal.

[0185] In step 1226, the training data generator 109 stores the corresponding target value estimate for each time window in the training database 107. In some embodiments, the training data generator 109 may pair the target value estimates with corresponding audio signals and store them in an audio database.

[0186] Figures 13A-13E This is a graphical representation illustrating the exemplary performance of a speech detection technique with a causal TCN described herein, according to an embodiment of the present invention. The test dataset used to evaluate the performance of the speech detection technique consists of 4041 audio signals, each lasting 6 seconds (6.7 hours in total). These audio signals were generated by mixing different types of speech signals from different speakers with different types of noise and reverberation. The speaker and noise conditions in the audio signals differ from those used to train the TCN. Target value estimates are labeled for the audio signals. The audio signals are processed frame-by-frame to simulate real-time processing for speech detection.

[0187] Table 2 below shows the overall classification metrics (at the frame level) for four different methods. These four methods include: (1) a speech detector developed for the WebRTC project (“WebRTC”); (2) a CNN-based real-time speech detection method for smartphones (“RT-CNN”); (3) a robust speech detection method combining DAE and CNN (“DAE-CNN”); and (4) a speech detection technique with causal TCN (“TCN”) described in this disclosure. The first three methods are used for comparative purposes only. Table 2 shows that the causal TCN described herein performs better than the other three methods in detecting speech signals.

[0188] Table 2

[0189] WebRTC 71.6％ 67.2％ 75.5％ RT-CNN 82.6％ 87.9％ 79.7％ DAE-CNN 88.6％ 86.7％ 91.3％ TCN 95.7％ 95.2％ 97.9％

[0190] exist Figures 13A-13EIn this study, the performance of causal TCN was also compared with that of WebRTC. (Reference) Figure 13A The figure shows an example of normalized audio signal 1302. The WebRTC classification results are represented by curve 1304. In curve 1304, a duration with a value of "1" indicates a speech segment (e.g., speech data is present), while a duration with a value of "0" indicates a non-speech segment (e.g., no speech data). The WebRTC classification results indicate that there are multiple speech segments in audio signal 1302. These WebRTC classification results are not entirely accurate because WebRTC misclassifies some non-speech segments as speech segments. On the other hand, the TCN classification results are represented by curve 1306, which is a straight line with a value of "0," indicating that there is no speech data in audio signal 1302. These classification results are correct because audio signal 1302 is primarily composed of noise.

[0191] Figure 13B Another example is shown, illustrating the normalized audio signal 1302, the WebRTC classification result (curve 1304), and the TCN classification result (curve 1306). It can be seen that curve 1306 correctly identifies the duration of the value "1" containing speech data. Curve 1304 indicates that WebRTC misclassified some non-speech segments as speech segments. Similar observations can be made from... Figure 13C-13E The details of what was obtained are omitted here. Figures 13A-13E This demonstrates that the causal TCN described in this paper is superior to the WebRTC method.

[0192] The speech detection techniques described in this paper can apply neural network classifiers to determine, in real-time or near real-time, whether a multi-channel audio input includes speech data within one or more time windows. Speech detection techniques can be helpful for various speech processing systems configured to detect human speech. Speech detection techniques can be applied to human-computer interaction (HMI), such as human-robot interaction (HRI), to detect a user's intent to communicate with a machine (e.g., a robot). For example, speech detection techniques can be used for speaker detection and localization (e.g., detecting the presence or absence of a speaker, and, if present, locating the speaker's position).

[0193] In the case of HRI, speech detection technology can be used to detect the start of a voice interaction using natural language, thereby determining the speaker's location and allowing the robot to turn towards the speaker. The automatic speech recognition engine can then be activated to listen to the speaker. This helps improve the efficiency of the automatic speech recognition engine. For example, if speech data is detected in the audio signal, the automatic speech recognition engine can be turned on and transcription of the audio signal can begin; if no speech data is detected in the audio signal, the automatic speech recognition engine can be turned off.

[0194] Understandably, speaker localization in HRI can make voice interaction between the speaker and the robot more natural and similar to human voice interaction. Specifically, when people communicate by voice, they typically face each other to obtain visual information in order to better understand the information being conveyed. Visual information (e.g., lip movements, facial expressions, etc.) helps in understanding the information. This human communication mechanism can be simulated in robots using cameras, microphones, and artificial intelligence algorithms. For example, one or more cameras can be placed on the robot's face, while a microphone array can be placed on the robot's head. By applying the speech detection techniques described in this paper, the presence of speech data can be detected using multi-channel audio input from the microphone array if the speaker begins to speak. The speaker can be localized, and the robot can then turn to that position so that one or more cameras are aimed at the speaker. For example, speech source localization techniques can be used to locate the speaker from multi-channel audio input obtained from the microphone array. With speech detection and speaker localization, the robot can only be triggered to turn towards the speaker when actual speech is detected, rather than responding to any received sound event, even if that sound event does not contain any speech.

[0195] Another aspect of this disclosure relates to a non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform the methods described above. The computer-readable medium may include volatile or non-volatile, magnetic, semiconductor-based, magnetic tape-based, optical, removable, non-removable, or other types of computer-readable media or computer-readable storage devices. For example, as disclosed, a computer-readable medium may be a storage device or memory module on which computer instructions are stored. In some embodiments, the computer-readable medium may be a disk or flash drive on which computer instructions are stored.

[0196] It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed systems and related methods. Other embodiments will be readily apparent to those skilled in the art, given the detailed description and practice of the disclosed systems and related methods.

[0197] This specification and examples are intended to be illustrative only, and the true scope is indicated by the claims and their equivalents.

[0198] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A speech detection method, comprising: Receive a multi-channel audio input consisting of a set of audio signals from a set of audio channels in the audio detection array; The multi-channel audio input is processed using a neural network classifier to generate a series of classification results within a series of time windows of the multi-channel audio input. The neural network classifier includes a causal temporal convolutional network configured to determine the classification result for each time window based on a portion of the multi-channel audio input within the corresponding time window and one or more time windows preceding the corresponding time window. There is no information leakage from the future to the past in the causal temporal convolutional network. The classification result is either a speech classification result or a non-speech classification result. as well as Based on this series of classification results, it is determined whether the multi-channel audio input includes one or more speech segments within this series of time windows.

2. The method according to claim 1, wherein, This causal temporal convolutional network contains one or more serially connected residual blocks, each containing an expanded one-dimensional (1D) convolution.

3. The method according to claim 2, wherein, The causal temporal convolutional network also includes a dense layer connected to the one or more serially connected residual blocks, the dense layer being configured to adjust the size of the output of the causal temporal convolutional network.

4. The method according to claim 1, wherein, The multi-channel audio input is processed using this neural network classifier to generate a series of classification results within a range of time windows of the multi-channel audio input, including: The audio signals are preprocessed to output a combined audio signal; The combined audio signal is divided into a series of frames within this series of time windows; and For each time window: Obtain the frame corresponding to the corresponding time window from this series of frames; The frame is applied to the neural network classifier to generate an estimate for the corresponding time window; and The classification result for the corresponding time window is determined based on this estimation result.

5. The method according to claim 4, wherein, Applying this frame to the neural network classifier to generate the estimation results for the corresponding time window includes: Generate a feature sequence for this frame, which includes the features of this frame and T-1 features from the previous T-1 frames, where T is a positive integer; The feature sequence is input into the neural network classifier to generate a value sequence, wherein the value sequence includes a value estimate for the current frame and T-1 value estimates for the T-1 previous frames; Update the value cache based on this value sequence; and The cached value is used to generate the estimated result for the corresponding time window.

6. The method according to claim 5, wherein, The feature sequence that generated this frame includes: The frame is processed using a high-pass filter; Calculate the Mel spectrum of this frame; and The Mel spectrum of this frame is concatenated with the T-1 Mel spectra of the previous T-1 frames to generate the feature sequence.

7. The method according to claim 1, wherein, The neural network classifier is used to process the multi-channel audio input to generate a series of classification results within a series of time windows of the multi-channel audio input, including: Each audio signal is processed using the neural network classifier to perform speech estimation based on the corresponding audio signal, thereby generating a set of speech estimates for the set of audio signals, wherein the set of speech estimates includes the corresponding set of estimates for each time window of the set of audio signals; The set of speech estimates from the set of audio signals are fused to generate the set of classification results within the set of time windows of the multi-channel audio input.

8. The method according to claim 1, wherein, Based on this series of classification results, it is determined whether the multi-channel audio input includes one or more speech segments within this series of time windows, including: For each time window, based on the classification results of the corresponding time window, determine whether there are speech segments in the multi-channel audio input during the corresponding time window.

9. A speech detection system, comprising: The memory is configured to store a multi-channel audio input acquired by an audio detection array, wherein the multi-channel audio input includes a set of audio signals from a set of audio channels in the audio detection array; and The processor is configured as follows: The multi-channel audio input is processed using a neural network classifier to generate a series of classification results within a series of time windows of the multi-channel audio input. The neural network classifier includes a causal temporal convolutional network configured to determine the classification result for each time window based on a portion of the multi-channel audio input within the corresponding time window and one or more time windows preceding the corresponding time window. There is no information leakage from the future to the past in the causal temporal convolutional network. The classification result is either a speech classification result or a non-speech classification result. as well as Based on this series of classification results, it is determined whether the multi-channel audio input includes one or more speech segments within this series of time windows.

10. A non-transitory computer-readable storage medium configured to store instructions that, in response to execution by a processor, cause the processor to perform a speech detection method, the method comprising: Receive a multi-channel audio input consisting of a set of audio signals from a set of audio channels in the audio detection array; The multi-channel audio input is processed using a neural network classifier to generate a series of classification results within a series of time windows of the multi-channel audio input. The neural network classifier includes a causal temporal convolutional network configured to determine the classification result for each time window based on a portion of the multi-channel audio input within the corresponding time window and one or more time windows preceding the corresponding time window. There is no information leakage from the future to the past in the causal temporal convolutional network. The classification result is either a speech classification result or a non-speech classification result. as well as Based on this series of classification results, it is determined whether the multi-channel audio input includes one or more speech segments within this series of time windows.

Images