Systems and methods for data enhancement and speech processing in dynamic acoustic environments.

By defining multiple acoustic variation models of speech signals in a dynamic acoustic environment, time-varying spectral changes are generated and applied to microphone arrays. This solves the problem that the spectral filtering effect of microphone arrays varies with the speaker and beamformer, thus improving the robustness and effectiveness of the speech processing system.

CN116868265BActive Publication Date: 2026-11-17MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202280015690.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-02-18
Filing Date
2022-02-17
Publication Date
2026-11-17
Estimated Expiration
2042-02-17

AI Technical Summary

Technical Problem

In dynamic acoustic environments, the spectral filtering effect of microphone arrays on speech signals varies with changes in speaker position and beamformer, leading to a decrease in the robustness of speech processing systems.

Method used

By defining models representing multiple acoustic variations of the speech signal, including changes in the relative positions of the speaker and microphone, adaptive beamforming, etc., multiple time-varying spectral changes are generated and applied to the feature coefficients of the reference signal. A machine learning model is then used for mapping and training to generate time-varying spectral enhancement feature coefficients.

Benefits of technology

It improves the robustness of the speech processing system in dynamic acoustic environments, enhances its robustness to noise and reverberation, and adapts to changes in speaker position and beamforming.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116868265B_ABST
    Figure CN116868265B_ABST
Patent Text Reader

Abstract

A method, computer program product, and computing system are provided for defining a model representing multiple acoustic variations of a speech signal, thereby defining multiple time-varying spectral modifications. These multiple time-varying spectral modifications can be applied to multiple characteristic coefficients in a target domain of a reference signal, thereby generating multiple time-varying spectral enhancement characteristic coefficients of the reference signal.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. non-provisional application serial number No. 17 / 178,734, filed February 18, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to data augmentation and speech processing systems and methods, and more specifically to systems and methods for data augmentation and speech processing in dynamic acoustic environments. Background Technology

[0004] Data augmentation allows for the generation of new training data for any machine learning system by enhancing existing data to represent new conditions. For example, data augmentation has been used to improve robustness to noise and reverberation, as well as other unpredictable properties of speech. In many cases, microphone arrays can be used to receive speech signals. However, microphone arrays can have the effect of filtering the speech spectrum in an angle-dependent manner. For example, if the speaker's position changes over time within the beam pattern of a beamformer, the speech may be affected by a time-varying filter. In the case of adaptive beamforming, the beamformer may attempt to dynamically turn toward the speaker, thus introducing another time variation. Summary of the Invention

[0005] In one implementation, a computer-implemented method is executed on a computing device and includes defining a model representing multiple acoustic variations of a speech signal, thereby defining multiple time-varying spectral modifications. These multiple time-varying spectral modifications can be applied to multiple characteristic coefficients in the target domain of a reference signal, thereby generating multiple time-varying spectral enhancement characteristic coefficients of the reference signal.

[0006] This can include one or more of the following features. Defining a model representing multiple acoustic variations of a speech signal can include one or more of the following: defining a model representing multiple acoustic variations of a speech signal associated with changes in the relative position of the speaker and microphone, and defining a model representing multiple acoustic variations of a speech signal associated with adaptive beamforming. Defining a model representing multiple acoustic variations of a speech signal includes one or more of the following: modeling multiple acoustic variations of a speech signal as a statistical distribution; and modeling multiple acoustic variations of a speech signal as a mathematical model representing multiple acoustic variations of a speech signal associated with a specific use case scenario. Defining a model representing multiple acoustic variations of a speech signal can include receiving one or more inputs associated with one or more of speaker movement and speaker orientation. Defining a model representing multiple acoustic variations of a speech signal can include generating a mapping of multiple acoustic variations to one or more feature coefficients in a target domain via a machine learning model. Applying multiple time-varying spectral modifications to multiple feature coefficients in the target domain of a reference signal may include simultaneously generating a mapping of multiple acoustic modifications to one or more feature coefficients in the target domain via a machine learning model, and applying the multiple time-varying spectral modifications to the multiple feature coefficients of the reference signal via the machine learning model. A speech processing system can be trained using the multiple time-varying spectral enhancement feature coefficients of the reference signal and one or more inputs associated with one or more of speaker position and speaker orientation. The speech processing system can be trained using the multiple time-varying spectral enhancement feature coefficients of the reference signal, thereby defining a trained speech processing system. Speech processing can be performed via the trained speech processing system, wherein the trained speech processing system is performed on at least one computing device.

[0007] In another implementation, a computer program product resides on a computer-readable medium and stores a plurality of instructions thereon. When executed by a processor, these instructions cause the processor to perform operations including: defining a model representing a plurality of acoustic variations of a speech signal, thereby defining a plurality of time-varying spectral modifications. These multiple time-varying spectral modifications can be applied to multiple characteristic coefficients in the target domain of a reference signal, thereby generating multiple time-varying spectral enhancement characteristic coefficients of the reference signal.

[0008] This can include one or more of the following features. Defining a model representing multiple acoustic variations of a speech signal can include one or more of the following: defining a model representing multiple acoustic variations of a speech signal associated with changes in the relative positions of the speaker and the microphone, and defining a model representing multiple acoustic variations of a speech signal associated with adaptive beamforming. Defining a model representing multiple acoustic variations of a speech signal can include one or more of the following: modeling multiple acoustic variations of a speech signal as a statistical distribution; and modeling multiple acoustic variations of a speech signal as a mathematical model representing multiple acoustic variations of a speech signal associated with a specific use case scenario. Defining a model representing multiple acoustic variations of a speech signal can include receiving one or more inputs associated with one or more of speaker movement and speaker orientation. Defining a model representing multiple acoustic variations of a speech signal can include generating a mapping of multiple acoustic variations to one or more feature coefficients in a target domain via a machine learning model. Applying multiple time-varying spectral changes to multiple feature coefficients in the target domain of a reference signal can include simultaneously generating a mapping of multiple acoustic variations to one or more feature coefficients in the target domain via a machine learning model, and applying multiple time-varying spectral changes to multiple feature coefficients in the reference signal via a machine learning model. A speech processing system can be trained using multiple time-varying spectral enhancement feature coefficients of a reference signal and one or more inputs associated with one or more of speaker position and speaker orientation. The trained speech processing system is defined by being trained using multiple time-varying spectral enhancement feature coefficients of the reference signal. Speech processing can be performed via the trained speech processing system, wherein the trained speech processing system is executed on at least one computing device.

[0009] In another implementation, a computing system includes at least one processor and at least one memory architecture coupled to the at least one processor, wherein the at least one processor is configured to define a model representing a plurality of acoustic variations of a speech signal, thereby defining a plurality of time-varying spectral changes, and wherein the at least one processor is further configured to apply the plurality of time-varying spectral changes to a plurality of characteristic coefficients in a target domain of a reference signal, thereby generating a plurality of time-varying spectral enhancement characteristic coefficients of the reference signal.

[0010] This can include one or more of the following features. Defining a model representing multiple acoustic variations of a speech signal can include one or more of the following: defining a model representing multiple acoustic variations of a speech signal associated with changes in the relative positions of the speaker and the microphone, and defining a model representing multiple acoustic variations of a speech signal associated with adaptive beamforming. Defining a model representing multiple acoustic variations of a speech signal can include one or more of the following: modeling multiple acoustic variations of a speech signal as a statistical distribution; and modeling multiple acoustic variations of a speech signal as a mathematical model representing multiple acoustic variations of a speech signal associated with a specific use case scenario. Defining a model representing multiple acoustic variations of a speech signal can include receiving one or more inputs associated with one or more of speaker movement and speaker orientation. Defining a model representing multiple acoustic variations of a speech signal can include generating a mapping of multiple acoustic variations to one or more feature coefficients in a target domain via a machine learning model. Applying multiple time-varying spectral changes to multiple feature coefficients in the target domain of a reference signal can include simultaneously generating a mapping of multiple acoustic variations to one or more feature coefficients in the target domain via a machine learning model, and applying multiple time-varying spectral changes to multiple feature coefficients in the reference signal via a machine learning model. A speech processing system can be trained using multiple time-varying spectral enhancement feature coefficients of a reference signal and one or more inputs associated with one or more of speaker position and speaker orientation. The trained speech processing system is defined by being trained using multiple time-varying spectral enhancement feature coefficients of the reference signal. Speech processing can be performed via the trained speech processing system, wherein the trained speech processing system is executed on at least one computing device.

[0011] Details of one or more implementations are set forth in the accompanying drawings and the following description. Other features and advantages will become apparent from the specification, drawings, and claims. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the data augmentation process and the speech processing process coupled to a distributed computing network;

[0013] Figure 2 yes Figure 1 A flowchart illustrating the data enhancement and speech processing processes;

[0014] Figure 3 It is based on Figure 1 A schematic diagram of a microphone array deployed in the monitored space as one implementation of the data augmentation process is being performed while the speaker is speaking;

[0015] Figure 4 It is based on Figure 1A schematic diagram of a microphone array deployed in the monitored space as one implementation of the data augmentation process is performed while the speaker is moving and speaking;

[0016] Figure 5 It is based on Figure 1 A schematic diagram of a microphone array deployed in the monitored space as one implementation of the data augmentation process occurs while the speaker is speaking and when the microphone array beam is beam-directed toward the speaker.

[0017] Figure 6 yes Figure 1 A flowchart illustrating the implementation of a data augmentation process;

[0018] Figure 7 yes Figure 1 A flowchart illustrating an implementation of the speech processing procedure;

[0019] Figure 8 It is based on Figure 1 A schematic diagram of a microphone array deployed in the monitored space as one implementation of the speech processing procedure while the speaker is speaking;

[0020] Figure 9 It is implemented based on one or more examples of this disclosure. Figure 1 An example diagram of a client electronic device.

[0021] The same reference numerals in various figures indicate the same elements. Detailed Implementation

[0022] Now for reference Figure 1 The diagram illustrates data augmentation process 10 and voice processing process 11, which may reside on and be executed by computing device 12, which may be connected to a network (e.g., network 14) (e.g., the Internet or a local area network). Examples of computing device 12 (and / or one or more client electronic devices mentioned below) may include, but are not limited to, personal computers(s), laptop computers(s), mobile computing devices(s), server computers, a series of server computers, mainframes(s), or computing clouds(s). Computing device 12 may run an operating system, such as, but not limited to, operating systems. Or a custom operating system. (Microsoft and Windows are registered trademarks of Microsoft Corporation in the U.S., other countries, or both; Mac and OS X are registered trademarks of Apple Inc. in the U.S., other countries, or both; Red Hat is a registered trademark of Red Hat Inc. in the U.S., other countries, or both; Linux is a registered trademark of Linus Torvalds in the U.S., other countries, or both).

[0023] The data augmentation process (such as...) will be discussed in more detail below. Figure 1 The data augmentation process 10) can define a model representing multiple acoustic variations of the speech signal, thereby defining multiple time-varying spectral changes. These multiple time-varying spectral changes can be applied to a reference signal using filtering operations, thereby generating a time-varying spectral augmented signal. Furthermore, speech processing procedures (such as...) Figure 1 The speech processing procedure 11) can receive one or more inputs indicating at least one of the relative positions of the speaker and the microphone array and the relative orientation of the speaker and the microphone array. It can receive one or more reference signals. The speech processing system can be trained using one or more inputs and reference signals.

[0024] The instruction set and subroutines of the data enhancement process 10 and / or the voice processing process 11 (which may be stored on a storage device 16 coupled to the computing device 12) may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within the computing device 12. The storage device 16 may include, but is not limited to: hard disk drives; flash drives; tape drives; optical drives; RAID arrays; random access memory (RAM); and read-only memory (ROM).

[0025] Network 14 may be connected to one or more secondary networks (e.g., network 18), examples of which may include, but are not limited to: local area network; wide area network; or intranet.

[0026] The data augmentation process 10 and / or the voice processing process 11 can be standalone applications that interface with smaller applications / applications accessed via client applications 22, 24, 26, 28, 66. In some embodiments, the data augmentation process 10 can be distributed, in whole or in part, in a cloud computing topology. In this way, computing device 12 and storage device 16 can refer to multiple devices that can also be distributed throughout network 14 and / or network 18.

[0027] Computing device 12 can execute speech recognition applications (e.g., speech recognition application 20), examples of which may include, but are not limited to, automatic speech recognition (ASR) programs and applications, speech-to-text (SST) programs and applications, computer speech recognition programs and applications, speech recognition programs and applications, in-vehicle voice command programs and applications, including programs and applications provided by Nuance Communications, Inc., Burlington, Massachusetts. Data augmentation process 10 and / or speech recognition application 20 can be accessed via client applications 22, 24, 26, 28, and 68. Data augmentation process 10 and / or speech processing process 11 can be standalone applications or can be small applications / applications / scripts / extensions that can interact with and / or execute within speech recognition application 20, components of speech recognition application 20, and / or one or more of client applications 22, 24, 26, 28, and 68. Speech recognition application 20 may be a standalone application or a mini-application / script / extension that can interact with and / or execute within the data augmentation process 10 and / or speech processing process 11, components of the data augmentation process 10, and / or client applications 22, 24, 26, 28, 68. One or more of client applications 22, 24, 26, 28, 68 may be standalone applications or mini-applications / scripts / extensions that can interact with and / or execute within the data augmentation process 10, speech processing process 11, and / or components of speech recognition application 20. Examples of client applications 22, 24, 26, 28, 68 may include, but are not limited to, applications that receive queries to search for content from one or more databases, servers, cloud storage servers, etc., text and / or graphical user interfaces, custom web browsers, plugins, application programming interfaces (APIs), or custom applications. The instruction sets and subroutines of client applications 22, 24, 26, 28, 68 (which may be stored on storage devices 30, 32, 34, 36 coupled to client electronic devices 38, 40, 42, 44) may be executed by one or more processors (not shown) and one or more memory architectures (not shown) incorporated in client electronic devices 38, 40, 42, 44.

[0028] Storage devices 30, 32, 34, and 36 may include, but are not limited to: hard disk drives; flash drives; tape drives; optical disk drives; RAID arrays; random access memory (RAM); and read-only memory (ROM). Examples of client electronic devices 38, 40, 42, and 44 (and / or computing device 12) may include, but are not limited to: personal computers (e.g., client electronic device 18), laptop computers (e.g., client electronic device 40), smart / data-enabled cellular phones (e.g., client electronic device 42), laptop computers (e.g., client electronic device 44), tablet computers (not shown), servers (not shown), televisions (not shown), smart televisions (not shown), media (e.g., video, photos, etc.) capture devices (not shown), and dedicated network devices (not shown). Each of client electronic devices 38, 40, 42, and 44 may execute an operating system, examples of which may include, but are not limited to: Mobile, Chrome OS, Blackberry OS, Fire OS, or a custom operating system.

[0029] One or more of client applications 22, 24, 26, 28, and 68 can be configured to implement some or all of the functions of data augmentation process 10 and / or voice processing process 11 (and vice versa). Therefore, data augmentation process 10 and / or voice processing process 11 can be a purely server-side application, a purely client-side application, or a hybrid server-side / client-side application executed collaboratively by one or more of client applications 22, 24, 26, 28, and 68 and / or data augmentation process 10 and / or voice processing process 11.

[0030] One or more of client applications 22, 24, 26, 28, and 68 can be configured to implement some or all of the functions of speech recognition application 20 (and vice versa). Therefore, speech recognition application 20 can be a purely server-side application, a purely client-side application, or a hybrid server-side / client-side application executed collaboratively by one or more of client applications 22, 24, 26, 28, and 68 and / or speech recognition application 20. Since some or all of the above-described functions can be implemented individually or in any combination by one or more of the client applications 22, 24, 26, 28, 68, the data augmentation process 10, the speech processing process 11, and the speech recognition application 20, any description of implementing such functions via one or more of the client applications 22, 24, 26, 28, 68, the data augmentation process 10, the speech processing process 11, the speech recognition application 20, or a combination thereof, and any description of any of the aforementioned interactions between one or more of the client applications 22, 24, 26, 28, 68, the data augmentation process 10, the speech processing process 11, the speech recognition application 20, or a combination thereof for implementing such functions, should be taken as examples only and not as limiting the scope of this disclosure.

[0031] Users 46, 48, 50, and 52 can access computing device 12 and data enhancement process 10 directly or indirectly via network 14 or via secondary network 18 (e.g., using one or more of client electronic devices 38, 40, 42, and 44). Furthermore, computing device 12 can be connected to network 14 via secondary network 18, as shown by dashed link 54. Data enhancement process 10 may include one or more user interfaces, such as browsers and text or graphical user interfaces, through which users 46, 48, 50, and 52 can access data enhancement process and / or voice processing process 11.

[0032] Various client electronic devices can be directly or indirectly coupled to network 14 (or network 18). For example, client electronic device 38 is shown as being directly coupled to network 14 via a hardwired network connection. Furthermore, client electronic device 44 is shown as being directly coupled to network 18 via a hardwired network connection. Client electronic device 40 is shown as being wirelessly coupled to network 14 via a wireless communication channel 56 established between client electronic device 40 and a wireless access point (i.e., WAP) 58, which is shown as being directly coupled to network 14. For example, WAP 58 may be IEEE 800.11a, 800.11b, 800.11g, etc. and / or Bluetooth* TM (including Bluetooth*) TMA low-power device is provided, capable of establishing a wireless communication channel 56 between client electronic device 40 and WAP 58. Client electronic device 42 is shown wirelessly coupled to network 14 via a wireless communication channel 60 established between client electronic device 42 and cellular network / bridge 62, which is shown directly coupled to network 14. In some implementations, audio recording device 64 may be wirelessly coupled to network 14 via a wireless communication channel 66 established between client electronic device 42 and cellular network / bridge 62, which is shown directly coupled to network 14. Storage device 70 may be coupled to audio recording system 64 and may include, but is not limited to: hard disk drive; flash drive; magnetic tape drive; optical disk drive; RAID array; random access memory (RAM); and read-only memory (ROM). User 72 may access computing device 12 and data enhancement process 10 (e.g., using one or more of audio recording systems 64) directly or indirectly through network 14 or through secondary network 18.

[0033] Some or all of the IEEE 800.1 lx specifications can use Ethernet protocols and Carrier Sense Multiple Access with Collision Avoidance (i.e., CSMA / CA) for path sharing. For example, various 800.1 lx specifications can use Phase Shift Keying (PSK) modulation or Complementary Code Keying (CCK) modulation. Bluetooth* TM (including Bluetooth*) TM Low Energy (LEE) is a telecommunications industry standard that allows devices such as mobile phones, computers, smartphones, and other electronic devices to be interconnected using short-range wireless connections. Other forms of interconnection can also be used (e.g., Near Field Communication (NFC)).

[0034] In some implementations consistent with this disclosure, systems and methods for data augmentation in dynamic acoustic environments can be provided. As described above and in some implementations, data augmentation can generally allow the generation of new training data for any machine learning system by enhancing existing data to represent new conditions and constraints. For example, data augmentation has been used to improve robustness to noise and reverberation, as well as other unpredictable properties of speech signals. In many cases, microphone arrays can be used to receive speech signals. However, microphone arrays can have the effect of filtering the speech spectrum in an angle-dependent manner. For example, if the speaker's position changes over time in the beam pattern of a beamformer, the speech may be affected by a time-varying filter. In the case of adaptive beamforming, the beamformer may attempt to dynamically turn toward the speaker, thus introducing another time variation. In some implementations, and as will be discussed in more detail below, the implementation of data augmentation process 10 can specifically augment training data to account for time variations and can be designed to model the physical variations expected when using a particular beamformer in a particular use case.

[0035] As mentioned above, and at least refer to Figures 2-9 The data enhancement process 10 can define a model 200 representing multiple acoustic variations of the speech signal, thereby defining multiple time-varying spectral changes. These multiple time-varying spectral changes 202 can be applied to a reference signal using filtering operations, thereby generating a time-varying spectral enhanced signal.

[0036] In some implementations, the data augmentation process 10 can define 200 models representing multiple acoustic variations of the speech signal, thereby defining multiple time-varying spectral changes. See also... Figure 3 Examples of microphone arrays (e.g., microphone array 300) include, in some implementations, multiple discrete microphone elements (e.g., microphone elements 302, 304, 306). While example microphone array 300 includes, for example, three microphone elements, it should be understood that within the scope of this disclosure, microphone array 300 may include any number of microphone elements. Examples of microphone array 300 may generally include microphone arrays for headsets, microphone arrays for computing devices, microphone arrays for camera systems, and / or combinations thereof. Therefore, it should be understood that within the scope of this disclosure, microphone arrays may include any configuration of discrete microphone elements.

[0037] In some implementations, a microphone array (e.g., microphone array 300) can be configured to receive various sound signals from the monitored space (i.e., the space adjacent to microphone array 300). For example... Figure 3As illustrated in the example, microphone array 300 can be configured to receive one or more speech signals (e.g., speech signal 308) from one or more speakers (e.g., speaker 310). In one example, it is assumed that the speaker (e.g., speaker 310) is stationary (i.e., speaker 310 does not move relative to microphone array 300 while speaking). In this example, each microphone element (e.g., microphone elements 302, 304, 306) can be configured to receive one or more speech signals (e.g., speech signal 308) from the speaker (e.g., speaker 310). In some implementations, the microphone array (e.g., microphone array 300) can allow multiple discrete microphone elements (e.g., microphone elements 302, 304, 306) to form a beamformer.

[0038] As is known in the art, beamforming is a signal processing technique used for sensor arrays to perform directional signal transmission or reception. In the example of audio signals, beamforming is achieved by combining multiple discrete microphone elements (e.g., microphone elements 302, 304, 306) in a microphone array (e.g., microphone array 300) such that signals at certain angles experience constructive interference while signals at other angles experience destructive interference. By using the combined signal from each microphone element, speech signals can be received with less noise and / or less reverberation. In this way, multiple microphone elements can generate a sensitivity beam that can be configured to pick up or receive speech signals more sensitively in a selected direction than in other areas. In some implementations, the spatial sensitivity for speech signals can be frequency-dependent. For example, the sensitivity beam can be narrower for receiving high-frequency signals and wider for receiving low-frequency signals. As will be discussed in more detail below, changes in the relative positions of a sound source (e.g., a speaker) and a microphone array (e.g., microphone array 300) can cause acoustic changes in the speech signal received by the microphone array. For example, movement of the sound source (e.g., a speaker) can cause acoustic changes in the speech signal received by the microphone array.

[0039] return Figure 3For example, microphone elements 302, 304, and 306 are configured as beamformers with higher sensitivity to receive speech signals (e.g., speech signal 308) at predefined locations within the monitored space. In this example, it is assumed that the beamformer created by microphone array 300 has a low-frequency sensitive beam (e.g., low-frequency beam pattern 312) and a high-frequency sensitive beam (e.g., high-frequency beam pattern 314). Although the terms "low" and "high" have been provided to describe beam patterns 312 and 314, it should be understood that any frequency can be applied within the scope of this disclosure, and the size of the beam pattern can decrease as the frequency increases from 0 Hz. Therefore, it will be understood that any number of frequencies can be represented by any number of different beam patterns. For example, consecutive frequencies can produce corresponding consecutive beam patterns. Figure 3 As shown in the example, when speaker 310 speaks at the center of the beamformer, low-frequency and high-frequency sound signals are within the sensitivity beam at each frequency.

[0040] However, assuming the speaker's position changes over time (e.g., the speaker moves within the monitored space or the speaker's head rotates in one direction), the speech signal (e.g., speech signal 308) may be affected by acoustic changes associated with the speaker's movement. In another example, assuming the microphone array position changes over time (e.g., the microphone array moves relative to the speaker within the monitored space), the speech signal (e.g., speech signal 308) may be affected by acoustic changes associated with the microphone array's movement.

[0041] In another example, it is assumed that a beamformer formed by a microphone array is configured to perform adaptive beamforming and beam steering by dynamically changing the beam pattern(s) and steering it toward the speaker. In this example, beam steering can introduce acoustic variations into the speech signal.

[0042] In some implementations, definition 200, representing a model of multiple acoustic variations of a speech signal, may include definition 204, representing a model of multiple acoustic variations of a speech signal associated with changes in the relative positions of the speaker and the microphone. See also... Figure 4 For example, and in some implementations, it is assumed that the speaker (e.g., speaker 310) moves within the monitored space while speaking. In this example, the movement of speaker 310 may cause the speech signal (e.g., speech signal 308) to be within the low-frequency beam pattern (e.g., low-frequency beam pattern 312) but outside the high-frequency beam pattern (e.g., high-frequency beam pattern 314). Thus, the movement of speaker 310 can introduce acoustic variations into the speech signal 308 received by microphone array 300.

[0043] In another example, assume that a microphone array (e.g., microphone array 300) is moving within the monitored space, while a speaker (e.g., speaker 310) is speaking (e.g., microphone array 300 is part of a robot or drone moving within the monitored space). In this example, the movement of microphone array 300 may result in the speech signal (e.g., speech signal 308) being within the low-frequency beam pattern but outside the high-frequency beam pattern. While examples of speaker movement and microphone array movement have been provided, it should be understood that data augmentation process 10 may define 204 a model representing multiple acoustic variations of the speech signal associated with any change in the relative positions of the speaker and the microphone.

[0044] In some implementations, definition 200, representing a model of multiple acoustic variations of a speech signal, may include definition 206, representing a model of multiple acoustic variations of a speech signal associated with adaptive beamforming. See also... Figure 5 For example, and in some implementations, it is assumed that microphone array 300 is configured for adaptive beamforming and beam steering. In this example, movement of the beam pattern (e.g., high-frequency beam pattern 314 and low-frequency beam pattern 312) can introduce acoustic variations into the speech signal 308. In some implementations, and as will be discussed in more detail below, data augmentation process 10 may define 206 a model representing multiple acoustic variations of the speech signal associated with adaptive beamforming.

[0045] In some implementations, and as described above, variations in the relative positioning of the speaker and microphone array, and / or adaptive beamforming, can introduce acoustic variations into the speech signal. In some implementations, these acoustic variations can be modeled as multiple time-varying spectral changes. For example, speech or audio signals can be represented or measured in the time domain, frequency domain, and / or modulation domain. In the time domain, the amplitude or power of the speech signal can be observed as a function of time. In the frequency domain, the amplitude or power of the speech signal can be observed as a function of the frequency of the speech signal. In the modulation domain, the power of the speech signal can be observed as the time variation of the frequency components of the speech signal.

[0046] In some implementations, the speech signal can be modeled in the modulation domain by extracting frequency components from the speech signal. In some implementations, the data augmentation process 10 can extract frequency components from the speech signal by applying a Short Time Fourier Transform (STFT) to the speech signal. While STFT is discussed as one way to extract frequency components from a speech signal, it should be understood that other transforms can be used to derive the modulation domain within the scope of this disclosure. Applying an STFT to the speech signal can include applying overlapping frames with an analysis window. For example, the speech signal (e.g., the received speech signal 308) can be recorded as a time waveform in the time domain by a microphone array (e.g., microphone array 300). The data augmentation process 10 can convert the time waveform of the speech signal into a sequence of short excerpts of the time waveform. The data augmentation process 10 can transform each short excerpt to the frequency domain by combining a Fourier transform with a window function, where such a window function is known in the art. Additionally, one or more bandpass filters can be applied, and the received speech signal can be converted into multiple speech signals of multiple frequency bands by the data augmentation process 10.

[0047] As is known in the art, the frequency band or range of a Fourier transform can be combined with a time window to form multiple time-spectral units. The power or amplitude of the speech signal can be defined relative to time and frequency within each time-spectral unit. As will be discussed in more detail below, multiple acoustic variations associated with time-varying changes or movements (e.g., changes in the relative positioning of the speaker and the microphone array and / or movements associated with adaptive beamforming and beam steering) can be realized when observed in the modulation domain.

[0048] In some implementations, the data augmentation process 10 can model 208 multiple acoustic variations of the speech signal as a statistical distribution. For example, the speech amplitude spectrum can vary according to a statistical distribution or model, such that, as will be discussed in more detail below, a scaling factor can be applied to each time-frequency data point, and the scaling factor is derived from a previously determined statistical distribution. In one example, the data augmentation process 10 can model 208 multiple acoustic variations of the speech signal (e.g., caused by changes in the relative positioning of the speaker and microphone array and / or adaptive beamforming) as a Gaussian distribution. In this example, and as will be discussed in more detail below, the data augmentation process 10 can represent multiple acoustic variations of the speech signal associated with changes in the relative positioning of the speaker and microphone array and / or adaptive beamforming by selecting scaling factors with different means and variances. While an example of a Gaussian distribution has been provided for the statistical distribution, it should be understood that, within the scope of this disclosure, the data augmentation process 10 can model 208 multiple acoustic variations of the speech signal as any statistical distribution. In some implementations, the data augmentation process 10 may allow the user to select (e.g., via a graphical user interface) a specific statistical distribution from multiple statistical distributions for modeling multiple acoustic variations.

[0049] In some implementations, defining 200 as a model representing multiple acoustic variations of a speech signal may include modeling 210 the multiple acoustic variations of the speech signal as a mathematical model representing multiple acoustic variations of the speech signal associated with a specific use case scenario. For example, the amplitude spectrum of the speech signal may vary according to a mathematical model matched to a specific use case scenario, such that a change in scaling factor can be applied to each time-frequency data point, where the change in scaling factor represents the actual acoustic variation observed in an example of the specific use case scenario. In some implementations, and as described above, the acoustic variation may depend on two elements: the beam pattern of the beamformer and the movement of the sound source within and outside the beamformer.

[0050] For example, if the use case scenario is a business meeting, the data augmentation process 10 can generate data augmentation by selecting time-varying factors that represent, for example, head movements during conversational following behavior. In this example, the data augmentation process 10 can receive speech signals via headphones worn by the speaker (e.g., speaker 310). Furthermore, a microphone array (e.g., microphone array 300) can be configured to receive speech signals (e.g., speech signal 308). As the speaker's head moves, the data augmentation process 10 can identify spectral changes between the speech signal recorded by the headphones and the speech signal recorded by the microphone array (e.g., microphone array 300). Therefore, the data augmentation process 10 can model 210 multiple acoustic variations associated with, for example, speaker head movements during conversational following behavior, into a mathematical model based at least in part on the identified spectral changes between the speech signal recorded by the headphones and the speech information recorded by the microphone array.

[0051] While an example of a business meeting with acoustic changes caused by head movement has been provided for a specific use case scenario, it should be understood that, within the scope of this disclosure, acoustic changes associated with any use case scenario can be modeled 210 as a mathematical model. In some implementations, the data augmentation process 10 may allow a user (e.g., via a graphical user interface) to select a specific use case scenario from multiple use case scenarios and / or to select a mathematical model from multiple mathematical models for modeling multiple acoustic changes.

[0052] In some implementations, the model defined by 200 representing multiple acoustic variations of the speech signal may include receiving 212 one or more inputs associated with one or more of the speaker's position and speaker orientation. For example, the data augmentation process 10 may receive 212 one or more inputs from various sensors (e.g., audiovisual sensors, camera systems, microphone array 300, etc.). In some implementations, one or more inputs may indicate the speaker's position and / or orientation (e.g., head orientation, body orientation, etc.) within the monitored space. For example, the data augmentation process 10 may receive 212 an input indicating the direction of arrival (DOA) angle estimated by the beamformer.

[0053] In another example, the data augmentation process 10 may receive 212 one or more inputs indicating the speaker's orientation (e.g., whether the speaker is facing the microphone array, whether the speaker has their back to the microphone array, etc.). For example, it is assumed that one or more inputs include a video of the speaker. In some implementations, one or more inputs may represent the user's orientation (e.g., head orientation, body orientation, etc.) as a numerical value (e.g., a two-digit number indicating whether the speaker's head is directly facing the microphone array, whether the speaker's head is facing 90 degrees to the left or right of the microphone array, etc.). While examples of the types of inputs associated with one or more of the speaker's position and speaker orientation have been provided, it should be understood that within the scope of this disclosure, the data augmentation process 10 may receive 212 various types of inputs. In some implementations, and as will be discussed in more detail below, one or more inputs indicating the speaker's orientation may be provided as input during the training of a speech processing system (e.g., an automatic speech recognition (ASR) system, a speech biometrics system, an emotion detection system, a medical symptom detection system, a hearing enhancement system).

[0054] Returning to the aforementioned use case scenario of a business meeting, the data augmentation process 10 may receive 212 one or more inputs associated with head movement during conversational following. In this example, a microphone array (e.g., microphone array 300) may be configured to receive speech signals (e.g., speech signal 308), and one or more inputs may indicate speaker head movement during the meeting. As the speaker's head moves, the data augmentation process 10 may identify spectral changes between the speech signal recorded by the headset and the speech signal recorded by the microphone array (e.g., microphone array 300). Therefore, the data augmentation process 10 may model 210 multiple acoustic variations associated with, for example, speaker head movement during conversational following as a mathematical model, based at least in part on the identified spectral changes between the speech signal recorded by the headset and the speech information recorded by the microphone array. Although an example using specific inputs associated with speaker head movement has been described above, it should be understood that 212 any input associated with any speaker movement and / or speaker orientation may be received and used to model 210 multiple acoustic variations associated with speaker movement and / or speaker orientation.

[0055] In some implementations, the model defined as 200 representing multiple acoustic variations of a speech signal may include mappings of 214 multiple acoustic variations to one or more feature coefficients in the target domain generated via a machine learning model. For example, the data augmentation process 10 may utilize a machine learning model (e.g., machine learning model 72) configured to learn to model one or more of the effects of motion on signal characteristics and adaptive beamforming on the output spectrum, for example in the short-time Fourier transform (STFT) domain or the Mel-frequency cepstral domain (MFC or Mel-frequency domain).

[0056] As is known in the art, machine learning models typically include algorithms that have been trained to recognize certain types of patterns (e.g., machine learning pattern 72). For example, based on the nature of the available signals, machine learning methods can generally be categorized into three types: supervised learning, unsupervised learning, and reinforcement learning. As is known in the art, supervised learning can include presenting a computing device with example inputs given by a “teacher” and their expected outputs, where the goal is to learn general rules that map the inputs to the outputs. In the case of unsupervised learning, no labels are provided to the learning algorithm, which must search for structures in the inputs on its own. Unsupervised learning can be an objective in itself (discovering hidden patterns in the data) or a means to that objective (feature learning). As is known in the art, reinforcement learning typically includes a computing device interacting in a dynamic environment where it must perform an objective (such as driving a vehicle or playing a game with an opponent). As it navigates the problem space, the program receives feedback similar to rewards and attempts to maximize those rewards. While three examples of machine learning methods have been provided, it should be understood that other machine learning methods are possible within the scope of this disclosure.

[0057] In some implementations, the machine learning model (e.g., machine learning module 72) can be configured to receive multiple acoustic variations as input and can be configured to output a mapping of the multiple acoustic variations to one or more feature coefficients in a target domain. For example, within the scope of this disclosure, the machine learning model (e.g., machine learning model 72) can be configured to generate a mapping of multiple acoustic variations to feature coefficients in the STFT domain, the Mel frequency domain, or any other domain.

[0058] Refer again Figure 4In one example, it is assumed that a machine learning model (e.g., machine learning model 72) receives multiple acoustic variations associated with the speaker's movement. In this example, the data augmentation process 10 can define a model 200 representing the multiple acoustic variations by generating 214 via the machine learning model (e.g., machine learning model 72) a mapping from the multiple acoustic variations associated with the speaker's (e.g., speaker 310) movement to one or more feature coefficients in a target domain (e.g., STFT domain, Mel frequency domain, etc.).

[0059] Refer again Figure 5 In another example, it is assumed that a machine learning model (e.g., machine learning model 72) receives multiple acoustic variations associated with adaptive beamforming. In this example, the data augmentation process 10 can define a model 200 representing the multiple acoustic variations by generating 214 via the machine learning model (e.g., machine learning model 72) a mapping of 214 to one or more characteristic coefficients of a target domain (e.g., STFT domain, Mel frequency domain, etc.).

[0060] In some implementations, generating a mapping of multiple acoustic variations associated with changes in the relative positioning of the speaker and microphone array and / or adaptive beamforming to one or more feature coefficients in the target domain can allow the data augmentation process 10 to generate multiple data augmentations associated with changes in the relative positioning of the speaker and microphone array and / or adaptive beamforming. For example, where a machine learning model maps multiple acoustic variations to one or more feature coefficients in the target domain, as will be discussed in more detail below, the data augmentation process 10 can apply the mapping of one or more feature coefficients 202 to the reference signal to generate data augmentations representing the acoustic variations. In other words, the data augmentation process 10 can directly apply one or more feature coefficients in the target domain to obtain augmented data, including degradations from the acoustic variations.

[0061] Suppose that data augmentation process 10 receives multiple acoustic variations associated with, for example, head movements of a speaker during a meeting. In this example, data augmentation process 10 can generate 214 mappings of the multiple acoustic variations associated with the speaker's head movements during the meeting to one or more feature coefficients in a target domain via a machine learning model (e.g., machine learning model 72). Therefore, data augmentation associated with the speaker's head movements during the meeting can be generated by changing the feature coefficients of these mappings in the target domain to represent other examples of the speaker's head movements during the meeting.

[0062] In another example, suppose data augmentation process 10 receives multiple acoustic variations associated with, for example, the beam steering of an adaptive beamformer. In this example, data augmentation process 10 may generate 214 mappings of the multiple acoustic variations associated with the beam steering of the adaptive beamformer to one or more feature coefficients in the target domain via a machine learning model (e.g., machine learning model 72). Thus, data augmentation associated with the beam steering of the adaptive beamformer can be generated by changing the feature coefficients of these mappings in the target domain to represent other examples of the beam steering of the adaptive beamformer. While two examples of sources of acoustic variations that can be mapped to feature coefficients in the target domain have been provided, it should be understood that, within the scope of this disclosure, multiple acoustic variations from any source can be mapped to feature coefficients.

[0063] In some implementations, data augmentation process 10 may apply multiple time-varying spectral changes 202 to a reference signal using filtering operations, thereby generating a time-varying spectral augmented signal. As described above, data augmentation process 10 may define 200 as a model representing multiple acoustic variations of a speech signal to define multiple time-varying spectral changes. In some implementations, data augmentation process 10 may generate data augmentation of these time-varying spectral changes, which represent how the speech signal is affected by, for example, variations in the relative positioning of the speaker and microphone array and / or adaptive beamforming. For example, data augmentation process 10 may apply multiple time-varying spectral changes 202 to a reference signal. In some implementations, the reference signal may typically include any signal to which time-varying spectral changes can be applied. For example, data augmentation process 10 may apply multiple time-varying spectral changes 202 to a speech signal (e.g., from a training data corpus). In this way, the reference signal can be augmented by data augmentation process 10 to include multiple time-varying spectral changes, thereby generating a time-varying spectral augmented signal.

[0064] In some implementations, applying multiple time-varying spectral changes 202 to a reference signal using filtering operations can include applying multiple time-varying spectral changes 216 to the reference signal using multiple time-varying parameters in time-domain filtering. For example, data augmentation process 10 can represent multiple time-varying spectral changes as multiple time-varying parameters in the time domain (e.g., multiple finite impulse response filters configured to represent or simulate time-varying spectral changes in the time domain). In this way, data augmentation process 10 can generate multiple finite impulse response filters to represent multiple time-varying spectral changes, and can apply multiple finite impulse response filters 216 to the reference signal to generate a time-varying spectral augmented signal.

[0065] In some implementations, applying multiple time-varying spectral changes 202 to a reference signal using filtering operations may include applying multiple time-varying spectral changes 218 to the reference signal using multiple time-varying multiplication factors in frequency domain filtering. For example, data augmentation process 10 may represent multiple time-varying spectral changes as multiple time-varying multiplication factors in the frequency domain. In some implementations, data augmentation process 10 may acquire a reference signal (e.g., from a data corpus) and use an STFT to transform the reference signal to the frequency domain. Data augmentation process 10 may apply 218 multiple time-varying spectral changes in the Fourier transform domain. For example, data augmentation process 10 may apply 218 multiple time-varying spectral changes 218 to the signal in the frequency domain using multiple time-varying multiplication factors. In some implementations, data augmentation process 10 may perform an inverse Fourier transform to generate a time-varying spectral augmented signal in the time domain with characteristics associated with multiple acoustic variations modeled by data augmentation process 10 (e.g., a model representing changes in the relative positioning of the speaker and microphone array and / or adaptive beamforming).

[0066] Although two examples of filtering operations for applying multiple time-varying spectral changes 202 to a reference signal have been provided, it should be understood that within the scope of this disclosure, any filtering operation can be used by the data augmentation process 10 to apply multiple time-varying spectral changes 202 to the reference signal.

[0067] In some implementations, by defining 200 to represent a model of multiple acoustic variations (e.g., variations associated with the relative positioning of the speaker and microphone array and / or adaptive beamforming) and applying multiple time-varying spectral changes 202 to a reference signal, data augmentation process 10 can generate training data from a reference signal with multiple acoustic variations to represent variations in the relative positioning of the speaker and microphone array and / or movements associated with adaptive beamforming and / or beam steering. In this way, and as will be discussed in more detail below, data augmentation process 10 can generate augmented training data (e.g., a time-varying spectral augmentation signal) that makes speech processing systems such as automatic speech recognition (ASR) systems (e.g., ASR system 74) less susceptible to time-varying spectral changes caused by variations in the relative positioning of the speaker and microphone array and / or adaptive beamforming.

[0068] In some implementations, the data augmentation process 10 may use a time-varying spectral augmentation signal to train the speech processing system 220, thereby defining the trained speech processing system. As described above, and in some implementations, the data augmentation process 10 may generate time-varying spectral changes associated with variations in the relative positioning of the speaker and microphone array and / or adaptive beamforming, which, when used in the training of the speech processing system (e.g., ASR system 74), allows the speech processing system to be more robust to such acoustic variations. In some implementations, and as will be discussed in more detail below, training the speech processing system 220 may typically include training one or more speech processing models (e.g., machine learning or neural network models) configured to process speech signals for various purposes. For example, the speech processing system may typically include an ASR system, a speech biometric system, an emotion detection system, a medical symptom detection system, a hearing enhancement system, etc. In one example, training the speech processing system 220 may include training the ASR system 220, which is configured to process speech signals to generate recognized speech. In this way, automatic speech recognition systems (e.g., ASR system 74) can be improved to recognize speech signals with acoustic variations caused by moving speakers and / or adaptive beamforming. While examples of ASR systems have been provided, it should be understood that any speech processing system can be trained within the scope of this disclosure.

[0069] In some implementations, the data augmentation process 10 may train the speech processing system 222 using a time-varying spectral augmentation signal and one or more inputs associated with one or more of speaker location and speaker orientation. For example, and as described above, the data augmentation process 10 may receive one or more inputs 212 associated with one or more of speaker location and speaker orientation. In some implementations, the data augmentation process 10 may provide one or more inputs associated with one or more of speaker location and speaker orientation for training the speech processing system 222 (e.g., ASR system 74). In this way, the speech processing system (e.g., ASR system 74) may be configured to consider a particular speaker location and / or speaker orientation at runtime, at least in part, based on one or more inputs received by the data augmentation process 10 212. For example, and as will be discussed in more detail below, in response to the detection of a particular speaker location and / or a particular speaker orientation at runtime (e.g., during operation of the speech processing system (e.g., ASR system 74)), the trained speech processing system (e.g., ASR system 74) may utilize different speech recognition models to consider the speaker location and / or speaker orientation.

[0070] In some implementations, 224 automatic speech recognition can be performed via a trained speech processing system executed on the same computing device used to train the speech processing system and / or on another computing device. Therefore, it will be appreciated that, within the scope of this disclosure, generating augmented data (e.g., time-varying spectral augmented signals), training the speech processing system with the augmented data, and performing speech processing via the trained speech processing system can be performed on the same computing device and / or discrete computing devices.

[0071] Also refer to Figure 6 For example, and in some implementations, the data augmentation process 10 can define 200 a model representing multiple acoustic variations of the speech signal, thereby defining multiple time-varying spectral changes. These multiple time-varying spectral changes can be applied 600 to multiple characteristic coefficients in the target domain of the reference signal, thereby generating multiple time-varying spectral enhancement characteristic coefficients of the reference signal.

[0072] As described above, and in some implementations, the data augmentation process 10 may define 200 a model representing multiple acoustic variations of a speech signal, thereby defining multiple time-varying spectral changes. For example, the data augmentation process 10 may model 208 the multiple acoustic variations of the speech signal as a statistical distribution. In another example, the data augmentation process 10 may model 210 the multiple acoustic variations of the speech signal as a mathematical model representing multiple acoustic variations of the speech signal associated with a specific use case scenario. In yet another example, the data augmentation process 10 may generate 214 mappings of multiple acoustic variations to one or more feature coefficients in a target domain via a machine learning model.

[0073] In some implementations, the data augmentation process 10 can apply multiple time-varying spectral changes 600 to multiple characteristic coefficients of the reference signal in the target domain, thereby generating multiple time-varying spectral augmentation characteristic coefficients of the reference signal. For example, multiple time-varying changes in the characteristic domain of a speech recognizer (e.g., ASR system 74) can be used to apply multiple acoustic variations modeled by, for example, a statistical distribution and / or mathematical model of a specific use case scenario, to the reference signal. In some implementations, the data augmentation process 10 can determine the characteristic coefficients of the reference signal in the target domain. In one example, it is assumed that the target domain of the speech recognizer (e.g., ASR system 74) is the Mel frequency domain. The data augmentation process 10 can determine the Mel frequency cepstral coefficients of the reference signal in the Mel frequency domain. In this example, the data augmentation process 10 can apply multiple time-varying spectral changes 600 to the Mel frequency cepstral coefficients of the reference signal. In this way, the data augmentation process 10 can generate multiple time-varying spectral augmentation characteristic coefficients of the reference signal that reflect multiple acoustic variations.

[0074] In some implementations, applying multiple time-varying spectral changes 600 to multiple characteristic coefficients of a target domain of a reference signal may include simultaneously generating 602 mappings of acoustic changes to one or more characteristic coefficients of the target domain via a machine learning model, and applying the multiple time-varying spectral changes to the multiple characteristic coefficients of the reference signal via the machine learning model. For example, and as described above, the machine learning model (e.g., machine learning model 72) may be configured to receive multiple acoustic changes as input and may be configured to output mappings of multiple acoustic changes to one or more characteristic coefficients of the target domain. In some implementations, the data augmentation process 10 may utilize the machine learning model (e.g., machine learning model 72) in an "end-to-end" configuration, in which the data augmentation process 10 simultaneously generates 602 mappings of acoustic changes to one or more characteristic coefficients of the target domain and applies these mappings to the characteristic coefficients of the reference signal. For example, the data augmentation process 10 may receive a reference signal and multiple acoustic changes as input to the machine learning model and output data augmentation of the reference signal including the multiple acoustic changes. In this way, the data augmentation process 10 can generate augmented data from the reference signal and one or more parameters associated with a specific acoustic change via a machine learning model (e.g., machine learning model 72).

[0075] In some implementations, the data augmentation process 10 may use multiple time-varying spectral augmentation feature coefficients of the reference signal to train the 604 speech processing system, thereby defining the trained speech processing system. As described above, and in some implementations, the data augmentation process 10 may generate time-varying spectral changes associated with variations in the relative positioning of the speaker and microphone array and / or adaptive beamforming, which, when used in the training of the speech processing system (e.g., ASR system 74), allows the automatic speech recognition system to be more robust to such acoustic variations. In some implementations, and as will be discussed in more detail below, training the 604 speech processing system may typically include training one or more speech processing models configured to process speech signals for various purposes. For example, the speech processing system may typically include an ASR system, a speech biometric system, an emotion detection system, a medical symptom detection system, a hearing enhancement system, etc. In one example, training the 604 speech processing system may include training a 604 ASR system configured to process speech signals to generate recognized speech. In this way, automatic speech recognition systems (e.g., ASR system 74) can be improved to recognize speech signals with acoustic variations caused by moving speakers and / or adaptive beamforming. While examples of ASR systems have been provided, it should be understood that any speech processing system can be trained within the scope of this disclosure.

[0076] In some implementations, the data augmentation process 10 may use time-varying spectral augmentation feature coefficients of a reference signal and one or more inputs associated with one or more of speaker location and speaker orientation to train the 606 speech processing system. For example, and as described above, the data augmentation process 10 may receive 212 one or more inputs associated with one or more of speaker location and speaker orientation. In some implementations, the data augmentation process 10 may provide one or more inputs associated with one or more of speaker location and speaker orientation for training the 606 speech processing system (e.g., ASR system 74). In this way, the ASR system 74 may be configured to consider a particular speaker location and / or speaker orientation at runtime, at least in part, based on one or more inputs received by the data augmentation process 10 212. For example, and as will be discussed in more detail below, in response to the detection of a particular speaker location and / or a particular speaker orientation at runtime (e.g., during operation of the speech processing system (e.g., ASR system 74)), the trained speech processing system (e.g., ASR system 74) may utilize different speech recognition models to consider speaker location and / or speaker orientation.

[0077] In some implementations, the data augmentation process 10 can be performed via a trained speech processing system, wherein the trained speech processing system is executed on at least one computing device. For example, embodiments of the data augmentation process 10 can be used in environmental speech applications with multiple remote microphones, such as those provided by Nuance Communications, Inc., Burlington, Massachusetts. Speech recognition software, or healthcare applications in which the voice of doctors and patients is acquired via a microphone array (e.g., using...). Ambient eXperience TM (DAX); Dragon is a registered trademark of Nuance Communications, Inc. in the U.S., other countries, or both. In some implementations, the data augmentation process 10 can be used to enhance ASR performance in various settings (e.g., voice-based vehicle control systems, voice-based dialogue systems, etc.). In some implementations, speech processing can be performed 608 via a trained speech processing system executed on the same computing device used to train the speech processing system and / or on another computing device. Therefore, it will be appreciated that within the scope of this disclosure, generating augmented data (e.g., time-varying spectral augmentation feature coefficients), training the speech processing system with the augmented data, and performing speech processing via the trained speech processing system can be performed on the same computing device and / or discrete computing devices.

[0078] As mentioned above, and at least refer to Figures 7-8The speech processing procedure 11 may receive 700 one or more inputs indicating at least one of the relative positions of the speaker and the microphone array and the relative orientation of the speaker and the microphone array. It may also receive 702 one or more reference signals. The speech processing system 704 may be trained using one or more inputs and reference signals.

[0079] In some implementations, speech processing procedure 11 may allow input or input signals indicative of the speaker's position relative to the microphone array and / or the speaker's orientation relative to the microphone array to be used to train and / or perform speech processing in a manner that takes into account dynamic acoustic variations. As mentioned above, in many cases, a microphone array can be used to receive speech signals. However, the microphone array may have the effect of filtering the speech spectrum in an angle-dependent manner. For example, if the speaker's position changes over time in the beam pattern of the beamformer, the speech may be affected by a time-varying filter. In the case of adaptive beamforming, the beamformer may attempt to dynamically turn toward the speaker, thus introducing another time variation. In some implementations, and as will be discussed in more detail below, the implementation of speech processing procedure 11 may specifically train the speech processing system to take into account these dynamic acoustic variations when performing speech recognition.

[0080] In some implementations, the speech processing procedure 11 may receive one or more inputs 700 indicating at least one of the relative positions of the speaker and the microphone array and the relative orientation of the speaker and the microphone array. As described above, and in some implementations, inputs different from the speech signals received at the microphone array may be received. Specifically, these inputs may indicate the relative positions of the speaker and the microphone array (e.g., the speaker's position relative to the microphone array within the monitored space) and / or the relative orientation of the speaker and the microphone array (e.g., the speaker's orientation relative to the microphone array). In this way and as will be discussed in more detail below, the speech processing procedure 11 may use the speaker's position and orientation to train a speech processing system (e.g., an ASR system, a speech biometric system, an emotion detection system, a medical symptom detection system, a hearing enhancement system) to take into account dynamic acoustic changes when performing speech processing.

[0081] In some implementations, receiving 700 or more inputs may include receiving 706 or more inputs from one or more sensors located within the monitored space. For example, the speech processing procedure 11 may utilize one or more sensors (e.g., audiovisual sensors, camera systems, microphone array 300, etc.) deployed within the monitored space to receive one or more inputs indicating the location of a speaker within the monitored space and / or the orientation of a speaker within the monitored space (e.g., head orientation, body orientation, etc.). Other examples of sensors may typically include RGB imaging systems, infrared imaging systems, ultraviolet systems, SONAR imaging systems, RADAR imaging systems, and thermal imaging systems. Therefore, it will be appreciated that any sensor can be used to receive these inputs within the scope of this disclosure.

[0082] Also refer to Figure 8 For example, and in some implementations, the speech processing procedure 11 may receive 700 input indicating the relative position or location of the speaker and the microphone array. For example, the data augmentation procedure 10 may receive 706 input indicating the direction of arrival (DOA) angle estimated by a beamformer (e.g., microphone array 300). In this example, the DOA may indicate the speaker's position within the monitored space. In another example, assume a camera system (e.g., camera system 800) is deployed within the monitored space. In this example, assume camera system 800 is configured to record video of a speaker within the monitored space. Assume speaker 310 moves within the monitored space while microphone array 300 is capturing speech signals. In this example, the speech processing procedure 11 may receive one or more inputs (e.g., input 802) from camera system 800 indicating the movement of the speaker within the monitored space, where these inputs (e.g., input 802) may indicate the speaker's position or location within the monitored space. While examples of receiving one or more inputs indicating the relative position of the speaker and microphone array from, for example, a camera system, have been provided, it should be understood that within the scope of this disclosure, one or more inputs indicating the relative position of the speaker and microphone array can be received from any sensor(s).

[0083] In some implementations, the speech processing procedure 11 may receive input indicating the relative orientation of the speaker and the microphone array. See again... Figure 8In some implementations, it is assumed that the speaker 310's head rotates while the camera system 800 and microphone array 300 are recording video and audio from the speaker 310. In this example, the camera system 800 may identify a change in the speaker 310's orientation (e.g., caused by head movement of the speaker 310). In response to identifying a change in the speaker 310's orientation, the speech processing procedure 11 may receive 706 one or more inputs indicating the speaker's relative orientation to the microphone array 300 within the monitored space. While examples of receiving one or more inputs indicating the relative orientation of the speaker and microphone array from the camera system 706 have been provided, it should be understood that within the scope of this disclosure, one or more inputs indicating the relative orientation of the speaker and microphone array can be received from any sensor(s).

[0084] In some implementations, receiving 700 or more inputs may include generating 708 or more inputs. For example, speech processing 11 may generate 708 or more analog inputs indicating at least one of the relative positions of the speaker and the microphone array and the relative orientation of the speaker and the microphone array. In some implementations, and as described above, speech processing 11 may enhance one or more reference or existing inputs to indicate a specific relative position and / or a specific relative orientation of the speaker and the microphone array. For example, and as described above, suppose speaker 310 moves from a first known position (e.g., known to speech processing 11) to a second known position (e.g., known to speech processing 11). In this example, speech processing 11 may generate one or more inputs indicating the relative positions of speaker 310 and microphone array 300. Specifically, speech processing 11 may generate one or more inputs indicating the first position and one or more inputs indicating the second position. In this way, if speaker 310 moves from the first position to the second position, speech processing 11 may generate 708 or more inputs that will be received by the sensor.

[0085] In another example, suppose speaker 310 moves from a first orientation (e.g., known to speech processing 11) to a second orientation (e.g., known to speech processing 11). In this example, speech processing 11 may generate one or more inputs indicating the relative orientation of speaker 310 and microphone array 300. Specifically, speech processing 11 may generate one or more inputs indicating the first orientation and one or more inputs indicating the second orientation. In this way, if speaker 708 moves from the first orientation to the second orientation, speech processing 11 may generate one or more inputs that 708 will receive by a sensor (or a set of sensors).

[0086] In some implementations, the speech processing procedure 11 may receive 702 one or more reference signals. For example, and as described above, the reference signals may typically include speech signals that can be received in real time, previously received and recorded speech signals (e.g., speech signals such as those saved to a speech signal corpus), and / or signals generated via data enhancement (e.g., via the data enhancement procedure 10 described above). In some implementations, and as described above, one or more reference signals may include acoustic variations introduced by changes in the relative position and / or orientation of the speaker and the microphone array.

[0087] In some implementations, receiving 702 one or more reference signals may include receiving 710 one or more speech signals from a microphone array. For example, speech processing 11 may receive 710 and store one or more speech signals within a defined time period. (See again) Figure 8 For example, and in some implementations, a reference signal 710 can be received while speaker 310 is speaking. In this example, microphone array 300 can be configured to receive one or more speech signals from speaker 310. In some implementations, speech processing procedure 11 can store one or more speech signals from the speaker for use in real-time or subsequent training on a speech processing system (e.g., ASR system 74).

[0088] In some implementations, receiving 702 one or more reference signals may include generating 712 one or more time-varying spectral enhancement signals. For example, and as described above, data enhancement process 10 may generate one or more time-varying spectral enhancement signals by defining 200 a model representing multiple acoustic variations of a speech signal, thereby defining multiple time-varying spectral changes, and applying 202 of the multiple time-varying spectral changes to the reference signal using filtering operations. In this way, time-varying spectral enhancement signals with characteristics associated with the multiple acoustic variations modeled by data enhancement process 10 (e.g., models representing changes in the relative positioning of the speaker and microphone array and / or adaptive beamforming) can be generated.

[0089] In some implementations, the speech processing procedure 11 can train the speech processing system 704 using one or more inputs and one or more reference signals. As described above, and in some implementations, training the speech processing system (e.g., ASR system 74) typically includes training the speech processing system with various inputs and desired outputs using one or more speech processing models. For example, one or more speech processing models typically include machine learning or neural network models configured to be trained for various types of speech processing. For example, the speech processing system may include an ASR system with one or more ASR models configured to be trained to recognize speech from input speech signals. Due to time-varying acoustic variations in the speech signal, including those from speaker movement and / or adaptive beamforming, ASR models trained for static conditions may not accurately recognize speech in these dynamic environments. While examples of ASR systems with ASR models have been described, it should be understood that any speech processing system with various speech processing models can be trained within the scope of this disclosure.

[0090] In some implementations, the automatic speech recognition process 11 can train the speech processing system 704 to receive and utilize these inputs during runtime when performing speech processing, by using inputs indicating the relative position and / or orientation of the speaker and microphone array. In this way, the speech processing process 11 can train the speech processing system 704 with additional inputs indicating the relative position and / or orientation of the speaker and microphone array to make it more robust to acoustic variations. As mentioned above, one or more inputs can be received from sensors other than the microphone elements of the microphone or microphone array, and can indicate the relative position and / or orientation of the speaker and microphone array.

[0091] In one example, the speech processing procedure 11 can train the 704 speech processing system using one or more inputs from one or more sensors located within the monitored space and one or more speech signals from a microphone array. In this example, the speech processing procedure 11 can train the 704 speech processing system (e.g., ASR system 74) using actual sensor data received from the microphone array and actual speech signals. In another example, the speech processing procedure 11 can train the 704 speech processing system using one or more inputs from one or more sensors located within the monitored space and one or more time-varying spectral enhancement signals. In this example, the speech processing procedure 11 can train the 704 speech processing system (e.g., ASR system 74) using actual sensor data and one or more time-varying spectral enhancement signals enhanced from one or more reference signals.

[0092] In another example, the speech processing procedure 11 may train the 704 speech processing system using one or more generated inputs and one or more speech signals from a microphone array. In this example, the speech processing procedure 11 may train the 704 speech processing system (e.g., ASR system 74) using enhanced inputs indicating the relative position and / or orientation of the speaker and the microphone array, and actual speech signals received from the microphone array. In another example, the speech processing procedure 11 may train the 704 speech processing system using one or more generated inputs and one or more time-varying spectral enhancement signals. In this example, the speech processing procedure 11 may train the 704 speech processing system (e.g., ASR system 74) using enhanced inputs indicating the relative position and / or orientation of the speaker and the microphone array, actual speech signals received from the microphone array, and one or more time-varying spectral enhancement signals enhanced from one or more reference signals. Therefore, it should be understood that any combination of enhanced and actual inputs and / or reference signals can be used to train the speech processing system within the scope of this disclosure.

[0093] In some implementations, training the 704 speech processing system using one or more inputs and one or more reference signals may include training a single speech processing model multiple times using various combinations of reference signals and inputs to account for numerous acoustic variations. For example, suppose the speech processing system (e.g., ASR system 74) includes a single speech processing model (e.g., ASR model 76). In this example, the speech processing procedure 11 may train the ASR model 76 with a first set of inputs and a first set of reference signals, wherein the first set of reference signals is associated with a first set of acoustic variations (e.g., generated by changes in the relative positions of the speaker and the microphone array and / or generated by adaptive beamforming). Continuing this example, the speech processing procedure 11 may train the ASR model 76 with a second set of inputs and a second set of reference signals, wherein the second set of reference signals is associated with a second set of acoustic variations (e.g., generated by changes in the relative positions of the speaker and the microphone array and / or generated by adaptive beamforming). In this example, the ASR model 76 may be trained to account for the first and second sets of acoustic variations, as well as the first and second sets of inputs. While examples of training a single speech processing model, such as two sets of inputs and two sets of reference signals, have been provided, it should be understood that within the scope of this disclosure, a single speech processing model can be trained using any number of inputs and any number of reference signals.

[0094] In some implementations, training the 704 speech processing system using one or more inputs and one or more reference signals may include training the 714 speech processing system using one or more speech signals received via a microphone array and one or more generated time-varying spectral enhancement signals. For example, speech processing procedure 11 may train the 714 speech processing system (e.g., ASR system 74) using a first portion of an actual speech signal and a second portion of a generated time-varying spectral enhancement signal. In one example, speech processing procedure 11 may receive 710, for example, 10 hours of field data from an actual speech signal, and may generate 712 or receive, for example, 90 hours of time-varying spectral enhancement signal. In this example, speech processing procedure 11 may train the 714 speech processing system using a speech signal received via a microphone array and a generated time-varying spectral enhancement signal. Although the examples use specific amounts of speech signals and generated time-varying spectral enhancement signals, it should be understood that any combination of speech signals and generated time-varying spectral enhancement signals can be used to train the 714 speech processing system within the scope of this disclosure.

[0095] In some implementations, training the speech processing system 704 using one or more input and reference signals may include training 716 multiple speech processing models for multiple acoustic variations associated with one or more speech signals. For example, when training the speech processing system, speech processing procedure 11 may train 716 different speech processing models. In one example, speech processing procedure 11 may train a first speech processing model (e.g., ASR model 76) with a first set of inputs and a first set of reference signals, wherein the first set of reference signals is associated with a first set of acoustic variations (e.g., caused by changes in the relative positions of the speaker and the microphone array and / or caused by adaptive beamforming).

[0096] Continuing this example, speech processing procedure 11 can train a second speech processing model (e.g., ASR model 78) using a second set of inputs and a second set of reference signals, wherein the second set of reference signals is associated with a second set of acoustic variations (e.g., generated by changes in the relative positions of the speaker and the microphone array and / or generated by adaptive beamforming). Speech processing procedure 11 can train a third speech processing model (e.g., ASR model 80) using a third set of inputs and a third set of reference signals, wherein the third set of reference signals is associated with a third set of acoustic variations (e.g., generated by changes in the relative positions of the speaker and the microphone array and / or generated by adaptive beamforming). Therefore, ASR system 74 can include, for example, three ASR models, which are trained separately to take into account the first set of acoustic variations, the second set of acoustic variations, and the third set of acoustic variations, as well as the first set of inputs, the second set of inputs, and the third set of inputs. While examples of, for example, three sets of inputs and three sets of reference signals have been provided for training, for example, three speech processing models, it should be understood that within the scope of this disclosure, any number of inputs and any number of reference signals can be used to train any number of speech processing models. In this way, many speech processing models can be trained to allow speech processing systems to process speech in a variety of dynamic acoustic environments (e.g., changes in the relative position of the speaker and the microphone array, adaptive beamforming, etc.).

[0097] In some implementations, the speech processing procedure 11 may receive 718 one or more runtime inputs indicating at least one of the relative positions of the speaker and the microphone array and the relative orientation of the speaker and the microphone array. As described above, and again referring to Figure 8 For example, runtime inputs (e.g., runtime input 804) can be received from various sensors deployed within the monitored space. For example, a camera system (e.g., camera system 800) may be deployed in the monitored space. As will be discussed in more detail below, speech processing procedure 11 may allow speech processing to be performed by a trained speech processing system based at least in part on one or more inputs received at runtime 718 (i.e., one or more runtime inputs).

[0098] In some implementations, the speech processing step 11 can receive the 720 speech signal via a microphone array. See again... Figure 8 Assume that speaker 310 is speaking in the monitored space. In this example, microphone array 300 can receive 720 speech signals 308 emitted by speaker 310. Although an example of receiving, for example, a single speech signal is provided, it should be understood that any number of speech signals 720 can be received within the scope of this disclosure.

[0099] In some implementations, speech processing procedure 11 can perform speech processing 722 via a trained speech processing system using one or more runtime inputs and speech signals. Continuing the example above, assume that the speech processing system (e.g., ASR system 74) is trained using one or more inputs and one or more reference signals, as described above. In this example, ASR system 74 can be configured to perform automatic speech recognition 722 on one or more received speech signals (e.g., speech signal 308) at least in part based on training ASR system 74 with one or more inputs (e.g., received and / or generated from sensors prior to training) and one or more reference signals (e.g., speech signals received and / or generated prior to training) to take into account acoustic variations present in one or more speech signals. Therefore, ASR system 74 can utilize one or more runtime inputs (e.g., runtime input 804) to help ASR system 74 identify specific acoustic variations within speech signal 308. For example, by utilizing runtime input 804, ASR system 74 can determine, for example, whether speaker 310 moves within the monitored space while speaking and / or whether microphone array 300 is performing adaptive beamforming to direct beamforming toward speaker 310. Therefore, speech processing procedure 11 can improve the ability of the trained ASR system to perform 722 more accurate speech recognition using one or more runtime inputs. As described above, and in some implementations, within the scope of this disclosure, 722 speech processing using the trained speech processing system can be performed on the same computing device that trained 704 and / or on a separate computing device. While examples of ASR systems have been described above, it should be understood that any speech processing system can be used to perform 722 speech processing within the scope of this disclosure.

[0100] In some implementations, performing speech processing 722 via a trained speech processing system using one or more runtime inputs and speech signals may include selecting 724 at least one trained speech processing model from a plurality of trained speech processing models, at least in part based on one or more runtime outputs and speech signals. For example, and as described above, it is assumed that speech processing procedure 11 trains 704 a plurality of speech processing models for a plurality of acoustic variations associated with one or more reference signals and one or more inputs. In this example, speech processing procedure 11 may use one or more runtime inputs to select 724 a trained automatic speech model for performing speech recognition on received speech signals. As described above, it is assumed that speech processing procedure 11 trains 704, for example, three speech processing models (e.g., ASR models 76, 78, 80) for three different combinations of inputs and / or reference signals. In this example, upon receiving 718 to runtime input 804 and receiving 720 to speech signal 308, speech processing procedure 11 may select 724 a trained ASR model (e.g., one of ASR models 76, 78, 80) from a plurality of ASR models. While examples of selecting 724 trained ASR models from, for example, three trained ASR models have been provided, it should be understood that, within the scope of this disclosure, any number of speech processing models can be selected from any number of trained speech processing models to perform speech processing.

[0101] In some implementations, the speech processing procedure 11 can select a speech processing model from multiple speech processing models using one or more runtime inputs. For example, the speech processing procedure 11 can select a specific speech processing mode from multiple trained speech processing models in response to receiving a specific runtime input or a set of runtime inputs corresponding to the inputs used to train the speech processing model. In this way, the speech processing procedure 11 can be configured to identify a specific speaker position and / or speaker orientation relative to the microphone array during runtime, and can use speech processing models trained using the same or similar inputs.

[0102] Also refer to Figure 9 The diagram illustrates a client electronic device 38. Although the client electronic device 38 is shown in the diagram, it is for illustrative purposes only and is not intended to limit this disclosure, as other configurations are possible. For example, any computing device capable of performing all or part of the data augmentation process 10 may be used instead. Figure 9 The client electronic device 38 in the system may include, but is not limited to, computing device 12 and / or client electronic devices 40, 42, 44.

[0103] Client electronic device 38 may include a processor and / or microprocessor (e.g., microprocessor 900) configured to, for example, process data and execute the aforementioned code / instruction set and subroutines. Microprocessor 900 may be coupled to the aforementioned storage devices (e.g., storage device 30) via a storage adapter (not shown). An I / O controller (e.g., I / O controller 902) may be configured to couple microprocessor 900 to various devices, such as keyboard 904, pointing / selection devices (e.g., mouse 906), custom devices, such as microphones (e.g., device 908), USB ports (not shown), and printer ports (not shown). A display adapter (e.g., display adapter 910) may be configured to couple display 912 (e.g., multiple CRT or LCD displays) to microprocessor 900, while a network controller / adapter 914 (e.g., Ethernet adapter) may be configured to couple microprocessor 900 to the aforementioned network 14 (e.g., the Internet or a local area network).

[0104] As will be understood by those skilled in the art, this disclosure can be embodied as a method, system, or computer program product. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which are generally referred to herein as “circuit,” “module,” or “system.” Furthermore, this disclosure can take the form of a computer program product on a computer-usable storage medium having computer-usable program code embodied on that medium.

[0105] Any suitable computer-usable or computer-readable medium may be used. A computer-usable or computer-readable medium can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, devices, or transmission media. More specific examples (not an exhaustive list) of computer-readable media may include: electrical connections having one or more wires, portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, transmission media such as those supporting the Internet or intranets, or magnetic storage devices. A computer-usable or computer-readable medium may also be paper or another suitable medium on which a program is printed, as the program can be electronically captured by optical scanning of, for example, paper or other media, and then compiled, interpreted, or otherwise processed as appropriate, and then stored in computer memory. In the context of this document, a computer-usable or computer-readable medium can be any medium capable of containing, storing, transmitting, propagating, or transporting a program for use by or in connection with an instruction execution system, apparatus, or device. Computer-usable media can include propagated data signals, wherein computer-usable program code is embodied in baseband form or as part of a carrier wave. Computer-usable program code can be transmitted using any suitable medium, including but not limited to the Internet, wired, fiber optic cable, RF, etc.

[0106] The computer program code used to perform the operations of this disclosure may be written in an object-oriented programming language, such as Java, Smalltalk, C++, etc. However, the computer program code used to perform the operations of this disclosure may also be written in a conventional procedural programming language, such as the "C" programming language or a similar programming language. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via a local area network / wide area network / Internet (e.g., network 14).

[0107] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer / special-purpose computer / other programmable data processing apparatus, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in the flowchart illustrations and / or block diagram blocks.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can instruct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions / actions specified in flowcharts and / or block diagram blocks.

[0109] Computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be executed on the computer or other programmable apparatus, thereby producing a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide to implement the functions / actions specified in the flowchart and / or block diagram blocks.

[0110] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a code module, segment, or portion comprising one or more executable instructions for implementing a specified logical function(s). It should also be noted that in some alternative implementations, the functions described in a block may not appear in the order shown in the figures. For example, in fact, two blocks shown consecutively may be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware, or a combination of dedicated hardware and computer instructions, that performs the specified functions or actions.

[0111] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a,” “an,” and “the” should also include the plural forms unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and / or “comprising” as used in this specification specify the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0112] The corresponding structures, materials, actions, and equivalents of all means or steps plus functional elements in the following claims are intended to include any structure, material, or action for performing the function in conjunction with other claimed elements of the specific claim. The description in this disclosure is for illustrative and descriptive purposes only and is not intended to be exhaustive or limiting to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of this disclosure. This embodiment was chosen and described in order to best explain the principles and practical application of this disclosure and to enable those skilled in the art to understand the various embodiments of this disclosure with various modifications suitable for the particular intended use.

[0113] Many implementations have already been described. Therefore, the disclosure of this application has been described in detail with reference to embodiments thereof, and it will be apparent that changes and variations are possible without departing from the scope of this disclosure as defined in the appended claims.

Claims

1. A computer-implemented method executed on a computing device, comprising: A model is defined to represent multiple acoustic variations of a speech signal, thereby defining multiple time-varying spectral changes, wherein defining the model representing the multiple acoustic variations of the speech signal includes: defining a model representing the multiple acoustic variations of the speech signal, wherein the multiple acoustic variations of the speech signal depend on changes in the beam pattern of an adaptive beamformer and the movement of multiple sound sources relative to the beam of the adaptive beamformer; and The multiple time-varying spectral changes are applied to multiple characteristic coefficients in the target domain of the reference signal, thereby generating multiple time-varying spectral enhancement characteristic coefficients of the reference signal.

2. The computer-implemented method of claim 1, wherein defining the model representing the plurality of acoustic variations of the speech signal comprises: Define a model representing multiple acoustic variations of the speech signal associated with changes in the relative positions of the speaker and the microphone.

3. The computer-implemented method of claim 1, wherein the model defining the plurality of acoustic variations of the speech signal comprises one or more of the following: The multiple acoustic variations of the speech signal are modeled using statistical distributions; and The following is used to model the multiple acoustic variations of the speech signal: a mathematical model representing the multiple acoustic variations of the speech signal associated with a specific use case scenario.

4. The computer-implemented method of claim 1, wherein defining the model representing the plurality of acoustic variations of the speech signal comprises: Receive one or more inputs associated with one or more of the speaker movement and speaker orientation.

5. The computer-implemented method of claim 1, wherein defining the model representing the plurality of acoustic variations of the speech signal comprises: The mapping of the multiple acoustic variations to one or more feature coefficients in the target domain is generated via a machine learning model.

6. The computer-implemented method of claim 5, wherein applying the plurality of time-varying spectral changes to the plurality of characteristic coefficients of the target domain of the reference signal comprises: The mapping of the plurality of acoustic variations to the one or more feature coefficients of the target domain is generated simultaneously via a machine learning model, and the plurality of time-varying spectral changes are applied to the plurality of feature coefficients of the reference signal via the machine learning model.

7. The computer-implemented method according to claim 4, further comprising: The speech processing system is trained using the plurality of time-varying spectral enhancement feature coefficients of the reference signal and the one or more inputs associated with one or more of speaker position and speaker orientation.

8. The computer-implemented method according to claim 1, further comprising: The speech processing system is trained using the plurality of time-varying spectral enhancement feature coefficients of the reference signal, thereby defining the trained speech processing system.

9. The computer-implemented method according to claim 8, further comprising: Speech processing is performed via the trained speech processing system, wherein the trained speech processing system is executed on at least one computing device.

10. A computer program product residing on a non-transitory computer-readable medium, the non-transitory computer-readable medium having a plurality of instructions stored thereon, the plurality of instructions, when executed by a processor, causing the processor to perform operations including: A model is defined to represent multiple acoustic variations of a speech signal, thereby defining multiple time-varying spectral changes, wherein the model representing the multiple acoustic variations of the speech signal includes: Define a model representing multiple acoustic variations of the speech signal, wherein the multiple acoustic variations of the speech signal depend on the changes in the beam pattern of the adaptive beamformer and the movement of multiple sound sources relative to the beam of the adaptive beamformer; as well as The multiple time-varying spectral changes are applied to multiple characteristic coefficients in the target domain of the reference signal, thereby generating multiple time-varying spectral enhancement characteristic coefficients of the reference signal.

11. The computer program product of claim 10, wherein defining the model representing the plurality of acoustic variations of the speech signal comprises: Define a model representing multiple acoustic variations of the speech signal associated with changes in the relative positions of the speaker and the microphone.

12. The computer program product of claim 10, wherein the model defining the plurality of acoustic variations of the speech signal comprises one or more of the following: The multiple acoustic variations of the speech signal are modeled using statistical distributions; and The following is used to model the multiple acoustic variations of the speech signal: a mathematical model representing the multiple acoustic variations of the speech signal associated with a specific use case scenario.

13. The computer program product of claim 10, wherein defining the model representing the plurality of acoustic variations of the speech signal comprises: Receive one or more inputs associated with one or more of the speaker movement and speaker orientation.

14. The computer program product of claim 10, wherein defining the model representing the plurality of acoustic variations of the speech signal comprises: The mapping of the multiple acoustic variations to one or more feature coefficients in the target domain is generated via a machine learning model.

15. The computer program product of claim 14, wherein applying the plurality of time-varying spectral changes to the plurality of characteristic coefficients of the target domain of the reference signal comprises: The mapping of the plurality of acoustic variations to the one or more feature coefficients of the target domain is generated simultaneously via a machine learning model, and the plurality of time-varying spectral changes are applied to the plurality of feature coefficients of the reference signal via the machine learning model.

16. The computer program product of claim 13, wherein the operation further comprises: The speech processing system is trained using the plurality of time-varying spectral enhancement feature coefficients of the reference signal and the one or more inputs associated with one or more of speaker position and speaker orientation.

17. The computer program product according to claim 10, further comprising: The speech processing system is trained using the plurality of time-varying spectral enhancement feature coefficients of the reference signal, thereby defining the trained speech processing system.

18. The computer program product according to claim 17, further comprising: Speech processing is performed via the trained speech processing system, wherein the trained speech processing system is executed on at least one computing device.

19. A computing system, comprising: Memory; as well as A processor configured to define a model representing multiple acoustic variations of a speech signal, thereby defining multiple time-varying spectral changes, wherein defining the model representing the multiple acoustic variations of the speech signal includes: defining a model representing the multiple acoustic variations of the speech signal, the multiple acoustic variations of the speech signal depending on changes in the beam pattern of an adaptive beamformer and movement of multiple sound sources relative to the beam of the adaptive beamformer, and wherein the processor is further configured to apply the multiple time-varying spectral changes to multiple feature coefficients in a target domain of a reference signal, thereby generating multiple time-varying spectral enhancement feature coefficients of the reference signal.

20. The computing system of claim 19, wherein defining the model representing the plurality of acoustic variations of the speech signal comprises: Define a model representing multiple acoustic variations of the speech signal associated with changes in the relative positions of the speaker and the microphone.

21. The computing system of claim 19, wherein the model defining the plurality of acoustic variations of the speech signal comprises one or more of the following: The multiple acoustic variations of the speech signal are modeled using statistical distributions; and The following is used to model the multiple acoustic variations of the speech signal: a mathematical model representing the multiple acoustic variations of the speech signal associated with a specific use case scenario.

Citation Information

Patent Citations

  • Speech recognition system

    CN110383377A

  • Audio augmentation using environmental data

    US10595149B1