Fine-grained dynamic learning of in-vehicle noise patterns for speech applications

Abstract

Aspects of fine-grained, dynamic noise pattern learning for speech applications include a vehicle with a body and cabin. The vehicle contains a processor coupled to a memory. The processor can be configured to embed multimodal data for environmental and vehicle data. The embedded acoustic data can originate from microphone recordings in the cabin. The processor can chain the embeddings to form a latent vector that characterizes the embeddings, thereby estimating a mean and variance of the latent vector using an adaptive time window. Using the mean and variance of the latent vector, the processor can identify a noise type, with noise type identification being fine-grained over the adaptive time window to accurately emulate vehicle noises.

Description

introduction

[0001] Existing in-vehicle noise cancellation techniques used in speech-based applications such as speech recognition are often limited in scope due to the restricted types of noise that are actually suppressed. Noise processing circuitry is typically confronted with noise from a variety of different sources. Existing protocols often fail to account for cacophonies resulting from subtle changes in the vehicle's rhythm, with noise enhancement primarily focused on the intended noise source, for example, the user's voice. As a result, the scope of enhancements in speech-based applications is often limited to the inflections in the user's voice, while other sources of interference are ignored.Even if existing techniques are capable of suppressing other noise sources, they are typically static, meaning they do not account for changes in the noise sources over time. These types of solutions result in a "one-size-fits-all" noise suppression scheme that does not account for ambient noise and noise attributable to a specific vehicle and road type, and is unable to dynamically account for changes in background noise in or near real-time.

[0002] The document FENG, X [ET AL]: On using heterogeneous data for vehicle-based speech recognition: A DNN-based approach. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 4385–4389, describes a way to improve a hybrid neural network system for a vehicle-based speech recognition task. Description

[0003] Aspects of this disclosure utilize in-vehicle sound pattern learning, which includes using static and dynamic input signals in the vehicle and information from multiple sources to create a learning model of in-vehicle sound patterns. This model can be used to improve the performance of voice applications such as virtual assistants, hands-free devices, and in-vehicle communications, to name a few. The multiple input sources may include vehicle information in a calibration database, climate control signals from a cluster, vehicle control signals from a vehicle control module, climate control signals from infotainment systems or radios, road conditions from global positioning systems (GPS), and application-based map data.

[0004] A variety of (multimodal) factors are considered to accurately and dynamically identify the noise type in a vehicle environment. These factors include vehicle data (e.g., audio signals captured by on-board microphones, speed, engine rpm, engine temperatures, turn signals, open or closed windows, open or closed sunroof, horns, and so on) and environmental data (weather data, roughness of road sections, road disturbance rating, and the like).

[0005] In one aspect of the present disclosure, a method for learning in-vehicle sound patterns for speech applications is disclosed. The method includes embedding multimodal data from ambient and vehicle data; embedding acoustic data from microphone recording data in a cabin of the vehicle; concatenating the embeddings to form a latent vector characterizing the embeddings; estimating a mean and variance of the latent vector using an adaptive time window; and identifying a sound type using the mean and variance of the latent vector, wherein the sound type identification is fine-grained over the adaptive time window to accurately emulate vehicle sounds.

[0006] In various embodiments, the environmental data includes weather data, road roughness, road disturbance ratings, or location-based application programming interfaces (APIs), and the vehicle data includes speed, engine speed per minute (RPM), engine temperature, turn signals on or off, windows open or down, a sunroof open or closed, or horns honking in the vehicle. In various embodiments, the sound type identification also includes calculating an interquartile range (IQR) of the mean and variance of the latent vector as a measure of a spread of both the mean and variance within a calibratable time window. The method may further include generating a fine-grained sound type identification based on the calculated IQR and the adaptive time window.

[0007] In various embodiments, the fine-grained sound type identification comprises one or more of the following elements: stationary mean and stationary variance; non-stationary mean and stationary variance; stationary mean and non-stationary variance; and non-stationary mean and non-stationary variance. The adaptive time window may be based on one or more of the following functions: an inverse logistic function, an inverse sigmoid function, or a combination of mean IQR and variance IQR. The identified sound types may be used for in-vehicle speech applications, including active noise cancellation or speech enhancement. The identified sound type may also be used for one or more out-of-vehicle applications, such as noise suppression or dereverberation.

[0008] According to the invention, a vehicle for learning interior cabin noise patterns for speech applications comprises a vehicle body with a cabin disposed therein; a memory; a processor coupled to the memory and configured to: embed multimodal data from environmental and vehicle data; embed acoustic data from microphone recording data in the cabin; concatenate the embeddings to form a latent vector characterizing the embeddings; estimate a mean and variance of the latent vector over an adaptive time window; and identify a noise type using the mean and variance of the latent vector, wherein the noise type identification is fine-grained to accurately emulate vehicle noise.

[0009] According to the invention, a system for learning in-vehicle sound patterns for speech applications comprises a vehicle body having a cabin disposed therein; a memory; a processor coupled to the memory, wherein the processor and the memory are coupled within the vehicle body and the processor is configured to: embed multimodal data from environmental and vehicle data; embed acoustic data from microphone recording data in the cabin; concatenate the embeddings to form a latent vector characterizing the embeddings; estimate a mean and variance of the latent vector over an adaptive time window; and identify a sound type using the mean and variance of the latent vector, wherein the sound type identification is fine-grained to accurately emulate vehicle sounds.

[0010] In various embodiments of the vehicle and system, the environmental data includes weather data, a roughness of road sections, a rating of road disturbances, or location-based application programming interfaces (APIs), and the vehicle data includes speed, engine speed per minute (RPM), engine temperature, turn signals on or off, windows open or closed, a sunroof open or closed, or horns honking in the vehicle. In various embodiments, the sound type identification also includes calculating an interquartile range (IQR) of the mean and variance of the latent vector as a measure of a spread of both the mean and variance within a calibratable time window. The method may further comprise generating a fine-grained sound type identification based on the calculated IQR and the adaptive time window.

[0011] In various embodiments of the vehicle and the system, the fine-grained sound type identification comprises one or more of the following elements: stationary mean and stationary variance; non-stationary mean and stationary variance; stationary mean and non-stationary variance; and non-stationary mean and non-stationary variance. The adaptive time window may be based on one or more of the following functions: an inverse logistic function, an inverse sigmoid function, or a combination of mean IQR and variance IQR. The identified sound types may be used for in-vehicle speech applications, including active noise cancellation or speech enhancement. The identified sound type may also be used for one or more out-of-vehicle applications, such as noise suppression or dereverberation.

[0012] The above summary is not intended to describe every embodiment or aspect of the present disclosure. Rather, the above summary merely provides examples of some of the novel concepts and features set forth herein. The above features and advantages, as well as other features and associated advantages of this disclosure, will be readily apparent from the following detailed description of the illustrated examples and representative modes for carrying out the present disclosure, taken in conjunction with the accompanying drawings and the appended claims. Furthermore, this disclosure expressly includes the various combinations and subcombinations of the elements and features set forth above and below. Brief description of the drawings

[0013] The accompanying drawings, which are incorporated in this specification, illustrate embodiments of the disclosure and, together with the description, explain the principles of the disclosure. Fig. 1 is a conceptual diagram describing techniques for dynamically learning vehicle noise patterns according to one aspect of the disclosure. Fig. 2 is a conceptual diagram describing techniques for dynamically learning in-vehicle noise patterns using a latent vector according to one aspect of the disclosure. Fig. Figure 3 is a flowchart illustrating techniques for identifying noise types based on mean and variance using interquartile ranges (IQR µ,σ2 ) designations, in accordance with an aspect of revelation. Fig. Figure 4 is a conceptual diagram illustrating techniques for determining an adaptive time window using inter-quartile range (IQR µ,σ2 ) designations, in accordance with an aspect of revelation. Fig. Figure 5 is a block diagram of a system for implementing statistical noise analysis for use in speech-based applications.

[0014] The accompanying drawings are not necessarily to scale and may present a simplified representation of various features of the present disclosure, including, for example, certain dimensions, orientations, positions, and shapes. In some cases, readily apparent features may be omitted from certain drawings so as not to unduly obscure the concepts of the disclosure. The details associated with such features will be determined in part by the particular intended application and use case environment. Detailed description

[0015] The present disclosure may be embodied in many different forms. Representative examples of the disclosure are illustrated in the drawings and are described in detail herein as non-limiting examples of the disclosed principles. To this end, elements and limitations described in the "Summary," "Introduction," "Summary," and "Detailed Description" sections, but not expressly set forth in the claims, should not be incorporated into the claims, individually or collectively, by implication, inference, or otherwise.

[0016] For the purposes of this description, the use of the singular includes the plural and vice versa, unless expressly excluded; the terms "and" and "or" apply both subjunctive and disjunctive; and the words "including," "containing," "comprising," "with," and the like mean "including without limitation." Furthermore, words of approximation such as "approximately," "almost," "substantially," "generally," "about," and so on may be used herein to mean "at, near, or almost at," or "within 0-5% of," or "within acceptable manufacturing tolerances," or logical combinations thereof. A component "configured" to perform a particular function is capable of performing the particular function without modification, rather than merely having the potential to perform the particular function after further modifications.In other words, if the hardware described is expressly configured to perform the specified function, it is specifically selected, created, implemented, used, programmed, and / or designed to perform the specified function.

[0017] The detailed description and the drawings or illustrations are supportive and descriptive of the present teachings, but the scope of the present teachings is defined solely by the claims. While some of the best modes and other embodiments for carrying out the present teachings have been described in detail, various alternative designs and embodiments for carrying out the present teachings exist, as defined in the appended claims. Furthermore, this disclosure expressly includes combinations and subcombinations of the elements and features presented above and below.

[0018] For disclosure purposes, the terms "granularity" and "fine-grained" generally refer to the number of data bits that characterize the time window and represent a set of corresponding data of similar length. The finer the grain, the more detailed the time window and / or the changes to an adaptive time window for a given data set. For example, the data might be road-generated noise. The time window might then represent the resolution of the acoustic noise, for example, the number of changes in the noise pattern.

[0019] The present subject matter describes techniques for dynamically learning sound patterns for voice applications. In the case of a vehicle, the disclosure describes the use of static and dynamic in-vehicle input signals and information transmitted from multiple sources to create an in-vehicle sound pattern learning model. This learning model can, in turn, be used to improve the performance of various voice applications through noise suppression and echo cancellation. Examples of such applications include virtual assistants, hands-free systems, and in-vehicle communication.Multiple input sources for the learning model may include vehicle information in a calibration database, climate control signals from a cluster, vehicle control signals from a vehicle control module, climate control signals from infotainment radios, and road conditions from navigation systems, Global Positioning Systems (GPS), and related navigation-based data.

[0020] In one aspect, various factors are considered to accurately and dynamically identify the noise type(s) present in the vehicle environment, and in particular, the noise types that degrade the quality of a speech transmission over smaller, changing (or finer-grained) adaptive time segments in a speech-based application. Such factors can be divided into vehicle data and environmental data. Vehicle data can encompass a wide range of data types.Specific, non-limiting examples of vehicle data include acoustic data (for example, audio signals captured by on-board microphones positioned with their respective inputs in the vehicle cabin), speed, engine revolutions per minute (RPM), engine temperature, the presence or absence of turn signals, whether one or more windows are open or closed, whether the sunroof is open, the vehicle horn honking, and a large number and variety of sounds that a vehicle can independently generate. An example of the latter category is whether the muffler adequately attenuates the engine noise of an internal combustion engine vehicle. Other examples include sounds generated by one or more circuits or mechanical devices that are so powerful that they acoustically reverberate through the cabin and into the affected resource.

[0021] In addition to vehicle data, as previously mentioned, environmental data may also be present that could negatively impact the voice application in question. Environmental data includes, for example, weather data (ranging from probable events like rain or thunderstorms to less probable events like tornadoes, earthquakes, and hurricanes, as well as other weather-based criteria), the roughness of road sections, the rating of road disturbances, screaming pedestrians or cyclists, motorcycles sharing the road, and the like.

[0022] Aspects of the disclosure include a system having a processor, memory for executing code, and application-based elements for executing on the processor, such as application programming interfaces (APIs). For purposes of this disclosure, a "processor" is construed to include one or more processors. Where more than one processor is involved, they may or may not be identical in design. They may run identical or different executable code and instruction sets. They may use identical or different architectures. Non-exhaustive examples include a multiple instruction, multiple data (MIMD) processor or a single instruction, multiple data (SIMD) processor. The processors may also be complex instruction set computer-based (CISC) processors or reduced instruction set computer-based (RISC) processors, or a combination thereof.The processor can be partially or exclusively customized for the vehicle.

[0023] The processor can be implemented in software, hardware, or a combination thereof. For example, the processor can include a digital signal processor (DSP) for executing instructions in hardware. In some embodiments, the control unit and memory can be implemented in the vehicle as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a system on a chip (SoC). Physically, the processor can be implemented in one or more electronic control units (ECUs) of the vehicle. The control units, in turn, can be networked or wired together under the coordination of a control unit, for example, a controller area network (CAN), or simply be permanently connected.In various embodiments, machine learning can be performed in a vehicle using one or more processor concepts that dynamically exchange data using cloud computing. The use of a cloud platform is advantageous in this context because vehicle products are increasingly implemented using software. This trend is driven, among other things, by increasing customer demand for new vehicle features and new functions. This trend also enables manufacturers to design and build new platforms by developing code that quickly and efficiently supports the capabilities of existing hardware. The cloud computing environment can further reduce storage costs and increase the speed of manufacturers' new developments.

[0024] It should be noted that while the use of the cloud is one method for implementing the principles of this disclosure, the principles need not be limited to the cloud, and in some embodiments, more traditional data-centric systems may be used. For example, two or more vehicles may be networked to share their processing and data-sharing resources, but without using a cloud platform. As previously mentioned, the processing and data caching capabilities required to implement the principles of this disclosure may be independent of a particular vehicle. One or more general-purpose or special-purpose processors, or a series of them, may be used in a vehicle for implementing machine learning.Various types of computer-readable media, such as flash memory, hard disks, and the like, can be loaded into high-speed memory. This may include, for example, code configured to be executed by a processor to process incoming sound signals for use in dynamic machine learning for speech applications. The vehicle in question may communicate with one of several relevant nodes via a cellular network, such as 3G / 4G / 5G, or another suitable network, such as a metropolitan area network (MAN), a wide area network (WAN), a local area network (LAN), Bluetooth, or another network protocol.

[0025] When the embodiments take a network-based approach using the cloud as a computing platform, the various features and functions of the vehicle may be stored in local memory during a machine learning session. The cloud used here may refer to the cloud platform specifically tailored to the vehicle's machine learning applications for noise reduction and echo cancellation. The data may be uploaded to the cloud and stored there until needed again. Likewise, data may be downloaded from the cloud when needed as data input or as executable code.

[0026] During an active phone call in a vehicle, a dedicated set of signal processing libraries can be used. For example, if the driver is using a Bluetooth device or Android Auto and requests a phone call connection, a separate library can be used. When noise reduction and echo cancellation techniques are used, the outgoing signal is typically defined before being transmitted to the cloud during an active phone call. To maintain good call performance, the signal must be optimized by suppressing unwanted noise and canceling echoes. When a buffer is opened and the speaker begins speaking, the resulting noise is pleasant to the human ear, making it ideal for the speech verification engine.Conversely, in conditions of high noise and echoes, the speech recognition engine may have difficulty recognizing the words spoken into the microphone.

[0027] As a result, disclosure aspects consider a potentially large number of data points that can cause even moderate vehicle noise to impact voice applications. To this end, one benefit of maintaining optimal sound is that an application should be aware of most of the environmental factors affecting vehicle acoustics at any given time. The better the application's noise cancellation platform understands the factors that negatively impact in-vehicle acoustics, the more intelligently the solution platform can eliminate these anomalies during a voice application. Because the voice cancellation application knows the vehicle's speed at any given time, it can calculate how much noise the vehicle would generate in terms of sound pressure level (SPL). In turn, the vehicle speed can be used to calculate the engine rpm.The application is designed to account for acoustic effects in the vehicle, such as the occurrence of a turn signal window. As described below with reference to . Fig. 5, the information relevant to these types of acoustic artifacts may be available on a vehicle bus 524 for storage and processing. The information may be in the form of, for example, an Extensible Markup Language (XML), a JavaScript Object Notation (JSON) format for representing structured and other non-acoustic data using JavaScript object syntax, or a C structure. The C structure is a user-defined data type that can be used to group elements of potentially different types into a single type. The above formats are exemplary, and other protocols are possible. In one aspect of the disclosure, the noise suppression and echo cancellation system uses regression analysis with fine-grained (highly and finely tunable) adaptive time windows to estimate noise and echo generation.While the data is being adaptively filtered, the processor can continue to make corrective adjustments to suppress the data and cancel the echoes.

[0028] For example, the processor in the ECU evaluates the different data types over their respective time windows to determine their impact on voice applications, if any are present. For example, if a sunroof and a left window are open, the processor can use this information to get a better estimate of the signal-to-noise ratio (SNR) in the vehicle cabin. Another example: based on the detected acoustic signatures, the processor can determine that the ventilation system is on or that the blower is running. Based on this knowledge, the processor can make further determinations, such as whether the air from either system is directed toward the driver's face or is being directed at them.Example sets of such events provide the echo cancellation element with immediate and ongoing knowledge of the acoustic artifacts requiring echo cancellation. The fact that the noisy environment can suddenly change or transform into a different noise, so that the filter constantly adapts to the noise in the new time window, is a feature of disclosure that enables dynamic, granular noise suppression and echo cancellation based on adaptive filtering, since the data stream is continuously provided and its size changes dynamically over time.

[0029] Various data streams can complement the adaptive filtering to enhance the overall effect. For example, in some embodiments, sensors or microphones can be installed in the rear of the vehicle to determine the type of terrain the vehicle is traveling on. The noise caused by road conditions can have a significant impact on the use of an in-vehicle voice application. In addition to this technique, the vehicle can obtain data about the road surface from the global positioning system (GPS) by considering the geographic relationship between two or more points. In the latter embodiments, the road information can exist alongside road information stored in a computer-readable medium in various modules of the vehicle, for example, a list of possible surfaces the vehicle may encounter en route to a destination.These methods can be comparable to autocorrelation metrics, where the system can change the coefficients and metrics to accommodate different input characteristics (for example, the terrain obtained from different sources such as a mapping program and a GPS). Noise can be represented as a probability distribution, also called a parametric representation of the distribution. An example of a probability distribution is the "Gaussian distribution," a bell-shaped curve that has a mean µ and a squared standard deviation σ. 2 contains.

[0030] The processor can use regression analysis to model the relationship between a dependent variable and one or more independent variables. Regression analysis can be used to determine the strength of the correlation between the variables and model the future relationship between them. In principle, regression analysis can be used in machine learning to predict future outcomes and analyze past results. In the context of machine learning, regression can enable the system to predict a continuous outcome (for example, a "y" value on a vertical axis) based on the value of "x" predictor variables.

[0031] The processor can often determine or identify relevant variables for a particular regression analysis using embeddings. Embedding different data types may also be referred to herein as "embedding." Various application programming interfaces ("APIs") provided for these analyses can enable the use of embeddings to measure the relatedness of text strings and other types of data strings. An embedding is a vector (list) of numbers. The distance between two vectors measures their relationship. In the example above, the processor can perform a regression analysis on existing data to identify the condition of the road surface.

[0032] Other information about the vehicle's operating status, such as the vehicle's speed, the state of the air conditioning system including the fan speed, the number of passengers in the vehicle cabin, and other information easily obtained from the vehicle's communication network, can be used to establish context. This information is used to build a regression model. Adequate amounts of training data are used to refine these regression models. This essentially involves tokenizing the input feature vectors, which are then transformed into relevant vector embeddings using a specialized neural network architecture, also known as "transformer" models.These embeddings lead to a more accurate estimate of the statistical system identification of vehicle noise and are therefore intended to contribute to more effective fine-grained noise suppression. When the data is finally converted into the corresponding vector embeddings, one can say that the data are "embedded" for the input data in question (acoustic noise, etc.) to form the embeddings, so the term can be used in its verb or noun form.

[0033] Fig. 1 is a conceptual diagram 100 describing techniques for dynamically learning sound patterns in a vehicle, according to one aspect of the disclosure. The techniques describe a solution for dynamically learning sound patterns that is implemented partially in the vehicle and, in one embodiment, partially in a cloud for storing dynamic libraries. As previously mentioned, the cloud need not be used in the method, as other storage media locally in the vehicle or at another networked location may be used instead.

[0034] As in Fig. 1, the data relevant to the application in question may come from multiple sources. Acoustic data, such as cabin noise and the user's voice, may be derived from one or more microphones 102 positioned as inputs at strategic locations throughout the vehicle cabin and routed via appropriate vehicle wiring 104 (one of which is shown) to a source of multimodal data embedding. Examples include vehicle data 120 from the vehicle 122, including speed, engine speed per minute (RPM), engine temperature, turn signals, window position (up / down), sunroof position (open / closed), horns, and the like. Another example is an HVAC or air conditioning system 124, which may control various other acoustic phenomena such as the type of ventilation or fan speed 126.In addition, the noise specific to the vehicle make and engine type 132 can be regularly stored in a calibration database 128 and retrieved as needed. External environmental data, including the roughness of road sections and the road disturbance rating 130, can be provided.

[0035] An embedding in the context of machine learning is a mapping or transformation of high-dimensional data into low-dimensional data, typically in the form of a vector with features related to the embedded data. In other words, a data embedding is a generally dense numerical representation of data, expressed as a vector. The set of formed vectors quantifies the similarities between categories. The data in the multimodal data embedding block 106 learns the representation or converts high-dimensional multimodal data (e.g., acoustic data, vehicle signals, fan speed, calibration database, and so on) into a single low-dimensional vector space. This enables more efficient and effective machine learning tasks such as classification, clustering, and query.Various approaches can be used for multimodal data embedding, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), long-term memory (LSTM), and geometric machine learning methods.

[0036] In addition, as in Fig. As further described in Figure 2, the embeddings in the multimodal data embedding block 106 are used to represent the respective noise type as a parametric vector. For system identification, the noise can be represented as a vector of parameters.

[0037] For example, multimodal data embedding allows for the integration of collective data from a large number of influential data sources. The multimodal nature of data embedding can significantly increase the accuracy of machine learning by providing a more realistic and comprehensive set of factors that would otherwise add noise and impede the usability of language applications.

[0038] Once the initial mean and variance estimates are determined in logic block 108 for specific types of noise to statistically represent the noise as a probability density function, the processor can use the accumulated data in an adaptive time window 110. The processor can perform a time window-based analysis to evaluate the statistical properties of the noise in logic block 12. Once the processor has determined this classification in block 12, noise suppression and echo cancellation can be performed more accurately and effectively.For example, the data reclassified in 12 may be used in the application in question to enhance the user's speech at the front end of the vehicle (114), it may be used for active noise cancellation (116), or it may be used to suppress or enhance acoustic signatures associated with other speech applications (118).

[0039] The two main advantages of the system in Fig. 1 compared to existing approaches is that, firstly, the types of relevant data are significantly expanded through sophisticated multimodal embedding (106). 'Sophisticated' refers to the fact that a variety of noise sources are considered in the machine learning model, which in turn increases the accuracy and precision of the result. Secondly, the data can be used to improve a wide variety of applications by employing more realistic models for noise suppression and cancellation where necessary. Thirdly, the large amount of acoustic and non-acoustic data that can be processed means that more user applications can benefit from initial data analysis.

[0040] Fig. 2 is a conceptual diagram describing techniques 200 for dynamically learning noise patterns in a vehicle 222 using a latent pattern vector 213 according to one aspect of the disclosure. While Fig. 1 provides a high-level overview of the system components and a description of various embodiments, Fig. 2 more detailed information on parametric estimation according to one embodiment. For example, in Fig. 2 which in Fig. 1 not explicitly shown embodiment of the embedding chain 207 added. Fig. 2 shows the relevant data related to the environmental data 201 and the vehicle data 205. As already mentioned, the factors required to accurately and dynamically identify the noise type in vehicle environments also include the vehicle data 205. In short, in Fig. 2, the processor scans a variety of data parameters to create a data distribution with a mean µ and a variance σ 2to be defined. Vehicle data includes, for example, the audio signals picked up by the microphones in the vehicle, the speed, the engine speed, the engine temperature, whether a turn signal is activated, whether the windows are open or closed, whether a sunroof is open or closed, whether a horn is honked, and the like. Examples of environmental data 201, on the other hand, can be the weather, the roughness of road sections, and the rating of road disturbances. Data embedding, as shown in logic blocks 203 and 209, for example, is a technique for representing data as points in an n-dimensional space such that similar data points cluster together. As already mentioned, embedding is generally a low-dimensional representation of higher-dimensional data.The microphones 202 can be positioned in the booth to generate acoustic data that is combined or matched with the remaining acoustic data from sources outside the booth.

[0041] The idea of ​​non-acoustic data embedding 203 is derived from supervised or unsupervised representation learning techniques (see Fig. 2). This method can also be generally referred to as parametric representation distribution. Non-acoustic data includes not only structured text data, but also time series data such as dynamic vehicle data (vehicle data that changes over time). Fig. Figure 2 shows the embedding concatenation 207 between the acoustic data embedding 209 and the non-acoustic data embedding 203. During embedding concatenation, the generated embeddings are combined into a single embedding. This can be done by simply concatenating the vectors representing the embeddings or by a more sophisticated method, such as weighted averaging. Once the non-acoustic data embedding 203 is concatenated with the acoustic data embedding 209, the resulting feature vector is passed to a latent fully connected layer 211 to calculate the mean µ and the variance σ. 2to estimate. The processor can use a deep learning mechanism in the AI ​​to determine a distribution. The distribution is represented by the latent vector 213. That is, the latent vector 213 is a calculation of the initial value of this distribution. In this example, the latent pattern vector 213 can be viewed as a range with two values—the mean µ and the variance σ. 2 . The mean µ and the variance σ 2 of the latent vector 213 can then be used by the processor to identify the noise type via the adaptive time window 215, as shown in logic block 217. For example, noise identification uses the mean µ and the variance σ 2of the noise during the applicable adaptive time window 215 to align the data in time and accurately identify the noise type. The use of a fine-grained adaptive time window that can rapidly change its width over time in small or large steps, as here, contrasts with existing implementations in vehicle noise suppression, which often use fixed time window sizes. However, these existing prediction methods are inaccurate because, unlike the adaptive time window presented here, the fixed time window is not adjusted to account for recent trends in the data. Examples of the need for an adaptive time window include changing road conditions and many others, including changing trends in the input data in general.

[0042] In the example of changing ambient or road noise, this time change can be very rapid. Here, the adaptive time window 215 allows the system to use an initial time window to identify what type of noise / environmental noise is present. The processor can use this initial time window to collect the noise samples and then estimate the mean and variance based on the noise type identification, as in logic block 217. The smaller the time window, the more accurate the noise predictions. If the processor is computationally efficient enough, it can select time samples of almost arbitrarily small length, which increases the accuracy of the distribution and thus the effectiveness of the noise reduction. Noise estimation is a statistically multivariate problem in which the system incorporates multivariate inputs. At the same time, the physical conditions that give rise to these variables change.For example, in a vehicle, the turn signal may suddenly be activated, a person coughs, the wind starts to blow, the wipers are turned on, raindrops fall on the windshield, and so on. Each of these events can be viewed and treated as a distribution with individually adaptive time windows.

[0043] The identified noise type can later be used as an enabler in various speech applications 219 in the vehicle that require information about the noise type, such as speech enhancement modules and active noise cancellation (see also Fig. 2).

[0044] As described below, the principles of disclosure enable dynamic, fine-grained machine learning of speech patterns while a vehicle is moving. Real-time noise patterns can then be applied to various speech applications. Currently, the "one-size-fits-all" imitation of road noise and other acoustic and non-acoustic artifacts often does not produce accurate results. As discussed in Fig. 2, both acoustic and non-acoustic data can be embedded (logic blocks 203, 209), using geometric deep learning to form a latent vector that provides an estimate of the mean µ and the variance σ 2 using the relevant time window 215 to account for the dynamic or changing nature of the input data. That is, once the processor has an initial estimate of the mean µ and the variance σ 2to statistically represent the noise as a probability function, the processor may use the time window-based analysis 215 to further estimate the statistical properties of the noise at different time intervals, as done in logic block 217.

[0045] After the processor has calculated the statistical properties of the mean µ and the variance σ 2 during the applicable time window to identify the noise type (logic block 217), the obtained statistical noise data may then be used in an in-vehicle voice application, such as voice enhancement, active noise cancellation, and so on, and similar techniques to dynamically suppress noise and cancel echoes, as in logic block 219. These applications may also be used as input to other voice-based applications to improve their overall performance.

[0046] It should be noted that the embedding of non-acoustic data 203, the embedding of acoustic data 209, and the embedding concatenation 207 result in a fully linked layer 211, which allows a latent vector 213 to incorporate the relevant data types required to accurately calculate the noise distribution for correction and enhancement purposes. Previous approaches that do not consider the various data types are less accurate or inaccurate.

[0047] In a further aspect of the disclosure, a method for estimating the mean µ and the variance σ 2disclosed. Based on an interquartile range (IQR) of the estimated mean and the variance as a measure of dispersion, four different noise types can be automatically determined. The IQR is a measure of statistical dispersion. Dispersion, in turn, is a measure of the spread of a distribution of data. The IQR dispersion is defined as the difference between the 75 th and 25 thPercentiles of the data. In the example of Gaussian waveforms described above, two such waveforms may have the same mean but be spread over a larger or smaller time window. When a waveform has little dispersion, the waveform tends to peak at one point and quickly decay to zero or a negligible value on either side of the peak. In contrast, a waveform with a larger dispersion means that the data is spread over a larger time window. IQR is a means of describing the magnitude of this dispersion using four different quadrants, as explained below. Advantageously, the processor can perform fine-grained noise type identification based on the calculated IQR and an adaptive time window.The noise types described below include stationary mean and variance, non-stationary mean and variance, stationary mean and non-stationary variance, and non-stationary mean and non-stationary variance.

[0048] Fig. 3 is a flowchart 300 describing techniques for identifying sound types based on mean and variance using IQR designations according to one aspect of the disclosure. Fig. Figure 3 shows techniques for estimating the mean and variance of a collection of input data and then using an IQR technique to determine a noise type. For disclosure purposes, the noise type result may be synonymous with the noise type when using an IQR model. Fig. 3 shows the interquartile ratio of the mean µ 231a and the interquartile ratio of the variance σ 2213b within a target threshold. In some cases, either the mean µ 231a or the variance σ 2 itself be stationary, or both values ​​can be stationary. Fig. 3 shows that in Fig. 2 and Fig. 3 used adaptive time windows 315 as well as the mean µ 231a and the variance σ 2 213b to calculate the IQR as a measure of dispersion for both the mean µ 231a and the variance σ 2 231b within a calibratable (e.g., fine-grained and dynamically adjustable) time window, as in logic block 317.

[0049] In logic block 319, the µ-based IQR values ​​(IQR µ ) and the σ 2 -based IQR values ​​(IQR σ2 ) with four possible thresholds characterized by the noise type results 330, 340, 350 and 360. In result 300, both IQR µ as well as IQR σ2below the respective target thresholds µth and σth, resulting in a curve with a stationary mean and a stationary variance, as shown in the graph for result 330. In result 340, only IQR exceeds µ its target threshold µth, which in this example leads to a curve with a non-stationary mean and a stationary variance. For result 350, only the variance exceeds the IQR σ2 its target threshold σth, resulting in a curve with a stationary mean but a non-stationary variance, as shown in the graph for result 350 in Fig. 3. For result 360, both IQR µ as well as IQR σ2 greater than their respective thresholds µth and σth, and no diagram results. Depending on the quartile or the results of the various comparisons, different levels of predictability can be determined.

[0050] In short, the embodiment of Fig. Figure 3 uses the four different types of noise to estimate the mean and variance. As mentioned earlier, the noise signature changes based on the adaptive time window. For example, if the signature has a constant mean, the result with a constant variance is a simple scenario. However, if the standard deviation (and thus the variance) or the mean changes, these two patterns may be distributions in their own right, both of which should be considered, as shown in results 340 and 350, respectively.

[0051] By taking the estimated mean and variance and comparing whether they exceed a threshold in a given quartile, the noise type can be identified with the appropriate characteristics of the mean and standard deviation and the accurate distributions.

[0052] Fig. Figure 4 is a conceptual diagram 400 describing techniques for determining an adaptive time window in accordance with one aspect of the disclosure. An IRQ range is a measure of where most values ​​lie in a data graph. The processor takes the mean 413a and the variance 413b obtained from the embodiment in Fig. 3 in each quarter of the IRQ to produce an inverse logistic function, an inverse sigmoid or Z-curve function ("function") 405. The IQR divides the data into quarters, with the lowest quarter of the IQR containing the smallest quarter of values ​​in the input data and the upper quarter containing the highest quarter of values, with the two intermediate quarters containing the middle part of the data.

[0053] In function 405, the y-axis represents the sum of the values ​​IQR µ and IQR µ 2obtained from the IRQ analyses in fields 420a and 420b. Each of fields 420a and 420b represents an adaptive time window. In field 420a, the input mean µ 413 is entered, with the range of the means defined by the lower half 406 and the upper half 408. The lower half 406 comprises the means µ 1-µ 3 and the upper half 408 the means µ 5-µ 7. The lower quarter 410 corresponds to a Q1 of µ 2, and the upper quarter 420 corresponds to a Q3 of µ 6. Thus, IQR µ = Q3 - Q1 from equation 440.2. Related to field 420b, in which the median σ 2 413 is entered, a similar analysis results in IQR σ2 = Q3 - Q1, where the lower half is 411 σ 2 1 - σ 2 3 and the upper half 413 σ 2 5 - σ 2 7. In this case, the lower quarter 461 corresponds to a value of σ 2 2 and the upper quarter 462 a value of σ 26 . Thus the sum y = IQR µ and IQR σ2 from fields 420a and 420b represent the y-axis of function 405. The x-axis of function 405 represents the applicable time window. The time window can be calculated as follows: x=Wln|y||y+1| where W is a predefined window value and y is the value calculated above for the y-axis. For clarity, note that the median value in field 420 corresponds to a mean of Q2 = µ4. Likewise, the median value in field 420b corresponds to a median of Q2 = σ 2 4 .

[0054] In the summary of Fig. 4, the processor retrieves the mean and variance obtained from Fig. 3 within each quarter of the IQR. Function 405 corresponds to the rate of change, or the first derivative, of the data. Thus, real-time input values ​​are input, and a non-negative derivative value should result at each inflection point. This technique corresponds to a sigmoid, or cost, function. Since one goal is to understand the distribution pattern, the sigmoid function can be used to define the distribution of the mean or the distribution of the standard deviation in cases where the two are non-stationary.

[0055] Fig.Figure 5 is a block diagram of a system 500 for implementing statistical noise analysis for use in speech-based applications. The system and apparatus of the disclosure may be implemented in a vehicle 123. In various configurations, when the cloud is used to deliver data and code to the vehicle via a suitable network, the apparatus may operate as a system including the vehicle 123 and a remote cloud. In some embodiments, the system may be implemented in more than one vehicle. Within the vehicle 123, the term "processor 504" itself may refer to one or more processors within one or more computing devices. Such computing devices may be, for example, a computer's motherboard, an electronic control unit (ECU), or an automotive microcontroller unit (MCU) within the vehicle 123. Depending on the vehicle's architecture and electronic configuration, the ECU and / or MCU may include the processor 504 and memory 508 (e.g., cache memory, dynamic random access memory (DRAM), static random access memory (SRAM), or the like). In other cases, the ECU (or MCU) is kept separate from these components, and optionally, the ECU or MCU may be hard-wired to these components.

[0056] Mass storage 512 may be a magnetic disk drive, a solid-state drive, or other form of non-volatile memory and may be used to retrieve data from the cloud via an antenna 522 over a suitable network 548 and under the control of processor 504. Returning to processor 504, this component can be broadly configured to include a variety of different processing devices. For example, processor 504 may be a general-purpose microprocessor (or a plurality thereof). In other embodiments, processor 504 may be a digital signal processor (DSP), an application-specific integrated circuit (ASIC) (in which other devices shown in this figure may be embedded), a field-programmable gate array (FPGA), a system-on-a-chip (SoC), or other processors or arrays thereof.Various processors within processor 504 may perform specific functions of the techniques disclosed herein.

[0057] Memory 508 may be DRAM or other faster memory into which executable code corresponding to active applications or payload data may be loaded. Portable storage 516 may be removable storage, such as flash memory or a USB drive. Portable storage 516 may include firmware or a means for uploading firmware to processor 504.

[0058] The output display 506 may include the display panel(s) on the vehicle's dashboard, such as one or more I / O touchpads indicating an infotainment system or a navigation system. The output display 506 may allow a user to control settings consistent with the techniques described herein. Furthermore, the vehicle 123 may include one or more microphones 510 within the cabin. If multiple microphones 510 are used, they may be positioned at various locations within the vehicle's interior so that the microphone's acoustic input can most efficiently capture sounds generated by the vehicle 123 or its occupants. A user interface 514 may also be included.The user interface 514 may display corresponding displays embedded as part of the output display 506, or it may include switches, knobs, and other control functions built into the instrument panel to allow the user (or a professional automotive technician) to interact with the algorithms executing in the processor 504 and perform other functions relevant to the system. Various peripherals 518 may also be used in conjunction with the system, such as Bluetooth devices, radios, or peripheral controllers to perform various functions.Examples include increasing the granularity of machine learning or adapting other controls related to similar functions, although these types of controls may also be implemented using the network 548, antenna 522, and transceiver 520 under the control of the processor 504 (for example, in firmware updates or responses to ECU network requests). Alternatively, these functions may also be controlled via the user interface 514.Furthermore, to the extent that the device incorporating one or more of the identified elements in the vehicle 123 is added externally to the vehicle (e.g., as part of retrofitting older vehicles, and so on), docking hardware 527 may be provided, for example, under the dashboard or in another accessible area, to allow the vehicle 123 to be modified to perform one or more of the techniques described herein.

[0059] The advantages of the techniques described here are comprehensive. Unlike existing implementations, the techniques of multimodal data embedding and concatenation can dramatically improve the accuracy of the machine learning process by providing data on a large number of acoustic and non-acoustic, vehicle- and environmental-related phenomena that would otherwise degrade the performance of a speech application. Latent parameter estimation ensures efficiency and enables the numerical transformation of the diverse data for efficient further processing. Furthermore, IQR-based noise type identification considers an essentially complete set of scenarios, depending on whether the mean and variance factors characterizing a noise distribution are active or stationary.The methods can also be performed using real-time vehicle control information, allowing the benefits to be fully realized even while the vehicle is moving. The techniques are also applicable to various voice applications, including virtual assistants, in-vehicle communication (for example, users can communicate information more clearly and quietly to other users in the vehicle, which can be particularly useful in larger vehicles with multiple rows of seats or in trucks), and hands-free devices. Furthermore, the adaptive nature of the time window allows the machine learning process to achieve very high temporal resolution, resulting in very accurate calculations of the expected noise and interference. This higher resolution allows the methods to enhance the user experience with speech applications and provide a visible improvement in performance. The IQR nature of the estimated mean and variance enables the identification of different noise patterns / types.

[0060] It should also be noted that the identified noise types need not only be used for various in-vehicle voice applications such as active noise cancellation and speech enhancement. The identified noise types can also be used for various voice applications outside the vehicle, such as noise suppression, dereverberation for improved performance of external voice assistance, and the like.

[0061] The detailed description and the drawings or illustrations are supportive and descriptive of the present teachings, but the scope of the present teachings is defined solely by the claims. While some of the best modes and other embodiments for carrying out the present teachings have been described in detail, various alternative designs and embodiments for carrying out the present teachings exist, as defined in the appended claims. Furthermore, this disclosure expressly includes combinations and subcombinations of the elements and features presented above and below.

Claims

[1] A vehicle (222) for learning interior cabin noise patterns for speech applications, comprising: a vehicle body with a cabin arranged therein; a RAM; a processor coupled with the memory and configured to: embed multimodal data from environmental and vehicle data; embed acoustic data (209) from microphone recording data from the booth; concatenate the embeddings to form a latent vector (213) characterizing the embeddings; to estimate a mean and a variance of the latent vector (213) over an adaptive time window (215); and identify a noise type using the mean and variance of the latent vector (213), wherein the noise type identification is fine-grained to accurately emulate vehicle noise. [2] Vehicle (222) according to claim 1, wherein: the environmental data includes weather data, a roughness of road sections, an assessment of road disturbances or location-based application programming interfaces, APIs, and Vehicle data may include speed, engine revolutions per minute, RPM, engine temperature, turn signals on or off, windows open or closed, a sunroof open or closed, or a vehicle horn honking. [3] The vehicle (222) of claim 1, wherein the processor is configured to identify the noise type using an interquartile range, IQR, of the mean and variance of the latent vector (213) as a measure of a spread of both the mean and variance within a calibratable time window (215). [4] The vehicle (222) of claim 3, wherein the processor is further configured to generate a fine-grained noise type identification based on the calculated interquartile range, IQR, and the adaptive time window (215). [5] The vehicle (222) of claim 4, wherein the fine-grained noise type identification comprises one or more of the following elements: stationary mean and stationary variance; non-stationary mean and stationary variance; stationary mean and non-stationary variance; or non-stationary mean and non-stationary variance. [6] A system (500) for learning sound patterns in the vehicle (222) for speech applications, comprising: a vehicle body with a cabin arranged therein; a RAM; a processor coupled to the memory, wherein the processor and the memory are coupled within the vehicle body, the processor configured to: embed multimodal data from environmental and vehicle data; embed acoustic data (209) from microphone recording data from the booth; concatenate the embeddings to form a latent vector (213) characterizing the embeddings; to estimate a mean and a variance of the latent vector (213) over an adaptive time window (215); and identify a noise type using the mean and variance of the latent vector (213), wherein the noise type identification is fine-grained to accurately emulate vehicle noise. [7] System (500) according to claim 6, wherein: the environmental data includes weather data, a roughness of road sections, an assessment of road disturbances or location-based application programming interfaces, APIs, and Vehicle data may include speed, engine revolutions per minute, RPM, engine temperature, turn signals on or off, windows open or closed, a sunroof open or closed, or a vehicle horn honking. [8] The system (500) of claim 6, wherein the processor is configured to identify the noise type using an interquartile range, IQR, of the mean and variance of the latent vector (213) as a measure of a spread of both the mean and variance within a calibratable time window (215). [9] The system (500) of claim 8, wherein the processor is further configured to generate a fine-grained noise type identification based on the calculated interquartile range, IQR, and the adaptive time window (215). [10] The system (500) of claim 9, wherein the fine-grained noise type identification comprises one or more of the following elements: stationary mean and stationary variance; non-stationary mean and stationary variance; stationary mean and non-stationary variance; and non-stationary mean and non-stationary variance.

Images