Fine-grained in-vehicle dynamic noise pattern learning for speech applications

Abstract

Aspects of fine-grained dynamic noise pattern learning for speech applications include a vehicle having a body with a cabin. Embedded within the vehicle is a processor coupled to the memory. The processor may be configured to embed multi-modal data for environmental data and vehicle data. The embedded acoustic data may be from data captured by a microphone in the cabin. The processor may concatenate the inserts to form implicit vectors characterizing the inserts to estimate the mean and variance of the implicit vectors using adaptive time windows. The processor may identify a noise type using the mean and variance of the implicit vectors, the noise type identification being fine-grained via an adaptive time window to accurately simulate vehicle noise.

Description

[0001] introduction Due to the limited types of noise actually suppressed, the prior art for in-vehicle noise suppression used with speech-based applications (such as voice recognition) is often limited in scope. Noise processing circuits are usually faced with noise from multiple different sources. In existing protocols, the cacophony caused by subtle changes in vehicle rhythm is often ignored, and noise improvement is mainly focused on the expected noise source (such as the user's voice). Therefore, the scope of improvement of speech-based applications is often limited to the pitch change (inflection) in the user's voice, while ignoring other interference sources. Even in the case where the prior art has the ability to suppress other interference sources, the technology is usually static because they do not consider the change over time in the noise source. These types of solutions lead to the following "one-size-fits-all" noise suppression scheme: the noise suppression scheme fails to consider ambient noise and the sound attributed to specific vehicles and road types, and lacks the ability to dynamically account for changes to background noise in real time or near real time. Summary of the invention

[0002] Aspects of the present disclosure use in-vehicle noise pattern learning, which includes creating an in-vehicle noise pattern learning model using static and dynamic in-vehicle input signals and information from multiple sources. The model can be used to improve the performance of voice applications such as virtual assistants, hands-free calling, and in-vehicle communications. Multiple input sources can include vehicle information in a calibration database, climate control signals from a car cluster, vehicle control signals from a vehicle control module, climate control signals from an infotainment system or radio, road conditions from a global positioning system (GPS), and application-based map data.

[0003] Consider multiple (multimodal) factors to accurately and dynamically identify noise types in the vehicle environment. These factors include vehicle data (e.g., audio captured by in-vehicle microphones, speed, engine RPM, engine temperature, turn signals, windows up or down, sunroof open or closed, honking, etc.), as well as environmental data (weather data, road section roughness, road interruption score, etc.).

[0004] In one aspect of the present disclosure, a method for in-vehicle noise pattern learning for speech applications is disclosed. The method includes: embedding multimodal data from environmental data and vehicle data; embedding acoustic data from data captured by a microphone in a vehicle cabin; concatenating the embeddings to form a latent vector representing the embeddings; using an adaptive time window to estimate the mean and variance of the latent vector; and using the mean and variance of the latent vector to identify noise types, the noise type identification being fine-grained via the adaptive time window to accurately simulate vehicle noise.

[0005] In various embodiments, the environmental data includes weather data, road segment roughness, road interruption score, or a location-based application programming interface (API), and the vehicle data includes speed, engine revolutions per minute (RPM), engine temperature, turn signal on or off, window up or down, sunroof open or closed, or honking a horn in the vehicle. In addition, in various embodiments, identifying the noise type includes calculating the interquartile range (IQR) of the mean and variance of the latent vector as a dispersion measure of both the mean and variance within a calibrable time window. The method may further include generating a fine-grained noise type identification based on the calculated IQR and the adaptive time window.

[0006] In various embodiments, the fine-grained noise type identification includes one or more of the following: a stationary mean and a stationary variance; a non-stationary mean and a stationary variance; a stationary mean and a non-stationary variance; and a non-stationary mean and a non-stationary variance. The adaptive time window can be based on one or more of the following: an inverse logistic function, a reverse sigmoid function, or a combined mean IQR and variance IQR. The identified noise type can be used for in-vehicle speech applications, including active noise cancellation or voice enhancement. The identified noise type can be further used for one or more out-of-vehicle applications, including denoising or dereverberation.

[0007] In another aspect of the present disclosure, a vehicle for in-cabin noise pattern learning for speech applications comprises: a vehicle body including a cabin arranged therein; a memory; a processor coupled to the memory and configured to: embed multimodal data from environmental data and vehicle data; embed acoustic data from data captured by a microphone in the cabin; cascade the embeddings to form a latent vector representing the embeddings; estimate the mean and variance of the latent vector via an adaptive time window; and identify a noise type using the mean and variance of the latent vector, wherein the noise type identification is fine-grained to accurately simulate vehicle noise.

[0008] In another aspect of the present disclosure, a system for in-vehicle noise pattern learning for speech applications includes: a vehicle body, including a cabin arranged therein; a memory; a processor coupled to the memory, the processor and the memory being coupled within the vehicle body, the processor being configured to: embed multimodal data from environmental data and vehicle data; embed acoustic data from data captured by a microphone in the cabin; cascade the embeddings to form a latent vector representing the embeddings; estimate the mean and variance of the latent vector via an adaptive time window; and identify a noise type using the mean and variance of the latent vector, wherein the noise type identification is fine-grained to accurately simulate vehicle noise.

[0009] In various embodiments of the vehicle and system, the environmental data includes weather data, road segment roughness, road interruption score, or a location-based application programming interface (API), and the vehicle data includes speed, engine revolutions per minute (RPM), engine temperature, turn signal on or off, window up or down, sunroof open or closed, or honking a horn in the vehicle. In addition, in various embodiments, identifying the noise type includes calculating the interquartile range (IQR) of the mean and variance of the latent vector as a measure of dispersion of both the mean and variance within a calibrable time window. The method may further include generating a fine-grained noise type identification based on the calculated IQR and the adaptive time window.

[0010] In various embodiments of the vehicle and system, the fine-grained noise type identification includes one or more of the following: stationary mean and stationary variance; non-stationary mean and stationary variance; stationary mean and non-stationary variance; and non-stationary mean and non-stationary variance. The adaptive time window can be based on one or more of the following: an inverse logistic function, a reverse sigmoid function, or a combined mean IQR and variance IQR. The identified noise type can be used for in-vehicle speech applications, including active noise cancellation or voice enhancement. The identified noise type can be further used for one or more out-of-vehicle applications, including denoising or de-reverberation.

[0011] The present application provides the following embodiments: 1. A method for in-vehicle noise pattern learning for speech applications, the method comprising: Embedding multimodal data from environmental data and vehicle data; acoustic data embedded in data captured by microphones in the cabin of the vehicle; concatenating the embeddings to form a latent vector representing the embeddings; Using an adaptive time window to estimate the mean and variance of the latent vector; and The noise type is identified using the mean and variance of the latent vector, and the noise type identification is fine-grained via the adaptive time window to accurately simulate vehicle noise.

[0012] 2. The method of embodiment 1, wherein: The environmental data includes weather data, road roughness, road disruption score, or location-based application programming interface (API), and The vehicle data includes speed, engine revolutions per minute (RPM), engine temperature, turn signal on or off, windows up or down, sunroof open or closed, or honking a horn in the vehicle.

[0013] 3. The method of embodiment 1, wherein identifying the noise type comprises calculating the interquartile range (IQR) of the mean and variance of the latent vector as a measure of dispersion of both the mean and variance within a calibrable time window.

[0014] 4. The method according to embodiment 3 further includes generating a fine-grained noise type identification based on the calculated IQR and the adaptive time window.

[0015] 5. A method according to embodiment 4, wherein the fine-grained noise type identification includes one or more of the following: a stationary mean and a stationary variance; a non-stationary mean and a stationary variance; a stationary mean and a non-stationary variance; or a non-stationary mean and a non-stationary variance.

[0016] 6. The method of embodiment 4, wherein the adaptive time window is based on one or more of: an inverse logistic function, an inverse sigmoid function, or a combined mean IQR and variance IQR.

[0017] 7. The method of embodiment 1, wherein the identified noise type is used for in-vehicle speech applications including at least one of active noise cancellation or voice enhancement.

[0018] 8. A vehicle for in-cabin noise pattern learning for speech applications, comprising: a vehicle body, including a carriage arranged therein; Memory; a processor coupled to the memory and configured to: Embedding multimodal data from environmental data and vehicle data; Embedding acoustic data from data captured by microphones in the vehicle cabin; concatenating the embeddings to form a latent vector representing the embeddings; estimating the mean and variance of the latent vector via an adaptive time window; and The noise type is identified using the mean and variance of the latent vector, where the noise type identification is fine-grained to accurately model vehicle noise.

[0019] 9. The vehicle of embodiment 8, wherein: The environmental data includes weather data, road roughness, road disruption score, or location-based application programming interface (API), and The vehicle data includes speed, engine revolutions per minute (RPM), engine temperature, turn signal on or off, windows up or down, sunroof open or closed, or honking a horn in the vehicle.

[0020] 10. A vehicle according to embodiment 8, wherein the processor is configured to use the interquartile range (IQR) of the mean and variance of the latent vector as a measure of dispersion of both the mean and variance within a calibrable time window to identify the noise type.

[0021] 11. The vehicle of embodiment 10, wherein the processor is further configured to generate a fine-grained noise type identification based on the calculated IQR and the adaptive time window.

[0022] 12. A vehicle according to embodiment 11, wherein the fine-grained noise type identification includes one or more of the following: a stationary mean and a stationary variance; a non-stationary mean and a stationary variance; a stationary mean and a non-stationary variance; or a non-stationary mean and a non-stationary variance.

[0023] 13. The vehicle of embodiment 11, wherein the adaptive time window is based on one or more of: an inverse logistic function, an inverse sigmoid function, or a combined mean IQR and variance IQR.

[0024] 14. The vehicle of embodiment 8, wherein the identified noise type is used for in-vehicle speech applications including at least one of active noise cancellation or voice enhancement.

[0025] 15. A system for in-vehicle noise pattern learning for speech applications, comprising: a vehicle body, including a carriage arranged therein; Memory; A processor coupled to the memory, the processor and the memory being coupled within the vehicle body, the processor being configured to: Embedding multimodal data from environmental data and vehicle data; Embedding acoustic data from data captured by microphones in the vehicle cabin; concatenating the embeddings to form a latent vector representing the embeddings; estimating the mean and variance of the latent vector via an adaptive time window; and The noise type is identified using the mean and variance of the latent vector, where the noise type identification is fine-grained to accurately model vehicle noise.

[0026] 16. The system of embodiment 15, wherein: The environmental data includes weather data, road roughness, road disruption score, or location-based application programming interface (API), and The vehicle data includes speed, engine revolutions per minute (RPM), engine temperature, turn signal on or off, windows up or down, sunroof open or closed, or honking a horn in the vehicle.

[0027] 17. A system according to embodiment 15, wherein the processor is configured to use the interquartile range (IQR) of the mean and variance of the latent vector as a measure of dispersion of both the mean and variance within a calibrable time window to identify the noise type.

[0028] 18. The system of embodiment 17, wherein the processor is further configured to generate a fine-grained noise type identification based on the calculated IQR and the adaptive time window.

[0029] 19. A vehicle according to embodiment 18, wherein the fine-grained noise type identification includes one or more of the following: a stationary mean and a stationary variance; a non-stationary mean and a stationary variance; a stationary mean and a non-stationary variance; and a non-stationary mean and a non-stationary variance.

[0030] 20. The vehicle of embodiment 15, wherein the identified noise type is used for one or more off-vehicle applications including denoising or dereverberation.

[0031] The above summary of the invention is not intended to represent every embodiment or every aspect of the present disclosure. On the contrary, the above summary of the invention merely provides examples of some novel concepts and features set forth herein. When considered in conjunction with the accompanying drawings and the appended claims, the above-mentioned features and advantages of the present disclosure and other features and attendant advantages will become apparent from the following detailed description of the illustrated examples and representative modes for implementing the present disclosure. In addition, the present disclosure explicitly includes various combinations and sub-combinations of the elements and features presented above and below. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate implementations of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0033] Figure 1 is a conceptual diagram describing a technique for dynamic in-vehicle noise pattern learning according to an aspect of the present disclosure.

[0034] Figure 2 is a conceptual diagram describing a technique for dynamic in-vehicle noise pattern learning using sample latent vectors according to an aspect of the present disclosure.

[0035] Figure 3 The present invention describes a method for using the interquartile range according to one aspect of the present disclosure. Flowchart of the technique for noise type identification based on mean and variance of the designation.

[0036] Figure 4 is a method for using the interquartile range according to one aspect of the present disclosure. A conceptual diagram specifying a technique for determining an adaptive time window.

[0037] Figure 5 is a block diagram of a system for implementing statistical noise analysis for use in speech-based applications.

[0038] The drawings are not necessarily drawn to scale and may present simplified representations of various features of the present disclosure, including, for example, specific dimensions, orientations, positions, and shapes. In some cases, well-known features may be omitted from certain drawings to avoid unduly obscuring the concepts of the present disclosure. Details associated with such features will be determined in part by the specific intended application and use case environment. DETAILED DESCRIPTION

[0039] The present disclosure allows for embodiments in many different forms. Representative examples of the present disclosure are shown in the accompanying drawings and described in detail herein as non-limiting examples of the disclosed principles. For this reason, elements and limitations described in the abstract, introduction, summary, and detailed description but not explicitly set forth in the claims should not be incorporated into the claims individually or collectively by implication, inference, or otherwise.

[0040] For the purposes of this description, unless otherwise stated, the use of the singular includes the plural, and vice versa, the terms "and" and "or" shall be both conjunctive and disjunctive, and the words "comprising", "containing", "including", "having", etc. shall mean "including but not limited to". In addition, approximate words such as "about", "almost", "substantially", "substantially", "approximately", etc. may be used herein in the sense of "at, close to, or nearly at", or "within 0-5% of..." or "within an acceptable manufacturing tolerance", or a logical combination thereof. As used herein, a component that is "configured to" perform a specified function is capable of performing the specified function without modification, not merely having the potential to perform the specified function after further modification. In other words, the described hardware, when explicitly configured to perform a specified function, is specifically selected, created, implemented, utilized, programmed and / or designed for the purpose of performing the specified function.

[0041] This detailed description and the accompanying drawings or figures are support and description of the present teaching, but the scope of the present teaching is limited only by the claims. Although some best modes and other embodiments for implementing the present teaching have been described in detail, there are various alternative designs and embodiments to practice the present teaching defined in the appended claims. In addition, the present disclosure explicitly includes combinations and sub-combinations of the elements and features presented above and below.

[0042] For purposes of this disclosure, the terms "granularity" and "fine granularity" generally refer to the number of data bits that characterize a time window and represent a corresponding amount of data of similar length. For a given amount of data, the finer the granularity, the more detailed the time window, and / or the more detailed the changes to the adaptive time window. The data may represent, for example, acoustic noise generated by a road. The time window may then represent the resolution of the acoustic noise, such as the number of times the noise pattern changes.

[0043] The subject matter herein describes a technique for dynamic noise pattern learning for speech applications. In the case of a vehicle, the disclosure describes the use of static and dynamic in-vehicle input signals and information transmitted from multiple sources to create an in-vehicle noise pattern learning model. The learning model can then be used to improve the performance of different speech applications using noise suppression and echo cancellation. Exemplary such applications include virtual assistants, hands-free calling, and in-vehicle communications. The multiple input sources of the learning model may include vehicle information in a calibration database, climate control signals from automotive instruments, vehicle control signals from a vehicle control module, climate control signals from an infotainment radio, and road conditions from a navigation-global positioning system (GPS)-and related navigation-based data.

[0044] On the one hand, a variety of different factors are considered to accurately and dynamically identify the type of noise (one or more) present in the vehicle environment, and in particular to accurately and dynamically identify the type of noise that will adversely affect the quality of voice transmission within a small change (or finer granularity) adaptive time period in a voice-based application. Such factors can be classified into vehicle data and environmental data. Vehicle data can cover a wide variety of data types. Some non-limiting examples of vehicle data include acoustic data (e.g., audio captured using an in-car microphone, which is positioned in the vehicle compartment together with its corresponding input), speed, engine revolutions per minute (RPM), engine temperature, the presence or absence of a turn signal, whether one or more windows are raised or lowered, whether the sunroof is opened, the honking of the vehicle horn, and a large number of and various sounds that the vehicle can independently emit. An example of the latter category is whether the muffler fully suppresses the sound from the engine in a combustion-based vehicle. Other examples include noise generated by one or more circuits or mechanical devices, which has sufficient magnitude to resonate acoustically through the compartment and enter the affected source.

[0045] In addition to vehicle data, as noted, environmental data may also exist and may adversely affect the subject speech application. Environmental data may include, for example, weather data (such as more likely events (such as rain or thunderstorms) to less likely events (such as tornadoes, earthquakes, and hurricanes), and other weather-based criteria), road segment roughness, road interruption scores, yelling pedestrians or cyclists, motorcycles sharing the road, etc.

[0046] Aspects of the present disclosure include a system having a processor, a memory for executing code, and application-based elements for running on the processor, such as an application programming interface (API). For the purposes of this disclosure, a "processor" is interpreted as including one or more processors. If more than one processor is involved, they can but need not be identical in structure. They can run the same or different executable codes and instruction sets. They can adopt the same architecture, or their architectures can be different. Non-exhaustive examples include a multiple instruction multiple data (MIMD) processor, or a single instruction multiple data (SIMD) processor. The processor may also be a processor based on a complex instruction set computer (CISC), or a processor based on a reduced instruction set computer (RISC), or some combination thereof. The processor may be partially or specifically customized for a vehicle.

[0047] The processor may be implemented in software, hardware, or a combination thereof. For example, the processor may include a digital signal processor (DSP) for executing commands in hardware. In some embodiments, the controller and memory may be implemented in the vehicle as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a system on chip (SoC). Physically, the processor may be implemented in one or more electronic control units (ECUs) of the vehicle. The ECU may then be networked or wired together under the coordination of a controller unit such as a controller area network (CAN), or it may simply be hardwired together. In various embodiments, machine learning in the vehicle may be implemented using one or more processor designs that use cloud computing to dynamically exchange data. Using a cloud platform in this context is beneficial because vehicle products are becoming increasingly implemented via software. This trend is based in part on meeting consumers' increased demand for establishing new vehicle features and enabling new functions. This trend also enables manufacturers to design and build new platforms by developing code that quickly and efficiently enhances the capabilities of existing hardware. Cloud computing environments can further reduce storage costs and increase the speed at which manufacturers make new designs.

[0048] It should be noted that, although using the cloud is a method of implementing the principles of the present disclosure, the principles do not need to be limited to the cloud, and more traditional data-centric systems can be used in some embodiments. For example, without adopting a cloud platform, two or more cars can be networked to divide their processing and data sharing resources. In addition, as mentioned above, the processing and data caching capabilities required to perform the principles of the present disclosure can be independent of a given vehicle. One or more general or special processors or arrays thereof can be used in the vehicle to implement machine learning. Various types of computer-readable media such as flash memory, hard disk, etc. can be loaded onto high-speed memory. This can include, for example, code configured to be executed by a processor for processing incoming noise signals for use in dynamic machine learning for voice applications. The subject vehicle can communicate with one of a series of related nodes via a cellular network (such as 3G / 4G / 5G) or another suitable network (such as a metropolitan area network (MAN), a wide area network (WAN), a local area network (LAN), Bluetooth or another network protocol).

[0049] In the case where an embodiment employs a network-based approach such that the cloud is used as a computing platform, various features and functions of the vehicle can be stored in local memory during the machine learning session. The cloud as used herein may refer to a cloud platform for noise reduction and echo cancellation that is specifically customized for machine learning applications of the vehicle. Data can be uploaded to the cloud, where it can be stored until it is needed again. Likewise, the data can be downloaded from the cloud when it is needed as data input or executable code.

[0050] During an active phone call in the vehicle, a dedicated set of signal processing libraries may be used. For example, when the driver is using a Bluetooth device or Android auto and requests a connection for a phone call, a separate library may be used. When noise suppression and echo cancellation techniques are used, the outgoing signal is typically defined during an active phone call before the signal goes to the cloud. To maintain good call performance, this means optimizing the signal by suppressing unwanted noise as well as eliminating echoes. When the buffer is turned on and the speaker starts speaking, if the ensuing noise is pleasant to the human ear, this means it will be ideal for the voice detection engine. Conversely, if there is a lot of static and echo, the speech engine may have difficulty recognizing the words being articulated into the microphone.

[0051] Thus, aspects of the present disclosure take into account a potentially large number of data points that may cause even moderate vehicle noise to affect voice applications. To this end, one benefit of maintaining optimal sound is that the application should be aware of most of the environmental factors that affect the vehicle acoustics at a given moment. The more the application's noise suppression platform understands the factors that adversely affect the acoustics in the vehicle, the smarter the solution platform can eradicate these anomalies during the voice session. Because the voice suppression application knows the speed of the vehicle at a given moment, the application can calculate how much noise in sound pressure level (SPL) the vehicle will produce. Furthermore, using inputs such as vehicle speed, the engine revolutions per minute (RPM) can be calculated. The application is designed to include acoustic effects in the vehicle, such as the presence of turn signal windows. As described below with reference to Figure 5 As described, information related to these types of acoustic artifacts may be available on the vehicle bus 524 for storage and processing. For example, the information may be in the form of an extensible markup language (XML), a JavaScript object representation (JSON) format for representing structured and other non-acoustic data using JavaScript object syntax, or a C-structure. The C-structure is a user-defined data type that can be used to group items of potentially different types into a single type. The above formats are exemplary in nature, and other protocols are also possible. In one aspect of the present disclosure, a noise suppression and echo cancellation system estimates the generation of noise and echo using a regression analysis that uses a fine-grained (highly and finely adjustable) adaptive time window. As the data is adaptively filtered, the processor may continue to make corrective adjustments to suppress the data and eliminate the echo.

[0052] Thus, for example, the processor in the ECU evaluates different data types over corresponding time windows to determine their impact on voice applications (if they exist). For example, when the sunroof and left window are opened, the processor can use this information to obtain a better estimate of the signal-to-noise ratio (SNR) in the vehicle cabin. As another example, the processor can use the identified acoustic features to determine whether the ventilation system is turned on or the fan is turned on. Based on this knowledge, the processor can make further determinations, such as whether the air from either system is directed toward or toward the driver's face. This exemplary set of occurrences provides the echo cancellation element with an immediate and ongoing understanding of acoustic artifacts that cause the need for echo cancellation. The fact that the noise environment may suddenly change or deform into different sounds so that the filter constantly adapts to the sounds in the new time window is a property of the present disclosure, which enables dynamic, granular noise suppression and echo cancellation based on adaptive filtering when the data stream is continuously provided and dynamically changes in magnitude over time.

[0053] Different data streams can augment the adaptive filtering to enhance the overall effect. For example, in some embodiments, sensors or microphones may be embedded in the rear portion of the vehicle for the purpose of determining the type of terrain on which the vehicle is traveling. The noise generated by the road type may have a significant ongoing impact on the use of voice applications in the vehicle. In addition to this technique, the vehicle may also obtain road surface data from a global positioning system (GPS) by considering the geographical relationship between two or more points. In these latter embodiments, the road information may coexist with road information stored in computer-readable media in different modules of the vehicle (such as a list of possible roads that the vehicle may encounter while on the way to the destination). These processes may be similar to auto-correlation metrics, where the system can change coefficients and metrics to reconcile different input characteristics (e.g., terrain collected from different sources such as a map program and a GPS). Noise can be represented as a probability distribution, also known as a parameter representation of a distribution. A "Gaussian" distribution is an example of a probability distribution, which is displayed as including a mean μ and a square of the standard deviation or σ. 2 The bell curve.

[0054] The processor may employ regression analysis to model the relationship between a dependent variable and one or more independent variables. Regression analysis may be used to determine the strength of the correlation between variables for modeling future relationships between them. More fundamentally, regression analysis may be used in machine learning to predict future outcomes as well as analyze past outcomes. In the context of machine learning, regression may allow the system to predict continuous outcomes (such as "y" values ​​on a vertical axis) based on the values ​​of an "x" predictor variable.

[0055] Processors can often use embeddings to determine or identify relevant variables for a particular regression analysis. Embedding various data types may also be referred to as "embeddings" in this article. Various application programming interfaces ("APIs") provided for these analyses can allow embeddings to be used to measure the relatedness of text strings and other types of data strings. Embeddings are vectors (lists) of numbers. The distance between two vectors measures their relationship. In the example above, the processor can use a regression analysis using existing data to identify the nature of the road being driven on.

[0056] As further information about the vehicle's operating conditions, such as the speed of the vehicle, the state of the HVAC system including the blower speed, the number of occupants in the vehicle cabin, and other information that can be easily collected from the vehicle communication network can be utilized to build a scenario. This information is used to build a regression model. These regression models are honed using a sufficient amount of training data. This is essentially the tokenization of the input feature vectors, which are then converted into relevant vector embeddings using a special neural network architecture (also known as a "transformer" model). These embeddings will lead to more accurate estimates of the statistical system signatures of vehicle noise, and will therefore contribute to more effective fine-grained noise elimination. When the data is finally converted into a relevant vector embedding, it can be said that the data is "embedded" for the input data in question (acoustic noise, etc.) to form an embedding so that the term can be used in its verb or noun form.

[0057] Figure 1 1 is a conceptual diagram 100 describing a technique for dynamic in-vehicle noise pattern learning according to an aspect of the present disclosure. The technique describes a solution for dynamic noise pattern learning that is implemented partially in the vehicle and, in one embodiment, partially in the cloud for storing a dynamic library. As noted, the cloud need not be used in the technique, as other storage media may instead be used locally in the vehicle or another networked location.

[0058] Continue to refer Figure 1, data relevant to the application in question may be derived from a plurality of sources. Acoustic data, such as cabin noise and user's voice, may be derived from one or more microphones 102 distributed as input at strategic locations in the vehicle cabin and channeled to a source of multimodal data embedding via corresponding vehicle lines 104 (one of which is shown). Examples may include vehicle data 120 from a vehicle 122, including speed, engine revolutions per minute (RPM), engine temperature, turn signals, window positioning (up vs. down), sunroof positioning (open vs. closed), horn, etc. As another example, an HVAC or air conditioning system 124 may control various other relevant acoustic phenomena, such as ventilation type or fan speed 126. In addition, noises specific to vehicle brands and engine types 132 may be regularly stored in a calibration database 128 and retrieved when needed. External environmental data may be provided, including road segment roughness and road interruption scores 130.

[0059] Embedding in the context of machine learning is the mapping or conversion of high-dimensional data into low-dimensional data, which is usually in the form of a vector with properties related to the embedded data. In other words, data embedding is usually a dense numerical representation of the data, which is expressed as a vector. The resulting set of vectors quantifies the similarity between categories.

[0060] The data in the multimodal data embedding block 106 learns the representation or converts the high-dimensional multimodal data (e.g., acoustic data, vehicle signals, fan speed, calibration database, etc.) into a single low-dimensional vector space. This allows for more efficient and effective machine learning tasks such as classification, clustering, and retrieval. Different methods can be used for multimodal data embedding, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory (LSTM), and geometric machine learning techniques.

[0061] In addition, if Figure 2 As further described in , the embedding in the multimodal data embedding block 106 can be used to represent the applicable noise type as a parameter vector. For system identification, the noise can be represented as a vector of parameters.

[0062] Thus, in multimodal data embedding, collective data from a large number of influential data sources can be embedded. The multimodal nature of data embedding can significantly increase the accuracy of machine learning by providing a more realistic and inclusive set of factors that would otherwise add noise and adversely affect the use of voice applications.

[0063] Once the initial estimates of the mean and variance for a particular type of noise at logic block 108 are determined to statistically represent the noise as a probability density function - the processor can use the accumulated data in adaptive time windows 110, and the processor can perform a time window-based analysis at logic block 12 to gauge the statistical characteristics of the noise. Once the processor determines the classification at 12, noise suppression and echo cancellation can be performed in a more accurate and efficient manner. For example, the newly classified data in 12 can be used in the application in question to enhance the user's voice at the front end of the vehicle (114), it can be used for active noise cancellation (116), or it can be used to eliminate or augment acoustic features in the context of other voice applications (118).

[0064] Figure 1 Two significant advantages of the system in over existing approaches include: First, the types of relevant data are significantly increased into a sophisticated multimodal embedding (106). "Sophisticated" refers to the fact that multiple noise sources are considered in the machine learning model, which in turn increases the accuracy and precision of the results. Second, the data can be used to enhance a wide range of applications, thereby using more realistic models to achieve noise suppression and cancellation where warranted. Third, the large amount of acoustic and non-acoustic data that can be processed means that more user applications can benefit from the initial data analysis.

[0065] Figure 2 2 is a conceptual diagram illustrating a technique 200 for dynamic noise pattern learning in a vehicle 222 using a sample latent vector 213 according to an aspect of the present disclosure. Figure 1 conveys high-level diagrams of system components and descriptions of various embodiments, but Figure 2 Provides more granular details of parameter estimation according to an embodiment. For example, Figure 2 Added Figure 1 The embedded cascade 207 embodiment is not explicitly shown. Figure 2 The relevant data in the context of environmental data 201 and vehicle data 205 is shown. As noted, factors required to accurately and dynamically identify noise types in a vehicle environment include vehicle data 205. In short, in Figure 2 In the process, the processor scans multiple data parameters to define the distribution of the data as mean μ and variance σ 2. Examples of vehicle data include audio captured by an in-vehicle microphone, speed, engine RPM, engine temperature, whether a turn signal is active, the up / down nature of a window, the open / closed nature of a sunroof, honking, etc. In contrast, examples of environmental data 201 may include weather, road section roughness, and a road interruption score. For example, data embedding as shown in logic blocks 203 and 209 is a technique for representing data as points in an n-dimensional space so that similar data points are clustered together. As noted, embedding is generally a low-dimensional representation of high-dimensional data. Microphone 202 may be positioned in the cabin to produce acoustic data that is combined or reconciled with the rest of the acoustic data from sources outside the cabin.

[0066] Still reference Figure 2 The idea of ​​non-acoustic data embedding 203 is derived from supervised or unsupervised representation learning techniques. This process can also be generally referred to as parameter representation distribution. Non-acoustic data includes not only structured text data, but also time series data, such as vehicle dynamics data (vehicle data that changes over time). Figure 2 An embedding concatenation 207 between the acoustic data embedding 209 and the non-acoustic data embedding 203 is shown. Embedding concatenation involves combining the generated embeddings into a single embedding. This can be done by simply concatenating the vectors representing the embeddings, or by using more sophisticated methods such as weighted averaging. Once the non-acoustic data embedding 203 is concatenated with the acoustic data embedding 209, the resulting feature vector is passed to a hidden fully connected layer 211 to estimate the mean μ and variance σ 2 The processor may use a deep learning mechanism in AI to determine the distribution. The distribution is represented by a sample latent vector 213. That is, the sample latent vector 213 is a calculation of the initial value of the distribution. In this example, the sample latent vector 213 can be considered to have two values ​​(mean μ and variance σ 2 ) area. The mean μ and variance σ of the sample latent vector 213 2 The noise type may then be identified by the processor over the adaptive time window 215 as shown in logic block 217. For example, the noise identification uses the mean μ and variance σ during the applicable adaptive time window 215. 2 The invention provides a method for predicting noise in a vehicle by using a fine-grained adaptive time window (such as herein) whose width can be changed rapidly over time in small or large increments. The method is contrasted with existing implementations in vehicle noise suppression, which often use fixed time window sizes. However, these existing prediction methods are inaccurate because, unlike the adaptive time windows disclosed herein, the fixed time windows are not adjusted to reflect more recent trends in the data. Examples of where adaptive time windows are needed include changing road conditions and many other situations, including generally changing trends in the input data.

[0067] In the example of changing environment or road noise, the time change may be very fast. Here, the adaptive time window 215 allows the system to use the first timing window to detect what type of noise / environmental noise is present. The processor can use this first timing window to collect noise samples and then estimate the noise type identification based on the mean and variance, as in logic block 217. The smaller the time window, the more accurate the noise prediction. If the processor is computationally efficient enough, the processor can select time samples of almost arbitrarily small lengths, which increases the accuracy of the distribution and therefore increases the effectiveness of noise cancellation. Noise estimation is a statistical multivariate problem in which the system is incorporating multivariate inputs. At the same time, the physical conditions that produce these variables are changing. As an example, in a vehicle, the turn signal may suddenly be activated, a person may cough, the wind may start blowing, the window wipers may be turned on, raindrops may fall on the windshield, etc. Each of these events can be considered and treated as a distribution with an individual adaptive time window.

[0068] Still reference Figure 2 The identified noise type may be later used as an enabler in different in-vehicle speech applications 219 that require information about the noise type, such as, for example, a voice enhancement module and active noise cancellation.

[0069] As further described herein, the principles of the present disclosure enable dynamic, fine-grained machine learning of speech patterns while a vehicle is being driven. The real-time noise patterns can then be applied to different speech applications. As currently performed, mimicking road noise and other acoustic and non-acoustic artifacts with a "one-size-fits-all" approximation often does not produce accurate results. Figure 2 As shown in FIG. 1 , geometric deep learning can be used to embed both acoustic data and non-acoustic data (logic blocks 203 , 209 ) to form a latent vector comprising the mean μ and variance σ using a correlation time window 215 2 to account for the dynamic or changing nature of the input data. That is, once the processor performs an estimate of the mean μ and variance σ 2 To statistically represent the noise as a probability function, the processor can then use time window based analysis 215 to further measure the statistical characteristics of the noise at different time periods, such as performed in logic block 217.

[0070] The processor uses the mean μ and variance σ during the applicable time window 2After identifying the noise type based on the statistical characteristics of the noise (logic block 217), the determined statistical noise data can then be used in in-vehicle voice applications such as, for example, voice enhancement, active noise cancellation, etc. and similar techniques for dynamically suppressing noise and eliminating echoes, as in logic block 219. These applications can further be used as input to other voice-based applications to improve their overall performance.

[0071] It should be noted that the non-acoustic data embedding 203, the acoustic data embedding 209, and the embedding cascade 207 produce a fully connected layer 211 that enables the sample latent vector 213 to incorporate relevant data types required to accurately calculate the distribution of noise for correction and enhancement purposes. Therefore, existing methods that fail to account for various types of data are less accurate or inaccurate.

[0072] In another aspect of the present disclosure, a method for estimating a mean μ and a variance σ is disclosed. 2 The invention relates to a technique for automatically determining four different noise types based on the interquartile range (IQR) of the estimated mean and variance as a measure of dispersion. IQR is a measure of statistical dispersion. Dispersion is in turn a measure of the spread of data distribution. IQR dispersion is defined as the difference between the 75th percentile and the 25th percentile of the data. In the example of the Gaussian waveform discussed above, two such waveforms may share the same mean, but may be distributed over a larger or smaller time window. In the case where the waveform has a small amount of dispersion, the waveform tends to peak at a certain point and rapidly drops to zero or a negligible value on either side of the peak. In contrast, a waveform with a larger dispersion means that the data is spread over a larger time window. IQR is a method of describing the magnitude of this dispersion using four different quadrants, as discussed further below. Beneficially, the processor can implement fine-grained noise type identification based on the calculated IQR and the adaptive time window. Noise types, as described further below, include stationary mean and stationary variance; non-stationary mean and stationary variance; stationary mean and non-stationary variance; and non-stationary mean and non-stationary variance.

[0073] Figure 3 3 is a flow chart describing a technique for mean and variance based noise type identification using IQR specification in accordance with an aspect of the present disclosure. Figure 3 A technique for estimating the mean and variance of a set of input data and then determining the noise type using an IQR technique is shown. For purposes of this disclosure, the noise type result may be synonymous with the noise type when using an IQR model. Figure 3The interquartile ratio and variance σ of the mean μ231a within the target threshold are shown. 2 213b is the quartile ratio. In some cases, the mean μ231a or the variance σ 2 Either one of can be stationary by itself, or both values ​​can be stationary. Figure 3 Shown in Figure 2 and 3 The adaptive time window 315 used in 2 213b, in order to calculate the IQR as the mean μ 231a and variance σ within a calibrable (e.g., fine-grained and dynamically adaptable) time window 2 231b's dispersion metric, as in logic block 317.

[0074] In logic block 319, the IQR (IQR μ ) and based on σ 2 IQR(IQR σ2 ) values ​​are compared to four possible thresholds, which are represented by noise type results 330, 340, 350, and 360. In result 300, IQR μ and IQR σ2 Both are determined to be less than the corresponding target thresholds μth and σth, resulting in a curve with a stationary mean and a stationary variance, as shown in the curve graph corresponding to result 330. In result 340, only IQR μ exceeds its target threshold μth, resulting in a curve with a non-stationary mean and a stationary variance in this example. σ2 exceeds its target threshold σth, resulting in a curve with a stationary mean but a non-stationary variance, such as Figure 3 As shown in the graph corresponding to result 350. Referring to result 360, IQR μ and IQR σ2 Both are larger than their corresponding thresholds μth and σth and no plot is obtained. Depending on the quartiles or results of different comparisons, various levels of predictability can be determined.

[0075] in short, Figure 3 The embodiment of uses four different types of noise to estimate the mean and variance. As noted, based on the adaptive time window, the noise feature changes. As an example, if the feature has a constant mean, then the constant variance result is a straightforward scenario. However, when the standard deviation (and therefore variance) is changing or the mean is changing, those two modes can be their own distributions, and both modes should be accounted for, as illustrated in results 340 and 350, respectively.

[0076] Using the estimated mean and variance and comparing whether they exceed thresholds in specified quartiles, noise types with suitable characteristics of mean and standard deviation and accurate distribution can be identified.

[0077] Figure 4 4 is a conceptual diagram 400 describing a technique for determining an adaptive time window according to an aspect of the present disclosure. The IRQ range is a measure of where most values ​​in a data graph are located. The processor takes the following actions in each quarter of the IRQ: Figure 3 The mean 413a and variance 413b calculated by the embodiment in the embodiment are used to generate an inverse logistic function, reverse sigmoid, or Z-curve function ("function") 405. The IQR divides the data into quarters, where the lowest quarter of the IQR includes the smallest quarter of the values ​​in the input data, and the upper quarter includes the highest quarter of the values, and where the two middle quarters include the middle half of the data.

[0078] In function 405, the y-axis represents IQRμ and IQRσ obtained from the IRQ analysis in blocks 420a and 420b. 2 The sum of the values. Each of boxes 420a and 420b represents an adaptive time window. Box 420a takes as input the input mean μ 413, where the spread of the mean is defined by the lower half 406 and the upper half 408. The lower half 406 includes the means μ1-μ3, and the upper half 408 includes the means μ5-μ7. The lower quarter 410 corresponds to Q1 of μ2, and the upper quarter 420 corresponds to Q3 of μ6. Therefore, according to equation 440.2, IQR μ = Q3–Q1. Reference where the median σ is entered 2 413, box 420b, similar analysis yields IQR σ2 =Q3–Q1, where the lower half 411 includes σ 2 1-σ 2 3, and the upper portion 413 includes σ 2 5-σ 2 7. In this case, the lower quarter 461 corresponds to the value σ 2 2, and the upper quarter 462 corresponds to the value σ 2 6. Therefore, the sum y from blocks 420a and 420b = IQR μ and IQR σ2 represents the y-axis of function 405. The x-axis of function 405 represents the applicable time window. The time window can be calculated using the following formula: Where W is the predefined window value, and y is the value calculated above for the y-axis. For clarity, it should be noted that in box 420, the median corresponds to the mean of Q2 = μ4. Similarly, in box 420b, the median corresponds to Q2 = σ 2 The median value of 4.

[0079] Summarize Figure 4 , the processor retrieves the Figure 3 The mean and variance are calculated. Function 405 is consistent with the rate of change or first derivative of the data. Therefore, real-time input values ​​are input and non-negative derivative values ​​should be generated at each inflection point. This technique is consistent with the sigmoid or cost function. Because one purpose is to understand the distribution pattern, the sigmoid function can be used to define the distribution of the mean or the distribution of the standard deviation in the case where both the distribution of the mean or the distribution of the standard deviation are not stationary.

[0080] Figure 5 is a block diagram of a system 500 for implementing statistical noise analysis for use in speech-based applications. The systems and apparatus of the present disclosure may be implemented in a vehicle 123. In various configurations, the apparatus may operate as a system including a vehicle 123 and a remote cloud when a cloud is used to provide data and code to the vehicle via an appropriate network. In some embodiments, the system may be implemented on more than one vehicle.

[0081] Within the vehicle 123, it is noted that the term processor 504 itself may refer to one or more processors within one or more computing devices. Such computing devices may include, for example, the motherboard of a computer, or an electronic control unit (ECU) or automotive microcontroller unit (MCU) within the vehicle 123. Depending on the architecture of the vehicle and its electronic configuration, the ECU and / or MCU may include a processor 504 and a memory 508 (such as a cache memory, a dynamic random access memory (DRAM), a static random access memory (SRAM), etc.). In other cases, the ECU (or MCU) is maintained separately from these components, and where applicable, the ECU or MCU may be hardwired to these components.

[0082] The mass storage device 512 may include a disk drive, a solid-state disk drive, or another form of non-volatile memory, and may be used to retrieve data from the cloud via the antenna 522 over a suitable network 548 and under the control of the processor 504. Referring back to the processor 504, the component may be broadly interpreted as including a variety of different processing devices. For example, the processor 504 may be a multi-purpose microprocessor (or multiple multi-purpose microprocessors). In other embodiments, the processor 504 may be a digital signal processor (DSP), an application specific integrated circuit (ASIC) (which may be embedded within other devices shown in the figure), a field programmable gate array (FPGA), a system on a chip (SoC), or other processors or arrays thereof. Different processors within the processor 504 may perform specific functions of the technology disclosed herein.

[0083] Referring again to the memory 508, the memory 508 may be a DRAM or another faster memory on which executable code corresponding to an active application may be loaded, or useful data may be loaded. The portable storage device 516 may include a removable storage device, such as a flash memory or a USB drive. The portable storage device 516 may include firmware or a device for uploading firmware to the processor 504. The output display 506 may include (one or more) display panels on the dashboard of the vehicle, such as one or more I / O touch pads representing an infotainment system or a navigation system. The output display 506 may enable a user to control settings corresponding to the technology herein. In addition, the vehicle 123 may include one or more in-cabin microphones 510. If multiple microphones 510 are used, they may be positioned in various parts of the interior of the vehicle so that the acoustic input of the microphones can capture the sounds emitted by the vehicle 123 or its passengers as efficiently as possible. A user interface 514 may also be included. The user interface 514 may represent a corresponding display embedded as part of the output display 506, or they may include switches, buttons, and other control features built into the dashboard so that the user (or a professional car dealer) can dock or interact with the algorithm executed in the processor 504 and perform other functions related to the system. Various peripheral equipment 518 may also be used in conjunction with the system, such as Bluetooth devices, radios, or peripheral controls for performing various functions. Examples may include increasing the granularity of machine learning, or adjusting other controls related to similar functions, although these types of controls may also be implemented using the network 548, antenna 522, and transceiver 520 under the control of the processor 504 (such as in a firmware update or in response to an ECU network request). Alternatively, these functions may be controlled by the user interface 514. In addition, insofar as the device including one or more identified elements in the vehicle 123 is added to the vehicle from the outside (e.g., during the process of upgrading an older vehicle, etc.), docking hardware 527 may be provided, such as under the dashboard or in another accessible area, so that the vehicle 123 can be modified to perform one or more techniques described herein.

[0084] The benefits of the technology described herein are extensive. Unlike in existing implementations, multimodal data embedding and embedding cascade techniques can significantly improve the accuracy of the machine learning process by providing data related to a large number of phenomena (acoustic and non-acoustic phenomena, vehicle phenomena and environmental phenomena), which would otherwise adversely affect the performance of voice applications. Hidden parameter estimation provides efficiency and enables data in various forms to be digitally converted for subsequent efficient processing. In addition, the noise type identification based on IQR depends on whether the mean and variance factors characterizing the noise distribution are active or stable to consider a substantially complete set of scenarios. The technology can also be performed using real-time vehicle control information, thereby allowing full benefits even when the vehicle is in motion. The technology can also be widely applicable to different voice applications, including but not limited to virtual assistants, in-car communications (e.g., allowing users to convey information to other users in the vehicle more clearly and with less noise, which may be particularly useful in larger vehicles with multiple rows of seats or in trucks) and hands-free calling.

[0085] In addition, the adaptive nature of the time window enables the machine learning process to have very high temporal resolution, resulting in very accurate calculations of expected noise and interference. In providing this higher resolution, the technology can improve the user's experience with voice applications and notice better visible changes in performance. The IQR nature of the estimated mean and variance enables identification of different noise patterns / types.

[0086] It should also be noted that the identified noise types need not only be used for different in-car voice applications, such as active noise cancellation and voice enhancement. The identified noise types can also be used for different out-of-car voice applications, such as denoising, de-reverberation to obtain better performance of external voice assistance, etc.

[0087] This detailed description and the accompanying drawings or figures are support and description of the present teaching, but the scope of the present teaching is limited only by the claims. Although some best modes and other embodiments for implementing the present teaching have been described in detail, there are various alternative designs and embodiments to practice the present teaching defined in the appended claims. In addition, the present disclosure explicitly includes combinations and sub-combinations of the elements and features presented above and below.

Claims

1. A method for in-vehicle noise pattern learning for speech applications, the method comprising: Embedding multimodal data from environmental data and vehicle data; acoustic data embedded in data captured by microphones in the cabin of the vehicle; concatenating the embeddings to form a latent vector representing the embeddings; Using an adaptive time window to estimate the mean and variance of the latent vector; as well as The noise type is identified using the mean and variance of the latent vector, and the noise type identification is fine-grained via the adaptive time window to accurately simulate vehicle noise.

2. The method according to claim 1, wherein: The environmental data includes weather data, road roughness, road disruption score, or location-based application programming interface (API), and The vehicle data includes speed, engine revolutions per minute (RPM), engine temperature, turn signal on or off, windows up or down, sunroof open or closed, or honking a horn in the vehicle.

3. The method of claim 1 , wherein identifying the noise type comprises calculating the interquartile range (IQR) of the mean and variance of the latent vector as a measure of dispersion of both the mean and variance within a calibrable time window. 4 . The method of claim 3 , further comprising generating a fine-grained noise type identification based on the calculated IQR and the adaptive time window.

5. The method of claim 4, wherein the fine-grained noise type identification comprises one or more of: a stationary mean and a stationary variance; a non-stationary mean and a stationary variance; a stationary mean and a non-stationary variance; or a non-stationary mean and a non-stationary variance.

6. The method of claim 4, wherein the adaptive time window is based on one or more of: an inverse logistic function, an inverse sigmoid function, or a combined mean IQR and variance IQR. 7 . The method of claim 1 , wherein the identified noise type is for in-vehicle speech applications including at least one of active noise cancellation or voice enhancement.

8. A vehicle for in-cabin noise pattern learning for speech applications, comprising: a vehicle body, including a carriage arranged therein; Memory; a processor coupled to the memory and configured to: Embedding multimodal data from environmental data and vehicle data; embedding acoustic data from data captured by microphones in the vehicle cabin; concatenating the embeddings to form a latent vector representing the embeddings; estimating the mean and variance of the latent vector via an adaptive time window; as well as The noise type is identified using the mean and variance of the latent vector, where the noise type identification is fine-grained to accurately model vehicle noise.

9. The vehicle of claim 8, wherein: The environmental data includes weather data, road roughness, road disruption score, or location-based application programming interface (API), and The vehicle data includes speed, engine revolutions per minute (RPM), engine temperature, turn signal on or off, windows up or down, sunroof open or closed, or honking a horn in the vehicle.

10. The vehicle of claim 8, wherein the processor is configured to identify the noise type using the interquartile range (IQR) of the mean and variance of the latent vector as a measure of dispersion of both the mean and variance within a calibratable time window.

Images