Procedures for voice assistance

Abstract

A method for voice assistance includes receiving audio data by a vehicle's control unit. The audio data indicates a voice command uttered in natural language by a vehicle user. The method also includes encoding the audio data into a latent space using a variational autoencoder encoder to generate encoded data. The method also includes receiving contextual data related to the voice command uttered by the vehicle user. The method further includes generating an expression from the encoded data and the contextual data using a variational autoencoder decoder. This expression is representative of the audio data. The method also includes commanding the vehicle's control unit to generate a response based on the expression generated by the variational autoencoder decoder.

Description

[0001] This description relates to voice assistance methods. In particular, this description relates to voice assistance methods in vehicles.

[0002] This introduction generally presents the context of the description. The work of the inventors named herein, to the extent described in this introduction, as well as aspects of the description that are not prior art at the time of the application, are neither expressly nor implicitly acknowledged as prior art against this description.

[0003] DE 10 2023 133 577 A1 describes systems and methods for training an audio processing system to perform speech audio encoding and decoding using neural networks. In particular, the systems and methods comprise receiving an audio sequence, the audio sequence including speech audio, generating pitch data representing the detected pitch within the audio sequence, passing the audio sequence through an audio encoder to generate a vector representation of the audio sequence, generating an encoded vector representation of the audio sequence by a vector quantizer using the vector representation of the audio sequence and a codebook of discrete vectors, and reconstructing the audio sequence using the pitch data and the encoded vector representation of the audio sequence by an audio decoder.

[0004] US 2022 / 0 189 456 A1 describes a model for disentangling linguistic content and speaking style, comprising a content encoder, a style encoder, and a decoder. The content encoder is configured to receive input speech as input and generate a latent representation of the linguistic content for the input speech output. The content encoder is trained to separate speech style information from the latent representation of the linguistic content. The style encoder is configured to receive the input speech as input and generate a latent representation of the speaking style for the input speech as output. The style encoder is trained to separate linguistic content information from the latent representation of the speech style.The decoder is configured to generate output speech based on the latent representation of linguistic content for the input language and the latent representation of speaking style for the same or another input language.

[0005] Tokenization-based sentence generation and context understanding for speech recognition have several limitations and constraints. For example, a homophonic token can introduce ambiguities into the speech recognition process. Therefore, it is desirable to develop methods and systems for voice assistance that do not rely on tokenization.

[0006] The present description describes a voice assistance method. The inventive method comprises receiving audio data by means of a vehicle control system of a vehicle. The audio data indicates a voice command uttered in natural language by a user of the vehicle. The method also comprises encoding the audio data in a latent space using an encoder of a variational autoencoder to generate encoded data. The method also comprises receiving contextual data relating to the voice command uttered by the vehicle user. The method also comprises generating an expression from the encoded data and the contextual data using a decoder of the variational autoencoder. The expression is representative of the audio data.The method also includes commanding the vehicle, using the vehicle controller, to generate a response based on the expression generated by the decoder of the variational autoencoder. The method described in this paragraph improves speech recognition technology and vehicle technology by recognizing natural speech uttered by a user without relying on lexical tokenization, which is prone to errors when the user utters language-specific traditional expressions (e.g., idioms, poems, slang, etc.). Because the method described in this section does not use lexical tokenization, this method improves natural language recognition by voice assistance applications, thereby improving speech recognition technology and voice assistance applications in vehicles.

[0007] According to one embodiment, the method does not comprise performing a lexical tokenization of the audio data.

[0008] According to a further embodiment, the method comprises reducing background noise from the audio data and detecting a voice in the audio data.

[0009] According to another embodiment, the encoder of the variational autoencoder is a first neural network that maps the audio data into the latent space. The audio data is in an input space. The decoder is a second neural network that maps the encoded data into the input space. The contextual data includes user speech data and data about external factors. The user speech data contains information about the user's tone of voice while uttering the voice command.

[0010] According to another embodiment, the external factor data includes the traffic condition around the vehicle when the user uttered the voice command, a date when the user uttered the voice command, and a time when the user uttered the voice command.

[0011] According to a further embodiment, the contextual data also includes data from the conversation history. The conversation history data includes information about the conversation history of the user who uttered the voice command.

[0012] According to a further embodiment, the contextual data serves as an input for the second neural network.

[0013] According to another embodiment, the response is generated based on a plurality of conditions.

[0014] According to another embodiment, the plurality of conditions includes reaction time and sentence length.

[0015] According to a further embodiment, the response comprises controlling an actuator of the vehicle based on the response.

[0016] In one application, the method according to the invention and the embodiments is implemented in a voice assistance system. The voice assistance system comprises a user interface with a microphone. The microphone is configured to capture a voice command uttered by a user of a vehicle. The voice assistance system further comprises a plurality of sensors. Each of the plurality of sensors is configured to collect contextual data. The voice assistance system further comprises a vehicle controller that communicates with the user interface and the plurality of sensors. The vehicle controller is programmed to execute the method described above.

[0017] The present description also describes, in one application, a tangible, non-transitory, machine-readable medium comprising machine-readable instructions that, when executed by a processor, cause the processor to perform the above-described method according to the invention and its embodiments.

[0018] The above features and advantages, as well as other features and advantages of the presently described system and method, are readily apparent from the detailed description, including the claims, and exemplary embodiments taken in conjunction with the accompanying drawings.

[0019] This description will be better understood from the detailed description and the accompanying drawings, in which: Fig. 1 is a schematic representation of a vehicle with a voice assistance system. Fig. 2 is a flowchart of a voice assistance process.

[0020] Reference will now be made in detail to several examples of the description illustrated in the accompanying drawings. Wherever possible, the same or similar reference numbers are used in the drawings and the description to refer to the same or similar parts or steps.

[0021] In relation to Fig. 1, a vehicle 10 includes a voice assistance system 39. The vehicle 10 generally includes a body 12 and a plurality of wheels 14 connected to the body 12. The vehicle 10 may be an autonomous vehicle. In the illustrated embodiment, the vehicle 10 is depicted as a sedan, but it should be understood that other vehicles, including trucks, coupes, sport utility vehicles (SUVs), boats, aircraft, recreational vehicles (RVs), etc., may be used.

[0022] The vehicle 10 also includes one or more sensors 24 connected to the body 12. The sensors 24 sense observable conditions of the external environment and / or the internal environment of the vehicle 10. As non-limiting examples, the sensors 24 may include one or more cameras, one or more light detection and ranging (LIDAR) sensors, one or more proximity sensors, one or more ultrasonic sensors, one or more thermal imaging sensors, global positioning system (GPS) transceivers, and / or other sensors. Each sensor 24 is configured to generate a signal indicative of the sensed observable conditions (i.e., sensor data) of the external environment and / or the internal environment of the vehicle 10. The signal is indicative of the sensor data sensed by the sensors 24.

[0023] The vehicle 10 includes a vehicle controller 34 that communicates with the sensors 24. The vehicle controller 34 includes at least one vehicle processor 44 and a non-transitory computer-readable vehicle storage device or medium 46. The vehicle processor 44 may be a custom or off-the-shelf processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among a plurality of processors associated with the vehicle controller 34, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally an instruction-executing device.The vehicle-readable storage device or medium 46 may include volatile and non-volatile memory, such as read-only memory (ROM), random-access memory (RAM), and keep-alive memory (KAM). KAM is persistent or non-volatile memory that can be used to store various operational variables while the vehicle processor 44 is powered off.The vehicle's computer-readable storage device or medium 46 may be implemented using a variety of storage devices such as PROMs (programmable read-only memory), EPROMs (electrical PROMs), EEPROMs (electrically erasable PROMs), flash memory, or other electrical, magnetic, optical, or combination storage devices capable of storing data, some of which represent executable instructions used by the vehicle controller 34 in controlling the vehicle 10. In particular, the vehicle controller 34 is programmed to execute the method 100 (. Fig. 2) as described in detail below.

[0024] The instructions may comprise one or more separate programs, each containing an ordered list of executable instructions for implementing logical functions. When executed by the vehicle processor 44, the instructions receive and process signals from sensors, execute logic, calculations, methods, and / or algorithms for automatically controlling the components of the vehicle 10, and generate control signals for automatically controlling the components of the vehicle 10 based on the logic, calculations, methods, and / or algorithms. Although in Fig. 1 illustrates a single vehicle controller 34, embodiments of the vehicle 10 may include a plurality of vehicle controllers 34 that communicate via a suitable communication medium or combination of communication media and that cooperate to process the sensor signals, perform logic, calculations, methods, and / or algorithms, and generate control signals to automatically control functions of the vehicle 10. The vehicle controller 34 is part of the voice assistance system 39.

[0025] The vehicle 10 also includes one or more actuators 26 connected to the vehicle controller 34. The actuators 26 control one or more vehicle functions, such as, but not limited to, the drive system, transmission system, steering system, radio, air conditioning, and braking system of the vehicle 10. In various embodiments, the vehicle functions may also include, but are not limited to, interior and / or exterior functions of the vehicle, such as, but not limited to, doors, trunk, and cabin functions such as ventilation, music, lighting, etc.

[0026] The vehicle 10 has a user interface 23 that communicates with the vehicle controller 34. The user interface 23 may be a touchscreen in the dashboard. The user interface 23 may include, among other things, an alert, such as one or more speakers 48 for emitting an audible tone, haptic feedback in a vehicle seat or other object, one or more displays, one or more microphones 50 (e.g., a microphone array), and / or other devices suitable for conveying a notification to the vehicle user of the vehicle 10. The microphones 50 are configured to capture voice commands from a user of the vehicle 10. The user interface 23 is in electronic communication with the vehicle controller 34 and is configured to receive inputs from a vehicle occupant (e.g., a vehicle driver or a vehicle passenger), such as voice commands.The user interface 23 may, for example, include a touchscreen and / or buttons configured to receive inputs from a person.

[0027] As explained further below, the voice assistance system 39 uses a generative model (e.g., a variational autoencoder). As described above, the encoder of the variational autoencoder encodes the audio data collected by the microphones 50 into a latent space to search for and construct the most relevant information for understanding the natural speech uttered by the user of the vehicle 10. The variational autoencoder also uses contextual data and data about external factors to maximize the accuracy of converting the natural speech from the user's utterance into a digital audio signal. In doing so, the voice assistance system 39 can also understand language-specific traditional expressions (e.g., poems, sayings, slang, or other colloquial language) by taking the external and contextual factors into account.Because the voice assistant system 39 considers external and contextual factors, it is not necessary to translate the source language into standard English and then translate it back into the source language. As a result, the voice assistant system 39 accurately converts the user's utterances into digital audio. Furthermore, the voice assistant system 39 does not rely on lexical tokenization. Tokenization-based sentence generation and context understanding for speech recognition have a number of limitations and restrictions. For example, a homophonic token can introduce ambiguity into the speech recognition process. Consequently, the voice assistant system 39 accurately converts the user's utterances into digital audio.

[0028] Fig.2 is a flowchart of a voice assistance method 100. The method 100 does not use lexical tokenization and begins in block 102. In block 102, the vehicle controller 34 receives audio data captured by the microphones 50. The audio data is indicative of a voice command uttered in natural language by a user of the vehicle 10. Furthermore, in block 102, the vehicle controller 34 reduces the noise from the audio data and detects the user's voice in the audio data. As a non-limiting example, a dual-ended compander noise reduction system may be used to reduce the noise from the audio data. A suitable automatic speech recognition (ASR) system may be used to detect the user's voice in the audio data captured by the microphones 50. In this method 100, the voice assistance system 39 is always listening.Accordingly, noise suppression is performed iteratively in each voice assistant cycle (e.g., every 1 millisecond). Method 100 then continues with block 104.

[0029] In block 104, the vehicle controller 34 encodes the audio data recorded by the microphones 50 into a latent space using an encoder of a variational autoencoder. The audio data thus serves as an input to the encoder. Encoding the audio data generates encoded data, which is compressed data. The audio data is located in an input space. The latent space is a low-dimensional space compared to the input space in order to minimize the effort required to convert the user's speech utterance into a digital waveform. The encoder of the variational autoencoder is a first neural network that maps the sound into the latent space. In other words, the encoder of the variational autoencoder encodes the speech wave signal into latent variables and vectors. This encoder of the variational autoencoder has been previously trained to cover multiple situations.For example, the encoder has been trained to recognize homophonic voices and their meanings. One voice and one voice can have multiple meanings. In method 100, the voice assistance system 39 introduces a third language to perform cross-referencing for better meaning matching. The encoder of the variational autoencoder is adaptive and therefore tracks voices in different dialects through the use of latent variables and vectors. Furthermore, the encoder of the variational autoencoder recognizes language-specific traditional expressions (e.g., idioms, poems, slang, etc.) using the contextual data. The contextual data may include conversation history data about the user. The conversation history data includes information about the conversation history of the user who uttered the voice command.Therefore, the contextual data, in addition to the audio data, can be an input to the first neural network that forms the encoder of the variational autoencoder. In other words, the audio data and the contextual data can be inputs to the first neural network that forms the encoder of the variable autoencoder. The voice assistance system 39 also recognizes and understands sounds that do not contain words but have meaning. The method 100 then continues with block 106.

[0030] In block 106, the vehicle controller 34 understands the context based on the encoded data. In particular, the vehicle controller 34 receives contextual data related to the voice command uttered by the vehicle user, for example, from the sensors 24. The contextual data resides in latent space and has no explicit speech-based meaning. The contextual data includes the user's speech data. The user's speech data, in turn, includes information about the user's tone of voice and mood while uttering the voice command. The contextual data may also include data about external factors. The data about external factors includes, among other things, the traffic conditions surrounding the vehicle 10 when the user uttered the voice command, a date on which the user uttered the voice command, and a time at which the user uttered the voice command.The contextual data may also include conversation history data. The conversation history data includes information about the conversation history of the user who uttered the voice command to facilitate understanding of a speech-based traditional expression. The method 100 then proceeds to block 108.

[0031] In block 108, the vehicle controller 34 generates a printout (in speech waveform format) from the encoded data and the contextual data using a pre-trained decoder of the variational autoencoder. The printout is representative of the audio data captured by the microphones 50. The decoder is a second neural network that maps the encoded data to the input space. The decoder can be trained using the audio data and contextual data from the vehicle 10. The decoder constructs a query of all contexts and, in parallel, generates all possible responses in real time. Some unlikely responses can be filtered out by considering the speech tone, context, and preferences, thereby increasing the accuracy of the decoder. The method 100 then proceeds to block 110.

[0032] In block 110, the vehicle controller 34 commands the vehicle to generate a response based on the expression generated by the variational autoencoder's decoder. The response is subject to certain conditions (e.g., response time, sentence length, tone of voice, etc.) to make the response as natural as possible according to the user's preferences and characteristics. The voice assistance system 39 can proactively generate the responses even if no speech information is detected. The vehicle controller 34 can command the speakers 49 to speak the responses. In addition, the vehicle controller 34 can control one or more actuators 26 to automatically control one or more vehicle functions (e.g., air conditioning, text messaging, music, shopping, anxiety reduction, switching to autopilot mode, etc.) upon user confirmation.The response may also include an explanation for the decision made when the vehicle 10 is in autopilot mode.

[0033] The drawings are simplified and not to scale. For the sake of simplicity and clarity, directional terms such as top, bottom, left, right, above, above, below, below, back, and front may be used in the drawings.

[0034] Embodiments of the present invention are described herein. However, it should be understood that the described embodiments are merely examples, and other embodiments may take various and alternative forms. The drawings are not necessarily to scale; some features may be exaggerated or reduced to show details of particular components. Therefore, specific structural and functional details described herein are not to be considered limiting, but merely as a representative basis for showing one skilled in the art how to variously employ the presently described system and method.As those skilled in the art will appreciate, various features illustrated and described with reference to one of the figures may be combined with features illustrated in one or more other figures to produce embodiments not explicitly illustrated or described. The illustrated feature combinations represent representative embodiments for typical applications. However, various combinations and modifications of the features consistent with the teachings of this description may be desirable for particular applications or implementations.

[0035] Embodiments of the present description may be described herein in terms of functional and / or logical block components and various processing steps. Such block components may be implemented by a variety of hardware, software, and / or firmware components configured to perform the specified functions. For example, an embodiment of the present description may utilize various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, or the like, capable of performing a variety of functions under the control of one or more microprocessors or other control devices.Furthermore, those skilled in the art will recognize that embodiments of the present description may be practiced in connection with a variety of systems and that the systems described herein are merely exemplary embodiments of the present description.

[0036] For the sake of brevity, techniques for signal processing, data fusion, signaling, control, and other functional aspects of the systems (and the individual operating components of the systems) are not described in detail here. Furthermore, the connecting lines depicted in the various figures are intended to represent exemplary functional relationships and / or physical couplings between the various elements. It should be noted that alternative or additional functional relationships or physical connections may be present in an embodiment of the present disclosure.

Claims

[1] Method (100) for voice assistance, comprising: Receiving (102) audio data by means of a vehicle control (34) of a vehicle (10), wherein the audio data indicates a voice command uttered by a user of the vehicle (10) in natural language; encoding (104) the audio data into a latent space using an encoder of a variational autoencoder to generate encoded data; Receiving (106) contextual data relating to the voice command uttered by the user of the vehicle (10); Generating (108), using a decoder of the variational autoencoder, an expression from the encoded data and the contextual data, the expression being representative of the audio data; and Commanding (110), using the vehicle controller (34), the vehicle (10) to generate a response based on the expression generated by the decoder of the variational autoencoder. [2] The method (100) of claim 1, wherein the method (100) does not comprise performing a lexical tokenization of the audio data. [3] The method (100) of claim 2, further comprising: Reducing background noise from the audio data; and detecting a voice in the audio data. [4] The method (100) of claim 3, wherein the encoder of the variational autoencoder is a first neural network that maps the audio data into the latent space, the audio data is in an input space, and the decoder is a second neural network that maps the encoded data into the input space, the contextual data includes user speech data, and the user speech data includes information about a tone of voice of the user while the user utters the voice command. [5] The method (100) of claim 4, wherein the contextual data includes data on external factors, the data on external factors including the traffic condition in the surroundings of the vehicle (10) when the user uttered the voice command, a date on which the user uttered the voice command, and a time at which the user uttered the voice command. [6] The method (100) of claim 5, wherein the contextual data comprises conversation history data, the conversation history data including information about a conversation history of the user who uttered the voice command. [7] The method (100) of claim 6, wherein the contextual data serves as an input to the second neural network. [8] The method (100) of claim 7, wherein the response is generated based on a plurality of conditions. [9] The method (100) of claim 8, wherein the plurality of conditions includes response time and sentence length. [10] The method (100) of claim 9, wherein the response comprises controlling an actuator (26) of the vehicle (10) based on the response.

Images