Biasing the interpretation of verbal utterances received in a vehicular environment

Abstract

Implementations described herein relate to various techniques for biasing interpretations of verbal utterances received in a vehicular environment. For example, implementations may receive a verbal utterance including a query from a user of a vehicle and obtain corresponding vehicle sensor data instances generated by a vehicle sensor(s) of the vehicle. Some implementations may determine to perform a search only on the first corpus data and not on the second corpus data to obtain a given response to the query based on various criteria, which may include at least the query, the corresponding vehicle sensor data instances, corresponding timestamps associated with the corresponding vehicle sensor data instances, and / or corresponding durations the user is associated with the vehicle. Additional or alternative implementations may perform a search on both the first corpus data and the second corpus data to obtain a given response based on the criteria.

Description

[Background technology]

[0001] Humans may engage in human-computer interactions using interactive software applications referred to herein as “automated assistants” (also referred to as “chatbots,” “conversational personal assistants,” “intelligent personal assistants,” “personal voice assistants,” “conversational agents,” etc.). For example, a human (sometimes referred to as a “user” when interacting with an automated assistant) may provide verbal natural language input (i.e., utterances) targeted to the automated assistant, which may be converted to text and then processed, and / or provided as textual (e.g., typed) natural language input targeted to the automated assistant. These verbal utterances and / or typed inputs often include assistant commands targeted to the automated assistant. The automated assistant typically responds to these assistant commands by providing responsive user interface output(s) (e.g., audible and / or visual user interface output), controlling smart device(s), and / or performing other action(s).

[0002] These automated assistants typically rely on a pipeline of components in interpreting and responding to these spoken utterances and / or typed input. For example, an automatic speech recognition (ASR) engine can process audio data corresponding to a user's spoken utterance and generate an ASR output, such as a transcription of the spoken utterance (i.e., a sequence of word(s) and / or a sequence of other token(s)). Additionally, a natural language understanding (NLU) engine can process the ASR output (or typed input) to generate an NLU output, including the user's intent in providing the spoken utterance and, optionally, slot value(s) for parameter(s) associated with the intent. Additionally, a fulfillment engine can be used to process the NLU output and generate a fulfillment output, such as a structured request to obtain content responsive to the spoken utterance.

[0003] In some cases, these automated assistants may be used in a particular environment. For example, a given automated assistant may be associated with an in-vehicle computing device of a user's vehicle. In this example, the given automated assistant may utilize the pipeline of components described above when interpreting and responding to these spoken utterances and / or typed inputs. However, in some of these cases, the given automated assistant does not consider being used in a vehicular environment when interpreting and responding to these spoken utterances and / or typed inputs. As a result, some of the spoken utterances associated with the vehicle's operations may be misinterpreted, leading to performing an incorrect action(s), providing an incorrect response content, etc. Furthermore, the user may provide additional spoken utterances and / or additional typed inputs to cause the correct action(s) to be performed and / or the correct response content to be provided, which increases the amount of user input and wastes computational resources. Summary of the Invention

[0004] Implementations described herein relate to various techniques for biasing an interpretation of a verbal utterance received in a vehicular environment. For example, implementations may receive a verbal utterance including a query from a user located in the user's vehicle and obtain corresponding vehicle sensor data instances generated by vehicle sensor(s) of the vehicle. In some implementations, based on at least the query and the corresponding vehicle sensor data instances, it may be determined to perform a search only on the first corpus data and not on the second corpus data to obtain a given response to the query. In these implementations, a search may be performed on the first corpus data to identify one or more candidate responses, and the given candidate response from among the one or more candidate responses may be provided for presentation to the user in response to the query. In other implementations, a search may be performed on both the first corpus data and the second corpus data to identify one or more candidate responses, and the given candidate response from among the one or more candidate responses may be provided for presentation to the user in response to the query. In these implementations, a selection of a given candidate response may be biased toward any candidate response obtained from a search on the first corpus data based on the corresponding vehicle sensor data instances. In particular, the first corpus data in these embodiments may correspond to vehicle-specific user manual corpus data provided by the vehicle's OEM, and the second corpus data in these embodiments may correspond to any corpus data that is not vehicle-specific, such as web-based corpus data.

[0005] For example, assume that a verbal utterance of "Which tire has low air pressure?" is received from a user of the vehicle while the user is within the vehicle. Further assume that a tire pressure sensor data instance of tire pressure data is obtained from a tire pressure sensor of the vehicle indicating that a tire on the passenger side of the vehicle is low on air. In some examples, the tire pressure sensor data instance may cause a light on the vehicle's dashboard to be illuminated and / or an alert to be provided for presentation to the user on a display of an in-vehicle computing device to make the user aware that the tire on the passenger side of the vehicle is low on air. Thus, interpretation of the verbal utterance provided by the user may be biased towards vehicle-specific user manual corpus data. As a result, a given candidate response to the query "The tire pressure is 28 psi, but according to the user manual it should be 32 psi" may be selected to provide an audible and / or visual presentation to the user in response to the query. Notably, in this example, the given candidate response may be selected in preference to one or more other candidate responses using various biasing criteria. In the absence of these various bias criteria, not only may vehicle-specific user manual corpus data be searched, but one or more other corpus data that are not vehicle-specific may also be searched.

[0006] In some implementations, this bias determination may be based on determining whether the verbal utterance including the query was received within a threshold duration during which the corresponding vehicle sensor data instance was generated and / or as indicated by a corresponding timestamp associated with the corresponding vehicle sensor data instance. The threshold duration may include, for example, a static duration (e.g., 30 seconds, 2 minutes, etc.) or a dynamic duration based on other factors. For example, further assume that the tire pressure sensor data instance, as described above, provided an alert for illuminating a dashboard light of the vehicle and / or presenting to a user. Further assume that the verbal utterance was received within 15 seconds of the dashboard light being illuminated and / or the alert being displayed. In this example, it may be determined that the verbal utterance includes a query associated with the dashboard light being illuminated and / or the alert being displayed. This thus allows the temporal relationship between the provided verbal utterance and the corresponding vehicle sensor data instance to be utilized in the biased interpretation of the verbal utterance.

[0007] In additional or alternative implementations, this bias determination may be made based on determining whether the verbal utterance including the query is associated with the corresponding vehicle sensor data instance. For example, assume again that the corresponding vehicle sensor data instance corresponds to a tire pressure data instance indicating that a passenger side tire is low on air, and a vehicle dashboard light is illuminated and / or an alert is provided for presentation to a user as described above, and a verbal utterance of "Which one is low on air?" is received. The audio data capturing the verbal utterance including the query may be processed using an automatic speech recognition (ASR) model to generate one or more recognized terms corresponding to the query. This may allow a determination to be made (e.g., using soft word matching, semantic word matching, and / or other techniques) that the recognized term "low on air" is associated with the passenger side tire that is low on air, taking into account the dashboard light and / or alert. This may thus allow a linguistic relationship between the provided verbal utterance and the corresponding vehicle sensor data instance to be additionally or alternatively utilized in a biased interpretation of the verbal utterance.

[0008] In additional or alternative implementations, the bias determination may be based on determining whether the user who provided the verbal utterance was associated with the vehicle for a threshold duration. The threshold duration may be indicative of, for example, the amount of time the user spent in the vehicle, the distance the user drove the vehicle, the number of times the user started the vehicle, the number of times a given corresponding vehicle sensor data instance generated by one or more sensors of the vehicle was obtained, and / or other factors. For example, when processing audio data capturing the verbal utterance including the query, speaker identification of the audio data may be performed to determine whether the user who provided the verbal utterance is a known user. Speaker identification may be performed using any known technique (e.g., text-dependent speaker identification, text-independent speaker identification, etc.) and using any known speaker identification model. Additional or alternative techniques may be utilized to determine whether the user who provided the verbal utterance is a known user, such as face identification based on processing visual data generated by a visual sensor(s) of the computing device or additional computing device, fingerprint identification based on processing fingerprint data generated by a fingerprint sensor(s) of the computing device or additional computing device, and / or any other technique.

[0009] Additionally, the identification information can be compared to the identification information of known users of the vehicle (e.g., accounts associated with the vehicle or automated assistant) to determine whether the user is a known user. In these implementations, if the user is a known user but has not been associated with the vehicle for the threshold duration, the interpretation of the verbal utterance can be biased toward user manual corpus data. Additionally or alternatively, if the user is not a known user, the interpretation of the verbal utterance can be biased toward user manual corpus data. However, in these implementations, if the user is a known user and has been associated with the vehicle for the threshold duration, the interpretation of the verbal utterance may not be biased toward user manual corpus data because the user is likely already familiar with the cause of the vehicle's dashboard light and / or the cause of the alert (e.g., low passenger side tire pressure). Thus, the duration the user has been associated with the vehicle can additionally or alternatively be utilized in biasing the interpretation of the verbal utterance.

[0010] In additional or alternative implementations, this biasing decision may be made based on determining whether the verbal utterance includes an explicit instruction to search only the first corpus data. For example, assume again that the corresponding vehicle sensor data instance corresponds to a tire pressure data instance indicating that the passenger side tire is low on air, and assume that a vehicle dashboard light is illuminated and / or an alert is displayed as described above. However, assume that a verbal utterance is received that states "Check the user manual to see which one has low pressure" (e.g., rather than simply "Which one has low pressure"). In this example, the query included in the verbal utterance also includes an explicit instruction to search only the first corpus data as indicated in the "user manual." This thus allows the user to additionally or alternatively provide an explicit instruction that may be utilized in biasing the interpretation of the verbal utterance.

[0011] It should be understood that while the techniques above are described as being implemented on an on-board computing device of a vehicle, this is for purposes of illustration and not limitation. For example, the techniques described herein may additionally or alternatively be implemented on a mobile computing device of a user of the vehicle. Further, the techniques described herein may additionally or alternatively be implemented by a remote computing device (e.g., a remote server or cluster of remote servers). However, in various implementations, on-device processing by the on-board computing device and / or mobile computing device may be prioritized to reduce latency in responding to queries contained in verbal utterances.

[0012] Moreover, while the techniques above are described with respect to utilizing corresponding vehicle sensor data instances in determining which of one or more corpus data to search and / or in causing a search to be performed on one or more identified corpus data, it should also be understood that this is for purposes of illustration and not intended to be limiting. For example, the techniques described herein may perform one or more of these searches without utilizing corresponding vehicle sensor data instances at all. For example, some vehicles do not include an in-vehicle computing device capable of implementing the automated assistant described herein and / or do not include a controller area network (CAN) bus for obtaining corresponding vehicle sensor data instances. Nonetheless, in these examples, an automated assistant implemented on a user's mobile computing device may utilize the same or similar techniques as those described herein to bias the interpretation of a verbal utterance by limiting the search space of queries associated with the user's vehicle.

[0013] The techniques described herein may provide various technical advantages. As a non-limiting example, the techniques described herein may enable a system to efficiently constrain a search space for identifying candidate response(s) to a query based on a query of vehicle sensor data and / or corresponding vehicle sensor data instance(s) generated by a vehicle's sensor(s). For example, the techniques described herein may perform a search on a given corpus data (e.g., vehicle-specific user manual corpus data) based on a query to identify a given candidate response to a query and bias the interpretation of the verbal utterance by avoiding the use of other corpus data based on various context signals and / or data. Also, for example, the techniques described herein may perform a search on multiple corpus data (e.g., vehicle-specific user manual corpus data and at least one additional non-vehicle-specific corpus data) but bias the selection of a given candidate response to a query toward those obtained from the given corpus data, thereby biasing the interpretation of the verbal utterance. As a result, the consumption of computational resources in processing the verbal utterance to identify a given candidate response may be reduced.

[0014] The above description is provided as a summary of only some of the embodiments disclosed herein. These and other embodiments are described in further detail herein. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 illustrates various aspects of the present disclosure and shows a block diagram of an exemplary hardware and software environment in which embodiments disclosed herein may be implemented. [Diagram 2] 2 illustrates an exemplary process flow for biasing an interpretation of verbal utterance(s) received in a vehicular environment illustrating various aspects of FIG. 1, according to various embodiments. [Diagram 3]1 shows a flowchart illustrating an exemplary method for biasing an interpretation of verbal utterance(s) received in a vehicular environment, according to various embodiments. [Figure 4] 1 shows a flowchart illustrating another exemplary method for biasing an interpretation of verbal utterance(s) received in a vehicular environment, according to various embodiments. [Diagram 5] 1 shows a flowchart illustrating yet another exemplary method for biasing an interpretation of verbal utterance(s) received in a vehicular environment, according to various embodiments. [Figure 6A] 1 illustrates various non-limiting examples of computing devices illustrating various user interactions that bias audio processing of oral speech in a vehicular environment, according to various implementations. [Figure 6B] 1 illustrates various non-limiting examples of computing devices illustrating various user interactions that bias audio processing of oral speech in a vehicular environment, according to various implementations. [Figure 7] 1 illustrates an exemplary architecture of a computing device in accordance with various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] 1, an environment in which one or more selected aspects of the present disclosure may be implemented is shown. An exemplary environment includes a number of computing devices 110. 1-N , a bias system 120 for automated assistant applications, a vehicle 100A, one or more original equipment manufacturer (OEM) applications 181, one or more first-party applications 182, and one or more third-party applications 183. 1-N, 120, 181, 182, and 183 may each communicate, for example, over one or more networks generally indicated at 195. The one or more networks may include wired or wireless networks, such as a local area network (LAN), including Wi-Fi, Bluetooth, near field communication, and / or other LANs, a wide area network (WAN), including the Internet, and / or other networks to facilitate communication between the components shown in FIG.

[0017] In various embodiments, a user may access a computing device 110 1-N 1. The computing device 110 may be configured to operate in a manner that interacts with the other components shown in FIG. 1-N may be used in, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, an in-vehicle computing device of a vehicle 100A (e.g., 110 N The vehicle may be an in-vehicle communication system, an in-vehicle entertainment system, and / or an in-vehicle navigation system, as shown with respect to FIG. 1, or a wearable device including a computing device such as a head mounted display ("HMD"), a "smart" watch, etc. that provides an augmented reality ("AR") or virtual reality ("VR") immersive computing experience. Additional and / or alternative computing devices may be provided.

[0018] Computing device 110 1-NEach of the computing devices 110 and bias systems 120 may include one or more memories for storing data and software applications (e.g., one or more of the OEM applications 181, one or more of the first party applications 182, and / or one or more of the third party applications 183), one or more processors for accessing the data and executing the software applications, and other components that facilitate communication over one or more of the networks 195. 1-N And / or operations performed by one or more of the biasing systems 120 may be distributed across multiple computer systems. For example, the biasing system 120 may be implemented as a computer program executing solely on or distributed across one or more computers at one or more locations that are communicatively connected to each other via, for example, one or more of the networks 195.

[0019] Component 110 1-N, 120, 181, 182, and 183 may include a variety of different components that may be used, for example, to bias an interpretation of verbal speech received in the vehicle environment, as described herein. For example, the computing device 1101 may include a user interface engine 1111 to detect and process user input (e.g., verbal speech, typed input, and / or touch input) directed to the computing device 1101. As another example, the computing device 1101 may include one or more sensors 1121 to generate corresponding sensor data. The one or more sensors may include, for example, a Global Positioning System ("GPS") sensor that generates GPS data, a vision component that generates visual data in the field of view of the vision component, a microphone that generates audio data based on verbal speech captured in the environment of the computing device 1101, and / or other sensors that generate corresponding sensor data.

[0020] As yet another example, the computing device 1101 can operate an input processing engine 1131 (which can be, e.g., standalone or part of another application, such as part of an automated assistant application) to interpret various user inputs received at the computing device 1101. For example, the input processing engine 1131 can capture oral speech and process audio data generated by the microphone(s) of the client device 1101 to generate ASR output using automatic speech recognition (ASR) model(s) (e.g., a recurrent neural network (RNN) model, a transformer model, and / or any other ML model capable of performing ASR). Additionally, the input processing engine 1131 can process the ASR output (or typed input) to generate NLU output using natural language understanding (NLU) model(s) (e.g., a long short-term memory (LSTM), a gated recurrent unit (GRU), and / or any other type of RNN or other ML model capable of performing NLU) and / or grammar-based NLU rule(s). Additionally, the input processing engine 1131 can process the NLU output using the fulfillment model(s) and / or fulfillment rule(s) to obtain one or more candidate responses responsive to the user input, such as action(s) for the automated assistant to perform based on the user input, content item(s) to be provided for presentation to the user based on the user input, etc. In implementations in which the textual content is rendered audibly in response to spoken utterances or typed input, the user interface engine 1111 can process the textual content using the text-to-speech model(s) to generate synthetic speech audio data including computer-generated synthetic speech capturing the content. The synthetic speech audio data can be audibly rendered for presentation to the user via speaker(s) of the computing device 1101.In embodiments in which visual content is visually rendered in response to spoken utterances or typed input, user interface engine 1111 can visually render the visual content for presentation to a user via the display of 1101.

[0021] In various implementations, the ASR output may include, for example, one or more speech hypotheses (e.g., term hypotheses and / or transcript hypotheses) predicted to correspond to the user's voice activity and / or oral speech captured in the audio data, one or more corresponding prediction values ​​(e.g., probabilities, log-likelihoods, and / or other values) for each of the one or more speech hypotheses, a number of phonemes predicted to correspond to the user's voice activity and / or oral speech captured in the audio data, and / or other ASR outputs. In some versions of these implementations, the input processing engine 1131 may select one or more of the speech hypotheses as recognized text corresponding to the oral speech (e.g., based on the corresponding prediction values).

[0022] In various implementations, the NLU output may include annotated recognized text, for example, including one or more annotations of the recognized text for one or more (e.g., all) of the terms of the recognized text. For example, the input processing engine 1131 may use a part-of-speech tagger (not shown) configured to annotate terms with their grammatical roles. Additionally or alternatively, the input processing engine 1131 may use an entity tagger (not shown) configured to annotate entity references in one or more segments of the recognized text. The entity references may include, for example, references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, places (real and fictional), etc. In some implementations, data about the entities may be stored in one or more databases, such as a knowledge graph (not shown). In some implementations, the knowledge graph may include nodes representing known entities (and, in some cases, entity attributes), as well as edges connecting the nodes to represent relationships between the entities. The entity tagger may annotate references to entities at a high level of granularity (e.g., to enable identification of all references to an entity class, such as a person) and / or at a low level of granularity (e.g., to enable identification of all references to a particular entity, such as a particular person). The entity tagger may rely on the content of the natural language input to resolve particular entities, and / or may optionally communicate with a knowledge graph or other entity database to resolve particular entities. As described herein, the NLU output may be utilized in determining whether the spoken utterance includes one or more salient terms that correspond to user manual corpus data.

[0023] Additionally or alternatively, the input processing engine 1131 may employ a co-reference resolver (not shown) configured to group or "cluster" references to the same entity based on one or more contextual cues. As one non-limiting example, the co-reference resolver may be utilized to resolve the term "that" in a natural language input of "What's that light?" to a particular light or indicator associated with the operation of the vehicle 100A based on corresponding sensor data instances generated by a vehicle sensor(s) that lead to the particular light or indicator associated with the operation of the vehicle 100A being generated. In some implementations, one or more components utilized by the input processing engine 1131 may rely on annotations from one or more other components utilized by the input processing engine 1131. For example, in some implementations, the entity tagger may rely on annotations from the co-reference resolver in annotating all references to a particular entity. Also, for example, in some implementations, the co-reference resolver may rely on annotations from the entity tagger in clustering references to the same entity.

[0024] As yet another example, the computing device 1101 may operate a bias system client 1141 (which may be, for example, standalone or part of another application, such as part of an automated assistant application) to interact with the bias system 120. Additionally, additional computing devices 110 N may take the form of an on-board computing device of the vehicle 100A. Although not shown, additional computing devices 110 N may include the same or similar components as computing device 1101. For example, additional computing device 110 NEach instance of the vehicle 100A may include a user interface engine for detecting and processing user input, one or more sensors for generating corresponding vehicle sensor data instances of the vehicle sensor data, an input processing engine, and / or a bias system client for interacting with the bias system 120. In this example, the one or more sensors may include a tire pressure sensor for generating tire pressure data for tires of the vehicle 100A, an air flow sensor for generating air flow data for an air conditioning system of the vehicle 100A, a vehicle speed sensor for generating vehicle speed data for the vehicle 100A, an energy sensor for generating energy source data for an energy source of the vehicle 100A, a transmission sensor for generating transmission data for a transmission of the vehicle 100A, and / or a vehicle sensor for interacting with the vehicle 100A and / or the on-board computing device 110 of the vehicle 100A. N 1 may include vehicle sensors, such as any other sensors integrated into the computing device 1101 and the in-vehicle computing device 110. N While only one computing device is shown, it should be understood that this is for illustrative purposes only and that additional or alternative computing devices may be provided.

[0025] In various implementations, the bias system 120 may include an interface engine 121, an input processing engine 122, a request processing engine 123, a user context engine 124, a vehicle context engine 125, a bias engine 126, a search engine 127, and a response engine 128, as shown in FIG. 1 . In some implementations, one or more of the engines 121-128 of the bias system 120 may be omitted. In some implementations, all or some aspects of one or more of the engines 121-128 of the bias system 120 may be combined. In some implementations, one or more of the engines 121-128 of the bias system 120 may be integrated into the computing device 110. 1-NIn some implementations, one or more of the engines 121-128 of the bias system 120, or any operating portion thereof, may be implemented with components that execute partially or completely remotely from one or more of the computing device 110. 1-N The present invention may be implemented in components that are executed partially or completely locally by one or more of the following:

[0026] 2, an exemplary process flow for biasing an interpretation of verbal utterance(s) received in a vehicle environment illustrating various aspects of FIG. 1 is shown. An interface engine 121 (not shown in FIG. 2) can facilitate data exchange between one or more of the engines 121-128 illustrated in FIGS. 1 and 2. By way of example, verbal utterances may be received from a user located within a vehicle 100A through a computing device (e.g., computing device 1101, in-vehicle computing device 110A, etc.). N Assume that the spoken utterance is received via the input processing engine 113 (e.g., a voice input device 114, a mobile terminal 115, a mobile phone 116, a mobile terminal 117, a mobile terminal 118, a mobile terminal 119, a mobile terminal 120, a mobile terminal 210, a mobile terminal 220, a mobile terminal 230, a mobile terminal 240, a mobile terminal 250, a mobile terminal 1-NThe request processing engine 122 can process the audio data to generate processed input data 202. For example, the audio data can be processed using automatic speech recognition (ASR) model(s), natural language understanding (NLU) model(s), speaker identification model(s), and / or other models to generate the processed input 202. The processed input data 202 can include, for example, ASR data, NLU data, and / or any other data resulting from processing the audio data 202. The request processing engine 122 can receive the processed input data 202 and determine that the verbal utterance includes a query based on analyzing the ASR data, NLU data, and / or other data included in the processed input data. Furthermore, the request processing engine 122 can generate request data 222 that can be utilized in performing one or more searches against one or more corpus data as described herein. Furthermore, the request processing can provide the request data to the search engine 127.

[0027] In various embodiments, the input processing engine 113 1-NWhile the audio data 201 is being processed by the bias engine 122 and other components of the bias system 120, the bias engine 126 may process various signals and data acquired by the bias system 120 to generate bias data 226 and determine whether to limit the search space of a query to one or more particular corpus data. These various signals and data may include, for example, user context signal(s) 224 acquired via the user context engine 124, vehicle context signal(s) 225 acquired via the vehicle context engine 125, original equipment manufacturer (OEM) data 281 acquired via one or more OEM applications 181, first party data 282 acquired via one or more first party applications 182, third party data 283 acquired via one or more third party applications 183, and / or various other signals and data that may be utilized in determining whether to limit the search space of a query to one or more particular corpus data.

[0028] For example, the user context signal(s) 224 obtained via the user context engine 124 may characterize a state of the user of the vehicle 100A. The state of the user of the vehicle 100A may include, for example, whether the user is driving the vehicle 100A, whether the user is a passenger in the vehicle 100A, or the user's level of engagement while the vehicle 100A is traveling. The state of the user of the vehicle 100A may additionally or alternatively include, for example, a starting point of the user, a destination point of the user, a route that the user is predicted to take from the starting point to the destination point, and / or any other signal that characterizes the state of the user. Thus, the user context signal(s) 224 may be received from the sensor(s) 1121 of the computing device 1101, the on-board computing device 110 NIt should be understood that the data may be obtained based on any other data, such as data generated by a sensor(s) of the vehicle 100A, a sensor(s) of the vehicle 100B, and / or first party data 282 from a first party application(s) 182, and / or third party data 283 from a third party application(s) 183.

[0029] Also, for example, the vehicle context signal(s) 225 obtained via the user context engine 125 can characterize the state of the vehicle 100A. The state of the vehicle 100A can include, for example, whether the vehicle 100A is powered and in an "on" state, whether the vehicle 100A is moving, whether the vehicle 100A is stopped, whether the vehicle 100A is parked, whether the vehicle 100A is following a particular route, or whether the vehicle 100A includes additional occupants other than the user of the vehicle 100A. The state of the vehicle 100A can additionally or alternatively include information associated with the operation of the vehicle 100A, such as, for example, the amount of energy source (e.g., gas, battery, etc.) available to the vehicle 100A, whether any dashboard indicators are illuminated, and / or any other signals that characterize the state of the vehicle. Thus, the vehicle context signal(s) 225 can be used by the in-vehicle computing device 110 to determine whether the vehicle 100A is in a powered "on" state. N The data may be obtained based on data generated by a sensor(s) of the vehicle 100A, a sensor(s) of the vehicle 100B, and / or any other data, such as OEM data 281 from one or more of the OEM applications 181.

[0030] Additionally, OEM data 281 may include any information accessible by an OEM application associated with vehicle 100A and usable by search engine 127 and / or response engine 128 in obtaining a given candidate response 228 to a query contained in the verbal utterance captured in audio data 201. Similarly, first party data 282 and third party data 283 may include any information accessible by first party application(s) 182 and third party application(s) 183, respectively, and usable by search engine 127 and / or response engine 128 in obtaining a given candidate response 228 to a query contained in the verbal utterance captured in audio data 201, such as user account data, application usage information, and / or other data. As used herein, the term "first party application" may refer to a software application developed and / or maintained by the same entity that develops and / or maintains the automated assistant and / or biasing system 120 described herein. Additionally, as used herein, the term “third-party application” may refer to a software application or system that is developed and / or maintained by an entity other than the one that develops and / or maintains the automated assistant and / or bias system 120 described herein.

[0031] The bias engine 126 may process these various signals and data, and / or other signals and data, to determine whether to limit the search space of the query to one or more particular corpus data. In other words, the bias engine 126 may process these various signals and data to determine whether one or more bias criteria are met. In response to determining that one or more of the bias criteria are met, the bias engine 126 may generate bias data 226 to provide the bias data 226 to the search engine 127 to alter the manner in which the response(s) are obtained. For example, as described in more detail below with respect to FIG. 3, the search engine 127 may limit a search performed based on at least the query embodied in the request data 222 to a given corpus data among the multiple corpora 127A. In this example, the search engine 127 may prevent any search from being performed on other corpus data. For example, the content item(s) 227 obtained in response to causing a search to be performed on the given corpus data may include information identified from the given corpus data. Also, for example, as described in more detail below with respect to Figure 4, search engine 127 may cause multiple searches to be performed against multiple corpora 127A. Thus, content item(s) 227 retrieved in response to causing multiple searches to be performed against multiple corpora 127A may include information identified from each of multiple corpora 127A. However, in this example, answer engine 128 may then bias the selection of a given answer to a query provided for presentation to a user against a given corpus data.

[0032] As described herein, the plurality of corpora 127A can include first corpus data and at least second corpus data in addition to the first corpus data. The first corpus data can correspond to user manual corpus data of a plurality of user manuals provided by the OEM of the vehicle 100A and / or other vehicles, for example. In contrast, the at least second corpus data can correspond to any other corpus data. In particular, the first corpus data can be defined or indexed at various granularities. For example, the user manual corpus data can be defined or indexed by the OEM, further defined or indexed by the year the vehicle was manufactured by the OEM, further defined or indexed by the manufacturer of the vehicle manufactured by the OEM, further defined or indexed by the model of the vehicle manufactured by the OEM, and further defined or indexed by a plurality of feature tags of the features of the vehicle manufactured by the OEM. The plurality of feature tags associated with any document in the first corpus data can define various features of the vehicle described in that document (e.g., a document describing a power seat, a document describing a heated seat, a document describing a sport mode, etc.). Thus, when oral utterances are received from a user within the vehicle 100A, the year of manufacture, manufacturer, model, and / or feature tags of the vehicle 100A may be transmitted to the bias system 120 for use in searching at least the first corpus data (e.g., as part of the request data 222 determined based on processing the oral utterance and / or as part of the bias data 226 determined based on the OEM data 281 received from the OEM application(s) 181).For example, when a user is in vehicle 100A and provides oral utterances, the year, manufacturer, model, and set of feature tags for vehicle 100A can be provided to bias system 120 and used to restrict a search against the user manual corpus data to relevant documents (e.g., documents associated with the year, manufacturer, and model of vehicle 100A and whose feature tags are a subset of the feature tags associated with vehicle 100A).

[0033] In various implementations, the first corpus data may additionally or alternatively be associated with a number of salient terms that may be present throughout the user manual corpus data and identified using various techniques (e.g., TF-IDF and / or other techniques across the user manuals included in the user manual corpus data). These salient terms may be stored in association with various user manuals of the user manual corpus data stored in the plurality of corpora 127A. For example, a first user manual of the user manual corpus data may include a first set of salient terms, and a second user manual of the user manual corpus data may include a second set of salient terms. Notably, some of these salient terms may overlap between different user manuals included in the user manual corpus data. In these implementations, the request data 222 may include an indication that one or more terms of the oral utterance (e.g., determined based on NLU output generated in processing the oral utterance) match one or more of the salient terms in the user manual corpus data using various techniques (e.g., soft matching, etc.). In these implementations, the presence of one or more terms of the oral utterance that match one or more of the salient terms in the user manual corpus data may be utilized as a biasing criterion in biasing searches performed on the first corpus data and / or the second corpus data. In some of these implementations, when an oral utterance is received from a user in the vehicle 100A, one or more terms of the oral utterance that match one or more of the salient terms in the owner manual corpus of the vehicle 100A may be transmitted to the biasing system 120 for use in searching at least the first corpus data (e.g., as part of the request data 222 determined based on processing the oral utterance and / or as part of the biasing data 226 determined based on the OEM data 281 received from the OEM application(s) 181).

[0034] The answer engine 128 can analyze the content item(s) 227 to identify a given candidate answer 228 that is provided for presentation to the user in response to a query contained in the verbal utterance captured in the audio data 201. In some implementations, the answer engine 128 can augment the content item(s) 227 with other content that is stored in the answer(s) database 128A. For example, assume that the given content item corresponds to "Tire pressure=32 psi." In this example, the automated assistant can estimate that the current tire pressure in the passenger side tire is 28 psi based on the vehicle context signal(s) 225. Assuming the query contained in the verbal utterance is related to tire pressure and is a question inquiring as to why a light associated with tire pressure was illuminated (e.g., based on at least the vehicle context signal(s) 225 and biased towards user manual corpus data), the answer engine 128 may augment the given content item to generate a given candidate answer 228 of "the tire pressure is 28 psi but should be 32 psi." Thus, the given candidate answer 228 may provide an auditory and / or visual presentation to the user using context-related signals in response to the query to bias the interpretation of the verbal utterance.

[0035] It should be understood that while the example of FIG. 2 is described with respect to a query contained in a verbal utterance captured in audio data 201A, this is for illustrative purposes and is not intended to be limiting. For example, the input processing engine 113 1-N / 122 may additionally or alternatively process the non-audio data 201B that captured the query. For example, a user may additionally or alternatively process the non-audio 201B, including text data, touch data, and / or any other non-audio data, to identify a query contained in the non-audio data 201B. Thus, even when a user provides a query via non-audio data, the biasing system 120 may use the same or similar techniques to identify and provide given candidate responses for presentation to the user of the vehicle.

[0036] 3, a flow chart illustrating an example method 300 for biasing an interpretation of verbal utterance(s) received in a vehicular environment is shown. For convenience, operations of method 300 are described with reference to a system that performs the operations. This system of method 300 may include at least one processor, at least one memory, and / or other component(s) of a computing device(s) (e.g., computing device(s) 110 of FIG. 1 ). 1-N 1, the biasing system 120 of FIG. 1, the computing device 710 of FIG. 7, remote server(s), and / or other computing devices. Although the operations of the method 300 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0037] In block 352, the system receives a verbal utterance including a query from a user via a computing device, the verbal utterance being provided while the user is in the user's vehicle. The computing device can be, for example, an on-board computing device of the vehicle, or a mobile computing device of a user of the vehicle communicatively connected to the vehicle and / or the on-board computing device of the vehicle. In some implementations, the verbal utterance may be received in response to the automated assistant being explicitly invoked at the computing device via a particular word or phrase (e.g., "Assistant," "Hey Assistant," etc.), the actuation of a hardware or software button of the computing device, a particular gesture or combination of gestures (e.g., hand movements, eye movements, gaze, and / or any one or combination thereof), and / or other techniques for explicitly invoking the automated assistant. In additional or alternative implementations, the verbal utterance may be received in response to the automated assistant being implicitly invoked at the computing device, for example, based on one or more contextual signals associated with the user and / or the user's vehicle.

[0038] For example, the verbal utterance including the query can be captured in audio data generated by one or more microphones of the computing device, and can optionally respond to an invocation of an automated assistant as described above. The system can use an automatic speech recognition (ASR) model to process the audio data capturing the verbal utterance including the query to generate ASR data for the query, such as one or more recognized terms corresponding to the query. Additionally, the system can use a natural language understanding (NLU) model to process the ASR data for the query to generate NLU data for the query, such as intent(s), slot value(s) for parameter(s) associated with the intent(s), and / or other NLU data. Based on the NLU data for the query, the system can determine that the verbal utterance includes the query. Additionally, as described below, the system can utilize the NLU data in causing a search to be performed against one or more corpus data.

[0039] In block 354, the system obtains corresponding vehicle sensor data instances of the vehicle sensor data, the corresponding vehicle sensor data instances being generated by one or more vehicle sensors of the user's vehicle. The one or more vehicle sensors of the user's vehicle may include, for example, tire pressure sensor(s) of the vehicle generating tire pressure data for tires of the vehicle, air flow sensor(s) of the vehicle generating air flow data for an air conditioning system of the vehicle, vehicle speed sensor(s) of the vehicle generating vehicle speed data for the vehicle, energy sensor(s) of the vehicle generating energy source data for an energy source of the vehicle, transmission sensor(s) of the vehicle generating transmission data for a transmission of the vehicle, and / or any other sensors integrated into the vehicle and / or on-board computing device, etc. In particular, each of these sensors may generate vehicle sensor data instances at a corresponding frequency, which may be the same for one or more sensors of the vehicle and / or may be different for one or more other sensors of the vehicle. In some implementations, the corresponding vehicle sensor data instance may be associated with a corresponding timestamp that corresponds to the time the corresponding vehicle sensor data instance was generated by one or more of the vehicle sensors and / or the time the corresponding vehicle sensor data instance was acquired by the system. This time may be a global time relative to a global clock (e.g., generated and / or acquired at 9:49 a.m.) or a relative time relative to a relative clock (e.g., generated and / or acquired two minutes after the user entered the vehicle).

[0040] In block 356, the system determines whether to perform a first search on the first corpus data and / or a second search on the second corpus data to identify one or more candidate responses to the query contained in the verbal utterance based on (i) the query and (ii) the corresponding vehicle sensor data instances. In other words, the system may determine whether to bias the search space utilized to identify one or more candidate responses by possibly restricting the search space based on the query and the corresponding vehicle sensor data instances. The first corpus data may correspond, for example, to user manual corpus data that is vehicle-specific and provided by the vehicle's OEM. Furthermore, the second corpus data may correspond, for example, to corpus data available to the system that is not vehicle-specific (e.g., not provided by the vehicle's OEM), such as web-based corpus data, knowledge graphs, and / or any other corpus data available to the system that is not vehicle-specific. Furthermore, the system may make this bias determination based on one or more bias criteria.

[0041] In some implementations, the system may make this biasing determination based on determining whether the verbal utterance including the query was received within a threshold duration during which the corresponding vehicle sensor data instance was generated and / or as indicated by a corresponding timestamp associated with the corresponding vehicle sensor data instance. The threshold duration may include, for example, a static duration (e.g., 30 seconds, 2 minutes, etc.) or a dynamic duration based on other factors. For example, assume that the corresponding vehicle sensor data instance corresponds to a tire pressure data instance indicating that a passenger side tire is low on air. As a result, an indication of the low tire may be provided for presentation to a user, such as a light on the vehicle's dashboard being illuminated and / or an alert being displayed on a display of the vehicle's onboard computing system. Further, assume that the verbal utterance "Which one has low pressure?" is received by the system within 15 seconds of the dashboard light being illuminated and / or the alert being displayed. In this example, the system may determine that the verbal utterance includes a query associated with a dashboard light being illuminated and / or an alert being displayed. Thus, the system can limit the search space of the query to the first corpus data corresponding to the vehicle-specific user manual corpus data when obtaining response content responsive to the query. In the absence of this temporal relationship between the verbal utterance and the corresponding vehicle sensor data instance, the system can search not only the vehicle-specific user manual corpus data but also one or more other corpus data that are not vehicle-specific when obtaining response content responsive to the query. In other words, the system can use this temporal relationship between the verbal utterance and the corresponding vehicle sensor data instance to infer that the user provided the verbal utterance to further inquire about the illuminated dashboard light and / or the displayed alert.

[0042] In additional or alternative implementations, the system may make this biasing determination based on determining whether the verbal utterance including the query is relevant to the corresponding vehicle sensor data instance. For example, assume again that the corresponding vehicle sensor data instance corresponds to a tire pressure data instance indicating that a passenger side tire is low on air, a vehicle dashboard light is illuminated and / or an alert is displayed, and a verbal utterance of "Which tire is low on air?" is received by the system. As described above, the system may use the ASR model to process the audio data capturing the verbal utterance including the query to generate one or more recognized terms corresponding to the query. This allows the system to make a determination that the recognized term "low on air pressure" is relevant to the passenger side tire, given that the vehicle dashboard light is illuminated and / or the alert is displayed, based on the corresponding vehicle sensors. In doing so, the system may utilize various word matching techniques (e.g., soft word matching, semantic word matching, and / or other techniques) to determine that the alert and the query are relevant. Thus, the system can limit the search space of the query to the first corpus data corresponding to the vehicle-specific user manual corpus data when obtaining response content responsive to the query. In the absence of this linguistic relationship between the query terms and the corresponding vehicle sensor data instances, the system can search not only the vehicle-specific user manual corpus data but also one or more other corpus data that are not vehicle-specific when obtaining response content responsive to the query. In other words, the system can additionally or alternatively utilize this linguistic relationship between the verbal utterance and the corresponding vehicle sensor data instances to infer that the user provided the verbal utterance to further inquire about the illuminated dashboard lights and / or the displayed alerts.

[0043] In additional or alternative implementations, the system may make this bias determination based on determining whether the user who provided the verbal utterance was associated with the vehicle for a threshold duration. The threshold duration may be indicative of, for example, the amount of time the user spent in the vehicle, the distance the user drove the vehicle, the number of times the user started the vehicle, the number of times a given corresponding vehicle sensor data instance generated by one or more sensors of the vehicle was obtained, and / or other factors. For example, in processing audio data capturing the verbal utterance including the query, the system may perform speaker identification of the audio data to determine whether the user who provided the verbal utterance is a known user. The system may perform speaker identification using any known technique (e.g., text-dependent speaker identification, text-independent speaker identification, etc.) and using any known speaker identification model. The system may additionally or alternatively utilize other techniques to determine whether the user providing the verbal utterance is a known user, such as facial identification based on processing of visual data generated by a visual sensor(s) of the computing device or the additional computing device, fingerprint identification based on processing of fingerprint data generated by a fingerprint sensor(s) of the computing device or the additional computing device, and / or any other technique.

[0044] Further, the system may compare the identification information with identities of known users of the vehicle (e.g., accounts associated with the vehicle or the automated assistant) to determine whether the user is a known user. In these embodiments, if the user is a known user but has not been associated with the vehicle for the threshold duration, the system may limit the search space of the query to the first corpus data corresponding to the user manual corpus data specific to the vehicle when obtaining response content responsive to the query. Additionally or alternatively, if the user is not a known user, the system may also limit the search space of the query to the first corpus data corresponding to the user manual corpus data specific to the vehicle when obtaining response content responsive to the query. In other words, the system may additionally or alternatively limit the search space to the user manual corpus data based on the user who provided the verbal utterance being a new owner of the vehicle (or a lessee of the vehicle) who may not be familiar with the cause of the vehicle's dashboard lights and / or the cause of the alerts being displayed. However, in these embodiments, if the user is a known user and has been associated with the vehicle for the threshold duration, the user is likely already familiar with what causes the vehicle's dashboard to light up and / or alert to appear, and so the system may not need to limit the search space of this query to the first corpus data corresponding to the vehicle-specific user manual corpus data when obtaining response content that responds to the query.

[0045] In additional or alternative implementations, the system may make this biasing determination based on determining whether the verbal utterance includes an explicit instruction to search only the first corpus data. For example, assume again that the corresponding vehicle sensor data instance corresponds to a tire pressure data instance indicating that the passenger side tire is low on air, and a light on the vehicle's dashboard is illuminated and / or an alert is displayed. However, assume that a verbal utterance is received by the system that states, "Check the user manual to see which one has low air pressure." In this example, the query included in the verbal utterance also includes an explicit instruction to search only the first corpus data as indicated in the "user manual."

[0046] In additional or alternative implementations, the system may make this biasing determination based on whether the verbal utterance was received while the user was in the vehicle. For example, assume that the verbal utterance "Which one has low air pressure" was received by the system while the user was in the vehicle. In this example, even if the system has determined to perform both a first search on the first corpus data and a second search on the second corpus data (e.g., as described with respect to the implementation of FIG. 4), the system may bias the selection of a given candidate response toward one or more of the candidate responses obtained from the first corpus data based on the verbal utterance received while the user was in the vehicle.

[0047] In additional or alternative implementations, the system may make this biasing determination based on whether the spoken utterance includes one or more terms determined to correspond to one or more salient terms. For example, assume that the spoken utterance "What is preconditioning" is received by the system while the user is in the vehicle. In this example, the NLU output generated based on processing the spoken utterance may include the term "preconditioning". Further, the system cross-references the term "preconditioning" with one or more salient terms of the user manual corpus data and determines (e.g., using various word matching techniques) that "preconditioning" is a salient term associated with one or more documents of the first corpus data. As a result, the system may perform a first search on the first corpus data but not on the second corpus data (e.g., as described with respect to FIG. 3). Additionally or alternatively, the system may perform a second search on the second corpus data (e.g., as described with respect to the embodiment of FIG. 4), but may bias the selection of a given candidate response toward one or more of the candidate responses obtained from the first corpus data based on oral utterances that include one or more salient terms from the user manual corpus data.

[0048] If, in the repetition of block 356, the system determines to perform a first search on the first corpus data but not a second search on the second corpus data, the system proceeds to block 358. In block 358, the system causes a first search to be performed on the first corpus data to identify one or more candidate responses to the query. In block 360, the system prevents a second search from being performed on the second corpus data to identify one or more of the candidate responses to the query. In other words, the system may generate a first search to be submitted on the first corpus data but may prevent any second search to be submitted on the second corpus data based on the NLU data of the query and the indication of the corresponding vehicle sensor data instances. For example, again assume that the spoken utterance "What's under pressure?" is received by the system and the system has determined to limit the search space to only the user manual corpus data. In this example, the system may generate a structured request to search the user manual corpus data based on the term "under pressure" and an indication that the tire pressure data instance indicates low air on the passenger side. In response, the system may identify one or more candidate responses based on one or more content items included in the user manual corpus data, such as what the tire pressure should be, what tire pressure would cause a vehicle dashboard light to be illuminated and / or an alert to be displayed, and / or any other content items included in the user manual corpus data obtained in response to the first search. In this example, the search may be further limited to a "tire" section of the user manual corpus data.

[0049] If, in the repetition of block 356, the system determines to perform a first search on the first corpus data and a second search on the second corpus data, the system proceeds to block 362. In block 362, the system causes a first search to be performed on the first corpus data to identify one or more candidate responses to the query. In block 364, the system causes a second search to be performed on the second corpus data to identify one or more of the candidate responses to the query. In other words, the system can generate a first search to be submitted on the first corpus data based on the NLU data of the query and the corresponding vehicle sensor data instances, and can also generate at least a second search to be submitted on the second corpus data based on the query. For example, the system causes a first search to be performed on the first corpus data to identify one or more candidate responses to the query in the same or similar manner as described above with respect to the operation of block 358 and the user manual corpus data. Additionally, the system also causes a second search against the second corpus data to identify one or more candidate responses to a query, such as a query submitted to a search engine and / or one or more applications. For example, a query may be submitted to a media application to obtain response content corresponding to "Under Pressure," a song by the band Queen.

[0050] In block 366, the system causes, via the computing device or the additional computing device, to provide a given candidate response from among the one or more candidate responses for presentation to the user. In some implementations, the one or more candidate responses may include only candidate responses based on the first corpus data (e.g., in implementations where the system reaches block 366 from block 360). In other implementations, the one or more candidate responses may include candidate responses based on the first corpus data and the second corpus data (e.g., in implementations where the system reaches block 366 from block 364). In some implementations, the system may select a given candidate response from among the one or more candidate responses based on a ranking of the one or more candidate responses. The system may rank the one or more candidate responses based on one or more ranking criteria. The one or more ranking criteria may include, for example, one or more terms of the query, corresponding vehicle sensor data instances, one or more user context signals characterizing a state of a user of the vehicle, one or more vehicle context signals characterizing a state of the vehicle, application data associated with one or more applications accessible at the computing device or the additional computing device, and / or other criteria. Further, the system may select a given candidate response based on the ranking and provide the given candidate response for audible and / or visual presentation to a user via the computing device or an additional computing device.

[0051] 4, a flow chart illustrating another exemplary method 400 for biasing an interpretation of verbal utterance(s) received in a vehicular environment is shown. For convenience, operations of method 400 are described with reference to a system that performs the operations. This system of method 400 may include at least one processor, at least one memory, and / or other component(s) of a computing device(s) (e.g., computing device(s) 110 of FIG. 1 ). 1-N 1, the biasing system 120 of FIG. 1, the computing device 710 of FIG. 7, remote server(s), and / or other computing devices. Although the operations of the method 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0052] At block 452, the system receives from a user, via a computing device, a verbal utterance including a query, the verbal utterance being provided while the user is within the user's vehicle. At block 454, the system obtains a corresponding vehicle sensor data instance of vehicle sensor data, the corresponding vehicle sensor data instance being generated by one or more vehicle sensors of the user's vehicle. The operations of blocks 452 and 454 of method 400 of FIG. 4 may be performed in the same or similar manner as described above with respect to the operations of blocks 352 and 354 of method 300 of FIG. 3.

[0053] At block 456, the system processes the verbal utterance to identify one or more candidate responses to the query contained in the verbal utterance. For example, as indicated at block 456A, the system may cause a first search to be performed on the first corpus data to identify one or more first candidate responses to the query. The system may cause the first search to be performed on the first corpus data to identify one or more first candidate responses to the query in the same or similar manner as described above with respect to the operations of blocks 358 and 360 of the method 300 of FIG. 3. Further, as indicated at block 456B, the system may cause a second search to be performed on the second corpus data to identify one or more second candidate responses to the query. The system may cause the second search to be performed on the second corpus data to identify one or more second candidate responses to the query in the same or similar manner as described above with respect to the operations of block 364 of the method 300 of FIG. 3. In particular, in contrast to method 300 of FIG. 3, method 400 of FIG. 4 does not require a determination of whether to limit the search space before causing any searches to be performed.

[0054] Rather, in block 458, the system determines whether to bias the selection of one or more of the candidate responses toward one or more of the first candidate responses identified based on the first search on the first corpus data. In other words, the system obtains one or more first candidate responses for the query based on the first search on the first corpus data and one or more second candidate responses for the query based on the second search on the second corpus data, and then determines whether to bias the selection of one or more of the first candidate responses toward one or more of the first candidate responses based on one or more bias criteria. The one or more bias criteria utilized in making this determination are described in more detail herein (e.g., with respect to the operations of block 356 of method 300 of FIG. 3).

[0055] In the iteration of block 458, if the system determines to bias the selection of one or more of the candidate responses toward one or more of the first candidate responses identified based on the first search on the first corpus data, the system proceeds to block 460. In block 460, the system selects one of the one or more first candidate responses from among the one or more candidate responses as a given candidate response to be provided for presentation to the user. In an embodiment in which the one or more first candidate responses include multiple candidate responses, the system may rank the one or more first candidate responses and, for example, select the top-ranked first candidate response as the given candidate response. The system may rank the one or more first candidate responses using various ranking criteria described herein.

[0056] In the iteration of block 458, if the system determines not to bias the selection of one or more of the candidate responses toward one or more of the first candidate responses identified based on the first search on the first corpus data, the system proceeds to block 462. In block 462, the system selects one of the one or more first candidate responses or one or more second candidate responses from among the one or more candidate responses as a given candidate response to be provided for presentation to the user. Similarly, the system may rank the one or more first candidate responses and the one or more second responses, for example, selecting the top-ranked one of the one or more first candidate responses and the one or more second candidate responses as the given candidate response. The system may rank the one or more first candidate responses using various ranking criteria described herein.

[0057] At block 464, the system causes the given candidate responses to be provided for presentation to the user via the computing device or an additional computing device. The system may cause the given candidate responses to be provided for presentation to the user in the same or similar manner as described above with respect to the operations of block 366 of method 300 of FIG.

[0058] 5, a flow chart illustrating another exemplary method 500 for biasing an interpretation of verbal utterance(s) received in a vehicular environment is shown. For convenience, operations of method 500 are described with reference to a system that performs the operations. This system of method 500 may include at least one processor, at least one memory, and / or other component(s) of a computing device(s) (e.g., computing device(s) 110 of FIG. 1 ). 1-N 1, the biasing system 120 of FIG. 1, the computing device 710 of FIG. 7, remote server(s), and / or other computing devices. Although the operations of the method 500 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0059] At block 552, the system receives a request from a user, via a computing device, to access user manual corpus data specific to the user's vehicle. At block 554, the system accesses user manual corpus data provided by the vehicle's OEM. The computing device can be, for example, the user's mobile computing device, and the request can be directed to an automated assistant that executes at least in part on the user's mobile computing device. In some implementations, the request can be included in text data or touch data provided by the user on a display of the computing device. For example, the system can receive the request based on the user providing a text input or touch input detected through an interface of an automated assistant application. In other implementations, the request can be included in a spoken utterance received in response to an automated assistant being explicitly or implicitly invoked, as described with respect to the operation of block 352 of method 300 of FIG. 3.

[0060] For example, the verbal utterance including the request can be captured in audio data generated by one or more microphones of the computing device, and can optionally respond to an invocation of the automated assistant as described above. The system can use an automatic speech recognition (ASR) model to process the audio data capturing the verbal utterance including the request to generate ASR data of the request, such as one or more recognized terms corresponding to the request. Additionally, the system can use a natural language understanding (NLU) model to process the ASR data of the request (or text data or touch data provided by the user) to generate NLU data of the request, such as intent(s), slot value(s) for parameter(s) associated with the intent(s), and / or other NLU data. Based on the NLU data of the request, the system can determine that the verbal utterance includes a request to access user manual corpus data specific to the user's vehicle. Additionally, the system can generate and send a request to a third-party application or agent associated with the vehicle's OEM. The request may include, for example, the make and model of the user's vehicle, the year of manufacture associated with the make and model of the user's vehicle, and / or other information that may be utilized in identifying the correct user manual corpus data for the user's vehicle. In this example, the system may access this information based on user profile data from the user's user profile and / or may prompt the user to provide this information. Additionally, the system may access user manual corpus data from a third party application or agent associated with the vehicle's OEM and responsive to the submission of the request.

[0061] In block 556, the system receives from the user, via the computing device, a verbal utterance including a query directed to the user manual corpus data. In some implementations, the verbal utterance may be received in response to an automated assistant being explicitly or implicitly invoked at the computing device, as described above with respect to the operation of block 552. In addition, the system may process audio data capturing the verbal utterance using at least the ASR model and / or the NLU model to determine that the verbal utterance includes a query directed to the user manual corpus data. In particular, in various implementations, the request received in block 552 and the verbal utterance received in block 556 may be received from the user as a single request.

[0062] In block 558, the system causes a search to be performed against at least the user manual corpus data to identify one or more candidate responses to the query contained in the verbal utterance, where the search against at least the user manual corpus data is based on the query and does not utilize any corresponding vehicle sensor data instances of the vehicle sensor data generated by one or more sensors of the vehicle. For example, assume that the user's vehicle is 1994 or later. In this example, the vehicle is unlikely to have an on-board computing device that can communicate with the system. As a result, the system may not be able to utilize the corresponding sensor data instances when causing a search to be performed against the user manual corpus data to identify one or more candidate responses to the query. Nevertheless, an automated assistant running on the user's mobile computing device may access the user manual corpus data of the vehicle to enable the user to query the user manual corpus data via the automated assistant.

[0063] At block 560, the system causes a given candidate response to be provided from among the one or more candidate responses for presentation to the user via the computing device. In some implementations, the system may select the given candidate response from among the one or more candidate responses based on a ranking of the one or more candidate responses. The system may rank the one or more candidate responses based on one or more ranking criteria. The one or more ranking criteria may include, for example, one or more terms of the query, a corresponding vehicle sensor data instance, one or more user context signals characterizing a state of a user of the vehicle, one or more vehicle context signals characterizing a state of the vehicle, application data associated with one or more applications accessible at the computing device or the additional computing device, and / or other criteria. Further, the system may select the given candidate response based on the ranking and cause the given candidate response to be provided for audible and / or visual presentation to the user via the computing device or the additional computing device.

[0064] It should be understood that while method 500 of FIG. 5 is described solely with respect to performing a search against user manual corpus data, this is for illustrative purposes and is not intended to be limiting. For example, the system may further perform additional searches against additional corpus data that is not vehicle specific. In these implementations, a given candidate response may be further selected from among any other candidate responses determined based on the search against the additional corpus data.

[0065] 6A and 6B, various non-limiting examples of computing devices illustrating various user interactions that bias speech processing of oral utterances in a vehicle environment are shown. With specific reference to FIG. 6A, the in-vehicle computing device 110 from FIG. N The in-vehicle computing device 110N 6 shows a display 620 having multiple different portions targeted for different applications. N For example, the display 620 of FIG. N is at least partially implemented by the in-vehicle computing device 110 N A display 620 for an automated assistant application running on the N First part 622 N and at least partially on-board computing device 110 N Display 620 for OEM applications in vehicle 100A N The second part 624 N and a display 620 directed to a third-party media application associated with an exemplary music streaming service. N The third part of 626 N The in-vehicle computing device 110 N Display 620 N Although the display 620 is shown in FIG. 6A as having a particular configuration (e.g., multiple different portions targeted to various applications), it should be understood that this is for illustrative purposes and is not intended to be limiting. N may be configured in any desired manner and may be specific to a particular OEM.

[0066] By way of example, assume that a corresponding vehicle sensor data instance corresponding to the tire pressure data instance of the tire pressure data is generated by one or more tire sensors of the vehicle 100A and indicates low tire pressure in one or more tires of the vehicle 100A. N The second part 624 N But based on tire pressure data instance 624A alerts "Tire pressure is low" N, displayed for presentation to the user. Further, assume that the user of the vehicle 100A invokes the automated assistant and receives the query 622A, “What’s under pressure?” N In this example, audio data capturing the verbal utterance is processed to determine whether the verbal utterance is a query 622A. N It can be determined that the set of variables includes:

[0067] Further, using various biasing techniques described herein (e.g., with respect to FIGS. 3 and 4), the automated assistant may select a query 622A N When providing the alert 624A N In particular, the automated assistant may determine that the user is not submitting a general query, but is instead requesting an explanation for the alert 624A. N For example, query 622A N and Alert 624A N The temporal relationship between (e.g., Alert 624A N is provided for presentation to the user within a threshold duration. N Query 622A N and Alert 624A N The linguistic relationship between (e.g., query 622A N and Alert 624A N and the like), a temporal relationship between the user and vehicle 100A (e.g., based on how long the user has owned vehicle 100A, based on the distance the user has driven vehicle 100A, based on the number of times the user has started the vehicle, etc.), and / or other biasing criteria.

[0068] Thus, in the example of FIG. 6A, the automated assistant would respond to query 622A N3. The search space for identifying one or more candidate responses to "the tire pressure is 28 psi, but according to the user manual it should be 32 psi" may be restricted to user manual corpus data as described with respect to FIG. N may be provided for visual (as shown in FIG. 6A) and / or audible presentation to the user. Additionally or alternatively, in the example of FIG. 6A, the automated assistant may provide query 622A N Both user manual corpus data and additional corpus data (e.g., the Internet, other applications, other databases, etc.) may be searched to identify one or more candidate responses to one or more given candidate responses 622B, as described with respect to FIG. N The selection of query 622A may be biased toward one or more candidate responses obtained using user manual corpus data. In these additional or alternative examples, the automated assistant may also N 622A, but may be related to N For example, in various embodiments, the automated assistant may generate a notification 626A that reads, "Click here to play Under Pressure by Queen." N , Display 620 for third-party media applications N The third part of 626 N can be further presented to the user via

[0069] With specific reference to Figure 6B, the in-vehicle computing device 1101 from Figure 1 is shown. While the computing device 1101 is shown as a mobile computing device of a user of the vehicle 100A, it should be understood that this is for purposes of illustration and not limitation. The computing device 1101 includes a display 6201 having various system interface elements 6811, 6821, and 6831 (e.g., hardware and / or software interface elements) that can be interacted with by a user and that cause the computing device 1101 to perform one or more actions. The display 6201 of the computing device 1101 allows a user to interact with the content displayed on the display 6201 by typing or touch input (e.g., by directing user input to the display 6201 or text interface element 6841) and / or by verbal input (e.g., by selecting the microphone interface element 6851 on the computing device 1101, or by simply speaking without necessarily selecting the microphone interface element 6851 (i.e., the automated assistant may monitor one or more terms or phrases, gesture(s), gaze(s), mouth movement(s), lip movement(s), and / or other conditions that enable verbal input)). Additionally, the automated assistant application may be implemented at least in part on the computing device 1101, as indicated by 6221.

[0070] For example, assume that the user of the vehicle 100A has already provided the automated assistant with a request to access user manual corpus data associated with the vehicle 100A. Further assume that the user has provided an input 622A1 (e.g., via spoken utterance or typed input) of “What is preconditioning?” In this example, the automated assistant may perform a search against the user manual corpus data provided by the vehicle's OEM to identify a given candidate response 622B1 of “In the context of a vehicle, preconditioning allows a vehicle to be preheated or precooled before entering the vehicle's cabin” without utilizing any vehicle sensor data instances of the vehicle sensor data. Absent the techniques described herein, the automated assistant may not return any definition in the context of the vehicle 100A. Nevertheless, other definitions of “preconditioning” may be provided to the user, as indicated by a notification 622C1 of “Click here for other definitions of preconditioning.” In particular, the example implementation of Figure 6B is particularly advantageous in the absence of an in-vehicle computing device as described with respect to Figure 6A and / or in the absence of the in-vehicle computing device 110N capable of running an automated assistant. It should be understood that Figures 6A and 6B are described with respect to specific examples and are not intended to be limiting.

[0071] 7, a block diagram of an example computing device 710 that may optionally be utilized to perform one or more aspects of the techniques described herein is shown. In some implementations, one or more computing devices, one or more vehicles, and / or other component(s) may make up one or more components of the example computing device 710.

[0072] Computing device 710 typically includes at least one processor 714 that communicates with a number of peripheral devices via a bus subsystem 712. These peripheral devices may include a storage subsystem 724, including, for example, a memory subsystem 725 and a file storage subsystem 726, user interface output devices 720, user interface input devices 722, and a network interface subsystem 716. The input and output devices enable user interaction with computing device 710. The network interface subsystem 716 provides an interface to external networks and is connected to corresponding interface devices in other computing devices.

[0073] The user interface input devices 722 may include a keyboard, a mouse, a trackball, a touchpad, a pointing device such as a graphics tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 710 or a communications network.

[0074] The user interface output devices 720 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 710 to a user or to another machine or computing device.

[0075] Storage subsystem 724 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 724 may include logic to perform selected aspects of the methods disclosed herein, as well as logic to implement the various components illustrated in Figures 1 and 2.

[0076] These software modules are typically executed by the processor 714 alone or in combination with other processors. The memory 725 used by the storage subsystem 724 may include a number of memories, such as a main random access memory (RAM) 730 for storing instructions and data during program execution, and a read-only memory (ROM) 732 in which fixed instructions are stored. The file storage subsystem 726 may provide persistent storage of program files and data files, and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of an embodiment may be stored by the file storage subsystem 726 in the storage subsystem 724 or on other machines accessible by the processor(s) 714.

[0077] Bus subsystem 712 provides a mechanism that allows the various components and subsystems of computing device 710 to communicate with each other as intended. Although bus subsystem 712 is illustrated generally as a single bus, alternative implementations of bus subsystem 712 may use multiple buses.

[0078] Computing device 710 can be of many different types, such as a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 710 shown in Figure 7 is intended only as a specific example to illustrate some implementations. Many other configurations of computing device 710 can have more or fewer components than the computing device shown in Figure 7.

[0079] In situations where the systems described herein may collect or monitor personal information about a user or utilize personal and / or monitored information, the user may be provided with an opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social behavior or activity, occupation, user preferences, or the user's current geographic location) or whether and / or how content that may be more relevant to the user is received from a content server. Additionally, certain data may be processed in one or more ways such that personally identifiable information is removed before it is stored or used. For example, the user's identity may be processed such that personally identifiable information about the user cannot be determined, or if geographic location information is obtained (such as down to the city, zip code, or state level), the user's geographic location may be generalized such that the user's specific geographic location cannot be determined. Thus, the user may control how information about the user is collected and / or used.

[0080] In some embodiments, a method implemented by one or more processors is provided, the method comprising: receiving, from a user, via a computing device, an oral utterance including a query, the oral utterance being provided while the user is located within the user's vehicle; acquiring corresponding vehicle sensor data instances of vehicle sensor data, the corresponding vehicle sensor data instances being generated by one or more vehicle sensors of the user's vehicle; and performing a first search against first corpus data based on (i) the query and / or (ii) the corresponding vehicle sensor data instances to identify one or more candidate responses to the query included in the oral utterance. determining whether to perform a search and / or a second search on the second corpus data; and in response to determining to perform the first search on the first corpus data but not the second search on the second corpus data, causing a first search to be performed on the first corpus data to identify one or more candidate responses to a query included in the verbal utterance, the first search being based on (i) the query and (ii) corresponding vehicle sensor data instances; and providing a given candidate response from among the one or more candidate responses for presentation to a user via the computing device or an additional computing device.

[0081] These and other embodiments disclosed herein can optionally include one or more of the following features.

[0082] In some implementations, the method may further include processing the audio data capturing the verbal utterance including the query with an automatic speech recognition (ASR) model to generate ASR data for the query, and processing the ASR data with a natural language understanding (NLU) model to generate NLU data for the query. In some versions of these implementations, causing a first search against a first corpus of data to identify one or more candidate responses to the query included in the verbal utterance may include submitting the NLU data for the query and an indication of the corresponding vehicle sensor data instances to the first corpus data, performing the first search against the first corpus data, and identifying one or more candidate responses based on content responsive to the NLU data for the query.

[0083] In some implementations, the first corpus data may correspond to vehicle-specific user manual corpus data provided by an original equipment manufacturer (OEM) of the vehicle, and the second corpus data may correspond to additional corpus data that is not vehicle-specific.

[0084] In some implementations, determining to perform a first search on the first corpus data but not to perform a second search on the second corpus data may include identifying a corresponding timestamp associated with a corresponding vehicle sensor data instance, where the corresponding timestamp associated with the corresponding vehicle sensor data instance corresponds to a time the corresponding vehicle sensor data instance was generated, determining whether the verbal utterance including the query was received within a threshold duration with respect to the time the corresponding vehicle sensor data instance was generated based on the corresponding timestamp associated with the corresponding vehicle sensor data instance, and determining to perform the first search on the first corpus data but not to perform the second search on the second corpus data in response to determining that the verbal utterance including the query was received within the threshold duration. In some versions of these implementations, the method may further include determining to perform the first search on the first corpus data and the second search on the second corpus data in response to determining that the verbal utterance including the query was received within the threshold duration. In some further versions of these embodiments, the method may further include, in response to determining to perform a first search on the first corpus data and a second search on the second corpus data, causing a first search to be performed on the first corpus data to identify one or more candidate responses to the query comprised in the verbal utterance, the first search being based on (i) the query and (ii) corresponding vehicle sensor data instances, and causing a second search to be performed on the second corpus data to identify one or more candidate responses to the query comprised in the verbal utterance, the second corpus data being added to the first corpus data, the second search being based on (i) the query but not based on (ii) the corresponding vehicle sensor data instances.In further versions of these embodiments, the method may further include selecting a given candidate response to be provided for presentation to a user via the computing device or the additional computing device based on a ranking of the one or more candidate responses. In further versions of these embodiments, the ranking of the one or more candidate responses is based on one or more of the one or more terms of the query, the corresponding vehicle sensor data instances, the one or more user context signals characterizing a state of a user of the vehicle, the one or more vehicle context signals characterizing a state of the vehicle, and application data associated with one or more applications accessible at the computing device or the additional computing device.

[0085] In some implementations, determining to perform a first search on the first corpus data but not a second search on the second corpus data may include determining that one or more terms of a query included in the verbal utterance are associated with a corresponding vehicle sensor data instance. In some versions of these implementations, the corresponding vehicle sensor data instance, once generated, may cause the computing device or the additional computing device to provide an indication of the corresponding vehicle sensor data instance for presentation to the user. In still some versions of these implementations, determining that one or more terms of a query included in the verbal utterance are associated with the corresponding vehicle sensor data instance may include determining that one or more terms of a query included in the verbal utterance are directed to an indication of the corresponding vehicle sensor data instance that is provided for presentation to the user via the computing device or the additional computing device.

[0086] In some implementations, the method may further include determining a duration for which the user is associated with the vehicle based on processing the verbal utterance including the query. Determining a duration for which the user is associated with the vehicle may include causing speaker identification to be performed based on processing the verbal utterance to determine if the user is a known user, and determining a duration for which the user is associated with the vehicle based on a user profile associated with the user in response to determining that the user is a known user. In some versions of these implementations, determining to perform a first search on the first corpus data but not a second search on the second corpus data may include determining whether a duration for which the user is associated with the vehicle meets a threshold duration, and determining to perform a first search on the first corpus data but not a second search on the second corpus in response to determining that the duration for which the user is associated with the vehicle does not meet the threshold duration. In further versions of these embodiments, the method may further include, in response to determining that the duration the user is associated with the vehicle meets the threshold duration, causing a first search to be performed on the first corpus data to identify one or more of candidate responses to the query comprised in the verbal utterance based on (i) the query and (ii) the corresponding vehicle sensor data instances, and causing a second search to be performed on second corpus data to identify one or more of candidate responses to the query comprised in the verbal utterance based on (i) the query, where the second corpus data adds to the first corpus data. In further versions of these embodiments, the method may further include selecting a given candidate response to be provided for presentation to the user via the computing device or an additional computing device based on a ranking of the one or more candidate responses.

[0087] In some embodiments, a method implemented by one or more processors is provided, the method including: receiving, from a user via a computing device, a verbal utterance including a query, the verbal utterance being provided by the user while located within the user's vehicle; obtaining a corresponding vehicle sensor data instance of vehicle sensor data, the corresponding vehicle sensor data instance being generated by one or more vehicle sensors of the user's vehicle; and processing the verbal utterance to identify a plurality of candidate responses to the query included in the verbal utterance. Processing the verbal utterance to identify a plurality of candidate responses to the query included in the verbal utterance includes: performing a first search on first corpus data to identify one or more first candidate responses of the plurality of candidate responses to the query included in the verbal utterance, the first search being based on (i) the query and (ii) a corresponding vehicle sensor data instance; and performing a second search on second corpus data to identify one or more second candidate responses of the plurality of candidate responses to the query included in the verbal utterance, the second corpus data being added to the first corpus data, the second search being based on (i) the query but not based on (ii) the corresponding vehicle sensor data instance. The method further includes selecting a given candidate response from among the plurality of candidate responses and providing the given candidate response for presentation to a user via the computing device or an additional computing device.

[0088] These and other embodiments disclosed herein can optionally include one or more of the following features.

[0089] In some implementations, the method may further include processing the audio data capturing the verbal utterance including the query with an automatic speech recognition (ASR) model to generate ASR data for the query, and processing the ASR data with a natural language understanding (NLU) model to generate NLU data for the query. In some versions of these implementations, performing a first search against the first corpus data to identify one or more first candidate responses to the query included in the verbal utterance may include submitting the NLU data of the query and an indication of corresponding vehicle sensor data instances to the first corpus data, performing the first search against the first corpus data, and identifying one or more first candidate responses based on the first content responsive to the NLU data of the query.

[0090] In some further versions of these embodiments, performing a second search against the second corpus data to identify one or more second candidate responses to the query contained in the verbal utterance may include submitting NLU data of the query to the second corpus data and performing a second search against the second corpus data, and identifying one or more second candidate responses based on second content responsive to the NLU data of the query.

[0091] In further additional or alternative versions of these embodiments, the first corpus data may correspond to vehicle-specific user manual corpus data provided by the vehicle's original equipment manufacturer (OEM), and the second corpus data may correspond to non-vehicle-specific web-based corpus data.

[0092] In still additional or alternative versions of these embodiments, selecting a given candidate response from among the multiple candidate responses may include ranking the multiple candidate responses, biasing the ranking of the multiple candidate responses toward one or more first candidate responses, and selecting the given candidate response from among the multiple candidate responses in the biased ranking of the multiple candidate responses. In further versions of these embodiments, the ranking of the multiple candidate responses is based on one or more terms of the query, the corresponding vehicle sensor data instances, one or more user context signals characterizing a state of a user of the vehicle, one or more vehicle context signals characterizing a state of the vehicle, application data associated with one or more applications accessible at the computing device or the additional computing device.

[0093] In some embodiments, a method implemented by one or more processors is provided, the method comprising: receiving, from a user, via a computing device, a verbal utterance including a query, the verbal utterance being received while the user is located within the user's vehicle; acquiring a corresponding vehicle sensor data instance of vehicle sensor data, the corresponding vehicle sensor data instance being generated by one or more vehicle sensors of the user's vehicle; determining a duration for which the user is associated with the vehicle based on processing the verbal utterance including the query; and determining one or more candidate responses to the query included in the verbal utterance based on (i) the query, (ii) the corresponding vehicle sensor data instance, and / or (iii) the user sensor data instance. determining whether to perform a first search on the first corpus data and / or a second search on the second corpus data based on the duration the user is associated with the vehicle not meeting a temporal threshold; and in response to determining to perform the first search on the first corpus data but not to perform the second search on the second corpus data, causing a first search to be performed on the first corpus data to identify one or more candidate responses to the query comprised in the oral utterance based on (i) the query and (ii) corresponding vehicle sensor data instances; and causing a given candidate response from among the one or more candidate responses to be provided for presentation to the user via the computing device or the additional computing device.

[0094] In some implementations, a method implemented by one or more processors is provided, the method comprising: receiving, from a user, via a computing device, a verbal utterance including a query, the verbal utterance being provided while the user is located within the user's vehicle; acquiring corresponding vehicle sensor data instances of vehicle sensor data, the corresponding vehicle sensor data instances being generated by one or more vehicle sensors of the user's vehicle; and identifying corresponding timestamps associated with the corresponding vehicle sensor data instances, the corresponding timestamps associated with the corresponding vehicle sensor data instances corresponding to times at which the corresponding vehicle sensor data instances were generated; and identifying one or more candidate responses to the query included in the verbal utterance, the method comprising: determining whether to perform a first search on the first corpus data and / or a second search on the second corpus data based on (ii) the corresponding vehicle sensor data instance and / or (iii) a corresponding timestamp associated with the corresponding vehicle sensor data instance; and in response to determining to perform the first search on the first corpus data but not the second search on the second corpus data, causing a first search to be performed on the first corpus data to identify one or more candidate responses to a query comprised in the oral utterance based on (i) the query and / or (ii) the corresponding vehicle sensor data instance; and causing a given candidate response from among the one or more candidate responses to be provided for presentation to a user via the computing device or an additional computing device.

[0095] In some embodiments, a method implemented by one or more processors is provided, the method including receiving, from a user, via a computing device, a request to access user manual corpus data specific to the user's vehicle; in response to receiving the request to access the user manual corpus data specific to the vehicle, accessing user manual corpus data from an original equipment manufacturer (OEM) of the vehicle; receiving, from the user, via the computing device, an oral utterance including a query directed to the user manual corpus data; causing a search to be performed against the user manual corpus data to identify one or more candidate responses to the query included in the oral utterance, the search against the user manual corpus data being (i) based on the query but (ii) not utilizing any corresponding vehicle sensor data instances of vehicle sensor data generated by one or more vehicle sensors of the vehicle; and causing a given candidate response from among the one or more candidate responses to be provided for presentation to the user via the computing device or an additional computing device.

[0096] In some embodiments, a method implemented by one or more processors is provided, the method including: receiving, from a user via a computing device, a verbal utterance including a query, the verbal utterance being provided by the user while located within the user's vehicle; obtaining a corresponding vehicle sensor data instance of vehicle sensor data, the corresponding vehicle sensor data instance being generated by one or more vehicle sensors of the user's vehicle; and processing the verbal utterance to identify a plurality of candidate responses to the query included in the verbal utterance. Processing the verbal utterance to identify a plurality of candidate responses to a query included in the verbal utterance includes: causing a first search to be performed on a first corpus data to identify one or more first candidate responses of the plurality of candidate responses to the query included in the verbal utterance, the first search being based on at least (i) the query; and causing a second search to be performed on a second corpus data to identify one or more second candidate responses of the plurality of candidate responses to the query included in the verbal utterance, the second corpus data being added to the first corpus data, the second search being based on (i) the query. The method further includes (ii) selecting a given candidate response from among the plurality of candidate responses based on a corresponding vehicle sensor data instance, the selecting being biased toward the one or more first candidate responses based on the corresponding vehicle sensor data instance, and causing the given candidate response to be provided for presentation to a user via the computing device or an additional computing device.

[0097] Further, some embodiments include one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), and / or tensor processing unit(s) (TPU(s)) of one or more computing devices, the one or more processors operable to execute instructions stored in associated memory, the instructions configured to perform any of the methods described above. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by the one or more processors to perform any of the methods described above. Some embodiments also include a computer program product including instructions executable by the one or more processors to perform any of the methods described above.

[0098] It will be understood that all combinations of the above concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

Claims

1. 1. A method implemented by one or more processors, comprising: receiving, from a user via a computing device, a verbal utterance including a query, the verbal utterance being provided while the user is within the user's vehicle; acquiring a corresponding vehicle sensor data instance of vehicle sensor data, the corresponding vehicle sensor data instance being generated by one or more vehicle sensors of the vehicle of the user; determining whether to perform a first search on a first corpus data and / or a second search on a second corpus data to identify one or more candidate responses to the query comprised in the verbal utterance based on (i) the query and / or (ii) the corresponding vehicle sensor data instances; in response to determining to perform the first search on the first corpus data but not to perform the second search on the second corpus data; performing a first search against the first corpus data to identify the one or more candidate responses to the query contained in the verbal utterance, the first search being based on (i) the query and (ii) the corresponding vehicle sensor data instances; providing, via the computing device or an additional computing device, a given candidate response from among the one or more candidate responses for presentation to the user; The method comprising:

2. processing the captured audio data including the query using an automatic speech recognition (ASR) model to generate ASR data for the query; processing the ASR data using a natural language understanding (NLU) model to generate NLU data for the query; The method of claim 1 further comprising:

3. performing the first search against the first corpus data to identify the one or more candidate responses to the query contained in the oral utterance, comprising: submitting the NLU data of the query and an indication of the corresponding vehicle sensor data instance to the first corpus data to perform the first search against the first corpus data; identifying the one or more candidate responses based on content responsive to the NLU data of the query; The method of claim 2 , comprising:

4. the first corpus data corresponds to user manual corpus data specific to the vehicle and provided by an original equipment manufacturer (OEM) of the vehicle; The method of any of claims 1 to 3, wherein the second corpus of data corresponds to additional corpus of data that is not specific to the vehicle.

5. Determining to perform the first search on the first corpus data but not the second search on the second corpus data includes: identifying a corresponding timestamp associated with the corresponding vehicle sensor data instance, the corresponding timestamp associated with the corresponding vehicle sensor data instance corresponding to a time at which the corresponding vehicle sensor data instance was generated; determining, based on the corresponding timestamp associated with the corresponding vehicle sensor data instance, whether the verbal utterance including the query was received within a threshold duration relative to a time when the corresponding vehicle sensor data instance was generated; in response to determining that the spoken utterance including the query was received within the threshold duration; determining to perform the first search on the first corpus data but not to perform the second search on the second corpus data; The method according to any one of claims 1 to 4, comprising:

6. in response to determining that the spoken utterance including the query was received within the threshold duration; determining to perform the first search on the first corpus data and to perform the second search on the second corpus data; The method of claim 5 further comprising:

7. in response to determining to perform the first search on the first corpus data and to perform the second search on the second corpus data; performing a first search against the first corpus data to identify one or more of the candidate responses to the query included in the oral utterance, the first search being based on (i) the query and (ii) the corresponding vehicle sensor data instances; performing the second search on the second corpus data to identify one or more of the candidate responses to the query included in the oral utterance, the second corpus data being appended to the first corpus data, the second search being (i) based on the query, but (ii) not based on the corresponding vehicle sensor data instances; The method of claim 6 further comprising:

8. 8. The method of claim 7, further comprising: selecting the given candidate response to be provided for presentation to the user via the computing device or an additional computing device based on a ranking of the one or more candidate responses.

9. The ranking of the one or more candidate responses comprises: one or more terms of the query; the corresponding vehicle sensor data instances; one or more user context signals characterizing a state of the user of the vehicle; one or more vehicle context signals characterizing a state of the vehicle; or Application data associated with one or more applications accessible on the computing device or the additional computing device. The method of claim 8, based on one or more of:

10. Determining to perform the first search on the first corpus data but not the second search on the second corpus data includes: The method of any of claims 1 to 4, comprising determining that one or more terms of the query contained in the spoken utterance are relevant to the corresponding vehicle sensor data instance.

11. 11. The method of claim 10, wherein the corresponding vehicle sensor data instance, once generated, causes the computing device or the additional computing device to provide an indication of the corresponding vehicle sensor data instance for presentation to the user.

12. Determining that one or more of the terms of the query included in the verbal utterance are associated with the corresponding vehicle sensor data instance includes:

12. The method of claim 11, comprising determining that one or more terms of the query included in the verbal utterance are directed to the indication of the corresponding vehicle sensor data instance provided for presentation to the user via the computing device or the additional computing device.

13. determining a duration for which the user is associated with the vehicle based on processing the oral utterance including the query, the determining the duration for which the user is associated with the vehicle comprising: performing speaker identification based on processing the oral utterance to determine if the user is a known user; in response to determining that the user is a known user, determining the duration that the user has been associated with the vehicle based on a user profile associated with the user; The method of any one of claims 1 to 4, further comprising:

14. Determining to perform the first search on the first corpus data but not the second search on the second corpus data includes: determining whether the duration that the user is associated with the vehicle meets a threshold duration; in response to determining that the duration the user is associated with the vehicle does not meet the threshold duration; determining to perform the first search on the first corpus data but not to perform the second search on the second corpus data; The method of claim 13, comprising:

15. in response to determining that the duration the user is associated with the vehicle satisfies the threshold duration; performing the first search against the first corpus data to identify one or more of the candidate responses to the query contained in the verbal utterance based on (i) the query and (ii) the corresponding vehicle sensor data instances; (i) performing the second search on the second corpus data to identify one or more of the candidate responses to the query contained in the oral utterance based on the query, the second corpus data being appended to the first corpus data; The method of claim 14 further comprising:

16. 16. The method of claim 15, further comprising: selecting the given candidate response to be provided for presentation to the user via the computing device or an additional computing device based on a ranking of the one or more candidate responses.

17. 1. A method implemented by one or more processors, comprising: receiving, from a user via a computing device, a verbal utterance including a query, the verbal utterance being provided while the user is within the user's vehicle; acquiring a corresponding vehicle sensor data instance of vehicle sensor data, the corresponding vehicle sensor data instance being generated by one or more vehicle sensors of the vehicle of the user; processing the spoken utterance to identify a plurality of candidate responses to the query contained in the spoken utterance, the processing of the spoken utterance to identify the plurality of candidate responses to the query contained in the spoken utterance comprising: performing a first search on first corpus data to identify one or more first candidate responses of the plurality of candidate responses to the query included in the oral utterance, the first search being based on (i) the query and (ii) the corresponding vehicle sensor data instances; performing a second search on second corpus data to identify one or more second candidate responses of the plurality of candidate responses to the query included in the oral utterance, the second corpus data being appended to the first corpus data, the second search being (i) based on the query, but (ii) not based on the corresponding vehicle sensor data instance; processing the oral utterance, selecting a given candidate response from among the plurality of candidate responses; providing, via the computing device or an additional computing device, the given candidate responses for presentation to the user; The method comprising:

18. processing the captured audio data including the query using an automatic speech recognition (ASR) model to generate ASR data for the query; processing the ASR data using a natural language understanding (NLU) model to generate NLU data for the query; 20. The method of claim 17, further comprising:

19. Performing the first search against the first corpus data to identify the one or more first candidate responses to the query contained in the oral utterance includes: submitting the NLU data of the query and an indication of the corresponding vehicle sensor data instance to the first corpus data to perform the first search against the first corpus data; identifying the one or more first candidate responses based on first content responsive to the NLU data of the query; 20. The method of claim 18, comprising:

20. performing the second search against the second corpus data to identify the one or more second candidate responses to the query contained in the oral utterance, submitting the NLU data of the query to the second corpus data and performing the second search against the second corpus data; identifying the one or more second candidate responses based on second content responsive to the NLU data of the query; 20. The method of claim 19, comprising:

21. the first corpus data corresponds to user manual corpus data specific to the vehicle and provided by an original equipment manufacturer (OEM) of the vehicle; The method of any of claims 17 to 19, wherein the second corpus of data corresponds to non-vehicle specific web-based corpus of data.

22. selecting the given candidate response from among the plurality of candidate responses; ranking the plurality of candidate responses; and biasing the ranking of the plurality of candidate responses toward the one or more first candidate responses; selecting the given candidate response from among the plurality of candidate responses based on the biased ranking of the plurality of candidate responses; 20. The method of claim 19, comprising:

23. The ranking of the plurality of candidate responses comprises: one or more terms of the query; the corresponding vehicle sensor data instances; one or more user context signals characterizing a state of the user of the vehicle; one or more vehicle context signals characterizing a state of the vehicle; or Application data associated with one or more applications accessible on the computing device or the additional computing device.

23. The method of claim 22, based on one or more of:

24. 1. A method implemented by one or more processors, comprising: receiving, from a user via a computing device, a verbal utterance including a query, the verbal utterance being received while the user is within the user's vehicle; acquiring a corresponding vehicle sensor data instance of vehicle sensor data, the corresponding vehicle sensor data instance being generated by one or more vehicle sensors of the vehicle of the user; determining a duration that the user has been associated with the vehicle based on processing the verbal utterance including the query; and determining whether to perform a first search on first corpus data and / or a second search on second corpus data to identify one or more candidate responses to the query contained in the verbal utterance based on (i) the query, (ii) the corresponding vehicle sensor data instance, and / or (iii) the duration that the user was associated with the vehicle not meeting a temporal threshold; in response to determining to perform the first search on the first corpus data but not to perform the second search on the second corpus data; performing the first search against the first corpus data to identify the one or more candidate responses to the query contained in the verbal utterance based on (i) the query and (ii) the corresponding vehicle sensor data instances; providing, via the computing device or an additional computing device, a given candidate response from among the one or more candidate responses for presentation to the user; The method comprising:

25. 1. A method implemented by one or more processors, comprising: receiving, from a user via a computing device, a verbal utterance including a query, the verbal utterance being provided while the user is within the user's vehicle; acquiring a corresponding vehicle sensor data instance of vehicle sensor data, the corresponding vehicle sensor data instance being generated by one or more vehicle sensors of the vehicle of the user; identifying a corresponding timestamp associated with the corresponding vehicle sensor data instance, the corresponding timestamp associated with the corresponding vehicle sensor data instance corresponding to a time at which the corresponding vehicle sensor data instance was generated; determining whether to perform a first search on a first corpus data and / or a second search on a second corpus data to identify one or more candidate responses to the query comprised in the verbal utterance based on (i) the query, (ii) the corresponding vehicle sensor data instance, and / or (iii) the corresponding timestamp associated with the corresponding vehicle sensor data instance; in response to determining to perform the first search on the first corpus data but not to perform the second search on the second corpus data; performing the first search against the first corpus data to identify the one or more candidate responses to the query contained in the verbal utterance based on (i) the query and / or (ii) the corresponding vehicle sensor data instances; providing, via the computing device or an additional computing device, a given candidate response from among the one or more candidate responses for presentation to the user; The method comprising:

26. 1. A method by one or more processors, comprising: receiving a request from a user, via a computing device, to access user manual corpus data specific to the user's vehicle; accessing the user manual corpus data from an original equipment manufacturer (OEM) of the vehicle in response to receiving the request to access the user manual corpus data specific to the vehicle; receiving, from the user via the computing device, a verbal utterance including a query directed to the user manual corpus data; performing a search against the user manual corpus data to identify one or more candidate responses to the query comprised in the oral utterance, the search against the user manual corpus data (i) being based on a query and (ii) not utilizing any corresponding vehicle sensor data instances of vehicle sensor data generated by one or more sensors of the vehicle; providing, via the computing device or an additional computing device, a given candidate response from among the one or more candidate responses for presentation to the user; The method comprising:

27. 1. A method implemented by one or more processors, comprising: receiving, from a user via a computing device, a verbal utterance including a query, the verbal utterance being provided while the user is within the user's vehicle; acquiring a corresponding vehicle sensor data instance of vehicle sensor data, the corresponding vehicle sensor data instance being generated by one or more vehicle sensors of the vehicle of the user; processing the spoken utterance to identify a plurality of candidate responses to the query contained in the spoken utterance, the processing of the spoken utterance to identify the plurality of candidate responses to the query contained in the spoken utterance comprising: performing a first search on first corpus data to identify one or more first candidate responses of the plurality of candidate responses to the query included in the oral utterance, the first search being based at least on (i) the query; performing a second search on second corpus data to identify one or more second candidate responses of the plurality of candidate responses to the query included in the oral utterance, the second corpus data being added to the first corpus data, the second search being based on (i) the query; processing the oral utterance, (ii) selecting a given candidate response from among the plurality of candidate responses based on the corresponding vehicle sensor data instance, the selecting including biasing the one or more candidate responses toward a first candidate response based on the corresponding vehicle sensor data instance; providing, via the computing device or an additional computing device, the given candidate responses for presentation to the user; The method comprising:

28. one or more processors; a memory storing instructions which, when executed, cause said one or more processors to perform the operations recited in any one of claims 1 to 27; Including, the system.

29. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform the operations recited in any of claims 1-27.

30. one or more processors; a memory storing instructions which, when executed, cause said one or more processors to perform the operations recited in any one of claims 1 to 27; an in-vehicle computing device comprising:

Images