Guiding vehicle client device to use functionality on device

Abstract

The invention discloses guiding a vehicle client device to use functionality on the device. Implementations set forth herein relate to ensuring useful responsiveness of any versions of vehicle computing devices that are still in operation while gradually eliminating versions of vehicle computing devices. Due to hardware limitations, certain features of the updated computing device may be unavailable to a previous version of the computing device. Implementations set forth herein eliminate crashed and wasted data transmissions caused by previous versions of computing devices that have not yet been upgraded or cannot be upgraded. The server device may respond to a particular intent request provided to the vehicle computing device despite the intent request being associated with an action that cannot be performed by a particular version of the vehicle computing device. In response, the server device may select to provide voice to text data and / or natural language understanding data to facilitate allowing the vehicle computing device to continue to utilize resources at the server device.

Description

Description of the case

[0001] This application is a divisional application of Chinese invention patent application No. 201980091340.0, filed on February 12, 2019. Technical Field

[0002] The present disclosure relates to directing a vehicle client device to use a function on the device. Background Art

[0003] Humans may engage in human-computer conversations with interactive software applications, referred to herein as "automated assistants" (also referred to as "digital agents," "chat programs," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," etc.). For example, humans (who may be referred to as "users" when they interact with the automated assistant) may provide commands and / or requests to the automated assistant using spoken natural language input (i.e., utterances) and / or by providing textual (e.g., typed) natural language input, which may in some cases be converted to text and then processed.

[0004] The automated assistant may be installed at a variety of different devices, such as, for example, mobile phones, smart home devices, and / or vehicles. Unlike mobile phones and other computing devices, vehicles can often be used by respective owners for an extended period of time (e.g., ten or more years) before the owner ultimately decides to purchase a replacement vehicle. During this period of ownership of the vehicle, the software installed at the vehicle may be subject to updates. For example, updates may be provided to the vehicle computing device to allow the vehicle computing device to respond to commands that can be processed by updated smart home devices and / or updated mobile phones. However, a user may choose not to install certain updates, resulting in incompatibility between the vehicle computing device and a remote server device, with which the vehicle computing device interacts in response to commands. In addition, after a period of time (e.g., three or more years), updates may no longer be provided for the vehicle computing device due to, for example, the end of the support life, the inability of the hardware of the vehicle computing device to execute new updates, and / or other factors. This may also result in incompatibility between the vehicle computing device and the remote server device. When the vehicle computing device becomes incompatible with the remote server device, the server device may respond to requests from the vehicle computing device that can no longer be interpreted by the vehicle computing device. This may result in the vehicle computing device failing to respond appropriately to various commands, and in the vehicle computing device wastefully transmitting various data to the server device and / or the server device wastefully transmitting various data to the vehicle computing device (because some server device responses will no longer be interpretable by the vehicle computing device). Some techniques attempt to address this issue by making the server device compatible with recent updates as well as previous versions of the vehicle computing device. However, providing such backward compatibility indefinitely may require significant storage, memory, and / or processor usage at the server device. Summary of the invention

[0005] The implementations described herein relate to techniques for processing spoken utterances received at a vehicle computing device, which techniques—although including operable software—correspond to versions that are gradually phased out by a server device, and / or any other support system. Gradually phasing out can result in a range of support for various server operations of hardware and / or software versions. These server operations may include, but are not limited to, speech-to-text processing, natural language understanding (e.g., intent identification and / or slot value identification), action generation, and / or action execution. Based on the version of hardware and / or software becoming obsolete relative to other newly released hardware and / or software, the server device may operate to gradually phase out the execution of one or more operations over time. As a result, since specific versions undergo phased out, specific operations (such as generating actions and / or slot values) for these specific versions may be performed locally and specifically. In this way, a computing device that typically operates longer than most other computing devices (e.g., a vehicle computing device) can still receive a certain amount of support from a server device over a longer period of time, although not corresponding to the latest version.

[0006] For example, the spoken utterance may include natural language content that the user typically uses to control another device in their device (such as a smart home device and / or a mobile phone), and the natural language content may specify the requested intent. When the vehicle computing device corresponds to the current version supported by the server device, the vehicle computing device may transmit the spoken utterance to the server device. In response, the vehicle computing device may receive data specifying the following items from the server device in one or more transmissions: text converted from the spoken utterance, one or more intents determined based on the text, one or more slot values ​​of one or more corresponding intents, and / or one or more actions of one or more intents. In some implementations, the slot for an intent and / or an action may refer to a required or optional parameter referenced when performing an operation according to the intent and / or the action. In addition, a slot value may be a value assigned to a specific slot of an intent and / or an action. For example, an intent such as a message intent may include a slot for specifying a recipient of a message, and the slot value may be a name (e.g., John Smith) and / or a phone number (e.g., 555-555-1234).

[0007] As described in detail herein, the specific data provided by the server device when the vehicle computing device corresponds to the current version can depend on the type of spoken utterance, which can be derived from the text of the spoken utterance and / or the intent determined based on the text. In addition, what data (if any data) is provided in response to a specific type of spoken utterance can change dynamically over time because the version of the vehicle computing device is no longer the current version fully supported by the server device. For example, if the vehicle computing device corresponds to a version that is not the latest version, the server device can at least perform speech-to-text processing of the audio data corresponding to the spoken utterance to generate text data anyway. The text data can optionally be provided to the vehicle computing device and used by it. However, the server device may not provide any further data, but instead instruct the vehicle computing device to rely on its own local engine (e.g., local NLU engine, local action engine) to facilitate satisfying the spoken utterance-this approach is contrary to the way of exclusively issuing an error to the user (e.g., "Please update your device"). In this way, although the vehicle computing device may have the ability to locally convert spoken utterances into text data, the vehicle computing device may still leverage the processing power of the server device to obtain better speech-to-text conversion even though the server no longer fully supports the version of the vehicle computing device.

[0008] In some implementations, the server device may transmit the text data to the vehicle computing device along with instructions for the vehicle computing device to no longer transmit similar intent requests to the server device. For example, the vehicle may include a subsystem that can be controlled via an automated assistant for a user operating a second version of the vehicle computing device, but cannot be controlled via an automated assistant for a user operating a first version of the vehicle computing device. The subsystem may be, but is not limited to, a network of fluid sensors connected to the vehicle computing device. The user may request the vehicle computing device to implement a specific intent associated with a network of three fluid sensors by providing a spoken utterance such as "Assistant, send the body shop my fluid sensor readings". The server device may process audio data corresponding to the spoken utterance and determine, based on the audio data and the version information, an action corresponding to an intent that is only supported by a later version (e.g., the second version) of the vehicle computing device (e.g., retrieving data from four fluid sensors). For example, an action supported by a latest version of the vehicle computing device may include an action grammar for sampling output signals from four different fluid sensors. Thus, if a vehicle computing device having only three fluid sensors receives the aforementioned action, the syntax of the action may cause the vehicle computing device to fail or otherwise perform an irrelevant task to facilitate the execution of an incorrect action. In response to the server device receiving the audio data and version information from the vehicle computing device, the server device may generate instructions and / or data for providing to the vehicle computing device so that the vehicle computing device no longer transmits audio data corresponding to similar types of intent requests to the server device. Instead, using these instructions, when the vehicle computing device determines that the user is requesting a similar type of intent request, the vehicle computing device may locally generate action data for the requested intent. This may conserve computing resources and network resources because the server device will no longer process audio data for intent requests corresponding to those actions that certain vehicle computing devices cannot perform due to hardware and / or software version limitations.

[0009] In some implementations, audio data corresponding to spoken utterances from a user may be processed at both the vehicle computing device and the server device. However, when the vehicle is able to determine the intent that the user is requesting, the vehicle computing device may respond to the intent request when the server device has previously requested that the vehicle computing device process such an intent. In addition, the vehicle computing device may operate to facilitate the generation of actions for the intent without further assistance from the server device, despite the presence of a network connection for allowing the vehicle computing device to communicate with the server device. In some implementations, when the vehicle computing device does not receive an instruction to no longer involve the server device in assisting the execution of certain intents, the vehicle computing device may still utilize certain capabilities of the server device, despite the server device not supporting all features of a particular version of the vehicle computing device's hardware and / or software.

[0010] As an example, a user may provide a spoken utterance to promote an intent to use information about a particular subsystem of a vehicle. The spoken utterance may be "Assistant, when do I need to buy new tires?". When the vehicle computing device has a network connection with a server device, audio data corresponding to the spoken utterance may be transmitted to the server device for processing. In response to receiving the audio data, the server device may determine the intent requested by the user and the version associated with the vehicle computing device. Based on the version of the vehicle computing device, the server device may determine that certain requests corresponding to the tire tread sensor and the version of the computing device are no longer supported by the server device, and therefore the server device will not generate intent data in response to determining the intent and version. For example, if the user has the latest version of the computing device, the server device will generate data representing the intent, as well as actions and / or slot values, so that the automated assistant at the vehicle computing device provides an estimate of when the user should change their tires based on the sensor output of the user's tire sensor (e.g., "You should change your tires in about 2 months, or in about 1600 miles").

[0011] However, because the vehicle computing device does not correspond to a fully supported version, the server device may provide natural language content and / or other data based on the server device processing the audio data. For example, the server device may provide the derived request data to the vehicle computing device, and the derived request data may at least characterize the natural language content of the spoken utterance (e.g., "Assistant, when do I need to buy new tires?"), and / or provide a message requesting the user to update their device. In response to receiving the vehicle computing device, the vehicle computing device may provide a message to assist the user (e.g., the message to the user may include: "Please update your vehicle computing device to receive access to all Assistant actions"). In addition, the vehicle computing device may locally generate appropriate actions to perform to facilitate the realization of the requested intent from the user. For example, the locally determined action may correspond to searching the Internet (e.g., "WEB_SEARCH_ACTION()"), while the action to be determined via the latest version of the vehicle computing device may correspond to retrieving and presenting data from at least a subsystem of the vehicle (e.g., "TIRE _ LIFETIME_ESTIMATE_ACTION()").

[0012] As an example, the vehicle computing device may receive the derived request data and, because the vehicle computing device has a network connection, perform an action (e.g., "WEB_SEARCH_ACTION()") to perform an internet search using the phrase "When do I need to buy new tires?" Based on the internet search, the automated assistant may provide natural language output via the interface of the vehicle computing device, rather than providing an error or exclusively indicating that the vehicle computing device cannot perform the requested action. For example, based on the internet search, the automated assistant may provide auditory natural language output such as "Some sources say you should change your tires every 5 years or every 50,000 miles." In this way, devices of interest such as vehicles that a user has used for many years may maintain certain functionality even though certain versions of these devices are no longer partially or completely supported by the manufacturer or other service provider (e.g., an entity that maintains corresponding and supporting server devices).

[0013] The above description is provided as an overview of some implementations of the present disclosure. Further descriptions of these implementations and other implementations are described in more detail below.

[0014] Other implementations may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform a method such as one or more of the methods described above and / or elsewhere herein. Yet other implementations may include a system of one or more computers and / or one or more robots including one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described above and / or elsewhere herein.

[0015] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A , Figure 1B and Figure 1C A diagram showing a server device responsive to various computing devices even as the server device proactively phases out certain versions of the computing devices.

[0017] Figure 2 A system is shown for phasing out support for computing device versions while ensuring useful responsiveness of any computing device versions still in use by users.

[0018] Figure 3 A method is shown for phasing out support for certain computing device versions while ensuring useful responsiveness of any computing device versions still in operation.

[0019] Figure 4 A method is shown for phasing out support for a particular computing device request while ensuring useful responsiveness to any requests that the computing device has previously responded to.

[0020] Figure 5 is a block diagram of an example computer system. DETAILED DESCRIPTION

[0021] Figure 1A , Figure 1B and Figure 1CThe views 100, 130, and 150 show the server device 114 responding to various vehicle computing devices despite the server device 114 actively phasing out support for certain versions of vehicle computing devices. Specifically, Figure 1A The vehicle computing device 106 is shown as being fully supported by a remote computing device 112, such as a server device 114. In other words, the version of one or more software and / or hardware components of the vehicle computing device 106 may correspond to the latest available version. As an example, the vehicle computing device 106 may correspond to a third version, and other previously available versions may include a first version and a second version. The server device 114 may be operable to provide a certain amount of support for each version, even though the third version is the latest version, in order to eliminate wasted transmissions and / or operations at other vehicle computing devices operating according to the first version or the second version.

[0022] like Figure 1A As provided in , a user may provide a spoken utterance 102 such as “Assistant, send my location and arrival time to my destination's phone number”. The spoken utterance may correspond to an action that is exclusively executable at a vehicle computing device operating according to the third version. Therefore, since the vehicle computing device 106 of the vehicle 104 corresponds to the third version, the server device 114 may operate to fully respond to the request from the user. In some implementations, the request from the user may correspond to an intent that is the same for all versions (e.g., the first version, the second version, and the third version), but the specific action for the requested intent may be different depending on the version associated with the vehicle computing device that receives the request. In other words, the identifier for the intent may be common to all versions of the software and / or hardware module. For example, the intent requested via the spoken utterance 102 may be identified as MESSAGE_INTENT(). However, the actions that can be performed according to the intent may be different for each version, and / or the syntax of each action may be different for each version. For example, a MESSAGE_DESSITINATION() action may be available to all versions of the vehicle computing device, however, the specific syntax of the MESSAGE_DESSITINATION() action may be different for the third version, at least relative to the first version. As an example, the action syntax of the third version may reference properties, values, APIs, and / or any other data that is not referenced by the action syntax of the same action of the first version.

[0023] In some implementations, the vehicle computing device 106 may rely on the server device 114 to select a particular action for execution at the vehicle computing device 106. The vehicle computing device 106 may determine locally whether to rely on the server device 114 to select the action, or to determine the action locally. For example, because the vehicle computing device 106 corresponds to the third and latest version, the vehicle computing device 106 may determine that the server device 114 can provide action data and / or slot value data when the user is requesting a particular intent. In response to the user providing a request for a message intent, the vehicle computing device 106 may perform speech-to-text processing on the spoken utterance 102, and / or may provide audio data corresponding to the spoken utterance 102 to the server device 114 for processing. The audio data may be transmitted over a network 108 such as the Internet via a wireless communication protocol such as cellular, 4G, 5G, LTE, Bluetooth, Wi-Fi, one or more remote protocols, and / or any other wired or wireless communication protocol.

[0024] In response to the server device 114 receiving the audio data, the server device 114 and / or another device communicating with the server device 114 may perform speech-to-text processing on the audio data. The text data obtained from the speech-to-text processing may characterize the natural language content of the spoken utterance 102. In some instances, the user may provide a spoken utterance 102 that includes one or more natural language words that specifically generate and / or identify a particular intent. For example, a natural language word (such as send, message, draft, and / or any other natural language word that may be associated with an electronic message) may be used to identify the message intent. The text data may be stored at the server device 114 as assistant data 118 and may be further processed via the auxiliary interaction engine 122 to generate natural language understanding (NLU) data corresponding to the text data. In some implementations, by generating NLU data, the assistant interaction engine 122 may identify the intent requested by the user via the spoken utterance 102.

[0025] In some implementations, when the vehicle computing device 106 receives the spoken utterance from the user, the vehicle computing device 106 may transmit audio data corresponding to the spoken utterance along with the version data to the server device 114. In response to receiving the version data, the version engine 120 of the server device 114 may determine a version corresponding to the vehicle computing device 106. In some implementations, the version may be based on one or more hardware components of the vehicle computing device 106, one or more software modules operating at the vehicle computing device 106, an automated assistant accessible via the vehicle computing device 106, and / or any other characteristics that may be associated with the vehicle computing device 106.

[0026] Based on identifying the requested intent and version corresponding to the vehicle computing device 106, the server device 114 may generate an appropriate action for the vehicle computing device 106 to perform. Alternatively or additionally, the server device 114 may also identify one or more slot values ​​for the intent, which may be used during the generation of the identified action. The server device 114 may use the version data 116 to identify one or more actions that may be performed using the vehicle computing device 106. For example, a third version corresponding to the vehicle computing device 106 may support retrieving application data from different applications to generate slot values ​​for a message. Thus, since the user requests that his location and arrival time be sent to a phone number associated with his destination, the server device may access other application data identifying the current location (e.g., "SLOT_VALUE_X"), the estimated arrival time (e.g., "SLOT_VALUE_Y"), and the phone number associated with the user's destination (e.g., "SLOT_VALUE_Z") with the user's permission. The action corresponding to this type of message may be identified as, for example, a multi-source message action (e.g., "ACTION_N"). The particular action may be selected by the server device 114 to be performed at the vehicle computing device 106 at least when the vehicle computing device 106 corresponds to the third version.

[0027] Thus, in response to the vehicle computing device 106 receiving the spoken utterance 102, and the vehicle computing device 106 providing the audio data and the version data to the server device 114, the server device 114 may provide the intent data 124 to the vehicle computing device 106. The intent data 124 may identify the intent (e.g., "INTENT()"), the action selected by the server (e.g., "ACTION_N"), and any slot values ​​identified by the server device 114 based on the utterance 102 and / or any other suitable data source. However, when the vehicle computing device does not correspond to the third version, the server device 114 and / or the vehicle computing device may process similar spoken utterances differently but in a manner that ensures a response to the spoken utterance by an automated assistant or another application.

[0028] In various implementations, the vehicle computing device 106 is integrated with the vehicle and directly drives the vehicle's speaker(s) that are also integrated with the vehicle. The vehicle computing device 106 may be original equipment to the vehicle, or may be an aftermarket installed accessory. The vehicle computing device 106 is integrated because it directly drives the vehicle's speaker(s) and / or cannot be removed from the vehicle without the use of specialized tools and / or requiring a significant amount of time and / or expertise. For example, the vehicle computing device 102 may be connected to the vehicle's controller area network (CAN) bus and / or may be powered via a vehicle-specific connector (e.g., not a 12V vehicle receptacle and not an easily accessible auxiliary standard plug).

[0029] Figure 1B View 130 shows a user providing a spoken utterance 132 to a vehicle computing device 140 corresponding to a second version. The vehicle computing device 140, and / or any other vehicle computing device discussed herein, may include one or more memory devices and one or more processors. In response to receiving the spoken utterance 132, the vehicle computing device 140 may communicate with the remote computing device 112 via a network 126 such as the Internet. The spoken utterance 132 may be similar to Figure 1A 102 , at least to illustrate the differences and how server device 114 may handle different versions of similar spoken utterances.

[0030] For example, because the vehicle computing device 140 corresponds to the second version and the third version is the latest version, the server device 114 may be at least relative to Figure 1A 132 and the second version of the vehicle computing device 140. The vehicle computing device 140 may transmit audio data representing the spoken utterance 132 and version data representing the vehicle computing device 140 as corresponding to the second version. In some implementations, the vehicle computing device 140 may locally determine the extent to which the server device 114 may assist the vehicle computing device 140 in responding to the specific intent request contained by the spoken utterance 132. Additionally or alternatively, based at least on the version data 116 accessible to the server device 114, the server device 114 may determine the extent to which the server device 114 may assist the vehicle computing device in responding to the specific intent request. The vehicle computing device 140 may provide the audio data and the version data to the server device 114 in response to determining that the spoken utterance 132 contains the intent request corresponding to the MESSAGE_INTENT and that the second version of the vehicle computing device 140 is at least partially supported with respect to the MESSAGE_INTENT.

[0031] As an example, the server device 114 may receive the audio data and the version data, and the version engine 120 may determine the extent to which the server device 114 can support the vehicle computing device 140 having the second version. The version engine 120 may compare the version data from the vehicle computing device 140 with the version data 116 to generate data characterizing the extent to which the server device 114 can support the vehicle computing device 140. Alternatively or additionally, the server device 114 may access text data characterizing the natural language content of the spoken utterance 132 to generate natural language understanding data of the spoken utterance 132. The auxiliary interaction engine 122 may use the text data to generate NLU data that characterizes the MESSAGE_INTENT and, optionally, one or more slot values ​​for implementing the intent request. In some implementations, the intent and / or one or more slot values ​​may be generated based on the assistant data 118 accessible to the server device 114.

[0032] Based on the server device 114 determining that the vehicle computing device 140 corresponds to the second version, the server device 114 may transmit data 128 to the vehicle computing device, and the data 128 may identify the intent (e.g., INTENT()) and speech-to-text data (e.g., STT_DATA (“Assistant, send…”), which may characterize the natural language content of the spoken utterance 132. If any local processing of the spoken utterance 132 remains in progress, by providing the speech-to-text data, computing resources may be preserved at the vehicle computing device 140 despite the vehicle computing device 140 corresponding to the second version instead of the third version. Furthermore, by allowing the server device 114 to bypass transmitting action data corresponding to the third version, network bandwidth may be preserved because less data is transmitted via the network 126. Furthermore, if the vehicle computing device 140 has already received the action data corresponding to the third version and attempted to process it, such processing would be immaterial, at least with respect to further fulfillment of the message intent.

[0033] When the vehicle computing device 140 receives the data 128, the vehicle computing device 140 may locally select an appropriate action for the intent. For example, in addition to identifying and / or generating a multi-source action intent, the vehicle computing device 140 may identify an action supported by one or more applications accessible via the vehicle computing device 140. For example, the vehicle computing device 140 may generate a LOCATION_MESSAGE action that can be executed by a map application with the help of an automated assistant. For example, the slot value of the LOCATION_MESSAGE action may include a current location and an estimated arrival time, which may be generated by a map application. However, because the vehicle computing device 140 corresponds to the second version, the vehicle computing device 140 may not be able to support an application that can provide a phone number for the mapped destination. Therefore, in response to the spoken utterance 132 and based on the data 128, the automated assistant may provide a response 142 such as "Ok, I'm sending a message with your location and estimated arrival time. Who would you like to send this message to?" In response, and to fulfill the remaining slot values, the user can provide another spoken utterance 132, such as "Albert Smith." When all slot values ​​for the LOCATION_MESSAGE action have been determined, the action can be executed at vehicle computing device 140. In this way, the automated assistant and / or vehicle computing device 140 can maintain a response to the requested intent without wasting computing resources despite the vehicle computing device 140 corresponding to the second version that is outdated relative to the third version.

[0034] Figure 1C A view 150 is shown of a user providing a spoken utterance 152 to a vehicle computing device 156 of a vehicle 154 corresponding to a first version supported by the server device 114. Specifically, Figure 1C It is shown that although the requested intent is associated with the first and latest version of the action, the server device 114 still causes the vehicle computing device 156 to respond to the user's context. Figure 1C It shows how the server device 114 may provide instruction data that may cause the vehicle computing device 156 to subsequently bypass transmitting certain data corresponding to certain intended requests.

[0035] according to Figure 1C, the user can provide a spoken utterance 152 such as "Assistant, send my location and estimated arrival time to my destinations phone number". Figure 1A and Figure 1B As discussed, the above-mentioned spoken utterances may be fully supported by the server device 114 or partially supported by the server device 114. However, in some cases, the server device 114 may use the version engine 120 and the version data 116 to determine that the server device 114 no longer supports one or more requested intents. As an example, in response to receiving audio data representing the spoken utterance 152, the server device 114 may determine that the vehicle computing device 156 corresponds to the first version and identify one or more intents requested by the user via the spoken utterance 152. If the identified intent corresponds to an intent that is no longer supported by the latest version (e.g., the third version), the server device 114 may further determine whether the vehicle computing device 156 has received guidance data regarding limiting the ability to submit such a request to the server device 114. Alternatively or additionally, the vehicle computing device 156 may transmit the audio data, the version data, and / or data representing the limitations of the vehicle computing device 156 with respect to a particular intent via the network 158. In response to receiving the restriction data, the server device 114 may determine the extent to which the server device 114 should and / or will respond to the data from the vehicle computing device 156.

[0036] For example, in response to receiving the audio data, data, and data characterizing the restriction, the server device 114 may determine that the vehicle computing device 156 is operating according to the first version, and the vehicle computing device 156 has been instructed to no longer request intent data, action data, and / or slot data for one or more intents and / or one or more actions. The server device 114 may process the audio data to generate natural language understanding data, which may characterize the one or more intents requested by the user. If one or more characterized intents correspond to one or more intents identified by the restriction data, the server device 114 may bypass generating intent data, action data, and / or slot data for the vehicle computing device 156 at least with respect to the one or more characterized intents. In addition, in addition to the server device 114 using resources to generate intent data, action data, and / or slot data, the server device 114 may generate speech-to-text data. Depending on the processing power of the local vehicle computing device 156 and / or how many applications are running at the vehicle computing device 156 at the same time, providing speech-to-text data from the server device 114 may reserve processing bandwidth at the vehicle computing device 156.

[0037] In some implementations, if the server device 114 determines that a version corresponding to the vehicle computing device 156 has undergone a phase-out, but has not yet been provided with instruction data, the server device 114 may transmit data 160 to the vehicle computing device 156, and the data 160 may include speech-to-text data and / or instruction data characterizing limitations of the particular version associated with the vehicle computing device 156. In response to receiving the instruction data, the vehicle computing device 156 may update its configuration so that it subsequently bypasses reliance on certain intent data, action data, and / or slot data, and / or bypasses querying the server device 114 for certain intent data, action data, and / or slot data.

[0038] In the case where the server device 114 confirms the limitations of the version of the vehicle computing device 156 and the server device 114 provides speech-to-text data, the vehicle computing device 156 can use the speech-to-text data or the locally generated speech-to-text data to facilitate the response to the user and / or otherwise implement the requested intent. For example, the vehicle computing device 156 can determine that the user has requested a specific intent based on the speech-to-text data. In addition, because the server device 114 has not selected a specific action for the intent, the vehicle computing device 156 can locally generate an action, such as a NEW_MESSAGE action. In addition, the vehicle computing device 156 can determine whether a slot value has been assigned to one or more slots required for the NEW_MESSAGE action. For example, if the "MESSAGE_BODY()" slot value is missing or otherwise unidentifiable by the vehicle computing device 156 using the speech-to-text data, the vehicle computing device 156 can provide a query 162 to the user. In response, the vehicle computing device 156 can provide a natural language output, such as "OK, what would you like the message to say?". Subsequently, the user may provide another spoken utterance 152, such as "I'm on my way". This interaction between the user and the vehicle computing device 156 may continue until all required slots have been assigned slot values, and / or until an action has been performed via the vehicle device 156. In this way, the vehicle computing device 156 may be responsive to the user and utilize at least some functionality of the server device 114 despite the vehicle computing device 156 corresponding to the first version, or a version that is not the latest version. This allows the vehicle computing device 156 to be responsive to the user for longer periods of use, while also eliminating wasted network communications and / or inefficient use of computing resources (e.g., memory and / or processing bandwidth).

[0039] Figure 2A system 200 is shown for ensuring useful responsiveness of any computing device version that is still in operation while phasing out versions of computing devices (e.g., vehicle computing devices and / or any other client devices). Automated assistant 204 can operate as part of an automated assistant application provided at one or more computing devices such as client device 218 and / or server device 202. A user can interact with automated assistant 204 via one or more assistant interfaces 220, which can include one or more of the following: a microphone, a camera, a touch screen display, a user interface, and / or any other device capable of providing an interface between a user and an application. For example, a user can initialize automated assistant 204 by providing verbal, textual, or graphical input to the assistant interface to cause automated assistant 204 to perform a function (e.g., provide data, control peripherals, access an agent, etc.). Client device 218 can include a display device, which can be a display panel including a touch interface for receiving touch input and / or gestures to allow a user to control the application of client device 218 and / or server device 202 via the touch interface.

[0040] In some implementations, the client device 218 may not have a display device but include an audio interface (e.g., a speaker and / or a microphone) to provide audible user interface output without providing graphical user interface output, and provide a user interface for receiving oral natural language input from a user, such as a microphone. For example, in some implementations, the client device 218 may include one or more tactile input interfaces, such as one or more buttons, and omit a display panel that provides graphics data from a graphics processing unit (GPU). In this way, a significant amount of energy and processing resources may be saved compared to a computing device that includes a display panel and a GPU.

[0041] The client device 218 can communicate with the server device 202 via a network 240, such as the Internet. The client device 218 can offload computing tasks to the server device 202 in order to preserve computing resources at the client device 218. For example, the server device 202 can host the automated assistant 204, and the client device 218 can transmit input received at one or more assistant interfaces 220 to the server device 202. However, in some implementations, the automated assistant 204 can be hosted at the client device 218. In various implementations, all or less than all aspects of the automated assistant 204 can be implemented on the remote computing device 242 and / or the client device 218. In some of those implementations, various aspects of the automated assistant 204 are implemented via the local automated assistant 222 of the client device 218 and interfaced with the server device 202, which can implement other aspects of the automated assistant 204. The server device 202 can optionally serve multiple users and their associated assistant applications via multiple threads. In implementations in which all or less than all aspects of automated assistant 204 are implemented via a local automated assistant 222 of client device 218, local automated assistant 222 may be an application separate from the operating system of client device 218 (e.g., installed “on top of” the operating system)—or may alternatively be implemented directly by the operating system of client device 218 (e.g., considered an application of the operating system, but integrated with the operating system).

[0042] In some implementations, automated assistant 204 and / or automated assistant 222 can include an input processing engine 206 that can employ a number of different modules to process input and / or output for client device 218. For example, input processing engine 206 can include speech processing engine 208 that can process audio data received at assistant interface 220 to identify text contained in the audio data. The audio data can be transmitted from, for example, client device 218 to server device 202 in order to preserve computing resources at client device 218.

[0043] The process for converting audio data into text may include a speech recognition algorithm that may employ a neural network and / or a statistical model to identify a group of audio data corresponding to a word or phrase. The text converted from the audio data may be parsed by a natural language understanding (NLU) / intent engine 210 and available to the automated assistant 204 as data that can be used to identify one or more intents requested by the user. In some implementations, the output data provided by the NLU / intent engine 210 may be provided to the action engine 214 to determine whether the user has provided input that corresponds to a specific action and / or routine that can be performed by the automated assistant 204 and / or an application or agent that can be accessed by the automated assistant 204. For example, the assistant data 216 may be stored as client data 230 at the server device 202 and / or the client device 218, and may include data that defines one or more actions that can be performed by the automated assistant 204 and / or the automated assistant 222, as well as any slot values ​​and / or other parameters involved in performing those actions.

[0044] When input processing engine 206 has determined that the user has requested a particular intent, routine, and / or action to be implemented and / or otherwise performed, slot engine 212 can determine one or more slot values ​​for a slot for the particular intent and / or action, and action engine 214 can then provide output to the user based on the particular intent, action, routine, and / or one or more slot values. For example, in some implementations, in response to user input, such as a gesture of assistant interface 220 of client device 218 directed toward vehicle 238, automated assistant 222 can cause data representing the gesture to be transmitted to server device 202, thereby allowing the server device to determine the intent and / or action that the user is intending for automated assistant 204 and / or automated assistant 222 to perform.

[0045] In some implementations, the client device 218 may be, but is not limited to, a desktop computer, a laptop computer, a portable computing device such as a cellular phone, a tablet computing device, a smart home device, and / or any other device that can communicate directly or indirectly with a server device. The client device 218 may correspond to a specific version that may change over time. For example, the client device 218 may correspond to a latest version for receiving support from the server device 202, or the client device 218 may correspond to a version that is not the latest version for receiving support from the server device 202. The version may correspond to one or more physical hardware components of the client device 218, an operating system of the client device 218, one or more applications available at the client device 218, an automated assistant 222, and / or any other device or module that may be associated with the client device 218.

[0046] In order for the server device 202 to phase out support for a particular version, the server device 202 may include a version engine 234 from which the server device 202 may determine restrictions on supporting a particular version. For example, using the version engine 234 and / or the server data 236, the server device 202 may determine that the client device 218 corresponds to a version that is fully supported by the server device 202. In other words, a version may be fully supported when there are no restrictions on responding to certain intent requests from the client device 218, at least relative to other versions.

[0047] As an example, when the client device 218 corresponds to a fully supported version and a user provides a spoken utterance to the assistant interface 220 of the client device 218 for a particular intent, the server device 202 can be responsible for characterizing the intent, generating appropriate actions, and optionally generating a slot value that can be used by the client device 218 to implement the particular intent. Alternatively, when the client device 218 corresponds to a partially supported version and a user provides a spoken event to the assistant interface 220 for the particular intent, the server device 202 can be responsible for characterizing the intent and optionally generating a slot value. The resulting data characterizing the intent and / or the slot value can then be provided back to the client device 218, which can employ the action engine 228 to generate an action that corresponds to the particular intent and is also supported by the version of the client device 218.

[0048] In some implementations, version engine 234 may determine that client device 218 corresponds to a version that is no longer supported by server device 202. For example, in response to receiving a spoken utterance from a user at assistant interface 220, client device 218 may transmit audio data corresponding to the spoken utterance and version data 232 to server device 202. Version data 232 stored at client device 218 may indicate one or more versions corresponding to client device 218, and / or one or more limitations that have been elaborated by one or more server devices. Limitations may include, but are not limited to, instructions that cause client device 218 not to rely on server device 202 to provide intent data, action data, slot data, and / or any other data corresponding to certain requests from the user. Using the version data, server device 202 may determine whether client device 218 corresponds to a version that server device 202 fully supports, partially supports, or no longer supports.

[0049] When the client device 218 corresponds to a version that is no longer supported by the server device 202, the server device 202 can process the audio data received from the client device 218 and provide text data representing the natural language content of the spoken utterance back to the client device 218. Thus, the client device 219 will not be provided with intent, action, and / or slot value support, but the speech-to-text service may still be available to the client device 218 when a network connection between the client device 218 and the remote computing device 242 is available. This can eliminate wasted processing at the client device 218, which may have less memory than the server device 202. For example, when the client device 218 corresponds to a significantly older version of hardware and therefore includes only 200 MB of RAM, but the server device 202 includes 8 GB of RAM, the server device 202 will be able to complete speech-to-text processing of the audio data in less time than the client device 218 can perform speech-to-text processing. As a result, while the client device 218 is employing the speech-to-text engine 224 to convert the audio data into text data, the client device 218 may receive the resulting text data from the server device 202. The client device 218 may then choose to terminate the ongoing local speech-to-text processing, at least because the text data generated by the server device 202 eliminates the need for the client device 218 to generate such data.

[0050] In some implementations, a client device 218 corresponding to a version partially supported by the server device 202 may locally process audio data for the spoken utterance and also transmit the audio data to the server device 202 for processing. The local processing of the audio data may be performed by an NLU / intent engine 226, which is capable of generating natural language understanding (NLU) data from text data representing the natural language content of the spoken utterance. The natural language understanding data may represent one or more requested intents. If the server device 202 generates the natural language understanding data in a shorter period of time than the client device 218, the client device 218 may rely on the server device 202 to obtain the natural language understanding data. However, thereafter, the client device 218 may cause the action engine 228 to process the natural language understanding data from the server device 202 to at least generate an appropriate action that can be performed by the version of the client device 218. The version data may represent one or more characteristics of one or more hardware components, one or more software components, and / or any other feature of the client device. For example, in some implementations, the version data may represent one or more operating specifications of the client device. Alternatively or additionally, the version data may characterize one or more operating specifications of an application and / or operating system of the client device 218. Alternatively or additionally, the version data may be specified by a manufacturer of the client device 218, a component of the client device, and / or a software manufacturer of an application or operating system of the client device 218. In some implementations, the version data may characterize accounts and / or subscription tiers corresponding to one or more users. For example, the version data may characterize one or more devices as corresponding to one or more account tiers (e.g., full membership and / or full service tier), and other version data may characterize one or more other devices as corresponding to one or more other account tiers (e.g., limited membership and / or limited service tier).

[0051] Figure 3 A method 300 is shown for ensuring useful responsiveness of any computing device versions still in operation while phasing out support for certain computing device versions. The method 300 may be performed by one or more computing devices, applications, and / or any other device or module that may be associated with an automated assistant. The method 300 may include an operation 302 of determining whether a spoken utterance is detected at a computing device. The computing device may be, but is not limited to, a vehicle computing device, a mobile computing device, a desktop computing device, a cellular device, a server device, and / or any other apparatus capable of operating as a computing device. The spoken utterance may be a spoken natural language input from a user of the computing device. The spoken utterance may be, for example, "Assistant, stream music."

[0052] From operation 302, method 300 may proceed to operation 304, which may include determining a version corresponding to a computing device. Alternatively, if no spoken utterance is detected, operation 302 may be repeated until a spoken utterance is detected at the computing device. However, with respect to operation 304, the version corresponding to the computing device may be a version for the entire computing device, one or more hardware components of the computing device, one or more software components of the computing device, and / or any other feature of the computing device that may be characterized by a version. For example, a computing device may operate a local automated assistant, which may correspond to a particular version. Thus, the local automated assistant may respond differently depending on the version of the local automated assistant. Similarly, the entire computing device may respond differently depending on a particular version corresponding to the entire computing device, which may optionally be integrated into a vehicle.

[0053] From operation 304, method 300 may proceed to operation 306, which may include determining whether the version of the computing device is fully supported by the server device. Operation 306 may be performed by a server device, which may receive version data from the computing device or otherwise access version data in response to determining that a spoken utterance is detected at the computing device. When it is determined that the version corresponding to the computing device is fully supported by the server device, method 300 may proceed to operation 308. Alternatively, when it is determined that the version corresponding to the computing device is not fully supported by the server device, method 300 may proceed to operation 310.

[0054] Operation 308 may include generating action data based on the natural language content of the spoken utterance. The action data may characterize one or more actions that can be performed by a particular version of the computing device and that are currently supported by the server device. For example, corresponding at least to the aforementioned example (e.g., "Assistant, stream music"), the action data may generate a STREAM_MUSIC action, which may correspond to a PLAY_MUSIC intent.

[0055] Operation 310 may include determining whether the version corresponding to the computing device is at least partially supported by the server device. When it is determined that the version corresponding to the computing device is at least partially supported by the server device, method 300 may proceed to operation 312. Operation 312 may include generating intent / NLU data and / or slot data based on natural language content. The intent / NLU data may characterize the intent requested by the user via the spoken utterance, and the slot data may characterize one or more slot values ​​that may be used when the computing device is performing one or more actions. For example, if the user has specified an artist to perform in their spoken utterance, the slot data may characterize the artist name. In this way, although the server device does not fully support the version corresponding to the computing device, the server device may still provide useful information to the computing device to eliminate the waste of computing resources at the computing device.

[0056] When the version corresponding to the computing device is not supported at least in part by the server device, the method 300 can proceed to operation 314. Operation 314 can include generating text data based on the natural language content of the spoken utterance. Because the computing device may have limited processing bandwidth and / or memory at least relative to the server device, the computing resources of the computing device can still be retained by relying on the server device to at least perform speech-to-text processing. It should be noted that when the version corresponding to the computing device is fully supported by the server device, operation 308, operation 312 and / or operation 314 can be performed by the server device for this particular version. In addition, when determining that the version corresponding to the computing device is supported at least in part by the server device, operation 312 and / or operation 314 can be performed by the server device for this particular version.

[0057] Method 300 can proceed from operation 314 to operation 316 for providing the generated data to a computing device. The generated data may correspond to action data, intent / NLU data, slot data, and / or text data, depending on the extent to which the server device supports the version corresponding to the computing device. Therefore, although the server device may not fully or partially support a particular version, the server device can still provide a certain amount of data that will help the computing device process the spoken utterance from the user. When the server device no longer supports the version corresponding to the computing device, the computing device can use the speech-to-text data provided from the server device to locally select one of our actions for implementing the request from the user. For example, with respect to the aforementioned example, when the server device does not fully or partially support the version corresponding to the computing device, the server device can provide text data (e.g., "STT_DATA('Assistant, stream music')") characterizing the natural language content of the spoken utterance. In response, the computing device can determine the intent requested by the user (e.g., "PLAY_MUSIC") and select an action (e.g., "SHUFFLE_MUSIC()") available via the computing device. In this way, wasteful communications between the server device and the computing device can be eliminated while also providing responsiveness to the user. Such benefits can be achieved despite the fact that the user does not employ a computing device corresponding to a version of the latest version, or a computing device corresponding to a version that is not even at least partially supported by the server device. It should be noted that support by the server device may refer to intent-specific support, but does not include intent-agnostic support, such as speech-to-text processing, which may be performed by the server device regardless of the natural language content of the spoken utterance from the user.

[0058] Figure 4 A method 400 is shown for ensuring useful responsiveness to any request that a computing device has previously responded to while phasing out support for a particular request. The method 400 may be performed by one or more computing devices, applications, and / or any other device or module that may be associated with an automated assistant. The method 400 may include an operation 402 of determining whether a spoken utterance is detected at a computing device. The computing device may be, but is not limited to, a vehicle computing device, a mobile computing device, a desktop computing device, a cellular device, a server device, and / or any other device capable of operating as a computing device. The spoken utterance may be a spoken natural language input from a user of the computing device. As an example, the spoken utterance may include a request for a computing device to control a peripheral device such as an IoT device (e.g., "Assistant, turn down the backlight of my smart TV").

[0059] From operation 402, method 400 may proceed to operation 404, which may include determining an intent requested via a spoken utterance. An intent may characterize a request from a user and / or may be associated with one or more different actions that can be performed by different corresponding computing devices. For example, an intent corresponding to the spoken utterance "Assistant, turn down the backlight of my smart TV" may correspond to an intent such as an ADJUST_TV_SETTING intent. At least some versions corresponding to the computing device may support actions for the intent, such as a BACKLIGHT_ADJUST action, however, other versions corresponding to the computing device will not support the BACKLIGHT_ADJUST action, but may perform a SETTING_ADJUST action.

[0060] Method 400 can proceed to operation 406 from operation 404, which can include generating and / or receiving text or data based on natural language content. In some implementations, both computing devices and server devices can perform speech-to-text processing in response to the computing device detecting spoken utterances. Specifically, the computing device can generate audio data in response to receiving spoken utterances, and provide the audio data to the server device with permission from the user, and also process the audio data locally. If the server device is unresponsive to the provision of audio data, the computing device can process the audio data locally to generate text data representing the natural language content of the spoken utterances. The context data received from the server device and / or from the computing device can be further processed to identify one or more intentions requested to be implemented.

[0061] Method 400 may proceed from operation 408 to operation 410, which may include determining whether the server device fully supports one or more identified intents. In some implementations, operation 408 may include determining whether the computing device has been instructed to no longer request support for one or more intents that have been identified from the natural language content of the spoken utterance. When the computing device determines that one or more intents are fully supported by the server device or the computing device has not been completely blocked from requesting support for one or more intents, method 400 may proceed from operation 408 to operation 410. Alternatively, when the computing device determines that one or more intents are not at least partially supported by the server device, or the computing device has been completely blocked from requesting support for one or more intents, method 400 may proceed from operation 408 to operation 412.

[0062] Operation 410 may include requesting action data based on the natural language content and / or one or more intents identified from the natural language content. For example, the computing device may identify the ADJUST_TV_SETTING intent, and in response to determining that the intent is fully supported by the server device given a version corresponding to the computing device, the computing device may request action data from the server device. For example, the action data may characterize a BACKLIGHT_ADJUST action that may be executable by the latest version relative to other versions operated via other computing devices.

[0063] From operation 408, method 400 may proceed to operation 412, which may include determining whether one or more identified intents are partially supported by the server device. In some implementations, operation 412 may include determining whether the computing device has been instructed to request only specific support for one or more intents that have been identified from the natural language content of the spoken utterance. When the computing device determines that one or more intents are at least partially supported by the server device, or the computing device is not limited in the type of support for one or more intents, method 400 may proceed from operation 412 to operation 414. Alternatively, when the computing device determines that one or more intents are not at least partially supported by the server device, or the computing device has been limited in requesting support for one or more intents, method 400 may proceed from operation 412 to operation 416.

[0064] Operation 414 may include requesting intent / NLU data and / or slot data from the server device. The intent / NLU data and / or slot data may be used by the computing device to identify a particular functionality that remains executable via a version corresponding to the computing device. For example, the server device and / or the computing device may identify the intent ADJUST_TV_SETTING, and the server device may identify slot data for an action to be selected by the computing device. For example, the slot data may include a slot value based at least on the natural language content of the spoken utterance, such as "backlight".

[0065] When it is determined that the intent is no longer supported by the server device, at least for the version corresponding to the computing device, method 400 may proceed from operation 412 to operation 416. Operation 416 includes locally selecting an action for implementing the requested intent. For example, the computing device may determine that the server device previously provided instructions for completely limiting the computing device's ability to request support for the ADJUST_TV_SETTING intent. Based on this determination, the computing device may locally select an action, SETTING_ADJUST(), which, when executed, may cause a menu to appear at the TV to allow the user to adjust the TV's settings.

[0066] In some implementations, method 400 may proceed from operation 410 to operation 414 so that the computing device utilizes support from the server device. Method 400 may proceed from operation 414 and / or operation 416 to operation 418. Operation 418 may include performing an action to facilitate the requested intention of the sensation. When the action is identified locally at the computing device, the execution may include performing the action selected locally. When the action is identified by the server device, the execution may include performing the action selected remotely. In this way, regardless of whether the version corresponding to the computing device is the latest version or not, the computing device will still respond to the spoken words from the user in a manner useful to the user, while also eliminating the waste of computing resources. For example, the action selected remotely may cause the BACKLIGHT_ADJUST action to be executed. Alternatively, the action computer selected locally may cause the SETTING_ACTION_ADJUST action to be executed, so that the menu appears at the TV or another device. When the version is at least partially supported by the server device for a specific intention and the server device provides slot data, the action selected locally may cause the menu to appear, and cause the submenu of the backlight adjustment part of the identification menu to appear. In this manner, partial support still allows the computing device to utilize the computing processes of the server device even though the computing device is not operating according to the latest version.

[0067] Figure 5 5 is a block diagram of an example computer system 510. Computer system 510 typically includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524, including, for example, a memory 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with computer system 510. Network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computer systems.

[0068] The user interface input devices 522 may include a keyboard, a pointing device such as a mouse, a trackball, a touch pad or a graphics tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a speech recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computer system 510 or over a communication network.

[0069] The user interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRD), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways to output information from the computer system 510 to a user or another machine or computer system.

[0070] The storage subsystem 524 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include logic for executing selected aspects of the method 300, the method 400, and / or the server device 114, the vehicle computing device 106, the vehicle 104, the vehicle computing device 140, the vehicle 134, the remote computing device 112, the vehicle computing device 156, the vehicle 154, the server device 202, the client device 218, and / or any other apparatus, module, and / or engine discussed herein.

[0071] These software modules are typically executed by the processor 514 alone or in conjunction with other processors. The memory 525 used in the storage subsystem 524 may include multiple memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 may provide permanent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement certain implemented functions may be stored in the storage subsystem 524 by the file storage subsystem 526, or in other machines accessible by the processor(s) 514.

[0072] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computer system 510 to communicate with each other as desired. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0073] Computer system 510 can be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The description of the computer system 510 depicted in FIG. 5 is intended only as a specific example for the purpose of illustrating some implementations. Many other configurations of the computer system 510 may have more Figure 5The computer system depicted in FIG.

[0074] In cases where the systems described herein collect or make available personal information about users (or, as they are often referred to herein, "participants"), users may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, the user's preferences, or the user's current geographic location) or to control whether and / or how content that may be more relevant to the user is received from a content server. Moreover, certain data may be processed in one or more ways before being stored or used to remove personally identifiable information. For example, the user's identity may be processed so that personally identifiable information cannot be determined for the user, or the user's geographic location may be summarized (such as to a city, zip code, or state level) in the case where geographic location information is obtained so that the user's specific geographic location cannot be determined. Thus, users may control how information about the user is collected and / or used.

[0075] In some implementations, a method implemented by one or more processors is described as including operations such as determining, based on audio data processed at a server device, that a user has provided a spoken utterance to an automated assistant interface of a vehicle computing device connected to a vehicle. The method may also include, in response to determining that the user has provided the spoken utterance, accessing version information associated with the vehicle computing device, wherein the version information indicates that the vehicle computing device corresponds to a specific version. The method may also include characterizing natural language content of the spoken utterance provided by the user based on processing the audio data. The method may also include determining, based on the natural language content characterizing the spoken utterance, whether at least a portion of the natural language content is fully supported by the server device for the specific version. The method may also include, in response to determining that at least a portion of the natural language content is not fully supported by the server device for the specific version, based at least on the version information: providing text data characterizing the natural language content of the spoken utterance provided by the user to the vehicle computing device, and causing the vehicle computing device to locally perform an action generated locally by the vehicle computing device based on the natural language content.

[0076] In some implementations, determining whether at least the portion of the natural language content is fully supported by the server device for the specific version includes: determining whether at least the portion of the natural language content includes one or more natural language terms corresponding to one or more intents indicated at the server device as not fully supported by the server device. In some implementations, the one or more intents are indicated at the server device as being fully supported for other vehicle computing device versions that do not include the specific version of the vehicle computing device. In some implementations, the text data represents the portion of the natural language content that includes the one or more natural language terms. In some implementations, the method may also include, in response to determining that at least the portion of the natural language content is fully supported by the server device for the specific version: generating action data at the server device, the action data identifying the intent requested by the user and another action supported by the specific version of the vehicle computing device, and providing the action data to the vehicle computing device. In some implementations, providing the action data to the vehicle computing device causes the vehicle computing device to perform another action using the provided action data.

[0077] In some implementations, the method may further include, in response to determining that at least the portion of the natural language content for the particular version is not fully supported by the server device, at least with respect to the version information: generating instruction data characterizing a limitation of the vehicle computing device with respect to the requested intent, and providing the instruction data to the vehicle computing device, wherein the instruction data causes the vehicle computing device to bypass requesting the server device to generate a specific action corresponding to another instance of the requested intent in response to subsequent user input. In some implementations, the method may further include, in response to determining that at least the portion of the natural language content for the particular version is not fully supported by the server device, and after providing the instruction data to the vehicle computing device: determining that another spoken utterance associated with the requested intent is received at the automated assistant interface of the vehicle computing device, and providing other text data to the vehicle computing device, wherein the other text data characterizes other natural language content of the other spoken utterance and ignores data characterizing a specific action for the requested intent.

[0078] In some implementations, the method may also include, in response to determining that at least the portion of the natural language content for the specific version is partially supported by the server device, at least with respect to the version information: providing natural language understanding (NLU) data to the vehicle computing device, the NLU data representing a specific intent requested by the user via the spoken utterance. In some implementations, the method may also include, in response to determining that at least the portion of the natural language content for the specific version is partially supported by the server device, at least with respect to the version information: providing slot data representing one or more slot values ​​for the specific intent to the vehicle computing device, and causing the vehicle computing device to perform the action using the one or more slot values, wherein the action is locally identified by the vehicle computing device based on the specific intent and the slot value. In some implementations, the spoken utterance is associated with a hardware subsystem of the vehicle, and the spoken utterance is received at the automated assistant interface while the user is riding in and / or driving the vehicle. In some implementations, the method may further include, after determining that the user provided the spoken utterance, determining that another spoken utterance is received at a separate vehicle computing device operating according to a currently supported version; and providing NLU data representing the requested intent, slot data representing one or more slot values ​​of the requested intent, and action data representing a separate action to be performed by the separate vehicle computing device based on another natural language content of the other spoken utterance.

[0079] In some implementations, the method may further include, before determining that the user provided the spoken utterance, determining that the specific version previously fully supported by the server device in communication with the vehicle computing device is fully supported by the server device. In some implementations, determining whether at least the portion of the natural language content is fully supported by the specific version includes determining the extent to which the specific version is supported by the server device.

[0080] In yet other implementations, a method implemented by one or more processors is described as including operations such as determining that natural language content contained in a spoken utterance received at a first vehicle computing device corresponds to a first intent request. The method may also include determining, based on determining that the natural language content corresponds to the first intent request, to what extent the first intent request is supported by a server device for a version corresponding to the first vehicle computing device. The method may also include generating first data representing the intent requested by the user based on the extent to which the first intent request is supported by the server device. The method may also include determining that another natural language content contained in another spoken utterance received at a second vehicle computing device corresponds to a second intent request. The method may also include determining, based on determining that the another natural language content includes the second intent request, to another extent to which the second intent request is supported by the server device for another version corresponding to the second vehicle computing device, wherein the version is different from the another version. The method may also include generating second data representing the other natural language content of another spoken utterance based on another extent to which the second intent request is supported by the server device. The method may also include providing the first data to the first vehicle computing device to facilitate causing the first vehicle computing device to implement the first intent request; and providing the second data to the second vehicle computing device to facilitate causing the second vehicle computing device to implement the second intent request. In some implementations, the another version corresponding to the second vehicle computing device is initially supported by the server device after the time when the version corresponding to the first vehicle computing device is initially supported by the server device.

[0081] In some implementations, the first intent request and the second intent request correspond to a type of vehicle hardware device, and wherein the first data further characterizes an action corresponding to the intent and an operation that can be performed by the vehicle hardware intent of the type. In some implementations, in response to the second vehicle computing device receiving the second data, the second data causes the vehicle hardware device of the type to perform the operation and / or a different operation. In some implementations, the vehicle hardware device of the type includes one or more sensors, one or more other computing devices, and / or one or more electromechanical devices.

[0082] In other implementations, a method implemented by one or more processors is described, including operations such as receiving data representing a spoken utterance provided by a user to a vehicle computing device of a vehicle via an interface in communication with the interface, wherein the spoken utterance corresponds to a request for the vehicle computing device to implement a specific intent. The method may also include determining a natural language content of the spoken utterance based at least on the data representing the spoken utterance provided by the user, wherein the natural language content includes one or more words corresponding to the specific intent. The method may also include determining the extent to which the one or more words are supported by a server device for the version based at least on the one or more words of the natural language content and a version corresponding to the vehicle computing device. The method may also include, in response to determining that the one or more words of the natural language content are fully supported by the server device for the version: providing a request for action data to the server device in communication with the vehicle computing device, and after the server device receives the request, receiving the action data from the server device, and causing the vehicle computing device to perform an action represented by the action data. The method may also include, in response to determining that the one or more terms of the natural language content for the version are partially supported by the server device: providing different requests for at least intent data from the server device, in response to the server device receiving the different requests, receiving the intent data from the server device, wherein the intent data characterizes the specific intent, and based on the intent data, causing the vehicle computing device to perform different actions associated with the specific intent.

[0083] In some implementations, the action corresponds to an application and the different actions correspond to different applications. In some implementations, both the action and the different actions correspond to a hardware subsystem of the vehicle. In some implementations, the method may also include, in response to determining that the one or more words of the natural language content for the version are no longer supported by the server device: providing a separate request for at least text data from the server device, in response to the server device receiving the separate request, receiving the text data from the server device, wherein the text data characterizes the natural language content of the spoken utterance, and based on the text data, causing the vehicle computing device to select a specific action for execution at the vehicle computing device.

[0084] Although several implementations have been described and shown herein, various other devices and / or structures for performing the functions described herein and / or obtaining the results and / or one or more advantages described herein may be utilized, and each of such changes and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications using this teaching. Those skilled in the art will recognize or be able to determine many equivalents of the specific implementations described herein using no more than routine experiments. Therefore, it should be understood that the aforementioned implementations are presented only by way of example, and within the scope of the appended claims and their equivalents, implementations may be implemented in a manner different from that specifically described and claimed. The implementations of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, if these features, systems, articles, materials, kits, and / or methods are not mutually contradictory, any combination of two or more of these features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A method implemented by one or more processors, the method comprising: determining that first natural language content contained in a first spoken utterance received at the first computing device corresponds to a first intent request; based on determining that the first natural language content corresponds to the first intent request, determining a first extent to which the first intent request is supported for a first version corresponding to the first computing device; generating first data including action data for the first intent request based on the first extent to which the first intent request is supported for the first version, the action data being executable by the first computing device to satisfy the first intent request; providing the first data to the first computing device, wherein providing the first data to the first computing device enables the first computing device to use the action data of the first data to satisfy the first intention request; determining that the first natural language content contained in audio data capturing a second spoken utterance received at a second computing device corresponds to the first intent request; determining, based on determining that the second natural language content includes the first intent request, a second degree to which the first intent request is supported by the server device for a second version corresponding to the second computing device, wherein the second version is different from the first version, and wherein the second degree is less than the first degree; generating second data based on the second degree to which the second intent request is supported by the server device, the second data not including the action data but including speech-to-text data generated by the server device based on speech-to-text processing performed on the audio data capturing the second spoken utterance; and providing the second data to the second computing device, Wherein, providing the second data to the second computing device causes the second computing device to locally perform an action generated locally by the second computing device based on the speech-to-text data of the second data.

2. The method according to claim 1, wherein: The first computing device executes the action data to perform an alternative action different from the action generated locally by the second computing device.

3. The method according to claim 2, wherein: Performing the alternative action includes utilizing sensor output from one or more sensors of the first computing device, and wherein performing the alternative action is independent of utilizing any sensor output from any sensor of the second computing device.

4. The method according to claim 3, wherein: The first computing device is a first vehicle computing device and the second computing device is a second vehicle computing device.

5. The method according to claim 1, wherein: The second computing device generates the action locally using a local natural language understanding (NLU) engine and a local action engine.

6. The method according to claim 1, wherein: The second computing device generates the action locally using a local action engine.

7. The method according to claim 1, further comprising: receiving the audio data and second device version data from the second computing device; as well as The second version is determined based on the second device version data received from the second computing device.

8. A method implemented by one or more processors of a vehicle computing device of a vehicle, the method comprising: Transmitting a first transmission to a server, the first transmission comprising: first audio data detected at the vehicle computing device and capturing an instance of a user uttering a particular request, and version data indicating a version of the vehicle computing device; In response to the first transmission, receiving from the server: action data generated by the server device based on processing the first audio data and generated by the server based on a determination at the server that the version data satisfies one or more conditions; in response to receiving the action data from the server, executing the action data to perform an action to satisfy the specific request; transmitting a second transmission to the server, the second transmission comprising: second audio data detected at the vehicle computing device and capturing additional instances of the user or additional users uttering the particular request, and the version data, wherein the second transmission occurs after the first transmission; In response to the second transmission, receiving from the server: data comprising speech-to-text data but lacking any motion data, wherein the speech-to-text data is generated by the server by performing speech-to-text processing on the audio data, and wherein the data lacks any motion data based on an additional determination at the server that the version data no longer satisfies the one or more conditions; and In response to receiving the speech-to-text data from the server, locally generating a local action using the speech-to-text data to satisfy the specific request, Wherein, the local action is different from the action.

9. The method according to claim 8, wherein: Executing the action data to perform the action includes utilizing sensor output from one or more sensors of the vehicle, and wherein the local action lacks utilizing any sensor output from any sensor of the vehicle.

10. The method according to claim 8, wherein: Generating the local action locally using the speech-to-text data includes generating the action based on processing the speech-to-text data using a local natural language understanding (NLU) engine to generate NLU data.

11. The method according to claim 10, wherein: Locally generating the local action using the speech-to-text data further includes generating the action based on processing the NLU data using a local action engine.

12. The method according to claim 11, wherein: Executing the action data to perform the action includes utilizing sensor output from one or more sensors of the vehicle, and wherein the local action lacks utilizing any sensor output from any sensor of the vehicle.

13. A method implemented by one or more processors of a vehicle computing device, the method comprising: Transmitting a first transmission from the vehicle computing device to a server device, the first transmission comprising: first request data based on first audio data detected at the vehicle computing device and capturing an instance of a user uttering a particular request, and version data indicating a version of the vehicle computing device; At the vehicle computing device, in response to the first transmission, receiving from the server: action data generated by the server device based on processing the request data; In response to receiving the action data from the server, executing the action data to satisfy the specific request; transmitting a second transmission to the server device, the second transmission comprising: second request data based on second audio data detected at the vehicle computing device and capturing additional instances of the user or additional users uttering the particular request, and the version data, wherein the second transmission occurs after the first transmission; At the vehicle computing device, in response to the second transmission, receiving from the server: instruction data to prevent the vehicle computing device from transmitting further request data for the particular request; and After receiving the instruction data: In response to detecting further audio data at the vehicle computing device capturing further verbal instances of the particular request, based on the instruction data, preventing any data from being transmitted to the server.

14. The method according to claim 13, wherein: The first request data includes the first audio data, and the second request data includes the second audio data.

15. The method according to claim 13, wherein: Receiving the instruction data from the server in response to the second transmission, in response to the server determining that the particular request is no longer supported by the server for the version indicated by the version data at the time of the second transmission.

16. The method of claim 13, further comprising in response to a further instance of the specific request: selecting locally at the vehicle computing device a local action to satisfy the specific request; and The specific request is satisfied locally at the vehicle computing device using the selected local action.

17. The method according to claim 16, wherein: The local action is different from a substitute action that is performed when performing the action data received from the server in response to the first transmission.

18. The method according to claim 16, wherein: Executing the action data to satisfy the specific request includes utilizing sensor output from one or more sensors of the vehicle, and wherein locally satisfying the specific request lacks utilizing any sensor output from any sensor of the vehicle.

19. The method according to claim 16, wherein: Locally satisfying the specific request includes using a local natural language understanding (NLU) engine to generate NLU data.

20. The method according to claim 19, wherein: Locally satisfying the specific request further includes processing the NLU data using a local action engine.

21. The method according to claim 20, wherein: Executing the action data to satisfy the specific request includes utilizing sensor output from one or more sensors of the vehicle, and wherein locally satisfying the specific request lacks utilizing any sensor output from any sensor of the vehicle.

22. A computing device in a vehicle, the computing device comprising: a memory storing instructions; one or more processors operable to execute the instructions to: Transmitting a first transmission from the vehicle computing device to a server device, the first transmission comprising: first request data based on first audio data detected at the vehicle computing device and capturing an instance of a user uttering a particular request, and version data indicating a version of the vehicle computing device; At the vehicle computing device, in response to the first transmission, receiving from the server: action data generated by the server device based on processing the first request data; In response to receiving the action data from the server, executing the action data to satisfy the specific request; transmitting a second transmission to the server device, the second transmission comprising: second request data based on second audio data detected at the vehicle computing device and capturing additional instances of the user or additional users uttering the particular request, and the version data, wherein the second transmission occurs after the first transmission; At the vehicle computing device, in response to the second transmission, receiving from the server: instruction data to prevent the vehicle computing device from transmitting further request data for the particular request; and After receiving the instruction data: In response to detecting further audio data at the vehicle computing device capturing further verbal instances of the particular request, based on the instruction data, preventing any data from being transmitted to the server.

23. The computing device of claim 22, wherein: The first request data includes the first audio data, and the second request data includes the second audio data.

24. The computing device of claim 22, wherein: Receiving the instruction data from the server in response to the second transmission, in response to the server determining that the particular request is no longer supported by the server for the version indicated by the version data at the time of the second transmission.

25. The computing device of claim 22, wherein: One or more of the processors are further operable to execute the instructions to: selecting locally at the vehicle computing device a local action to satisfy the specific request; and The specific request is satisfied locally at the vehicle computing device using the selected local action.

26. The computing device of claim 25, wherein: The local action is different from a substitute action that is performed when performing the action data received from the server in response to the first transmission.

27. The computing device of claim 25, wherein: In executing the action data to satisfy the specific request, one or more of the processors will utilize sensor outputs from one or more sensors of the vehicle, and wherein locally satisfying the specific request lacks utilizing any sensor output from any sensor of the vehicle.

28. The computing device of claim 25, wherein: When the specific request is satisfied locally, one or more of the processors will use a local natural language understanding (NLU) engine to generate NLU data.

29. The computing device of claim 28, wherein: When the specific request is satisfied locally, one or more of the processors will process the NLU data using a local action engine.

30. The computing device of claim 29, wherein: In executing the action data to satisfy the specific request, one or more of the processors will utilize sensor outputs from one or more sensors of the vehicle, and wherein locally satisfying the specific request lacks utilizing any sensor output from any sensor of the vehicle.

Images