Voice Control of a Vehicle System

Abstract

Examples for voice control of a vehicle system are disclosed. An example system for controlling a vehicle includes: a microphone; at least one vehicle component; and an interface system communicatively coupled to the microphone and the at least one vehicle component, the interface system being configured to: detect speech of a spoken command issued by an occupant of the vehicle via the microphone; transmit the speech to at least one of a virtual personal assistant (VPA) of the vehicle and a VPA of a mobile device located within the vehicle; receive a control signal from at least one of the VPA of the vehicle and the VPA of the mobile device, the control signal being generated based on the spoken command; and transmit the control signal to the at least one vehicle component to control the at least one vehicle component.

Description

Technical Field

[0001] The present disclosure relates to an in-vehicle interface system and associated control. Background Art

[0002] A vehicle may include an in-vehicle computing system, such as a host unit of an infotainment system, which may provide multimedia and control functions. For example, the in-vehicle computing system may provide navigation, multimedia playback, telephone, social media interaction, mobile device charging, and / or other functions, and receive user input to control elements of the vehicle and the in-vehicle computing system. To provide such functions, the in-vehicle computing system includes complex and expensive processing resources and memory resources capable of executing an operating system and various applications. Additionally, considering such extended functionality, the control of such an in-vehicle computing system may be too complex for a user whose attention should be focused on safely operating the vehicle. Summary of the Invention

[0003] To improve the control of an in-vehicle computing system, the in-vehicle computing system may accept voice commands as user input to control elements of the vehicle and the in-vehicle computing system. However, since the basic functions of the in-vehicle computing system may already utilize most of the processing power and memory resources of the in-vehicle computing system, the in-vehicle computing system may not be able to support a complex voice control system. To overcome the limitations of the in-vehicle computing system, the processing power and memory resources of a mobile device located within the vehicle may be utilized to provide complex voice control of vehicle components and the in-vehicle computing system.

[0004] Embodiments of methods for an in-vehicle system and controlling a vehicle system using a mobile device in communication with the in-vehicle system are disclosed. An example system for controlling a vehicle includes: a microphone; at least one vehicle component; and an interface system communicatively coupled to the microphone and the at least one vehicle component, the interface system being configured to: detect speech of a spoken command issued by an occupant of the vehicle via the microphone; transmit the speech to at least one of a virtual personal assistant (VPA) of the vehicle and a VPA of a mobile device located within the vehicle; receive a control signal from at least one of the VPA of the vehicle and the VPA of the mobile device, the control signal being generated based on the spoken command; and transmit the control signal to the at least one vehicle component to control the at least one vehicle component.

[0005] An example method of a vehicle system for controlling a vehicle using a mobile device includes: receiving, at an in-vehicle interface system, speech of an occupant of the vehicle via a microphone; transmitting the speech from the in-vehicle interface system to the mobile device, wherein the mobile device processes the speech to determine a command; receiving, at the in-vehicle interface system, a control signal from the mobile device, the control signal being generated by the mobile device based on the command; and transmitting the control signal from the in-vehicle interface system to at least one vehicle system of the vehicle to control the at least one vehicle system.

[0006] An example system for a vehicle system for controlling a vehicle includes: a mobile device located within the vehicle; and an interface system integrated into the vehicle and communicatively coupled to the vehicle system and the mobile device, the interface system being configured to: receive, via a microphone, speech of a command issued by an occupant of the vehicle; transmit the speech from the interface system to the mobile device; receive, at the interface system, a control signal from the mobile device; and transmit the control signal from the interface system to the vehicle system for controlling the vehicle system, wherein the mobile device is configured to receive the speech from the interface system, convert the speech to natural language text, identify the command in the natural language text, generate the control signal based on the command, and transmit the control signal to the interface system. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present disclosure may be better understood by reading the following description of non-limiting embodiments with reference to the accompanying drawings, in which:

[0008] Figure 1 is a partial view of a vehicle cabin including an in-vehicle computing system and a mobile device, according to one or more embodiments of the present disclosure;

[0009] Figure 2 is a block diagram of an interface system within a vehicle, according to one or more embodiments of the present disclosure;

[0010] Figure 3 is a flowchart of a method for selecting a virtual personal assistant of a vehicle or a mobile device to control one or more vehicle systems, according to one or more embodiments of the present disclosure;

[0011] Figure 4 is a flowchart of a method for using a virtual personal assistant of a vehicle and a mobile device to concurrently process voice commands to control one or more vehicle systems, according to one or more embodiments of the present disclosure; and

[0012] Figure 5A flowchart of a method for switching between a vehicle and a virtual personal assistant of a mobile device according to one or more embodiments of the present disclosure. Detailed Description

[0013] A virtual personal assistant (VPA) may include an automatic speech recognition system that detects spoken commands in spoken speech and performs actions in response to the spoken commands. For example, the VPA may detect a spoken command (e.g., "turn up the volume," "roll down the window," "navigate to the nearest gas station," "play classical music," etc.) and then execute the command if possible. While some functions (such as adjusting audio settings or adjusting the window) may be easily performed by an in-vehicle computing system, other functions (such as navigating to a specific location or playing a specific song by a specific musician on a stereo) may not be easily performed by an in-vehicle computing system. As described above, a mobile device may include available processing resources and / or storage resources that are equal to or exceed those typically used to provide functions associated with an in-vehicle computing system. To provide reliable voice control of a vehicle system, the present disclosure provides systems and methods for selectively routing spoken commands to a VPA installed on a mobile device or to a VPA of an in-vehicle computing system.

[0014] Figure 1 An example partial view of the interior of the passenger compartment 100 of a vehicle 102 is shown, in which a driver and / or one or more passengers may be seated. Figure 1 The vehicle 102 may be a motor vehicle including drive wheels (not shown) and an internal combustion engine 104. The internal combustion engine 104 may include one or more combustion chambers that may receive intake air via an intake passage and discharge combustion gases via an exhaust passage. The vehicle 102 may be a road vehicle, as well as other types of vehicles. In some examples, the vehicle 102 may include a hybrid propulsion system that includes an energy conversion device that may be operable to absorb energy from vehicle movement and / or the engine and convert the absorbed energy into a form suitable for storage by an energy storage device. The vehicle 102 may include a fully electric vehicle incorporating a fuel cell, a solar capture element, and / or other energy storage systems that power the vehicle.

[0015] As shown, the dashboard 106 may include various displays and controls accessible to the driver of the vehicle 102, such as a touchscreen 108 of an in-vehicle interface system (e.g., an infotainment system), an audio system control panel, and an instrument cluster 110. While Figure 1The example system shown includes audio system controls that can be performed via a user interface of the vehicle interface system (such as touchscreen 108) without a separate audio system control panel. However, in other embodiments, vehicle 102 can include an audio system control panel that can include controls for a conventional vehicle audio system (such as a radio, CD player, MP3 player, etc.). The audio system controls can include features for controlling one or more aspects of the audio output via speakers 112 of the vehicle speaker system. For example, the audio system controls can control the volume of the audio input, the distribution of sound among the various speakers of the vehicle speaker system, the equalization of the audio signal, and / or any other aspect of the audio output.

[0016] The instrument cluster 110 can include various gauges, such as a fuel gauge, tachometer, speedometer, and odometer, as well as indicators and warning lights. The steering wheel 114 can protrude from the dashboard 106 below the instrument cluster 110. Optionally, the steering wheel 114 can include controls 116 that can be used in conjunction with the touchscreen 108 to navigate the features of the vehicle interface system and control the vehicle interface system. As an illustrative example, the controls 116 can include a button 117 that activates or initiates voice control of the vehicle interface system. For example, in response to the button 117 being pressed and / or while the button 117 is being pressed, the microphone 126 can record one or more utterances of the driver or other passengers of the vehicle. If the one or more utterances recorded via the microphone 126 include voice commands, then the one or more utterances can then be used to navigate or control one or more features of the vehicle interface system.

[0017] In addition to Figure 1 the components depicted, it should be understood that the dashboard 106 can include additional components, such as door and window controls, a cigarette lighter that can also be used as a low-voltage power outlet, a glove box, and / or any other suitable elements. In one or more embodiments, in-vehicle climate control via the climate control system vents 118 can be performed using the touchscreen 108, and thus a separate climate control interface may not be included in the dashboard 106. However, in alternative embodiments, a separate climate control interface can be provided.

[0018] Compartment 100 may include one or more sensors for monitoring the vehicle, the user, and / or the environment. For example, compartment 100 may include one or more pressure sensors 120 mounted on the seats, which are configured to measure the pressure applied to the seats to determine the presence of a user. Compartment 100 may include one or more door sensors 122, which are configured to monitor door activities, such as opening and / or closing of the doors, locking of the doors, operation of the windows of the doors, and / or any other suitable door activity events. A humidity sensor 124 may be included to measure the moisture content of compartment 100. A microphone 126 may be included to receive user input in the form of voice commands, enable the user to make a phone call, and / or measure the ambient noise in compartment 100. It should be understood that Figure 1 the placement of the sensors shown is exemplary, and one or more additional or alternative sensors may be located in the engine compartment, on the outer surface of the vehicle, and / or other suitable locations for providing information about the operation of the vehicle, the environmental conditions of the vehicle, the users of the vehicle, etc.

[0019] Compartment 100 may also include one or more user objects stored in the vehicle before, during, and / or after travel, such as mobile device 128. The mobile device may include a smart phone, a tablet computer, a laptop computer, a portable media player, and / or any suitable mobile computing device. Mobile device 128 may be connected to the in-vehicle interface system via communication link 130. Communication link 130 may be wired (e.g., via Universal Serial Bus [USB], Mobile High-Definition Link [MHL], High-Definition Multimedia Interface [HDMI], etc.) or wireless (e.g., via Bluetooth, WI-FI, Near Field Communication [NFC], etc.), and is configured to provide two-way communication between mobile device 128 and the in-vehicle interface system. For example, communication link 130 may provide sensor signals and / or control signals from the in-vehicle system and touch screen 108 to mobile device 128, and may provide control and / or display signals from mobile device 128 to the in-vehicle system and touch screen 108. Communication link 130 may also provide power from the in-vehicle power supply to mobile device 128 to charge the internal battery of mobile device 128.

[0020] Although mobile device 128 is shown as being spatially separated from the in-vehicle interface system and connected via a substantially external communication link (e.g., a cable or a radio frequency signal), it should be understood that a mobile device storage structure 132 or other storage structure may be formed in dashboard 106 or another location in the vehicle to hold mobile device 128 in a specific position. Mobile device storage structure 132 may include an integrated connector 134 to which mobile device 128 may be attached or "docked" for providing a substantially internal communication link between mobile device 128 and the interface system.

[0021] Figure 2 FIG. 204 shows a block diagram of an in-vehicle interface system 204 and associated components with which the in-vehicle interface system 204 communicates. The in-vehicle interface system 204 may be located in and / or integrated within a vehicle 202, such as Figure 1 in vehicle 102. The in-vehicle interface system 204 may communicate with one or more components within the vehicle, including but not limited to vehicle systems connected via an in-vehicle interconnect, such as a controller area network (CAN) bus 206. It should be understood that any suitable number of interconnects and / or combinations of interconnects may be used to permit communication between the in-vehicle interface system 204 or simply the interface system 204 and various in-vehicle components, including but not limited to CAN bus, media-oriented system transport (MOST) bus, Ethernet-based interconnects, etc. The interconnects may communicate directly with the in-vehicle components and / or may communicate with such components via an intervening processor. In some embodiments, one or more in-vehicle components may communicate directly with the interface system 204 without communicating with or as a supplement to the interface system 204 via the CAN bus 206.

[0022] The interface system 204 may transfer information from the CAN bus to an external mobile device 250 to utilize the memory 252 and processor 254 and other computing resources of the mobile device 250 to control various in-vehicle components of the vehicle 202. In some embodiments, the interface system 204 may be connected to the mobile device 250 via a wired connection. In additional or alternative embodiments, the interface system 204 may include a wireless gateway and / or protocol 236 to enable a wireless connection between the interface system 204 and the mobile device 250. The mobile device 250 similarly includes a protocol 256 for enabling communication between the interface system 204 and the mobile device 250. The protocols 236 and 256 may include standard protocols, such as SmartDeviceLink (SDL), or proprietary protocols, for example.

[0023] A display 210 may display images to provide visual feedback related to navigation, media playback, telephone, vehicle system control (e.g., cabin temperature control, cruise control settings, odometer output / settings, vehicle diagnostic control, vehicle operation mode control, etc.) and / or other vehicle-related functions. In some embodiments, the display 210 may include a touch screen 214, such as Figure 1 touch screen 108, to enable receipt of user input for controlling vehicle-related functions via the display 210.

[0024] The mobile device 250 can receive signals from the interface system 204 and / or the display 210, process the signals based on non-transitory instructions stored on the memory 252 and executed by the processor 254, and output display signals and / or control signals to the display 210 and / or the interface system 204. In some embodiments, the mobile device 250 can communicate with the remote service 295 via the network 290. In such an embodiment, the mobile device 250 can utilize the computing resources of the remote service 295 and / or additional information stored at the remote service 295 to execute and / or assist in processing signals from the interface system 204. For example, the mobile device 250 can utilize the remote service 295 to assist in generating a response to a specific user input. Although Figure 2 only one remote service is shown, it should be understood that the mobile device 250 can communicate with one or more remote services including the remote service 295 during operation.

[0025] As shown, the mobile device 250 is separated from the vehicle 202 and not integrated within the vehicle 202, but the mobile device 250 can be located within the vehicle cabin during travel and simultaneously connected to the interface system 204 (e.g., via a communication link such as Figure 1 communication link 130). In contrast, Figure 2 each of the other systems and components shown within the vehicle 202 in Figure 2 can be integrated within the vehicle. The mobile device 250 can include computing resources (e.g., the memory 252 and the processor 254) separate from the computing resources integrated within the vehicle 202 (e.g., the computing resources utilized by the components shown within the vehicle 202 of

[0026] The display signal can control the output of the display 210, while the control signal can control one or more other vehicle systems 216 that communicate with the interface system 204. For example, the vehicle systems 216 can include controllable elements related to the engine and / or auxiliary components, such as windshield wipers, windows, doors / door locks, headlights, air conditioning systems, etc. The control signal can also control the audio output at one or more speakers 218 of the vehicle 202. For example, the control signal can adjust audio output characteristics, such as volume, equalization, audio image (e.g., the configuration of an audio signal for generating an audio output that seems to originate from one or more defined locations for the user), audio distribution among the multiple speakers 218, etc.

[0027] For example, user input can be received via one or more in-vehicle controls 220 as an additional or alternative input source for transmission to the interface system 204 and / or the mobile device 250, which is an optional touchscreen of the display 210. The in-vehicle controls 220 can include any suitable user-actuable elements inside and / or on the vehicle, including but not limited to any number of buttons, sliders, switches, knobs, levers, joysticks, keypads, pedals, etc. or combinations thereof. For example, the in-vehicle controls 220 can include control elements inside the passenger compartment of the vehicle 202, such as steering wheel controls (e.g., audio system controls mounted on the steering wheel, cruise controls, windshield wiper controls, headlight controls, turn signal controls, etc.), dashboard controls, accelerator pedal / brake pedal / clutch pedal, gearshift, door / window controls located in the driver's door or passenger's door, seat controls, cabin light controls, audio system controls, cabin temperature controls, etc. Control elements located outside the vehicle (e.g., controls for the security system) can also be connected to the interface system 204 via the CAN bus 206. The control elements of the in-vehicle controls 220 can be physically and permanently located on and / or in the vehicle for receiving user input, even when the mobile device 250 is removed from the vehicle.

[0028] As a specific example, the in-vehicle controls 220 can include a microphone 221 integrated inside the passenger compartment of the vehicle 202 that records or senses user input, such as voice control commands routed to the interface system 204 and / or the mobile device 250. As another example, the steering wheel controls and / or dashboard controls of the in-vehicle controls 220 can include Figure 1The button 117 for initiating voice control depicted therein, where pressing the button 117 indicates that the speech sensed by the microphone 221 includes a voice command. In this way, speech not intended for voice control of vehicle components is not inadvertently processed by the interface system 204 and / or the mobile device 250 to control one or more components of the vehicle 202. Additionally or alternatively, the user may indicate that the speech is intended for voice control by speaking a wake word for voice control (e.g., a specific voice data string, which may be a word, phrase, tone, or other predefined sound). For example, the interface system 204 and / or the mobile device 250 may monitor the audio recorded by the microphone 221 and specifically process the audio to detect the wake word without otherwise processing or disposing of the audio, and in response to detecting the wake word spoken by an occupant of the vehicle 202, the interface system 204 and / or the mobile device 250 may activate the corresponding VPAs 230 and 258 to process the speech sensed by the microphone 221 and following the wake word or trigger word, as further described herein.

[0029] The CAN bus 206 may communicate with one or more vehicle sensors 222 to provide an indication of the operating state and / or condition of the vehicle to the interface system 204. The interface system 204 may pass sensor signals from the vehicle sensors 222 to the mobile device 250 to provide the mobile device 250 with context information related to the operation and / or environment of the vehicle and / or the user. The vehicle sensors 222 may include any suitable sensors, such as the in-cabin sensors, vehicle operating state sensors (e.g., sensors monitoring fuel storage, engine temperature, oil storage, engine speed, vehicle speed, transmission / gear setting, tire pressure, traction, diagnostic information, etc.) and environmental sensors (e.g., rearview camera and / or other sensors monitoring environmental conditions such as temperature, humidity, pressure, road surface / condition, etc.) described above with respect to Figure 1 After receiving user input and / or sensor signals at the mobile device 250, the mobile device 250 may determine adjustments to the display 210 and / or control instructions for one or more vehicle systems 216.

[0030] Information regarding the power state and / or power control signal may be exchanged between the power module 224 and the mobile device 250 via the CAN bus 206 and the interface system 204. For example, the mobile device 250 may receive information from the power module 224 and update the display 210 to provide the user with feedback regarding the power state of one or more vehicle components. The mobile device 250 may additionally or alternatively provide a control signal to the power module 224 based on user input, information received from the vehicle sensors 222, and / or information received from the power module 224.

[0031] Antenna 226 can provide an AM / FM radio signal to interface system 204 for processing at mobile device 250 and / or interface system 204. For example, mobile device 250 can control speaker 218 to control the audio broadcast of an AM / FM radio station, and control display 210 to display information related to the AM / FM radio station and / or the content being broadcast (e.g., information accompanying the audio broadcast via a radio data system).

[0032] To communicate with the various vehicle systems and mobile device 250 as described above, in some embodiments, interface system 204 can include one or more modules for resolving different communication protocols and other issues that may arise when passing signals between disparate systems. For example, interface system 204 can include a video switching module (not shown) to synchronize and / or format the video feed from a rearview camera for display on display 210. As another example, interface system 204 can include an encoder (not shown) and / or a decoder (not shown) for converting signals received from one system and / or device into a format compatible with or otherwise interpretable by a target system and / or device. It should be understood that one or more of the above modules are optional, and in some embodiments, interface system 204 can pass received signals to another system and / or device without providing any adjustment to the signals.

[0033] Interface system 204 includes a device platform 238 that includes an infotainment platform that provides primitives for communicating with components of vehicle 202. For example, device platform 238 can generate a user interface for display via display 210 and / or communicate with one or more components of vehicle 202, including but not limited to speaker 218 via audio amplifier 219, vehicle sensors 222, power module 224, vehicle system 216, in-vehicle controls 220, and / or antenna 226, for providing commands to and / or receiving signals from one or more of the vehicle components.

[0034] Interface system 204 includes a virtual personal assistant (VPA) 230 for supporting a voice dialogue between a user of vehicle 202 and interface system 204. VPA 230 includes a speech processing module that processes voice commands such as those received from microphone 221. Thus, VPA 230 performs automatic speech recognition on the received spoken utterances received via microphone 221 to convert the spoken utterances into natural language text and processes the natural language text to identify a spoken command from the spoken utterances. VPA 230 can also provide an audio response, for example, by using text-to-speech, which can be output via speaker 218.

[0035] To implement the commands of the voice commands recognized by the VPA 230, the interface system 204 also includes a VPA service module 232 that provides access to the device functions. In particular, the VPA service module 232 provides a directory of device functions and interfaces for performing the device functions, and the directory maps the commands to the appropriate components of the vehicle 202. For example, the voice speech received by the microphone 221 can be processed by the VPA 230 to recognize the command "change the equalizer to jazz" in the voice speech. The VPA service module 232 links the commands of the voice speech recognized by the VPA 230 to the corresponding device functions that can be executed by the device platform 238, for example. For example, the VPA service module 232 can route the command "change the equalizer to jazz" recognized by the VPA 230 to the device platform 238, which in turn controls the audio equalization settings of the audio amplifier 219 and / or the speaker 218 to a combination of preset and optimized equalization settings for jazz music. Alternatively, instead of providing the command to the device platform 238, in some examples, the VPA service module 232 can communicate directly with and control one or more components of the vehicle 202 according to the command.

[0036] The interface system 204 also includes a VPA switch 234 for arbitrating between the VPA 230 of the vehicle 202 and the VPA 258 of the mobile device 250. The VPA switch 234 routes the speech sensed by the microphone 221 to, for example, the VPA 230 of the interface system 204, the VPA 258 of the mobile device 250, or both the VPA 230 and the VPA 258. Various methods for routing speech to one or more of the VPA 230 and the VPA 258 are further described herein with respect to Figures 3 to 5 Further description.

[0037] As described above, the mobile device 250 includes a VPA 258 for supporting a voice conversation between the user of the vehicle 202 and the interface system 204. Similar to the VPA 230 of the interface system 204, the VPA 258 of the mobile device 250 includes a speech processing module that processes voice commands, such as voice commands received from the microphone 221 and / or from a microphone (not shown) of the mobile device 250. Therefore, the VPA 258 performs automatic speech recognition on the received voice speech received via the microphone 221 to convert the voice speech into natural language text, and processes the natural language text to identify the utterance commands from the voice speech. The VPA 258 can also provide an audio response, for example, by using text-to-speech, which can be output via the speaker 218.

[0038] The mobile device 250 also includes a VPA service module 260 that provides a directory of device functions and interfaces for performing device functions, the directory mapping commands to appropriate components of the vehicle 202. The VPA service module 260 can also map commands to the mobile device platform 262.

[0039] In some examples, the VPA 258 of the mobile device 250 and the VPA 230 of the vehicle 202 can be the same. In other examples, the VPA 258 can include extended functionality relative to the VPA 230. As a specific example, the VPA 230 can be equipped with a limited vocabulary dedicated to the control of various vehicle components, and thus the VPA 230 may be limited to enabling voice control of vehicle components. In contrast, the VPA 258 can be equipped with a larger vocabulary not limited to controlling various vehicle components. Additionally, the VPA 258 can access an even larger vocabulary by communicating with a remote service 295 via the network 290. As an example, the VPA 230 can support adjusting the control of the audio amplifier 219 and the speaker 218, such as by adjusting the volume or equalization, and further, the VPA 230 can support controlling the device platform 238 to change radio transmissions received via the antenna 226. However, if the user verbally commands the VPA 230 to play a specific song by a specific artist, the vocabulary of the VPA 230 may be too limited to understand the command. Additionally, the functionality of the device platform 238 may not support streaming music not stored locally. In contrast, the VPA 258 may be able to process the spoken command to play a specific song by a specific artist and may also be able to stream the specific song via the mobile device platform 262 and / or the remote service 295.

[0040] Figure 3 is a flowchart of an example method 300 for routing a user's speech to a virtual personal assistant (VPA) of a vehicle or a mobile device. The method 300 can be executed by a system in the vehicle, such as an in-vehicle interface system. Additionally, one or more actions of the method 300 can be executed by the in-vehicle interface system, while other actions of the method 300 can be executed by a mobile device located in the vehicle and communicatively coupled to the in-vehicle interface system.

[0041] Method 300 begins at 305. At 305, method 300 includes receiving speech via a microphone. The speech includes one or more words spoken by a person in the vehicle, which may include a spoken command to control one or more components of the vehicle. A VPA switch of the in-vehicle interface system, such as VPA switch 234 of interface system 204, may receive the speech via the microphone, which may include a microphone of the vehicle, such as microphone 126 or microphone 221, or optionally may include a microphone of a mobile device. Additionally, method 300 may receive the speech in response to detecting a wake word spoken by a user or occupant of the vehicle. Method 300 may receive the speech spoken after the wake word and may complete the reception of the speech in response to an additional trigger word or in response to a threshold duration elapsing after the speech is received, without sensing additional speech of the user via the microphone. As another example, method 300 may receive the speech in response to detecting that a voice control button is pressed (such as button 117 being pressed). Additionally, VPA switch 234 may receive the speech via the microphone. As further described herein, VPA switch 234 determines in method 300 whether to route the speech to the vehicle VPA or to the mobile device VPA.

[0042] After receiving the speech, method 300 proceeds to 310. At 310, a VPA switch, such as VPA switch 234, determines whether a vehicle VPA, such as VPA 230 of interface system 204, exists or is present. For example, a vehicle VPA, such as VPA 230, may include an optional component of a vehicle, such as vehicle 202, and thus in some cases, the vehicle may not be equipped with a vehicle VPA to reduce the cost of the vehicle. However, interface system 204 may include VPA switch 234 for routing speech in the case where a vehicle VPA is ultimately installed in the vehicle. For example, method 300 thus determines whether the vehicle VPA is installed in the vehicle and connected to VPA switch 234.

[0043] If a vehicle VPA exists (“yes”), method 300 proceeds to 315. At 315, the VPA switcher routes the speech to the vehicle VPA to generate a response including one or more control signals based on the speech. For example, method 300 may utilize VPA 230 to process the speech to perform automatic speech recognition. That is, VPA 230 converts the speech into natural language text. VPA 230 may further process the natural language text to identify commands in the speech. As an example, VPA 230 may compare the natural language text derived from the speech with commands in a command directory stored in VPA service module 232. VPA 230 can thus determine that at least a portion of the natural language text corresponds to a command. In some examples, VPA 230 may determine that the natural language text does not include a command even if a portion of the natural language text does not match any command in the command directory stored in VPA service module 232. In any case, VPA 230 generates a response to the speech, the response including control signals to be transmitted by VPA service module 232 to one or more components.

[0044] Proceeding to 320, method 300 includes controlling one or more elements of the vehicle based on the response generated at 315. In cases where the speech does not include a command or at least a valid command for controlling a vehicle component, the response may indicate that the speech does not include a valid command, and thus method 300 may control one or more elements of the vehicle, such as display device 210 and / or speaker 218, to notify the user that the speech does not include a valid command. For example, method 300 may control display device 210 to display a message indicating that the speech does not include a valid command, and / or method 300 may control speaker 218 to replay a tone or voice message indicating that the speech does not include a valid command. Method 300 then returns. Thus, in some examples, method 300 may use the vehicle VPA to control one or more components of the vehicle in response to speech spoken by a vehicle user and based on the speech.

[0045] However, referring again to 310, if a vehicle VPA does not exist (“no”), method 300 proceeds to 325. At 325, method 300 includes determining whether a mobile device, such as mobile device 128 or 250, is connected and active. For example, a mobile device may be determined to be connected after establishing a wired or wireless connection to an interface system (such as Figure 2 interface system 204) and / or after completing a data transaction between the mobile device and the interface system (e.g., receiving data from the mobile device, performing a handshake operation, etc.). A mobile device may be determined to be active when it is running a vehicle interface application and / or is otherwise capable of receiving signals from the vehicle, processing the received signals, and transmitting signals to control one or more vehicle systems.

[0046] In response to determining at 325 that the mobile device is not connected and / or inactive (“NO”), the method 300 proceeds to 327. At 327, the method 300 includes outputting an alert to connect and / or activate the mobile device. For example, the method 300 may be displayed on a display device (e.g., Figure 2 Additionally or alternatively, the method 300 may be performed via one or more speakers integrated in the vehicle (e.g., Figure 2 The method 300 then returns. Thus, if the vehicle VPA is not present and the mobile device is not connected and inactive, the method 300 may not control one or more components of the vehicle based on speech.

[0047] Referring again to 325, if the method 300 determines that the mobile device is connected and active ("yes"), the method 300 continues to 330. At 330, the method 300 includes transmitting the speech to the mobile device. For example, for the interface system 204, the VPA switch 234 routes the speech to the mobile device 250 via the protocol 236. The mobile device 250 then utilizes the VPA installed on the mobile device (e.g., Figure 2 The method 300 may include receiving a response generated by the VPA of the mobile device from the mobile device 250. For example, the VPA service module 260 of the mobile device 250 may transmit the response to the interface system 204 of the vehicle 202 via the protocol 256 of the mobile device 250. The interface system 204 receives the response, for example, via the protocol 236, and at 340, the method 300 includes controlling one or more elements or components of the vehicle according to the response. The method 300 then returns.

[0048] Thus, in some examples, the VPA switch 234 may route speech to the vehicle VPA 230 or to the nomadic device VPA 258 depending on whether the vehicle VPA 230 is installed in the vehicle 202 .

[0049] However, in some examples where both the vehicle VPA 230 and the nomadic device VPA 258 are present in the vehicle 202, the VPA switch 234 may route speech to both the VPA 230 and 258. As an example, Figure 4It is a flowchart of an exemplary method 400 for parallel speech processing according to an embodiment. In particular, method 400 involves simultaneously or in parallel processing speech using the vehicle VPA of the in-vehicle interface system 402 and the mobile device VPA of the mobile device 404, and controlling one or more vehicle components 406 based on the outputs of the vehicle VPA and / or the mobile device VPA. The vehicle VPA of the in-vehicle interface system 402 may correspond to the VPA 230 of the interface system 204, and the mobile device VPA of the mobile device 404 may correspond to the VPA 258 of the mobile device 250 described above. One or more vehicle components 406 may correspond to one or more components of the vehicle 202 described above, including but not limited to the display 210, the vehicle system 216, the speaker 218 and / or the audio amplifier 219, the in-vehicle controls 220, the vehicle sensors 222, the power module 224, the antenna 226, the device platform 238, etc.

[0050] Method 400 begins at 410. At 410, the interface system 402 receives speech sensed, for example, by a microphone (such as microphone 221) in the vehicle at a VPA switch such as the VPA switch 234. At 412, the VPA switch transmits the speech from the interface system 402 to the mobile device 404, for example, via protocols 236 and 258. At 420, the mobile device 404 receives the speech from the interface system 402. At 422, the mobile device 404 processes the speech using the mobile device VPA to generate a mobile device VPA response. At 424, the mobile device 404 transmits the mobile device VPA response to the interface system 402. At 426, the interface system 402 receives the mobile device VPA response.

[0051] Meanwhile, when the mobile device 404 processes speech as described above, method 400 proceeds from 410 to 415. At 415, the interface system 402 processes the speech using the vehicle VPA to generate a vehicle VPA response. At 430, the interface system 402 evaluates the vehicle VPA response and the mobile device VPA response to determine one or more control signals for one or more target vehicle components specified by the responses. As an illustrative example, the VPA switch may evaluate the vehicle VPA response and the mobile device VPA response based on which response is received first. As another example, the VPA switch may evaluate the vehicle response and the mobile device VPA response to determine if the responses are different. If the responses are the same, the VPA switch uses either response as the control signal. However, if the responses are different, the VPA switch may determine that one response is better than the other and may select the best response as the control signal. For example, the speech may include a spoken command to navigate to a specific location with a location name. The vehicle VPA response may include, for example, a control signal to start the navigation system in the device platform, and a control signal for the display device and / or speaker to indicate that the VPA does not understand the location name. Meanwhile, the mobile device VPA response may include a control signal to start the navigation system to navigate to the specific location. Since the mobile device VPA response is a complete response while the vehicle VPA response is an incomplete response, the VPA switch may select the mobile device VPA response as the control signal. As yet another example, the VPA switch may use the first received response as the control signal and may evaluate the second received response to determine if the second response is better than the first response. For example, in the above example where the mobile device VPA accurately parses the location name while the vehicle VPA fails to parse the location name, the VPA switch may first receive the vehicle VPA response, so the interface system 402 may start the navigation system with an indication that the location name is not parsed, and then when the mobile device VPA response is received, the interface system 402 may overwrite the first response with the second response such that the navigation system navigates to the specific location.

[0052] Accordingly, continuing at 435, the interface system 402 transmits one or more control signals to one or more target vehicle components. At 440, one or more of the target vehicle components in the vehicle component 406 perform one or more actions based on the control signals. Method 400 then returns.

[0053] Thus, in some examples, speech may be processed by both the vehicle VPA and the mobile device VPA, and one or more vehicle components may subsequently be controlled based on the response from the vehicle VPA, the response from the mobile device VPA, or the responses from both.

[0054] In other examples, the VPA switcher may selectively route speech to the vehicle VPA or the mobile device VPA according to the user's intention. As an illustrative and non-limiting example, Figure 5 is a flowchart of a method 500 for selectively allocating speech between a vehicle VPA and a mobile device VPA. In particular, method 500 involves determining the intention of the speech, routing the speech to the vehicle VPA of the vehicle interface system 502 or the mobile device VPA of the mobile device 504 according to the intention of the speech, and controlling one or more vehicle components 506 according to the output of the vehicle VPA and / or the mobile device VPA. The vehicle VPA of the vehicle interface system 502 may correspond to the VPA 230 of the interface system 204, and the mobile device VPA of the mobile device 504 may correspond to the VPA 258 of the mobile device 250 described above. One or more vehicle components 506 may correspond to one or more components of the vehicle 202 described above, including but not limited to the display 210, the vehicle system 216, the speaker 218 and / or the audio amplifier 219, the vehicle controls 220, the vehicle sensors 222, the power module 224, the antenna 226, the device platform 238, etc.

[0055] Method 500 begins at 510. At 510, the interface system 502 receives speech. At 515, the interface system 502 predicts the user intent based on at least a portion of the speech. For example, the VPA switcher of the interface system 502 can be configured to convert an initial portion of the speech (e.g., one or more of the foremost words of the speech) into natural language text and determine whether the natural language text of the initial portion includes a command word. Such a command word in the initial portion of the speech can indicate the user's intent. For example, if the initial portion of the speech includes the word "navigate" or "direction", the VPA switcher can determine that the user intent is to obtain directions or assistance in navigating to a given location. As another example, if the initial portion of the speech includes the words "increase" or "decrease", the VPA switcher can determine that the user intent is to adjust the settings of a vehicle system (such as an audio system, one or more vehicle controls, or one or more vehicle systems) (e.g., adjust the cruise control speed, adjust the climate control settings, adjust the volume, etc.). Thus, the VPA switcher can include a list of command words linked to different expected functions. At 520, the interface system 502 determines whether the vehicle VPA supports the intent. For example, the list of command words linked to different expected functions can also specify whether the expected function can be performed by the vehicle VPA. As an example, functions related to controlling vehicle components such as vehicle system 216, vehicle controls 220, audio amplifier 219, and speaker 218 can generally be performed by the vehicle VPA. However, other functions (such as playing a song by a particular musician or navigating to a specific location) may require a larger vocabulary to generate a response or may require interaction with a remote service (such as remote service 295). The vocabulary of the vehicle VPA and the capabilities of the device platform 238 may be limited in these functions, so in some examples, the vehicle VPA may not support such an expected function.

[0056] Thus, if the vehicle VPA supports the intent ("yes"), method 500 continues to 522. At 522, the interface system 502 uses the vehicle VPA to process the speech to generate a vehicle VPA response. At 525, one or more vehicle components 506 perform an action based on the vehicle VPA response. Method 500 then returns.

[0057] However, again referring to 520, if the vehicle VPA does not support the intent ("no"), method 500 continues to 530. At 530, the interface system 502 transmits the speech to the mobile device 504. At 535, the mobile device 504 receives the speech. At 540, the mobile device uses the mobile device VPA to process the speech to generate a mobile device VPA response. At 545, the mobile device 504 transmits the mobile device VPA response to the interface system 502.

[0058] At 550, the interface system 502 receives the mobile device VPA response. At 555, one or more vehicle components 506 perform one or more actions in accordance with the mobile device VPA response. Method 500 then returns.

[0059] Accordingly, a method for voice control of vehicle components includes selectively routing spoken speech to an in-vehicle VPA system or to a mobile device VPA system according to a user's intent. According to one embodiment, a system for controlling a vehicle is provided. The system includes a microphone, at least one vehicle component, and an interface system communicatively coupled to the microphone and the at least one vehicle component. The interface system is configured to: detect, via the microphone, speech of a spoken command issued by a vehicle occupant; transmit the speech to at least one of a virtual personal assistant (VPA) of the vehicle and a VPA of a mobile device located within the vehicle; receive a control signal from at least one of the vehicle's VPA and the mobile device's VPA, the control signal being generated according to the spoken command; and transmit the control signal to the at least one vehicle component to control the at least one vehicle component.

[0060] A description of embodiments has been presented for purposes of illustration and description. Suitable modifications and changes may be made in light of the above description, or may be acquired by practicing the methods. For example, unless otherwise stated, one or more of the methods described may be performed by suitable devices and / or combinations of devices, such as the vehicle system and mobile device described above with respect to Figure 1 and Figure 2 described. The methods may be performed by executing stored instructions using one or more logic devices (e.g., processors) in combination with one or more hardware elements such as storage devices, memories, hardware network interfaces / antennas, switches, actuators, clock circuits, etc. The methods and associated actions described may also be performed in various orders other than the order described in this application, in parallel, and / or simultaneously. The systems described are exemplary in nature and may include additional elements and / or omit elements. The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various systems and configurations and other features, functions, and / or properties disclosed.

[0061] As used in this application, an element or step recited in the singular and preceded by the word "a" or "an" is to be understood as not excluding a plurality of said elements or steps, unless such exclusion is specified. Additionally, a reference to "one embodiment" or "an example" of the present disclosure is not to be construed as excluding the existence of additional embodiments that also include the recited features. The terms "first," "second," "third," etc. are used only as labels and are not intended to impose numerical requirements or a particular positional order on their objects. The appended claims particularly point out the subject matter regarded as novel and non-obvious from the foregoing disclosure.

Claims

1. A system for controlling a vehicle, which comprises: a microphone; at least one vehicle component; and an interface system communicatively coupled to the microphone and the at least one vehicle component, the interface system being configured to: detect, via the microphone, the speech of a spoken command issued by an occupant of the vehicle; determine whether a virtual personal assistant (VPA) of the vehicle is present within the vehicle; in response to determining that the VPA of the vehicle is present within the vehicle, transmit the speech to both the VPA of the vehicle and the VPA of a mobile device located within the vehicle, receive a first control signal from the VPA of the vehicle, receive a second control signal from the VPA of the mobile device, and determine a control signal for the at least one vehicle component by evaluating the first control signal and the second control signal; in response to determining that the VPA of the vehicle is not present within the vehicle, transmit the speech to the VPA of the mobile device, and receive a control signal from the VPA of the mobile device; and transmit the control signal to the at least one vehicle component to control the at least one vehicle component.

2. The system according to claim 1, wherein the interface system is further configured to determine whether the mobile device is connected to and active with the interface system, and display a message to connect the mobile device via a display device communicatively coupled to the interface system in response to determining that the mobile device is not connected to and not active with the interface system.

3. The system according to claim 1, wherein the interface system includes the VPA of the vehicle, and wherein the VPA of the vehicle and the VPA of the mobile device are configured to perform automatic speech recognition on the speech to identify the spoken command.

4. The system according to claim 1, wherein the interface system is further configured to predict the intention of the occupant upon receiving the speech, and transmit the speech to the VPA of the vehicle or the VPA of the mobile device according to the intention.

5. The system according to claim 4, wherein the interface system is configured to predict the intention of the occupant upon receiving the speech by converting at least one word of the speech into text and comparing the at least one word with a list of commands supported by the VPA of the vehicle.

6. The system according to claim 5, wherein the interface system is configured to transmit the speech to the VPA of the vehicle if the at least one word matches at least one command in the command list, and otherwise transmit the speech to the VPA of the mobile device.

7. The system according to claim 1, wherein the at least one vehicle component includes one or more of a speaker system of the vehicle, an engine system of the vehicle, an auxiliary system of the vehicle, a cruise control system of the vehicle, a display system of the vehicle, and a climate control system of the vehicle.

8. A method for a vehicle system for controlling a vehicle using a mobile device, the method comprises: receiving, at an in-vehicle interface system via a microphone, the speech of a voice command of an occupant of the vehicle; Determine whether a virtual personal assistant (VPA) of the vehicle exists within the vehicle; In response to determining that the VPA of the vehicle exists within the vehicle, transmit the speech from the in-vehicle interface system to both the VPA of the vehicle and the VPA of a mobile device located within the vehicle, receive a first control signal from the VPA of the vehicle and a second control signal from the VPA of the mobile device at the in-vehicle interface system, and determine a control signal for the vehicle system by evaluating the first control signal and the second control signal; In response to determining that the VPA of the vehicle does not exist within the vehicle, transmit the speech from the in-vehicle interface system to the VPA of the mobile device and receive a control signal from the VPA of the mobile device at the in-vehicle interface system; and Transmit the control signal from the in-vehicle interface system to at least one vehicle system of the vehicle to control the at least one vehicle system.

9. The method according to claim 8, further comprising: Predict an expected command using the in-vehicle interface system when receiving the speech; Determine that the in-vehicle interface system supports the expected command; and In response to determining that the in-vehicle interface system does not support the expected command, transmit the speech to the mobile device.

10. The method according to claim 9, further comprising, in response to determining that the in-vehicle interface system supports the expected command, not transmitting the speech to the mobile device.

11. The method according to claim 10, further comprising, in response to determining that the in-vehicle interface system supports the expected command: Process the speech using the in-vehicle interface system to determine the command; Generate a second control signal using the in-vehicle interface system according to the command; and Transmit the second control signal from the in-vehicle interface system to the at least one vehicle system to control the at least one vehicle system.

12. The method according to claim 8, wherein evaluating the first control signal and the second control signal to determine the control signal includes receiving the first control signal before the second control signal and selecting the first control signal as the final control signal, or receiving the second control signal before the first control signal and selecting the second control signal to determine the control signal.

13. A system for controlling a vehicle system of a vehicle, the system comprising: A mobile device located within the vehicle; An interface system integrated into the vehicle and communicatively coupled to the vehicle system and the mobile device, the interface system being configured to: Receive speech of a command issued by an occupant of the vehicle via a microphone; Determine whether the interface system includes a speech recognition system; In response to determining that the interface system includes the speech recognition system, transmit the speech from the interface system to both the virtual personal assistant (VPA) of the vehicle and the VPA of the mobile device, receive a first control signal from the VPA of the vehicle and a second control signal from the VPA of the mobile device at the interface system, and determine a control signal for the vehicle system by evaluating the first control signal and the second control signal; In response to determining that the interface system does not include a speech recognition system, transmit the speech from the interface system to the VPA of the mobile device, and receive a control signal from the VPA of the mobile device at the interface system; and Transmit the control signal from the interface system to the vehicle system to control the vehicle system; wherein the mobile device is configured to: Receive the speech from the interface system; Convert the speech into natural language text; Identify the command in the natural language text; Generate the control signal according to the command; And Transmit the control signal to the interface system.

14. The system according to claim 13, wherein the interface system is configured to transmit the speech to the mobile device in response to determining that the mobile device is communicatively coupled to the interface system and is active.

15. The system according to claim 13, wherein the mobile device is configured to access a remote service via a network to generate the control signal.

16. The system according to claim 13, wherein the vehicle system includes one or more of a display system, an audio system, a cruise control system, a climate control system, one or more steering wheel controls, one or more dashboard controls, one or more vehicle sensors, and the device platform of the interface system.

Images