PRELOAD AND DELAYED CHARGING RESULTS OF IN-VEHICLE DIGITAL ASSISTANCE WANTED SEARCH

Abstract

In-vehicle system, comprehensive: an audio input; a transceiver; and a processor, programmed to Receiving a voice request from a vehicle occupant via the audio input, Evaluate the query locally to obtain initial local search results from a local digital assistant on the vehicle. Presenting the initial search results, Using the send-receiver in parallel with the presentation of the initial search results to simultaneously evaluate the query using one or more remote digital assistants to receive historical and new search results, and Merging local and remote results in the initial search results presentation to provide additional search results for delayed loading.

Description

TECHNICAL FIELD

[0001] Aspects of the disclosure generally relate to preloading and deferred loading results from in-vehicle voice searches to one or more digital assistants. GENERAL STATE OF THE ART

[0002] Digital assistants generally refer to application programs that understand natural language commands and perform tasks for the user. Virtual assistants are typically cloud-based programs that require internet-connected devices and / or applications to function. As a result, such systems can exhibit significant latency in responding to requests. However, the practical utility of digital assistants continues to improve, and the adoption of such systems continues to grow.

[0003] US 2016 / 0 357 853 A1 discloses a system intended to provide enhanced functionality on a terminal device, such as a smartphone. The system comprises a voice input module for receiving voice input from a user for a search query. A local search and a remote search are then performed, and the user is presented with combined search results consisting of the local search results and the remote search results. A speech recognition-based search system for big data searches in DFS systems is disclosed in US 2018 / 0 089 258 A1. Another voice-based search system for searching databases with digital content is known, among other things, from US 2017 / 0 147 585 A1. US 2011 / 0 047 605 A1 discloses an identification of a computer system user based on voice information. SUMMARY

[0004] The in-vehicle system according to the present invention includes an audio output, a transceiver, and a processor. The processor is programmed to receive a voice query from a vehicle occupant via the audio input, evaluate the query locally to provide initial search results, and use the transceiver in parallel to simultaneously evaluate the query using one or more remote digital assistants, to receive historical and new search results, and to merge the local and remote results to provide additional search results for delayed loading.

[0005] The method according to the present invention includes receiving a voice query from a vehicle occupant via a microphone installed in the vehicle, evaluating the query locally to provide initial search results, using a network connection in parallel to evaluate the query simultaneously using one or more digital assistants to receive historical and new search results, and merging the local results and the remote results to perform a lazy loading of additional search results after the initial search results have been provided.

[0006] In one or more illustrative examples, a non-transitory computer-readable medium containing instructions that, when executed by one or more processors, cause the one or more processors to receive a voice query from a vehicle occupant via a microphone installed in the vehicle, evaluate the query locally to provide initial search results, use a network connection in parallel to simultaneously evaluate the query using one or more digital assistants to receive historical and new search results, and merge the local results and the remote results to perform a lazy loading of additional search results after the initial search results are provided. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 shows an example diagram of a system configured to provide pre-load and deferred load results from in-vehicle voice searches using one or more digital assistant services; Fig. 2 illustrates an example data flow diagram for processing pre-loading and delayed loading of results from in-vehicle voice searches using one or more digital assistant services; Fig. 3 illustrates an example data flow diagram for identity processing for use in pre-loading and delayed loading of results from in-vehicle voice searches; and Fig. 4 illustrates an example data flow diagram for using the Fig. 3 for use in pre-loading and delayed loading of results from in-vehicle voice searches. DETAILED DESCRIPTION

[0007] While detailed embodiments of the present invention are disclosed herein as required, it is to be understood that the disclosed embodiments are merely exemplary of the invention that may be embodied in various and alternative forms. The figures are not necessarily to scale; some features may be exaggerated or reduced to show details of particular components. Accordingly, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the present invention.

[0008] Current consumer trends in the automotive industry demonstrate increased use of in-vehicle voice-based search. This is leading to a desire to integrate digital assistant technologies into the vehicle context and deliver voice search results with minimal latency. Latency in communication between in-vehicle digital assistant components and their respective cloud components outside the vehicle is a concern (e.g., for digitally assisted navigation functions, voice-based in-vehicle purchasing, etc.). Accordingly, improvements in latency can lead to a better end-user experience.

[0009] This disclosure provides a standardized approach for pre-loading and lazy loading voice search result content relevant to the user's search in the appropriate in-vehicle digital assistant (a user may use multiple digital assistants). This proactive pre-loading and lazy loading of search results relies on analytics learned from previous voice searches and searches relevant to the user. Search optimization and search results may be based on one or more of the following factors: (i) natural language support; (ii) subjectivity; (iii) synonymy; (iv) homonymy; (v) runtime frequency; (vi) relevance models; (vii) end-to-end system performance; (viii) result summaries; and (ix) dynamically merging cloud-based results with local results at runtime.

[0010] A user (such as a vehicle driver or other vehicle occupant) makes a voice search query. An in-vehicle search component receives the voice-based search query and performs the search with one or more digital assistants to obtain search results. An initial search is performed locally, while the digital assistant can interact with the speaker. In parallel with the initial local search, the digital assistant(s) perform(s) a parallel search with appropriate cloud components and can load historical search results and search results based on relevance. These received search results are then sent through an optimization algorithm to narrow down / fine-tune the data into a single result.This in-vehicle search component may use an optimization algorithm by connecting to an optimization cloud component to narrow the received results. In one example, criteria for narrowing the search may include one or more of the following: (i) natural language support; (ii) subjectivity; (iii) synonymy; (iv) homonymy; (v) runtime frequency; (vi) relevance models; (vii) end-to-end system performance; (viii) result summaries; and (ix) dynamic merging of cloud-based results with local results at runtime; (x) lazy loading and dynamic merging at runtime based on initial customer responses; and (xi) an end-to-end framework that thereby reduces the latency of voice search results being presented. Additional aspects of the disclosure are discussed in detail herein.

[0011] Fig. 1 shows an example diagram of a system 100 configured to provide pre-load and deferred load results from in-vehicle voice searches using one or more digital assistant services. A vehicle 102 may include various types of passenger vehicles, such as soft-road vehicles (CUVs), sport utility vehicles (SUVs), trucks, recreational vehicles (RVs), boats, aircraft, or other mobile machines for transporting people or goods. Telematics services may include, but are not limited to, navigation, route guidance, vehicle diagnostics, local business searches, accident notifications, and hands-free calling. In one example, the system 100 may include the SYNC system manufactured by Ford Motor Company of Dearborn, Michigan. It should be noted that the illustrated system 100 is merely an example, and more, fewer, and / or differently arranged elements may be used.

[0012] A head unit controller 104 may include one or more memories 108 and one or more processors 106 configured to execute instructions, commands, and other routines to support the processes described herein. For example, the head unit controller 104 may be configured to execute instructions from vehicle applications 110 to provide functions such as navigation, crash notification, satellite radio decryption, and hands-free calling. Such instructions and other data may be maintained non-transitoriously using a variety of types of electronically readable storage media 112. The computer-readable medium 112 (also referred to as a processor-readable medium or memory) includes any non-transitory medium (e.g.,a tangible medium) involved in providing instructions or other data readable by the processor 106 of the main unit controller 104. Computer-executable instructions may be compiled or interpreted by computer programs produced using a variety of programming languages ​​and / or technologies, including, among others, and either alone or in combination, JAVA, C, C++, C#, OBJECTIVE C, FORTRAN, PASCAL, JAVA SCRIPT, PYTHON, PERL, and PL / SQL.

[0013] The head unit controller 104 may be equipped with various features that allow vehicle occupants to connect to the head unit controller 104. For example, the head unit controller 104 may include an audio input 114 configured to receive voice commands from vehicle occupants via a connected microphone 116, and an auxiliary audio input 118 configured to receive audio signals from connected devices. The auxiliary audio input 118 may be a physical connection, such as a power cable or fiber optic cable, or a wireless input, such as a BLUETOOTH audio connection or Wi-Fi connection. In some examples, the audio input 114 may be configured to provide audio processing features, such as pre-amplifying low-level signals and converting analog inputs to digital data for processing by the processor 106.

[0014] The head unit controller 104 may also provide one or more audio outputs 120 to an input of an audio subsystem 122 with audio playback functionality. In other examples, the head unit controller 104 may provide platform audio via the audio output 120 to an occupant through the use of one or more dedicated speakers (not illustrated). The audio output 120 may include, for example, system-generated sounds, pre-recorded sounds, navigation prompts, other system prompts, or warning signals.

[0015] The audio module 122 may include an audio processor 124 configured to perform various operations on audio content received from a selected audio source 126 and to stage audio received from the audio output 120 of the head unit controller 104. The audio processors 124 may be one or more computing devices capable of processing audio and / or video signals, such as a computer processor, a microprocessor, a digital signal processor, or any other device, set of devices, or other mechanisms capable of performing logical operations. The audio processor 124 may operate in conjunction with memory to execute instructions stored in the memory.The instructions may be in the form of software, firmware, computer code, or a combination thereof, and, when executed by the audio processors 124, may provide audio detection and audio generation functionality. The instructions may also provide audio cleanup (e.g., noise reduction, filtering, etc.) prior to processing the received audio signal. The memory may be any form of one or more data storage devices, such as volatile memory, non-volatile memory, electronic memory, magnetic memory, optical memory, or any other form of data storage device.

[0016] The audio subsystem may also include an audio amplifier 128 configured to receive a processed signal from the audio processor 124. The audio amplifier 128 may be any circuit or stand-alone device that receives audio input signals of relatively small magnitude and outputs similar audio signals of relatively larger magnitude. The audio amplifier 128 may be configured to provide playback through vehicle speakers 130 or headphones (not shown).

[0017] The audio sources 126 may include, for example, decoded amplitude-modulated (AM) or frequency-modulated (FM) radio signals and audio signals from audio playbacks of compact disks (CDs) or digital versatile disks (DVDs). The audio sources 126 may also include audio data received by the main unit controller, such as audio content generated by the main unit controller 104, from flash memory drives 104. decrypted audio content connected to a USB (Universal Serial Bus) subsystem 132 of the head unit controller 104; and audio content passed through the head unit controller 104 from the auxiliary audio input 118. For example, the audio sources 126 may also include Wi-Fi streaming audio, USB streaming audio, BLUETOOTH streaming audio, Internet streaming audio, TV audio, as some other examples.

[0018] The main unit controller 104 may utilize a voice interface 134 to provide a hands-free interface to the main unit controller 104. The voice interface 134 may support speech recognition of audio data received via the microphone 116 according to a standard grammar describing available command functions and generating voice prompts for output via the audio module 122. The voice interface 134 may utilize probabilistic speech recognition techniques using the standard grammar compared to the input speech. In many cases, the voice interface 134 may include a default user profile setting for use by the speech recognition functions to enable the speech recognition to be tuned to provide good results on average, resulting in a positive experience for the maximum number of initial users.In some cases, the system may be configured to temporarily mute or otherwise override the audio source specified by an input selector when an audio prompt may be issued by the head unit controller 104 and another audio source 126 is selected for playback.

[0019] The microphone 116 may also be used by the head unit controller 104 to detect the presence of conversations between vehicle occupants within the vehicle interior. In one example, the head unit controller 104 may perform voice activity detection by filtering audio samples received from the microphone 116 to a frequency range where early speech formants are typically found (e.g., between 240 and 2400 Hz), and then applying the results to a classification algorithm configured to classify the samples as either speech or non-speech. The classification algorithm may use various types of artificial intelligence algorithms, such as pattern matching classifiers and K-nearest neighbor classifiers, as some examples.

[0020] The head unit controller 104 may also receive inputs from controls 136 of a human-machine interface (HMI) configured to provide interaction between occupants and vehicle 102. For example, the head unit controller 104 may interface with one or more buttons or other HMI controls configured to invoke functions on the head unit controller 104 (e.g., audio buttons on the steering wheel, a talk button, controls on the instrument panel, etc.). The head unit controller 104 may also drive or otherwise communicate with one or more displays 138 configured to provide visual output to vehicle occupants via a video controller 140.In some cases, the display 138 may be a touchscreen that is also configured to receive touch-based input from the user via the video controller 140, whereas in other cases, the display 138 may simply be a display without the capability for touch-based input.

[0021] The head unit controller 104 may also be configured to communicate with other components of the vehicle 102 via one or more in-vehicle networks 142. The in-vehicle networks 142 may include, for example, one or more of the following: a vehicle controller area network (CAN), an Ethernet network, or a media oriented system transfer (MOST). Through the in-vehicle networks 142, the head unit controller 104 may communicate with other systems in the vehicle 102, such as a telematics controller 144 with an embedded modem 145 (may not be present in some configurations), a global positioning system (GPS) module 146 configured to provide the current location and heading of the vehicle 102, and various vehicle electronic control units (ECUs) 148 configured to cooperate with the head unit controller 104.As some non-limiting possibilities, the vehicle ECU 148 may include a powertrain control module configured to control the engine's operating components (e.g., idle control, fuel delivery components, emission control components, etc.) and monitor the engine's operating components (e.g., status of engine diagnostic codes); a body control module configured to manage various power control functions, such as exterior lighting, interior lighting, keyless entry, remote start, and check the status of access points (e.g., closure status of the hood, doors, and / or trunk of the vehicle 102); a two-way radio device configured to communicate with key fobs or other local devices of the vehicle 102; and a climate control management module configured to control and monitor heating and cooling system components (e.g.,Compressor clutch and blower fan control, temperature sensor information, etc.).

[0022] As illustrated, the audio module 122 and HMI controls 136 may communicate with the head unit controller 104 via a first in-vehicle network 142-A, and the telematics controller 144, GPS module 146, and vehicle ECU 148 may communicate with the head unit controller 104 via a second in-vehicle network 142-B. In other examples, the head unit controller 104 may be connected to more or fewer in-vehicle networks 142. Additionally or alternatively, one or more HMI controls 136 or other components may be connected to the head unit controller 104 via other in-vehicle networks 142 that differ from the networks depicted, or directly without connection to any in-vehicle network 142.

[0023] The head unit controller 104 may also be configured to communicate with mobile devices 152 of the vehicle occupants. The mobile devices 152 may be any of various types of portable computing devices, such as cellular phones, tablet computers, smart watches, laptop computers, portable music players, or other devices capable of communicating with the head unit controller 104. In many examples, the head unit controller 104 may include a wireless transceiver 150 (e.g., a BLUETOOTH module, a ZIGBEE transceiver, a WLAN transceiver, an IrDA transceiver, an RFID transceiver, etc.) configured to communicate with a compatible wireless transceiver 154 of the mobile device 152.Additionally or alternatively, the main unit controller 104 may communicate with the mobile device 152 via a wired connection, such as a USB connection between the mobile device 152 and the USB subsystem 132. In some examples, the mobile device 152 may be battery-powered, while in other cases, the mobile device 152 draws at least a portion of its power from the vehicle 102 via the wired connection.

[0024] A communications network 156 may provide communications services, such as packet-switched network services (e.g., Internet access, VoIP communications services), to devices connected to the communications network 156. An example of a communications network 156 may include a cellular network. Mobile devices 152 may provide network connectivity to the communications network 156 via a device modem 158 of the mobile device 152. To enable communications over the communications network 156, mobile devices 152 may be associated with unique device identifiers (e.g., mobile device numbers (MDNs), Internet Protocol (IP) addresses, etc.) to identify communications of the mobile devices 152 over the communications network 156.In some cases, occupants of the vehicle 102 or devices authorized to connect to the head unit controller 104 may be identified by the head unit controller 104 according to the paired device data 160 maintained in the storage medium 112. For example, the paired device data 160 may indicate the unique device identifiers of the mobile devices 152 previously paired with the head unit controller 104 of the vehicle 102, so that the head unit controller 104 can automatically reconnect to the mobile devices 152 referenced in the paired device data 160 without user intervention.

[0025] When a mobile device 152 supporting network connectivity is paired and connected to the head unit controller 104, the mobile device 152 may enable the head unit controller 104 to utilize the network connectivity of the device modem 158 to communicate with the various remote computing devices over the communications network 156. In one example, the head unit controller 104 may utilize a data-over-voice plan or a data plan of the mobile device 152 to communicate information between the head unit controller 104 and the communications network 156. Additionally or alternatively, the head unit controller 104 may utilize the telematics controller 144 to communicate information between the head unit controller 104 and the communications network 156 without using any communications facilities of the mobile device 152.

[0026] Similar to the main unit controller 104, the mobile device 152 may include one or more processors 185 configured to execute instructions from mobile applications 174 loaded from a storage medium 188 of the mobile device 152 into a memory 186 of the mobile device 152. In some examples, the mobile applications 174 may be configured to communicate with the main unit controller 104 via the wireless transceiver 154 and with other network services via the device modem 158.

[0027] Each of the digital assistant service 162, the identity service 164, and the search optimization service 166 may include various types of computing devices, such as a computer workstation, a server, a desktop computer, a virtual server instance running on a mainframe server, or another computer system and / or computing device, such as cloud servers, virtual servers, and / or services or software. Similar to the main unit controller 104, the one or more devices of the digital assistant service 162, the identity service 164, and the search optimization service 166 may each include memory in which computer-executable instructions may be maintained, where the instructions may be executable by one or more processors of the devices.It should also be noted that while the identity service 164 is illustrated as being cloud-based, in other examples, the identity service 164 may be located in the vehicle 102. In still other examples, identity services 164 may be located in the vehicle 102 and in the cloud, and the vehicle 102 may have the capability to use one or both to perform identity determinations.

[0028] A digital assistant can refer to a software agent designed to perform tasks or services for a person. In some examples, digital assistants can receive voice input from a user, convert the voice input into requests for actions, perform the actions, and return results of the actions to the user. Possible actions that can be performed by the digital assistants based on a voice request can include providing search results, ordering goods or services, selecting media for playback, adjusting home automation settings, and so on.

[0029] The digital assistant services 162 include various cloud-based virtual assistants, which are software agents designed to perform tasks or services for an individual. As some examples, the digital assistant services 162 may include the digital assistant Siri provided by Apple, Inc., Cupertino, California, the digital assistant Cortana provided by Microsoft Corp., Redmond, Washington, and the digital assistant Alexa provided by Amazon.com Inc., Seattle, Washington.

[0030] The digital assistant agents 168 may comprise client-side, in-vehicle software installed on the memory 112 of the head unit controller 104 and configured to provide digital assistant functionality to occupants of the vehicle 102. In some examples, the digital assistant agents 168 may be standalone, while in other cases, the digital assistant agents 168 may be networked components installed in the head unit controller 104 configured to communicate with cloud-based digital assistant services 162 via the communications network 156.

[0031] The search manager 170 may include client-side, in-vehicle software installed on the memory 112 of the head unit controller 104 and configured to provide a search query management function to occupants of the vehicle 102. For example, the search manager 170 may receive queries from the vehicle occupants, e.g., via the microphone 116 and, in some cases, using the services of the voice interface 134 to convert spoken words into text. The search manager 170 may also perform a local search function in response to receiving the search to provide initial search results, utilize the digital assistant 168 to utilize the digital assistant functionality to provide additional results for the query, process and optimize the search results, including lazy loading of the additional results from the digital assistant 168, and present the results to the occupant.The search manager 170 may also be configured to utilize the lazy loading capability to begin rendering results while the parallel results have not arrived, are in the process of arriving, or are not yet sorted.

[0032] The search optimization service 166 may include functionality to sort, select, modify, or otherwise enhance raw search results to improve the relevance of the results to a user. In one example, the search optimization service 166 may be trained according to previous searches and / or other search queries relevant to the user. The search optimization service 166 may perform search optimization on search results based on one or more of the following factors: (a) natural language support; (b) subjectivity; (c) synonymy; (d) homonymy; (e) runtime frequency; (f) relevance models; (g) end-to-end system performance; (h) result summaries; and (i) dynamically merging cloud-based results with local results at runtime. In some examples, the search manager 170 may utilize the services of the search optimization service 166 to optimize the search results for presentation to occupants.

[0033] The main unit controller 104 may include a device linking interface 172 to enable the integration of functionality of mobile applications 174 configured to communicate with a device linking application kernel 176 executed by the mobile device 152. In some examples, the mobile applications 174 that support communication with the device linking interface 172 may statically link or otherwise integrate the functionality of the device linking application kernel 176 with the binary representation of the mobile applications 174.In other examples, the mobile applications 174 that support communication with the device linking interface 172 may access an application programming interface (API) of a common or separate device linking application core 176 to enable communication with the device linking interface 172.

[0034] The integration of functionality provided by the device linking interface 172 may include, for example, the ability of mobile applications 174 executed by the mobile device 152 to integrate additional voice commands into the grammar of commands available via the voice interface 134. For example, using the device linking interface 172, additional digital assistants available to the mobile device 152 may be made available for use by the main unit controller 104.

[0035] The device link interface 172 may also provide the mobile applications 174 with access to vehicle information available to the head unit controller 104 via the in-vehicle networks 142. The device link interface 172 may also provide the mobile applications 174 with access to the vehicle display 138. An example of a device link interface 172 may be the SYNC APPLINK component of the SYNC system provided by Ford Motor Company, Dearborn, Michigan. Other examples of device link interfaces 172 may include MIRRORLINK, APPLE CARPLAY, and ANDROID AUTO.

[0036] The mobile device 152 may be configured to provide identity information 178 from the mobile device 152 to the main unit controller 104. In one example, the identity information 178 may include keys or other information specific to a user stored in the memory 188 of the mobile device 152. In another example, the identity information 178 may include a passcode, biometrics, or other information received from a user of the mobile device 152.

[0037] The identity processor 180 may include software installed in the vehicle configured to manage the determination of identities of vehicle occupants who submit search requests to the search manager 170. The identity processor 180 may use data, such as the identity information 178, to determine an identity of a vehicle occupant who initiates a search request.

[0038] The identity service 164 may include various networked services for managing electronic or digital identities. For example, the identity service 164 may capture and record user credentials, maintain a database of user identities, and manage the assignment and removal of user access privileges to managed resources. Accordingly, the identity service 164 may be used to perform operations such as authenticating and validating a user's identity and confirming that a user has access to a particular service. The identity processor 180 may communicate with the identity service 164 to confirm the identity of a vehicle occupant making a query and to further support the personalization of search result optimizations performed by the search manager 170 in response to processing a search query.

[0039] Fig. Figure 2 illustrates an example dataflow diagram 200 for processing pre-loading and delayed loading results from in-vehicle voice searches using one or more digital assistant services. In one example, dataflow diagram 200 may be performed using system 100, discussed in detail above.

[0040] As shown, at 202, a voice search request is received by vehicle 102. In one example, a vehicle occupant may request that certain information be displayed, such as the best local restaurant for sushi or the availability of parking at a local parking garage. At 204, the received speech is translated into a form suitable for the search. In some cases, this translation may include voice interface 134 using probabilistic speech recognition techniques compared to the input speech to determine a textual representation of the input.

[0041] At 206, the vehicle 102 performs the search 206. Notably, this search may involve the use of multiple digital assistants, both local to the vehicle 102 and remote from the vehicle 102. Generally, an initial search is performed locally while the connected digital assistant interacts with the speaking occupant. In parallel with an initial local search, the connected digital assistants perform a parallel search with appropriate cloud components and may load historical and relevance search results. In the illustrated example, the search includes sending the input query to four digital assistant agents 168, two of which are connected agents communicating with remote digital assistant services 162 and two of which are local digital assistant agents 168 operating using the computing resources of the vehicle 102.More specifically, digital assistant agent 168A communicates with digital assistant service 162A via communication network 156, and digital assistant agent 168B communicates with digital assistant service 162B via communication network 156. Digital assistant agents 168C and 168D process the input request locally. Note that this is only an example, and more or fewer digital assistant agents 168 may be used in other cases. For example, there may be more or fewer digital assistant agents 168 located locally in the vehicle, and there may be more or fewer digital assistant agents 168 communicating with remote services.

[0042] In particular, due to the increased processing, storage, and accessible data resources available to digital assistant services 162, the networked digital assistants may provide better or more detailed results than the resources available locally within the vehicle 102. However, the latency between requesting the results and receiving the results may be greater for the networked digital assistants compared to the digital assistants executing locally for the vehicle 102. As shown in diagram 200, the dashed lines in the remainder of the diagram indicate the path for the results received from the local digital assistants, while the solid lines in the remainder of the diagram indicate the path for the results received from the networked digital assistants. First, the results are received from the local digital assistants 168C and 168D in response to the search at 206.These initial results are processed at 208. For example, at 208, the initial results from multiple local digital assistants 168 may be deduplicated or otherwise aggregated into a unified list of results. At 210, these results may be optimized. For example, the order and visibility of the results may be improved based on historical insights, relevance determinations, or other factors that can be used to adjust the order or whether items are even included in the results. Search optimization may be based on one or more of the following factors: (i) natural language support; (ii) subjectivity; (iii) synonymy; (iv) homonymy; (v) runtime frequency; (vi) relevance models; (vii) end-to-end system performance; and / or (viii) result summaries.These results are then translated at 212 into a format suitable for provision to occupants of vehicle 102. For example, a best result may be translated into text for display or input into a text-to-speech algorithm, or a set of results may be formatted into a list format for selection by a vehicle operator. At 214, the translated results are sent to the HMI of vehicle 102. For example, the results may be provided to display 138. In another example, the results may be provided to audio subsystem 122 after being processed from text to speed via speech interface 134. These initial results are presented to the occupant at 216.

[0043] Returning to the flow of connected digital assistants, connected results may be received at 206. These results may include historical search results indicating results provided for the same or similar queries (in some cases, these results were provided for the same occupant to determine the identity services used, discussed in more detail below), as well as search results sorted by relevance. At 218, the historical results are processed, and at 220, the relevance results are processed. This processing at 218 and 220 may include, for example, deduplication or some other combination of results into a single unified listing.

[0044] Again, the search results are optimized at 210, but this time including results from the connected digital assistants. Furthermore, the optimization at 210 (e.g., re-optimization) may further include performing cloud-based search optimization at 222. This re-optimization may include results currently presented to the user for a seamless user experience. For example, if the method is in the middle of displaying choice two of the local agent list, the re-optimization may improve the remainder of choice two that was not presented (e.g., to obtain better wording) and may completely replace choices three and beyond, without the user noticing that subsequent choices have spontaneously changed.Because additional results are received at 206, this step may occur multiple times within a single list being presented. In one example, criteria for narrowing the search may include one or more of the following: (i) natural language support; (ii) subjectivity; (iii) synonymy; (iv) homonymy; (v) runtime frequency; (vi) relevance models; (vii) end-to-end system performance; (viii) result summaries; and (ix) dynamic merging of cloud-based results with local results at runtime; (x) lazy loading and dynamic merging at runtime based on initial customer responses; and (xi) an end-to-end framework that thereby reduces the latency of voice search results being presented. In one example, the optimization algorithm may narrow or otherwise fine-tune the data to a single result or a set of results.These optimized results are retranslated at 212 and then sent as optimized search results at 224 to the vehicle HMI for presentation as delayed search results at 226. Accordingly, the latency associated with using the remote digital assistant services 162 is masked by the initial results provided more quickly by the local search, thereby improving the latency of voice search results being presented.

[0045] Fig. 3 illustrates an exemplary dataflow diagram 300 for identity processing for use in pre-loading and delayed loading results from in-vehicle voice searches. Accordingly, diagram 300 shows additional details of the processing described above related to the identity aspects of providing search results. In one example, the in-vehicle identity processing described with reference to diagram 300 may be performed by computing elements of head unit controller 104, while cloud identity processing may be performed by computing elements of identity service 164 or other cloud-based computing components.

[0046] As discussed above, a voice search query 202 is received. In addition to processing the words of the voice search query 202, the voice search query 202 may also be used to identify the speaker. At 302, the vehicle 102 may perform voice identity processing to identify the speaker. For example, the user may be identified using pre-stored vocal fingerprint data. Other channels may also be used to determine identity. For example, the user may be identified using a camera, via visual biometrics, and / or via a fingerprint scanner or other biometric sensor.At 304, the vehicle 102 may perform device identity processing, for example, via recognition of a key fob assigned to a user or via identity information 178 received from a user's mobile device 152. At 306, the vehicle 102 may perform profile-based identity processing. e.g., based on the user's biometrics, seat sensor information, entertainment settings in use, a user's driving profile, a navigation profile including destinations frequented by the user or specific to the user, and so on. At 308, the vehicle 102 may perform HMI-based identity processing, including, for example, receiving a passcode or other identifying information via the HMI of the vehicle 102.

[0047] To accurately identify a person, multi-factor identification may be used with one or more of these channels. Each of these channels may send the results to pre-identity processing at 310. Pre-identity processing may use an algorithm to identify a person using one or more identity channels depending on the feature requirements. For example, one feature may use only one identity channel to identify a user, while other features may utilize two or more channels to more accurately identify a user. In some cases, pre-identity processing may interact with identity service 164, shown at 312, to identify the vehicle occupant.For example, the identity service 164 may use an identity data store 314 to match the identity characteristics identified by the channels with one or more user identities stored in the data store 314. Depending on the channels used, it is possible that the pre-identity processing may return more than one possible user.

[0048] If pre-identity processing 310 results in more than one possible identity, pre-identity processing 310 may forward the identified user information to identity processing 316. Identity processing 316 may use additional algorithms to more accurately narrow the list to a unique identity. If identity processing 316 is unable to narrow down to a single identity, identity processing 316 may attempt to identify the user using a pseudonym (e.g., a role such as vehicle operator, passenger, employee, etc.). If identity processing 316 cannot identify even a pseudonym, the user may be identified as anonymous, and information regarding the error may be logged in the identity store.As with pre-identity processing, identity processing may also interact with the cloud-based identity decision processing 312 of the identity service 164 to identify users.

[0049] Fig. 4 illustrates an exemplary data flow diagram 400 for using the Fig.3 for use in pre-loading and delayed loading results from in-vehicle voice searches. Diagram 400 accordingly shows additional details of the processing described above regarding the use of identity aspects to provide search results. In one example, the in-vehicle digital assistant processing described with reference to diagram 400 may be performed by computing elements of head unit controller 104, while the cloud digital assistant processing may be performed by computing elements of digital assistant services 162 or other cloud-based computing components.

[0050] At 402, the voice search query 202 is received. Specifically, as indicated in diagrams 300 and 400 under (A), this may be the same voice search query 202 that was processed for identity purposes in diagram 300. At 404, the user is identified. Furthermore, as indicated in diagrams 300 and 400 under (B), this identity may be the identity determined according to the further discussion with reference to diagram 300.

[0051] At 406, the vehicle 102 performs a preload of historical results by relevance. In one example, and as discussed above, this may include using one or more digital assistant agents 168 to communicate over the communication network 156 with one or more respective digital assistant services 162 at 408 to retrieve the historical results. Similarly, at 410, the vehicle 102 performs a request for new search results by relevance. In one example, and as discussed above, this may include using one or more digital assistant agents 168 to communicate over the communication network 156 with one or more respective digital assistant services 162 at 412 to retrieve the relevance search results.

[0052] Furthermore, these historical queries and new search queries may be informed by the identity determination references at 404. For example, the vehicle 102 may determine at 414 whether a specific user identity has been identified, with reference to the identity received at 404. If so, control transfers to 416 to provide this identity information to the preloading of historical results by relevance 406 and also to the request for new search results by relevance 410. In this way, these searches may provide historical data and new results in the user's context. For example, the historical data could include results previously presented to the user or previously selected by the user.Alternatively, the new results could be ordered or selected based on user preferences, which may be available according to a search profile for the user tied to the user's identity. Referring again to 414, if no specific user identity was provided, control transfers to step 418 to determine whether a generic identity has been provided. The generic identity, in one example, may indicate a pseudonym, such as a role of the speaker, such as the speaker being a vehicle operator, passenger, employee, etc. If such a pseudonym is available, control transfers to step 416 to provide this information for the search operations.This allows search queries to return results tailored to a person in the same role, even if the results cannot be tailored to the specific user.

[0053] If no generic identity is available at 418, control transfers to 422 to determine whether a group identity or group location can be provided or identified. For example, a group of users of which the user is a member, or a group of users, or location-based customization may be used to display results near the user, even if no specific or generic identity of the user is known. If no group or location is identified at 422, control transfers to 424 to identify the user as anonymous. This information may also be provided to searches at 416 to allow searches to be performed anonymously without being tailored to the specific user or user role. Additionally, the user of an anonymous identity may be logged for searches at 424.For example, as indicated in diagrams 300 and 400 at (C), this indication of an anonymous identity may be recorded as a log entry in the identity data store at 314. This information may allow the system to recall the ambiguous identities, for example, to improve future user identification algorithms.

[0054] Thus, system 100 can provide a standardized approach for pre-loading and delayed loading search result content relevant to the user's search. Such techniques can be used for voice searches, as discussed above. Furthermore, such techniques may also be applicable to other search contexts, such as searching based on an HMI click, typed text, or other non-speech input approaches. In these other approaches, the results may still be returned by voice in some cases, even if the search was not triggered by voice. This proactive pre-loading and delayed loading of search results, including local search augmented by connected search results, can enable a reduction in latency in delivering search results to a user, thereby improving the user's overall search experience.

[0055] Computing devices described herein, such as main unit controller 104 and mobile device 152, generally include computer-executable instructions, where the instructions may be executed by one or more computing devices, such as those listed above. Computer-executable instructions may be compiled or interpreted by computer programs created using a variety of programming languages ​​and / or technologies, including, among others, either alone or in combination, JAVA™, C, C++, C#, VISUAL BASIC, JAVASCRIPT, PYTHON, JAVASCRIPT, PERL, PL / SQL, etc. In general, a processor (e.g., a microprocessor) receives instructions, e.g., from memory, a computer-readable medium, etc., and executes those instructions, thereby performing one or more processes, including one or more of the processes described herein.Such instructions and other data may be stored and transmitted using a variety of computer-readable media.

[0056] With respect to the acts, systems, methods, heuristics, etc. described herein, it is understood that while the steps of such acts, etc., have been described as occurring according to a particular order, such acts may be performed with the described steps in an order different from the order described herein. It is also understood that certain steps could be performed concurrently, other steps could be added, or certain steps described herein could be omitted. In other words, the descriptions of acts in this specification are for the purpose of illustrating particular embodiments and should in no way be construed to limit the claims.

[0057] Accordingly, it is to be understood that the foregoing description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided will become apparent upon reading the foregoing description. The scope should be determined not by reference to the foregoing description, but by reference to the appended claims, along with the full scope of equivalents valid for such claims. It is anticipated and intended that there will be future developments in the technologies discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. Overall, it is understood that the application is susceptible to modification and variation.

[0058] All terms used in the claims are intended to be given their broadest reasonable constructions and their common meanings as known to those skilled in the art familiar with the technologies described herein, unless expressly indicated otherwise. In particular, the use of the singular articles, such as "a," "an," "the," "the," "the," etc., is to be interpreted to refer to one or more of the listed elements, unless a claim contains an express limitation to the contrary.

[0059] The Summary of Disclosure is provided to give the reader a quick overview of the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Additionally, it will be apparent from the foregoing Detailed Description that various features are grouped together in various embodiments for the purpose of simplifying the disclosure. This method of disclosure should not be interpreted to reflect an intent that the claimed embodiments require more features than expressly recited in each claim. Rather, the subject matter of the invention lies in fewer than all features of a single disclosed embodiment, as reflected in the following claims.Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as separately claimed subject matter.

[0060] While exemplary embodiments are described above, these embodiments are not intended to describe all possible forms of the invention. Rather, the terms used in the description are words of description rather than limitation, and it is understood that various changes may be made without departing from the spirit and scope of the invention. Furthermore, the features of various implementing embodiments may be combined to form further embodiments of the invention.

[0061] According to the present invention, a system is provided comprising: an audio input, a transceiver; and a processor programmed to receive a voice query from a vehicle occupant via the audio input, evaluate the query locally to provide initial local search results, and use the transceiver in parallel to evaluate the query using one or more remote digital assistants simultaneously, to receive historical and new search results, and to merge the local results and the remote results to provide additional search results for lazy loading.

[0062] According to one embodiment, the processor is further programmed to: determine an identity of the vehicle occupant using one or more channels of user identity information; and provide the identity of the vehicle occupant to the one or more remote digital assistants to obtain historical and recent search results tailored to the identity of the vehicle occupant.

[0063] According to one embodiment, the one or more channels of user identity information comprise one or more of the following channels: a voice identity channel configured to determine user identity via voice pressure, a device identity channel configured to determine user identity via identity information received from a user device, a profile-based identity channel configured to determine user identity via profile information representing the characteristics of users of the system, and a human-machine input identity channel configured to determine user identity via receiving HMI inputs from users.

[0064] According to one embodiment, the processor is further programmed to: in response to the identity of the vehicle occupant not being uniquely determined, determine a generic identity of the vehicle occupant as a pseudonym indicative of a role of the occupant; and provide the identity of the vehicle occupant to the one or more remote digital assistants to receive historical and recent search results tailored to the role of the vehicle occupant.

[0065] According to one embodiment, the processor is further programmed to: in response to the role of the vehicle occupant not being clearly determined, indicate to the one or more remote digital assistants that the vehicle occupant is anonymous to receive historical and new search results for an anonymous user without identity information.

[0066] According to one embodiment, the processor is further programmed to optimize the historical and recent search results using cloud-based search optimization. According to one embodiment, the cloud-based search optimization includes one or more of the following features: (i) natural language support; (ii) subjectivity; (iii) synonymy; (iv) homonymy; (v) runtime frequency; and (vi) relevance models.

[0067] According to the present invention, a method comprises receiving a voice query from a vehicle occupant via a vehicle-mounted microphone; evaluating the query locally to provide initial search results; using a network connection in parallel to simultaneously evaluate the query using one or more digital assistants to receive historical and new search results; and merging the initial results and the remote results to perform lazy loading of additional search results after the initial search results have been provided.

[0068] According to one embodiment, the invention is further characterized by: determining an identity of the vehicle occupant using one or more channels of user identity information; and providing the identity of the vehicle occupant to the one or more remote digital assistants to obtain historical and recent search results tailored to the identity of the vehicle occupant.

[0069] According to one embodiment, the one or more channels of user identity information comprise one or more of the following channels: a voice identity channel configured to determine user identity via voice pressure, a device identity channel configured to determine user identity via identity information received from a user device, a profile-based identity channel configured to determine user identity via profile information representing the characteristics of users of the system, and a human-machine input identity channel configured to determine user identity via receiving HMI inputs from users.

[0070] According to one embodiment, the invention is further characterized by: in response to the identity of the vehicle occupant not being uniquely determined, determining a generic identity of the vehicle occupant as a pseudonym indicative of a role of the occupant; and providing the identity of the vehicle occupant to the one or more remote digital assistants to receive historical and new search results tailored to the role of the vehicle occupant. According to one embodiment, the invention is further characterized by: in response to the role of the vehicle occupant not being uniquely determined, indicating to the one or more remote digital assistants that the vehicle occupant is anonymous to receive historical and new search results for an anonymous user without identity information.

[0071] According to one embodiment, the invention is further characterized by optimizing the historical and new search results using cloud-based search optimization.

[0072] According to one embodiment, cloud-based search optimization includes one or more of the following features: (i) natural language support; (ii) subjectivity; (iii) synonymy; (iv) homonymy; (v) runtime frequency; and (vi) relevance models.

[0073] According to the present invention, a non-transitory computer-readable medium is provided containing instructions that, when executed by one or more processors, cause the one or more processors to receive a voice query from a vehicle occupant via a microphone installed in the vehicle, evaluate the query locally to provide initial local search results, use a network connection in parallel to simultaneously evaluate the query using one or more digital assistants to receive historical and new search results, and merge the local search results and the results from the remote digital assistants to perform a lazy loading of additional search results after the initial search results have been provided.

[0074] According to one embodiment, the invention is further characterized by instructions that, when executed by the one or more processors, cause the one or more processors to: determine an identity of the vehicle occupant using one or more channels of user identity information; and provide the identity of the vehicle occupant to the one or more remote digital assistants to obtain historical and recent search results tailored to the identity of the vehicle occupant.

[0075] According to one embodiment, the one or more channels of user identity information comprise one or more of the following channels: a voice identity channel configured to determine user identity via voice pressure, a device identity channel configured to determine user identity via identity information received from a user device, a profile-based identity channel configured to determine user identity via profile information representing the characteristics of users of the system, and a human-machine input identity channel configured to determine user identity via receiving HMI inputs from users.

[0076] According to one embodiment, the invention is further characterized by instructions that, when executed by the one or more processors, cause the one or more processors to: in response to the identity of the vehicle occupant not being uniquely determined, determine a generic identity of the vehicle occupant as a pseudonym indicative of a role of the occupant; and provide the identity of the vehicle occupant to the one or more remote digital assistants to receive historical and recent search results tailored to the role of the vehicle occupant.

[0077] According to one embodiment, the invention is further characterized by instructions that, when executed by the one or more processors, cause the one or more processors to: in response to the role of the vehicle occupant not being uniquely determined, indicate to the one or more remote digital assistants that the vehicle occupant is anonymous to receive historical and new search results for an anonymous user without identity information.

[0078] According to one embodiment, the invention is further characterized by instructions that, when executed by the one or more processors, cause the one or more processors to optimize the historical and recent search results using cloud-based search optimization with one or more of the following features: (i) natural language support; (ii) subjectivity; (iii) synonymy; (iv) homonymy; (v) runtime frequency; and (vi) relevance models.

Claims

[1] In-vehicle system comprising: an audio input; a transceiver; and a processor programmed to Receiving a voice request from a vehicle occupant via the audio input, Evaluate the request locally to obtain initial local search results from a local digital assistant locally on the vehicle, Presenting the initial search results, Using the transceiver in parallel with the presentation of the initial search results to simultaneously evaluate the query using one or more remote digital assistants to receive historical and new search results, and Merging the local results and the remote results in the presentation of the initial search results to provide additional search results for lazy loading. [2] The in-vehicle system of claim 1, wherein the processor is further programmed to: Determining an identity of the vehicle occupant using one or more channels of user identity information; and Providing the identity of the vehicle occupant to the one or more remote digital assistants to obtain historical and recent search results tailored to the identity of the vehicle occupant. [3] The in-vehicle system of claim 2, wherein the one or more channels of user identity information comprise one or more of the following channels: a voice identity channel configured to determine user identity via voice pressure, a device identity channel configured to determine user identity via identity information received from a user device, a profile-based identity channel configured to determine user identity via profile information representing the characteristics of users of the system, and a human-machine input identity channel configured to determine user identity via receiving HMI inputs from users. [4] The in-vehicle system of claim 2, wherein the processor is further programmed to: in response to the vehicle occupant's identity not being clearly determined, determining a generic identity of the vehicle occupant as a pseudonym that indicates a role of the occupant; and Providing the identity of the vehicle occupant to the one or more remote digital assistants to receive historical and recent search results tailored to the role of the vehicle occupant. [5] The in-vehicle system of claim 4, wherein the processor is further programmed to: in response to the role of the vehicle occupant not being clearly determined, indicate to the one or more remote digital assistants that the vehicle occupant is anonymous to receive historical and new search results for an anonymous user without identity information. [6] The in-vehicle system of claim 1, wherein the processor is further programmed to optimize the historical and recent search results using cloud-based search optimization. [7] The in-vehicle system of claim 6, wherein the cloud-based search optimization comprises one or more of the following features: (i) natural language support; (ii) subjectivity; (iii) synonymy; (iv) homonymy; (v) runtime frequency; and (vi) relevance models. [8] Method comprising: Receiving a voice request from a vehicle occupant of a vehicle via an in-vehicle microphone; Evaluating the query locally on the vehicle to obtain initial search results; Presenting the initial search results; Using a network connection in parallel with the presentation of the initial search results to simultaneously evaluate the query using one or more remote digital assistants to receive historical and new search results; and Merging the initial results and the remote results in the presentation of the initial search results to perform lazy loading of additional search results after the initial search results have been provided. [9] The method of claim 8, further comprising: Determining an identity of the vehicle occupant using one or more channels of user identity information; and Providing the identity of the vehicle occupant to the one or more remote digital assistants to obtain historical and recent search results tailored to the identity of the vehicle occupant. [10] The method of claim 9, wherein the one or more channels of user identity information comprise one or more of the following channels: a voice identity channel configured to determine user identity via voice pressure, a device identity channel configured to determine user identity via identity information received from a user device, a profile-based identity channel configured to determine user identity via profile information representing the characteristics of users of the system, and a human-machine input identity channel configured to determine user identity via receiving HMI inputs from users. [11] The method of claim 9, further comprising: in response to the vehicle occupant's identity not being clearly determined, determining a generic identity of the vehicle occupant as a pseudonym that indicates a role of the occupant; and Providing the identity of the vehicle occupant to the one or more remote digital assistants to receive historical and recent search results tailored to the role of the vehicle occupant. [12] The method of claim 11, further comprising, in response to the role of the vehicle occupant not being clearly determined, indicating to the one or more remote digital assistants that the vehicle occupant is anonymous to receive historical and new search results for an anonymous user without identity information. [13] The method of claim 8, further comprising optimising the historical and recent search results using cloud-based search optimization that includes one or more of the following features: (i) natural language support; (ii) subjectivity; (iii) synonymy; (iv) homonymy; (v) runtime frequency; and (vi) relevance models. [14] Vehicle, in particular motor vehicle, comprising an in-vehicle system designed according to one of claims 1-7.

Images