Multilevel distributed AI assistant

Abstract

This invention provides a method for operating an artificial intelligence (AI) assistant and a computing device for doing so. [Solution] The method includes receiving user input including a user question, generating a first vector representing the user question, matching the first vector with a second vector representing a question stored in the computing device's question-answer database on the computing device, and selectively escalating the user question for processing by a machine learning model. The question-answer database includes a plurality of questions extracted from a knowledge base related to the computing device, and the corresponding answers for each of the plurality of questions.

Description

Technical Field

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 730,885, filed on December 11, 2024, under the title "Multi-Level Distributed AI Assistant", which is hereby incorporated herein by reference in its entirety.

[0002] The present disclosure relates generally to the field of voice assistant applications, and more particularly to voice assistant applications that utilize artificial intelligence (AI).

Background Art

[0003] In the implementation of voice assistants, artificial intelligence (AI) models often face a trade-off between latency and accuracy. Device-based solutions can enable low-latency responses, but the limited processing power and memory of the device may limit the response accuracy. Cloud-based solutions offer higher processing power and memory capacity but may significantly increase latency.

[0004] In many applications, a specific set of responses is required. As one example, home appliances may implement a knowledge database to assist in troubleshooting by answering user questions. The complexity of a fully cloud-based solution for an AI model such as a general-purpose large language model (LLM) may be infeasible.

[0005] There is a need for an AI assistant architecture that adjusts the trade-off between latency and processing accuracy by enabling fast inference on the device for everyday queries and selectively escalating to a more powerful model only when necessary. This approach improves responsiveness, reduces cloud dependency, and enhances user privacy.

Summary of the Invention

[0006] This abstract is provided in a concise form to introduce the selection of concepts further described below in modes for carrying out the invention. This abstract is not intended to identify any important or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0007] A method and a computing device are disclosed. One innovative aspect of the subject matter of this disclosure is implementable in a method for operating an artificial intelligence assistant. The method includes receiving user input including a user question, generating a first vector representing the user question, matching the first vector with a second vector representing a question stored in a question-answer database on a computing device, and selectively escalating the user question for processing by a machine learning model. The question-answer database includes a plurality of questions extracted from a knowledge base associated with the computing device and the corresponding answers for each of the plurality of questions.

[0008] Another innovative aspect of the subject of this disclosure can be implemented in a computing device comprising a processing system and memory. The memory, when executed by the processing system, stores instructions for the computing device to receive user input, including a user question, generate a first vector representing the user question, match the first vector with a second vector representing a question stored in a question-answer database in the computing device, and selectively escalate the user question for processing by a machine learning model. The question-answer database includes a number of questions extracted from a knowledge base related to the computing device and the answers corresponding to those questions. [Brief explanation of the drawing]

[0009] This embodiment is illustrated illustratively and is not intended to limit the scope of the accompanying drawings.

[0010] [Figure 1]Figure 1 illustrates an assistant system in which an aspect of this disclosure may be implemented.

[0011] [Figure 2] Figure 2 illustrates a block diagram of an exemplary assistant engine based on a partial implementation.

[0012] [Figure 3] Figure 3 illustrates an example of the operational flow for a first-line query to an artificial intelligence (AI) assistant, as demonstrated by some implementations.

[0013] [Figure 4] Figure 4 illustrates an example of the operation flow of a secondary query to an AI assistant in some implementations.

[0014] [Figure 5] Figure 5 illustrates an example of the operation flow of a third-level query to an AI assistant in some implementations.

[0015] [Figure 6] Figure 6 shows a block diagram of the assistant system based on a partial implementation.

[0016] [Figure 7] Figure 7 illustrates a flowchart illustrating one example of how to operate an AI assistant using a partial implementation. [Modes for carrying out the invention]

[0017] The following description includes many specific details, such as examples of specific components, circuits, and processes, in order to fully understand the disclosure. The term “coupled” as used herein means directly connected or connected via one or more intervening components or circuits. Furthermore, a specific terminology is used in the following description for illustrative purposes to fully understand the aspects of the disclosure. However, it will be apparent to those skilled in the art that these specific details are not necessarily required to implement the embodiments. In other examples, well-known circuits and devices are shown in block diagram form to avoid obscuring the disclosure. Some parts of the embodiments for carrying out the following inventions are presented in the form of procedures, logic blocks, processes, or other symbolic representations of operations on data bits in computer memory. Interconnections between circuit elements or software blocks may be shown as buses or single signal lines. Each bus may alternatively be a single signal line, and each single signal line may alternatively be a bus. Also, a single line or bus may represent one or more of countless physical or logical mechanisms for communication between components.

[0018] As will be apparent from the following description, unless otherwise specifically stated, throughout this application, descriptions using terms such as “access,” “receive,” “transmit,” “use,” “select,” “determine,” “normalize,” “multiply,” “average,” “monitor,” “compare,” “apply,” “update,” “measure,” and “derive” refer to the operations and processes by which a computer system or similar electronic computing device manipulates and transforms data represented as physical quantities (electronic quantities) in its registers or memory, and converts it into other data represented as similar physical quantities in the computer system’s memory, registers, or other information storage, transmission, and display devices.

[0019] Unless otherwise specifically stated, the technologies described herein may be implemented in hardware, software, firmware, or any combination thereof. Any configuration described as a module or component may be implemented as a single integrated logical device or as separate but interoperable logical devices. When implemented in software, the technology may be realized by a non-temporary computer-readable storage medium containing instructions that, at least in part, perform one or more of the methods described above at runtime. The non-temporary computer-readable storage medium may form part of a computer program product, including packaging.

[0020] Non-transient processor-readable storage media may consist of random access memory (RAM), including synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, and other known storage media. Furthermore, or alternatively, the technology may be at least partially implemented by processor-readable communication media that transmit or transmit code in the form of instructions or data structures, and which can be accessed, read, and / or executed by a computer or other processor.

[0021] Various exemplary logic blocks, modules, circuits, and instructions described in connection with the implementations disclosed herein may be executed by one or more processors. As used herein, the term “processor” may mean any general-purpose processor, conventional processor, controller, microcontroller, and / or state machine capable of executing scripts or instructions of one or more software programs stored in memory.

[0022] The device may implement an assistant application to assist the user with various tasks. For example, consumer electronics may implement an assistant application to assist the user, for example, in answering questions regarding the operation of the electronics or troubleshooting. The assistant application often implements an artificial intelligence (AI) model.

[0023] There may be a trade-off between latency and accuracy in the AI model. The AI model on the device may be able to provide responses with lower latency. However, due to the limitations of the computing resources (e.g., processing power, memory) on the device, the accuracy of those responses may be limited. On the other hand, the AI model implemented in a cloud-based system may be able to provide more accurate responses because of the abundant available computing resources. However, those responses may be provided with higher latency than on the device.

[0024] Describe an approach that enables escalating AI inferences to a more performant model (e.g., a model that can freely use more computing resources) as needed, considering the natural conversation flow with the user. The purpose of this approach is to provide a lightweight and fluid user interaction method that can mainly reside on the device without relying on the cloud infrastructure for primary or secondary inquiries. This allows most responses to be provided with low latency on the device and provides a path to naturally transfer inquiries to a more performant in-device model through a secondary processing method. If necessary, the system can also connect to an even more performant AI model on the network (local or cloud).

[0025] Therefore, aspects of this disclosure relate to the operation of an AI assistant. A computing device may receive user input, which may include a user question. The computing device may generate a first vector representing the user question and compare the first vector with a second vector representing a question stored in a question-answer database on the computing device. The question-answer database includes multiple questions extracted from a knowledge base related to the computing device, and the corresponding answers for each of those questions. The computing device may selectively escalate the user question for processing by a machine learning model.

[0026] Individual implementations of the subject matter described in this disclosure can be implemented to achieve one or more of the following potential benefits. By using a question-and-answer database on the device in the first stage and escalating to a more powerful model in the second or third stage as needed, the AI ​​assistant can provide low-latency responses to user input in more cases and use a more powerful model as needed (at the trade-off of higher latency). This allows the AI ​​assistant to provide low-latency responses to the majority of user inputs while escalating to more advanced, but resource-intensive and / or high-latency, inputs. This novel approach improves upon current technologies that only offer either low-latency, limited responses or high-latency communication through direct communication with cloud-based services.

[0027] Figure 1 shows an assistant system 100 in which an aspect of the present disclosure may be implemented. The assistant system 100 includes a computing device 110, one or more networks 130, and a cloud assistant system 120.

[0028] The computing device 110 is a device configured to implement the assistant engine 112. The computing device 110 may include a processing system, memory, one or more input devices, one or more output devices, and a network interface. The computing device 110 may implement the assistant engine 112 in conjunction with one or more of these components to provide AI assistant functionality in the computing device 110. In some implementations, the computing device 110 may be a desktop computer, a laptop computer, a mobile phone, a smartphone, a media device (e.g., a smart speaker), a game console, a consumer or home appliance (e.g., a washing machine, dryer, stove, oven, refrigerator, freezer, etc.), or another device having a processing system, memory, one or more input devices, one or more output devices, and a network interface.

[0029] The computing device 110 may include one or more input devices (e.g., integrated into the device or connected communicably) such as a keyboard, mouse, trackpad, touchscreen, touchpad, imaging sensor (e.g., camera), or microphone. The computing device 110 may also include one or more output devices (e.g., integrated into the device or connected communicably) such as a display device or an audio output device (e.g., speaker). In some implementations, the computing device 110 includes a microphone configured to receive audio input and an audio output device configured to output audio output.

[0030] The computing device 110 may include one or more networks 130 and network interfaces configured to connect to one or more remote systems via the networks 130, such as a cloud assistant system 120. The networks 130 may include, but are not limited to, a local area network, a wide area network, an ad hoc network, a cellular network, or the internet.

[0031] The assistant engine 112 is configured to provide AI assistant functionality on the computing device 110. This AI assistant functionality may include, for example, providing answers or performing operations on the computing device 110 in response to user questions or requests. In some implementations, the assistant engine 112 may receive user input 102, generate an output 104 in response to the user input 102 using AI, and output the output 104. In some implementations, the assistant engine 112 may comprise software components (e.g., machine learning algorithms and programs) and / or hardware components (e.g., one or more processing units, memory, storage, etc.) configured to implement the AI ​​assistant functionality, and background data used for its functionality (e.g., knowledge datasets, data for machine learning models, etc.). In some implementations, the assistant engine 112 implements one or more models (including, but not limited to, large-scale language models (LLMs) or neural network models) related to the AI ​​assistant functionality on the computing device 110. For example, the assistant engine 112 may process user input 102 based on an LLM to generate output 104.

[0032] In some implementations, the assistant engine 112 may receive user input 102 via an input device (not shown) on the computing device 110. User input 102 may be, for example, a speech uttered by the user or a text input entered by the user. User input 102 may include a question or request asked by the user. The assistant engine 112 may process the user input 102 and generate an embedding vector representing the question or request. Depending on the modality of the user input 102 (e.g., text or speech), this processing may include performing speech recognition processing on the user input 102. The assistant engine 112 may determine a response (e.g., a text answer to the question) and output that response as output 104 in the same or a different modality as the user input 102. For example, the assistant engine 112 may output the text answer as text displayed on a display device, or output the utterance converted from the text answer via a speaker.

[0033] The cloud assistant system 120 is configured to provide cloud-based AI assistant functionality to the computing device 110. Cloud-based AI assistant functionality may include, for example, responding with answers to user questions and requests received from a device (e.g., the computing device 110) located remotely from the cloud assistant system 120. In some implementations, the cloud-based assistant system 120 may receive user questions sent from a remote device (e.g., the computing device 110), generate an output responding to the question using AI, and send that output back to the remote device for output to the user. In some implementations, the cloud assistant system 120 may include software components (e.g., machine learning algorithms and programs) and / or hardware components (e.g., one or more servers, distributed or cloud computing systems) configured to implement the cloud-based assistant functionality, and background data used for that functionality (e.g., knowledge datasets, data for machine learning models, etc.). In some implementations, the cloud assistant system 120 implements one or more models (e.g., large-scale language models (LLMs) or neural network models) associated with its cloud-based AI assistant functionality. In some implementations, the LLM implemented by the cloud assistant system 120 may be a larger version of the LLM implemented by the assistant engine 112 on the computing device 110.

[0034] In some implementations, the user may input a question to the assistant engine 112 using one of several modalities. For example, the user may input the question as text or speak it as an utterance. The assistant engine 112 receives the question as user input 102. Depending on the modality of user input 102, the assistant engine 112 may preprocess user input 102 to obtain the text of the question. For example, if user input 102 is an utterance, the assistant engine 102 may perform speech recognition to convert user input 102 into text. If user input 102 is already in text format (for example, if the user typed user input 102), the assistant engine 102 may skip preprocessing. In some implementations, the assistant engine 112 may process the text corresponding to user input 102 and convert the text into an embedding vector. The embedding vector may be a high-dimensional vector that embodies the meaning of the text. Thus, the embedding vector of a user question may embody the meaning of the text of the question. By converting the question text into an embedding vector, the assistant engine 112 may be able to identify matches with the user question based on semantic similarity (measured, for example, by vector similarity), in addition to or instead of keyword matching.

[0035] In some implementations, the assistant engine 112 may attempt to determine the response to a user question through multiple stages or levels of queries, which may correspond to levels of escalation of the question. In a primary or first-level query, the assistant engine 112 may determine the response to a user question by searching the question-answer database (e.g., the QA database 232 in Figure 2) to identify the question in the database that best matches the embedding vector of the input question. In some implementations, the assistant engine 112 performs a semantic search using the embedding vector of the user question to identify the question that best matches in the database. In some implementations, the question may be stored in the question-answer database as an embedding vector, or as text that can be converted into an embedding vector. The assistant engine 112 may identify the question in the question-answer database that best fits the user question based on vector similarity (e.g., cosine similarity) between the embedding vector of the user question and the embedding vector of the question in the question-answer database. Once the most relevant question is identified, the assistant engine 112 retrieves the answer corresponding to that most relevant question from the database and may output that answer as output 104 in the same or a different modality as the user input 102. For example, if the user speaks the user input 102 as an utterance, the assistant engine 112 may also output output 104 as an utterance.

[0036] In some implementations, the assistant engine 112 may receive user input indicating whether the user is satisfied with the answer output by the assistant engine 112. For example, the assistant engine 112 may prompt the user to indicate whether they found the outputted answer useful. If the user is satisfied with the outputted answer, the assistant engine 112 terminates the primary query and returns to a standby state waiting for the next user question. If the user is not satisfied with the outputted answer, or if the assistant engine 112 is unable to output an answer (for example, if the assistant engine 112 cannot identify the most matching question in the question-answer database, or if the assistant engine 112 cannot identify a question in the question-answer database whose vector similarity to the user question exceeds a predetermined threshold), the assistant engine 112 may escalate the question to a secondary query or a second-level query.

[0037] In some implementations, the question-and-answer database may be generated based on a knowledge base related to the computing device 110 (e.g., knowledge base 224 or 234 in Figure 2). This knowledge base includes one or more documents related to the computing device 110. The question-and-answer database may include specific questions and corresponding answers generated from the content of said documents. Such documents may include user and / or support supplementary materials for the computing device 110. This includes, but is not limited to, user manuals, technical support articles, troubleshooting guides, technical specifications, quick start guides, and / or similar materials. The computing device 110 (e.g., Assistant Engine 112) or a remote system (e.g., Cloud Assistant System 120) may use machine learning techniques to analyze said one or more documents and determine one or more specific questions and corresponding answers from the content of the documents. In some implementations, the computing device 110 or a remote system may use a Large-Scale Language Model (LLM) to analyze said documents and extract a set of questions and corresponding answers from said documents for inclusion in the question-and-answer database. When executed by a remote device, the remote system may send the question-and-answer database to the assistant engine 112 for storage on the computing device 110.

[0038] For example, with respect to an assistant engine 112 implemented as a computing device 110 in a home appliance, the cloud assistant system 120 may generate a question-and-answer database from the accompanying documentation for the home appliance (e.g., user manual, quick start guide, technical specifications, etc.). The cloud assistant system 120 may analyze the accompanying documentation using LLM and extract one or more specific questions and corresponding answers to add to the question-and-answer database. The cloud assistant system 120 may send the question-and-answer database to the assistant engine 112 for storage on the computing device 110. When the accompanying documentation is updated (e.g., when a new version of the user manual is published) and / or at regular intervals, the cloud assistant system 120 may re-analyze the accompanying documentation to extract new questions and answers, update previously extracted questions and answers, or otherwise update the question-and-answer database. The cloud assistant system 120 may send the updated question-and-answer database to the assistant engine 112 for storage on the computing device 110. Thus, the question-and-answer database represents a collection of specific questions and answers derived from text-based knowledge related to the computing device 110.

[0039] In some implementations, the assistant engine 112 executes the primary query on the computing device 110 without sending it to a remote device (e.g., a cloud assistant system 120 via the network 130) for the purpose of executing the primary query. For example, the assistant engine 112 may receive user input 102 on the computing device 110, preprocess the user input 102 (e.g., convert the user input 102 into a format suitable for querying the question-answer database), search the question-answer database, and generate output 104. Thus, the assistant engine 112 can execute the primary query with less latency compared to sending the question directly to the cloud assistant system 120 to determine the response.

[0040] In some implementations, in a secondary or second-level query, the assistant engine 112 may determine a response to a user question by searching a knowledge base related to the computing device 110. In some implementations, the assistant engine 112 may perform a semantic search on the knowledge base, which may include one or more documents related to the computing device 110 (e.g., a user manual). The assistant engine may convert text chunks from the knowledge base into embedding vectors and compare the embedding vectors of the user question with those embedding vectors in the knowledge base text. In some implementations, the assistant engine 112 may search the knowledge base using LLM-based search extension generation, in which case the assistant engine 112 searches for text chunks related to the user question and uses them as context for generating an answer to the user question.

[0041] In some implementations, similar to primary queries, the assistant engine 112 may receive user input indicating whether the user is satisfied with the answer output by the assistant engine 112 for a secondary query. If the user is satisfied with the outputted answer, the assistant engine 112 may terminate the secondary query and return to a standby state waiting for the next user question. If the user is not satisfied with the outputted answer, or if the assistant engine 112 is unable to output an answer (for example, if the assistant engine 112 could not identify the most matching question based on a semantic search of the knowledge base, or if the semantic search performed by the assistant engine 112 could not identify a text chunk in the knowledge base whose vector similarity to the user question exceeds a predetermined threshold), the assistant engine 112 may escalate the question to a tertiary query or a third-level query.

[0042] In some implementations, the assistant engine 112 executes secondary queries on the computing device 110 without making outbound transmissions to remote devices (e.g., the cloud assistant system 120 via the network 130) for the purpose of executing secondary queries. For example, the assistant engine 112 may perform a semantic search using search extension generation based on the LLM to identify an answer and output the answer. Search extension generation and the semantic search (including the associated LLM processing) are performed on the computing device 110. Data related to the LLM and knowledge base are also stored and accessed on the computing device 110. Therefore, the assistant engine 112 can execute secondary queries without incurring the latency associated with sending the question to the cloud assistant system 120 to determine the response.

[0043] In some implementations, in a tertiary or third-level query, the assistant engine 112 may determine a response to a user query by sending the query to the cloud assistant system 120. In some implementations, the assistant engine 112 may connect to the cloud assistant system 120 via an application programming interface (API). The assistant engine 112 may use the API to send the query to the cloud assistant system 120 via one or more networks 130. The cloud computing system 120 may use LLM or other appropriate machine learning models or techniques to identify the answer to the query. For example, the cloud computing system 120 may use LLM to analyze a knowledge base and optionally other resources related to the computing device 110 to identify the answer. Once the cloud assistant system 120 has identified an answer, it may use the API to send the answer back to the assistant engine 112 via the network 130. The assistant engine 112 may output the answer as output 104.

[0044] In some implementations, a cloud computing system 120, which has more computing resources at its disposal than the assistant engine 112 on the computing device 110, may be able to identify answers with higher accuracy compared to the assistant engine 112. For example, an LLM implemented by the cloud computing system 120 may be more highly trained and have more computing resources (e.g., processing power, memory, storage) available to it compared to an LLM implemented by the assistant engine 112. However, when a question is sent to the cloud assistant system 120 and an answer is received from the cloud assistant system 120, latency occurs that may not occur in primary and secondary queries related to communication between the computing device 110 and the cloud assistant system 120. Therefore, a user question may be escalated to a tertiary query if the user does not obtain an acceptable answer in the primary and secondary queries, and not otherwise.

[0045] Figure 2 is a block diagram showing an example of the assistant engine 112 with some implementations. Figure 2 shows the assistant engine 112 of Figure 1 in more detail. As shown in the figure, the assistant engine 112 includes a user interface module 240 and an assistant module 210.

[0046] The user interface module 240 is configured to detect user input to the assistant engine 112 and to perform preprocessing on such user input (e.g., user input 102) to convert it into text suitable for the assistant module 210 (e.g., text of a user question). The user interface module 240 is also configured to output a response to a user question received from the assistant module 210. In some implementations, the user interface module 240 may include a voice activity detection module 242, a speech recognition module 244, and a speech synthesis module 246.

[0047] In some implementations, user input 102 may include utterances spoken by the user. These utterances may be captured by the microphone of the computing device 110. The voice activity detection module 242 is configured to detect utterances (for example, user utterances included in ambient noise) from the sounds captured by the microphone.

[0048] The speech recognition module 244 is configured to convert the utterance detected in the user input 102 into text. The speech recognition module 244 may use machine learning or artificial intelligence-based techniques to convert the utterance detected in the user input 102 into question text 206. In some implementations, the speech recognition module 244 runs locally on the computing device 110 using a model on the device, without communication with remote devices or systems. An example of a speech recognition model that can run locally without communication with remote devices is "Moonshine". The user interface module 240 may send the question text 206 to the assistant module 210.

[0049] In some implementations, the user interface module 240 may detect hotwords in user input 102 or question text 206. The assistant engine 112 may require a predefined hotword or wake-up word (or a similar phrase) to precede an utterance addressed to the assistant engine 112 to indicate that the question is indeed addressed to the assistant engine 112. Therefore, the user interface module 240 may detect hotwords (e.g., "Hey Assistant," "Hey Siri," "OK Google," etc.) to distinguish utterances directed to the assistant engine 112 (e.g., questions) from other utterances. If the user interface module 240 detects a hotword in user input 102 or the corresponding question text 206, it may send the question text 206 to the assistant module 210. If the user interface module 240 does not detect a hotword, it may ignore user input 102 and wait for the next user input. In some other implementations, the assistant module 210 may perform hotword detection on the question text 206 instead of the user interface module 240.

[0050] In some implementations, the user interface module 240 may have the capability to accept text-based user input 102 instead of speech or voice. For example, the user interface module 240 may have a graphical user interface (GUI) that can be displayed on the display device of the computing device 110. The user may input a question as text via the GUI using a touch-detecting surface (e.g., a touchscreen, touchpad, etc.), one or more physical buttons, one or more physical dials, or other suitable input devices of the computing device 110. If the user input 102 is entered as text, the user interface module 240 may bypass the voice activity detection module 242 and the speech recognition module and send the user input 102 as question text 206 to the assistant module 210.

[0051] The speech synthesis module 246 is configured to convert the response 208 received from the assistant module 210 into speech. The assistant module 210 may send the text of the response 208 in response to the question text 206 to the user interface module 240. The speech synthesis module 246 may convert the text of the response 208 into speech and output the converted response as output 104 via the audio output device (e.g., speaker) of the computing device 110. In addition to or instead of outputting the response 208 as speech, the user interface module 240 may output the text of the response 208 as text. In some implementations, the speech synthesis module 246 runs locally on the computing device 110 using a model on the device without communication with remote devices or systems. An example of a speech recognition model that can run locally without communication with remote devices is "Piper".

[0052] The assistant module 210 comprises a sentence conversion module 212, a QA search module 214, a local LLM module 216, and a cloud module 218. The sentence conversion module 212 is configured to convert the question text 206 into an embedding vector 207 that represents its meaning. The sentence conversion module 212 sends the embedding vector 207 to the QA search module 214.

[0053] In the initial query for a user question, the QA search module 214 searches the QA database 232 stored in the computing device 110 to find a question that matches the embedding vector 207 (for example, the question with the highest similarity based on cosine similarity). In some implementations, questions in the QA database 232 are stored as embedding vectors. Therefore, the QA search module 214 may search the QA database 232 by comparing the embedding vector of the question with the embedding vector 207. In implementations where questions in the QA database 232 are stored as text, the question text from the QA database may be converted to an embedding vector and compared with the embedding vector 207.

[0054] In some implementations, the QA database 232 is generated by the cloud assistant system 120. The cloud assistant system 120 may use machine learning models and technologies (e.g., LLM) available in the cloud assistant system 120 to analyze documents related to the computing device 110 stored in the knowledge base 224 (e.g., manuals, support articles, user guides, technical specifications, etc.) and extract specific questions and corresponding answers related to the computing device 110. For example, regarding a washing machine, a specific question might be "What are the default washing settings for the 'cold water wash' mode preset?" and the corresponding answer might be "Normal soil level, cold water, medium spin, 1 hour." The QA database 232 may contain one or more specific questions related to the computing device 110 (which may be stored in the database 232 as text and / or embedded vectors) and their corresponding answers (which may also be stored in the database 232 as text and / or embedded vectors).

[0055] When a matching question is found in the QA database 232, the QA module 214 may retrieve the answer 208 corresponding to the matching question from the QA database 232. The answer 208 may be stored as text in the QA database 232. The QA module 214 sends the answer 208 to the user interface module 240. The user interface module 240 may output output 104 containing the text of the answer 208, or it may output an utterance converted from the text of the answer 208 by the speech synthesis module 246. In some implementations, the user interface module 240 may prompt the user to indicate whether the answer 208 is satisfactory or at least acceptable to the user. If the user indicates that the answer 208 is acceptable, the assistant module 210 may terminate the primary query and enter a waiting state in preparation for the next question.

[0056] If the user indicates that they cannot accept answer 208, or if the QA search module 214 cannot find a matching question in the QA database 232, the assistant module 210 may escalate the user question to a secondary query, which may be handled by the local LLM module 216.

[0057] The local LLM module 216 is configured to determine the answer to a user question based on the knowledge base 234. The local LLM module 216 searches the knowledge base 234 for a match with the embedding vector 207. If a match is found, the local LLM module 216 may retrieve the corresponding answer from the knowledge base 234. In some implementations, the local LLM module 216 searches the knowledge base 234 using a search-enhanced generation technique based on an LLM local to the computing device 110. The local LLM module 216 may retrieve chunks of text from the knowledge base 234, convert those chunks into embedding vectors, and compare those embedding vectors with the embedding vector 207. Based on these comparisons, the local LLM module 216 may identify the answer to the user question and output that answer as answer 208 to the user interface module 240.

[0058] In some implementations, Knowledge Base 234 includes one or more documents related to the computing device 110. Examples of such documents include user manuals, quick start guides, troubleshooting guides, support articles, technical specifications, and other user and support supplementary materials related to the computing device 110.

[0059] In some implementations, the user interface module 240 may prompt the user to indicate whether the answer identified by the local LLM module 216 is satisfactory to the user, or at least acceptable. If the user indicates that the answer identified by the local LLM module 216 is acceptable, the assistant module 210 may terminate the secondary query and enter a waiting state for the next question. If the user indicates that the answer identified by the local LLM module 216 is unacceptable, or if the local LLM module 216 cannot identify an answer from the knowledge base 234, the assistant module 210 may further escalate the user question to a tertiary query that may be handled by the cloud module 218.

[0060] The cloud module 218 is configured to communicate with the cloud assistant system 120 via one or more networks (for example, network 130 in Figure 1). In some implementations, the cloud module 218 implements an Application Programming Interface (API). The cloud module 218 may send and / or receive communications to and from the cloud assistant system 120 via the API. When the assistant module 210 escalates a user question to a tertiary query, the cloud module 218 may use the API to send an embedding vector 207 to the cloud assistant system 120. The cloud assistant system 120 may determine the answer to the user question by analyzing the knowledge base 224 using the LLM.

[0061] In some implementations, the cloud-based knowledge base 224 contains other resources and information related to the computing device 110, along with supplementary materials related to the computing device 110. For example, in the case of consumer electronics, these other resources and information may include web pages and forum messages discussing the consumer electronics. In some implementations, the knowledge base 234, which is stored locally on the computing device 110, is part of the knowledge base 224.

[0062] Figure 3 illustrates an example of an operational flow 300 for a first-line query to an artificial intelligence (AI) assistant in some implementations. For illustrative purposes, flow 300 may show the operational flow of an AI assistant implemented in a computing device (e.g., computing device 110) (e.g., implemented by assistant engine 112). The computing device is a consumer electronics product, but this is not intended to limit it. Therefore, the following description may refer to various embodiments (e.g., elements, components, etc.) shown in Figure 1-2. In some implementations, a custom or product-specific question-and-answer database may be generated for devices such as consumer electronics products to address potential product-related issues. The AI ​​assistant within a consumer electronics product may omit the ability to answer general knowledge questions such as "What is the capital of Bolivia?" and instead be designed to specialize in product-specific questions, such as "Why aren't my dishes clean?" or "Is it safe to put plastic in the dishwasher?" in the case of a dishwasher. Operational flow 300 shows the flow of an AI assistant implemented as part of a dishwasher, washing machine or dryer, refrigerator, or other types of consumer electronics products not specifically mentioned. In particular, operation flow 300 shows the flow of a first-line inquiry to an AI assistant implemented in a home appliance.

[0063] As shown in the diagram, flow 300 includes a step in which the user utters a user utterance 310. The user utterance may include a speech command (e.g., a user question) and may be input to an AI assistant (e.g., an assistant engine 112). Home appliances may have a microphone or other input device for recording and processing speech input. The assistant engine 112 may perform speech activity detection 320 (e.g., via a speech activity detection module 242) to detect the user utterance and trigger further processing of the user utterance.

[0064] The assistant engine 112 may perform speech recognition processing 330 on the user utterance 310, converting the user utterance into text 312. The speech recognition processing may be performed by a commercially available solution or a proprietary solution, but is not limited to this. In some implementations, the speech recognition processing may be performed on-device (e.g., using an on-device speech recognition model such as the "Moonshine" model). The converted text 312 may be input to a semantic search 340. In some implementations, the text 312 may contain a user question included in the user utterance 310, which could be an example of the question text 206 in Figure 2. In some implementations, the assistant engine 112 (e.g., a sentence conversion module 212) may perform sentence conversion 344 on the text 312, converting the text 312 into an embedding vector (e.g., an embedding vector 207). The QA search module 214 may perform a semantic search 340 using the embedding vector of the text 312.

[0065] QA database 345 may be generated offline. In some implementations, QA database 345 is an example of QA database 232 in Figure 2. QA database 345 may contain a set of pre-generated questions and answers. Pre-generated questions and answers may be generated (e.g., extracted) from a knowledge base (e.g., knowledge base 224) of information related to the consumer electronics product (e.g., supplementary materials). This information may include user manuals, technical specifications, support articles, and other reference materials. A cloud-based system (e.g., cloud assistant system 120) may generate QA database 345 (343) using LLM (e.g., by analyzing the knowledge base using LLM) and send QA database 345 to the consumer electronics product for storage.

[0066] For semantic search 340, a set of questions 341 may be retrieved based on the QA database 345. The QA search module 214 may retrieve questions 341 from the QA database 345 as input to semantic search 340. In some implementations, questions 341 may be in embedding vector format, with the QA database 345 storing the embedding vectors for the questions. In some implementations, if questions 341 are in text format, the assistant engine 112 may also perform sentence conversion 344 on the text of questions 341 (e.g., using the sentence conversion module 312) to convert it into an embedding vector. The QA search module 214 may perform semantic search 340 using the embedding vectors of text 312 and the embedding vectors of questions 341. For example, the QA search module 214 may match the embedding vectors of text 312 with the embedding vectors of questions 341. Semantic search 340 may match the user question in text 312 with the nearest or most similar question 341 stored in the QA database 345.

[0067] The QA search module 214 performs a lookup operation 350 to retrieve an answer 342 from the QA database 345. The retrieved answer may be an answer corresponding to a question matched in the semantic search 340, and may represent the closest match based on a given quality metric.

[0068] When an answer is found in the QA database 345, the assistant module 210 may send the retrieved answer 342 (for example, as answer 208) to the speech synthesis module 246. The speech synthesis module 246 may perform a speech synthesis process 360 to convert the retrieved answer 342 from text to speech and play the speech output 362 (for example, output 104) on an audio output device, including but not limited to speakers and headphones. In some implementations, the speech synthesis process may be performed on-device on a consumer electronics product (for example, using an on-device speech synthesis model such as the "Piper" model). The audio playback device may have a communication protocol (including but not limited to Bluetooth) for playing the speech output 362 on a remote device (for example, a device paired with the computing device 110). In some implementations, the speech synthesis module 246 may cache the audio data of converted speech from one or more previously retrieved answers on the computing device 110. If there is cached audio data on the computing device 110 for the acquired response 342, the speech synthesis module 246 may output the cached audio data instead of performing a speech synthesis operation on the acquired response 342.

[0069] If the lookup operation 350 fails to match the user with an answer (for example, because the semantic search 340 fails to match the user question with question 341), or if the user is not satisfied with the outputted answer, the assistant engine 112 may escalate the user question to a secondary query (370) as shown in Figure 4 below.

[0070] Figure 4 illustrates an example of the operation flow 400 for a secondary query to an AI assistant in some implementations. Operation flow 400 follows the example illustrated in Figure 3. In particular, operation flow 400 illustrates the flow of a secondary query when the primary query in operation flow 300 is escalated. Therefore, the following description may refer to various aspects (e.g., elements, components, etc.) shown in Figures 1-3.

[0071] A user question 405 (for example, in embedded vector format converted from text 312) may be input to a search extension generation operation 420. Search extension generation 420 may include a semantic search 410 based on a local (device-based) LLM 430. The semantic search 410 may include atomic text fragments organized as vectors based on their meaning. The atomic text fragments may include text chunks from a knowledge base 412 (e.g., knowledge base 234), which may contain user manuals and other supplementary materials related to consumer electronics. These text fragments may be converted from text to embedded vectors (411) by a sentence conversion module 212. In some implementations, search extension generation 420 is performed by an assistant module 210, which includes a local LLM module 216.

[0072] The answer may be generated based on the search extension generation 420. The assistant module 210 may send the answer to the user interface module 240, which may test the answer 440 by prompting the user to indicate whether the answer is appropriate, or at least acceptable or satisfactory. If the user indicates that the answer is inappropriate, the user question may be escalated to a tertiary query (460) as illustrated in Figure 5 below, based on the user's feedback. Furthermore, if the answer is insufficient based on feedback from the local LLM (for example, if the confidence level output by the local LLM falls below a threshold), the user question may be escalated to a tertiary query (460).

[0073] If the response is deemed appropriate, the assistant module 210 may send the response generated by the search extension generation 420 to the speech synthesis module 246. The speech synthesis module 246 may perform speech synthesis processing 450 to convert the response from text to speech and play the speech output 470 (e.g., output 104) on an audio output device, including but not limited to speakers and headphones. In some implementations, the speech synthesis processing may be performed on-device on a consumer electronics product (e.g., using an on-device speech recognition model such as the "Piper" model). The audio playback device may include a communication protocol (including but not limited to Bluetooth) for playing the speech output 470 on a remote device (e.g., a device paired with the computing device 110).

[0074] Figure 5 illustrates an example of the operation flow 500 for a tertiary query to an AI assistant in some implementations. Operation flow 500 follows the example shown in Figure 3-4. In particular, operation flow 500 illustrates the flow of a tertiary query when the secondary query in operation flow 400 is escalated. Therefore, the following explanation may refer to various aspects (e.g., elements, components, etc.) shown in Figure 1-4.

[0075] The assistant module 210 (e.g., cloud module 218) may call API 511 to send the user question 501 in embedded vector format to the cloud assistant system 540 (e.g., cloud assistant system 120). The cloud assistant system 540 may use LLM 510 to analyze the knowledge base 542 (e.g., knowledge base 224) and generate a response (e.g., answer 502) to the user question 501. The cloud assistant system 540 may call API 522 to send the generated answer 502 back to the assistant module 210. The assistant module 210 may send the answer 502 to the speech synthesis module 246. Furthermore, in some implementations, the assistant module 210 may feed back the question 502 and answer 502 to the QA database 345 (not shown in Figure 5) (512).

[0076] The speech synthesis module 246 may perform speech synthesis processing 520 to convert the response 502 from text to speech and play the speech output 530 (e.g., output 104) on an audio output device, including but not limited to speakers and headphones. In some implementations, speech synthesis processing may be performed on-device on a consumer electronics product (e.g., using an on-device speech model such as the "Piper" model). The audio playback device may include a communication protocol (including but not limited to Bluetooth) for playing the speech output 530 on a remote device (e.g., a device paired with the computing device 110).

[0077] Figure 6 shows a block diagram of the assistant system 600 in some implementations. In some implementations, the assistant system 600 may be an example of the computing device 110 in Figure 1.

[0078] The assistant system 600 comprises an input / output interface 610, a network interface 612, a processing system 620, and a memory 630. The input / output interface may include one or more input devices, output devices, or one or more interfaces for communicating with input / output devices. The network interface 612 may include one or more interfaces for communicating with one or more remote devices and networks such as local area networks, wide area networks, and cellular networks, or for communicating with one or more local devices, via wired or wireless connections. More specifically, with respect to this disclosure, the network interface 610 may connect the assistant system 600 to a remote assistant system such as the cloud assistant system 120 in Figure 1 in a communicable manner.

[0079] Memory 630 may include a QA datastore 531 configured to store a question-and-answer database (e.g., QA database 232) and a local LLM datastore 632 configured to store model data associated with a local large-scale language model (e.g., for execution by a local LLM module 216). Memory 630 may also include non-temporary computer-readable media (such as one or more non-volatile memory elements like EPROM, EEPROM, flash memory, or hard drives) that may store at least the following software (SW) modules: • User input receiving SW module 634 that receives user input including user questions. • Vector generation SW module 636 generates a first vector representing the user question. A vector matching SW module 638 that matches a first vector with a second vector representing a question stored in a question-answer database (e.g., QA data store 631), and • Escalating SW module 640 that selectively escalates user questions for processing by machine learning models.

[0080] Each software module includes instructions that, when executed by the processing system 620, cause the assistant system 600 to perform the corresponding function.

[0081] The processing system 620 may include one or more suitable processors capable of executing scripts or instructions of one or more software programs stored in the assistant system 600 (for example, in memory 630). For example, the processing system 620 may execute a vector generation SW module 636 to generate vectors representing user questions. Similarly, the processing system 620 may execute a vector matching SW module 638 to match vectors representing user questions with vectors representing questions stored in the QA data store 631.

[0082] Figure 7 illustrates a flowchart showing an example of a method 700 for operating an AI assistant, based on some implementations. Method 700 may be performed by a computing device 110, as described, for example, with reference to Figure 1-2.

[0083] As shown in the figure, in block 702, the computing device receives user input. The user input includes a user question. In block 704, the computing device generates a first vector representing the user question.

[0084] In block 706, the computing device matches a first vector with a second vector representing a question stored in a question-answer database within the computing device. The question-answer database contains multiple questions extracted from a knowledge base related to the computing device, and the corresponding answers associated with those questions. In block 708, the computing device selectively escalates user questions for processing by a machine learning model.

[0085] In some embodiments, user input may include utterances corresponding to user questions, and the computing device may convert these utterances into text corresponding to the user questions.

[0086] In some embodiments, the computing device may convert the text of the user's question into a first vector.

[0087] In some embodiments, a knowledge base related to a computing device may include supplementary materials related to that computing device.

[0088] In some configurations, multiple questions and their associated answers are extracted from a knowledge base based on a large-scale language model (LLM).

[0089] In some embodiments, a set of questions and their respective answers are extracted from a knowledge base by a system located remotely from the computing device and transmitted from that remote system to the computing device.

[0090] In some embodiments, a computing device may determine which vector is closest to a first vector from among several vectors representing multiple questions.

[0091] In some embodiments, a computing device may retrieve an answer corresponding to a second vector from a question-and-answer database and output an answer corresponding to the second vector.

[0092] In some embodiments, a computing device may convert the text of the response corresponding to the second vector into speech and output that speech as audio.

[0093] In some embodiments, a computing device may not escalate a user question if the user response indicates that the outputted answer is acceptable, and may escalate a user question if the user response indicates that the outputted answer is unacceptable.

[0094] In some embodiments, computing devices may perform semantic searches of knowledge bases based on escalated user questions and large language models (LLMs).

[0095] In some embodiments, computing devices may selectively send user queries to a cloud-based system for LLM-based processing by the cloud-based system.

[0096] Those skilled in the art will understand that information and signals can be represented using a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips, which may be mentioned throughout the above description, can be represented by voltage, electric current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0097] Furthermore, those skilled in the art will understand that various exemplary logic blocks, modules, circuits, and algorithmic steps described in relation to the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly demonstrate this hardware- and software compatibility, various exemplary components, blocks, modules, circuits, and steps are described above in terms of their functionality as a whole. Whether such functionality is implemented as hardware or software depends on the design constraints imposed on the individual application and the overall system. Those skilled in the art may implement the described functionality in various ways for individual applications, but such decisions should not be construed as resulting in a deviation from the scope of the disclosure.

[0098] Methods, sequences, or algorithms described in relation to the embodiments disclosed herein may be implemented directly in hardware, in software modules executed by a processor, or in a combination of both. The software modules may be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is connected to the processor so that the processor can read information from and write information to the storage medium. Alternatively, the storage medium may be integrated with the processor.

[0099] This specification has described implementations with reference to specific examples. However, it is clear that various changes and modifications are possible without departing from the broad scope of disclosure set forth in the attached claims. Therefore, this specification and the drawings should be interpreted as illustrative rather than restrictive.

Claims

1. A method for operating an artificial intelligence (AI) assistant on a computing device, Receiving user input, including user questions, To generate a first vector representing the user question, The first vector is compared with a second vector representing a question stored in the question-answer database on the computing device, Selectively escalating the user questions for processing by a machine learning model, Includes, The aforementioned question-and-answer database includes a plurality of questions extracted from a knowledge base related to the computing device, and the respective answers corresponding to the plurality of questions. method.

2. The user input includes an utterance corresponding to the user question, The method further includes converting the utterance into text corresponding to the user question using the computing device, The method according to claim 1.

3. Generating the first vector includes converting the text of the user question into the first vector. The method according to claim 1.

4. The knowledge base relating to the computing device includes supplementary materials relating to the computing device, The method according to claim 1.

5. The aforementioned multiple questions and their respective answers are extracted from the knowledge base based on a Large-Scale Language Model (LLM). The method according to claim 1.

6. The aforementioned multiple questions and their respective answers are extracted from the knowledge base by a system located remotely from the computing device and transmitted from the remote system to the computing device. The method according to claim 1.

7. Matching the first vector with the second vector includes the computing device determining the vector closest to the first vector from among the vectors representing the plurality of questions. The method according to claim 1.

8. Obtaining the answer corresponding to the second vector from the aforementioned question and answer database, Outputting the answer corresponding to the second vector, This also includes, The method according to claim 1.

9. Outputting the answer corresponding to the second vector is Converting the text of the response corresponding to the second vector into speech, Outputting the aforementioned utterance as speech, including, The method according to claim 8.

10. Selectively escalating the aforementioned user questions The user question will not be escalated if the user response indicates that the outputted answer is acceptable. The user question is escalated if the user response indicates that the outputted answer is unacceptable. including, The method according to claim 8.

11. Processing the escalated user questions by the machine learning model includes performing a semantic search of the knowledge base on the computing device based on the escalated user questions and a large-scale language model (LLM). The method according to claim 1.

12. To selectively send the user questions to the cloud-based system for processing by the LLM-based cloud-based system, This also includes, The method according to claim 11.

13. Processing system and, Memory coupled to one or more processors, A computing device equipped with, When the memory is executed by the processing system, the computing device, Receive user input including user questions, A first vector representing the user question is generated, In the computing device, the first vector is compared with a second vector representing a question stored in the question-answer database. Selectively escalate the user questions for processing by a machine learning model. It stores instructions, The aforementioned question-and-answer database includes a plurality of questions extracted from a knowledge base related to the computing device, and the respective answers corresponding to the plurality of questions. Computing device.

14. The user input includes an utterance corresponding to the user question, When the instruction is executed by the one or more processors, the computing device converts the utterance into text corresponding to the user question, and converts the text corresponding to the user question into a first vector. The computing device according to claim 13.

15. The knowledge base relating to the computing device includes accompanying documentation relating to the computing device, The computing device according to claim 13.

16. The aforementioned multiple questions and their respective answers are extracted from the knowledge base based on a Large-Scale Language Model (LLM). The computing device according to claim 13.

17. When the instruction is executed by the one or more processors, the computing device, The answer corresponding to the second vector is obtained from the aforementioned question and answer database. Output the answer corresponding to the second vector. The computing device according to claim 13.

18. When the instruction is executed by the one or more processors, the computing device, If the user response indicates that the outputted answer is acceptable, the user question will not be escalated. If the user response indicates that the outputted answer is unacceptable, the user question is escalated. The computing device according to claim 17.

19. When the instruction is executed by one or more processors, the computing device performs a semantic search of the knowledge base based on the escalated user question and the Large Language Model (LLM). The computing device according to claim 13.

20. When the instruction is executed by the one or more processors, the computing device selectively sends the user query to the cloud-based system for processing by the LLM-based cloud-based system. The computing device according to claim 19.

Images