System and method for managing execution plans for artificial intelligence based assistance devices

Abstract

A method for managing an execution plan for an artificial intelligence (AI)-based assistant device includes receiving a first input indicative of a voice command from a user; determining, based on the first input, a first context associated with the AI-based assistant device and an intent of the user; generating, based on the first context, a first execution plan and a second execution plan; generating a first timeline connecting the first execution plan and the second execution plan; detecting, based on the first timeline and a second context of the AI-based assistant device or a second context of the user, a change from the first context; generating an updated execution plan based on the change; and generating a second timeline by modifying the first timeline based on the updated execution plan.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence (AI)-based assistive devices, and more specifically, to a method and system for managing execution plans for AI-based assistive devices. Background Technology

[0002] In recent years, artificial intelligence (AI)-based assistive devices (referred to as devices in this paper) have revolutionized the way users interact with technology. These devices are small gadgets equipped with programming / software that allow users to perform tasks via voice commands. These devices perform multiple functions, such as setting reminders and alarms, playing music, controlling smart home appliances, or providing weather updates. These devices ensure hands-free control, enhancing the user experience.

[0003] Currently, to operate these devices, users typically issue commands for each task and receive responses. However, the operations performed by these devices have specific limitations. For example, when a user issues a command to the device to perform an operation, the user only receives a response to that operation—e.g., a single input and a single output—until execution is complete. The user receives responses without any further active involvement or assistance from the device. The execution plan created for each command provided by the user is static and independent of other commands. The execution plan is created corresponding to the command and is not adaptive. This means that after executing a command, the user must independently think and interact with the device to perform subsequent tasks. For example, as... Figure 1 As shown, the user issues command 102 to set an alarm for 5 a.m. to go to location X. In this case, the device simply sets alarm 104 to 5 a.m. and does not offer the user any further active participation, advice, or assistance.

[0004] This results in a lack of continuity and assistance beyond a single command, requiring users to repeatedly interact with the device for each subsequent task, thus increasing user discomfort.

[0005] Previous solutions in this field have focused on predicting future user activity based on past events, typically revolving around the user's physical environment or activities such as sleep patterns or exercise habits. These methods do not involve a comprehensive analysis of potential tasks that the user may not have performed or even considered in the past.

[0006] In view of the above problems, it is advantageous to provide an improved system and method to overcome one or more of the aforementioned problems / challenges associated with the device. Summary of the Invention

[0007] Technical solution According to one aspect of this disclosure, a method for managing an execution plan for an artificial intelligence (AI)-based assistive device includes: receiving a first input indicating a voice command from a user; determining a first context associated with the AI-based assistive device and the user's intent based on the first input; generating a first execution plan and a second execution plan based on the first context; generating a first timeline connecting the first execution plan and the second execution plan; detecting changes from the first context based on the first timeline and a second context of the AI-based assistive device or a second context of the user; generating an updated execution plan based on the changes; and generating a second timeline by modifying the first timeline based on the updated execution plan.

[0008] According to one aspect of this disclosure, a method for managing an execution plan for an artificial intelligence (AI)-based assistive device includes: receiving a first input via a microphone indicating a voice command from a user; determining a first context associated with the AI-based assistive device and the user's intent based on the first input; generating a first execution plan based on the first context; and generating one or more second execution plans based on the first context, wherein the first execution plan indicates one or more first tasks to be performed in response to the first context, and the one or more second execution plans indicate one or more second tasks to be performed in response to the first context; generating a first timeline connecting the first execution plan and the one or more second execution plans; detecting changes in the first context based on the first timeline and at least one of a second context of the AI-based assistive device or a second context of the user; generating an updated execution plan based on the changes; and generating a second timeline by modifying the first timeline based on the updated execution plan.

[0009] According to one aspect of this disclosure, a system for managing execution plans for an artificial intelligence (AI)-based assistive device is provided. The system includes: a memory storing instructions; and at least one processor in communication with the memory, wherein the instructions, when executed by the at least one processor, cause the system to: receive a first input via a microphone indicating a voice command from a user; determine a first context associated with the AI-based assistive device and the user's intent based on the first input; generate a first execution plan based on the first context, and generate one or more second execution plans based on the first context, wherein the first execution plan indicates one or more first tasks to be performed in response to the first context, and the one or more second execution plans indicate one or more second tasks to be performed in response to the first context; generate a first timeline connecting the first execution plan and the one or more second execution plans; detect changes in the first context based on the first timeline and at least one of a second context of the AI-based assistive device or a second context of the user; generate an updated execution plan based on the changes; and generate a second timeline by modifying the first timeline based on the updated execution plan. Attached Figure Description

[0010] The foregoing and other features of the embodiments will become more apparent when read in conjunction with the accompanying drawings and the following detailed description of the embodiments. In the drawings, the same reference numerals denote the same elements.

[0011] Figure 1 The operation performed by an artificial intelligence (AI)-based assistive device is shown; Figure 2 An environment for managing an execution plan for an artificial intelligence (AI)-based assistive device, according to embodiments of the present disclosure, is illustrated. Figure 3 A block diagram of a system for managing the execution plan of an AI-based assistive device according to an embodiment of the present disclosure is shown; Figure 4 An architecture for managing an execution plan for an AI-based assistive device, according to embodiments of the present disclosure, is shown. Figure 5A and Figure 5B The process of determining a first context, a first execution plan, and one or more second execution plans by a system according to an embodiment of the present disclosure is illustrated; Figure 5C and Figure 5D A first execution plan and one or more second execution plans according to embodiments of the present disclosure are shown; Figure 6 The process of generating a first timeline and a second timeline by a system according to an embodiment of the present disclosure is illustrated; Figure 7 The operation of a dynamic execution plan generator according to embodiments of the present disclosure is illustrated; Figure 8 A flowchart depicting a method for managing an execution plan for a device according to embodiments of the present disclosure is shown; Figure 9 An example timeline representation of a use case scenario implementation of a system and method for generating a first execution plan based on a first input from a user, according to embodiments of this disclosure, is shown; and Figure 10A and Figure 10B Another example timeline representation of a use case scenario according to embodiments of this disclosure is shown. Detailed Implementation

[0012] To facilitate an understanding of the principles of this disclosure, reference will now be made to various embodiments, and these embodiments will be described using language. However, it will be understood that this is not intended to limit the scope of the disclosure, and such changes and further modifications in the illustrated systems, as well as such further applications of the principles of the disclosure as illustrated herein, are considered to be common knowledge of those skilled in the art to which this disclosure pertains.

[0013] Those skilled in the art will understand that the foregoing general description and the following detailed description are for the purpose of interpreting this disclosure and not limiting it.

[0014] Whether a feature or element is limited to being used only once, it may still be referred to as “one or more features” or “one or more elements” or “at least one feature” or “at least one element”. Unless otherwise stated by restrictive language (including, but not limited to, “may have one or more…” or “requires one or more elements”), the use of the terms “one or more” or “at least one” for a feature or element does not preclude the absence of that feature or element.

[0015] The expressions “at least one of A, B and C” and “at least one of A, B or C” both indicate “A”, “B” only, “C” only, both “A and B”, both “A and C”, both “B and C”, and all of “A, B and C”.

[0016] This document refers to several "Examples". It should be understood that the embodiments are examples of implementing any feature and / or element of this disclosure. Several embodiments have been described to explain one or more potential ways in which the features and / or elements of this disclosure satisfy the requirements of uniqueness, utility, and non-obviousness.

[0017] The use of phrases and / or terms including, but not limited to, “first embodiment,” “another embodiment,” “optional embodiment,” “one embodiment,” “embodiment,” “multiple embodiments,” “some embodiments,” “other embodiments,” “further embodiments,” “additional embodiments,” “additional embodiments,” or other variations thereof, does not necessarily refer to the same embodiment. Unless otherwise stated, one or more specific features and / or elements described in connection with one or more embodiments may be found in one embodiment, or in more than one embodiment, or in all embodiments, or may not be found in any embodiment. Throughout this document, although one or more features and / or elements may be described in the context of a single embodiment only, or in the context of more than one embodiment, or in the context of all embodiments, features and / or elements may alternatively be provided individually or in any suitable combination, or not at all. Conversely, any features and / or elements described in the context of different embodiments may be implemented to exist together in the context of a single embodiment.

[0018] All the details set forth herein are used in the context of some embodiments and should therefore not be construed as limiting this disclosure.

[0019] The terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process or method that includes a list of steps may include not only those steps but may also include other steps not expressly listed or inherent to such process or method. The inclusion of one or more devices, subsystems, elements, structures, or components beginning with “comprising” does not exclude the presence of other devices, subsystems, elements, structures, components, or additional devices, subsystems, elements, structures, or components, unless further constraints are imposed.

[0020] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0021] For clarity, the first digit of the reference numeral for each component in this disclosure indicates the drawing number, in which the corresponding component is shown. For example, at least in Figure 1 The reference numerals shown begin with the number "1". Similarly, at least in Figure 2 The reference numerals that begin with the number "2" are shown in the figure.

[0022] Figure 2 An environment 200 for managing an execution plan for an artificial intelligence (AI)-based assistive device 202, according to an embodiment of the present disclosure, is shown. Figure 3 A block diagram 300 of a system 204 for managing the execution plan of an AI-based assistive device 202 according to an embodiment of the present disclosure is shown.

[0023] In embodiments, without departing from the scope of this disclosure, the artificial intelligence (AI)-based assistive device (referred to herein as device) 202 can be any device capable of operating based on commands / instructions received from a user. When device 202 receives commands / instructions from a user to perform some task, in this case, device 202 generates output solely based on the commands / instructions provided by the user, lacking continuity and assistance beyond a single command. System 204 can communicate with device 202. System 204 can be configured to operate based on multiple inputs, such that system 204 manages the execution plan for device 202. Based on this management, device 202 can provide customized initiative, assistance, and suggestions to the user based on the task. Thus, user comfort is ensured. In another embodiment, without departing from the scope of this disclosure, system 204 can be deployed within device 202.

[0024] In an embodiment, system 204 may include, but is not limited to, at least one processor 304 (referred to herein as processor 304), memory 308, and multiple modules 312, as well as other examples explained in detail in subsequent paragraphs.

[0025] System 204 may include input / output (I / O) interface 338 and transceiver 340. In some embodiments where system 204 is implemented as a standalone entity in a server / cloud architecture, system 204 may communicate with multiple devices to receive data from each of the multiple devices, and the details provided below regarding system 204 and device 202 also apply to system 204 and the multiple devices.

[0026] In an example embodiment, processor 304 may be operatively coupled to each of I / O interface 338, multiple modules 312, transceiver 340, and memory 308. In one embodiment, processor 304 may include a graphics processing unit (GPU) and / or an artificial intelligence engine (AIE). In one embodiment, processor 304 may include at least one data processor for executing processes in a virtual memory area network. For example, processor 304 may include a dedicated processing unit (such as an integrated system (bus) controller), a memory management control unit, a floating-point unit, a graphics processing unit, or a digital signal processing unit. In one embodiment, processor 304 may include a central processing unit (CPU), a graphics processing unit (GPU), or both. Processor 304 may be one or more general-purpose processors, digital signal processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), servers, networks, digital circuits, analog circuits, combinations thereof, or other means now known or hereafter developed for analyzing and processing data. Processor 304 may run software programs (such as manually generated (e.g., programmed) code) to perform desired operations.

[0027] Processor 304 may be configured to communicate with one or more input / output (I / O) devices via I / O interface 338. In some embodiments, processor 304 may use I / O interface 338 to communicate with device 202. In some embodiments, I / O interface 338 may be implemented within device 202. I / O interface 338 may employ Code Division Multiple Access (CDMA), High-Speed ​​Packet Access (HSPA+), Global System for Mobile Communications (GSM), Long Term Evolution (LTE), WiMax, etc. In embodiments, I / O interface 338 may use devices such as, but not limited to, a display, keyboard, mouse, touchscreen, microphone, speaker, etc., to enable input to system 204 and output from system 204.

[0028] Using I / O interface 338, system 204 can communicate with one or more I / O devices (device 202), whereby system 204 manages execution plans for device 202. For example, input devices may be antennas, microphones, touchscreens, touchpads, storage devices, transceivers, or video devices / sources. For example, output devices may be video displays (e.g., cathode ray tube (CRT), liquid crystal displays (LCD), light-emitting diodes (LEDs), plasma displays, plasma display panels (PDPs), organic light-emitting diode displays (OLEDs), etc.) or audio speakers.

[0029] Processor 304 can be configured to communicate with a communication network via a network interface. In an embodiment, the network interface may be I / O interface 338. The network interface can be connected to a communication network to enable connection between system 204 and device 202. The network interface may employ a connection protocol, including but not limited to, direct connection, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), Transmission Control Protocol / Internet Protocol (TCP / IP), Token Ring, or IEEE 802.11a / b / g / n / x. For example, the communication network may include, but is not limited to, direct interconnect, local area network (LAN), wide area network (WAN), wireless network (e.g., using Wireless Application Protocol), or the Internet. Using the network interface and the communication network, system 204 can communicate with other devices. For example, the network interface may employ a connection protocol, including but not limited to, direct connection, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), Transmission Control Protocol / Internet Protocol (TCP / IP), Token Ring, or IEEE 802.11a / b / g / n / x.

[0030] Transceiver 340 may be configured to receive signals from and / or send signals to device 202. In one embodiment, database may be configured to store information, and processor 304 may perform one or more functions for managing execution plans for device 202.

[0031] In some embodiments, memory 308 may be communicatively coupled to processor 304. Memory 308 may be configured to store data and instructions executable by processor 304 to perform one or more methods disclosed herein throughout. In one embodiment, memory 308 may be located within device 202. In another embodiment, memory 308 may be located within system 204, remote from device 202. In yet another embodiment, memory 308 may communicate with processor 304 via a bus within system 204. In yet another embodiment, memory 308 may be located remotely from processor 304 and may communicate with processor 304 via a network. Memory 308 may include, but is not limited to, non-transitory computer-readable storage media (such as various types of volatile and non-volatile storage media), including but not limited to random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, magnetic tape or disk, optical media, etc.

[0032] In one example, memory 308 may include a cache or random access memory for processor 304. In alternative examples, memory 308 may be separate from processor 304 (such as a processor cache), system memory, or other memory. Memory 308 may be an external storage device or database for storing data. Memory 308 is operable to store instructions that can be executed by processor 304. The functions, actions, or tasks shown or described in the figures may be performed by a programmed processor 304 for executing instructions stored in memory 308. These functions, actions, or tasks are independent of a particular type of instruction set, storage medium, processor, or processing strategy, and may be performed by software, hardware, integrated circuits, firmware, microcode, etc., operating individually or in combination. Similarly, processing strategies may include multiprocessing, multitasking, parallel processing, etc.

[0033] In some embodiments, multiple modules 312 may be included within memory 308. Memory 308 may also include a database for storing data. The multiple modules 312 may include an instruction set executable to cause system 204 and its processor 304 to perform any one or more of the methods / processes disclosed herein. The multiple modules 312 may be configured to perform steps of this disclosure using data stored in the database. For example, the multiple modules 312 may be configured to perform… Figures 4 to 7 The technology disclosed in China.

[0034] In this embodiment, each of the plurality of modules 312 may be a hardware unit that is external to the memory 308. The memory 308 may include an operating system for performing one or more tasks of the system 204, such as one or more tasks performed by a general-purpose operating system.

[0035] In one example, multiple modules 312 may include a receiving module 314, a generating module 316, a separating module 318, a determining module 320, an identifying module 322, a providing module 324, a fusing module 326, an extracting module 328, an aggregating module 330, a sending module 331, a processing module 332, an analysis module 334, an execution module 335, and a detection module 336. Each of modules 314-336 can communicate with each other. Each of modules 314-336 can communicate with the processor 304.

[0036] This disclosure envisions a computer-readable medium that includes instructions or receives and executes instructions in response to a propagated signal. Instructions can be sent or received over a network via a communication port or interface or using a bus. The communication port or interface can be part of processor 304 or can be a separate component. The communication port can be created in software or can be a physical connection in hardware form.

[0037] The communication port can be configured to connect to a network, external media, a display, or any other component or combination thereof in the system. The connection to the network can be a physical connection (such as a wired Ethernet connection) or can be established wirelessly. Similarly, additional connections to other components of system 204 can be physical or can be established wirelessly. The network can be directly connected to the bus. For the sake of brevity, the architecture and standard operation of memory 308, processor 304, transceiver 340, and I / O interface 338 are not discussed in detail.

[0038] In this embodiment, the operation of system 204 for managing execution plans for device 202 is explained in detail. Processor 304, in conjunction with receiving module 314, generating module 316, separating module 318, determining module 320, identifying module 322, providing module 324, fusing module 326, extracting module 328, aggregating module 330, sending module 331, processing module 332, analyzing module 334, execution module 335, and detection module 336, can be configured to perform the combination in subsequent paragraphs. Figures 4 to 7 The operation is explained in the paragraph.

[0039] Figure 4 An architecture of a system 204 for managing an execution plan for a device 202, according to an embodiment of the present disclosure, is shown. Figure 5A and Figure 5B The process of determining a first context, a first execution plan, and one or more second execution plans by system 204 according to an embodiment of the present disclosure is illustrated. Figure 5C and Figure 5D A first execution plan and one or more second execution plans are shown according to embodiments of the present disclosure. Figure 6The process of generating a first timeline and a second timeline by system 204 according to an embodiment of the present disclosure is illustrated. Figure 7 The operation of a dynamic execution plan generator 604 according to an embodiment of the present disclosure is illustrated.

[0040] In the embodiments, for the sake of brevity, they may be explained in combination. Figures 3 to 7 .

[0041] Reference Figure 3 , Figure 4 and Figure 5A In one embodiment, the receiving module 314 may be configured to receive a first input from a user instructing a voice command. For example, the voice command may instruct the user to set an alarm for 5:00 AM to travel to location XYZ. In another embodiment, without departing from the scope of this disclosure, the first input may be in the form of text or any other input modality. Voice input may be obtained via I / O interface 338 based on audio signals or audio data received via an input device (such as an antenna, microphone, audio transceiver, video device, video transceiver, or other audio or video source).

[0042] In an embodiment, the generation module 316 may be configured to generate a text format corresponding to the received first input by converting the received first input into a text format via at least one of predefined technologies. The at least one of the predefined technologies may include, but is not limited to, automatic speech recognition technology 406. Automatic speech recognition technology 406 may be defined as a technology that enables system 204 to recognize a user's voice commands and process them into text format. The separation module 318 may be configured to separate the generated text format into multiple domains by classifying the generated text format via at least one of the predefined technologies. The at least one of the predefined technologies may include, but is not limited to, natural level understanding (NLU) technology 404. NLU technology 404 is defined as a technology that helps 204 understand and interpret text format / voice commands in a context-appropriate manner. In an embodiment, without departing from the scope of this disclosure, the multiple domains may include, for example, search, reminders, and alarm clocks.

[0043] Based on the first input, a first context associated with device 202 can be determined in coordinator unit 407 of system 204. In embodiments, for example, coordinator unit 407 may be configured to provide seamless integration of multiple data sources, neural networks, and different modules to determine the first context. The operations performed to determine the first context are explained in detail in the following paragraphs. 3 In one embodiment, the determining module 320 may be configured to determine a first context associated with the device 202 and the user intent based on the first input. In another embodiment, the generating module 316 may be configured to generate a word embedding for the received first input by converting the discrete text format corresponding to the received first input into word embeddings via an embedding model. In another embodiment, without departing from the scope of this disclosure, the discrete text format corresponding to the received first input may be converted into word embeddings using at least one predefined technique (e.g., a frequency-based model technique or word2Vec technique). The identifying module 322 may be configured to identify multiple data points corresponding to the received first input based on a vector search of the generated word embeddings in a predefined vector database. In yet another embodiment, without departing from the scope of this disclosure, the identifying module 322 may be configured to identify multiple data points based on performing multiple search operations (e.g., reordering, filtering, etc.). The multiple data points indicate multiple services associated with the first input. In another embodiment, without departing from the scope of this disclosure, for example, the vector database of supported external services may include information about weather or navigation.

[0044] The generation module 316 can be configured to generate a predetermined number of priority data corresponding to the received first input from the identified multiple data entries. The generation module 316 can be configured to generate the predetermined number of priority data by reordering the identified multiple data entries based on the received first input via a reordering model. In embodiments, the reordering of the identified multiple data entries can be performed using, but is not limited to, various RAG techniques without departing from the scope of this disclosure. The predetermined number of priority data indicates the data with the highest priority compared to the remaining data entries.

[0045] The providing module 324 can be configured to provide multiple external services from multiple external service providers when multiple external services are requested based on a predetermined amount of generated priority data and a received first input. The predetermined amount of priority data and the received first input can be provided to a neural network 506 of system 204. In an embodiment, the neural network 506 may include a large language model. The neural network 506 can process the predetermined amount of priority data and the received first input, and generate multiple external services corresponding to a user-customized context based on this processing. The providing module 324 can receive multiple external services. The providing module 324 can be configured to provide multiple external services from multiple external service providers when multiple external services are requested. The multiple external services may indicate multiple actions associated with the first input (e.g., weather updates or navigation).

[0046] The determination module 320 can be configured to determine a first context for device 202 based on the fusion of multiple external services, multiple predefined custom data, and multi-device environment (MDE) data provided in the context builder 510 of the coordinator unit 407. The context builder 510 can obtain user-defined data and MDE data from at least one vector database and collect multiple external services provided. The context builder 510 can use the multiple external services, user-defined data, and MDE data provided to generate the first context. In an embodiment, the determined first context may include a 360-degree analysis of the user's lifestyle around a first input. In an embodiment, without departing from the scope of this disclosure, for example, multiple predefined custom data may include, but is not limited to, information about alarms, events, tasks, locations, usage patterns, or time contexts. In an embodiment, without departing from the scope of this disclosure, for example, MDE data may include, but is not limited to, automatic appliances, service usage history, patterns, or correlations.

[0047] Based on the determined first context, a first execution plan and one or more second execution plans can be generated in the coordinator unit 407. According to embodiments of this disclosure, the execution plan may be in the form of a scheduling diagram. (Refer to subsequent paragraphs) Figures 3 to 5D This section provides a detailed explanation of the operations performed to generate a first execution plan and one or more second execution plans.

[0048] Reference Figure 3 , Figure 4 and Figure 5B In an embodiment, the generation module 316 may be configured to generate a first execution plan and one or more second execution plans based on a determined first context. The first execution plan may indicate one or more tasks to be performed in response to the determined first context of the first input. In an embodiment, the first execution plan may indicate one or more tasks to be performed based on a voice command from the first input. One or more second execution plans may indicate one or more tasks to be performed in response to the determined first context of the first input. In an embodiment, one or more second execution plans may indicate one or more tasks predicted to be performed within the determined first context of the first input.

[0049] In an embodiment, the fusion module 326 may be configured to fuse the determined first context and the result of a query corresponding to the received first input. In an embodiment, without departing from the scope of this disclosure, the query may be, but is not limited to, a cypher query. The first input may be received by a cypher generator 514. The cypher generator 514 may generate a query based on the first input. The query may be provided to a natural language to semantic network converter 516. The semantic network / knowledge graph may be configured to provide multiple predefined activities in the natural language to semantic network converter 516 based on the query and the first input. A user may perform multiple predefined activities based on the first input and the query. The natural language to semantic network converter 516 may be configured to process the query based on multiple predefined activities to generate the result of the query. The fusion module 326 may be configured to fuse the determined first context and the result of a query corresponding to the received first input. In an embodiment, without departing from the scope of this disclosure, the determined first context provides unstructured information, and the result of the query provides a structured query. In an embodiment, the knowledge graph may enhance the operation of system 204 by providing a structured yet flexible way to manage domain knowledge and user interactions. The knowledge graph provides permissible utterance sequences and related domain connections. For the utterance “Set an alarm at 5 AM tomorrow to go to ABC place”, it can be (:Alarm)-[:RELATED_TO]->(:Travel)”.

[0050] The fused, determined first context and the query results can be provided to neural network 522 along with multiple instructions / cues for further processing. In embodiments, for example, the fused, determined first context and query results may include comprehensive user context, domain connections, and relevant domain utterance sequences. In embodiments, without departing from the scope of this disclosure, neural network 522 may be a timeline-based artificial intelligence multiple model. In embodiments, neural network 522 may be configured to generate multiple future activities to be performed corresponding to the received first input based on multiple instructions / cues. For example, the multiple future activities may be performed by the user's smart device or service, thereby providing a comprehensive prediction of future activities.

[0051] Extraction module 328 can be configured to extract multiple future activities to be executed corresponding to the received first input, and multiple categories corresponding to multiple predefined activities. Extraction module 328 can be configured to extract the multiple future activities and multiple categories based on fusion using predefined techniques. Predefined techniques may include, but are not limited to, at least one of fine-tuning techniques, adapter techniques, and rag techniques. Generation module 316 can be configured to generate a first execution plan and one or more second execution plans based on the extracted multiple future activities and multiple categories.

[0052] In the embodiments, reference is made to Figure 5C The first execution plan and one or more second execution plans can be specifically defined as one of the following: any capability workflow or dependency resolution workflow of session device 202, domain capability mapping with different instantiation strategies, input validation, error handling scenario implementation, or planning for task execution through actions with input action definitions and output action definitions. The first execution plan and one or more second execution plans can also be defined as endpoints connected to an application programming interface (API) of an external server to accept action inputs and return expected outputs, actions, input / output concepts, types, to define dialogues, define selection rules, and input / output validation.

[0053] Reference Figure 5D A first execution plan and one or more second execution plans can be created all at once after the first captured input is executed. The first execution plan and the one or more second execution plans can be interdependent, where the current execution plan may depend on the execution plan generated by previous statements / use cases. Because use cases can be modified based on execution results and adaptive context information, one or more second execution plans can also be dynamically generated to incorporate existing first execution plans. The generated first execution plan and one or more second execution plans are adaptive and context-dependent.

[0054] In the embodiments, reference is made to Figure 4 and Figure 5B The separation module 318, together with the response optimizer 524, can be configured to separate a first execution plan and one or more second execution plans into multiple predetermined groups. These predetermined groups may include an active multi-domain task group, a personalized dynamic recommendation group, and a multi-device assistance group. This configuration customizes the output format (e.g., a first execution plan and one or more second execution plans for seamless integration). In an embodiment, the active multi-domain task group may include tasks / future activities that can be prompted to the user by system 204 for execution. Here, the trigger may originate from system 204 rather than the user. In an embodiment, the personalized dynamic recommendation group may indicate recommendations that can be displayed to the user as prompts, and further, the user may decide on the trigger action. In an embodiment, the multi-device assistance group may indicate activities that can be associated with connected smart home devices.

[0055] Reference Figure 3 , Figure 4 and Figure 6 In an embodiment, the generation module 316 may be configured to generate a first timeline connecting a first execution plan and one or more second execution plans. Without departing from the scope of this disclosure, the operations performed to generate the first timeline may be executed in the timeline aggregator unit 418 of system 204.

[0056] In an embodiment, aggregation module 330 may be configured to aggregate the first execution plan and one or more second execution plans after separating them. Aggregation module 330 may be configured to aggregate the first execution plan and one or more second execution plans. Aggregation module 330 may be configured to perform aggregation based on multiple parameters by aggregator cache 602. For example, the multiple parameters may include location, time, type of device 202, or state of device 202. Sending module 331 may be configured to send the aggregated first execution plan and one or more second execution plans sequentially to dynamic execution plan generator 604, wherein dynamic execution plan generator 604 is located in dynamic execution plan unit 420 connected to the timeline of system 204. The aggregation process satisfies all conditions, such as a single user intent being decomposed into a multi-intent execution system 204.

[0057] Processing module 332 can be configured to process the sent first execution plan and one or more second execution plans, along with multiple predetermined factors, via neural network 614 in dynamic execution plan generator 604. These predetermined factors may include, but are not limited to, priority strategies, parameter dependency trackers, and text mappings. In embodiments, neural network 614 can be configured to process and extract meaningful information from the execution plans of various activities. Neural network 614 can employ a variety of different techniques (e.g., tokenization, named entity recognition, and dependency resolution) to process and interpret the sent first execution plan and one or more second execution plans. This network 614 essentially bridges the gap between raw data and actionable insights, contributing to effective decision-making and strategic planning.

[0058] Reference Figure 6 and Figure 7 After being processed by neural network 614, the first execution plan and one or more second execution plans, along with multiple predetermined factors, can be sent to dependency mapper 616 for further processing. Dependency mapper 616, equipped with information extraction structuring unit 702 and symbolic reasoning engine 704, can be configured to analyze the interdependencies between the first execution plan and one or more second execution plans. Information extraction structuring unit 702 can be configured to extract and identify information (such as actions, objects, and entities) from the first execution plan and one or more second execution plans. Information extraction structuring unit 702 can be configured to structure the extracted information in a manner that facilitates reasoning and understanding of each execution plan in the first execution plan and one or more second execution plans. For example, when information extraction structuring unit 702 is fed "nodes" containing actions (such as "navigation" or "finding") and associated locations or objects, it returns an organized array of "activities," categorizing actions into "types" and specifying corresponding locations or targets.

[0059] Symbolic reasoning engine 704 can be configured to establish connections between a first execution plan and one or more second execution plans. Symbolic reasoning engine 704 can be a neurosymbolic artificial intelligence (AI) module that applies various rules to structured data. Engine 704 receives structured data from information extraction and structuring unit 702 and applies rules (e.g., temporal dependencies, causal relationships, and contextual grouping) to find relationships between the first execution plan and one or more second execution plans. Engine 704 transforms the input into a more refined output by identifying connections based on the applied rules. Engine 704 enables system 204 to understand the context and dependencies between the first execution plan and one or more second execution plans, thereby achieving a more efficient and logical decision-making process.

[0060] A first execution plan and one or more second execution plans can be sent to a connected execution plan generator 618. The connected execution plan generator 618 can obtain structured data from the symbolic inference engine 704 and integrate the first execution plan and one or more second execution plans into a single connected execution plan. The generator 618 uses relationships and connections between activities to generate this unified plan. The generator 618's inputs include "activities" and "applied rules," and therefore outputs the interconnected first execution plan and one or more second execution plans. Thus, depending on the connected execution plan generator 618, the generation module 316 can be configured to generate the interconnected first execution plan and one or more second execution plans based on scheduler flags and a first timeline of the interconnected first execution plan and one or more second execution plans.

[0061] In an embodiment, the separation module 318 may be configured to separate an interconnected first execution plan and one or more second execution plans based on a plurality of predetermined segments. The plurality of predetermined segments may include a context monitoring service segment / context monitoring segment 620, a status monitoring service segment / status monitoring segment 622, an execution plan validator segment 624, and an execution scheduler segment 626. In an embodiment, the context monitoring service segment 620 may facilitate the acquisition of runtime context data or the continuous observation and analysis of the context environment associated with the interconnected first execution plan and one or more second execution plans. The status monitoring service segment 622 may facilitate the continuous observation and analysis of the status of the interconnected first execution plan and one or more second execution plans. The execution plan validator segment 624 may facilitate the correctness of the interconnected first execution plan and one or more second execution plans. The execution scheduler segment 626 may facilitate the scheduling of the interconnected first execution plan and one or more second execution plans. In the event of any changes / updates in the interconnected first execution plan and one or more second execution plans, the changes / updates can be reverted to the dynamic execution plan generator 604 for further processing, and the execution scheduler segment 626 can indicate the scheduling of the changes / updates.

[0062] In embodiments, without departing from the scope of this disclosure, the separation of interconnected first execution plans and one or more second execution plans can be performed in the timeline monitoring and scheduler unit 422 of system 204. In embodiments, without departing from the scope of this disclosure, the timeline monitoring and scheduler 422 can ensure that events, tasks, or plans are tracked over time and that execution scheduling is managed.

[0063] Analysis module 334 can be configured to analyze the verification of a first execution plan and one or more second execution plans of a discrete interconnect based on the user's real-time state and a determined first context. In an embodiment, without departing from the scope of this disclosure, the analysis of the verification of the first execution plan and one or more second execution plans of the discrete interconnect can be performed in the context event execution unit 424 of system 204.

[0064] Execution module 335 can be configured to execute a first execution plan and one or more second execution plans on at least one user device 628, based on a separate interconnection. Execution module 335 can be configured to execute when the first execution plan and one or more second execution plans of the separate interconnection are verified. The first execution plan and one or more second execution plans of the separate interconnection can be marked as expired, and the first execution plan and one or more second execution plans of the separate interconnection are invalid. Without departing from the scope of this disclosure, for example, at least one user device 628 may include, but is not limited to, a smartphone, laptop computer, watch, wearable device, hearing device, television, or refrigerator. The above operations can be performed in the context event execution unit 424 of system 204. For example, the context event execution unit 424 can facilitate interaction with different user devices to execute the first execution plan and one or more second execution plans of the separate interconnection on at least one user device 628 based on a first timeline, such as scheduling. The context event execution unit 424 can send information to the user's user device 628 to cause the user device 628 to execute at least one task of the first execution plan, an updated execution plan, or one or more second execution plans. One or more tasks or one or more execution plans based on a first timeline can be executed on at least one user device 628.

[0065] In an embodiment, the detection module 336 may be configured to detect changes from the first context of device 202 based on at least one of a generated first timeline and a second context of device 202 or a second context of a user. In an embodiment, the detection module 336 may be configured to detect changes from a determined first context of device 202. The detection module 336 may be configured to detect changes via context monitoring service technology based on associating the generated first timeline with the second context of device 202 and / or the second context of a user. The detection module 336 may be configured to detect changes in multiple variables associated with the generated first execution plan and one or more second execution plans. The detection module 336 may be configured to detect changes via state monitoring service technology or execution plan validator technology based on associating the generated first timeline with the second context of device 202 and / or the second context of a user. The multiple variables may include the initial state of at least one user device 628, the type of at least one user device 628, and the plan provided in the generated first execution plan and one or more second execution plans.

[0066] In an embodiment, the generation module 316 may also be configured to generate an updated execution plan based on detected changes. The updated execution plan may indicate one or more third tasks to be performed in response to the detected changes. The updated execution plan may indicate one or more third tasks to be performed in response to a second context of device 202 and / or a second context of the user. In an embodiment, the updated execution plan indicates an extension of one or more second execution plans or a replacement of one or more second execution plans.

[0067] In embodiments, the generation module 316 may also be configured to generate a second timeline by modifying or updating the first timeline based on an updated execution plan. In embodiments, the second timeline may be generated based on modifying or updating at least one of the first execution plan or one or more second execution plans using an updated execution plan. The second timeline may be generated based on modifying or updating one or more second execution plans connected to the first execution plan using an updated execution plan. The second timeline may be generated based on connecting the updated execution plan with the first execution plan and one or more second execution plans. In embodiments, without departing from the scope of this disclosure, the execution plan may be the first execution plan or an already generated execution plan.

[0068] For example, the second timeline can be output to the user via a display or speaker through input / output (I / O) interface 338. Output via the display may include a graphical representation of the second timeline. Output via the speaker may include audio generated based on a text-to-speech engine or model that produces synthesized speech audio based on a text representation of the second timeline. One or more tasks or execution plans based on the second timeline can be executed on at least one user device 628.

[0069] Figure 8 A flowchart depicting a method 800 for managing an execution plan for device 202 according to an embodiment of the present disclosure is shown. Method 800 includes... Figure 8 The series of operations shown in steps 802 to 814. Method 800 can be executed by system 204 in conjunction with module 312, in conjunction with Figures 3 to 7 The details are explained, and for the sake of brevity in this disclosure, will not be repeated here. Method 800 begins with step 802.

[0070] In step 802, method 800 includes: receiving a first input from a user indicating a voice command.

[0071] Method 800 includes: generating a text format corresponding to the received first input by converting the received first input into a text format via at least one of predefined techniques. At least one of the predefined techniques may be implemented via an Automatic Speech Recognition (ASR) model 406. Method 800 also includes: separating the generated text format into multiple domains by classifying the generated text format via at least one of the predefined techniques. At least one of the predefined techniques may be implemented via a Natural Level Understanding (NLU) model 404.

[0072] In step 804, method 800 includes: determining a first context associated with the AI-based assistive device 202 and the user intent based on a first input. Method 800 includes: generating a word embedding corresponding to the received first input by converting a separated text format corresponding to the received first input into a word embedding. Method 800 includes: identifying multiple pieces of data corresponding to the received first input based on a vector search of the generated word embeddings in a predefined vector database. The multiple pieces of data may indicate multiple services associated with the first input. Method 800 includes: generating a predetermined number of priority data corresponding to the received first input from the identified multiple pieces of data by reordering them based on the received first input. The predetermined number of priority data may indicate data with the highest priority compared to the remaining data in the multiple pieces of data. Method 800 includes: providing multiple external services from multiple external service providers when requesting multiple external services based on the generated predetermined number of priority data and the received first input. The multiple external services may indicate multiple operations associated with the first input. Method 800 includes determining a first context for device 202 based on the fusion of multiple external services, multiple pre-defined custom data, and multi-device environment (MDE) data.

[0073] In step 806, method 800 includes: generating a first execution plan based on a determined first context, and generating one or more second execution plans based on the determined first context. The first execution plan indicates one or more tasks to be executed in response to the determined first context from a first input. The one or more second execution plans may indicate one or more tasks to be executed in response to the determined first context from a first input.

[0074] Method 800 includes: fusing a determined first context and the result of a query corresponding to a received first input. Method 800 includes: extracting, based on the fusion, multiple future activities to be executed corresponding to the received first input and multiple categories corresponding to the multiple predefined activities using predefined techniques. The predefined techniques may include at least one of fine-tuning techniques, adapter techniques, and rag techniques. Method 800 includes: generating a first execution plan and one or more second execution plans based on the extracted multiple future activities and multiple categories.

[0075] Method 800 includes separating a first execution plan and one or more second execution plans into multiple predetermined groups. The multiple predetermined groups may include an active multi-domain task group, a personalized dynamic recommendation group, and a multi-device assistance group.

[0076] In step 808, method 800 includes: generating a first timeline that connects a first execution plan and one or more second execution plans.

[0077] Method 800 includes: after separating a first execution plan and one or more second execution plans, aggregating the first execution plan and one or more second execution plans by an aggregator cache 602 based on multiple parameters. The multiple parameters may include position and time. Method 800 includes: sequentially sending the aggregated first execution plan and one or more second execution plans to a dynamic execution plan generator 604. Method 800 includes: processing the sent first execution plan and one or more second execution plans, along with multiple predetermined factors, through a neural network 614 in the dynamic execution plan generator 604. The multiple predetermined factors may include a priority strategy, a parameter dependency tracker, and a text mapping. Method 800 includes: generating interconnected first execution plans and one or more second execution plans, and a first timeline connecting the interconnected first execution plans and one or more second execution plans.

[0078] Method 800 includes: separating a first execution plan and one or more second execution plans of an interconnect based on a plurality of predetermined segments. The plurality of predetermined segments may include a context monitoring service segment 620, a state monitoring service segment 622, an execution plan validator segment 624, and an execution scheduler segment 626. Method 800 includes: analyzing the verification of the separated first execution plan and one or more second execution plans of the interconnect based on the user's real-time state and a determined first context. Method 800 includes: executing the separated first execution plan and one or more second execution plans of the interconnect on at least one of user devices 628 when the separated first execution plan and one or more second execution plans of the interconnect are verified. At least one of the user devices 628 may include a smartphone, a laptop computer, and a watch.

[0079] In step 810, method 800 includes: detecting changes from the determined first context of device 202 based on at least one of the generated first timeline and a second context of device 202 or a second context of the user.

[0080] Method 800 includes: detecting changes from a determined first context of device 202 based on associating a generated first timeline with a second context of device 202 and / or a second context of a user, via context monitoring service technology. Method 800 also includes: detecting changes in multiple variables associated with a generated first execution plan and one or more second execution plans based on associating the generated first timeline with a second context of device 202 and / or a second context of a user, via state monitoring service technology or execution plan validator technology. The multiple variables may include the initial state of at least one of user devices 628, the type of at least one of user devices 628, and the plan provided in the generated first execution plan and one or more second execution plans.

[0081] In step 812, method 800 includes: generating an updated execution plan based on the detected changes. The updated execution plan may indicate one or more third tasks to be performed in response to the detected changes. The updated execution plan may indicate one or more third tasks to be performed in response to a second context of device 202 and / or a second context of the user. In embodiments, the updated execution plan may indicate an extension of one or more second execution plans or a replacement of one or more second execution plans.

[0082] In step 814, method 800 includes generating a second timeline by modifying or updating a first timeline based on an updated execution plan. In embodiments, the second timeline may be generated based on modifying or updating at least one of a first execution plan or one or more second execution plans using an updated execution plan. The second timeline may be generated based on modifying or updating one or more second execution plans connected to the first execution plan using an updated execution plan. The second timeline may be generated based on connecting an updated execution plan to the first execution plan and one or more second execution plans.

[0083] Method 800 may further include: sending information to the user's user equipment 628 to cause the user equipment 628 to execute at least one task of a first execution plan, an updated execution plan, and one or more second execution plans.

[0084] Figure 9An example timeline representation of a use case scenario implementation of a system 204 and method 800 for generating a first execution plan based on a first input from a user, according to embodiments of this disclosure, is shown. As depicted in the figure, the user provides a command (e.g., "voice input for setting an alarm for 5 a.m. tomorrow to go to location XYZ"). Then, as... Figure 9 As shown, the method 800 and system 204 disclosed herein determine the context from a first input and generate a timeline with various tasks and recommendations to assist the user without any further input or commands from the user. The method 800 and system 204 may take into account various dynamics associated with user commands; for example, if tomorrow is a rainy day, the method 800 and system 204 may also include suggesting to the user whether he / she wants to postpone his / her trip to location XYZ. In one embodiment, for example, the method 800 and system 204 disclosed herein may generate tasks and recommendations based on usage patterns and dynamics, such as smart water heater and air conditioning (AC) operation.

[0085] Figure 10A and Figure 10B Another example timeline representation of a use case scenario implementation of method 800 and system 204 according to embodiments of this disclosure is shown, to dynamically generate task suggestions based on a first input from a user and to execute a task based on the dynamically generated task suggestions. As depicted in the figure, the user provides a command (e.g., “Remind me to visit W's place at 7 pm tomorrow”). Then, as... Figure 10A As shown, the methods 800 and system 204 disclosed herein determine the context from the first input and generate a timeline with various tasks and recommendations to assist the user without any further input or commands from the user. For example, methods 800 and system 204 may generate a timeline based on data associated with the user to plan for tomorrow accordingly by retrieving the context that tomorrow is W's birthday from various data sources, including all possible information and events related to W (such as birthdays or anniversaries). Methods 800 and system 204 disclosed herein may also consider overlapping events and schedules around the time frame of the first input, and intend to consider such overlapping events. Figure 10B As shown, in the scenario, a new event (such as a venue change), for example, "changing the venue for the 7 PM event to ABC Resort," is detected. The disclosed method 800 and system 204 dynamically adjust the timeline according to the change to assist the user without requiring any further user input. The timeline can also consider any pre-planned or overlapping events (such as meetings), which can be identified based on user-related data from various sources and devices when generating timelines and suggestions for the user. The provided descriptions are intended for informational purposes, and any interpretation or application of the results should take into account the context and details of the test scenario.

[0086] This disclosure ensures technological advancements in the fields of assistive technologies and recommender systems. The technical advantage of the method disclosed herein lies in its ability to intelligently predict a series of tasks a user wants to perform, thereby managing the execution plan for device 202. This involves the dynamic generation of a first execution plan and one or more second execution plans across multiple intentions, domains, and devices, as well as timeline creation, which generates timelines of tasks, recommendations, and assistive commands tailored to the user's needs and context. These generated timeline branches adjust and expand as activities progress, ensuring a comprehensive approach to task management. The disclosed system 204 and method 800 ensure the dynamic connection of the first execution plan and one or more second execution plans based on contextual dependencies, priorities, and the order in which execution occurs, thereby ensuring the prediction of multiple future activities based on user input and providing a personalized and proactive user experience. Furthermore, system 204 dynamically adjusts / modifies tasks and schedules based on relevance information in the multi-device environment (MDE) and incoming timeline activities, ensuring adaptability and efficiency. This runtime adaptive timeline generation considers user scenarios and preferences, resulting in a personalized and proactive user experience. In summary, this 360-degree AI framework manages the execution plan for device 202, providing end-to-end support to users by taking into account all angles and dynamics, thereby offering customized initiative, assistance, and suggestions throughout the user's journey.

[0087] Previously known solutions did not take into account user-related environmental factors (such as user intent, the domain of the event, and data from multiple devices), and therefore could not adapt and scale as the activity progressed or was performed. These methods could not perceive, adapt, and execute based on the correlation between different tasks and activities that the user might perform or that need to be performed according to changes in user preferences or specific events, which is overcome by the system 204 and method 800 disclosed in this disclosure.

[0088] According to embodiments of this disclosure, a method for managing an execution plan for an artificial intelligence (AI)-based assistive device may include: receiving a first input via a microphone indicating a voice command from a user. The method may include: determining a first context associated with the AI-based assistive device and the user's intent based on the first input. The method may include: generating a first execution plan based on the first context, and generating one or more second execution plans based on the first context, wherein the first execution plan indicates one or more first tasks to be performed in response to the first context, and the one or more second execution plans indicate one or more second tasks to be performed in response to the first context. The method may include: generating a first timeline connecting the first execution plan and the one or more second execution plans. The method may include: detecting changes in the first context based on the first timeline and at least one of a second context of the AI-based assistive device or a second context of the user. The method may include: generating an updated execution plan based on the changes. The method may include: generating a second timeline by modifying the first timeline based on the updated execution plan.

[0089] According to embodiments of this disclosure, before determining the first context, the method may include: generating a text format corresponding to the first input by converting the first input into text based on an Automatic Speech Recognition (ASR) model. The method may also include: separating the text format into multiple domains by classifying the text format based on a Natural Level Understanding (NLU) model.

[0090] According to embodiments of this disclosure, determining the first context may include: generating a word embedding corresponding to the first input by converting a separate text format corresponding to the first input into a word embedding. Determining the first context may include: identifying multiple data points corresponding to the first input based on a vector search of the word embeddings in a predefined vector database, wherein the multiple data points indicate multiple services associated with the first input. Determining the first context may include: generating a predetermined number of priority data points corresponding to the first input from the multiple data points by reordering them based on the first input, wherein the predetermined number of priority data points indicate one or more data points that have the highest priority compared to the remaining data points. Determining the first context may include: when requesting multiple external services based on the predetermined number of priority data points and the first input, providing information corresponding to multiple external services from multiple external service providers, wherein the information corresponding to the multiple external services indicates multiple operations associated with the first input. Determining the first context may include: determining the first context by fusing information corresponding to multiple external services, multiple predetermined custom data points, and multi-device environment (MDE) data.

[0091] According to embodiments of this disclosure, generating a first execution plan and one or more second execution plans may include: fusing a first context and the results of a query corresponding to a first input. Generating the first execution plan and one or more second execution plans may include: extracting activity information corresponding to multiple future activities to be executed based on at least one of fine-tuning techniques, adapter techniques, or rag techniques, wherein the activity information corresponds to the first input and multiple categories of predefined activities. Generating the first execution plan and one or more second execution plans may include: generating the first execution plan and one or more second execution plans based on the activity information and multiple categories.

[0092] According to embodiments of this disclosure, the method may further include: separating a first execution plan and one or more second execution plans into multiple predetermined groups. The multiple predetermined groups may include an active multi-domain task group, a personalized dynamic recommendation group, and a multi-device assistance group.

[0093] According to embodiments of this disclosure, the operation of generating a first timeline may include: after separating a first execution plan and one or more second execution plans into multiple predetermined groups, aggregating the first execution plan and one or more second execution plans based on multiple parameters via an aggregator cache, wherein the multiple parameters include position and time. The operation of generating the first timeline may include: sequentially sending the aggregated first execution plan and one or more second execution plans to a dynamic execution plan generator.

[0094] According to embodiments of this disclosure, the operation of generating a first timeline may include: processing an aggregated first execution plan and one or more second execution plans via a neural network of a dynamic execution plan generator based on multiple predetermined factors including a priority strategy, a parameter dependency tracker, and a text mapping. The operation of generating the first timeline may also include: generating interconnected first execution plans and one or more second execution plans, and a first timeline connecting the interconnected first execution plans and one or more second execution plans.

[0095] According to embodiments of this disclosure, the method may include: separating an interconnected first execution plan and one or more second execution plans based on a plurality of predetermined segments, wherein the plurality of predetermined segments may include a context monitoring service segment, a state monitoring service segment, an execution plan validator segment, and an execution scheduler segment. The method may include: analyzing the verification of the separated interconnected first execution plan and one or more second execution plans based on the user's real-time state and a first context. The method may include: based on the verification of the separated interconnected first execution plan and one or more second execution plans, sending information to the user's user equipment to cause the user equipment to execute the separated interconnected first execution plan and one or more second execution plans on the user equipment.

[0096] According to embodiments of this disclosure, the operation of detecting changes may include: detecting changes from a first context based on the association of a first timeline with at least one of a second context or a user's second context via a context monitoring service. The operation of detecting changes may also include: detecting changes in a plurality of variables associated with a first execution plan and one or more second execution plans based on the association of a first timeline with a second context or a user's second context via a state monitoring service or an execution plan validator. The plurality of variables may include the initial state of at least one user device, the type of at least one user device, and the plan provided in at least one of the first execution plan and one or more second execution plans.

[0097] According to embodiments of this disclosure, an updated execution plan may indicate an extension of one or more second execution plans or a replacement of one or more second execution plans.

[0098] According to embodiments of this disclosure, a system for managing execution plans for an artificial intelligence (AI)-based assistive device is provided. The system includes a memory storing instructions and at least one processor communicating with the memory. When executed by the at least one processor, the instructions cause the system to: receive a first input via a microphone indicating a voice command from a user. When executed by the at least one processor, the instructions cause the system to: determine a first context associated with the AI-based assistive device and the user's intent based on the first input. When executed by the at least one processor, the instructions cause the system to: generate a first execution plan based on the first context, and generate one or more second execution plans based on the first context, wherein the first execution plan indicates one or more first tasks to be performed in response to the first context, and the one or more second execution plans indicate one or more second tasks to be performed in response to the first context. When executed by the at least one processor, the instructions cause the system to: generate a first timeline connecting the first execution plan and the one or more second execution plans. When executed by the at least one processor, the instructions cause the system to: detect changes from the first context based on the first timeline and at least one of a second context of the AI-based assistive device or a second context of the user. When executed by at least one processor, the instructions cause the system to: generate an updated execution plan based on changes. When executed by at least one processor, the instructions also cause the system to: generate a second timeline by modifying a first timeline based on the updated execution plan.

[0099] According to embodiments of this disclosure, when executed by at least one processor, the instructions enable the system to perform the following operations: generate a text format corresponding to the first input by converting the first input into text based on an Automatic Speech Recognition (ASR) model. When executed by at least one processor, the instructions enable the system to perform the following operations: separate the text format into multiple domains by classifying the text format based on a Natural Level Understanding (NLU) model.

[0100] According to embodiments of this disclosure, when executed by at least one processor, the instructions cause the system to perform the following operations: First, generate a word embedding corresponding to the first input by converting a separated text format corresponding to the first input into a word embedding. Second, when executed by at least one processor, the instructions cause the system to perform the following operations: First, identify multiple data points corresponding to the first input based on a vector search of the word embeddings in a predefined vector database, wherein the multiple data points indicate multiple services associated with the first input. Third, when executed by at least one processor, the instructions cause the system to perform the following operations: First, generate a predetermined number of priority data points corresponding to the first input from the multiple data points by reordering the multiple data points based on the first input, wherein the predetermined number of priority data points indicate one or more data points that have the highest priority compared to the remaining data points. Fourth, when executed by at least one processor, the instructions cause the system to perform the following operations: When requesting multiple external services based on the predetermined number of priority data points and the first input, provide information corresponding to multiple external services from multiple external service providers, wherein the information corresponding to the multiple external services indicates multiple operations associated with the first input. When executed by at least one processor, the instruction enables the system to determine a first context based on the fusion of information corresponding to multiple external services, multiple predefined custom data, and multi-device environment (MDE) data.

[0101] According to embodiments of this disclosure, when executed by at least one processor, the instructions cause the system to perform the following operations: merging a first context and the result of a query corresponding to a first input. When executed by at least one processor, the instructions cause the system to perform the following operations: extracting activity information corresponding to a plurality of future activities to be executed based on at least one of fine-tuning techniques, adapter techniques, or rag techniques, wherein the activity information corresponds to the first input and a plurality of predefined activities of various categories. When executed by at least one processor, the instructions cause the system to perform the following operations: generating a first execution plan and one or more second execution plans based on the activity information and the plurality of categories.

[0102] According to embodiments of this disclosure, when executed by at least one processor, the instructions cause the system to perform the following operation: separate a first execution plan and one or more second execution plans into multiple predetermined groups. The multiple predetermined groups may include an active multi-domain task group, a personalized dynamic recommendation group, and a multi-device assistance group.

[0103] According to embodiments of this disclosure, when executed by at least one processor, the instructions cause the system to perform the following operations: after separating a first execution plan and one or more second execution plans into multiple predetermined groups, aggregate the first execution plan and one or more second execution plans via an aggregator cache based on multiple parameters, wherein the multiple parameters include position and time. When executed by at least one processor, the instructions cause the system to perform the following operations: sequentially send the aggregated first execution plan and one or more second execution plans to a dynamic execution plan generator.

[0104] According to embodiments of this disclosure, when executed by at least one processor, the instructions cause the system to: process an aggregated first execution plan and one or more second execution plans via a neural network of a dynamic execution plan generator based on multiple predetermined factors including a priority strategy, a parameter dependency tracker, and a text map. When executed by at least one processor, the instructions also cause the system to: generate interconnected first execution plans and one or more second execution plans, and a first timeline connecting the interconnected first execution plans and one or more second execution plans.

[0105] According to embodiments of this disclosure, when executed by at least one processor, the instructions cause the system to: separate an interconnected first execution plan and one or more second execution plans based on a plurality of predetermined segments, wherein the plurality of predetermined segments may include a context monitoring service segment, a state monitoring service segment, an execution plan validator segment, and an execution scheduler segment. When executed by at least one processor, the instructions cause the system to: analyze and verify the separated interconnected first execution plan and one or more second execution plans based on the user's real-time state and a first context. When executed by at least one processor, the instructions cause the system to: based on the verification of the separated interconnected first execution plan and one or more second execution plans, send information to the user's user equipment to cause the user equipment to execute the separated interconnected first execution plan and one or more second execution plans on the user equipment.

[0106] According to embodiments of this disclosure, when executed by at least one processor, the instructions cause the system to: detect changes from a first context based on an association between a first timeline and at least one of a second context or a user's second context, via a context monitoring service. When executed by at least one processor, the instructions cause the system to: detect changes in a plurality of variables associated with a first execution plan and one or more second execution plans based on an association between a first timeline and a second context or a user's second context, via a state monitoring service or an execution plan verifier. The plurality of variables may include the initial state of at least one user device, the type of at least one user device, and the plan provided in at least one of the first execution plan and one or more second execution plans.

[0107] According to embodiments of this disclosure, an updated execution plan may indicate an extension of one or more second execution plans or a replacement of one or more second execution plans.

[0108] According to embodiments of this disclosure, a non-transitory computer-readable storage medium for storing instructions is provided. The instructions are executable by at least one processor, causing the at least one processor to: receive a first input via a microphone indicating a voice command from a user. When executed by at least one processor, the instructions cause the at least one processor to: determine a first context associated with an AI-based assistive device and a user intent based on the first input. When executed by at least one processor, the instructions cause the at least one processor to: generate a first execution plan based on the first context, and generate one or more second execution plans based on the first context, wherein the first execution plan indicates one or more first tasks to be performed in response to the first context, and the one or more second execution plans indicate one or more second tasks to be performed in response to the first context. When executed by at least one processor, the instructions cause the at least one processor to: generate a first timeline connecting the first execution plan and the one or more second execution plans. When executed by at least one processor, the instructions cause the at least one processor to: detect changes in the first context based on the first timeline and at least one of a second context of the AI-based assistive device or a second context of the user. When executed by at least one processor, the instructions cause the at least one processor to: generate an updated execution plan based on the changes. When executed by at least one processor, the instruction enables at least one processor to perform the following operation: generate a second timeline by modifying a first timeline based on an updated execution plan.

[0109] Unless otherwise stated, the use of the singular includes the plural, and the use of "or" means "and / or". Furthermore, the use of the terms "comprising" or "having" is not limiting. Any scope described herein will be understood to include all values ​​between endpoints. For example, features of the disclosed embodiments may be combined or rearranged to produce additional embodiments within the scope of this disclosure.

[0110] Although at least one example embodiment has been presented in the foregoing detailed description, it should be understood that variations may exist.

Claims

1. A method (800) for managing an execution plan for an AI-based assistive device (202), the method (800) comprising: Receive the first input (802) indicating a voice command from the user via the microphone; Based on the first input, a first context (804) is determined that is associated with the AI-based assistive device (202) and the user intent. A first execution plan is generated based on the first context, and one or more second execution plans are generated based on the first context, wherein the first execution plan indicates one or more first tasks to be executed in response to the first context, and the one or more second execution plans indicate one or more second tasks to be executed in response to the first context (806). Generate a first timeline connecting the first execution plan and the one or more second execution plans (808). Changes are detected from the first context based on at least one of the first timeline and the second context of the AI-based assistive device (202) or the second context of the user (810). An updated execution plan (812) is generated based on the changes; and The second timeline is generated by modifying the first timeline based on the updated execution plan (814).

2. The method (800) as described in claim 1, wherein, Determining the first context (804), the method (800) includes: The word embedding corresponding to the first input is generated by converting the separated text format corresponding to the first input into a word embedding. Based on vector search of the word embedding in a predefined vector database, multiple data points corresponding to the first input are identified, wherein the multiple data points indicate multiple services associated with the first input; A predetermined number of priority data items corresponding to the first input are generated from the plurality of data items by reordering the plurality of data items based on the first input, wherein the predetermined number of priority data items indicate one or more data items that have the highest level compared with the remaining data items in the plurality of data items; When requesting multiple external services based on the predetermined number of priority data and the first input, information corresponding to the multiple external services is provided from multiple external service providers, wherein the information corresponding to the multiple external services indicates multiple operations associated with the first input; The first context is determined by fusing information corresponding to the multiple external services, multiple pre-defined custom data, and multi-device environment MDE data.

3. The method (800) as claimed in claim 1 or claim 2, wherein, The operations of generating the first execution plan and the one or more second execution plans (806) include: Merge the first context and the query result corresponding to the first input; Activity information corresponding to multiple future activities to be executed is extracted based on at least one of fine-tuning technology, adapter technology, or rag technology, wherein the activity information corresponds to the first input and multiple categories of predefined activities; and The first execution plan and the one or more second execution plans are generated based on the activity information and the multiple categories.

4. The method (800) as claimed in any one of claims 1 to 3, further comprising: The first execution plan and the one or more second execution plans are separated into multiple predetermined groups, wherein the multiple predetermined groups include an active multi-domain task group, a personalized dynamic recommendation group, and a multi-device assistance group.

5. The method (800) according to any one of claims 1 to 4, wherein, The operation of generating the first timeline (808) includes: After separating the first execution plan and the one or more second execution plans into multiple predetermined groups, the first execution plan and the one or more second execution plans are aggregated via an aggregator cache (602) based on multiple parameters, wherein the multiple parameters include position and time; and Send the aggregated first execution plan and one or more second execution plans in sequence to the dynamic execution plan generator (604).

6. The method (800) according to any one of claims 1 to 5, wherein, The operation of generating the first timeline (808) includes: Based on multiple predetermined factors including a priority strategy, a parameter dependency tracker, and a text mapping, the aggregated first execution plan and one or more second execution plans are processed via a neural network (614) of the dynamic execution plan generator (604); and Generate an interconnected first execution plan and one or more second execution plans, and a first timeline connecting the interconnected first execution plan and one or more second execution plans.

7. The method (800) as claimed in any one of claims 1 to 6, further comprising: A first execution plan and one or more second execution plans are separated and interconnected based on multiple predetermined segments, wherein the multiple predetermined segments include a context monitoring service segment (620), a status monitoring service segment (622), an execution plan verifier segment (624), and an execution scheduler segment (626). Verification of the separated interconnected first execution plan and one or more second execution plans based on the user's real-time state and the first context; and Based on the verification of a first execution plan and one or more second execution plans based on the separate interconnection, information is sent to the user's user equipment (628) to cause the user equipment to execute the first execution plan and one or more second execution plans based on the separate interconnection on the user equipment.

8. A computer-readable storage medium for storing instructions, wherein, When the instruction is executed by at least one processor, the at least one processor causes the at least one processor to perform the following operations: The first input, indicating a voice command from the user, is received via microphone; Based on the first input, a first context associated with the AI-based assistive device (202) and the user intent is determined; A first execution plan is generated based on the first context, and one or more second execution plans are generated based on the first context, wherein the first execution plan indicates one or more first tasks to be executed in response to the first context, and the one or more second execution plans indicate one or more second tasks to be executed in response to the first context; Generate a first timeline connecting the first execution plan and the one or more second execution plans; Changes are detected from the first context based on at least one of the first timeline and the second context of the AI-based assistive device (202) or the second context of the user; An updated execution plan is generated based on the changes; and The second timeline is generated by modifying the first timeline based on the updated execution plan.

9. A system (204) for managing an execution plan for an AI-based assistive device (202), the system (204) comprising: Memory (308), store instructions; At least one processor (304) communicates with the memory (308). When the instruction is executed by the at least one processor (304), it causes the system to perform the following operations: The first input, indicating a voice command from the user, is received via microphone; Based on the first input, a first context associated with the AI-based assistive device (202) and the user intent is determined; A first execution plan is generated based on the first context, and one or more second execution plans are generated based on the first context, wherein the first execution plan indicates one or more first tasks to be executed in response to the first context, and the one or more second execution plans indicate one or more second tasks to be executed in response to the first context; Generate a first timeline connecting the first execution plan and the one or more second execution plans; Changes are detected from the first context based on at least one of the first timeline and the second context of the AI-based assistive device (202) or the second context of the user; An updated execution plan is generated based on the changes; and The second timeline is generated by modifying the first timeline based on the updated execution plan.

10. The system (204) as claimed in claim 9, wherein, When the instructions are executed by the at least one processor (304), the system performs the following operations: The word embedding corresponding to the first input is generated by converting the separated text format corresponding to the first input into a word embedding. Based on vector search of the word embedding in a predefined vector database, multiple data points corresponding to the first input are identified, wherein the multiple data points indicate multiple services associated with the first input; A predetermined number of priority data items corresponding to the first input are generated from the plurality of data items by reordering the plurality of data items based on the first input, wherein the predetermined number of priority data items indicate one or more data items that have the highest level compared with the remaining data items in the plurality of data items; When requesting multiple external services based on the predetermined number of priority data and the first input, information corresponding to the multiple external services is provided from multiple external service providers, wherein the information corresponding to the multiple external services indicates multiple operations associated with the first input; The first context is determined by fusing information corresponding to the multiple external services, multiple pre-defined custom data, and multi-device environment MDE data.

11. The system (204) as claimed in claim 9 or claim 10, wherein, When the instructions are executed by the at least one processor (304), the system performs the following operations: Merge the first context and the query result corresponding to the first input; Activity information corresponding to multiple future activities to be executed is extracted based on at least one of fine-tuning technology, adapter technology, or rag technology, wherein the activity information corresponds to the first input and multiple categories of predefined activities; and The first execution plan and the one or more second execution plans are generated based on the activity information and the multiple categories.

12. The system (204) according to any one of claims 9 to 11, wherein, When the instructions are executed by the at least one processor (304), the system also performs the following operations: The first execution plan and the one or more second execution plans are separated into multiple predetermined groups, wherein the multiple predetermined groups include an active multi-domain task group, a personalized dynamic recommendation group, and a multi-device assistance group.

13. The system (204) according to any one of claims 9 to 12, wherein, When the instructions are executed by the at least one processor (304), the system performs the following operations: After separating the first execution plan and the one or more second execution plans into multiple predetermined groups, the first execution plan and the one or more second execution plans are aggregated via an aggregator cache (602) based on multiple parameters, wherein the multiple parameters include position and time; and Send the aggregated first execution plan and one or more second execution plans in sequence to the dynamic execution plan generator (604).

14. The system (204) according to any one of claims 9 to 13, wherein, When the instructions are executed by the at least one processor (304), the system performs the following operations: Based on multiple predetermined factors including priority strategy, parameter dependency tracker and text mapping, the aggregated first execution plan and one or more second execution plans are processed via the neural network (614) of the dynamic execution plan generator (604); as well as Generate an interconnected first execution plan and one or more second execution plans, and a first timeline connecting the interconnected first execution plan and one or more second execution plans.

15. The system (204) according to any one of claims 9 to 14, wherein, When the instructions are executed by the at least one processor (304), the system performs the following operations: Changes are detected from the first context by means of a context monitoring service, based on the association between the first timeline and at least one of the second context or the user's second context; as well as By means of a state monitoring service or an execution plan verifier, changes in a plurality of variables associated with the first execution plan and the one or more second execution plans are detected based on associating the first timeline with a second context or the second context of the user, wherein the plurality of variables include the initial state of at least one of the user devices, the type of at least one of the user devices, and the plan provided in at least one of the first execution plan and the one or more second execution plans.

Images