Method for managing dynamic database for providing personalized service, and electronic device for performing same

Abstract

This method for managing a dynamic database for providing a personalized service may comprise the steps of: acquiring user input information about an electronic device; acquiring context information related to the personalized service from the user input information, the context information including implicit information determined from the user input information; and updating the dynamic database on the basis of the context information.

Description

Method for managing a dynamic database for providing personalized services and electronic devices for performing the same

[0001] The present disclosure relates to a method for managing a dynamic database for providing personalized services and an electronic device for performing the same.

[0002] In modern society, electronic devices, such as smartphones and other portable terminals, are increasingly diversifying their functionality as demand for personalized services grows. Electronic devices can provide services to users based on predefined information, and databases, such as those for knowledge graph modeling, can be dynamically generated or updated to reflect real-time user input and interactions. However, existing knowledge graphs have limitations due to their complex structures and large amounts of stored information, making it difficult to create or update nodes and edges within the knowledge graph in response to real-time user input.

[0003] According to one aspect of the present disclosure, a method for managing a dynamic database for providing a personalized service may include a step of obtaining user input information for an electronic device, a step of obtaining context information related to the personalized service from the user input information, the context information including implicit information determined from the user input information, and a step of updating the dynamic database based on the context information.

[0004] According to one aspect of the present disclosure, an electronic device includes at least one processor including a memory storing at least one instruction and a processing circuit, wherein the at least one processor executes, alone or in cooperation, a program stored in the memory or at least one instruction, thereby obtaining user input information for the electronic device, obtaining context information related to the personalized service from the user input information, and wherein the context information includes implicit information determined from the user input information, and updating a dynamic database based on the context information.

[0005] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0006] FIG. 1 is a diagram illustrating an overall flow of an electronic device managing a database based on user input information according to one embodiment of the present disclosure.

[0007] FIG. 2 is a diagram illustrating the structure of a knowledge graph based on 4W1R (subject information, event information, location information, time information, and reason information) according to one embodiment of the present disclosure.

[0008] FIG. 3 is a diagram illustrating the structure of a 4W1R-based knowledge graph according to one embodiment of the present disclosure.

[0009] FIG. 4 is a drawing for explaining modules included in an electronic device according to one embodiment of the present disclosure.

[0010] FIG. 5 is a drawing for explaining a hardware configuration included in an electronic device according to one embodiment of the present disclosure.

[0011] FIG. 6 is a diagram illustrating a situation in which a user makes a call through an electronic device according to one embodiment of the present disclosure.

[0012] FIG. 7 and FIG. 8 are diagrams illustrating a knowledge graph generated by an electronic device according to one embodiment of the present disclosure based on the call content of FIG. 6.

[0013] FIG. 9 is a diagram illustrating a situation in which a user makes a call through an electronic device according to one embodiment of the present disclosure.

[0014] FIG. 10 is a diagram illustrating an example of an electronic device according to an embodiment of the present disclosure providing an updated knowledge graph based on the call content of FIG. 9 and an answer to a user's question.

[0015] FIG. 11 is a diagram illustrating a situation in which a user makes a call through an electronic device according to one embodiment of the present disclosure.

[0016] FIG. 12 is a diagram illustrating an example of an electronic device according to an embodiment of the present disclosure providing a knowledge graph updated based on the call content of FIG. 11 and a response to a user's request.

[0017] FIG. 13 is a diagram illustrating a process in which an electronic device stores a schedule based on a message according to one embodiment of the present disclosure.

[0018] FIG. 14 is a diagram illustrating a process in which an electronic device according to one embodiment of the present disclosure performs a search based on chat content and displays search results.

[0019] FIG. 15 is a diagram illustrating a process in which an electronic device according to one embodiment of the present disclosure searches for a schedule based on chat content and displays the search results.

[0020] FIG. 16 is a diagram illustrating a process in which an electronic device automatically generates an answer based on an image according to one embodiment of the present disclosure.

[0021] FIG. 17 is a diagram illustrating a process in which an electronic device, according to one embodiment of the present disclosure, identifies an object pointed by a user based on the context of a screen and responds to a user's request.

[0022] FIG. 18 is a diagram illustrating a process in which an electronic device according to one embodiment of the present disclosure checks an appointment time indicated by a user based on the context of the screen and sets an alarm.

[0023] FIG. 19 is a diagram illustrating a process in which an electronic device according to one embodiment of the present disclosure checks and stores information in chunk units based on the context of the screen.

[0024] FIG. 20 is a diagram illustrating a process in which an electronic device according to one embodiment of the present disclosure controls an air conditioner based on user input.

[0025] FIG. 21 is a diagram illustrating a knowledge graph updated by an electronic device according to a user's operation of controlling an air conditioner according to an embodiment of the present disclosure.

[0026] FIG. 22 is a diagram illustrating a process of an electronic device according to one embodiment of the present disclosure generating a knowledge graph when storing a photograph and searching for the photograph at a user's request.

[0027] FIGS. 23 and 24 are flowcharts illustrating a method for managing a dynamic database for providing personalized services according to embodiments of the present disclosure.

[0028] FIGS. 25 to 28 are flowcharts illustrating a method for providing personalized services based on a dynamic database according to embodiments of the present disclosure.

[0029] FIG. 29 is a diagram illustrating a situation in which a user makes a call through an electronic device according to one embodiment of the present disclosure.

[0030] FIG. 30 is a diagram for explaining a process in which an electronic device according to one embodiment of the present disclosure creates a knowledge graph based on user input information (call content of FIG. 29) collected through a call app, and saves a schedule based on the knowledge graph.

[0031] FIG. 31 is a diagram illustrating a situation in which a user makes a call through an electronic device according to one embodiment of the present disclosure.

[0032] FIG. 32 is a diagram illustrating a knowledge graph generated by an electronic device according to one embodiment of the present disclosure based on user input information (call content of FIG. 31) collected through a call app.

[0033] FIG. 33 is a diagram illustrating a situation in which a user transfers a congratulatory gift requested by the user to another person through an electronic device according to one embodiment of the present disclosure.

[0034] FIG. 34 is a diagram illustrating a knowledge graph generated by an electronic device according to one embodiment of the present disclosure based on user input information (transfer information) collected through a mobile banking app.

[0035] FIG. 35 is a diagram illustrating an example of an electronic device according to one embodiment of the present disclosure classifying and storing context information based on reason information included in the context information.

[0036] FIG. 36 is a diagram illustrating a method for an electronic device according to one embodiment of the present disclosure to obtain context information from user input information collected through multiple apps, and to store and manage the context information based on reason information included in the context information.

[0037] FIG. 37 is a diagram illustrating an example in which an electronic device according to one embodiment of the present disclosure provides a personalized service (to-do list) to a user based on context information linked to reason information.

[0038] FIG. 38 is a flowchart illustrating a method for managing a dynamic database for providing personalized services according to one embodiment of the present disclosure.

[0039] The embodiments described in this disclosure and the configurations illustrated in the drawings are merely exemplary embodiments, and various modifications may be made without departing from the spirit and scope of the present disclosure.

[0040] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0041] The terms described below are defined based on the functions of this disclosure and may vary depending on the intent or custom of the user or operator. These terms should be interpreted based on the contents of this specification.

[0042] Some components in the attached drawings are schematically illustrated. The dimensions of each component do not reflect the actual size. Identical or corresponding components in each drawing are given the same reference numbers.

[0043] The advantages and features of the present disclosure, and methods for achieving them, will become clearer with reference to the embodiments described below in detail with the accompanying drawings. The present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The disclosed embodiments are provided to ensure that the disclosure is complete and to fully inform those skilled in the art of the scope of the disclosure. An embodiment of the present disclosure may be defined according to the claims. Like reference numerals denote like elements throughout the present disclosure.

[0044] In one embodiment of the present disclosure, each block of the flowchart drawings and combinations of the flowchart drawings can be performed by computer program instructions. The computer program instructions can be installed on a processor of a computer or other programmable data processing apparatus, and the instructions executed by the processor of the computer or other programmable data processing apparatus can create means for performing the functions described in the flowchart block(s). The computer program instructions can also be stored in a computer-available or computer-readable memory that can direct a computer or other programmable data processing apparatus to implement the functions in a particular manner, and the instructions stored in the computer-available or computer-readable memory can also produce an article of manufacture that includes instruction means for performing the functions described in the flowchart block(s). The computer program instructions can also be installed on a computer or other programmable data processing apparatus.

[0045] Each block in the flowchart diagram may represent a module, segment, or portion of code that includes one or more executable instructions for performing a specified logical function(s). In one embodiment of the present disclosure, the functions described in the blocks may occur out of order. For example, two blocks depicted in succession may be executed substantially simultaneously or, depending on the function, may be executed in reverse order.

[0046] The term '~ unit' used in one embodiment of the present disclosure may represent a software or hardware component such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC), and the '~ unit' may perform a specific role. The '~ unit' is not limited to software or hardware. The '~ unit' may be configured to be on an addressable storage medium and may be configured to play one or more processors. In one embodiment, the '~ unit' may include components such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided by a specific component or a specific '~ unit' may be combined to reduce the number or separated into additional components. In one embodiment of the present disclosure, the '~ unit' may include one or more processors.

[0047] The meanings of terms used in this disclosure are explained.

[0048] "Personalized service" may refer to a service for a user determined based on the user's use of the electronic device. For example, if a user makes a call using the electronic device, a personalized service may be provided that summarizes and saves the call. For example, if a chat message exchanged between the user and the electronic device includes information about a schedule, a personalized service may be provided that automatically saves the schedule. For example, if a user determines that control of a home appliance or the like is necessary based on the voice input into the electronic device, a personalized service may be provided that automatically controls the device. Electronic devices according to embodiments of the present disclosure may also provide various personalized services, such as route search, automatic response generation, and photo search. Terms such as "customized service" or "service" may be used instead of "personalized service."

[0049] "User input information" may refer to information contained in various forms of user input to an electronic device. "User input information" may also refer to information collected while a user uses an electronic device. "User input information" may refer to information that a user inputs into an electronic device, information about the user's usage patterns, or information about the actions a user performs through the electronic device. For example, text entered by a user through the CLI (Command Line Interface) of an electronic device may be user input information. For example, voice information recorded during a phone call or text information stored during a chat using an electronic device such as a smartphone may be user input information.

[0050] 'User input information' may include not only text or voice input directly by the user, but also text or images displayed on the screen of an electronic device such as a smartphone while the user is using the device (e.g., chat messages or images displayed on the screen, photos included in SNS posts, app execution screens, etc.).

[0051] "User input information" may include not only information directly entered by the user into the electronic device, but also "all information obtained by the electronic device during the user's use of the device." For example, if a user uses an electronic device to chat with another user, "user input information" may include not only the chat messages entered into the electronic device, but also the chat messages received from other users via the electronic device.

[0052] 'User input information' may also include prompts that ask the user a question or request the electronic device to perform an action.

[0053] According to one embodiment of the present disclosure, "user input information" may be multimodal information such as text, images, or audio. Terms such as "input data" or "device usage information" may be used instead of "user input information."

[0054] "Context information" can refer to information used to perform knowledge graph-based reasoning based on various user requests. The type of information included in context information can be determined based on the type of task to be performed through the knowledge graph and the performance of the electronic device (memory capacity, computing speed, etc.). "Context information" can include explicit information and implicit information determined based on explicit information.

[0055] The term "PEFT model" can refer to a fine-tuned model using the Parameter Efficient Fine Tuning (PEFT) technique. PEFT is a fine-tuning technique that efficiently saves computational resources and processing time by updating only a subset of parameters of a pre-trained model, rather than all of them. Terms such as "personalized model" or "user-specific AI model" can be used instead of "PEFT model."

[0056] Embodiments of the present disclosure will be described in detail with reference to the drawings.

[0057] The present disclosure relates to a method for managing a dynamic database for providing personalized services and to an electronic device for performing the same. The present disclosure also relates to information stored in the dynamic database, the structure of the dynamic database, and methods for creating and updating the dynamic database.

[0058] The present disclosure includes a method for providing personalized services based on a dynamic database and an electronic device for performing the same. The present disclosure includes performing knowledge graph-based inference based on user input information (e.g., voice information recorded from a call, text information stored from a chat, prompts asking questions or requesting the performance of specific actions, etc.) and providing personalized services to the user based on the inference results.

[0059] Personalizing electronic devices using neural network models like the Large Language Model (LLM) may require the management (creation and updating) of a database (e.g., a knowledge graph) to store relevant information. Conventional knowledge graphs, however, have complex structures and store a large amount of information, leading to significant memory requirements and high computational costs.

[0060] Existing knowledge graphs have limitations, such as difficulty in real-time updates and implementation on-device. Databases like existing knowledge graphs only store explicit information (e.g., who, what, where, and when—subject information, event information, location information, and time information). Therefore, they are unable to respond to user requests that utilize non-explicit information (e.g., information obtainable through inference).

[0061] To solve the above problems, we present a database structure, a method for managing the database, and a method for performing knowledge graph-based inference based on such a database.

[0062] FIG. 1 is a diagram illustrating an overall flow of an electronic device managing a database based on user input information according to an embodiment of the present disclosure. As illustrated in FIG. 1 , according to embodiments of the present disclosure, the electronic device can extract context information from user input information and store it in a database in a preset structure (e.g., a 4W1R structure to be described later - subject information, event information, location information, time information, and reason information). According to embodiments of the present disclosure, by simplifying the database, it is expected that the memory usage and processing time required when managing (creating and updating) the database or performing knowledge graph-based inference based on the database will be reduced. Managing a database according to embodiments of the present disclosure has the advantages of enabling implementation on-device and real-time updates.

[0063] According to embodiments of the present disclosure, the database may have the characteristics of a 'dynamic database' as it can be updated in real time according to added user input information.

[0064] For detailed explanation, we assume that the database is implemented in the form of a knowledge graph, a widely used form of device personalization. It should be apparent that the embodiments of the present disclosure can be applied to various types of databases, in addition to knowledge graphs.

[0065] According to embodiments of the present disclosure, an electronic device can identify context from a user's input (e.g., various multimodal inputs such as voice, text, and image), and create or update a knowledge graph or provide personalized services based on the identified context.

[0066] First, we explain the information stored in the knowledge graph and the structure of the knowledge graph, and then we explain the process of managing the knowledge graph and providing personalized services based on the knowledge graph.

[0067] 1. Information stored in the knowledge graph (database)

[0068] As illustrated in FIG. 1, according to one embodiment of the present disclosure, an electronic device can acquire contextual information from information collected while a user uses the electronic device, such as user input information, and store the information in a knowledge graph. The electronic device can also determine the context of the user input information and acquire contextual information from the determined context.

[0069] "User input information" can refer to information a user inputs into an electronic device, information about the user's usage patterns, or information about the user's actions performed via the electronic device. For example, text entered by a user through the CLI of an electronic device can be user input information. For example, voice information recorded during a phone call or text information saved from a chat using an electronic device such as a smartphone can be user input information. Since user input information can take various forms, such as text, images, and audio, an electronic device that creates or updates a knowledge graph based on user input information can support multimodality.

[0070] 'Context information' may refer to information for performing knowledge graph-based inference according to various requests of users, and the type of information included in the context information may be determined according to the type of task to be performed through the knowledge graph, the performance of the electronic device (memory capacity, operation speed, etc.), etc. According to one embodiment of the present disclosure, the context information may include explicit information and implicit information that may be determined based on the explicit information. According to one embodiment of the present disclosure, the context information may include information on 4W1R (subject information, event information, location information, time information, reason information) as described below.

[0071] According to one embodiment of the present disclosure, explicit information may include information about 'Who', 'What', 'Where', and 'When' (subject information, event information, location information, temporal information), and implicit information may include information about 'Reason'. For example, according to one embodiment of the present disclosure, an electronic device may store information expressed as 4W1R (information in a 4W1R structure) in a knowledge graph.

[0072] According to one embodiment of the present disclosure, an electronic device can generate a 4W1R vector based on information extracted or determined from user input information, and store the generated 4W1R vector in a knowledge graph. For example, the electronic device can generate the 4W1R vector based on the user's intention and the context included in the user input information, and the user's intention may be determined from the context included in the user input information. In this case, the 4W1R vector may be a vector including information in a 4W1R structure.

[0073] According to one embodiment of the present disclosure, an electronic device can extract explicit information (4W) from user input information and determine implicit information (1R). For example, the electronic device can infer 1R based on 4W extracted from the user input information. For example, the electronic device can determine 1R based on the overall context of the user input information. For example, the context of the user input information, rather than 4W, may be considered when determining 1R.

[0074] According to one embodiment of the present disclosure, 1R may be explicitly included in the user input information. The electronic device can extract 1R from the user input information.

[0075] According to one embodiment of the present disclosure, the electronic device may utilize a generative model, such as an LLM, when determining 1R. For example, the electronic device may obtain 1R by performing knowledge graph-based reasoning based on user input information.

[0076] While non-explicit information (1R) is not stored in existing knowledge graphs, the knowledge graph according to one embodiment of the present disclosure stores non-explicit information (1R), thus having the advantage of being able to output more specific and accurate results when performing knowledge graph-based inference.

[0077] Since only context information is stored in the knowledge graph according to one embodiment of the present disclosure, the knowledge graph becomes lightweight, and the electronic device can quickly perform processing such as searching, creating, merging, or modifying the knowledge graph in real time.

[0078] An example of how an electronic device obtains contextual information from user input is provided.

[0079] According to one embodiment of the present disclosure, when a user makes a call using an electronic device (e.g., a smartphone), the electronic device recognizes the call audio (user input information) to understand the content of the call and extracts explicit information (4W) from the call content. The electronic device can extract information such as who the user is meeting with, the location and time of the meeting, etc. The electronic device can extract explicit information from the call content.

[0080] According to one embodiment of the present disclosure, an electronic device can determine implicit information (1R) based on the overall context of the call. For example, the electronic device can determine information about why the user made the appointment, such as the purpose or reason for the appointment. The electronic device can determine 1R based on the 4W extracted from the call content, or it can determine 1R by considering the overall context of the call content.

[0081] So far, we have described information stored in a knowledge graph according to one embodiment of the present disclosure. The structure of the knowledge graph according to one embodiment of the present disclosure will now be described.

[0082] 2. Structure of the knowledge graph (database)

[0083] FIG. 2 is a diagram illustrating the structure of a 4W1R-based knowledge graph according to one embodiment of the present disclosure. Referring to FIG. 2, the knowledge graph may include only nodes without edges. The five nodes included in the knowledge graph may each correspond to subject information, event information, location information, time information, and reason information. For example, the nodes of the knowledge graph may store subject information, event information, location information, time information, and reason information, respectively. The location of each node may vary from that illustrated in FIG. 2. At least one of the nodes included in the knowledge graph may store an attribute. The method of storing an attribute in a node is described in detail in the section below on embodiments for controlling a device (e.g., a home appliance, etc.).

[0084] The knowledge graph illustrated in Figure 2 has a structure that only includes nodes, but knowledge graphs can also be implemented with a structure that includes nodes and edges. A knowledge graph with a different structure will be described with reference to Figure 3.

[0085] FIG. 3 is a diagram illustrating the structure of a 4W1R-based knowledge graph according to one embodiment of the present disclosure. Referring to FIG. 3, the knowledge graph may include edges and nodes, and three nodes may correspond to Who, What, and Where (subject information, event information, and location information), respectively, and two edges may correspond to When and Reason (time information and reason information), respectively. For example, the nodes of the knowledge graph may store Who, What, and Where information (subject information, event information, and location information), respectively, and the edges may store When and Reason information (time information and reason information), respectively. The positions of each node and the types of nodes connected by the edges may be varied differently from those illustrated in FIG. 3. At least one of the nodes and edges included in the knowledge graph may store an attribute.

[0086] The electronic device can obtain context information expressed as 4W1R from user input information and store the context information in a node or edge of the knowledge graph illustrated in FIG. 2 or FIG. 3.

[0087] The knowledge graph may be implemented in various structures other than those illustrated in FIGS. 2 and 3.

[0088] 3. Management of knowledge graphs (databases) and provision of personalized services based on knowledge graphs.

[0089] An electronic device according to one embodiment of the present disclosure can acquire context information from context collected based on user input information (e.g., calls, chats, searches, etc.), structure the acquired context information, and store it in a knowledge graph in real time. The electronic device can perform knowledge graph-based inference using the knowledge graph thus generated, and can provide personalized services (e.g., call summarization, schedule management, health information provision, or device control, etc.) using the knowledge graph.

[0090] Referring to FIG. 4, a process of generating and updating a knowledge graph based on user input information by an electronic device according to one embodiment of the present disclosure is described, and then a process of providing a personalized service based on the knowledge graph is described.

[0091] (1) Creation and update of knowledge graph (database)

[0092] FIG. 4 is a drawing for explaining modules included in an electronic device according to one embodiment of the present disclosure.

[0093] The components (100, 200, 300) included in the electronic device (1000) of FIG. 4 are components classified based on their functions or roles. The components (100, 200, 300) of the electronic device (1000) of FIG. 4 may be software components implemented by the processor (1300) of the electronic device (1000) executing a program stored in the memory (1400), which will be described later with reference to FIG. 5, or may be virtual components for which no matching hardware device actually exists. The operations performed by the processor (1300) of the electronic device (1000) executing a program or instruction stored in the memory (1400) may be classified into a plurality of groups based on their functions or purposes, and the entities that perform the operations included in each of the classified groups may be expressed as the components of FIG. 4. The operations described as being performed by the components (100, 200, 300) of the electronic device (1000) illustrated in FIG. 4 can be seen as actually being performed by the processor (1300) of the electronic device (1000) executing a program or instruction stored in the memory (1400).

[0094] In FIG. 4, one electronic device (1000) is illustrated as including all of the components (100, 200, 300), but this is not limited thereto, and at least some of the components (100, 200, 300) may be implemented to be included in a separate device, or one of the components may be implemented to be included in another component. The components (100, 200, 300) included in the electronic device (1000) according to one embodiment of the present disclosure may be hardware components or software components, and may be implemented in the form of various electronic devices (e.g., one electronic device or a combination of two or more electronic devices).

[0095] An electronic device (1000) according to one embodiment of the present disclosure may be a terminal (e.g., a smartphone, a laptop, a desktop, etc.) of a user (1), but is not limited thereto, and may also be a server that performs communication with the terminal of the user (1).

[0096] An electronic device (1000) according to one embodiment of the present disclosure may include a Parameter Efficient Fine-Tuning (PEFT) model obtained by fine-tuning a pre-trained model according to the PEFT technique. The electronic device (1000) may include a collaborative PEFT model that is fine-tuned to personalize the electronic device using a dynamic knowledge graph, and may include a master PEFT model (100) and an agent PEFT model (200) as illustrated in FIG. 4. The electronic device (1000) according to one embodiment of the present disclosure has an advantage in on-device implementation by using a collaborative PEFT model composed of a plurality of models.

[0097] The master PEFT model (100) may be a fine-tuned model that determines whether an action corresponding to a user input needs to be performed, what action needs to be performed, what information is required to perform the action, etc. The agent PEFT model (200) may be a fine-tuned model that performs a specific action based on the judgment results of the master PEFT model (100). The master PEFT model (100) and the agent PEFT model (200) may be referred to as a master model and an agent model, respectively.

[0098] The master PEFT model (100) may include a context reasoner (110), a 4W1R generator (120), a semantic reasoner (130), a preferred action generator (140), and an agent model classifier (150). The operations of the detailed components included in the master PEFT model (100) are described in detail below.

[0099] The agent PEFT model (200) may include a plurality of agents (210, 220, 230). The agents (210, 220, 230) are configured to provide services to the user (1) by performing various categories of operations (e.g., search, shopping, schedule management, photo search, SNS post upload, etc.). For example, each of the agents (210, 220, 230) may be a model fine-tuned according to the PEFT technique to perform specific operations for each category. The agents (210, 220, 230) may execute apps of their respective categories, thereby performing operations that match the intention of the user (1) or operations required by the user (1).

[0100] For example, a first agent (210) may execute an app in the search category (an app providing a search function) and perform a necessary search based on user input information. For example, a second agent (220) may execute an app in the shopping category (an app providing an online shopping function) and display a page for purchasing necessary items on the screen of an electronic device (1000) based on user input information. The agent PEFT model (200) may also include agents for performing various other operations.

[0101] A dynamic knowledge graph can be stored in the database (300), and as described above, the database (300) can be implemented to store information in various structures and forms other than the knowledge graph.

[0102] The electronic device (1000) can transmit user input information received from the user (1) to the master PEFT model (100). For example, the electronic device (1000) can transmit voice information recorded from a phone call between the user (1) and another person to the master PEFT model (100). For example, the electronic device (1000) can transmit text information entered by the user (1) while chatting with another person to the master PEFT model (100). For example, the electronic device (1000) can transmit an image displayed on the screen to the master PEFT model (100).

[0103] According to one embodiment of the present disclosure, the electronic device (1000) can extract voice information, text information, image information, etc. from a user input using a Content Capture Service (CCS) and transmit the extracted voice information, text information, image information, etc. to a master PEFT model (100) as user input information. The electronic device (1000) can also extract voice information, text information, image information, etc. from a user input using an on-device LLM such as a multimodal encoder and transmit the extracted voice information, text information, image information, etc. to a master PEFT model (100).

[0104] The context reasoner (110) can determine whether an action corresponding to user input information needs to be performed. The context reasoner (110) can determine which action to perform based on the context of the user input information. The context reasoner (110) can understand the context of the user input information and, based on the result, determine whether an action such as creating or updating a knowledge graph, executing an app, or controlling a device needs to be performed. For example, an action corresponding to user input information may refer to an action of creating or updating a knowledge graph. An action corresponding to user input information may refer to an action of processing a user's (1) request (e.g., answering a question) based on the knowledge graph. An action corresponding to user input information may refer to an action such as executing an app to search for a location, moving to a page for purchasing an item, saving or changing a schedule, uploading a SNS post, or searching for a photo. An action corresponding to user input information may also refer to an action of controlling a home appliance, such as adjusting the set temperature of an air conditioner.

[0105] For example, the context reasoner (110) may determine that the knowledge graph needs to be created or updated if the user's (1) call or chat content includes information that needs to be reflected in the knowledge graph. For example, the context reasoner (110) may determine that an app needs to be executed for search or shopping based on information contained in the user's (1) call or chat content. The context reasoner (110) may also determine that control of a home appliance is necessary based on analysis of the user's (1) voice command. Embodiments are described in detail below.

[0106] If the context reasoner (110) determines that an action corresponding to user input information needs to be performed, it can request the generation of a 4W1R vector while transmitting the user input information or the context information extracted from the user input information to the 4W1R generator (120). As described below, a 4W1R vector is required to create or update a knowledge graph, execute an app, or control a device. A '4W1R vector' is a vector that includes context information of a 4W1R structure extracted from user input information. For example, a vector obtained by performing an embedding transformation on context information expressed as 4W1R is a 4W1R vector. The method of generating a 4W1R vector is described in detail in the 4W1R generator section below.

[0107] The context reasoner (110) can terminate the process by stopping the execution of the master PEFT model (100) if it determines that there is no need to perform an action corresponding to the user input information.

[0108] The 4W1R generator (120) can generate a 4W1R vector based on context information acquired from user input information. For example, the 4W1R generator (120) can acquire context information expressed in a 4W1R structure based on the intention and context of the user (1) extracted or determined from the user input information, and can generate a 4W1R vector using the acquired context information. According to one embodiment of the present disclosure, the 4W1R generator (120) can summarize user input information, then acquire context information from the summarized information, and generate a 4W1R vector using the acquired context information.

[0109] The 4W1R generator (120) can receive context information from the context reasoner (110) or obtain context information from user input information received from the context reasoner (110).

[0110] The 4W1R generator (120) can perform a 4W1R vector by tokenizing context information according to the 4W1R structure and performing embedding transformation on the resulting tokens. The 4W1R generator (120) can extract information corresponding to Who, What, Where, When, and Reason (subject information, event information, location information, time information, and reason information) among the context information, separate the extracted information into a plurality of tokens, and then perform embedding transformation on the separated tokens.

[0111] The 4W1R generator (120) may not be able to obtain some of the information corresponding to 4W1R. For example, the context information may not include information corresponding to at least one of Who, What, Where, When, and Reason (subject information, event information, location information, time information, and reason information). The 4W1R generator (120) may leave the information that it has not obtained blank and generate a 4W1R vector. Since a default value is predetermined for at least one of the 4W1R, the 4W1R generator (120) may apply the default value to information that is not included in the context information to generate a 4W1R vector.

[0112] The semantic reasoner (130) can determine whether a knowledge graph needs to be created or updated based on the 4W1R vector. If the determination indicates a knowledge graph needs to be created or updated, the semantic reasoner (130) can create or update the knowledge graph based on the 4W1R vector.

[0113] The semantic reasoner (130) can determine whether it is necessary to create or update a knowledge graph by comparing the information included in the 4W1R vector received from the 4W1R generator (120) with the information included in the knowledge graph previously stored in the database (300). The semantic reasoner (130) can check whether an event of the same context as an event included in the 4W1R vector is stored in an existing knowledge graph, and if an event of the same context is not stored in the existing knowledge graph, it can create a new knowledge graph, and if an event of the same context is stored in the existing knowledge graph, it can update the knowledge graph with the information included in the 4W1R vector.

[0114] According to one embodiment of the present disclosure, the semantic reasoner (130) may store 4W1R vectors in the database (300) through vector quantization. The semantic reasoner (130) may store a codebook generated by clustering 4W1R vectors in the database (300), and may store the 4W1R vectors in the database (300) by mapping them to their respective corresponding code indices. The code indices refer to indices corresponding to the center points (code vectors) of a plurality of clusters included in the codebook.

[0115] Of course, the semantic reasoner (130) can store the 4W1R vector in the database (300) as is without quantizing it. However, if the semantic reasoner (130) stores the 4W1R vector in the knowledge graph using vector quantization, it is expected that the storage space will be saved and the amount of computation to be processed when creating and updating the knowledge graph will be reduced.

[0116] According to one embodiment of the present disclosure, the semantic reasoner (130) may classify context information based on reason information (e.g., 1R) included in context information (e.g., information included in a 4W1R vector) and store the classified context information in the database (300). For example, the semantic reasoner (130) may store the context information by linking it with at least one context information previously stored in the database (300) based on the reason information. If the first context information acquired from the user input information includes the first reason information and the database (300) already has second context information including the second reason information stored, the semantic reasoner (130) may compare the first reason information and the second reason information, and if the first reason information and the second reason information match each other, the first context information may be linked to the second context information and stored.

[0117] The reason information included in the context information may include information regarding the reason for the event corresponding to the context information. The reason information may be inferred from user input information corresponding to the context information. The context information includes explicit information (e.g., 4W) and implicit information (e.g., 1R), and the implicit information may include reason information. Specific examples of reason information are described in detail below with reference to the drawings.

[0118] If the semantic reasoner (130) classifies contextual information based on reason information and stores it in the database (300), contextual information obtained from user input information collected through various apps installed on the electronic device (1000) can be interconnected and stored. As a result, the electronic device (1000) can provide personalized services to the user by considering the interconnected contextual information together. For example, the electronic device (1000) can provide personalized services, such as processing a user's request or responding to a user's question, based on the results of information shared between multiple apps.

[0119] (2) Service provision and device control based on knowledge graph (database)

[0120] The semantic reasoner (130) can determine whether to perform an action, such as executing an app or controlling a device, based on the knowledge graph stored in the database (300). The knowledge graph stored in the database (300) can be generated or updated based on the 4W1R vector received from the 4W1R generator (120) as described above. The semantic reasoner (130) can also determine whether to perform an action, such as executing an app or controlling a device, based on the result of comparing the information included in the 4W1R vector received from the 4W1R generator (120) with the information included in the knowledge graph stored in the database (300).

[0121] In summary, the semantic reasoner (130) can determine whether to perform an action such as running an app or controlling a device based on at least one of the 4W1R vectors received from the 4W1R generator (120) and the knowledge graph stored in the database (300).

[0122] If the judgment result indicates that an action such as launching an app or controlling a device is required, the semantic reasoner (130) can request the preferred action generator (140) to perform the required action.

[0123] The semantic reasoner (130) can determine whether a specific action (e.g., search, shopping, schedule management, photo search, SNS post upload, etc.) should be performed based on the characteristics and content of the information included in the 4W1R vector and the knowledge graph. For example, if the 4W1R vector contains information about a new schedule (appointment) that is not included in the knowledge graph, the semantic reasoner (130) can determine that a calendar app should be executed to save the schedule. Or, for example, if the 4W1R vector or the knowledge graph contains information that requires the purchase of an item, the semantic reasoner (130) can determine that an online shopping app should be executed to purchase the item. If the semantic reasoner (130) needs to execute an app for search, shopping, schedule management, photo search, or SNS post upload, it can request the preferred action generator (140) to perform the necessary action.

[0124] In addition, the semantic reasoner (130) can control home appliances such as air conditioners based on the 4W1R vector received from the 4W1R generator (120) and the knowledge graph stored in the database (300). For example, if the semantic reasoner (130) determines that the user (1) feels the current temperature is hot based on the information included in the 4W1R vector, it can turn on the air conditioner and adjust the set temperature based on the information included in the knowledge graph.

[0125] According to one embodiment of the present disclosure, a representative 4W1R vector related to device control may be stored in the knowledge graph of the database (300), and a semantic reasoner (130) may control the device based on the representative 4W1R vector. The 'representative 4W1R vector' may mean a vector including information for controlling the device, and a representative 4W1R vector corresponding to each device may exist. For example, if a user (1) usually adjusts the set temperature of an air conditioner to 24 degrees, the representative 4W1R vector corresponding to the air conditioner may include information indicating that the set temperature of the air conditioner should be adjusted to 24 degrees.

[0126] The preferred action generator (140) can determine whether a preferred action of the user (1) exists in relation to the action requested from the semantic reasoner (130). The preferred action of the user (1) may refer to the form of the action preferred by the user (1) by category. For example, if the user (1) prefers to use App A when searching, the preferred action of the user (1) corresponding to the category of search may be 'executing App A'. Preferred actions corresponding to each category may exist.

[0127] If the user's (1) preferred action is not found as a result of the judgment, the preferred action generator (140) can launch or recommend an appropriate app to perform the requested action (e.g., recommend a shopping app if the user needs to purchase an item).

[0128] If the judgment result indicates that the user's (1) preferred action exists, the preferred action generator (140) can request the agent model classifier (150) to perform the preferred action. For example, the preferred action generator (140) can generate a deep link for executing an app frequently used by the user (1) and transmit it to the agent model classifier (150). At this time, the deep link for executing an app frequently used by the user (1) can be referred to as a 'preferred deep link'. The preferred action generator (140) can also transmit information about the user's (1) preferred action to the agent model classifier (150).

[0129] The agent model classifier (150) can select an agent to perform a preferred action of the user (1) and cause the selected agent to perform the preferred action. For example, the agent model classifier (150) can select at least one agent (210, 220, 230) for various categories included in the agent PEFT model (200) based on information about the preferred deep link or preferred action received from the preferred action generator (140). The selected agent can execute an app preferred by the user (1) and perform a necessary action through information about the preferred deep link or preferred action.

[0130] Each of the multiple agents (210, 220, 230) included in the agent PEFT model (200) may have a corresponding category, and the agent model classifier (150) may select an agent of a category corresponding to a preferred deep link or preferred action. For example, if the preferred deep link or preferred action is for search, the agent model classifier (150) may select an agent of the search category.

[0131] The operations of the semantic reasoner (130), the preferred action generator (140), the agent model classifier (150), and the agent PEFT model (200) are described below with specific examples.

[0132] According to the first embodiment, when the semantic reasoner (130) determines that a location search is necessary and requests the preferred action generator (140) to search for a location, the preferred action generator (140) can determine whether a preferred action of the user related to the search exists. If the user (1) has frequently used App A in the past when searching, the preferred action generator (140) can determine that the execution of App A is the preferred action of the user (1), and can generate a preferred deep link for executing App A and transmit it to the agent model classifier (150). The agent model classifier (150) can select the first agent (210) of the search category based on the received preferred deep link. The first agent (210) can execute App A through the preferred deep link and search for a location in App A.

[0133] According to the second embodiment, when the semantic reasoner (130) determines that a purchase of an item is necessary and requests the preference action generator (140) to purchase the item, the preference action generator (140) can determine whether the user (1) has a preference action related to shopping. If the user (1) has frequently used the B app when shopping in the past, the preference action generator (140) can determine that the execution of the B app is the user's (1) preference action, and can generate a preference deep link for executing the B app and transmit it to the agent model classifier (150). The agent model classifier (150) can select the second agent (220) of the shopping category based on the received preference deep link. The second agent (220) can execute the B app through the preference deep link and display a page for purchasing an item in the B app on the screen.

[0134] The results of an electronic device performing an operation according to the process described above can be stored in a database (300). User preferences identified through performing the operation multiple times can also be stored in the database (300) and used as a reference when performing the next operation.

[0135] 4. Hardware configuration of electronic devices

[0136] FIG. 5 is a diagram illustrating a hardware configuration included in an electronic device according to one embodiment of the present disclosure. Referring to FIG. 5, an electronic device (1000) according to one embodiment may include a communication interface (1100), an input / output interface (1200), a processor (1300), and a memory (1400). The components of the electronic device (1000) are not limited to the examples described above, and the electronic device (1000) may include more or fewer components than the components described above. Some or all of the communication interface (1100), the input / output interface (1200), the processor (1300), and the memory (1400) may be implemented in the form of a single chip.

[0137] The communication interface (1100) is a configuration for transmitting and receiving signals (such as control commands and data) with an external device via wire or wirelessly, and may be implemented to include a communication chipset that supports various communication protocols. The communication interface (1100) may receive signals from the outside and output them to the processor (1300), or transmit signals output from the processor (1300) to the outside. The electronic device (1000) may communicate with external devices via the communication interface (1100).

[0138] The input / output interface (1200) may include an input interface (e.g., a touch screen, a keyboard, a microphone, etc.) for receiving commands or information from a user (1), and an output interface (e.g., a display panel, a speaker, etc.) for displaying the result of an operation according to a user's command or the status of the electronic device (1000). According to one embodiment of the present disclosure, the electronic device (1000) may receive an input from the user (1) through the input / output interface (1200), and when an operation is completed, may output the result of performing the operation (e.g., a location search result, etc.) through the input / output interface (1200).

[0139] The processor (1300) controls a series of processes to operate the electronic device (1000) according to the embodiments described below, and may be composed of one or more processors. The one or more processors included in the processor (1300) may be circuitry such as a System on Chip (SoC), an Integrated Circuit (IC), etc. The one or more processors included in the processor (1300) may be a general-purpose processor such as a Central Processing Unit (CPU), a Micro Processor Unit (MPU), an Application Processor (AP), a Digital Signal Processor (DSP), a graphics-only processor such as a Graphics Processing Unit (GPU), a Vision Processing Unit (VPU), an artificial intelligence-only processor such as a Neural Processing Unit (NPU), or a communication-only processor such as a Communication Processor (CP). When the one or more processors included in the processor (1300) are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing artificial intelligence models.

[0140] The processor (1300) can write data to the memory (1400) or read data stored in the memory (1400), and in particular, process data according to predefined operation rules or artificial intelligence models by executing a program or at least one instruction stored in the memory (1400). The processor (1300) can perform operations described in the following embodiments, and operations described as being performed by the electronic device (1000) or modules (100, 200, 300) included in the electronic device (1000) in this specification can be regarded as being performed by the processor (1300) unless otherwise specifically described.

[0141] The memory (1400) is a configuration for storing various programs or data, and may be configured as a storage medium such as a ROM, a RAM, a hard disk, a CD-ROM, and a DVD, or a combination of storage media. The memory (1400) may not exist separately and may be configured to be included in the processor (1300). The memory (1400) may be configured as a volatile memory, a non-volatile memory, or a combination of volatile memory and non-volatile memory. A program or at least one instruction for performing operations according to embodiments described below may be stored in the memory (1400). The memory (1400) may also provide stored data to the processor (1300) upon request of the processor (1300).

[0142] Hereinafter, specific embodiments of an electronic device (1000) managing a knowledge graph based on user input and providing services or controlling the device based on the knowledge graph will be described. If necessary, the operations performed by the electronic device (1000) will be described using the configurations of the electronic device (1000) illustrated in FIG. 4.

[0143] 5. Description of Specific Examples

[0144] (1) An embodiment of summarizing and saving call (chat) content

[0145] If the electronic device (1000) according to one embodiment of the present disclosure is a device (e.g., a smartphone) that supports call and chat (message) functions, the electronic device (1000) can summarize and store the contents of the user's calls and chats, and create or update a knowledge graph accordingly.

[0146] According to one embodiment of the present disclosure, the electronic device (1000) can summarize the call content into a few sentences at the end of each call and display the summarized content in a call list along with information for identifying the caller (e.g., the caller's name or phone number). The electronic device (1000) can also extract keywords from the summarized call content and display the extracted keywords in the call list. The electronic device (1000) can also generate a title corresponding to the call content and display it in the call list.

[0147] The electronic device (1000) can generate or update a knowledge graph by reflecting summarized call content or keywords extracted from the call content. For example, the electronic device (1000) can obtain context information in a 4W1R structure from summarized call content or extracted keywords, and store the obtained context information in the knowledge graph.

[0148] The electronic device (1000) may perform necessary actions, such as saving a schedule, based on the call content. For example, if the context information obtained from the call content includes information about a new schedule (e.g., creating a new meeting appointment or changing the location or time of an existing meeting appointment), the electronic device (1000) may store the information about the new schedule in the knowledge graph and save the new schedule in the calendar app.

[0149] Referring to FIGS. 6 to 12, a specific embodiment of an electronic device (1000) managing a knowledge graph and storing a schedule based on call content will be described.

[0150] FIG. 6 is a diagram illustrating a situation in which a user makes a call using an electronic device according to one embodiment of the present disclosure. Referring to FIG. 6, users A and B make calls using their respective electronic devices (1000, 2000). FIG. 6 depicts the content of the call between users A and B.

[0151] The electronic device (1000) can obtain context information from the conversation between User A and User B, and create or update a knowledge graph based on the obtained context information. The electronic device (1000) can also save a schedule in a calendar app based on the updated knowledge graph.

[0152] The electronic device (1000) can obtain the contents of a call between users A and B as user input information. For example, the electronic device (1000) can record and store the voices of users A and B talking. The electronic device (1000) can also check the voices of users A and B talking in real time and store them as text information. The electronic device (1000) can obtain the contents of a call between users A and B as user input information through various methods.

[0153] The electronic device (1000) can obtain context information from acquired user input information (call content) and store the obtained context information in a knowledge graph. According to one embodiment of the present disclosure, the electronic device (1000) can obtain context information in a 4W1R structure from call content.

[0154] Referring to the call content illustrated in FIG. 6, User A is having a conversation with User B to make a meeting appointment, and mentions the place, time, and reason for the meeting. Referring to the call content, it can be seen that User A and User B have agreed to meet at COEX tomorrow evening. The electronic device (1000) can determine the reason for User A meeting User B by considering the overall context of the call content. Since User A said, "The wedding invitations came out," the electronic device (1000) can know that User A is planning to meet User B and give him the invitations. The electronic device (1000) can obtain context information in a 4W1R structure from the call content as follows.

[0155] Who: User A and User B

[0156] What: Meeting

[0157] Where: COEX

[0158] When: Tomorrow evening

[0159] Reason: To give out wedding invitations

[0160] According to one embodiment of the present disclosure, a context reasoner (110) of an electronic device (1000) can summarize a call content, obtain context information of a 4W1R structure as described above from the summarized call content, and transmit the obtained context information to a 4W1R generator (120). The 4W1R generator (120) can generate a 4W1R vector by performing embedding on the received context information of a 4W1R structure, and transmit the generated 4W1R vector to a semantic reasoner (130). The semantic reasoner (130) can generate a knowledge graph based on the received 4W1R vector.

[0161] The electronic device (1000) can generate a knowledge graph containing 4W1R information. The electronic device (1000) can store 4W1R information in knowledge graphs of various structures. Examples of storing 4W1R information in knowledge graphs of two structures described with reference to FIGS. 2 and 3 are illustrated in FIGS. 7 and 8.

[0162] Referring to Figure 7, Who (User A and User B) (subject information), What (meeting) (event information), Where (COEX) (location information), When (tomorrow evening) (time information), and Reason (to give an invitation) are all expressed as nodes in the knowledge graph, and the knowledge graph does not include edges. The knowledge graph illustrated in Figure 7 is the result of storing the 4W1R information obtained from the call content of Figure 6 in the knowledge graph having the structure described with reference to Figure 2.

[0163] Referring to Fig. 8, among the 4W1R information, Who (User A and User B) (subject information), What (meeting) (event information), and Where (COEX) (location information) are expressed as nodes of the knowledge graph, and When (tomorrow evening) (time information) and Reason (to give out invitations) can be expressed as edges of the knowledge graph. The knowledge graph illustrated in Fig. 8 is the result of storing the 4W1R information obtained from the call content of Fig. 6 in the knowledge graph having the structure described with reference to Fig. 3.

[0164] After generating the knowledge graph of FIG. 7 or FIG. 8, the electronic device (1000) can store a schedule based on the knowledge graph. For example, among the agents included in the agent PEFT model (200) of the electronic device (1000), an agent corresponding to a schedule management category can store a new schedule in the calendar app based on the knowledge graph.

[0165] It is assumed and explained that the electronic device (1000) uses a knowledge graph having the structure shown in FIG. 2 and FIG. 7.

[0166] As shown in Fig. 6, after users A and B have made a call, users A and C then make a call using their respective electronic devices (1000, 3000) as shown in Fig. 9. Fig. 9 shows the contents of the call between users A and C.

[0167] The electronic device (1000) can obtain context information from the call content between User A and User C and update the knowledge graph based on the obtained context information. The electronic device (1000) can also save the schedule in the calendar app based on the updated knowledge graph.

[0168] The process of an electronic device (1000) obtaining the call content between user A and user C as user input information and obtaining context information of a 4W1R structure from the call content is as described above with reference to FIG. 6.

[0169] Referring to the call content illustrated in Figure 9, User A proposes to User C to meet with User B, mentioning the location, time, and reason for the meeting. The call content reveals that User A has agreed to meet with Users B and C at COEX tomorrow evening at 7:00 PM, and that User A intends to give Users B and C wedding invitations.

[0170] Compared to the call content previously depicted in Figure 6, the Who and When fields (subject information and time information) have been updated. Therefore, the electronic device (1000) can obtain context information in a 4W1R structure from the call content as follows. (The underlined portion indicates the updated portion compared to the existing context information.)

[0171] Who: User A, User B, and User C

[0172] What: Meeting

[0173] Where: COEX

[0174] When: Tomorrow at 7 p.m.

[0175] Reason: To give out wedding invitations

[0176] According to one embodiment of the present disclosure, a context reasoner (110) of an electronic device (1000) can summarize a call content, obtain context information of the above 4W1R structure from the summarized call content, and transmit the obtained context information to a 4W1R generator (120). The 4W1R generator (120) can generate a 4W1R vector by performing embedding on the received context information of the 4W1R structure, and transmit the generated 4W1R vector to a semantic reasoner (130).

[0177] The semantic reasoner (130) can determine whether the knowledge graph needs to be updated based on the result of comparing the received 4W1R vector with the previously generated knowledge graph of FIG. 7. By comparing the 4W1R vector generated based on the call content of FIG. 9 with the knowledge graph of FIG. 7, the semantic reasoner (130) can determine that the knowledge graph needs to be updated since the Who and When items (subject information and time information) contain new information. The semantic reasoner (130) can update the knowledge graph based on the received 4W1R vector.

[0178] The knowledge graph updated by the electronic device (1000) based on the call content of FIG. 9 is illustrated in FIG. 10. FIG. 10 illustrates an example in which the electronic device (1000) answers a question posed by a user.

[0179] The semantic reasoner (130) of the electronic device (1000) can add new information not stored in the knowledge graph among the information included in the 4W1R vector received from the 4W1R generator (120). Referring to FIG. 10, the electronic device (1000) added 'User C' to the node corresponding to Who (subject information) among the nodes of the knowledge graph, and added '7 PM' to the node corresponding to When (time information). In this way, only the latest information can be stored for all 4W1R items in the updated knowledge graph.

[0180] According to one embodiment of the present disclosure, the electronic device (1000) can lighten the knowledge graph by storing only the latest information in each node of the knowledge graph and not storing unnecessary existing information (e.g., the appointment time before the change in a situation where the appointment time has been changed) in the knowledge graph. The electronic device (1000) according to one embodiment of the present disclosure has the advantage of being able to create or update a knowledge graph in real time and providing services to users in real time based on the results of performing knowledge graph-based inference based on the knowledge graph.

[0181] The electronic device (1000) can update a schedule based on the knowledge graph after updating the knowledge graph as illustrated in FIG. 10. For example, among the agents included in the agent PEFT model (200) of the electronic device (1000), an agent corresponding to a schedule management category can update the schedule stored in the calendar app based on the knowledge graph.

[0182] When a user inputs a question as a prompt, the electronic device (1000) can perform knowledge graph-based inference based on the knowledge graph and provide an answer based on the result. For example, when user A inputs a voice question to the electronic device (1000), the electronic device (1000) can determine an answer to user A's question based on the knowledge graph and output the determined answer as a voice or display it as text on the screen.

[0183] As illustrated in FIG. 10, if user A inputs a question to the electronic device (1000) such as “Who will I meet tomorrow and why?”, the electronic device (1000) can perform knowledge graph-based inference based on the knowledge graph stored in the database (300) and generate an answer based on the result.

[0184] The electronic device (1000) can check the information stored in the node corresponding to Who (subject information) and the node corresponding to Reason in the knowledge graph to generate an answer to User A's question. Based on the result of checking the information stored in the knowledge graph, the electronic device (1000) can generate the answer, "Meet with User B and User C to deliver the wedding invitation."

[0185] Since existing knowledge graphs only store explicit information (information corresponding to the 4Ws), they have limitations in being able to adequately respond when non-explicit information is required to generate a question answer. For example, if an electronic device (1000) had stored the conversation of user A in an existing knowledge graph, when the question shown in FIG. 10 was entered, the electronic device (1000) would not have been able to generate an answer to the question "why" user A meets users B and C.

[0186] In the embodiments illustrated in FIGS. 6 to 12, information about the Reason can be relatively easily identified from the conversation between users. However, in some cases, information about the Reason can be obtained only by identifying the context of the entire conversation and making a decision based on the identified context. According to one embodiment of the present disclosure, the electronic device (1000) can also acquire information about the Reason from the conversation using a neural network model such as LLM. Information about the Reason can be classified as non-explicit information among contextual information.

[0187] As shown in Fig. 9, after User A and User C have made a call, a situation in which User A and User B make a call again using their respective electronic devices (1000, 2000) is depicted in Fig. 11. Fig. 11 shows the contents of the call between User A and User B.

[0188] The process by which the electronic device (1000) obtains context information of the 4W1R structure from the call content between user A and user B is as described above with reference to FIG. 6.

[0189] Referring to the call transcript depicted in Figure 11, User B proposes to User A to change the meeting time and location, and User A agrees. The call transcript reveals that the meeting time between Users A and B has been changed to Friday evening, and the meeting location has been changed to Gangnam Station.

[0190] Compared to the call content previously depicted in Figure 9, the Where and When items (location information and time information) have been updated. Therefore, the electronic device (1000) can obtain context information in a 4W1R structure from the call content as follows. (The underlined portion indicates the updated portion compared to the existing context information.)

[0191] Who: User A, User B, and User C

[0192] What: Meeting

[0193] Where: Gangnam Station

[0194] When: Friday 7pm

[0195] Reason: To give out wedding invitations

[0196] According to one embodiment of the present disclosure, a context reasoner (110) of an electronic device (1000) can summarize a call content, obtain context information of the above 4W1R structure from the summarized call content, and transmit the obtained context information to a 4W1R generator (120). The 4W1R generator (120) can generate a 4W1R vector by performing embedding on the received context information of the 4W1R structure, and transmit the generated 4W1R vector to a semantic reasoner (130).

[0197] The semantic reasoner (130) can determine whether the knowledge graph needs to be updated based on the result of comparing the received 4W1R vector with the previously generated knowledge graph of FIG. 10. By comparing the 4W1R vector generated based on the call content of FIG. 11 with the knowledge graph of FIG. 10, the semantic reasoner (130) can determine that the knowledge graph needs to be updated since the Where and When items (subject information and time information) contain new information. The semantic reasoner (130) can update the knowledge graph based on the received 4W1R vector.

[0198] The knowledge graph updated by the electronic device (1000) based on the call content of FIG. 11 is illustrated in FIG. 12. In addition, FIG. 12 illustrates an example in which the electronic device (1000) answers a question posed by a user.

[0199] The semantic reasoner (130) of the electronic device (1000) can add new information not stored in the knowledge graph among the information included in the 4W1R vector received from the 4W1R generator (120). Referring to FIG. 12, the electronic device (1000) changed the information of the node corresponding to When among the nodes of the knowledge graph to 'Friday evening 7 PM' and changed the information of the node corresponding to Where to 'Gangnam Station'. In this way, only the latest information can be stored for all 4W1R items in the updated knowledge graph.

[0200] The electronic device (1000) can update a schedule based on the knowledge graph after updating the knowledge graph as illustrated in FIG. 12. For example, among the agents included in the agent PEFT model (200) of the electronic device (1000), an agent corresponding to a schedule management category can update the schedule stored in the calendar app based on the knowledge graph.

[0201] The electronic device (1000) can perform knowledge graph-based reasoning based on a knowledge graph when a user inputs a request as a prompt, and provide an answer based on the result.

[0202] As illustrated in FIG. 12, if user A inputs a request to the electronic device (1000) saying, “Tell me about my upcoming appointment,” the electronic device (1000) can perform knowledge graph-based inference based on the knowledge graph stored in the database (300) and generate an answer based on the result.

[0203] The electronic device (1000) can check the information stored in each node corresponding to Who, What, Where, When, and Reason (subject information, event information, location information, time information, and reason information) in the knowledge graph to generate a response to the request of user A. Based on the result of checking the information stored in the knowledge graph, the electronic device (1000) can generate a response such as, "I have agreed to meet with user B and user C at Gangnam Station at 7 PM on Friday. The purpose of the meeting is for me to give wedding invitations to user B and user C."

[0204] While the example so far has focused on users making calls using electronic devices, the process described above can also be applied to chatting using electronic devices. For example, the electronic device (1000) can obtain contextual information from the user's chat content and create or update a knowledge graph based on that contextual information.

[0205] (2) An embodiment of automatically saving a schedule based on a message

[0206] According to one embodiment of the present disclosure, the electronic device (1000) can perform necessary actions based on a received message. For example, if the message includes information for making a restaurant reservation, the electronic device (1000) can store the reservation schedule in a calendar app. FIG. 13 is a diagram illustrating a process of an electronic device storing a schedule based on a message according to one embodiment of the present disclosure. The first screen (1310) of FIG. 13 may be displayed on the input / output interface (1200) of the electronic device (1000).

[0207] Referring to FIG. 13, the first screen (1310) displays a received message, and the message includes reservation information for OO Restaurant.

[0208] When an electronic device (1000) receives a message, it can automatically update its knowledge graph based on the message and save the schedule in a calendar app. Upon receiving a message, the electronic device (1000) can also update its knowledge graph and save the schedule in a calendar app at the user's request. For example, as illustrated in FIG. 13, a button (hereinafter, "AI button") (1311) for AI function support is displayed on the first screen (1310). When a user selects the AI ​​button (1311) while a message is displayed on the screen, the electronic device (1000) can save the schedule based on the message.

[0209] The electronic device (1000) can obtain context information (1321) of the following 4W1R structure from a message (user input information) displayed on the first screen (1310).

[0210] Who: User A

[0211] What: Reservation

[0212] Where: OO Restaurant

[0213] When: Friday, November 1, 2024, 7:00 PM

[0214] Reason: N / A

[0215] The electronic device (1000) can generate a 4W1R vector based on context information (1321) of the 4W1R structure and update a knowledge graph (1320) based on the generated 4W1R vector. Subsequently, the electronic device (1000) can store a schedule in a calendar app (1330) based on the updated knowledge graph (1320).

[0216] The detailed operations performed by the detailed components of the electronic device (1000) in the process of updating the knowledge graph and saving the schedule based on the message are as follows.

[0217] The 4W1R generator (120) generates a 4W1R vector based on context information (1321) of the 4W1R structure, and the semantic reasoner (130) can update a knowledge graph (1320) stored in a database (300) based on the generated 4W1R vector.

[0218] The semantic reasoner (130) can determine whether additional actions need to be taken based on the updated knowledge graph (1320). Since new reservation information has been added to the knowledge graph (1320), the semantic reasoner (130) can determine that a schedule update is necessary and request the preferred action generator (140) to update the schedule.

[0219] The preferred action generator (140) checks whether the user frequently uses an app when managing his / her schedule, and if the user frequently uses the calendar app (1330), it can generate a deep link to execute the calendar app (1330) and transmit it to the agent model classifier (150) along with the reservation information.

[0220] The agent model classifier (150) can select an agent corresponding to a schedule management category from among the agents included in the agent PEFT model (200) and transmit a deep link for executing a calendar app (1330) to the selected agent along with reservation information. The selected agent can execute the calendar app (1330) via the received deep link and save the schedule based on the reservation information.

[0221] Since the electronic device (1000) automatically saves the schedule in the calendar app when the received message contains schedule information, the user does not need to save the schedule one by one, which improves convenience.

[0222] (3) An embodiment of displaying the execution screen of another app based on the chat content.

[0223] According to one embodiment of the present disclosure, the electronic device (1000) can perform necessary actions based on the chat content while the user is chatting. For example, the electronic device (1000) can determine which app needs to be run and which action needs to be performed based on the chat content, and can execute the app and perform the action based on the determination result. The electronic device (1000) can display the results of the app execution and the action overlapping the screen where the chat message is displayed, thereby allowing the user to continue chatting and check the results of the action execution.

[0224] According to one embodiment of the present disclosure, if the electronic device (1000) determines that information retrieval is necessary based on analysis of the user's chat content, the electronic device may execute a search app to perform a search and display the search results on the screen. FIG. 14 is a diagram illustrating a process in which an electronic device according to one embodiment of the present disclosure performs a search based on chat content and displays the search results. The first screen (1410) and the second screen (1420) of FIG. 14 may be displayed on the input / output interface (1200) of the electronic device (1000).

[0225] Referring to Figure 14, the first screen (1410) displays chat messages exchanged between a user and another user.

[0226] The electronic device (1000) can analyze chat content in real time and perform a search if deemed necessary. The electronic device (1000) can also perform a search based on the chat content when the user selects the AI ​​button (1411) while a chat message is displayed.

[0227] The electronic device (1000) can obtain context information (1430) of the following 4W1R structure from the chat content (user input information) displayed on the first screen (1410).

[0228] Who: N / A

[0229] What: Restaurant Information

[0230] Where: OO Restaurant

[0231] When: Present

[0232] Reason: Recommendation from User B

[0233] The electronic device (1000) can generate a 4W1R vector based on context information (1430) of the 4W1R structure and perform a search based on the generated 4W1R vector. Subsequently, the electronic device (1000) can display the search result (1421) overlapping the chat message, as shown in the second screen (1420).

[0234] The detailed operations performed by the detailed components of the electronic device (1000) in the process of performing a search based on the chat content are as follows.

[0235] The 4W1R generator (120) generates a 4W1R vector based on context information (1430) of the 4W1R structure, and the semantic reasoner (130) can determine a necessary action based on the generated 4W1R vector. Since the 4W1R vector includes the content that user B recommended OO restaurant, the semantic reasoner (130) can determine the user's intention that the user will want to check information about the recommended OO restaurant, and thus determine that a search for OO restaurant is necessary. The semantic reasoner (130) can request a search for OO restaurant to the preferred action generator (140).

[0236] The preferred action generator (140) checks whether there is an app that the user frequently uses when performing a search, and if the user has frequently used app A (1440) in the past when searching, it can generate a deep link to execute app A (1440) and transmit it to the agent model classifier (150) along with information for identifying the search target (OO restaurant).

[0237] The agent model classifier (150) can select an agent corresponding to a search category from among the agents included in the agent PEFT model (200), and transmit a deep link for executing App A (1440) to the selected agent along with information about the search target. The selected agent can execute App A (1440) through the received deep link, search for Restaurant OO, and display the search results (1421) on the screen.

[0238] According to one embodiment of the present disclosure, if the electronic device (1000) determines that a schedule search is necessary based on analysis of the user's chat content, the electronic device may perform a search and display the search results on the screen. FIG. 15 is a diagram illustrating a process in which an electronic device according to one embodiment of the present disclosure searches for a schedule based on chat content and displays the search results. The first screen (1510) and the second screen (1520) of FIG. 15 may be displayed on the input / output interface (1200) of the electronic device (1000).

[0239] Referring to Figure 15, the first screen (1510) displays chat messages exchanged between a user and another user.

[0240] The electronic device (1000) can analyze chat content in real time and, if deemed necessary, search for a schedule. The electronic device (1000) can also search for a schedule based on the chat content when the user selects the AI ​​button (1511) while a chat message is displayed.

[0241] The electronic device (1000) can obtain context information (1530) of the following 4W1R structure from the chat content (user input information) displayed on the first screen (1510).

[0242] Who: N / A

[0243] What: Class reunion schedule

[0244] Where: N / A

[0245] When: Present

[0246] Reason: Schedule confirmation required

[0247] The electronic device (1000) can generate a 4W1R vector based on context information (1530) of the 4W1R structure and search for a schedule based on the generated 4W1R vector. Subsequently, the electronic device (1000) can display the schedule search results (1521) overlapping the chat message, as shown in the second screen (1520).

[0248] The detailed operations performed by the detailed components of the electronic device (1000) in the process of searching for a schedule based on the chat content are as follows.

[0249] The 4W1R generator (120) generates a 4W1R vector based on context information (1530) of the 4W1R structure, and the semantic reasoner (130) can determine a necessary action based on the generated 4W1R vector. Since the 4W1R vector includes information indicating that a reunion schedule needs to be confirmed, the semantic reasoner (130) can determine that a reunion schedule needs to be searched. The semantic reasoner (130) can request a reunion schedule search from the preferred action generator (140).

[0250] The preferred action generator (140) can check whether the user frequently uses an app when managing his / her schedule, and if the user frequently uses a calendar app (1540), it can generate a deep link to execute the calendar app (1540) and transmit it to the agent model classifier (150) along with information for identifying the search target (class reunion schedule).

[0251] The agent model classifier (150) can select an agent corresponding to a schedule management category among the agents included in the agent PEFT model (200), and transmit a deep link for executing a calendar app (1540) to the selected agent along with information for identifying the search target. The selected agent can execute the calendar app (1540) via the received deep link to search for a class reunion schedule, and display the search results (1521) on the screen.

[0252] If the electronic device (1000) determines that the user wants to search for information or schedules based on the context of the chat content, it automatically performs a search and provides the results, so that the user's convenience can be expected to be improved.

[0253] (4) An embodiment of automatically generating answers, etc. based on multimodal input

[0254] According to one embodiment of the present disclosure, the electronic device (1000) can automatically perform operations such as generating a response based on multimodal input such as text, images, audio, etc. For example, when the electronic device (1000) receives an image from a counterparty through a chat app, the electronic device (1000) can automatically generate a response based on the image and recommend it to the user. FIG. 16 is a diagram illustrating a process of an electronic device automatically generating a response based on an image according to one embodiment of the present disclosure. The first screen (1610) and the second screen (1620) of FIG. 16 may be displayed on the input / output interface (1200) of the electronic device (1000).

[0255] The first screen (1610) of Fig. 16 displays chat messages exchanged between a user and another user. Referring to Fig. 16, when a user sent the message "What are you doing?", the other user sent an image (1612) in response.

[0256] The electronic device (1000) can analyze the received image (1612) in real time and automatically generate an appropriate response based on the analysis results. When the user selects the AI ​​button (1611) while the image (1612) is displayed on the screen, the electronic device (1000) can analyze the image (1612) and generate a response thereto.

[0257] The electronic device (1000) can obtain context information (1630) of a 4W1R structure based on the chat content (user input information) displayed on the first screen (1610). Since the other party sends an image (1612) in response to the user sending a message “What are you doing?” on the first screen (1610), the electronic device (1000) can determine that a response to the image (1612) is required. The electronic device (1000) can generate information (reason information) in the Reason item as “response generation required.” As a result of checking the background or object included in the image (1612), the electronic device (1000) can determine that the image (1612) is a photo taken of the Han River, generate information in the What item as “photo taken of the Han River,” and generate information (location information) in the Where item as “Han River.”

[0258] As a result, the electronic device (1000) can obtain context information (1630) of the following 4W1R structure from the chat content displayed on the first screen (1610).

[0259] Who: User B

[0260] What: Photos taken along the Han River

[0261] Where: Han River

[0262] When: Present

[0263] Reason: Need to generate an answer

[0264] The electronic device (1000) can generate a 4W1R vector based on the context information (1630) of the 4W1R structure acquired as described above, and can generate an answer for the image (1612) based on the 4W1R vector. Since the image (1612) is a photo of the Han River, the electronic device (1000) can generate “Did you go to the Han River?” (1621) and “The weather is so nice!” (1622) as answer candidates and display them on the second screen (1620). The user can select one of the answer candidates (1621, 1622) displayed on the second screen (1620) or directly write a new answer to respond to the other party.

[0265] The electronic device (1000) can generate a 4W1R vector based on the result of analyzing an image displayed on the screen, and generate an answer based on the 4W1R vector, thereby generating an answer that matches the background or object included in the image.

[0266] The function of an electronic device (1000) according to one embodiment of the present disclosure to generate a response or the like based on multimodal input can also be utilized in cases where comments corresponding to SNS posts are automatically generated or reviews for purchased items are automatically written.

[0267] (5) An embodiment of recognizing an entity referred to by a user based on the context of the screen.

[0268] According to one embodiment of the present disclosure, the electronic device (1000) can accurately identify an object pointed to by a user based on the context of the screen. For example, when a user inputs a question or request using terms that are difficult to specify, such as "here," "this," or "that," the electronic device (1000) can identify the intended object of the user by understanding the context of the screen and respond to the question or request.

[0269] 1) Search for the route to the cafe searched by the user.

[0270] FIG. 17 is a diagram illustrating a process in which an electronic device, according to one embodiment of the present disclosure, identifies an object pointed to by a user based on the context of the screen and responds to the user's request. The first screen (1710) of FIG. 17 may be displayed on the input / output interface (1200) of the electronic device (1000).

[0271] The first screen (1710) of FIG. 17 displays the results of a search for a cafe by user A. When user A searches for AA Cafe through a search app or map app installed on an electronic device (1000), the location of AA Cafe, cafes located nearby, and detailed information about AA Cafe may be displayed together, as in the first screen (1710).

[0272] Referring to FIG. 17, the upper half of the first screen (1710) displays a map, on which the locations of AA Cafe (1711), BB Cafe (1712), and CC Cafe (1713) are displayed. The lower half of the first screen (1710) displays detailed information (e.g., business hours, price range, etc.) (1714) about AA Cafe (1711).

[0273] When a user A inputs a voice command to the electronic device (1000) such as “Give me directions to here” while the first screen (1710) is displayed on the electronic device (1000), the electronic device (1000) must determine where the “here” intended by the user A is in order to process the request. The map on the first screen (1710) displays several cafes (1711, 1712, 1713), but the user A has not specified which cafe it is, so the electronic device (1000) cannot determine the destination indicated by the user A based on the voice command alone.

[0274] The electronic device (1000) can determine the intended destination of user A by considering the overall context of the first screen (1710). The electronic device (1000) can check and analyze the entire image of the first screen (1710) and determine the intended destination of user A based on the result. Since the lower half of the first screen (1710) displays detailed information (1714) about AA Cafe (1711), the electronic device (1000) can determine that user A searched for AA Cafe (1711) and therefore AA Cafe (1711) is the intended destination of user A.

[0275] The electronic device (1000) can obtain context information (1730) of the 4W1R structure as follows based on the result of determining that the intended destination of user A is AA Cafe (1711).

[0276] Who: User A

[0277] What: Directions to AA Cafe

[0278] Where: AA Cafe

[0279] When: Present

[0280] Reason: I want to search for a path

[0281] The electronic device (1000) can generate a 4W1R vector based on the context information (1730) of the 4W1R structure acquired as above, and search for a path to the AA cafe based on the 4W1R vector.

[0282] The 4W1R generator (120) can generate a 4W1R vector based on context information (1730) of the 4W1R structure and transmit it to the semantic reasoner (130). Since the 4W1R vector includes information indicating a desire to search for a path to the AA cafe, the semantic reasoner (130) can request the preferred action generator (140) to search for a path to the AA cafe.

[0283] If there is a navigation app that user A frequently uses when searching for a route, the preferred action generator (140) can generate a deep link to run the app and transmit it to the agent model classifier (150) along with information for identifying the search target (AA Cafe).

[0284] The agent model classifier (150) can select an agent corresponding to a search category from among the agents included in the agent PEFT model (200) and transmit a deep link for executing a navigation app to the selected agent along with information about the search target. The selected agent can execute the navigation app via the received deep link to search for a route to the AA cafe and display the search results on the screen of the electronic device (1000).

[0285] 2) Confirm and save the appointment time set by the user based on the chat message

[0286] FIG. 18 is a diagram illustrating a process in which an electronic device, according to one embodiment of the present disclosure, recognizes an appointment time indicated by a user based on the context of the screen and sets an alarm. The first screen (1810) of FIG. 18 may be displayed on the input / output interface (1200) of the electronic device (1000).

[0287] The first screen (1810) of Figure 18 displays chat messages exchanged between User A and another user. Referring to Figure 18, User A exchanged chat messages to set an appointment time, and multiple times (7:00, 7:30, 8:00) were mentioned during the process of setting the appointment time.

[0288] When a user A inputs a voice command to the electronic device (1000) such as “Set an alarm for the appointment time” while the first screen (1810) is displayed on the electronic device (1000), the electronic device (1000) must determine the “appointment time” indicated by the user A in order to process the request. The chat message on the first screen (1810) includes multiple times (7:00, 7:30, 8:00), and since the user A did not clearly state which time among them is the appointment time, the electronic device (1000) cannot determine the appointment time indicated by the user A based on the voice command alone.

[0289] The electronic device (1000) can determine the intended appointment time of user A by considering the overall context of the first screen (1810). By analyzing the content of the chat message, the electronic device (1000) can determine that user A engaged in a conversation with another user to set an appointment time, and that the ultimately determined appointment time is 7:30 AM (1822) in the morning (1821).

[0290] The electronic device (1000) can obtain context information (1830) of the 4W1R structure as follows based on the result of determining that the appointment time intended by user A is 7:30 AM.

[0291] Who: User A and User B

[0292] What: Meeting appointment

[0293] Where: N / A

[0294] When: 7:30 a.m.

[0295] Reason: I want to set an alarm for the appointment time.

[0296] The electronic device (1000) can generate a 4W1R vector based on the context information (1830) of the 4W1R structure acquired as above, and set an alarm at the appointed time (7:30 AM) based on the 4W1R vector.

[0297] 3) Check and save the information displayed on the screen in chunks.

[0298] FIG. 19 is a diagram illustrating a process in which an electronic device, according to one embodiment of the present disclosure, checks and stores information in chunks based on the context of the screen. The first screen (1910) of FIG. 19 may be displayed on the input / output interface (1200) of the electronic device (1000).

[0299] The first screen (1910) of Fig. 19 is the screen displayed when accessing the homepage promoting the exhibition. When user A accessed the homepage of the exhibition called 'AI EXPO KOREA' through an electronic device (1000), the first screen (1910) was displayed on the electronic device (1000).

[0300] The first screen (1910) displays various information related to the exhibition. The first area (1911) displays the name of the exhibition, the second area (1912) indicates the exhibition period, and the third area (1913) indicates the location of the exhibition.

[0301] When a user A inputs a voice command, "Register a schedule to the calendar," into the electronic device (1000) while the first screen (1910) is displayed on the electronic device (1000), the electronic device (1000) must obtain information regarding the "schedule" indicated by the user A in order to process the request. The first screen (1910) displays information in multiple areas (1911, 1912, 1913), and if the electronic device (1000) cannot connect these pieces of information, the schedule information may be stored inaccurately. For example, the duration of an exhibition may be accurately stored, but the name or location of the exhibition may be incorrectly stored.

[0302] An electronic device (1000) according to one embodiment of the present disclosure can increase the accuracy of information displayed in multiple areas (1911, 1912, 1913) by checking the information in chunk units. The electronic device (1000) can recognize information displayed on a first screen (1910) as information related to a single entity (e.g., an event, an exhibition, etc.) by linking the information with each other.

[0303] The electronic device (1000) can obtain context information (1930) of a 4W1R structure as follows based on the exhibition-related information obtained from the first screen (1910).

[0304] Who: User A

[0305] What: AI EXPO KOREA

[0306] Where: Hall A COEX

[0307] When: May 14 (Wed) - May 16 (Fri)

[0308] Reason: I want to save the schedule to my calendar.

[0309] The electronic device (1000) can generate a 4W1R vector based on the context information (1930) of the 4W1R structure acquired as described above, and register information about the exhibition (name, location, period) in the calendar app based on the 4W1R vector.

[0310] (6) Embodiment of controlling the device

[0311] According to one embodiment of the present disclosure, an electronic device (1000) can control another device based on user input information. For example, the electronic device (1000) can control the operation of a home appliance, such as an air conditioner, based on a knowledge graph stored in a database (300) and user input. FIG. 20 is a diagram illustrating a process of an electronic device controlling an air conditioner based on user input according to one embodiment of the present disclosure.

[0312] Referring to FIG. 20, when a voice of user A saying “It’s hot” in the living room is input to an electronic device (1000), the electronic device (1000) can obtain context information (2020) of a 4W1R structure as follows based on the user input information (user A’s voice input).

[0313] Who: User A

[0314] What: Air conditioner

[0315] Where: Living room

[0316] When: Present

[0317] Reason: I feel hot

[0318] When the 4W1R generator (120) generates a 4W1R vector based on the context information (2020) of the 4W1R structure and transmits it to the semantic reasoner (130), the semantic reasoner (130) can determine that user A currently feels that the temperature in the living room is hot based on the 4W1R vector. The semantic reasoner (130) can determine that the air conditioner (2010) located in the living room needs to be controlled, and can check whether the information required for controlling the air conditioner (2010) is stored in the knowledge graph of the database (300).

[0319] If a representative 4W1R vector corresponding to an air conditioner (2010) is stored in the knowledge graph, the semantic reasoner (130) can extract context information (2030) of the following 4W1R structure from the representative 4W1R vector.

[0320] Who: User A

[0321] What: Air conditioner

[0322] Where: Living room

[0323] When: N / A

[0324] Reason: Set temperature 24 degrees

[0325] The context information (2030) of the above 4W1R structure includes the meaning that user A prefers to adjust the set temperature of the air conditioner (2010) located in the living room to 24 degrees. The semantic reasoner (130) can adjust the set temperature of the air conditioner (2010) to 24 degrees based on the representative 4W1R vector corresponding to the air conditioner (2010).

[0326] According to one embodiment of the present disclosure, a representative 4W1R vector corresponding to an air conditioner (2010) can be generated or updated based on the history of user A controlling the air conditioner (2010).

[0327] According to one embodiment of the present disclosure, an electronic device (1000) can store information related to device control, such as an air conditioner, in a knowledge graph when a user directly controls the device. For example, the electronic device (1000) can obtain context information in a 4W1R structure from the user's action of controlling the device, convert the obtained context information into a 4W1R vector, and store it in a knowledge graph. FIG. 21 is a diagram illustrating a knowledge graph updated by an electronic device according to one embodiment of the present disclosure based on the user's action of controlling an air conditioner.

[0328] Assume that User A enters home at 6:00 PM and sets the temperature of the air conditioner in the living room to 24 degrees. The electronic device (1000) can obtain the following 4W1R structured context information from User A's actions.

[0329] Who: User A

[0330] What: Air Conditioner (Device Information)

[0331] Where: Living room

[0332] When: 6 p.m.

[0333] Reason: Status information (surrounding environment data)

[0334] The electronic device (1000) can generate a knowledge graph as illustrated in FIG. 21 based on the 4W1R vector converted from the acquired context information.

[0335] Referring to Fig. 21, information on 4W1R items is stored in each node, and related properties are added to the node corresponding to What and the node corresponding to Reason. Air conditioner device information is added to the node corresponding to What, and information on the surrounding environment is added to the node corresponding to Reason. Since information such as the user's habits, patterns, and preferences regarding air conditioner control are stored in the knowledge graph generated in this way, the electronic device (1000) can control the air conditioner according to the surrounding environment in the future.

[0336] The knowledge graph generated in this manner can be used for future device control. For example, the electronic device (1000) can control the air conditioner at optimal efficiency by considering the surrounding environment, such as temperature and humidity, and the air conditioner's performance. The electronic device (1000) can also generate a representative 4W1R vector corresponding to the air conditioner based on the knowledge graph.

[0337] (7) Example of searching for photos

[0338] According to one embodiment of the present disclosure, an electronic device (1000) can generate a knowledge graph corresponding to stored photos and search for photos based on the knowledge graph. FIG. 22 is a diagram illustrating a process in which an electronic device according to one embodiment of the present disclosure generates a knowledge graph when storing photos and searches for photos at a user's request.

[0339] When user A takes a photo (2210) using an electronic device (1000), the electronic device (1000) can obtain context information (2221) of a 4W1R structure from the taken photo. For example, the electronic device (1000) can determine that the person is user A (Who - subject information) by recognizing the face (2211) of the person included in the photo (2210). The electronic device (1000) can determine the location and time (Where and When - location information and time information) at which the photo (2210) was taken based on GPS information and time information. In addition, the electronic device (1000) can determine the information (reason information) of the Reason item as 'taking a photo to leave a memory after hiking' based on the situation in which the photo (2210) was taken. As a result, the electronic device (1000) can obtain context information (2221) of the 4W1R structure as follows.

[0340] Who: User A

[0341] What: Photo

[0342] Where: Cheonggyesan

[0343] When: October 12, 2024, 11:00 AM

[0344] Reason: To take pictures to preserve memories after hiking.

[0345] The 4W1R generator (120) generates a 4W1R vector based on context information (2221) of the 4W1R structure as above and transmits it to the semantic reasoner (130), and the semantic reasoner (130) can generate a knowledge graph (2220) based on the received 4W1R vector.

[0346] When user A inputs a voice command to the electronic device (1000) such as "Search for photos taken while hiking," the electronic device (1000) can check the knowledge graphs stored in the database (300) and determine that the photo (2210) was taken while hiking based on the Where item (location information) and the Reason item (reason information) of the 4W1R vector stored in the knowledge graph (2220). The electronic device (1000) can display the photo (2210) on the screen in response to the command from user A.

[0347] The electronic device (1000) can create a corresponding knowledge graph according to the method described above when storing all photos acquired through various methods as well as photos taken directly.

[0348] 6. Describe the process by referring to the flowcharts.

[0349] Referring to the flowcharts of FIGS. 23 and 24, a method for managing a dynamic database for providing personalized services by an electronic device according to embodiments of the present disclosure will be described. Referring to the flowcharts of FIGS. 25 to 28, a method for providing personalized services by an electronic device according to embodiments of the present disclosure based on a dynamic database will be described.

[0350] Since the steps included in the flowcharts of FIGS. 23 to 28 are performed by the electronic device (1000) of FIGS. 4 and 5, the contents described above with reference to FIGS. 1 to 22 may be equally applied to FIGS. 23 to 28 even if omitted below.

[0351] Referring to FIG. 23, at step 2301, the electronic device can obtain user input information for the electronic device. For example, the electronic device can store the user's input for the electronic device or obtain the user input information by checking the screen displayed on the electronic device. The user input information may include at least one of information entered by the user into the electronic device, information regarding the user's usage pattern, information regarding actions performed by the user via the electronic device, or information displayed on the screen of the electronic device while the user is using the electronic device.

[0352] In step 2302, the electronic device can obtain context information for providing personalized services from user input information. According to one embodiment of the present disclosure, the context information may include non-explicit information that can be determined from the user input information. The context information may be information used for knowledge graph-based inference to provide personalized services. The detailed steps included in step 2302 are illustrated in FIG. 24.

[0353] Referring to FIG. 24, at step 2401, the electronic device may obtain one or more explicit pieces of information from user input information. According to one embodiment of the present disclosure, the explicit pieces of information may include information about who, what, where, and when.

[0354] At step 2402, the electronic device can use the generative model to determine implicit information based on at least one of the contextual information and explicit information of the user input information. According to one embodiment of the present disclosure, the implicit information may include information regarding the reason. The electronic device can determine the implicit information included in the contextual information based on the overall context of the user input information.

[0355] Returning to Figure 23, at step 2303, the electronic device can update the dynamic database based on context information. According to one embodiment of the present disclosure, the electronic device can store only the most recent information for each item of context information in the dynamic database. The dynamic database may be a knowledge graph including nodes corresponding to each item of context information. According to one embodiment of the present disclosure, the electronic device can store information for controlling another device in the dynamic database based on its history of controlling that device.

[0356] Referring to FIG. 25, at step 2501, the electronic device can obtain user input information for the electronic device. For example, the electronic device can store the user's input for the electronic device or obtain the user input information by checking the screen displayed on the electronic device. The user input information may include at least one of information input by the user into the electronic device, information regarding the user's usage pattern, information regarding actions performed by the user via the electronic device, or information displayed on the screen of the electronic device while the user is using the electronic device.

[0357] In step 2502, the electronic device may select the type of personalized service to provide based on the context of the user input information. According to one embodiment of the present disclosure, the personalized service may include at least one of call summary, chat summary, schedule storage, device control, photo search, or automatic response generation. According to one embodiment of the present disclosure, if the user input information includes information about a new schedule, the electronic device may select a personalized service that updates the schedule in a schedule management app. According to one embodiment of the present disclosure, if the user input information includes call or chat content, the electronic device may select a personalized service that summarizes and stores the call or chat content. Detailed steps included in step 2502 are illustrated in FIG. 26.

[0358] Referring to Figure 26, in step 2601, the electronic device can understand the context of user input information in real time. Subsequently, in step 2602, the electronic device can perform knowledge graph-based reasoning based on the results of the context understanding. Finally, in step 2603, the electronic device can select the type of personalized service based on the results of the knowledge graph-based reasoning.

[0359] Returning to FIG. 25, in step 2503, the electronic device can provide the type of personalized service selected in step 2502 based on the dynamic database stored in the electronic device. According to one embodiment of the present disclosure, if the user input information includes a request for a specific entity, the electronic device can determine the entity the user is referring to based on the screen context, and then process the request using the determined entity. According to one embodiment of the present disclosure, if the user input information includes a request for control of another device, the electronic device can access the dynamic database, obtain control information corresponding to the other device, and then control the other device based on the control information. Detailed steps included in step 2503 are illustrated in FIGS. 27 and 28, respectively.

[0360] Referring to FIG. 27, at step 2701, the electronic device can determine whether there is a user-preferred app related to the type of personalized service selected at step 2502.

[0361] If the determination reveals a user's preferred app, the electronic device can generate a deep link to launch the user's preferred app at step 2702. Subsequently, in step 2703, the electronic device can transmit the deep link to the agent model. Finally, in step 2704, the agent model of the electronic device can launch the user's preferred app via the deep link, thereby providing a personalized service.

[0362] If the determination results indicate that the user does not have a preferred app, the electronic device may recommend an app related to the selected personalized service to the agent model at step 2705. In step 2706, the agent model of the electronic device may use the recommended app to provide the personalized service.

[0363] Referring to Figure 28, in step 2801, the electronic device can obtain context information from user input information. Next, in step 2802, the electronic device can convert the context information into an embedding vector. Finally, in step 2803, the electronic device can provide a personalized service based on the results of comparing the embedding vector with a dynamic database.

[0364] According to one embodiment of the present disclosure, the electronic device (1000) may provide a personalized service to a user by comprehensively considering information collected through various apps installed on the electronic device (1000). For example, the electronic device (1000) may obtain context information from user input information collected through the app, and classify and store the context information based on reason information included in the context information (information regarding the reason for an event corresponding to the context information). The electronic device (1000) may respond to a user's question or request by considering context information classified into the same category together.

[0365] There may be an event corresponding to user input information or context information obtained from the user input information, and the context information may include information regarding the reason for the event (reason information). The reason information may be determined from the user input information or context information.

[0366] 7. An embodiment of classifying and storing context information based on reason information.

[0367] FIGS. 29 to 35 are drawings for explaining a method of classifying and storing information acquired through multiple apps in an electronic device based on the reason for an event according to one embodiment of the present disclosure.

[0368] Referring to Figure 29, users A and C make calls using their respective electronic devices (1000, 3000). Figure 29 illustrates the call between users A and C.

[0369] The electronic device (1000) can obtain context information from the call content between User A and User C, and create or update a knowledge graph based on the obtained context information. The electronic device (1000) can also save a schedule in a calendar app based on the updated knowledge graph.

[0370] The electronic device (1000) can obtain the call content between User A and User C as user input information, obtain context information from the obtained user input information (call content), and store the obtained context information in a knowledge graph. The electronic device (1000) can obtain context information in a 4W1R structure from the call content.

[0371] Referring to the call content illustrated in Figure 29, User A is talking to User C about User C's wedding, mentioning the location and time of the wedding. From the call content, it can be seen that User C is planning to get married at Hotel R on January 15th of next year (2025). Accordingly, the electronic device (1000) can obtain context information in a 4W1R structure from the call content as follows.

[0372] Who: User C

[0373] What: Wedding

[0374] Where: R Hotel

[0375] When: January 15, 2025

[0376] Reason: User C's wedding

[0377] Among the information contained in the above context information, 1R (User C's wedding) corresponds to reason information. Reason information can be included in the previously described implicit information and can be determined from user input information (call content).

[0378] According to one embodiment of the present disclosure, a context reasoner (110) of an electronic device (1000) can summarize a call content, obtain context information of a 4W1R structure as described above from the summarized call content, and transmit the obtained context information to a 4W1R generator (120). The 4W1R generator (120) can generate a 4W1R vector by performing embedding on the received context information of a 4W1R structure, and transmit the generated 4W1R vector to a semantic reasoner (130). The semantic reasoner (130) can generate a knowledge graph based on the received 4W1R vector.

[0379] The first knowledge graph (3010) generated by the semantic reasoner (130) based on the context information obtained from the call content of FIG. 29 is illustrated in FIG. 30. Referring to FIG. 30, Who (User C) (subject information), What (wedding) (event information), Where (Hotel R) (location information), When (January 15, 2025) (time information), and Reason (User C's wedding) (reason information) are stored in the nodes of the first knowledge graph (3010), respectively.

[0380] The semantic reasoner (130) may determine that a schedule needs to be saved in the calendar app (3020) based on the first knowledge graph (3010) and request the preference action generator (140) to save the schedule in the calendar app (3020). The preference action generator (140) may generate a deep link for executing the calendar app (3020) and transmit it to the agent model classifier (150) together with information included in the first knowledge graph (3010). The agent model classifier (150) may select an agent corresponding to a schedule management category among a plurality of agents (210, 220, 230) included in the agent PEFT model (200) and transmit the deep link for executing the calendar app (3020) and information included in the first knowledge graph (3010) to the selected agent. The selected agent can run the calendar app (3020) and save the wedding schedule of user C included in the first knowledge graph (3010).

[0381] When User C's wedding schedule is saved in the calendar app (3020), the electronic device (1000) can determine whether the wedding schedule overlaps with the schedule already saved in the calendar app (3020), and reflect the result when providing personalized services. For example, if the business trip schedule already saved in the calendar app (3020) overlaps with User C's wedding schedule, the electronic device (1000) can provide User A with a notification regarding the overlapping schedule.

[0382] The semantic reasoner (130) can classify and store the first knowledge graph (3010) based on reason information (1R) when storing it in the database (300). For example, the semantic reasoner (130) can link context information with the same reason information and store it as a knowledge graph. Alternatively, the semantic reasoner (130) can link context information that is similar beyond a certain standard, even if the reason information is not identical, and store it as a knowledge graph. In summary, the semantic reasoner (130) can classify and store context information with matching reason information into the same category.

[0383] When storing the first knowledge graph (3010) illustrated in FIG. 30 in the database (300), the semantic reasoner (130) can connect it with another knowledge graph that includes 1R identical to or similar to 'User C's wedding'.

[0384] A method of connecting information collected through various apps installed on an electronic device (1000) based on reason information is described in more detail below with reference to FIGS. 31 to 35.

[0385] Referring to Figure 31, users A and B make calls using their respective electronic devices (1000, 2000). Figure 31 illustrates the call between users A and B.

[0386] The electronic device (1000) can obtain the call content between user A and user B as user input information, obtain context information from the obtained user input information (call content), and store the obtained context information in a knowledge graph. The electronic device (1000) can obtain context information in a 4W1R structure from the call content.

[0387] Referring to the call content illustrated in Figure 31, User A is requesting User B to deliver a congratulatory gift. Referring to the call content, it can be seen that User A is unable to attend User C's wedding due to a business trip and is requesting User B to deliver the congratulatory gift. The electronic device (1000) can obtain context information in a 4W1R structure from the call content as follows.

[0388] Who: User B

[0389] What: Request for congratulatory money

[0390] Where: N / A

[0391] When: January 15, 2025

[0392] Reason: Unable to attend User C's wedding due to business trip

[0393] According to one embodiment of the present disclosure, a context reasoner (110) of an electronic device (1000) can summarize a call content, obtain context information of a 4W1R structure as described above from the summarized call content, and transmit the obtained context information to a 4W1R generator (120). The 4W1R generator (120) can generate a 4W1R vector by performing embedding on the received context information of a 4W1R structure, and transmit the generated 4W1R vector to a semantic reasoner (130). The semantic reasoner (130) can generate a knowledge graph based on the received 4W1R vector.

[0394] A second knowledge graph (3210) generated by the semantic reasoner (130) based on the context information obtained from the conversation content of FIG. 31 is illustrated in FIG. 32. Referring to FIG. 32, the nodes of the second knowledge graph (3210) each store Who (User B) (subject information), What (request for congratulatory money delivery) (event information), Where (N / A) (location information), When (January 15, 2025) (time information), and Reason (cannot attend User C's wedding due to business trip) (reason information).

[0395] The semantic reasoner (130) can classify and store the second knowledge graph (3210) in the database (300) based on the reason information (1R). The reason information of the second knowledge graph (3210) includes a common element called 'User C's wedding' and thus matches the reason information of the first knowledge graph (3010). Therefore, the semantic reasoner (130) can store the second knowledge graph (3210) by connecting it with the first knowledge graph (3010).

[0396] FIG. 33 is a diagram illustrating a situation in which a user transfers a congratulatory gift requested by the user to another person via an electronic device according to one embodiment of the present disclosure. If User A transfers 100,000 won to User B on December 20, 2024, via a mobile banking app installed on the electronic device (1000), User A's electronic device (1000) can obtain user input information (100,000 won transferred to User B on December 20, 2024) from the mobile banking app.

[0397] The electronic device (1000) can acquire context information from user input information acquired through a mobile banking app and store the context information in a knowledge graph. At this time, the context information can be acquired by considering information acquired through other apps. For example, the electronic device (1000) can acquire context information from user input information (e.g., remit 100,000 won to user B on December 20, 2024) by considering information included in the second knowledge graph (3210) of FIG. 32, which is already stored in the database (300).

[0398] Referring to the second knowledge graph (3210) of FIG. 32, the context reasoner (110) of the electronic device (1000) can determine that user A has asked user B to send 100,000 won as a wedding gift for user C. In this situation, since user A has sent 100,000 won to user B, the context reasoner (110) can determine that the 100,000 won sent to user B is a wedding gift for user C. The context reasoner (110) can obtain context information of the 4W1R structure as follows.

[0399] Who: User B

[0400] What: Send 100,000 won

[0401] Where: N / A

[0402] When: December 20, 2024

[0403] Reason: Request for wedding congratulatory money for User C

[0404] When the context reasoner (110) obtains context information of the above 4W1R structure and transmits it to the 4W1R generator (120), the 4W1R generator (120) can generate a 4W1R vector by performing embedding on the received context information of the 4W1R structure and transmit the generated 4W1R vector to the semantic reasoner (130). The semantic reasoner (130) can generate a knowledge graph based on the received 4W1R vector.

[0405] In the situation illustrated in FIG. 33, the third knowledge graph (3410) generated by the semantic reasoner (130) is illustrated in FIG. 34. Referring to FIG. 34, the nodes of the third knowledge graph (3410) each store Who (User B) (subject information), What (send 100,000 won) (event information), Where (N / A) (location information), When (December 20, 2024) (time information), and Reason (request for wedding congratulatory money for User C) (reason information).

[0406] When storing the third knowledge graph (3410) in the database (300), the semantic reasoner (130) can connect it to another knowledge graph based on 1R (request for delivery of wedding gift money for user C). Since the first knowledge graph (3010) of FIG. 30 and the second knowledge graph (3210) of FIG. 32 are similar to 1R of the third knowledge graph (3410) in that both 1R includes a common element called 'user C's wedding', the semantic reasoner (130) can connect and store the third knowledge graph (3410) to the first knowledge graph (3010) and the second knowledge graph (3210).

[0407] FIG. 35 is a diagram illustrating an example of an electronic device according to one embodiment of the present disclosure classifying and storing context information based on reason information included in the context information.

[0408] When storing context information in a knowledge graph, the electronic device (1000) can classify and store the context information based on the reason information included in the context information. As a result, context information containing identical or similar reason information (e.g., 1R) can be classified into the same category and stored in the database (300).

[0409] Referring to Figure 35, the first knowledge graph (3010), the second knowledge graph (3210), and the third knowledge graph (3410), which include reason information related to 'User C's wedding', are classified and stored as a first category (3510). Similarly, knowledge graphs including reason information related to 'business trip' may be classified and stored as a second category (3520), and knowledge graphs including reason information related to 'travel to Australia' may be classified and stored as a third category (3530).

[0410] The electronic device (1000) can provide personalized services to users based on knowledge graphs (context information) stored and categorized into multiple categories. For example, when user A inputs a question into the electronic device (1000), the electronic device (1000) can generate an answer by considering knowledge graphs within the same category.

[0411] If user A asks the electronic device (1000) “Why did I send 100,000 won to B?”, the electronic device (1000) can generate an answer such as “I cannot attend C’s wedding due to overlapping business trips, so I sent 100,000 won to B and asked him to deliver the congratulatory money on my behalf” by considering the information stored in the knowledge graphs (3010, 3210, 3410) included in the first category (3510). In this way, the electronic device (1000) can answer the user’s question or process the user’s request by considering the knowledge graphs included in each category (3510, 3520, 3530).

[0412] 8. An embodiment of providing personalized services based on contextual information classified based on the reason for the event.

[0413] FIG. 36 and FIG. 37 are diagrams for explaining a method of classifying context information based on the reason for an event in an electronic device according to one embodiment of the present disclosure and providing a personalized service based on the classified context information.

[0414] Referring to FIG. 36, when user A saves a wedding dress fitting schedule in a calendar app installed on an electronic device (1000), the electronic device (1000) can obtain first context information (3610) of the following 4W1R structure.

[0415] Who: User A

[0416] What: Wedding dress fitting

[0417] Where: N / A

[0418] When: November 30, 2024

[0419] Reason: My wedding

[0420] When user A chats with user B through an electronic device (1000) and makes an appointment to deliver an invitation, the electronic device (1000) can obtain second context information (3620) of the following 4W1R structure.

[0421] Who: Users A and B

[0422] What: Meeting

[0423] Where: COEX

[0424] When: December 5, 2024

[0425] Reason: To give out my wedding invitations

[0426] When user A receives a message guiding a wedding photo shoot schedule through an electronic device (1000), the electronic device (1000) can obtain third context information (3630) of the following 4W1R structure.

[0427] Who: User A

[0428] What: Wedding photography

[0429] Where: K Studio

[0430] When: December 10, 2024

[0431] Reason: Preparing for my wedding

[0432] The electronic device (1000) can classify and store context information based on reason information included in the context information. Since the reason information of the first context information (3610), the second context information (3620), and the third context information (3630) includes the common element of "my wedding," the electronic device (1000) can classify and store the first context information (3610), the second context information (3620), and the third context information (3630) into the same category. For example, the electronic device (1000) can store the first context information (3610), the second context information (3620), and the third context information (3630) by linking them to each other.

[0433] The electronic device (1000) can provide a personalized service to user A by comprehensively considering context information classified into the same category.

[0434] According to one embodiment of the present disclosure, an electronic device (1000) may provide a to-do list including events corresponding to contextual information classified into the same category. Referring to FIG. 37, the to-do list (3710) includes events falling under the category "My Wedding."

[0435] According to one embodiment of the present disclosure, the electronic device (1000) automatically stores schedules corresponding to context information in a calendar app, and may store the schedules in a manner that is categorized by the category to which the context information belongs. For example, the electronic device (1000) may automatically store schedules corresponding to the second context information (3620) and the third context information (3630) in a calendar app, and may indicate that the schedules are related to "My Wedding" through a title or color, etc.

[0436] The electronic device (1000) can also provide personalized services by using interconnected context information in various other ways.

[0437] 9. Describe the process using the flowchart.

[0438] FIG. 38 is a flowchart illustrating a method for managing a dynamic database for providing personalized services according to one embodiment of the present disclosure.

[0439] Referring to FIG. 38, in step 3801, the electronic device can obtain user input information through at least one app installed on the electronic device. For example, the electronic device can store user input received while the user is using the app, or obtain user input information by checking the screen displayed on the electronic device while the user is using the app. In this case, the user input information may include at least one of information input by the user into the electronic device, information about the user's usage pattern of the electronic device, information about actions performed by the user through the electronic device, or information displayed on the screen of the electronic device while the user is using the electronic device.

[0440] At step 3802, the electronic device can obtain contextual information for providing personalized services from user input information. The contextual information may be used for knowledge graph-based inference to provide personalized services.

[0441] According to one embodiment of the present disclosure, context information may include reason information, and the reason information may include information regarding the reason for an event corresponding to the context information. The reason information may be inferred from user input information corresponding to the context information.

[0442] According to one embodiment of the present disclosure, context information may include explicit and implicit information. Implicit information may be inferred from user input information and may include reason information.

[0443] According to one embodiment of the present disclosure, an electronic device can obtain one or more explicit pieces of information from user input information, and determine reason information based on at least one of the context of the user input information or the obtained explicit pieces of information using a generative model.

[0444] According to one embodiment of the present disclosure, explicit information may include information about who, what, where, and when. Reason information may include information about why an event corresponding to the context information occurred.

[0445] In step 3803, the electronic device can classify the context information and store it in a dynamic database based on the reason information included in the context information. According to one embodiment of the present disclosure, the electronic device can store the context information by linking it to at least one piece of context information previously stored in the database based on the reason information.

[0446] If the first context information obtained from the user input information includes the first reason information, and the database already stores second context information including the second reason information, the electronic device can compare the first reason information and the second reason information, and if the first reason information and the second reason information match each other, store the first context information by linking it to the second context information.

[0447] After step 3803, the electronic device may provide a personalized service to the user based on the interconnected contextual information. For example, the electronic device may provide a personalized service by considering a single piece of contextual information and at least one piece of contextual information linked to that piece of contextual information.

[0448] According to one embodiment of the present disclosure, an electronic device may store in memory a fine-tuned collaborative Parameter-Efficient Fine-Tuning (PEFT) model to personalize the use of the electronic device based on a dynamic knowledge graph stored in a dynamic database. The PEFT model may include a master PEFT model and an agent PEFT model. A personalized service may be performed by determining one or more instructions to be performed based on the master PEFT model and executing the personalized service based on the one or more instructions through the agent PEFT model.

[0449] A method for managing a dynamic database for providing a personalized service according to one embodiment of the present disclosure may include the steps of obtaining user input information for an electronic device, obtaining context information related to the personalized service from the user input information, and updating a dynamic database based on the context information. The context information may include implicit information that can be determined from the user input information.

[0450] According to one embodiment, the context information may be information used in knowledge graph based reasoning to provide the personalized service.

[0451] According to one embodiment, the step of obtaining the context information may include the step of obtaining one or more explicit information from the user input information and the step of determining the non-explicit information based on at least one of the context determined from the user input information or the one or more explicit information based on a generation model.

[0452] According to one embodiment, the non-explicit information included in the context information may be determined based on the context indicated by the user input information.

[0453] According to one embodiment, the one or more explicit pieces of information may include event information and at least one of subject information, location information, or time information, and the non-explicit information may include information indicating a reason for an event corresponding to the event information.

[0454] According to one embodiment, the step of updating the dynamic database may store the latest information corresponding to the subject information, the event information, the location information, the time information, and the reason information in the dynamic database.

[0455] According to one embodiment, the dynamic database may be a knowledge graph including nodes corresponding to the subject information, the event information, the location information, the time information, and the reason information.

[0456] According to one embodiment, the step of updating the dynamic database may store information for controlling another device in the dynamic database based on a usage history of the electronic device controlling the other device.

[0457] According to one embodiment, the user input information may include at least one of information input by the user into the electronic device, information based on a pattern of the user using the electronic device, information based on an action performed by the user through the electronic device, or information displayed on a screen of the electronic device based on the electronic device by the user.

[0458] According to one embodiment, the step of obtaining the user input information may obtain the user input information by storing the user's input for the electronic device or checking a screen displayed on the electronic device.

[0459] An electronic device according to one embodiment of the present disclosure includes at least one processor including a memory storing at least one instruction and a processing circuit, wherein the at least one processor executes, alone or in cooperation, a program stored in the memory or at least one instruction, thereby obtaining user input information for the electronic device, obtaining context information related to the personalized service from the user input information, and wherein the context information includes implicit information determined from the user input information, and updating a dynamic database based on the context information.

[0460] According to one embodiment, the context information may be information used in knowledge graph based reasoning to provide the personalized service.

[0461] According to one embodiment, when obtaining the context information, the electronic device may obtain one or more explicit information from the user input information, and then, based on a generation model, determine the non-explicit information based on at least one of the context determined from the user input information or the one or more explicit information.

[0462] In one embodiment, the non-explicit information may be determined based on the context indicated by the user input information.

[0463] According to one embodiment, the one or more explicit pieces of information may include event information and at least one of subject information, location information, or time information, and the non-explicit information may include reason information indicating a reason for an event corresponding to the event information.

[0464] According to one embodiment, when updating the dynamic database, the electronic device may store only the latest information corresponding to the subject information, the event information, the location information, the time information, and the reason information in the dynamic database.

[0465] According to one embodiment, the dynamic database may be a knowledge graph including nodes corresponding to the subject information, the event information, the location information, the time information, and the reason information.

[0466] According to one embodiment, the electronic device may store information for controlling another device in the dynamic database based on a usage history of the electronic device controlling the other device when updating the dynamic database.

[0467] According to one embodiment, the user input information may include at least one of information input by the user into the electronic device, information based on a pattern of the user using the electronic device, information based on an action performed by the user through the electronic device, or information displayed on a screen of the electronic device based on the electronic device by the user.

[0468] A method for managing a dynamic database for providing a personalized service according to one embodiment of the present disclosure may include the steps of: obtaining user input information through at least one app installed on an electronic device; obtaining context information from the user input information, the context information being related to the personalized service and including information indicating a reason for an event; classifying the context information based on reason information included in the context information to obtain classified context information, storing the classified context information in a dynamic database; and executing the personalized service on the electronic device based on the context information and at least one other context information connected to the context information.

[0469] According to one embodiment, the step of obtaining the context information and storing it in the dynamic database may include obtaining linked context information by identifying at least one connection between the context information and at least one other context information based on the reason information, and storing the linked context information in the dynamic database.

[0470] According to one embodiment, when first context information obtained from the user input information includes first reason information and second context information including second reason information is stored in the dynamic database, the step of classifying the context information and storing it in the dynamic database may include the step of comparing the first reason information with the second reason information, and when it is confirmed that the first reason information and the second reason information correspond to each other, the step of obtaining the linked context information by linking the first context information to the second context information, and storing the linked context information in the dynamic database.

[0471] According to one embodiment, the reason information may be determined based on the content of user input information corresponding to the context information.

[0472] According to one embodiment, the context information includes explicit information and implicit information determined from the user input information, and the implicit information may include the reason information.

[0473] According to one embodiment, the step of obtaining the context information may include the step of determining the context from the user input information, the step of obtaining one or more explicit pieces of information from the user input information, and the step of determining the reason information based on at least one of the context or the obtained explicit pieces of information based on a generation model.

[0474] According to one embodiment, the explicit information may include event information and at least one of subject information, location information, or time information, and the reason information may include information indicating an event corresponding to the event information and a reason for the event.

[0475] According to one embodiment, the user input information may include at least one of information input by the user into the electronic device, information based on a usage pattern of the electronic device by the user, information based on an operation performed by the user on the electronic device, or information displayed on a screen of the electronic device based on the use of the electronic device by the user.

[0476] According to one embodiment, the step of obtaining the user input information may include storing the user's input for the at least one app or checking a screen displayed on the electronic device corresponding to the at least one app.

[0477] According to one embodiment, an electronic device may include a memory that stores a fine-tuned collaborative Parameter-Efficient Fine-Tuning (PEFT) model to personalize the use of the electronic device based on a dynamic knowledge graph stored in a dynamic database. The PEFT model may include a master PEFT model and an agent PEFT model. The step of executing a personalized service may include the step of determining one or more instructions to be performed based on the master PEFT model, and executing the personalized service through the agent PEFT model based on the one or more instructions.

[0478] An electronic device according to one embodiment of the present disclosure comprises at least one processor including a memory storing at least one instruction and a processing circuit, wherein the at least one processor executes a program or at least one instruction stored in the memory, alone or in cooperation, so that the electronic device obtains user input information through at least one app installed in the electronic device, obtains context information including information indicating a reason for an event and related to a personalized service from the user input information, and then classifies the context information based on reason information included in the context information to obtain classified context information, stores the classified context information in a dynamic database, and executes a personalized service on the electronic device based on the context information and at least one other context information connected to the context information.

[0479] According to one embodiment, the electronic device can obtain linked context information by identifying at least one connection between the context information and at least one other context information based on the reason information, by executing the at least one instruction alone or cooperatively by the at least one processor, and store the linked context information in the dynamic database.

[0480] According to one embodiment, when first context information obtained from the user input information includes first reason information, and second context information including second reason information is stored in the dynamic database, when the at least one instruction is executed alone or cooperatively by the at least one processor, the electronic device can obtain the linked context information by linking the first context information to the second context information when it is confirmed that the first reason information and the second reason information correspond to each other after comparing the first reason information and the second reason information, and store the linked context information in the dynamic database.

[0481] According to one embodiment, the reason information may be determined based on the content of user input information corresponding to the context information.

[0482] According to one embodiment, the context information includes explicit information and implicit information determined from the user input information, and the implicit information may include the reason information.

[0483] According to one embodiment, the electronic device can determine a context from the user input information, obtain one or more explicit pieces of information from the user input information, and then, based on a generative model, determine the reason information based on at least one of the context or the obtained explicit pieces of information, by executing the at least one instruction alone or cooperatively by the at least one processor.

[0484] In one embodiment, the explicit information may include information about who, what, where, and when (subject information, event information, location information, and time information), and the reason information may include information about why an event corresponding to the context information occurs.

[0485] According to one embodiment, the user input information may include at least one of information input by the user into the electronic device, information based on a usage pattern of the user using the electronic device, information about an action performed by the user through the electronic device, or information displayed on a screen of the electronic device based on the use of the electronic device by the user.

[0486] A method for providing a personalized service based on a dynamic database according to one embodiment of the present disclosure includes the steps of obtaining user input information for an electronic device, selecting a type of personalized service to be provided based on a context of the user input information, and providing the selected type of personalized service based on a dynamic database stored in the electronic device, wherein the dynamic database is updated based on context information obtained from the user input information, and the context information may include implicit information that can be inferred from the user input information.

[0487] According to one embodiment, the step of providing the personalized service may include the steps of determining whether there is an app preferred by the user related to the selected type of personalized service, if there is an app preferred by the user, the step of generating a deep link for executing the app, the step of transmitting the generated deep link to an agent model, and the step of the agent model executing the app preferred by the user through the deep link, thereby providing the personalized service.

[0488] According to one embodiment, the step of providing the personalized service may include the step of obtaining the context information from the user input information, the step of converting the context information into an embedding vector, and the step of providing the personalized service based on a result of comparing the embedding vector with the dynamic database.

[0489] According to one embodiment, if the user input information includes a request for a specific entity, the step of providing the personalized service may include the step of determining the entity indicated by the user based on the context of the screen of the electronic device and the step of processing the request using the determined entity.

[0490] According to one embodiment, if the user input information includes a request for control of another device, the step of providing the personalized service includes the step of accessing the dynamic database to obtain control information corresponding to the other device and the step of controlling the other device based on the control information, wherein the control information can be updated based on a history of the electronic device controlling the other device.

[0491] According to one embodiment, the personalized service may include at least one of call summary, chat summary, schedule storage, device control, photo search, or automatic response generation.

[0492] According to one embodiment, the step of selecting the type of personalized service may include the step of identifying the context of the user input information in real time, the step of performing knowledge graph-based reasoning based on the result of identifying the context, and the step of selecting the type of personalized service based on the result of performing the knowledge graph-based reasoning.

[0493] In one embodiment, the step of selecting the type of personalized service may include selecting a personalized service that updates the schedule in an app for managing the schedule if the user input information includes information about a new schedule.

[0494] According to one embodiment, the step of selecting the type of personalized service may include selecting a personalized service that summarizes and saves the call content or chat content if the user input information includes the call content or chat content.

[0495] In one embodiment, the non-explicit information included in the context information may be determined based on the overall context of the user input information.

[0496] An electronic device according to one embodiment of the present disclosure includes a memory storing a program or at least one instruction and at least one processor, and the at least one processor executes the program or at least one instruction stored in the memory, whereby the electronic device obtains user input information for the electronic device, selects a type of personalized service to be provided based on a context of the user input information, and then provides the selected type of personalized service based on a dynamic database stored in the electronic device, wherein the dynamic database is updated based on context information obtained from the user input information, and the context information may include implicit information that can be inferred from the user input information.

[0497] According to one embodiment, in providing the personalized service, the electronic device determines whether there is an app preferred by the user related to the selected type of personalized service, and if there is an app preferred by the user, generates a deep link for executing the app, transmits the generated deep link to an agent model, and then the agent model executes the app preferred by the user through the deep link, thereby providing the personalized service.

[0498] According to one embodiment, the electronic device may obtain context information from the user input information, convert the context information into an embedding vector, and provide the personalized service based on a result of comparing the embedding vector with the dynamic database in providing the personalized service.

[0499] According to one embodiment, if the user input information includes a request for a specific entity, the electronic device may determine the entity indicated by the user based on the context of the screen of the electronic device, and then process the request using the determined entity when providing the personalized service.

[0500] According to one embodiment, if the user input information includes a request for control of another device, the electronic device, in providing the personalized service, accesses the dynamic database, obtains control information corresponding to the other device, and then controls the other device based on the control information, and the control information may be updated based on a history of the electronic device controlling the other device.

[0501] According to one embodiment, the personalized service may include at least one of call summary, chat summary, schedule storage, device control, photo search, or automatic response generation.

[0502] According to one embodiment, when selecting the type of personalized service, the electronic device may determine the context of the user input information in real time, perform knowledge graph-based reasoning based on the result of determining the context, and then select the type of personalized service based on the result of performing the knowledge graph-based reasoning.

[0503] According to one embodiment, when selecting the type of personalized service, the electronic device may select a personalized service that updates the schedule in an app for managing the schedule if the user input information includes information about a new schedule.

[0504] According to one embodiment, when selecting the type of personalized service, the electronic device may select a personalized service that summarizes and stores the call content or chat content if the user input information includes the call content or chat content.

[0505] Various embodiments of the present disclosure may be implemented or supported by one or more computer programs, and the computer programs may be formed from computer-readable program code and embodied in a computer-readable medium. In the present disclosure, "application" and "program" may refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, associated data, or portions thereof suitable for implementation in computer-readable program code. "Computer-readable program code" may include various types of computer code, including source code, object code, and executable code. "Computer-readable medium" may include various types of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), a hard disk drive (HDD), a compact disc (CD), a digital video disc (DVD), or various types of memory.

[0506] Additionally, a device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, a 'non-transitory storage medium' is a tangible device and may exclude wired, wireless, optical, or other communication links that transmit temporary electrical or other signals. Meanwhile, this 'non-transitory storage medium' does not distinguish between cases where data is permanently stored in the storage medium and cases where it is temporarily stored. For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored. A computer-readable medium may be any available medium that can be accessed by a computer, and may include both volatile and non-volatile media, and removable and non-removable media. A computer-readable medium includes a medium on which data can be permanently stored and a medium on which data can be stored and later overwritten, such as a rewritable optical disk or an erasable memory device.

[0507] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0508] The above description of the present disclosure is for illustrative purposes only, and those skilled in the art will appreciate that the present disclosure can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present disclosure. For example, suitable results can be achieved even if the described techniques are performed in a different order than the described method, and / or components of the systems, structures, devices, circuits, etc. described are combined or combined in a different form than the described method, or are replaced or substituted by other components or equivalents. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined form.

[0509] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. A method for managing a dynamic database for providing personalized services, A step of obtaining user input information for an electronic device; A step of obtaining context information related to the personalized service from the user input information, wherein the context information includes implicit information determined from the user input information; and A method comprising the step of updating a dynamic database based on the context information.

2. In paragraph 1, The above context information is, A method characterized in that the information is used for knowledge graph based reasoning to provide the above personalized service.

3. In either paragraph 1 or paragraph 2, The step of obtaining the above context information is: A step of obtaining one or more explicit information from the user input information; and A method characterized by comprising a step of determining the non-explicit information based on at least one of a context determined from the user input information or the one or more explicit pieces of information based on a generative model.

4. In any one of paragraphs 1 to 3, The above non-explicit information is, A method characterized in that it is determined based on the context indicated by the user input information.

5. In any one of paragraphs 1 to 4, The one or more explicit pieces of information include event information, and at least one of subject information, location information, or time information, A method characterized in that the above non-explicit information includes reason information indicating a reason for an event corresponding to the above event information.

6. In any one of paragraphs 1 to 5, The step of updating the above dynamic database is: A method characterized in that the latest information corresponding to the subject information, the event information, the location information, the time information, and the reason information is stored in the dynamic database.

7. In any one of paragraphs 1 to 6, The above dynamic database is, A method characterized in that the knowledge graph includes nodes corresponding to the subject information, the event information, the location information, the time information, and the reason information.

8. In any one of paragraphs 1 to 7, The step of updating the above dynamic database is: A method characterized in that the electronic device stores information for controlling another device in the dynamic database based on the usage history of controlling the other device.

9. In any one of paragraphs 1 to 8, The above user input information is: A method characterized in that it includes at least one of information input by a user into the electronic device, information based on a pattern of use of the electronic device by the user, information based on an action performed by the user through the electronic device, or information displayed on a screen of the electronic device based on the electronic device by the user.

10. In any one of paragraphs 1 to 9, The step of obtaining the above user input information is: A method characterized by storing the user's input for the electronic device or obtaining the user input information by checking the screen displayed on the electronic device.

11. In electronic devices (1000), A memory (1400) in which at least one instruction is stored; and At least one processor (1300) comprising a processing circuit, The electronic device (1000) executes a program or at least one instruction stored in the memory (1400) by the at least one processor (1300) alone or in cooperation. Obtain user input information for the above electronic device (1000), Obtain context information related to the personalized service from the user input information, wherein the context information includes implicit information determined from the user input information, An electronic device that updates a dynamic database based on the above context information.

12. In paragraph 11, The above context information is, An electronic device characterized in that the information is used for knowledge graph based reasoning to provide the above personalized service.

13. In either of paragraphs 11 or 12, The electronic device (1000) is configured such that the at least one instruction is executed singly or cooperatively by the at least one processor. After obtaining one or more explicit information from the above user input information, An electronic device characterized in that the non-explicit information is determined based on a context determined from the user input information or at least one of the one or more explicit pieces of information based on a generative model.

14. In any one of paragraphs 11 to 13, The above non-explicit information is, An electronic device characterized in that it is determined based on the context indicated by the user input information.

15. In any one of paragraphs 11 to 14, The one or more explicit pieces of information include event information, and at least one of subject information, location information, or time information, An electronic device characterized in that the above-mentioned non-explicit information includes reason information indicating a reason for an event corresponding to the above-mentioned event information.

Images