Method for managing personal knowledge graph, and user device using same

Abstract

Disclosed is a method performed by an electronic device in order to manage a personal knowledge graph, the method comprising: the electronic device detecting an instance including an incomplete node having missing knowledge attribute information in a personal knowledge graph; the electronic device identifying candidate knowledge attribute information corresponding to the missing knowledge attribute information of the incomplete node; and the electronic device performing the task of completing the incomplete node on the basis of the identified candidate knowledge attribute information.

Description

How to manage a personal knowledge graph and user devices that utilize it

[0001] It relates to a method for managing a personal knowledge graph and a user device that utilizes it.

[0002] AI-based technologies are being utilized in diverse fields across industries. A variety of AI models are being developed and utilized across various sectors, and the application of AI-based solutions is rapidly increasing not only in manufacturing but also in robotics, transportation / logistics, healthcare, education, pharmaceuticals / biotechnology, and other industries. The adoption of such AI-based technologies is leading to enhanced competitiveness for both companies and countries.

[0003] To enhance the performance of AI models, knowledge bases can be used for AI model training and inference. One example of a knowledge base is a knowledge graph, which is a graph-based data structure. Interest in building and utilizing knowledge graphs is growing.

[0004] The above description is provided solely as background information to aid understanding of the present disclosure. No judgment or assertion is made as to whether any of the above content constitutes prior art to the present disclosure.

[0005] One embodiment of the present disclosure aims to address at least the problems and / or drawbacks mentioned above, and to provide at least the advantages described below. Accordingly, one embodiment of the present disclosure provides a method for managing a personal knowledge graph by completing incomplete nodes containing missing knowledge attribute information in the personal knowledge graph, and a user device utilizing the method.

[0006] One embodiment of the present disclosure will be set forth in part in the description that follows, and in part, as will be apparent from the description or may be learned by practice of the described embodiment.

[0007] According to one embodiment of the present disclosure, a method for managing a personal knowledge graph, performed by an electronic device, is provided. The method for managing a personal knowledge graph includes a step in which the electronic device detects an instance of an incomplete node having missing knowledge attribute information in the personal knowledge graph. Furthermore, the method for managing a personal knowledge graph includes a step in which the electronic device identifies candidate knowledge attribute information corresponding to the missing knowledge attribute information of the incomplete node. Furthermore, the method for managing a personal knowledge graph includes a step in which the electronic device completes the incomplete node based on the identified candidate knowledge attribute information.

[0008] According to one embodiment of the present disclosure, an electronic device is provided. The electronic device includes a memory comprising one or more storage media storing instructions, and one or more processors communicatively connected to the memory. The instructions, when individually or jointly executed by the one or more processors, cause the electronic device to detect an instance including an incomplete node having missing knowledge attribute information in a personal knowledge graph. Furthermore, the instructions, when individually or jointly executed by the one or more processors, cause the electronic device to identify candidate knowledge attribute information corresponding to the missing knowledge attribute information of the incomplete node. Furthermore, the instructions, when individually or jointly executed by the one or more processors, cause the electronic device to perform a task of completing the incomplete node based on the identified candidate knowledge attribute information.

[0009] According to one embodiment of the present disclosure, one or more non-transitory computer-readable storage media store one or more computer programs, wherein the one or more computer programs include computer-executable instructions that, when individually or jointly executed by one or more processors of an electronic device, cause the electronic device to perform operations. The one or more computer programs include computer-executable instructions that instruct the electronic device to perform a step of detecting an instance including an incomplete node having missing knowledge attribute information in a personal knowledge graph. Furthermore, the one or more computer programs include computer-executable instructions that instruct the electronic device to perform a step of identifying candidate knowledge attribute information corresponding to the missing knowledge attribute information of the incomplete node. Furthermore, the one or more computer programs include computer-executable instructions that instruct the electronic device to perform a step of completing the incomplete node based on the identified candidate knowledge attribute information.

[0010] Aspects, advantages, and key features of the present disclosure will become apparent to those skilled in the art from the following detailed description taken in conjunction with the accompanying drawings.

[0011] The above and other aspects, features, and advantages of one embodiment of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0012] FIG. 1 is a diagram illustrating an AI platform based on a personal knowledge graph according to one embodiment of the present disclosure.

[0013] FIG. 2 is a diagram illustrating an operation of a user device constructing a personalized database based on a knowledge graph according to one embodiment of the present disclosure.

[0014] FIGS. 3A and 3B are diagrams illustrating a content ontology as an example of an ontology used by a user device to build a personal knowledge graph according to one embodiment of the present disclosure.

[0015] FIG. 4 is a diagram illustrating an operation of a user device according to one embodiment of the present disclosure to provide a personal knowledge graph-based service to a user.

[0016] FIG. 5A, FIG. 5B, and FIG. 5C are diagrams illustrating a process of a user device searching for a photo using a search term based on knowledge attribute information through a personal knowledge graph-based search application according to one embodiment of the present disclosure.

[0017] FIG. 6 is a diagram illustrating an operation of a user device managing a personal knowledge graph according to one embodiment of the present disclosure.

[0018] FIG. 7 is a flowchart illustrating a method for managing a personal knowledge graph according to one embodiment of the present disclosure.

[0019] FIG. 8 is a diagram illustrating an instance including a complete node according to a given class and an instance including an incomplete node having missing knowledge attribute information in a personal knowledge graph according to one embodiment of the present disclosure.

[0020] FIG. 9 is a diagram illustrating various types of embedding vectors used to infer candidate knowledge attribute information corresponding to missing knowledge attribute information according to one embodiment of the present disclosure.

[0021] FIG. 10 is a diagram illustrating a process of completing an incomplete node by replacing the incomplete node with a completed node according to one embodiment of the present disclosure.

[0022] FIG. 11 is a diagram illustrating a process of completing an incomplete node by updating missing knowledge attribute information of an incomplete node with candidate knowledge attribute information confirmed in a completed node according to one embodiment of the present disclosure.

[0023] FIG. 12 is a block diagram illustrating a user device according to one embodiment of the present disclosure.

[0024] FIG. 13 is a block diagram illustrating the configuration and operation of a user device according to one embodiment of the present disclosure.

[0025] Throughout the drawings, the same reference numbers are used to indicate identical components.

[0026] The following description, with reference to the attached drawings, is provided to facilitate a comprehensive understanding of one embodiment of the present disclosure as defined by the claims and their equivalents. While various specific details may be included to facilitate understanding, they are to be considered merely exemplary. Accordingly, those skilled in the art will appreciate that various changes and modifications can be made to the one embodiment described herein without departing from the scope and spirit of the present disclosure. Furthermore, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0027] The terms and words used in the following description and claims are not limited to their bibliographic meanings and may be used solely to facilitate a clear and consistent understanding of the present disclosure. Therefore, the following description of one embodiment of the present disclosure is provided for illustrative purposes only and is not intended to limit the present disclosure, which is defined by the appended claims and their equivalents.

[0028] The singular form should be understood to include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more such component surfaces.

[0029] The terms used in this disclosure are selected from widely used, common terms, taking into account the functions of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, in which case their meanings will be described in detail in the relevant description. Therefore, the terms used in this disclosure should not be defined simply as names, but rather based on the meanings of the terms and the overall content of the disclosure.

[0030] Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described herein. Furthermore, terms containing ordinal numbers, such as "first" or "second," used herein may be used to describe various components, but these components should not be limited by these terms. These terms are used solely to distinguish one component from another.

[0031] When a part of the specification is said to "include" a component, unless otherwise specifically stated, this does not exclude other components but rather implies the inclusion of other components. Furthermore, terms such as "part" and "module" used in the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software.

[0032] The artificial intelligence-related functions according to the present disclosure are operated via a processor and memory. The processor may be comprised of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, an AP, a Digital Signal Processor (DSP), a graphics-only processor such as a GPU or a Vision Processing Unit (VPU), or an artificial intelligence-only processor such as an NPU. One or more processors control the processing of input data according to predefined operating rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0033] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is trained using a learning algorithm using a plurality of learning data, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0034] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and performs neural network operations through operations between the operation results of the previous layer and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

[0035] In this disclosure, the term "knowledge graph" refers to a knowledge base in graph form, which is a method for managing and exploring knowledge information, and which is based on a knowledge base that stores knowledge information and a graph that expresses it so that it can be analyzed in a network structure. A knowledge graph is a graph model that implements the knowledge accumulated in a knowledge base as a relationship between nodes and edges. A knowledge graph can be used to integrate data using a graph data model, topology, etc. To enable knowledge to be interconnected and integrated using a knowledge graph, a schema is implemented through an ontology, and a structure and dictionary (terminology) that can be shared with each other are used.

[0036] Semantic information, including common sense and factual knowledge, is organized into nodes and edges, and by referencing this, various types of data can be converted into the form of a knowledge graph. Methods for representing knowledge graphs include, but are not limited to, the Labeled Property Graph (LPG), in which nodes and edges can each have properties, and the Resource Description Framework (RDF), which expresses relationships in a triple structure of subject-predicate-object. Knowledge graphs can be created by recognizing entities from various data, linking them to appropriate entities in an existing knowledge base (entity linking), and extracting relationships between entities.

[0037] Knowledge graphs can be leveraged to enhance the performance of artificial intelligence. They can be used in models such as graph neural networks (GNNs) and graph convolution neural networks (GCNs), and can be used to provide explanations for results in explainable artificial intelligence (XAI).

[0038] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0039] Each block of a flowchart and its combination can be executed by one or more programs containing instructions. One or more programs may be stored entirely in a single memory, or one or more programs may be divided into different parts stored in different memories.

[0040] Any function or operation described herein may be processed by a single processor or a combination of processors. A single processor or a combination of processors is a circuit that performs processing and includes circuits such as an application processor (AP, e.g., a central processing unit (CPU)), a communication processor (CP, e.g., a modem), a graphics processing unit (GPU), a neural processing unit (NPU) (e.g., an artificial intelligence (AI) chip), a Wi-Fi chip, a Bluetooth chip, a global positioning system (GPS) chip, a near field communication (NFC) chip, a connectivity chip, a sensor controller, a touch controller, a fingerprint sensor controller, a display drive integrated circuit (IC), an audio codec chip, a universal serial bus (USB) controller, a camera controller, an image processing IC, a microprocessor unit (MPU), a system on a chip (SoC), an integrated circuit (IC), and the like.

[0041] The present disclosure will be described in detail with reference to the attached drawings below.

[0042] FIG. 1 is a diagram illustrating an AI platform based on a personal knowledge graph according to one embodiment of the present disclosure.

[0043] Referring to Figure 1, it shows that an AI platform based on a personal knowledge graph is provided to a user through a process of building a personalized database based on a knowledge graph (S110) and a process of providing a personalized AI service to a user through a service application based on a knowledge graph (S120).

[0044] In the process of building a personalized database based on a knowledge graph (S110), the user device (100) can create a personal knowledge graph in a structured form by referencing a general knowledge graph in a structured form, such as common sense and factual knowledge, with user data acquired from the user device (100). The user device (100) can convert various types of unstructured data into the form of a knowledge graph, build a personalized database based on a knowledge graph, i.e., a personal knowledge graph, and store it in storage.

[0045] In the process of providing personalized AI services to users through a knowledge graph-based service application (S120), the user device (100) can provide various services using a personalized database stored in storage, i.e., a personal knowledge graph. The user device (100) can use the personal knowledge graph to provide recommendation services, assistant services, QA (Question Answering) services, etc.

[0046] For example, if the user is performing an action, the user device (100) can recommend music that the user enjoys listening to during the action through a recommendation service, or provide information about related past experiences or upcoming events (e.g., calendar schedules) as well as knowledge information based on a personal knowledge graph through an assistant service. The user device (100) can identify the user's behavioral patterns from a personalized database, and if the user shows a pattern that is different from the user's usual behavioral patterns, the user device (100) can provide a solution service related to the cause of the different pattern through a recommendation service. The user device (100) can provide a customized answer to the user's question based on a personalized database through a QA service. The user device (100) can utilize the personalized database to perform a journaling function that manages and describes the user's daily routine or special events through an assistant service.

[0047] FIG. 2 is a diagram for explaining an operation of a user device (100) according to one embodiment of the present disclosure to build a personalized database based on a knowledge graph.

[0048] A user device (100) according to one embodiment of the present disclosure may be an electronic device capable of processing data. For example, the user device (100) may be an electronic device such as a smartphone, smart glasses, a wearable device, a digital camera, a laptop, an Augmented Reality (AR) device, or a Virtual Reality (VR) device. The user device (100) may be equipped with various types of neural network models. For example, the user device may be equipped with at least one model such as a Convolution Neural Network (CNN), a Graph Convolution Neural Network (GCN), a Graph Neural Network (GNN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), or a Bidirectional Recurrent Deep Neural Network (BRDNN), and may also use these models in combination.

[0049] A user device (100) according to one embodiment of the present disclosure can construct a personalized database based on a knowledge graph by converting various types of unstructured data into the form of a knowledge graph. The user device (100) can acquire various types of data and convert them into triple-format data. The user device (100) can map the converted data with an ontology stored in a semantic memory, thereby reflecting structured information in the form of an ontology into a personal knowledge graph. An ontology is a type of dictionary that defines terms that conceptualize data and the relationships between terms. An ontology can be expanded by adding external knowledge by converting it into triple-format data.

[0050] According to one embodiment of the present disclosure, data acquired from the user device (100) may be metadata about content such as text, photos, videos, music, etc., collected in the user device (100). Data acquired from the user device (100) may be stored in the user device in the form of metadata about the application used or metadata about the time, place, weather, etc. of an event occurrence. Data acquired from the user device (100) may include at least one of data input from a user in the user device (100), data sensed by the user device (100), data received from the outside by the user device (100), and data processed by the user device (100).

[0051] The user device (100) can process unstructured data acquired from the user device (100) into structured information and store it in a personal knowledge graph. The user device (100) can prepare semantic information in a structured form in advance as an ontology in a semantic memory. The user device (100) can refer to semantic information of various types of ontologies stored in the semantic memory to process unstructured data acquired from the user device (100) into structured information and store it in a personalized database. For example, the user device (100) can convert various types of unstructured data into the form of a knowledge graph and store it in a personal knowledge graph.

[0052] Referring to FIG. 2, a process of a user device (100) collecting metadata about content generated or provided by the user device (100) and constructing a personalized database in the form of a knowledge graph is illustrated as an example. For the sake of specific explanation, FIG. 2 merely illustrates the process of constructing a personal knowledge graph based on user content as an example; however, the present disclosure is not limited thereto and can be applied to other types of personal knowledge graphs.

[0053] The user device (100) can collect metadata about content from a contact provider, a message provider, a media provider, CMH (content management hub) data, etc. in the content collector part. The content collector is not limited to the example illustrated in FIG. 4, and can also collect metadata from various types of applications installed in the user device (100). The content collector can include a post-processing module that can classify character types, parse text, classify image types, and recognize objects in images. A place type collector and a weather collector can collect metadata about places or weather from external information acquired from the user device (100).

[0054] The user device (100) can convert metadata about content and metadata corresponding to places or weather related to the content, etc., into triple-format data through a content encoder in the memory core part. The recognizer can map the triple-format data with an ontology stored in the semantic memory, infer standardized information in accordance with the ontology format, and reflect it in a personal knowledge graph. As illustrated in FIG. 2, various types of ontologies, such as content ontology, user activity ontology, environment ontology, and relationship ontology, may be prepared in advance in the semantic memory according to the purpose, and the recognizer can search by integrating various ontologies prepared in the semantic memory.

[0055] Referring to FIG. 2, the process of building a personal knowledge graph based on user content by collecting metadata about content generated or provided by the user device (100), converting it into a triple format, and then mapping it with a content ontology is illustrated as an example of building a personalized database based on a knowledge graph. The personal knowledge graph based on user content can represent instances of content collected using photos or videos owned by the user, message information exchanged by the user, contact information, etc., using nodes and edges. For example, a message provider and a media provider can transmit data about card payment information paid on January 1st and photos taken on January 1st received by the user device (100) to a content encoder. The transmitted data can be connected to the ontology of the personal knowledge graph as an instance of an event that occurred on January 1st through text and metadata analysis.

[0056] FIG. 3A and FIG. 3B are diagrams illustrating a content ontology as an example of an ontology used by a user device (100) to build a personal knowledge graph according to one embodiment of the present disclosure.

[0057] As previously described, in the process of constructing a personal knowledge graph based on user content, the user device (100) collects metadata regarding content generated or provided by the user device (100), converts it into a triple format, and then maps the converted data to a content ontology stored in a semantic memory, thereby generating data in a standardized form conforming to the ontology format. At this time, the content ontology may be utilized in the process of constructing a personal knowledge graph based on user content.

[0058] Referring to Figures 3a and 3b, an example of a content ontology is illustrated. A content ontology can be used to model content consumed by users. A content ontology defines data items that can be collected for each content type, and these are called "classes." A content ontology can be composed of nodes corresponding to each class and edges expressing the relationships between each class, and lower classes can inherit properties of their upper classes. Through data collection, actual data values ​​applied to data items defined in a class are called instances, and each data item becomes knowledge property information of the corresponding instance. Instances can be created corresponding to each class in the ontology. For example, in the content ontology of Figures 3a and 3b, an instance corresponding to the highest level, "content," can be created, or instances at lower levels, such as "person" or "environment," can be created. The knowledge attribute information of an instance can be recorded as actual data values, or actual data values ​​can be recorded in predefined data items by referencing classes of the same or different types of ontology. For example, in FIGS. 3A and 3B, the media object can record actual data values ​​that match the data type for data items such as name, start time, end time, and creation date. In addition, the media object can record actual data values ​​for the data item of the content location according to the data items defined in the class called 'place' in the environment ontology. In addition, the media object can record actual data values ​​for the data item of the author according to the data items defined in the class called 'person' in the content ontology.

[0059] When a user remembers content, the information remembered together can be the knowledge attribute information of each instance in a personal knowledge graph based on the user content, so that the data items that can be collected can be defined in advance in the class. When all data values ​​for the data items defined in the class are collected without omission from various metadata related to the content, the user device (100) can provide a user-friendly content search according to the user's memory method by using a personal knowledge graph based on the user content and a query that includes the knowledge attribute information. If metadata regarding some objects that constitute the content are missing or some metadata regarding an object are missing, some values ​​of the knowledge attribute information of the instance of the content may be missing or empty, so the user device (100) may not be able to search the content.

[0060] According to one embodiment of the present disclosure, a user device (100) can model user content through user modeling-based data collection using a content ontology. For example, the user device (100) can use the content ontology to obtain standardized data in an ontology format for photos stored in the user device (100), thereby generating a personal knowledge graph based on the user content.

[0061] FIG. 4 is a diagram for explaining an operation of a user device (100) according to one embodiment of the present disclosure to provide a personal knowledge graph-based service to a user.

[0062] According to one embodiment of the present disclosure, a user device (100) can provide personalized services to users through a personal knowledge graph-based service application. The user device (100) can build a personalized knowledge graph-based database.

[0063] Continuing with the description of FIGS. 2, 3a, and 3b, a personalized database based on a knowledge graph is described below, assuming that a personal knowledge graph based on user content has been constructed. An embodiment in which a user device (100) provides a content search service using a personal knowledge graph based on user content and a query including a search term entered by the user is described below, but is not limited to such a service.

[0064] According to one embodiment, the user device (100) may receive user input regarding a search term by executing a content search service in the content finder part, and the service logic module may transmit the search term entered by the user to a retriever. The text search module may transmit the result of performing simple text matching on the search term to the service logic module. The service logic module may provide search results based on the result of combining the query results for a personal knowledge graph based on user content and the result of performing text matching, and may apply a ranking to the search results.

[0065] The user device (100) can derive the Internationalized Resource Identifier (IRI) of the search term transmitted from the service logic module through the entity encoder in the memory core part. The entity encoder can support multilingual processing of the search term or support synonym processing by using a knowledge graph vocabulary database and a synonym database to convert the search term into a standard language registered in the knowledge graph and derive the IRI corresponding to the search term. The searcher can construct a query using the IRI corresponding to the search term and receive a query result for a personal knowledge graph based on user content. The query result for the personal knowledge graph based on user content can include a list of IRIs of content related to the IRI included in the query, knowledge attribute information of each content, type information or score information of the knowledge attribute information, type information of the content, etc. The knowledge attribute information of each content can be set in advance up to knowledge attribute information of an n-hop relationship with the content. The type information of knowledge attribute information can be either predefined data item information (field information) for each class when constructing a content ontology, or separate type information defined for each content node. Searchers can output associative search terms based on knowledge attribute information and / or type information of knowledge attribute information, along with search results based on query results.

[0066] If the user input for a search term consists of multiple search terms, the service logic module can transmit the multiple search terms entered by the user to the retriever. Searches using multiple search terms may include simultaneously entering the first and second search terms, or additionally entering the second search term after the first search term.

[0067] In one embodiment, when a first search term and a second search term are input at the same time, the retriever can derive an IRI corresponding to the first search term and an IRI corresponding to the second search term through an entity encoder. The retriever can construct a query using the IRI corresponding to the first search term and a query using the IRI corresponding to the second search term, and after receiving the results of each query, obtain a final result corresponding to the intersection (or union) of the results of each query.

[0068] In one embodiment, when a second search term is additionally input after a first search term is input, the retriever can cache the query results of the query using the IRI corresponding to the first search term. When the second search term is additionally input, the retriever can obtain the final result of the multi-search term by deriving the IRI corresponding to the second search term and filtering the query results stored in the cache using the IRI corresponding to the second search term. When using the cache for a search using multiple search terms, the method of filtering the query results stored in the cache can be expected to provide a faster response than the method of calculating the intersection of the two query results.

[0069] When a user input for a search term is a multi-search term using an operation symbol, the service logic module can transmit the multi-search term using the operation symbol input by the user to the retriever. The retriever can parse the multi-search term using the operation symbol and process the first search term, the operation symbol, and the second search term separately. The retriever can perform a predefined operation according to the type of the operation symbol on the first query result of a query using an IRI corresponding to the first search term and the second query result of a query using an IRI corresponding to the second search term. For example, when the operation symbol is '+', a result combining the first and second query results can be output. When the operation symbol is '-', a result removing the second query result from the first query result can be output. When inputting multiple search terms, at least one operation symbol can be used, parentheses can be input, and an operation symbol corresponding to a user-defined operation can be used.

[0070] Meanwhile, if there is no instance corresponding to the search term entered by the user in the personal knowledge graph, it is possible to preset whether to output query results for instances corresponding to a higher class than the class of the instance corresponding to the search term. Furthermore, in the knowledge graph of an external server, node information can be collected using the class information of the instance corresponding to the search term, and by linking with the personal knowledge graph, related search terms in which the instance exists can be extracted to provide search results. In addition, an ontology containing class information of the instance corresponding to the search term can be downloaded from an external server, and by linking with the personal knowledge graph, related search terms in which the instance exists can be extracted to provide search results.

[0071] FIG. 5a, FIG. 5b, and FIG. 5c are diagrams for explaining a process of searching for a photo using a search term based on knowledge attribute information through a personal knowledge graph-based search application by a user device (100) according to one embodiment of the present disclosure.

[0072] Referring to Figure 5a, the user interface screen (hereinafter, "navigator") of a personal knowledge graph-based search application is illustrated. The navigator may be composed of a search window, a knowledge property type tab, a knowledge property category window, and a search results window, each of which may be modified in position, size, shape, etc.

[0073] The search box can receive user input regarding search terms. Users can enter search terms directly into the search box using the virtual keyboard overlapping the bottom of the screen, or by selecting associated search terms based on knowledge attribute information using the Knowledge Attribute Type tab and Knowledge Attribute Category window.

[0074] The knowledge attribute type tab may be composed of multiple tabs, each categorized by the type of knowledge attribute information of each instance of the personal knowledge graph. Referring to FIG. 5a, the knowledge attribute type tab may include tabs corresponding to location, time, person, topic, and type, respectively, depending on the type of knowledge attribute information, but is not limited thereto. For example, if the knowledge attribute information relates to the time at which content was filmed, the corresponding instance may be classified under the time tab. The knowledge attribute type tab may be displayed always or upon user interaction, depending on the user's settings, and may be provided in various ways, such as a drop-down, pop-up, or new window. Among the multiple tabs constituting the knowledge attribute type tab, a tab selected by the user may be displayed so as to be distinct from other tabs.

[0075] The knowledge attribute category window can categorize the knowledge attribute information of instances classified into each tab by knowledge attribute type tab and provide information about the category name and the number of instances belonging to the corresponding category. The knowledge attribute information of an instance can utilize actual data values ​​or information inferred from actual data values. For example, if the knowledge attribute information of an instance is 'wife', it can be classified as 'family' through inference. If the knowledge attribute information of a photo classified into each tab by knowledge attribute type tab is unknown, the user device (100) can infer candidate knowledge attribute information and utilize the inferred candidate knowledge attribute information, provide a thumbnail of the photo, or utilize the knowledge attribute type information as the knowledge attribute information.

[0076] Category names can be used as search terms, and when a category name provided in the knowledge attribute category window is selected, the category name is automatically entered as a search term in the search window, and instances belonging to the category can be provided in the search results window. Referring to Fig. 5a, information about the category name and the number of instances belonging to the category can be expressed in the form of a word box, but is not limited thereto, and can also be provided in the form of a word cloud or a three-dimensional solid.

[0077] The Knowledge Attribute Type tab and Knowledge Attribute Category window can be activated when the user enters a search term, or can be activated without entering a search term based on the user's current specific action, predicted future action, or certain contextual information.

[0078] The search results window provides search results corresponding to the search term entered in the search window. Referring to Figure 5a, in an embodiment of a photo search using a personal knowledge graph-based search application, the search results window may provide thumbnails of photos corresponding to the search term and information regarding the total number of photos corresponding to the search term.

[0079] Referring to FIGS. 5a, 5b, and 5c, the navigator according to one embodiment is shown in the process of searching for a photo using a search term based on knowledge attribute information through a personal knowledge graph-based search application by a user device (100).

[0080] Referring to FIG. 5a, the user device (100) may receive a search term 'Children's Day' from the user in the search window. The user device (100) may derive the IRI of the search term 'Children's Day' and construct a query using the IRI of the search term 'Children's Day' to receive a query result for a personal knowledge graph based on user content. The query result for the personal knowledge graph based on user content may include a list of IRIs of 28 photos related to the IRI of 'Children's Day', type information of knowledge attribute information of each photo, knowledge attribute information of photos classified into each tab by knowledge attribute type tab, score information of each knowledge attribute information, information indicating that the type information of the content is a photo, etc. Referring to FIG. 5a, the user device (100) may display 28 photos as search results for 'Children's Day' in the search results window, and categorize the knowledge attribute information of photos classified into each tab by knowledge attribute type tab in the knowledge attribute category window to provide information on the category name and the number of photos belonging to the corresponding category. Referring to Fig. 5a, the knowledge attribute information of the photos classified by the time tab of the knowledge attribute type tab is categorized into 'clear day', 'afternoon', 'last year', 'the year before last', 'morning', 'last month', 'this year', and 'OO month', and information about the number of photos belonging to each category along with the category name can be displayed in the knowledge attribute category window. The category names 'clear day', 'afternoon', 'last year', 'the year before last', 'morning', 'last month', 'this year', and 'OO month' can be utilized as search words that the user can associate. When 'last year' is selected among the category names provided in the knowledge attribute category window of Fig. 5a, 'last year' is additionally entered as a search word in the search window, as illustrated in Fig. 5b, and 9 photos belonging to the 'last year' category can be provided in the search results window.When a user selects the topic tab in the knowledge attribute type tab, as illustrated in FIG. 5b, the knowledge attribute information of the photos classified by the topic tab of the knowledge attribute type tab is categorized into 'person', 'top / overcoat', 'soup', 'food', 'meal', 'lunch', and 'group', and information about the number of photos belonging to each category along with the category name can be displayed in the knowledge attribute category window. The category names 'person', 'top / overcoat', 'soup', 'food', 'meal', 'lunch', and 'group' can be utilized as search words that the user can associate with. When 'meal' is selected among the category names provided in the knowledge attribute category window of FIG. 5b, 'meal' is additionally entered as a search word in the search window, as illustrated in FIG. 5c, and two photos belonging to the 'meal' category can be provided in the search results window.

[0081] Meanwhile, when a user input regarding a search term is a multi-search term, for example, when 'Children's Day' is input as a first search term and 'last year' is input as a second search term, the user device (100) can obtain a query result of a query using an IRI corresponding to 'Children's Day' and a query result of a query using an IRI corresponding to 'last year', and then provide a result as shown in FIG. 5b as a final result corresponding to the intersection of each query result.

[0082] As previously discussed, the user device (100) can search a personal knowledge graph by constructing a query using an IRI corresponding to a search term entered by the user. The query can detect instances with values ​​corresponding to the search term by checking the knowledge attribute information values ​​of each instance constituting the personal knowledge graph. If any of the instances constituting the personal knowledge graph have missing knowledge attribute information values, it is impossible to confirm whether the value corresponds to the search term, resulting in failure to detect appropriate instances or outputting results that omit instances that should actually be detected. In other words, to obtain good search results, it is important to know the knowledge attribute information values ​​of all instances constituting the personal knowledge graph. Therefore, it is necessary to manage the personal knowledge graph so that no instances constituting the personal knowledge graph have knowledge attribute information with some or all missing values ​​(hereinafter, “missing knowledge attribute information”). Below, a method for managing a personal knowledge graph so that no instances include incomplete nodes with missing knowledge attribute information is described.

[0083] FIG. 6 is a diagram for explaining an operation of a user device (100) managing a personal knowledge graph according to one embodiment of the present disclosure.

[0084] The user device (100) can manage a personal knowledge graph in the memory core part. The user device (100) can determine whether an instance constituting the personal knowledge graph has missing knowledge property information, determine whether the missing knowledge property information can be improved, and fill in the missing knowledge property information with an appropriate value. The personal knowledge graph manager can include a missing knowledge property information detector, a candidate knowledge property information detector, and a completion task performer.

[0085] A missing knowledge attribute detector can determine whether all knowledge attribute information values ​​of instances constituting a personal knowledge graph are complete without omissions. If at least one knowledge attribute information value of an instance is null, the missing knowledge attribute detector can determine that there is missing knowledge attribute information and determine that the knowledge attribute information of the instance is incomplete. Even if the knowledge attribute information value of an instance is not null, if there is a missing value, the missing knowledge attribute detector can determine that there is missing knowledge attribute information and determine that the knowledge attribute information of the instance is incomplete. For example, if there are three people in a photo, and data for only one of them is extracted to form the knowledge attribute information value about the person, the missing knowledge attribute detector can determine that there are missing values ​​for the remaining two people, and determine that the knowledge attribute information about the person in the photo is incomplete. The missing knowledge attribute information detector can determine whether there is missing knowledge attribute information based on whether the value of any knowledge attribute information is a null value based on a graph data model, and can determine whether there is missing knowledge attribute information by analyzing the content itself to determine whether there are unidentified objects. The missing knowledge attribute information detector can detect instances including incomplete nodes having missing knowledge attribute information in a personal knowledge graph when a user is not using the user device (100) (e.g., when the user is sleeping or the user device (100) is charging) or when the user device (100) is completing a predetermined task.

[0086] When an instance containing an incomplete node with missing knowledge attribute information is detected, the candidate knowledge attribute information detector can identify candidate knowledge attribute information to improve the missing knowledge attribute information. The candidate knowledge attribute information detector can search for a complete node corresponding to the incomplete node in the personal knowledge graph and identify candidate knowledge attribute information corresponding to the missing knowledge attribute information in the complete node. Alternatively, the candidate knowledge attribute information detector can infer candidate knowledge attribute information corresponding to the missing knowledge attribute information.

[0087] A completion task performer can complete an incomplete node. The completion task performer can complete an incomplete node if there is user input or if the accuracy of candidate knowledge attribute information corresponding to the missing knowledge attribute information satisfies a predetermined condition. If the completion task performer determines that a completed node retrieved from the personal knowledge graph is identical to the incomplete node, the completion task performer can replace the incomplete node with the retrieved completed node. When the incomplete node is replaced with a completed node, edges connected to the completed node are connected, forming a connection point that allows knowledge attribute information to be shared between the two contents. If the completion task performer determines that a completed node retrieved from the personal knowledge graph is not identical to the incomplete node, the completion task performer can update the missing knowledge attribute information of the incomplete node with the candidate knowledge attribute information identified in the retrieved completed node. Alternatively, the completion task performer can update the missing knowledge attribute information of the incomplete node with the candidate knowledge attribute information inferred by the candidate knowledge attribute detector. If the accuracy of the candidate knowledge attribute information is above a certain level, the completion task performer can automatically update the value of the missing knowledge attribute information of the instance including the incomplete node with the value of the candidate knowledge attribute information. Alternatively, the completion task performer can control the user device (100) to provide a user interface for confirming whether to update the value of the knowledge attribute information of the instance including the incomplete node with the value of the candidate knowledge attribute information, in order to update the missing knowledge attribute information. The completion task performer can control the user device (100) to provide a user interface for the user to directly input the value of the missing knowledge attribute information of the instance including the incomplete node. The completion task performer can update the missing knowledge attribute information based on the update approval input by the user or the value of the persistence attribute information.

[0088] FIG. 7 is a flowchart illustrating a method for managing a personal knowledge graph according to one embodiment of the present disclosure.

[0089] Referring to FIG. 7, in step S710, the user device (100) can detect instances including incomplete nodes with missing knowledge attribute information in a personal knowledge graph. The personal knowledge graph can be generated using information acquired from the user device (100), and various types of personal knowledge graphs can be generated. For example, a personal knowledge graph based on user content can represent instances of content collected using photos or videos owned by the user, message information exchanged by the user, contact information, etc., using nodes and edges. An incomplete node refers to a node in which at least one knowledge attribute information among the knowledge attribute information list defined in the class corresponding to the instance has a null value, or a node in which some or all of the knowledge attribute information has a missing value even if the value is not a null value. The user device (100) can determine whether an instance includes an incomplete node based on a graph data model by determining whether any knowledge attribute information has a null value, or can determine whether an instance includes an incomplete node by analyzing the content itself to determine whether there are unidentified objects.

[0090] FIG. 8 is a diagram illustrating an instance including a complete node according to a given class and an instance including an incomplete node having missing knowledge attribute information in a personal knowledge graph according to one embodiment of the present disclosure.

[0091] Referring to Figure 8, the user class defines name, gender, and age as collectable data items. These collectable data items can serve as knowledge attribute information for the user instance. Figure 8 illustrates an example in which a first instance corresponding to a first user and a second instance corresponding to a second user are created by applying actual data values ​​to the data items defined in the class through data collection.

[0092] The first instance corresponding to the first user has the knowledge attribute information 'Name = Karina', 'Gender = Female', and 'Age = 15', and is a complete node because it does not contain any missing knowledge attribute information. On the other hand, the second instance corresponding to the second user has the knowledge attribute information 'Name' as 'Karina', but 'Gender' and 'Age' have null values, so it is an incomplete node because it contains missing knowledge attribute information.

[0093] Referring to FIG. 7, according to one embodiment, the user device (100) may determine a search category for a personal knowledge graph based on the user's behavior type. The user device (100) may determine a different search category to search only a portion of the personal knowledge graph or the entire personal knowledge graph based on whether the user is sleeping, resting, using the user device (100), or charging. The search category corresponding to the user's behavior type may be determined by the length of the search time or the number of search instances. The user device (100) may determine whether there are incomplete nodes with missing knowledge attribute information in each instance within the determined search category. For example, if the user is sleeping, the user device (100) may allow sufficient time to search the personal knowledge graph, and thus may search the entire personal knowledge graph for instances containing incomplete nodes. Since the user device (100) can only give a short time to search the personal knowledge graph when the user is expected to use the user device (100) soon, it can search for instances that include incomplete nodes only for a part of the personal knowledge graph.

[0094] According to one embodiment, when the user device (100) detects multiple incomplete nodes in a personal knowledge graph, if the missing knowledge attribute information included in each of the different incomplete nodes has the same value, it can assign the same tag, and if the knowledge attribute information has different values, it can assign different tags.

[0095] At step S720, the user device (100) can identify candidate knowledge attribute information corresponding to the missing knowledge attribute information of the incomplete node. The user device (100) can search for or infer candidate knowledge attribute information corresponding to the missing knowledge attribute information from the personal knowledge graph to determine the data value for the missing knowledge attribute information.

[0096] According to one embodiment, the user device (100) may search for a completed node corresponding to an incomplete node in the personal knowledge graph based on the relationships between nodes in the personal knowledge graph and the knowledge attribute information included in the incomplete node. The user device (100) may identify a node having a similar connection relationship to the incomplete node based on the connection relationship between the incomplete node and other nodes in the personal knowledge graph. The user device (100) may identify a node including the same knowledge attribute information based on knowledge attribute information of which data value is known among the knowledge attribute information list included in the incomplete node. The user device (100) may search for a node determined to be identical to or most similar to the incomplete node in the personal knowledge graph based on the relationships between nodes in the personal knowledge graph and the knowledge attribute information included in the incomplete node. When there is an instance including an incomplete node, the user device (100) can search for a complete node corresponding to the incomplete node through a graph embedding method based on graph information related to the incomplete node, an embedding method based on characteristic information of the instance, or a logic-based method capable of determining identity or similarity between nodes. The user device (100) can check candidate knowledge attribute information corresponding to missing knowledge attribute information in the searched complete node.

[0097] According to one embodiment, the user device (100) may infer candidate knowledge attribute information corresponding to missing knowledge attribute information based on at least one of graph information and instance feature information related to the incomplete node. The user device (100) may obtain knowledge attribute information having a known data value from a list of knowledge attribute information of an incomplete node having the missing knowledge attribute information. The user device (100) may obtain image feature information or text feature information by analyzing an image or text of an instance including the incomplete node.

[0098] FIG. 9 is a diagram illustrating various types of embedding vectors used to infer candidate knowledge attribute information corresponding to missing knowledge attribute information according to one embodiment of the present disclosure.

[0099] When there is an instance including an incomplete node in the personal knowledge graph, the user device (100) may use any one of a graph embedding vector, an image embedding vector, and / or a text embedding vector, and a multi-modal embedding vector to infer candidate knowledge attribute information corresponding to the missing knowledge attribute information of the incomplete node.

[0100] Referring to FIG. 9, a graph information-based neural network can be trained by including information about ontology and instances, and then inputting graph information related to incomplete nodes to output a graph embedding vector. The graph information-based neural network can be a model in the form of a neural network composed of multiple layers.

[0101] A feature information-based neural network can receive feature information of an instance, such as image feature information or text feature information, as input, and output an image embedding vector and / or a text embedding vector. Image feature information may be information indicating features related to the form, such as shape, form, and color, identified from an image. Text feature information may be information indicating features related to the meaning, such as letters, numbers, and symbols identified from text. A feature information-based neural network may be a model in the form of a neural network composed of multiple layers.

[0102] A multi-modal based neural network can output a multi-modal embedding vector by concatenating a graph embedding vector based on graph information related to an incomplete node with an image embedding vector and / or text embedding vector of feature information of an instance. The graph embedding vector, which is the output of a graph information-based neural network, and the image embedding vector and / or text embedding vector, which are the output of a feature information-based neural network, can be input to the multi-modal based neural network. The multi-modal embedding vector is an embedding vector created by normalizing and then concatenating embedding vectors of different modalities, and can reflect the characteristics of the graph embedding vector, the image embedding vector, and / or the text embedding vector.

[0103] Referring back to FIG. 7, in one embodiment, the user device (100) can infer candidate knowledge attribute information corresponding to the missing knowledge attribute information by using a graph embedding vector based on graph information related to an incomplete node. For example, if the knowledge attribute information about the person (artist) who took photo 'A' is the missing knowledge attribute information, the user device (100) can infer candidate knowledge attribute information corresponding to the missing knowledge attribute information based on the node of photo 'B' that is clustered closest to the incomplete node of photo 'A' by using a graph embedding vector for the background or place extracted from photo 'A', a graph embedding vector for the time when photo 'A' was taken, etc. The user device (100) can infer candidate knowledge attribute information corresponding to the knowledge attribute information about the person who took photo 'A' by extracting knowledge attribute information about the person who took photo 'B' from the node of photo 'B'.

[0104] In one embodiment, the user device (100) may classify an instance using an image embedding vector or a text embedding vector of the feature information of the instance, thereby inferring candidate knowledge attribute information corresponding to missing knowledge attribute information. For example, if the knowledge attribute information regarding the time at which photo 'A' was taken is missing knowledge attribute information, the user device (100) may extract the background of photo 'A' and generate an image embedding vector of the background, or extract text regarding the time from photo 'A' and generate a text embedding vector. The user device (100) may compare the generated image embedding vector or text embedding vector with embedding vectors by time zone, and classify the time zone at which photo 'A' was taken, thereby inferring candidate knowledge attribute information corresponding to the knowledge attribute information regarding the time at which photo 'A' was taken.

[0105] In one embodiment, the user device (100) may infer candidate knowledge attribute information corresponding to missing knowledge attribute information by using a graph embedding vector based on graph information related to an incomplete node and a multi-modal embedding vector based on an image embedding vector or a text embedding vector of feature information of an instance. For example, if the knowledge attribute information about the location where photo 'A' was taken is missing knowledge attribute information, the user device (100) may use a multi-modal embedding vector that combines a graph embedding vector based on knowledge attribute information about the time when photo 'A' was taken and an image embedding vector for a person or background extracted through image analysis of photo 'A'. The user device (100) may use the multi-modal embedding vector to extract knowledge attribute information about the location where photo 'A' was taken from a photo that includes the same person or background as photo 'A' and was taken at a similar time when photo 'A' was taken, thereby inferring candidate knowledge attribute information about the location where photo 'A' was taken. For another example, if the knowledge attribute information about the location where photo 'A' was taken is missing knowledge attribute information, the user device (100) can use a multi-modal embedding vector that combines a graph embedding vector based on the knowledge attribute information about the time when photo 'A' was taken and a text embedding vector for text extracted through text analysis of photo 'A'. The user device (100) can use the multi-modal embedding vector to extract knowledge attribute information about the location where photo 'A' was taken from a photo that contains the same text as the text extracted from photo 'A' and was taken at a similar time as the time when photo 'A' was taken, thereby inferring candidate knowledge attribute information about the location where photo 'A' was taken.

[0106] At step S730, the user device (100) can perform a task of completing an incomplete node based on the confirmed candidate knowledge attribute information.

[0107] According to one embodiment, the user device (100) may perform an operation to complete an incomplete node if the accuracy of candidate knowledge attribute information corresponding to user input or missing knowledge attribute information satisfies a predetermined condition. For example, the user device (100) may perform an operation to complete an incomplete node automatically or through user input based on the accuracy of the candidate knowledge attribute information. If the accuracy of the candidate knowledge attribute information is below a predetermined threshold, the user device (100) may perform an operation to complete an incomplete node based on the confirmed candidate knowledge attribute information after obtaining an input of approval or modification from the user. Alternatively, if the accuracy of the candidate knowledge attribute information is above a predetermined threshold, the user device (100) may automatically perform an operation to complete an incomplete node based on the confirmed candidate knowledge attribute information.

[0108] According to one embodiment, when the user device (100) searches for a completed node corresponding to an incomplete node in the personal knowledge graph and confirms candidate knowledge attribute information corresponding to missing knowledge attribute information in the searched completed node, the user device (100) may determine whether the searched completed node is identical to the incomplete node and, based on the determination result, perform an operation of completing the incomplete node. For example, the user device (100) may determine whether the searched completed node is identical to the incomplete node using key knowledge attribute information. Alternatively, the user device (100) may determine whether the searched completed node is identical to the incomplete node by comprehensively comparing the knowledge attribute information. If the user device (100) determines whether the searched completed node is identical to the incomplete node and the two nodes are identical, the user device (100) may replace the incomplete node with the searched completed node, and if the two nodes are not identical, the user device may update the missing knowledge attribute information with the candidate knowledge attribute information confirmed in the searched completed node. The task of completing an incomplete node can be completed as the incomplete node is replaced by a completed node or as missing knowledge attribute information is updated.

[0109] FIG. 10 is a diagram illustrating a process of completing an incomplete node by replacing the incomplete node with a completed node according to one embodiment of the present disclosure.

[0110] Referring to Figure 10, it can be seen that the instance corresponding to the first user is the first node, which is a complete node, and the instance corresponding to the second user is the second node, which is an incomplete node. The second node has the value "Karina" for "Name", but "Gender" and "Age" have null values, so it contains missing knowledge attribute information and is therefore an incomplete node.

[0111] The user device (100) can search for a completed node corresponding to a second node, which is an incomplete node, in the personal knowledge graph and check candidate knowledge attribute information corresponding to knowledge attribute information missing from the first node, which is a searched completed node. The user device (100) can determine whether the first node is identical to the second node, which is an incomplete node, and, based on the determination result, perform a task of completing the incomplete node. For example, the user device (100) can determine whether the first node is identical to the second node through an algorithm that can determine the similarity of nodes on the knowledge graph based on whether the first node has key knowledge attribute information with the same value as the second node or knowledge attribute information with different values, and the relationship with commonly connected nodes (the third node, the fourth node, and the fifth node). If the first node is identical to the second node, which is an incomplete node, the user device (100) can delete the second node and replace the connection relationship for the second node with the first node.

[0112] FIG. 11 is a diagram illustrating a process of completing an incomplete node by updating missing knowledge attribute information of an incomplete node with candidate knowledge attribute information confirmed in a completed node according to one embodiment of the present disclosure.

[0113] Referring to Figure 11, it can be seen that the instance corresponding to the first user is the first node, which is a complete node, and the instance corresponding to the second user is the second node, which is an incomplete node. The second node has the values ​​'Name = Karina' and 'Age = 16', but since the 'Gender' value is null, it contains missing knowledge attribute information and is therefore an incomplete node.

[0114] The user device (100) can search for a completed node corresponding to a second node, which is an incomplete node, in the personal knowledge graph and check candidate knowledge attribute information corresponding to the missing knowledge attribute information in the first node, which is the searched completed node. The user device (100) can determine whether the first node is the same as the second node, which is an incomplete node, and, based on the determination result, perform a task of completing the incomplete node. In the case of FIG. 11, the user device (100) can determine that the first node is not the same as the second node because the first node has the same name as the second node as the knowledge attribute information, but the ages have different values. The user device (100) can update the missing knowledge attribute information 'gender' in the second node with the candidate knowledge attribute information checked in the first node. In order to update missing knowledge attribute information, the user device (100) may provide a user interface that allows the user to confirm whether to update the value of the missing knowledge attribute information of an instance including an incomplete node to the value of candidate knowledge attribute information, or a user interface that allows the user to directly input the value of the missing knowledge attribute information, and in response, may receive an update approval or a value of the persistence attribute information from the user. Referring again to FIG. 7, according to one embodiment, when the user device (100) infers candidate knowledge attribute information corresponding to the missing knowledge attribute information, the user device (100) may update the missing knowledge attribute information with the inferred candidate knowledge attribute information. As the missing knowledge attribute information is updated, the task of completing the incomplete node may be completed.

[0115] FIG. 12 is a block diagram illustrating a user device (100) according to one embodiment of the present disclosure. FIG. 13 is a block diagram for explaining the configuration and operation of a user device (100) according to one embodiment of the present disclosure.

[0116] Referring to FIG. 12, a user device (100) according to one embodiment may include, but is not limited to, a memory (110) and a processor (120), and general-purpose configurations may be further added. For example, as illustrated in FIG. 13, the user device (100) may further include, in addition to the memory (110) and the processor (120), a sensing unit (130), a communication unit (140), and an input / output unit (150). Hereinafter, each configuration will be described in detail with reference to FIGS. 12 and 13.

[0117] The memory (110) according to one embodiment can store a program for processing and controlling the processor (120), and can store data and information input to or generated from the user device (100). The memory (110) can store instructions, data structures, and program codes that can be read by the processor (120). Operations performed by the processor (120) can be implemented by executing instructions or program codes stored in the memory (110).

[0118] The memory (110) according to one embodiment may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), and may include a non-volatile memory including at least one of a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk, and a volatile memory such as a RAM (Random Access Memory) or an SRAM (Static Random Access Memory).

[0119] According to one embodiment, the memory (110) may store one or more instructions and / or programs that control the user device (100) to train a neural network model or utilize a neural network model.

[0120] According to one embodiment, the processor (120) can control operations or functions so that the user device (100) can perform tasks by executing instructions or programmed software modules stored in the memory (110). The processor (120) can be composed of hardware components that perform arithmetic, logic, and input / output operations and signal processing. The processor (120) can control the overall operations of the user device (100) by executing one or more instructions stored in the memory (110). The processor (120) can control the sensing unit (130), the communication unit (140), and the input / output unit (150) including at least one sensor by executing programs stored in the memory (110).

[0121] The processor (120) may include various processing circuits and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits, including at least one processor. “At least one processor” may be individually and / or collectively configured to perform various functions described herein. As used herein, “processor,” “at least one processor,” and “one or more processors” may be configured to perform multiple functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, “at least one processor” may include a combination of processors that perform various functions of the disclosed functions in a distributed manner. “At least one processor” may execute program instructions to achieve or perform various functions.

[0122] The processor (120) according to one embodiment may be configured with at least one of, for example, a Central Processing Unit, a microprocessor, a Graphic Processing Unit, Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), an Application Processor, a Neural Processing Unit, or an artificial intelligence processor designed with a hardware structure specialized for processing artificial intelligence models, but is not limited thereto. Each processor constituting the processor (120) may be a dedicated processor for performing a predetermined function.

[0123] An artificial intelligence (AI) processor according to one embodiment may perform calculations and control to process tasks set for a user device (100) to perform using an artificial intelligence (AI) model. The AI ​​processor may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as part of a general-purpose processor (e.g., a CPU or application processor) or a graphics-only processor (e.g., a GPU) and mounted on the user device (100).

[0124] The sensing unit (130) according to one embodiment may include a plurality of sensors configured to detect information about the surrounding environment of the user device (100). For example, the sensing unit (130) may include, but is not limited to, a camera (131), a temperature / humidity sensor (132), an infrared sensor (133), a barometric pressure sensor (134), a position sensor (135), a gyroscope sensor (136), etc. The function of each sensor can be intuitively inferred from its name by those skilled in the art, and thus is briefly described below.

[0125] The camera (131) according to one embodiment may include a stereo camera, a mono camera, a wide-angle camera, an around-view camera, or a 3D vision sensor. The lidar sensor (132) may detect the distance to an object and various physical properties by shining a laser on the target. The temperature / humidity sensor (132) may measure the temperature or humidity of the location where the user device (100) is located. The infrared sensor (133) may be either an active infrared sensor that detects changes by emitting infrared rays and blocking the light, or a passive infrared sensor that does not have a light emitter and only detects changes in infrared rays received from the outside. The barometric pressure sensor (134) may measure the barometric pressure of the location where the user device (100) is located. The location sensor (135) may detect the location of the user device (100). For example, the location sensor (135) may be a global positioning system (GPS). The gyroscope sensor (136) may detect angular velocity. The gyro sensor (136) can be used to measure the position of the user device (100) and set the moving direction of the user device (100).

[0126] The communication unit (140) may include one or more components that enable the user device (100) to communicate with an external device, such as a server or other electronic device. For example, the communication unit (140) may include, but is not limited to, a short-range wireless communication unit (141), a mobile communication unit (142), etc.

[0127] The short-range wireless communication unit (141) may include, but is not limited to, a Bluetooth communication unit, a BLE (Bluetooth Low Energy) communication unit, a near field communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an Ant+ communication unit, a WFD (Wi-Fi Direct) communication unit, a UWB (ultra wideband) communication unit, an infrared (IrDA, infrared Data Association) communication unit, a microwave (uWave) communication unit, etc.

[0128] The mobile communication unit (142) transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signals may include various types of data, such as voice call signals, video call signals, or text / multimedia message transmission and reception.

[0129] The input / output unit (150) may include an input unit (151) and an output unit (153). The input / output unit (150) may be a separate input unit (151) and an output unit (153), or may be an integrated unit, such as a touch screen. The input / output unit (150) may receive input information from a user and provide output information to the user.

[0130] The input unit (151) may refer to a means for a user to input data for controlling the user device (100). For example, the input unit (151) may be a key pad, a touch panel (contact electrostatic capacitance type, pressure resistive film type, infrared detection type, surface ultrasonic conduction type, integral tension measurement type, piezo effect type, etc.), a microphone, etc. In addition, the input unit (151) may include a jog wheel, a jog switch, etc., but is not limited thereto.

[0131] The output unit (153) can output an audio signal, a video signal, or a vibration signal, and the output unit (153) can include a display unit, an audio output unit, and a vibration motor. The display unit can display information processed in the user device (100). For example, the display unit can display a user interface for receiving a user's operation. When the display unit and the touchpad are configured as a touch screen in a layered structure, the display unit can be used as an input device in addition to an output device. The display unit can include at least one of a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, and a 3D display. Depending on the implementation form of the user device (100), the user device (100) can include two or more display units. The audio output unit can output audio data stored in the memory (110). The audio output unit can output an audio signal related to a function performed in the user device (100). The audio output section may include a speaker, a buzzer, etc.

[0132] According to one embodiment of the present disclosure, a user device (100) includes a memory (110) storing at least one instruction and at least one processor (120) operatively connected to the memory (110) to execute at least one instruction. The processor (120) may execute at least one instruction to load and execute a command or code for a given module.

[0133] According to one embodiment of the present disclosure, the processor (120) of the user device (100) can control the user device (100) by executing at least one instruction so that the operation of the user device (100) described using the drawings above can be performed.

[0134] According to one embodiment of the present disclosure, the processor (120) of the user device (100) may execute at least one instruction to detect an instance including an incomplete node having missing knowledge attribute information in a personal knowledge graph. The personal knowledge graph may be generated using information acquired from the user device (100), and various types of personal knowledge graphs may be generated. For example, a personal knowledge graph based on user content may represent instances of content collected using photos or videos owned by the user, message information exchanged by the user, contact information, etc., using nodes and edges. An incomplete node refers to a node in which at least one knowledge attribute information among the knowledge attribute information lists defined in the class corresponding to the instance has a null value, or a node in which some or all of the knowledge attribute information has a missing value even if the knowledge attribute information has a non-null value.

[0135] According to one embodiment, the processor (120) may determine a search category for a personal knowledge graph based on the user's behavior type. For example, the processor (120) may analyze the user's behavior type and, based on the analyzed behavior type, determine a search category to search only a portion of the personal knowledge graph or the entire personal knowledge graph. The search category corresponding to the user's behavior type may be determined in advance. The search category corresponding to the user's behavior type may be determined by the length of the search time or the number of search instances. The processor (120) may determine whether there are incomplete nodes with missing knowledge attribute information for each instance within the determined search category.

[0136] According to one embodiment, when the processor (120) detects a plurality of incomplete nodes in a personal knowledge graph, if the missing knowledge attribute information included in each of the different incomplete nodes has the same value, it can assign the same tag, and if the knowledge attribute information has different values, it can assign different tags.

[0137] The processor (120) can execute at least one instruction to identify candidate knowledge attribute information corresponding to the missing knowledge attribute information of an incomplete node. The processor (120) can search for or infer candidate knowledge attribute information corresponding to the missing knowledge attribute information from the personal knowledge graph to determine the data value for the missing knowledge attribute information.

[0138] According to one embodiment, the processor (120) may search for a complete node corresponding to an incomplete node in the personal knowledge graph based on the relationships between nodes in the personal knowledge graph and the knowledge attribute information included in the incomplete node. The processor (120) may identify a node having a similar connection relationship to the incomplete node based on the connection relationship between the incomplete node and other nodes in the personal knowledge graph. The processor (120) may identify a node containing the same knowledge attribute information based on knowledge attribute information of which data value is known among the knowledge attribute information list included in the incomplete node. The processor (120) may search for a node in the personal knowledge graph that is determined to be identical to or most similar to the incomplete node based on the relationships between nodes in the personal knowledge graph and the knowledge attribute information included in the incomplete node. The processor (120) may identify candidate knowledge attribute information corresponding to the missing knowledge attribute information in the searched complete node.

[0139] According to one embodiment, the processor (120) may infer candidate knowledge attribute information corresponding to the missing knowledge attribute information based on at least one of graph information and instance feature information related to the incomplete node. For example, the processor (120) may obtain knowledge attribute information having a known data value from a list of knowledge attribute information of an incomplete node having the missing knowledge attribute information. The processor (120) may obtain image feature information or text feature information by analyzing an image or text of an instance including the incomplete node.

[0140] In one embodiment, the processor (120) may infer candidate knowledge attribute information corresponding to the missing knowledge attribute information by using a graph embedding vector based on graph information related to an incomplete node. For example, if the first knowledge attribute information is the missing knowledge attribute information, the user device (100) may infer candidate knowledge attribute information corresponding to the first knowledge attribute information, which is the missing knowledge attribute information, based on a node clustered closest to the incomplete node including the missing knowledge attribute information by using a graph embedding vector for the second knowledge attribute information, a graph embedding vector for the third knowledge attribute information, etc. The processor (120) may infer candidate knowledge attribute information corresponding to the first knowledge attribute information by extracting knowledge attribute information corresponding to the first knowledge attribute information from a node clustered closest to the incomplete node.

[0141] In one embodiment, the processor (120) may classify the instance using the image embedding vector or text embedding vector of the feature information of the instance, thereby inferring candidate knowledge attribute information corresponding to the missing knowledge attribute information. For example, if the first knowledge attribute information is missing knowledge attribute information, the processor (120) may extract a background as feature information from the instance and generate an image embedding vector of the background, or extract text as feature information from the instance and generate a text embedding vector. The processor (120) may classify the attributes of the instance using the generated image embedding vector or text embedding vector, thereby inferring candidate knowledge attribute information corresponding to the first knowledge attribute information.

[0142] In one embodiment, the processor (120) may infer candidate knowledge attribute information corresponding to the missing knowledge attribute information by using a graph embedding vector based on graph information related to an incomplete node and a multi-modal embedding vector based on an image embedding vector of feature information of an instance. For example, if the first knowledge attribute information is missing knowledge attribute information, the processor (120) may infer candidate knowledge attribute information corresponding to the first knowledge attribute information by using a multi-modal embedding vector that combines a graph embedding vector of the second knowledge attribute information and an image embedding vector of a person or background extracted through image analysis.

[0143] In one embodiment, the processor (120) may infer candidate knowledge attribute information corresponding to the missing knowledge attribute information by using a multi-modal embedding vector based on a graph embedding vector based on graph information related to an incomplete node and a text embedding vector based on feature information of an instance. For example, if the first knowledge attribute information is missing knowledge attribute information, the processor (120) may infer candidate knowledge attribute information corresponding to the first knowledge attribute information by using a multi-modal embedding that combines a graph embedding vector of the second knowledge attribute information and a text embedding vector extracted through text analysis.

[0144] The processor (120) can perform a task of completing an incomplete node based on the identified candidate knowledge attribute information by executing at least one instruction.

[0145] According to one embodiment, when the processor (120) searches for a completed node corresponding to an incomplete node in the personal knowledge graph and confirms candidate knowledge attribute information corresponding to missing knowledge attribute information in the searched completed node, the processor (120) may determine whether the searched completed node is identical to the incomplete node and, based on the determination result, perform a task of completing the incomplete node. For example, the processor (120) may determine whether the searched completed node is identical to the incomplete node based on key knowledge attribute information. Alternatively, the processor (120) may determine whether the searched completed node is identical to the incomplete node by comprehensively comparing the knowledge attribute information. As a result of determining whether the searched completed node is identical to the incomplete node, if the two nodes are identical, the processor (120) may replace the incomplete node with the searched completed node, and if the two nodes are not identical, the missing knowledge attribute information may be updated with the candidate knowledge attribute information confirmed in the searched completed node. The task of completing an incomplete node can be completed as the incomplete node is replaced by a completed node or as missing knowledge attribute information is updated.

[0146] According to one embodiment, if the processor (120) infers candidate knowledge attribute information corresponding to the missing knowledge attribute information, the processor (120) may update the missing knowledge attribute information with the inferred candidate knowledge attribute information. As the missing knowledge attribute information is updated, the task of completing the incomplete node may be completed.

[0147] According to one embodiment, the processor (120) may perform a task of completing an incomplete node if the accuracy of candidate knowledge attribute information corresponding to user input or missing knowledge attribute information satisfies a predetermined condition. For example, the processor (120) may perform a task of completing an incomplete node automatically or upon user confirmation based on the accuracy of the candidate knowledge attribute information. If the accuracy of the candidate knowledge attribute information is below a predetermined threshold, the processor (120) may obtain an input of approval or modification from the user through the input / output unit (150) and then perform a task of completing an incomplete node based on the confirmed candidate knowledge attribute information. Alternatively, if the accuracy of the candidate knowledge attribute information is above a predetermined threshold, the processor (120) may automatically perform a task of completing an incomplete node based on the confirmed candidate knowledge attribute information.

[0148] Meanwhile, embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and include both volatile and nonvolatile media, removable and non-removable media. Furthermore, computer-readable media may include computer storage media and communication media. Computer storage media include both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media may typically include computer-readable instructions, data structures, or other data in a modulated data signal, such as program modules.

[0149] Additionally, a computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0150] According to one embodiment, a method according to one embodiment of the present disclosure may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0151] According to one embodiment of the present disclosure, a computer-readable recording medium is provided, which records a program for executing a method for constructing a personal knowledge graph in the user device, a method for providing a service using the personal knowledge graph, or a method for managing the personal knowledge graph.

[0152] According to one embodiment of the present disclosure, a method for managing a personal knowledge graph is provided. The method for managing a personal knowledge graph may include a step (S710) of detecting instances of an incomplete node having missing knowledge attribute information in the personal knowledge graph. Furthermore, the method for managing a personal knowledge graph may include a step (S720) of identifying candidate knowledge attribute information corresponding to the missing knowledge attribute information of the incomplete node. Furthermore, the method for managing a personal knowledge graph may include a step (S730) of completing the incomplete node based on the identified candidate knowledge attribute information.

[0153] Additionally, according to one embodiment of the present disclosure, the step of verifying candidate knowledge attribute information (S720) may include a step of searching for a complete node corresponding to an incomplete node in the personal knowledge graph based on the relationships between nodes in the personal knowledge graph and the knowledge attribute information included in the incomplete node. Furthermore, the step of verifying candidate knowledge attribute information (S720) may include a step of verifying candidate knowledge attribute information corresponding to missing knowledge attribute information in the searched complete node.

[0154] In addition, the step (S730) of performing the task of completing an incomplete node may include a step of determining whether the searched complete node is identical to the incomplete node. In addition, the step (S730) of performing the task of completing an incomplete node may include a step of replacing the incomplete node with the searched complete node if the two nodes are identical, and a step of updating the missing knowledge attribute information with the candidate knowledge attribute information identified in the searched complete node if the two nodes are not identical.

[0155] Additionally, according to one embodiment of the present disclosure, the step of verifying candidate knowledge attribute information (S720) may include a step of inferring candidate knowledge attribute information corresponding to missing knowledge attribute information based on at least one of graph information related to an incomplete node and characteristic information of an instance.

[0156] In addition, the step of inferring candidate knowledge attribute information can infer candidate knowledge attribute information corresponding to missing knowledge attribute information by using a graph embedding vector based on graph information related to an incomplete node and a multi-modal embedding vector based on an image embedding vector or a text embedding vector of feature information of an instance.

[0157] Additionally, the step (S730) of performing a task of completing an incomplete node may include a step of updating missing knowledge attribute information with inferred candidate knowledge attribute information.

[0158] In addition, according to one embodiment of the present disclosure, the step (S730) of performing a task of completing an incomplete node may perform a task of completing an incomplete node when there is a user input or the accuracy of candidate knowledge attribute information corresponding to missing knowledge attribute information satisfies a predetermined condition.

[0159] Additionally, according to one embodiment of the present disclosure, the step (S710) of detecting instances containing incomplete nodes may include a step of determining a search category based on the user's behavior type for the personal knowledge graph. Furthermore, the step (S710) of detecting instances containing incomplete nodes may include a step of determining whether incomplete nodes containing missing knowledge attribute information exist for each instance within the determined search category.

[0160] In addition, according to one embodiment of the present disclosure, the step (S710) of detecting an instance including an incomplete node may further include a step of assigning the same tag if the missing knowledge attribute information included in each of the different incomplete nodes has the same value, and assigning different tags if the missing knowledge attribute information has different values.

[0161] According to one embodiment of the present disclosure, a computer-readable recording medium having recorded thereon a program for executing a method for managing the above-described personal knowledge graph is provided.

[0162] According to one embodiment of the present disclosure, a user device (100) for managing a personal knowledge graph is provided. The user device (100) may include a memory (110) storing at least one instruction and at least one processor (120) operatively connected to the memory (110) and configured to execute at least one instruction. In addition, the at least one processor (120) may execute at least one instruction to detect an instance including an incomplete node having missing knowledge attribute information in a personal knowledge graph. In addition, the at least one processor (120) may execute at least one instruction to identify candidate knowledge attribute information corresponding to the missing knowledge attribute information of the incomplete node. In addition, the at least one processor (120) may execute at least one instruction to perform a task of completing the incomplete node based on the identified candidate knowledge attribute information.

[0163] Additionally, according to one embodiment of the present disclosure, at least one processor (120) may execute at least one instruction to search for a complete node corresponding to an incomplete node in the personal knowledge graph based on relationships between nodes in the personal knowledge graph and knowledge attribute information included in the incomplete node. Furthermore, at least one processor (120) may execute at least one instruction to identify candidate knowledge attribute information corresponding to missing knowledge attribute information in the searched complete node.

[0164] Additionally, at least one processor (120) may execute at least one instruction to determine whether the searched complete node is identical to the incomplete node. Additionally, at least one processor (120) may execute at least one instruction to determine whether the two nodes are identical, thereby replacing the incomplete node with the searched complete node. If the two nodes are not identical, the missing knowledge attribute information may be updated with the candidate knowledge attribute information identified in the searched complete node.

[0165] Additionally, according to one embodiment of the present disclosure, at least one processor (120) may execute at least one instruction to infer candidate knowledge attribute information corresponding to missing knowledge attribute information based on at least one of graph information related to an incomplete node and feature information of an instance.

[0166] Additionally, at least one processor (120) may execute at least one instruction to infer candidate knowledge attribute information corresponding to missing knowledge attribute information by using a graph embedding vector based on graph information related to an incomplete node and a multi-modal embedding vector based on an image embedding vector or a text embedding vector of feature information of an instance.

[0167] Additionally, at least one processor (120) may execute at least one instruction to classify an instance using an image embedding vector or a text embedding vector of feature information of the instance, thereby inferring candidate knowledge attribute information corresponding to missing knowledge attribute information.

[0168] Additionally, at least one processor (120) may execute at least one instruction to update missing knowledge attribute information with inferred candidate knowledge attribute information.

[0169] Additionally, according to one embodiment of the present disclosure, at least one processor (120) may execute at least one instruction to perform a task of completing an incomplete node if the accuracy of candidate knowledge attribute information corresponding to user input or missing knowledge attribute information satisfies a predetermined condition.

[0170] Additionally, according to one embodiment of the present disclosure, at least one processor (120) may execute at least one instruction to determine a search category based on a user's behavior type for a personal knowledge graph. Furthermore, at least one processor (120) may execute at least one instruction to determine whether there are incomplete nodes with missing knowledge attribute information for each instance within the determined search category.

[0171] Additionally, according to one embodiment of the present disclosure, at least one processor (120) may execute at least one instruction to assign the same tag to missing knowledge attribute information included in each of different incomplete nodes if the missing knowledge attribute information has the same value, and to assign different tags to missing knowledge attribute information if the missing knowledge attribute information has different values.

[0172] The above description of the present disclosure is provided for illustrative purposes only, and those skilled in the art will readily appreciate that modifications to other specific forms can be made without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, components described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined manner.

[0173] While the present disclosure has been illustrated and described with reference to various embodiments, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present disclosure as defined by the claims and their equivalents.

Claims

1. A step (S710) in which an electronic device detects an instance including an incomplete node having missing knowledge attribute information in a personal knowledge graph; A step (S720) in which the electronic device checks candidate knowledge attribute information corresponding to the missing knowledge attribute information of the incomplete node; and A step (S730) in which the electronic device performs a task of completing the incomplete node based on the above-mentioned confirmed candidate knowledge attribute information; A method for managing a personal knowledge graph, performed by an electronic device, comprising:

2. In paragraph 1, The step (S720) of checking the above candidate knowledge attribute information is: A step of searching for a completed node corresponding to the incomplete node in the personal knowledge graph based on the relationship between nodes in the personal knowledge graph and the knowledge attribute information included in the incomplete node; and A step of checking the candidate knowledge attribute information corresponding to the missing knowledge attribute information in the searched completion node; A method comprising:

3. In paragraph 1 or 2, The step (S730) of performing the task of completing the above-mentioned incomplete node is as follows: A step of determining whether the searched completed node is identical to the incomplete node; and A method comprising the step of replacing the incomplete node with the searched complete node if the two nodes are identical as a result of the above judgment, and updating the missing knowledge attribute information with the candidate knowledge attribute information confirmed in the searched complete node if the two nodes are not identical.

4. In any one of paragraphs 1 to 3, The step (S720) of checking the above candidate knowledge attribute information is: A method comprising a step of inferring candidate knowledge attribute information corresponding to the missing knowledge attribute information based on at least one of graph information related to the incomplete node and feature information of the instance.

5. In any one of paragraphs 1 to 4, The step of inferring the above candidate knowledge attribute information is: A method for inferring candidate knowledge attribute information corresponding to the missing knowledge attribute information by using a graph embedding vector based on graph information related to the incomplete node and a multi-modal embedding vector based on an image embedding vector or a text embedding vector of feature information of the instance.

6. In any one of paragraphs 1 to 5, The step (S730) of performing the task of completing the above-mentioned incomplete node is as follows: A method comprising the step of updating the missing knowledge attribute information with the inferred candidate knowledge attribute information.

7. In any one of paragraphs 1 to 6, The step (S730) of performing the task of completing the above-mentioned incomplete node is as follows: A method for completing an incomplete node when there is a user input or the accuracy of the candidate knowledge attribute information corresponding to the missing knowledge attribute information satisfies a predetermined condition.

8. In any one of paragraphs 1 to 7, The above detecting step (S710) is A step of determining a search category according to the user's behavior type for the above personal knowledge graph; and A step of determining whether there is an incomplete node having the missing knowledge attribute information for each instance within the determined search category; A method comprising:

9. In any one of paragraphs 1 to 8, The above detecting step (S710) is A method further comprising the step of assigning the same tag to the missing knowledge attribute information contained in each different incomplete node if the missing knowledge attribute information has the same value, and assigning different tags to the missing knowledge attribute information if the missing knowledge attribute information has different values.

10. A memory (110) including one or more storage media for storing instructions; and comprising one or more processors (120) communicatively connected to the above memory, The above instructions, when individually or jointly executed by one or more processors (120), cause the electronic device (100) to: An electronic device (100) that detects an instance including an incomplete node having missing knowledge attribute information in a personal knowledge graph, identifies candidate knowledge attribute information corresponding to the missing knowledge attribute information of the incomplete node, and performs a task of completing the incomplete node based on the identified candidate knowledge attribute information.

11. In paragraph 10, The above instructions, when individually or jointly executed by one or more processors (120), cause the electronic device (100) to: An electronic device (100) that searches for a complete node corresponding to the incomplete node in the personal knowledge graph based on the relationship between nodes in the personal knowledge graph and the knowledge attribute information included in the incomplete node, and checks the candidate knowledge attribute information corresponding to the missing knowledge attribute information in the searched complete node.

12. In paragraph 10 or 11, The above instructions, when individually or jointly executed by one or more processors (120), cause the electronic device (100) to: Determine whether the searched completed node is identical to the incomplete node, An electronic device (100) that, as a result of the above judgment, if the two nodes are identical, replaces the incomplete node with the searched complete node, and if the two nodes are not identical, updates the missing knowledge attribute information with the candidate knowledge attribute information confirmed in the searched complete node.

13. In any one of paragraphs 10 to 12, The above instructions, when individually or jointly executed by one or more processors (120), cause the electronic device (100) to: An electronic device (100) that infers candidate knowledge attribute information corresponding to the missing knowledge attribute information based on at least one of graph information related to the incomplete node and characteristic information of the instance.

14. In any one of paragraphs 10 to 13, The above instructions, when individually or jointly executed by one or more processors (120), cause the electronic device (100) to: An electronic device (100) that determines a search category according to a user's behavior type for the personal knowledge graph, and determines whether there is an incomplete node having the missing knowledge attribute information for each instance within the determined search category.

15. In one or more non-transitory computer-readable storage media, One or more computer programs stored in the computer-readable storage medium, when individually or jointly executed by one or more processors of the electronic device, cause the electronic device to: A step of detecting instances containing incomplete nodes having missing knowledge attribute information in a personal knowledge graph; A step of checking candidate knowledge attribute information corresponding to the missing knowledge attribute information of the incomplete node; and A step of performing a task of completing the incomplete node based on the above-mentioned confirmed candidate knowledge attribute information; A computer-readable storage medium containing computer-executable instructions that instruct a computer to perform a task.

Images