Method for providing journal based on personal knowledge graph and user device using same

Abstract

Disclosed is a method for providing a journal based on a personal knowledge graph, the method comprising: generating a journal using a personal knowledge graph based on correlated user data; displaying, in a first area of a display screen, a first image generated on the basis of first user data used to generate the journal and text of the journal; obtaining a user input for modifying the first image displayed in the first area; and changing the text of the journal on the basis of the modification of the first image.

Description

Method for providing a journal based on a personal knowledge graph and a user device utilizing the same

[0001] The present invention relates to a method for providing a journal based on a personal knowledge graph and a user device using the same.

[0002] AI-based technologies are being utilized in diverse fields across industries. A variety of AI models are being developed and utilized in various fields. AI-based solutions are rapidly expanding beyond manufacturing to include robotics, transportation / logistics, healthcare, education, pharmaceuticals / biotechnology, and more. The adoption of such AI-based technologies is leading to enhanced competitiveness for companies and countries.

[0003] To improve the performance of AI models, knowledge bases can be used for AI model training and inference. One example of a knowledge base is a knowledge graph, which is a graph-type data structure.

[0004] Recently, there has been a lot of interest in on-device artificial intelligence technology that collects and processes information on user devices such as smartphones, rather than sending the information collected from user devices such as smartphones to a server for analysis and then sending it back to the user devices.

[0005] The information described in this background art section may be information already known or derived by the inventor prior to or during the completion of the embodiments of this application, or may be technical information acquired during the completion of the embodiments. Therefore, this background art section may include information that does not correspond to publicly known prior art.

[0006] Additional aspects are partly described in the description below, partly will be obvious from the description, or may be learned by practice of the disclosed embodiments.

[0007] According to one embodiment of the present disclosure, a method for providing a journal based on a personal knowledge graph is provided. The method for providing a journal based on a personal knowledge graph includes a step of generating a journal using a personal knowledge graph based on user data that is interrelated. Furthermore, the method for providing a journal based on a personal knowledge graph includes a step of displaying a first image generated based on first user data used to generate the journal and the text of the journal in a first area of ​​a display screen. Furthermore, the method for providing a journal based on a personal knowledge graph includes a step of obtaining a user input for modifying the first image displayed in the first area. Furthermore, the method for providing a journal based on a personal knowledge graph includes a step of modifying the text of the journal based on the modification of the first image.

[0008] According to one embodiment of the present disclosure, a computer-readable recording medium storing instructions that, when executed by at least one processor, cause a device to perform a method for providing a journal based on a personal knowledge graph is provided.

[0009] According to one embodiment of the present disclosure, a user device providing a journal based on a personal knowledge graph is provided. The user device includes a memory including one or more storage media for storing instructions, at least one processor including a processing circuit, and an input / output device for outputting a display screen. The instructions, when executed by the at least one processor, cause the user device to generate a journal using a personal knowledge graph based on correlated user data. Furthermore, the instructions, when executed by the at least one processor, cause the user device to display a first image generated based on first user data used to generate the journal and the text of the journal in a first area of ​​the display screen. Furthermore, the instructions, when executed by the at least one processor, cause the user device to obtain a user input for modifying the first image displayed in the first area. Furthermore, the instructions, when executed by the at least one processor, cause the user device to change the text of the journal based on the modification of the first image.

[0010] The above and other aspects, features, and advantages of one embodiment of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0011] FIG. 1 is a diagram illustrating an artificial intelligence platform based on a personal knowledge graph according to one embodiment of the present disclosure.

[0012] FIG. 2A is a diagram illustrating an operation of a user device constructing a personalized database based on a knowledge graph according to one embodiment of the present disclosure.

[0013] FIG. 2b is a diagram illustrating the use of a personal knowledge graph generated in a user device according to one embodiment of the present disclosure.

[0014] FIG. 3 is a diagram illustrating a personal knowledge graph generated in a user device according to one embodiment of the present disclosure.

[0015] FIG. 4 is a flowchart illustrating a method for generating a personal knowledge graph in a user device according to one embodiment of the present disclosure.

[0016] FIG. 5 is a diagram illustrating a first data group and a second data group clustered from user data related to multiple applications in a user device according to one embodiment of the present disclosure.

[0017] FIG. 6 is a flowchart illustrating a process of acquiring and expanding a second data group in a user device according to one embodiment of the present disclosure.

[0018] FIG. 7 is a diagram illustrating expansion of a second data group in a user device according to one embodiment of the present disclosure.

[0019] FIG. 8 is a flowchart illustrating a process of obtaining a personal knowledge graph based on interrelated user data in a user device according to one embodiment of the present disclosure.

[0020] FIG. 9 is a diagram illustrating an operation of a user device providing a personal knowledge graph-based service according to one embodiment of the present disclosure.

[0021] FIG. 10 is a diagram illustrating a home screen and a journal provision screen of a journal application executed on a user device according to one embodiment of the present disclosure.

[0022] FIG. 11 is a flowchart illustrating a method for providing a journal based on a personal knowledge graph according to one embodiment of the present disclosure.

[0023] FIG. 12 is a diagram illustrating a process of editing a journal by modifying a first image in a user device according to one embodiment of the present disclosure.

[0024] FIG. 13 is a diagram illustrating a process of editing a journal by modifying a first image in a user device according to one embodiment of the present disclosure.

[0025] FIG. 14 is a diagram illustrating a process of editing a journal by modifying a first image in a user device according to one embodiment of the present disclosure.

[0026] FIG. 15 is a diagram illustrating a process of editing a journal by modifying text of the journal in a user device according to one embodiment of the present disclosure.

[0027] FIG. 16 is a diagram illustrating a process of editing a journal by modifying text of the journal in a user device according to one embodiment of the present disclosure.

[0028] FIG. 17 is a flowchart illustrating a method for providing a journal based on a personal knowledge graph according to one embodiment of the present disclosure.

[0029] FIG. 18 is a drawing showing a second image for journal editing displayed in a second area of ​​a display screen of a user device according to one embodiment of the present disclosure.

[0030] FIG. 19 is a diagram illustrating editing a journal using a second image in a user device according to one embodiment of the present disclosure.

[0031] FIG. 20 is a diagram illustrating editing a journal by selecting a tone to apply to the journal in a user device according to one embodiment of the present disclosure.

[0032] FIG. 21 is a drawing showing a second image for journal editing displayed in a second area of ​​a display screen of a user device according to one embodiment of the present disclosure.

[0033] FIG. 22 is a diagram illustrating editing a journal using a second image in a user device according to one embodiment of the present disclosure.

[0034] FIG. 23 is a block diagram illustrating a user device according to one embodiment of the present disclosure.

[0035] FIG. 24 is a block diagram illustrating the configuration and operation of a user device according to one embodiment of the present disclosure.

[0036] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the drawings, identical components are designated by the same reference numerals, and redundant descriptions thereof will be omitted. The embodiments described herein are merely exemplary, and the present disclosure is not limited thereto and may be implemented in various forms. Unless the context clearly dictates otherwise, the singular form should be understood to include the plural. Technical or scientific terms used herein may have the same meaning as commonly understood by those skilled in the art.

[0037] Hereinafter, terms used in this specification will be briefly described, and the present disclosure will be described in detail. In this disclosure, the expression "at least one of a, b, or c" may include "a," "b," "c," "a and b," "a and c," "b and c," "all of a, b, and c," or variations thereof.

[0038] The terms used in this disclosure are selected from widely used, common terms, taking into account the functions of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, in which case their meanings will be described in detail in the relevant description. Therefore, the terms used in this disclosure should not be defined simply as names, but rather based on the meanings of the terms and the overall content of the disclosure.

[0039] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the art described herein. Furthermore, terms including ordinal numbers, such as "first" or "second," used herein may be used to describe various components, but such components should not be limited by such terms. Such terms are used solely to distinguish one component from another. The operations of a method may be performed in any suitable order unless the order is explicitly stated. Furthermore, all exemplary terms (e.g., etc.) are merely intended to elaborate technical ideas, and the scope of the present invention is not limited by such examples or terms unless otherwise defined by the claims.

[0040] When a part of the specification is said to "include" a component, unless otherwise specifically stated, this does not exclude other components but rather implies the inclusion of other components. Furthermore, terms such as "part" and "module" used in the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software.

[0041] The artificial intelligence-related functions according to the present disclosure are operated via a processor and memory. The processor may be comprised of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, an AP, a Digital Signal Processor (DSP), a graphics-only processor such as a GPU or a Vision Processing Unit (VPU), or an artificial intelligence-only processor such as an NPU. One or more processors control the processing of input data according to predefined operating rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0042] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is trained using a learning algorithm using a plurality of learning data, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0043] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and performs neural network operations through operations between the operation results of the previous layer and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

[0044] In this disclosure, the term "knowledge graph" refers to a knowledge base in graph form, which is a method for managing and exploring knowledge information, and which is based on a knowledge base that stores knowledge information and a graph that expresses it so that it can be analyzed in a network structure. A knowledge graph is a graph model that implements the knowledge accumulated in a knowledge base as a relationship between nodes and edges. A knowledge graph can be used to integrate data using a graph data model, topology, etc. To enable knowledge to be interconnected and integrated using a knowledge graph, a schema is implemented through an ontology, and a structure and dictionary (terminology) that can be shared with each other are used.

[0045] Semantic information, including common sense and factual knowledge, is organized as connections between nodes and edges, and by referencing this, various types of data can be converted into the form of a knowledge graph. Methods for representing a knowledge graph include, but are not limited to, the Labeled Property Graph (LPG), in which nodes and edges can each have properties, and the Resource Description Framework (RDF), which expresses relationships in a triple-fact structure of subject-predicate-object. A knowledge graph can be created by recognizing entities from various data, connecting them to appropriate entities in an existing knowledge base (entity linking), and extracting relationships between entities.

[0046] Knowledge graphs can be leveraged to enhance the performance of artificial intelligence. They can be used in models such as graph neural networks (GNNs) and graph convolution neural networks (GCNs), and can be used to provide explanations for results in explainable artificial intelligence (XAI).

[0047] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0048] The present disclosure will be described in detail with reference to the attached drawings below.

[0049] FIG. 1 is a diagram illustrating an artificial intelligence (AI) platform based on a personal knowledge graph according to one embodiment of the present disclosure.

[0050] Referring to FIG. 1, it shows that an AI platform based on a personal knowledge graph is provided to a user through a process of building a personalized database based on a knowledge graph (e.g., S110) and a process of providing a personalized AI service to a user through a service application based on a knowledge graph (e.g., S120).

[0051] In the process of building a personalized database based on a knowledge graph (e.g., S110), the user device (100) can create a personal knowledge graph in a structured form by referencing a general knowledge graph in a structured form, such as common sense and factual knowledge, with user data acquired from the user device (100). The user device (100) can convert various types of unstructured data into the form of a knowledge graph to build a personalized database based on a knowledge graph, i.e., a personal knowledge graph, and store it in storage. The user device (100) can database the user data acquired from the user device (100) and manage it in storage.

[0052] In the process of providing personalized AI services to users through a knowledge graph-based service application (e.g., S120), the user device (100) can provide various services using the personal knowledge graph stored in storage and databased user data. The user device (100) can use the personal knowledge graph to provide recommendation services, assistant services, QA (Question Answering) services, etc.

[0053] For example, if the user is performing an action, the user device (100) can recommend music that the user enjoys listening to during the action through a recommendation service, or provide information about related past experiences or upcoming events (e.g., calendar schedules) as well as knowledge information based on a personal knowledge graph through an assistant service. The user device (100) can identify the user's behavioral patterns from a personalized database, and if the user shows a pattern that is different from the user's usual behavioral patterns, the user device (100) can provide a solution service related to the cause of the different pattern through a recommendation service. The user device (100) can provide a customized answer to the user's question based on a personalized database through a QA service. The user device (100) can utilize the personalized database to provide a journal service that manages and describes the user's daily routine or special events.

[0054] FIG. 2a is a diagram for explaining an operation of a user device (100) constructing a personalized database based on a knowledge graph according to one embodiment of the present disclosure.

[0055] A user device (100) according to one embodiment of the present disclosure may be an electronic device capable of processing data. For example, the user device (100) may be an electronic device such as a smartphone, smart glasses, a wearable device, a digital camera, a laptop, an Augmented Reality (AR) device, or a Virtual Reality (VR) device. The user device (100) may be equipped with various types of neural network models. For example, the user device may be equipped with at least one model such as a Convolution Neural Network (CNN), a Graph Convolution Neural Network (GCN), a Graph Neural Network (GNN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), or a Bidirectional Recurrent Deep Neural Network (BRDNN), and may also use these models in combination.

[0056] A user device (100) according to one embodiment of the present disclosure can construct a personalized database based on a knowledge graph by converting various types of unstructured data into the form of a knowledge graph. The user device (100) can acquire various types of data and convert them into triple-format data. The user device (100) can map the converted data with an ontology stored in a semantic memory, and reflect data in a structured form that conforms to the ontology format into a personal knowledge graph. (For example, PKG of FIG. 2A) An ontology is a type of dictionary that defines terms that conceptualize data and the relationships between terms. An ontology can be expanded by adding external knowledge by converting it into triple-format data.

[0057] Data acquired from a user device (100) according to one embodiment of the present disclosure may be user data related to an application executed in the user device (100) or metadata regarding an event occurrence time, location, weather, etc., stored in the user device (100). The data acquired from the user device (100) may include at least one of data input from a user in the user device (100), data sensed by the user device (100), data received from the outside by the user device (100), and data processed by the user device (100).

[0058] The user device (100) can process unstructured data acquired from the user device (100) into structured data and store it in a personal knowledge graph. The user device (100) can prepare semantic information in a structured form in advance as an ontology. The user device (100) can refer to semantic information of various types of ontologies to process unstructured data acquired from the user device (100) into structured data and store it in a personalized database. For example, the user device (100) can convert various types of unstructured data into the form of a knowledge graph and store it in a personal knowledge graph.

[0059] Referring to FIG. 2a, an example is provided of a process in which a user device (100) collects user data related to an application executed on the user device (100) and builds a personalized database in the form of a knowledge graph.

[0060] In the data collection part of the user device (100), a user data collector (1030) can collect user data from a contact provider, a message provider, a media provider, CMH (content management hub) data, other data providers, etc. Each provider can store raw data-level data provided from at least one application. For example, a message provider can store user data in the form of raw data received from a text message application or a chat application. The user data collector (1030) is not limited to the example illustrated in FIG. 2A, and can collect user data from various types of applications, including third-party applications installed on the user device (100). A place data collector (1010) and a weather data collector (1020) can collect metadata about a place or weather from external information obtained from a user device (100). In FIG. 2A, the place data collector (1010), the weather data collector (1020), and the user data collector (1030) are distinguished, but are not limited thereto, and may be integrated into a single collector.

[0061] Each of the data collectors (1010, 1020, 1030) collects log data or raw data-level data. For example, the user data collector (1030) can collect the photo file itself from a photo management application that manages photos taken by a camera. The user data collector (1030) can receive user data transmitted by each application according to a protocol predefined by each domain or provider. When an event such as addition, change, or deletion of user data stored in each application occurs, the user data collector (1030) can receive the added user data, changed user data, or deleted user data from each application.

[0062] In the memory core part of the user device (100), the analyzer (1040) can generate a new form of user data by using part or all of the data collected by each of the data collectors (1010, 1020, 1030).

[0063] The analyzer (1040) can generate user data at the action information or activity information level from the data collected by each of the data collectors (1010, 1020, 1030). The analyzer (1040) can infer and determine the user's action of taking a photo from a photo, the user's action of eating food from a receipt paid at a restaurant, and the user's action of staying from the Global Positioning System (GPS)-based location information of the user's current location, thereby generating user data at the action information or activity information level. The action information is user data corresponding to the user's action (unit of action), and the activity information may be user data composed of a series of continuous action information. The activity information may have attribute data such that the start time and end time are different values, whereas the action information may have attribute data regarding the time at which the user's action occurred. For example, watching a movie can be considered activity information because the time at which the movie starts to be watched is different from the time at which the movie ends, whereas making a payment can be considered action information because the payment time is instantaneous. However, action information can also be processed as a type of activity information with the same start time and end time.

[0064] The analyzer (1040) can generate user data at the level of a personal knowledge graph. The analyzer (1040) can generate a personal knowledge graph by obtaining action information or activity information based on data collected by each of the data collectors (1010, 1020, 1030), connecting entities related to the action information or activity information as nodes, and storing attribute information of each node. For example, the personal knowledge graph can be composed of a lower layer containing user data at the low data level, a middle layer containing user data at the level of action information or activity information, and an upper layer corresponding to the title or topic level of the personal knowledge graph encompassing the action information or activity information. The uppermost layer of the personal knowledge graph can be connected to multiple nodes corresponding to activity information included in the middle layer. Each node corresponding to activity information can be connected to multiple nodes corresponding to user data at the low data level included in the lower layer as attribute data of each activity information. Nodes corresponding to activity information in a personal knowledge graph can be connected to each other, and nodes corresponding to user data can be connected to nodes corresponding to other user data. Nodes corresponding to similar user data within one or more personal knowledge graphs can be clustered and managed as a single group. For example, photos containing similar user faces can be managed as a single group, and when attribute information for one photo is entered, all photos within the group can share the entered attribute information. When a photo within a group is deleted, all photos within the group can be deleted from one or more personal knowledge graphs. For another example, a personal knowledge graph can represent user data generated within a given period of time as a root node with the given period information, and each node connected to the root node.The root node can be navigated through a given period of time. The root node can be connected to each individual knowledge graph, each of which has sub-period information as a sub-root node.

[0065] The encoder (1050) converts the user data generated or processed by the analyzer (1040) into triple-format data and maps it to a predetermined ontology, thereby inferring a standardized form of data that conforms to the ontology format, and can build a personal knowledge graph in the storage (1060) or reflect it in an existing personal knowledge graph. Various types of ontologies, such as content ontology, activity information ontology, environment ontology, and relationship ontology, may be prepared in advance in the user device (100). The encoder (1050) can integrate and use various ontologies prepared in the user device (100). For example, the encoder (1050) can convert card payment information paid on January 1 and user data related to a photo taken on January 1 into a triple format, and identify the converted user data as activity information of a purchase made on January 1 and activity information of taking a photo using an activity information ontology, thereby creating or updating a personal knowledge graph based on activity information in a standardized form.

[0066] The storage (1060) can store a personal knowledge graph. The user device (100) can manage the personal knowledge graph by building a personalized database based on the knowledge graph in the storage (1060). The user device (100) can record and store data related to events occurring by the user in the form of a personal knowledge graph at regular intervals. The user device (100) can store and manage a personal knowledge graph in which each node represents an object related to an event, in order to record and manage events occurring in the user device (100) using various types of user data.

[0067] FIG. 2b is a diagram illustrating the use of a personal knowledge graph generated in a user device (100) according to one embodiment of the present disclosure.

[0068] Referring to FIG. 2B, a user device (100) according to an embodiment of the present disclosure may collect user data acquired from the user device (100) and generate a personal knowledge graph on the topic of 'restaurant exploration'. Based on the collected user data, the user device (100) may generate a node of activity information called 'staying', a node of activity information called 'eating', and a node of activity information called 'taking pictures', and may connect the nodes of each activity information to generate a personal knowledge graph called 'restaurant pizza'. The user device (100) may store and manage the personal knowledge graph generated in this way in a personalized database based on a knowledge graph.

[0069] Since each node of the activity information constituting the personal knowledge graph stores attribute data regarding the start time and end time, the user device (100) can use the personal knowledge graph to determine whether there is an overlapping time between each activity information. If there is overlapping activity information in the same time zone, the user device (100) can utilize the attribute data regarding one activity information as attribute data regarding another activity information. The user device (100) can infer new information by integrating and using the raw data-level user data of multiple activity information from activity information with overlapping performance times using a predetermined inference model. For example, referring to FIG. 2B, from the activity information "Stay", it can be determined that the user was at a place called Italian Table in Gangnam-gu, Seoul from 2:50 PM to 3:50 PM on May 18. From the activity information "Eat", it can be determined that the user ate with his friend Daniel from 3:00 PM to 3:50 PM on May 18. From the activity information called 'taking pictures', it can be seen that the user took several pictures related to pizza from 3:20 PM to 3:25 PM on May 18. From this, it can be inferred that from 3:20 PM to 3:25 PM on May 18, the time when the pizza was taken, the user took pictures of pizza while eating with his friend Daniel at a place called Italian Table in Gangnam-gu, Seoul. Therefore, when the user searches for a place where he ate pizza or a person he ate pizza with on the user device (100), the user can use the personal knowledge graph called 'pizza restaurant' to confirm that the place where he ate pizza is 'Italian Table in Gangnam-gu, Seoul' and the person he ate pizza with is 'his friend Daniel'.

[0070] FIG. 3 illustrates a personal knowledge graph generated in a user device (100) according to one embodiment of the present disclosure.

[0071] According to one embodiment of the present disclosure, a user device (100) can collect user data related to each application and classify the collected data according to predetermined criteria. For example, the user device (100) can cluster and classify user data that was collected at a similar time or location. The user device (100) can identify user action information or activity information from user data related to various applications using an action information ontology or an activity information ontology. The user device (100) can obtain a personal knowledge graph with each node representing the user's action information or activity information.

[0072] According to one embodiment of the present disclosure, a user device (100) can generate a personal knowledge graph that connects nodes of activity information from a specific time period. Looking at the personal knowledge graph illustrated in FIG. 3 , it can be seen that nodes of activity information generated based on user data generated on May 18th among user data associated with each application are connected.

[0073] Looking at each node within the personal knowledge graph, we can see that a node for activity information called "Watching a Video" is created based on user data related to the title, artist information, playback start time, and playback end time of a video played through a content playback application. A node for activity information called "Meeting" is created based on user data related to the title, start time, and end time of an event recorded through a schedule management application. A node for activity information called "Staying" is created based on user data related to the address and points of interest of a location recorded through a GPS-based application. A node for activity information called "Calling" is created based on user data related to the person called, the start time, and the end time of the call through a phone application. A node for activity information called "Taking Pictures" is created based on user data related to the photo taken through a photo management application, the subject of the photo, the start time, and the end time of the photo taken, and the information about the person stored in the contacts application. Through the messaging app, we can see that a node for activity information called "purchasing" has been created based on user data related to payment information (payment target, payment card, payment amount) and payment time. The activity information for each node in the personal knowledge graph is related to activity information that occurred on May 18th.

[0074] By utilizing a personal knowledge graph in a service, a user device (100) can provide customized services and personalized experiences to individual users. As illustrated in FIG. 3, utilizing a personal knowledge graph based on user data with recognized correlations in a service can eliminate noise information, providing improved service quality or providing a new type of service based on user data with recognized correlations. Below, a method for generating a personal knowledge graph based on user data with recognized correlations and a method for providing a service using a personal knowledge graph based on user data with recognized correlations are described.

[0075] FIG. 4 is a flowchart illustrating a method for generating a personal knowledge graph in a user device (100) according to one embodiment of the present disclosure.

[0076] Referring to FIG. 4, the user device (100) can obtain a first data group clustered from user data related to a first application (e.g., S410). The first application is an application installed on the user device (100), and may be a schedule management application, a chat application, a photo management application, a GPS-based application (e.g., a map application), an SNS application, a health information management application, a message application, a phone application, a weather forecast application, a content playback application, etc.

[0077] The first data group may include one or more user data. For example, if the user data is a chat message, the user device (100) may cluster the chat messages based on information regarding the time of creation of the chat message or information regarding the chat partner. Based on the time of creation of each chat message, the user device (100) may obtain chat messages for a predetermined period of time as the first data group. Based on information regarding the chat partner, the user device (100) may obtain chat messages with a specific partner as the first data group.

[0078] For example, if the user data is a photograph, the user device (100) can cluster the photographs based on at least one of information about the time of photo creation (time of capture), information about the location where the photograph was captured, and information about the photographed object. The user device (100) can obtain photographs captured consecutively as a first data group based on the time of capture. The user device (100) can obtain photographs captured at the same location as a first data group. The user device (100) can obtain photographs including the same object as a first data group based on the photographed object. The user device (100) can also obtain one photograph as a first data group.

[0079] FIG. 5 is a diagram for explaining a first data group and a second data group clustered from user data related to multiple applications in a user device (100) according to one embodiment of the present disclosure.

[0080] Referring to Figure 5, user data generated by each of a schedule management application, a chat application, a photo management application, a GPS-based application, and a SNS application are illustrated. For example, a schedule management application can generate schedule information by allowing a user to record schedules. A chat application can generate chat messages while chatting with others. A photo management application can store photos or videos created by a user taking photos or videos. A GPS-based application can generate location information at predetermined time intervals. An SNS application can generate SNS feeds by allowing a user to update content on the SNS.

[0081] As illustrated in FIG. 5, chat messages related to a chat application may be continuously exchanged with a counterpart, or may be sent or received intermittently or as a single message. In the case of chat messages continuously exchanged or chat messages with a specific counterpart, there is a high possibility that they are user data with mutual correlation. The user device (100) can cluster chat messages generated continuously or chat messages exchanged with a specific counterpart over a certain period of time to obtain a first chat message data group and a second chat message data group.

[0082] For photos or videos associated with photo management applications, a single photo or video may be captured at a specific moment or location, or a series of continuous shots or a combination of both. Since even a single shot can be meaningful, even a single photo needs to be processed as a data group. The user device (100) can cluster one or more photos or videos to obtain a first image data group and a second image data group.

[0083] In the case of schedule information related to a schedule management application, location information related to a GPS-based application, information about an SNS feed related to an SNS application, etc., the user device (100) can obtain a data group by clustering one or more user data.

[0084] The user device (100) may obtain any data group among the clustered data groups as the first data group. For example, the user device (100) may obtain any data group clustered from user data related to any application as the first data group.

[0085] Referring again to FIG. 4, the user device (100) can check the generation period (hereinafter, the first generation period) corresponding to the first data group based on the generation time of the user data included in the first data group. (Example: S420)

[0086] For example, the user device (100) may determine the period from the creation time of the earliest user data among the user data belonging to the first data group to the creation time of the latest user data as the first creation period. Referring again to FIG. 5, the user device (100) may obtain the first chat message data group clustered from the user data related to the chat application as the first data group. The user device (100) may determine the period from the creation time of the earliest chat message among the chat messages belonging to the first chat message data group to the creation time of the latest chat message as the first creation period (T1_C).

[0087] For example, the user device (100) can identify a period of time before and after the creation time of user data belonging to the first data group as the first creation period. Referring again to FIG. 5, the user device (100) can obtain a second image data group clustered from user data related to a photo management application as the first data group. The user device (100) can determine a period of time before and after the creation time of a photo belonging to the second image data group as the first creation period (T1_P).

[0088] Referring again to FIG. 4, the user device (100) may obtain at least one second data group corresponding to the first generation period, clustered from user data related to at least one second application different from the first application (e.g., S430).

[0089] Referring back to FIG. 5, if the user device (100) acquires the first chat message data group as the first data group, the first schedule data group, the first image data group, the first location information data group, and the SNS feed data group generated in the first generation period (T1_C) corresponding to the first chat message data group can be acquired as the second data group. The generation periods of each of the first schedule data group, the first image data group, the first location information data group, and the SNS feed data group generated in the first generation period (T1_C) corresponding to the first chat message data group do not exceed the range of the first generation period (T1_C) corresponding to the first chat message data group.

[0090] If the user device (100) acquires the second image data group as the first data group, the second schedule data group, the second chat message data group, and the second location information data group generated in the first generation period (T1_P) corresponding to the second image data group can be acquired as the second data group. The generation periods of each of the second schedule data group, the second chat message data group, and the second location information data group generated in the first generation period (T1_P) corresponding to the second image data group do not exceed the range of the first generation period (T1_C) corresponding to the second image data group.

[0091] If the second generation period corresponding to the second data group exceeds the range of the first generation period corresponding to the first data group, the user device (100) can acquire the second data group by expanding the range of the first generation period. In this regard, a detailed description will be provided with reference to FIGS. 6 and 7.

[0092] FIG. 6 is a flowchart illustrating a process of expanding and acquiring a second data group in a user device (100) according to one embodiment of the present disclosure.

[0093] Referring to FIG. 6, the user device (100) can extract user data corresponding to the first generation period corresponding to the first data group among user data related to at least one second application (e.g., S610). If the user data related to the second application is continuously generated before the start time and after the end time of the first generation period corresponding to the first data group, the user device (100) can extract user data that is continuously generated outside the first generation period corresponding to the first data group.

[0094] The user device (100) may cluster extracted user data related to at least one second application based on information about the time of generation to obtain at least one second data group (e.g., S620). The at least one second data group may include at least one data group related to the second application and at least one data group related to the third application.

[0095] The user device (100) can determine whether the second generation period corresponding to at least one second data group is outside the first generation period. (For example, S630) The user device (100) can check the period from the generation time of the earliest generated user data among the user data belonging to at least one second data group to the generation time of the latest generated user data to determine whether it is outside the range of the first generation period corresponding to the first data group. The user device (100) can determine whether the range of the generation period of at least one data group related to the second application and the range of the generation period of at least one data group related to the third application are outside the range of the first generation period corresponding to the first data group related to the first application.

[0096] The user device (100) may extend the first generation period based on the second generation period if the second generation period corresponding to at least one second data group exceeds the first generation period (e.g., S640). The user device (100) may repeat the process of acquiring at least one second data group for the extended first generation period. The user device (100) may end the process of acquiring at least one second data group if the second generation period corresponding to at least one second data group does not exceed the first generation period.

[0097] FIG. 7 is a diagram for explaining expansion of a second data group in a user device (100) according to one embodiment of the present disclosure.

[0098] Referring to FIG. 7, a user device (100) can obtain a clustered image data group (a first data group) from user data related to a photo management application (a first application). The user device (100) can identify a first generation period (T1) corresponding to the image data group. The user device (100) can obtain a schedule data group corresponding to the first generation period (T1) clustered from schedule information related to a schedule management application. The user device (100) can obtain a first chat message data group and a second chat message data group corresponding to the first generation period clustered from chat messages related to a chat application. The user device (100) can obtain a first location information data group corresponding to the first generation period (T1) clustered from location information related to a GPS-based application. Since there is no information regarding an SNS feed related to an SNS application corresponding to the first generation period (T1), the user device (100) cannot obtain the SNS feed data group corresponding to the first generation period (T1) from the information regarding the SNS feed related to the SNS application. Accordingly, the user device (100) can obtain the schedule data group, the first chat message data group, the second chat message data group, and the first location information data group corresponding to the first generation period (T1) as the second data group.

[0099] At this time, it can be seen that the generation period corresponding to the first chat message data group and the generation period corresponding to the second chat message data group are outside the first generation period (T1) corresponding to the video data group. The user device (100) can extend the first generation period (T1) to a new first generation period (T2) based on the generation period corresponding to the first chat message data group and the generation period corresponding to the second chat message data group. The user device (100) can obtain at least one second data group corresponding to the extended first generation period (T2).

[0100] The user device (100) can obtain a schedule data group corresponding to an extended first generation period (T2) clustered from schedule information related to a schedule management application. The user device (100) can obtain a first chat message data group and a second chat message data group corresponding to an extended first generation period (T2) clustered from chat messages related to a chat application. The user device (100) can obtain a first location information data group and a second location data group corresponding to an extended first generation period (T2) clustered from location information related to a GPS-based application. The user device (100) can obtain an SNS feed data group corresponding to an extended first generation period (T2) clustered from information about an SNS feed related to an SNS application. Consequently, as the existing first generation period (T1) is changed to the extended first generation period (T2), the user device (100) can further obtain a second location information data group and an SNS feed data group as a second data group.

[0101] Referring again to FIG. 4, the user device (100) can obtain a personal knowledge graph based on user data that is interrelated from the first data group and at least one second data group (e.g., S440).

[0102] Interrelationship refers to the formation or establishment of a relationship between user data. For example, if "User Data A" and "User Data B" have a causal or sequential relationship based on a specific point in time, "User Data A" and "User Data B" can be considered interrelated. If "User Data C" and "User Data D" share common attributes, such as being related to the same subject or generated in the same location, "User Data C" and "User Data D" can be considered interrelated.

[0103] Personal knowledge graphs based on user data can be used to represent user characteristics, tendencies, preferences, and preferences, or to explain specific events related to the user. A personal knowledge graph based on user data with recognized interrelationships can be optimized for providing personalized, knowledge-graph-based services.

[0104] FIG. 8 is a flowchart illustrating a process of obtaining a personal knowledge graph based on interrelated user data in a user device (100) according to one embodiment of the present disclosure.

[0105] Referring to FIG. 8, the user device (100) can generate a personal knowledge graph based on a first data group and at least one second data group. (e.g., S810) The user device (100) can generate a personal knowledge graph by obtaining action information or activity information based on all user data included in the first data group and at least one second data group, connecting the action information or activity information to each node, and storing attribute information of each node.

[0106] The user device (100) can determine the interrelationship of each node in the personal knowledge graph. (e.g., S820) The user device (100) can determine the interrelationship of each node based on at least one of the presence or absence of a node connected to the first node based on user data of the first data group (first determination criterion), the number of nodes connected to the first node (second determination criterion), the presence or absence of a node connected to the second node connected to the first node (third determination criterion), and the number of nodes connected to the second node (fourth determination criterion).

[0107] The user device (100) can obtain the correlation between each node based on a predetermined function that takes as input at least one of the digitized values ​​according to each judgment criterion or the output value of a predetermined learning model. The predetermined function may be a function that outputs a value through a predetermined operation among the digitized values ​​according to the judgment criterion or selects a value according to a predetermined rule. Alternatively, the predetermined function may be a function that calculates an average, such as an arithmetic mean, a harmonic mean, a geometric mean, a weighted mean, etc., of the digitized values ​​according to the judgment criterion, or that outputs a minimum or maximum value. Alternatively, the predetermined function may be a function that takes as input at least one of the digitized values ​​according to the judgment criterion and outputs a predetermined calculation result. The predetermined learning model may be a correlation determination model in the form of a deep learning model or a machine learning model that takes as input at least one of the digitized values ​​according to the judgment criterion. However, if the user device (100) cannot determine a digitized value according to all judgment criteria, the user device (100) may determine a predetermined value in advance as a value indicating the correlation.

[0108] The user device (100) can remove nodes that are not interconnected from the personal knowledge graph. (e.g., S830) The user device (100) can remove nodes that have values ​​indicating interconnection lower than a predetermined standard and nodes at a level lower than that from the personal knowledge graph.

[0109] The user device (100) can determine whether a personal knowledge graph has uniqueness. (e.g., S840) The user device (100) can generate sentences to be input into a large language model or a machine learning-based model that determines uniqueness from user data of each node reflected in the personal knowledge graph. The user device (100) can input the generated sentences into a large language model or a machine learning-based model that determines uniqueness, thereby determining whether a personal knowledge graph has uniqueness and determining a topic or title of the personal knowledge graph.

[0110] The user device (100) can store a personal knowledge graph based on user data that is interrelated when there is a specificity in the personal knowledge graph (e.g., S850). Even if the personal knowledge graph is based on user data that is interrelated, if it is determined that there is no specificity in the personal knowledge graph, the user device (100) does not store the personal knowledge graph.

[0111] FIG. 9 is a diagram for explaining an operation of a user device (100) providing a personal knowledge graph-based service according to one embodiment of the present disclosure.

[0112] According to one embodiment of the present disclosure, in the service providing portion of the user device (100), the service logic module (1090) can execute a predetermined service according to a user's input through the service request module (1095). The service request module (1095) can receive a user's input, such as a request to execute a predetermined application, an input of a search word, or a touch on a user interface screen to set a specific item, letter, number, period, etc. For example, the service logic module (1090) can request a personal knowledge graph or user data of the personal knowledge graph related to a retriever (1080) according to a request to execute a predetermined application or an input of a search word input by the user. The service logic module (1090) can provide a predetermined service by using a personal knowledge graph based on user data that is interrelated or user data acquired from the personal knowledge graph.

[0113] In the memory core portion of the user device (100), the retriever (1080) can query the personal knowledge graph stored in the storage (1060) or search for user data included in the personal knowledge graph. The retriever (1080) may include a query engine or a search engine. The retriever (1080) can retrieve a personal knowledge graph by date or a personal knowledge graph of a specific date or a specific period from the storage (1060) and transmit the retrieved knowledge graph to the service logic module (1090). The retriever (1080) can retrieve a personal knowledge graph corresponding to a search word or query or user data included in the personal knowledge graph from the storage (1060) and transmit the retrieved knowledge graph to the service logic module (1090). The retriever (1080) can check the IRI (Internationalized Resource Identifier) ​​of the representative word for the entered search word using an embedding vector comparison method or a thesaurus DB. The receiver (1080) can construct a query using IRI and receive query results for a personal knowledge graph from the storage (1060).

[0114] The decoder (1070) performs the reverse role of the encoder (1050) of FIG. 2A. The decoder (1070) can obtain data in a form that can be utilized by the service logic module (1090) from the standardized data obtained from the personal knowledge graph.

[0115] FIG. 10 is a diagram illustrating a home screen and a journal provision screen of a journal application executed on a user device (100) according to one embodiment of the present disclosure.

[0116] The user device (100) can execute a journal application based on user input. The journal application is installed on the user device (100) and can process and verify user data stored on the user device (100) to create a journal. The user device (100) can create a journal using a personal knowledge graph based on interrelated user data through the journal application.

[0117] Referring to FIG. 10, the home screen of a journal application provided on a display screen of a user device (100) is shown. The home screen of the journal application can provide representative images related to journals for each date to indicate whether a journal has been created on that date. For example, as illustrated in FIG. 10, the home screen of the journal application can display a preview image related to a journal for each date from today to a date corresponding to a predetermined period in the past, if there is a journal created for that date. If there is no journal created for that date, the home screen of the journal application can display only the date without a preview image.

[0118] The user device (100) can switch from the home screen of the journal application to the journal provision screen based on a user input. If a user inputs selecting a specific date on the home screen of the journal application, the user device (100) can display the journal provision screen on the display screen to show the journal corresponding to that date. For example, as illustrated in FIG. 10, if a user inputs selecting a representative image corresponding to the 18th on the home screen of the journal application, the user device (100) can display the journal corresponding to the 18th on the journal provision screen.

[0119] Referring to Figure 10, the journal can be displayed in a first area of ​​the display screen. The first area may correspond to one area of ​​the display screen divided into a predetermined number of sections. For example, if the display screen is divided into two sections, either vertically or horizontally, the first area may correspond to one area of ​​the two-sectioned display screen.

[0120] The first area of ​​the display screen is a journal area for displaying the journal. The first area of ​​the display screen may be a portion of the entire display screen. The first area of ​​the display screen may be a region of a predetermined size or a region of variable size that may vary depending on the content of the generated journal. The first area may be composed of a text display area and a first image display area. The position, size, shape, style, etc. of the text display area and the first image display area may be modified depending on the generated journal.

[0121] The text display area can display the journal's title and its contents as text at the top of the first area. The journal's title and contents may be generated using first user data extracted from a personal knowledge graph based on interrelated user data.

[0122] The first image display area can output a first image based on the first user data used to generate the journal. The first user data can be composed of at least one piece of user data, and the first image can be composed of at least one image element.

[0123] Referring to Fig. 10, the first image may be composed of multiple image elements, each of which may be an image, photograph, graphic, or the like. Each image element may be displayed in a different size depending on the degree to which the user data corresponding to the image element is related to the content of the journal. Each image element may be displayed to ensure a minimum size that can be visually confirmed by the user. If all image elements cannot be displayed in the first image display area of ​​the display screen, all image elements may be viewed by scrolling left and right or up and down.

[0124] FIG. 11 is a flowchart illustrating a method for providing a journal based on a personal knowledge graph according to one embodiment of the present disclosure.

[0125] Referring to FIG. 11, the user device (100) can create a journal using a personal knowledge graph based on user data that is interrelated (e.g., S1110). The user device (100) can obtain the text of the journal in sentence form using a journal creation model that inputs the personal knowledge graph based on the user data that is interrelated. The journal creation model is a Large Language Model (LLM), and can input user data (hereinafter, “first user data”) corresponding to some or all of the user data used to create the personal knowledge graph, and output the text of the journal in sentence form corresponding to the first user data. The user device (100) can obtain the text of the journal corresponding to the first user data and the first image based on the first user data.

[0126] The user device (100) may display a first image and the text of the journal based on the first user data used to create the journal in a first area of ​​the display screen (e.g., S1120). The first user data may be user data extracted from a personal knowledge graph based on criteria such as the level of correlation between user data or the time, place, and target of creation of the user data. The first user data may include a plurality of user data. The first image may include a plurality of image elements corresponding to the plurality of user data. The first image may include image elements such as a representative image, a photograph, or a graphic, such as a thumbnail or a preview image.

[0127] For example, the first user data may be at least one piece of activity information obtained from a personal knowledge graph based on interrelated activity information. The first image may be represented by image elements such as a representative image, photo, or graphic based on the at least one piece of activity information. Alternatively, the first image may include a knowledge graph with at least one piece of activity information as a node. Each node of the knowledge graph may be represented by an image element such as a representative image, photo, or graphic based on the activity information.

[0128] The user device (100) can obtain a user's input for modifying a first image displayed in a first area. (S1130) The user device (100) can obtain a user's input for modifying the first image. The user's input for modifying the first image may be to modify any one of a plurality of image elements constituting the first image. For example, the user's input for modifying the first image may be to modify the first image by changing at least one of the type, position, state, and number of image elements constituting the first image. When the first image includes a knowledge graph based on activity information composed of action information corresponding to a user's action and a series of continuous action information, the user's input for modifying the first image may be to modify the knowledge graph by changing at least one of the type, position, state, and number of nodes of the knowledge graph.

[0129] The user device (100) can change the text of the journal based on the modification of the first image displayed in the first area. (For example, S1140) The user device (100) can change the text of the journal based on the change of at least one of the type, position, state, and number of image elements constituting the first image. The first user data used to create a journal in a personal knowledge graph can be changed based on the modification of the first image, and further, the existing personal knowledge graph can also be changed. The user device (100) can obtain the text of the changed journal using a journal generation model that inputs the first user data changed based on the modification of the first image. The journal generation model can input the first user data changed in response to the modified first image, and output the text of the journal in the form of sentences corresponding to the changed first user data.

[0130] FIG. 12 is a diagram for explaining a process of editing a journal by modifying a first image in a user device (100) according to one embodiment of the present disclosure.

[0131] A user device (100) may create a journal using a personal knowledge graph based on interrelated user data, and display a first image and text of the journal based on the first user data used to create the journal in a first area of ​​a display screen. The first image may include a plurality of image elements corresponding to a plurality of user data, and each image element may be an image, a photograph, a graphic, or the like.

[0132] According to one embodiment of the present disclosure, the user device (100) can obtain a user input for selecting a first image element from among image elements constituting a first image. For example, the user can select the first image element from among a plurality of image elements output in a first image display area by tapping or double-tapping via a touch screen of the user device (100). The first image element can be adjusted in color, brightness, contrast, etc. to be distinguished from other image elements in order to indicate a selected state to the user.

[0133] The user device (100) can change the text of the journal so that the text corresponding to the first image element among the text of the journal is identifiable. In response to a user input selecting the first image element, the user device (100) can change the text of the journal so that the text corresponding to the first image element among the text of the journal is displayed distinctly from other text. For example, the user device (100) can output the text corresponding to the selected first image element among the text of the journal by adjusting the color, brightness, contrast, etc. so that the text is identifiable from other text by using a journal generation model that inputs first user data that has been changed according to the changed state of the first image element.

[0134] Referring to FIG. 12, the user device (100) obtains a user input for selecting an image related to a cultural center schedule among the image elements constituting the first image, and changes the color of the text of the corresponding part so that the text corresponding to the cultural center schedule among the text of the journal is identified.

[0135] According to one embodiment of the present disclosure, the user device (100) can obtain a user input for deleting a first image element from among the image elements constituting the first image. For example, the user can delete the first image element from among the plurality of image elements output in the first image display area by long-pressing the first image element through the touch screen of the user device (100). The deleted first image element can be replaced with another image element.

[0136] The user device (100) may change the text of the journal so that the text corresponding to the first image element is deleted from the text of the journal. In response to a user input deleting the first image element, the user device (100) may change the text of the journal so that the text corresponding to the first image element is excluded from the content of the journal. For example, the user device (100) may change the text of the journal so that the text corresponding to the deleted first image element is deleted using a journal generation model that inputs first user data that has been changed according to the deletion of the first image element.

[0137] Referring to FIG. 12, the user device (100) obtains a user input to delete an image related to the cultural center schedule among the image elements constituting the first image by long-pressing it, and changes the contents of the journal by deleting the text corresponding to the cultural center schedule among the texts of the journal.

[0138] FIG. 13 is a diagram for explaining a process of editing a journal by modifying a first image in a user device (100) according to one embodiment of the present disclosure.

[0139] The user device (100) may create a journal using a personal knowledge graph based on interrelated user data, and display a first image and text of the journal based on the first user data used to create the journal in a first area of ​​the display screen. The first user data may be action information and activity information obtained from the personal knowledge graph. The first image may include a knowledge graph based on the action information or activity information. In this case, each node of the knowledge graph may be expressed as an image element, such as an image, photo, or graphic, based on the action information or activity information.

[0140] Unlike the first image of Fig. 12, which displays image elements such as images, photographs, and graphics arranged in an enumerated manner, the first image of Fig. 13 may include a knowledge graph based on activity information. Each node of the knowledge graph may be an image element such as an image, photograph, or graphic based on activity information.

[0141] According to one embodiment of the present disclosure, a user device (100) may create a journal using a personal knowledge graph based on interrelated user data, and then, when text of the journal is directly entered based on a user input, an image element corresponding to the directly entered text may be displayed as a first node of a knowledge graph based on the first user data used to create the journal. The first node may be adjusted in color, brightness, contrast, etc., so as to be distinguished from other nodes of the knowledge graph.

[0142] According to one embodiment of the present disclosure, the user device (100) may obtain a user input for moving the position of a first node of a knowledge graph within a first region. For example, the user device (100) may receive a user input for dragging a first node through a touch screen, and may change and display the position of the first node according to the direction of the drag. In response to the user input for moving the position of the first node, the user device (100) may display an edge between the first node and at least one node of the knowledge graph based on the mutual correlation between the first node and other nodes of the knowledge graph. For example, if the first node has a higher mutual correlation with a third node than with a second node of the knowledge graph, the edge connecting the first node and the third node may be displayed with a thicker or darker line than the edge connecting the first node and the second node. The user may drag and drop the first node to a position where an edge is formed from another node of the knowledge graph with respect to the first node.

[0143] The user device (100) can change the text of the journal based on the modification of the knowledge graph according to the changed position of the first node. In response to a user input for moving the position of the first node within the first area, the user device (100) can change the text of the journal based on the modification of the knowledge graph according to the changed position of the dragged and dropped first node.

[0144] For example, as illustrated in FIG. 13, if a user drags and drops a first node related to cultural center schedule information among the nodes of the knowledge graph, the user device (100) can change the text of the journal based on the modification of the knowledge graph according to the changed position of the first node. The user device (100) can use a journal creation model that inputs the changed knowledge graph according to the changed position (or connection relationship with other nodes) of the first node. In the example of FIG. 13, if the nodes that were connected around the node corresponding to the outing per household are connected around the node showing the son's face photo as the first node is moved, the user device (100) can change the order of the contents and text of the journal.

[0145] FIG. 14 is a diagram for explaining a process of editing a journal by modifying a first image in a user device (100) according to one embodiment of the present disclosure.

[0146] A user device (100) may create a journal using a personal knowledge graph based on interrelated user data, and display a first image and text of the journal based on the first user data used to create the journal in a first area of ​​a display screen. The first user data may be action information and activity information obtained from the personal knowledge graph. The first image may represent a knowledge graph based on the action information or activity information. In this case, each node of the knowledge graph may be expressed as an image element, such as an image, photo, or graphic, based on the action information or activity information.

[0147] The first image of Figure 14 may represent a knowledge graph based on activity information. Each node of the knowledge graph may be an image element, such as an image, photo, or graphic, based on the activity information.

[0148] According to one embodiment of the present disclosure, a user device (100) can obtain user input for modifying user data associated with a first node of a knowledge graph. For example, the user device (100) can receive a user input for spreading the first node via a touch screen and display the first node by changing it into a plurality of sub-nodes, each of which displays user data. The user can select a sub-node corresponding to the user data to be modified and modify the user data or delete or add the sub-node.

[0149] The user device (100) can change the text of the journal based on the modification of the knowledge graph according to the modified user data. In response to a user input modifying the user data corresponding to the first node, the user device (100) can change the text of the journal based on the modification of the knowledge graph according to the modified user data.

[0150] For example, as illustrated in FIG. 14, if a user selects a first sub-node among the sub-nodes of the first node to modify user data related to time information and modifies the time information, the user device (100) can change the text of the journal based on the modification of the knowledge graph according to the modified time information, using a journal generation model that inputs user data reflecting the modified time information. In the example of FIG. 14, if the time information of the outing is modified to a time before the cultural center schedule, the content of the journal can be changed to indicate that the cultural center schedule was carried out after the outing.

[0151] FIG. 15 is a diagram for explaining a process of editing a journal by modifying the text of the journal in a user device (100) according to one embodiment of the present disclosure.

[0152] The user device (100) may create a journal using a personal knowledge graph based on interrelated user data, and display a first image and text of the journal based on the first user data used to create the journal in a first area of ​​the display screen. The first user data may be action information and activity information obtained from the personal knowledge graph. The first image may include a knowledge graph based on the action information or activity information. In this case, each node of the knowledge graph may be expressed as an image element, such as an image, photo, or graphic, based on the action information or activity information.

[0153] The first image of Figure 15 may include a knowledge graph based on activity information. Each node of the knowledge graph may be an image element, such as an image, photograph, or graphic, based on the activity information. In the example of Figure 15, the text display area displays the journal title "Family Dining Out" along with the text content of the journal entry about going out and dining out with family. The first image represents a knowledge graph including the first and second nodes, each corresponding to the activity information used to create the journal, "Going Out" and "Dining Out."

[0154] According to one embodiment of the present disclosure, the user device (100) can obtain a user input that spreads each node of the knowledge graph represented by the first image through a touch screen. In response to the user input, the user device (100) can display detailed user data corresponding to 'going out' and 'eating out' by changing each of the first node and the second node into a plurality of sub-nodes, which are lower-level structures. In the example of FIG. 15, in relation to the activity information 'going out', detailed user data such as the time of starting the movement, the time of ending the movement, the departure location, the arrival location, and the location around the arrival location can be expressed as a plurality of sub-nodes. In relation to the activity information 'eating out', detailed user data such as the time of eating out, the person eating out with, and the dining location can be expressed as a plurality of sub-nodes. It can be seen that the first node 'going out' is activity information that corresponds to the prerequisite of the second node 'eating out'.

[0155] The user device (100) can manage the user data text of the knowledge graph and the corresponding IRI by mapping them as a mapping table. The user device (100) can perform a relational search to receive query results for the knowledge graph by constructing a query using the IRI related to a word or sentence selected by the user. When the user selects a word or sentence, the user device (100) can obtain recommended words or sentences from the knowledge graph through a relational search.

[0156] In the example of FIG. 15, since the user actually parked in the hospital building's parking lot and then went to the restaurant, the journal records the user's trip as having gone to the hospital and then eaten out, based on the navigation information collected from the user device (100). Since it would be natural to change "hospital" to "restaurant" in the context, the user may wish to edit the journal text. When the user taps the text portion they wish to edit, the user device (100) can provide replacement words or sentences in a drop-down menu or new window based on the knowledge graph. The user device (100) can change the journal text based on the user's input for editing the text.

[0157] FIG. 16 is a diagram for explaining a process of editing a journal by modifying the text of the journal in a user device (100) according to one embodiment of the present disclosure.

[0158] Descriptions of aspects identical or similar to those described above may be omitted.

[0159] Referring to Figure 16, the first node, 'going out', is activity information that corresponds to the prerequisite of the second node, 'eating out', so it can be seen that a sentence based on 'going out', which is activity information corresponding to the first node, is in a relationship that corresponds to the prerequisite of a sentence based on 'eating out', which is activity information corresponding to the second node.

[0160] According to one embodiment of the present disclosure, the user device (100) may obtain a user input by long-pressing a sentence based on the activity information "eating out" through a touch screen to select it. In response to the user input, the user device (100) may display that the sentence based on the activity information "eating out" and the sentence based on the activity information "going out" are in a prerequisite relationship. If the user deletes the sentence based on the activity information "eating out," the user device (100) may request feedback from the user as to whether to also delete or keep the sentence based on the activity information "going out" that is in a prerequisite relationship. The user device (100) may delete the sentence selected and deleted by the user and its related sentences from the journal through a relationship search.

[0161] Figure 17 is a flowchart illustrating a method for providing a personal knowledge graph-based journal according to one embodiment of the present disclosure. Descriptions of aspects identical or similar to those described above may be omitted.

[0162] Referring to FIG. 17, the user device (100) can create a journal using a personal knowledge graph based on interrelated user data. (e.g., S1710) The user device (100) can create a journal using a personal knowledge graph based on interrelated user data through a journal creation model.

[0163] The user device (100) can display a first image and the text of the journal based on the first user data used to create the journal in a first area of ​​the display screen. (Example: S1720) The user device (100) can obtain the text of the journal corresponding to the first user data obtained from the personal knowledge graph and the first image based on the first user data, and output them in a first area of ​​the display screen.

[0164] The user device (100) can display a second image based on second user data that can be used to change the text of the journal in a second area of ​​the display screen (e.g., S1730).

[0165] The user device (100) may obtain second user data from the personal knowledge graph. The second user data may be user data that is related to the first user data but has not been used to create a journal. The second user data may be user data related to text that can be added to the journal. The second user data may be candidate user data that can be used to add or modify text in the journal. The second user data may be user data remaining from the personal knowledge graph, excluding the first user data. The second user data may include multiple user data. The second image may include multiple image elements corresponding to the multiple user data. The second image may include image elements such as representative images, photos, and graphics, such as thumbnails or preview images.

[0166] For example, the second user data may be at least one activity piece obtained from a personal knowledge graph based on interrelated activity pieces, and may be candidate activity pieces that can be used to modify the text of the journal. The second image may be represented by image elements, such as images, photographs, or graphics, based on at least one candidate activity piece. Alternatively, the second image may be represented by image elements corresponding to action information or activity information based on the second user data, and may be an image, photograph, or graphic based on action information or activity information.

[0167] The user device (100) can obtain a user input for modifying the first image using the second image (e.g., S1740).

[0168] The user's input for modifying the first image may be to modify the first image using any one of a plurality of image elements constituting the second image. For example, the user's input for modifying the first image may be to change at least one of the type, position, state, and number of image elements constituting the first image using any one of a plurality of image elements constituting the second image.

[0169] When the first image includes a knowledge graph based on action information and activity information, the user's input for modifying the first image may be to modify the knowledge graph of the first image by using any one of the plurality of image elements constituting the second image. For example, the user's input for modifying the first image may be to modify the knowledge graph by changing at least one of the type, position, status, and number of nodes of the knowledge graph of the first image by using any one of the plurality of image elements constituting the second image.

[0170] The user device (100) can change the text of the journal based on the modification of the first image. (Example: S1750) The user device (100) can change the text of the journal based on a change in at least one of the type, position, status, and number of image elements constituting the first image. The user device (100) can change the text of the journal based on a change in at least one of the type, position, status, and number of nodes of the knowledge graph of the first image. The user device (100) can obtain the text of the changed journal using a journal generation model that inputs first user data that has been changed according to the modification of the first image.

[0171] FIG. 18 is a drawing showing a second image for journal editing displayed in a second area of ​​a display screen of a user device (100) according to one embodiment of the present disclosure.

[0172] Referring to FIG. 18, a journal can be displayed on a first area of ​​a display screen. The first area can correspond to one area of ​​a display screen divided into a predetermined number of areas. The first area of ​​the display screen is a journal area for providing a journal. The first area can be composed of a text display area and a first image display area. The text display area can display the contents of the journal as text along with the title of the journal at the top of the first area. The first image display area can display a first image based on first user data used to create the journal. The first user data can be composed of at least one piece of user data, and the first image can be composed of at least one image element. Each image element can be an image, a photograph, a graphic, etc.

[0173] The user device (100) can switch from the journal provision screen to the journal editing screen based on user input. For example, as illustrated in FIG. 18, the user can request the journal editing screen via a menu located in the upper right corner of the first area. If the user requests the journal editing screen, the user device (100) can provide the journal editing screen.

[0174] The journal editing screen may be provided by dividing the display screen into a first area and a second area. As illustrated in FIG. 18, the user device (100) may designate the remaining area of ​​the divided display screen, other than the first area, as the second area, but this is not a limitation. The first area and the second area may be positioned in a vertical or horizontal relationship, depending on how the display screen is divided.

[0175] A second area of ​​the display screen can display a second image based on second user data that can be used to modify the text of the journal. The second area of ​​the display screen can be a portion of the entire display screen other than the first area. The second area of ​​the display screen can be a region of a predetermined size or a region of a variable size that can vary depending on the size of the first area. The second area can include a second image display area. The second image display area can output a second image based on second user data that can be used to modify the text of the journal. The second user data can include at least one piece of user data, and the second image can be composed of at least one image element. As illustrated in FIG. 18, the second image can include multiple image elements, each of which can be an image, a photograph, a graphic, etc. Each image element can be displayed in a different size depending on the degree to which the user data corresponding to the image element is related to the content of the journal. An image element that is more likely to be used in editing the journal can be displayed larger.

[0176] FIG. 19 is a drawing showing editing a journal using a second image in a user device (100) according to one embodiment of the present disclosure.

[0177] Referring to FIG. 19, a first image displayed in a first area may represent at least one image element based on first user data used to generate a journal, and a second image displayed in a second area may include at least one image element based on second user data that may be used to change text of the journal.

[0178] The user device (100) can obtain a user input for moving a second image element among the image elements constituting the second image displayed in the second area to the first area. In response to the user input for moving the second image element among the image elements constituting the second image to the first area, the user device (100) can adjust the positions of the image elements constituting the first image according to the position of the second image element. Referring to FIG. 19, when the second image element identified as 'step count 6,985' is moved from the second area to the first area, it can be seen that the image element corresponding to the photo in the movement direction of the second image element is gradually reduced and displayed.

[0179] The user device (100) may obtain a user input to position the second image element within the first area so that it becomes an image element of the first image. Referring to FIG. 19, the user may drag and drop the second image element to a predetermined location in the first area. As a result, the second image element disappears from the second area, and an image element identified as 'step count 6,985' is positioned together with the existing image elements of the first image within the first area.

[0180] The user device (100) can change the text of the journal based on the modification of the first image according to the movement of the second image element. Since the second image element becomes the image element of the first image, the first image can be modified by changing at least one of the type, position, state, and number of the image elements of the first image. The user device (100) can change the text of the journal based on the modification of the first image according to the position of the dragged and dropped second image element. The user device (100) can obtain the text of the modified journal using a journal generation model that inputs the user data changed according to the modification of the first image. Referring to FIG. 19, it can be seen that the text 'I walked 6985 steps that day' has been added to the contents of the journal in the text display area.

[0181] FIG. 20 is a diagram illustrating editing a journal by selecting a tone to be applied to the journal in a user device (100) according to one embodiment of the present disclosure.

[0182] Referring to FIG. 20, the user device (100) may display a plurality of tone mode buttons for changing the tone of the text in the journal in a second area of ​​the display screen. For example, as illustrated in FIG. 20, tone mode buttons such as "polite," "friendly," and "playful" may be provided in the second area; however, the types of tone mode buttons are not limited thereto, and the locations at which the tone mode buttons are provided are not limited thereto.

[0183] The user device (100) can obtain a user input selecting a first speech mode button among a plurality of speech mode buttons. For example, the user device (100) can obtain a user input selecting a 'playful' speech mode button among a plurality of speech mode buttons.

[0184] The user device (100) can change the text of the journal according to the tone corresponding to the selected first tone mode button. The user device (100) can obtain the text of the journal according to the tone selected by the user through the journal generation model. The journal generation model, which is a large language model, can create a journal based on information converted to text from a personal knowledge graph and the tone mode requested by the user. For example, as illustrated in FIG. 20, the user device (100) can change the existing journal text to the changed journal text and display it by adding information about the tone mode to the large language model according to the 'playful' tone mode button selected by the user.

[0185] FIG. 21 is a drawing showing a second image for journal editing displayed in a second area of ​​a display screen of a user device (100) according to one embodiment of the present disclosure.

[0186] Referring to FIG. 21, a journal may be displayed in a first area of ​​a display screen. The first area may be composed of a text display area and a first image display area. The text display area may display the contents of the journal in text form along with the journal title at the top of the first area. The first image display area may display a first image based on first user data used to create the journal. The first user data may include at least one piece of user data and may be action information or activity information obtained from a personal knowledge graph. The first image may include a knowledge graph based on the action information or activity information. In this case, each node of the knowledge graph may be expressed as an image element, such as an image, photo, or graphic, based on the action information or activity information.

[0187] The user device (100) can switch from the journal provision screen to the journal editing screen based on user input. For example, as illustrated in FIG. 21, the user can request the journal editing screen via a menu located in the upper right corner of the first area. If the user requests the journal editing screen, the user device (100) can provide the journal editing screen.

[0188] As illustrated in FIG. 21, the user device (100) may display a second image based on second user data that can be used to change the text of a journal in a second area of ​​a split display screen, other than the first area. The second area of ​​the display screen may be a portion of the entire display screen, other than the first area. The second area of ​​the display screen may be a area of ​​a predetermined size, or may be a area of ​​a variable size that can change depending on the size of the first area. The second area may include a second image display area. The second image display area may output a second image based on second user data that can be used to change the text of the journal. The second user data may include at least one piece of user data, and may be action information or activity information obtained from a personal knowledge graph. The second image may represent an image element corresponding to action information or activity information based on the second user data that can be used to add or modify the text of the journal. Each image element may be an image element, such as an image, photograph, or graphic, based on action information or activity information.

[0189] FIG. 22 is a drawing showing editing a journal using a second image in a user device (100) according to one embodiment of the present disclosure.

[0190] Referring to FIG. 22, the first image displayed in the first area may include a knowledge graph based on action information and activity information based on first user data used to generate the journal, and the second image may include an image element corresponding to action information or activity information based on second user data that may be used to change the text of the journal.

[0191] The user device (100) may obtain a user input for moving a second image element among the image elements constituting the second image displayed in the second area to the first area so as to become a node of the knowledge graph of the first image. In response to the user input for moving a second image element among the image elements constituting the second image to the first area, the user device (100) may display an edge based on a mutual correlation between the second image element and a node of the knowledge graph of the first image between the second image element and at least one node of the knowledge graph. Referring to FIG. 22, when the user moves the second image element identified as 'Payment details on May 18' from the second area to the first area, a first edge is formed between a node identified as 'one-chae-dang outing' of the knowledge graph of the first image and the second image element, and a second edge is formed between a node identified as 'son's photo' of the knowledge graph of the first image and the second image element. At this time, since the correlation between the user data corresponding to the second image element and the user data corresponding to the node identified as 'one-chae-dang outing' in the knowledge graph of the first image is higher than the correlation between the user data corresponding to the second image element and the user data corresponding to the node identified as 'son's photo' in the knowledge graph of the first image, the first edge may be displayed with a line that is thicker or darker in color than the second edge. Since the correlation between the user data corresponding to the second image element and the user data corresponding to the node identified as 'cultural center schedule' in the knowledge graph of the first image does not satisfy a predetermined condition, an edge may not be formed between the node identified as 'cultural center schedule' in the knowledge graph of the first image and the second image element.

[0192] The user device (100) may obtain a user input to position the second image element within the first region so that it becomes a node of the knowledge graph of the first image. Referring to FIG. 22, the user may drag and drop the second image element to a location where an edge is formed to connect the second image element to a node identified as "One-Chae-Dang Outing" in the knowledge graph of the first image. As a result, the second image element disappears from the second region, and a node identified as "May 18th Payment Details" in the knowledge graph of the first image is created.

[0193] The user device (100) can change the text of the journal based on the modification of the knowledge graph of the first image according to the movement of the second image element. Since the second image element becomes a node of the knowledge graph of the first image, at least one of the type, position, status, and number of nodes of the knowledge graph of the first image can be changed, thereby modifying the knowledge graph of the first image. The user device (100) can change the text of the journal based on the modification of the knowledge graph of the first image according to the position of the dragged and dropped second image element. The user device (100) can obtain the text of the modified journal using a journal generation model that inputs user data changed according to the modification of the knowledge graph of the first image. Referring to FIG. 22, it can be seen that the text 'I ate at a restaurant near Hanchaedang' has been added to the contents of the journal in the text display area.

[0194] Fig. 23 is a block diagram illustrating a user device (100) according to one embodiment of the present disclosure. Fig. 24 is a block diagram for explaining the configuration and operation of a user device (100) according to one embodiment of the present disclosure.

[0195] Referring to FIG. 23, a user device (100) according to one embodiment may include, but is not limited to, a memory (110) and a processor (120), and general-purpose configurations may be further added. For example, as illustrated in FIG. 24, the user device (100) may further include, in addition to the memory (110) and the processor (120), a sensing unit (130), a communication device (140), and an input / output device (150). Hereinafter, each configuration will be described in detail with reference to FIGS. 23 and 24.

[0196] The memory (110) according to one embodiment can store a program for processing and controlling the processor (120), and can store data and information input to or generated from the user device (100). The memory (110) can store instructions, data structures, and program codes that can be read by the processor (120). Operations performed by the processor (120) can be implemented by executing instructions or program codes stored in the memory (110).

[0197] The memory (110) according to one embodiment may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), and may include a non-volatile memory including at least one of a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk, and a volatile memory such as a RAM (Random Access Memory) or an SRAM (Static Random Access Memory).

[0198] According to one embodiment, the memory (110) may store one or more instructions and / or programs that control the user device (100) to train a neural network model or utilize a neural network model.

[0199] According to one embodiment, the processor (120) can control operations or functions so that the user device (100) can perform tasks by executing instructions or programmed software modules stored in the memory (110). The processor (120) can be composed of hardware components that perform arithmetic, logic, and input / output operations and signal processing. The processor (120) can control the overall operations of the user device (100) by executing one or more instructions stored in the memory (110). The processor (120) can control a sensing unit (130) including at least one sensor, a communication device (140), and an input / output device (150) by executing programs stored in the memory (110).

[0200] The processor (120) may include various processing circuits and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits, including at least one processor. “At least one processor” may be individually and / or collectively configured to perform various functions described herein. As used herein, “processor,” “at least one processor,” and “one or more processors” may be configured to perform multiple functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, “at least one processor” may include a combination of processors that perform various functions of the disclosed functions in a distributed manner. “At least one processor” may execute program instructions to achieve or perform various functions.

[0201] The processor (120) according to one embodiment may be configured with at least one of, for example, a Central Processing Unit, a microprocessor, a Graphic Processing Unit, Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), an Application Processor, a Neural Processing Unit, or an artificial intelligence processor designed with a hardware structure specialized for processing artificial intelligence models, but is not limited thereto. Each processor constituting the processor (120) may be a dedicated processor for performing a predetermined function.

[0202] An artificial intelligence (AI) processor according to one embodiment may perform calculations and control to process tasks set for a user device (100) to perform using an artificial intelligence (AI) model. The AI ​​processor may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as part of a general-purpose processor (e.g., a CPU or application processor) or a graphics-only processor (e.g., a GPU) and mounted on the user device (100).

[0203] The sensing unit (130) according to one embodiment may include a plurality of sensors configured to detect information about the surrounding environment of the user device (100). For example, the sensing unit (130) may include, but is not limited to, a camera (131), a temperature / humidity sensor (132), an infrared sensor (133), a barometric pressure sensor (134), a position sensor (135), a gyroscope sensor (136), etc. The function of each sensor can be intuitively inferred from its name by those skilled in the art, and thus is briefly described below.

[0204] The camera (131) according to one embodiment may include a stereo camera, a mono camera, a wide-angle camera, an around-view camera, or a 3D vision sensor. The lidar sensor (132) may detect the distance to an object and various physical properties by shining a laser on the target. The temperature / humidity sensor (132) may measure the temperature or humidity of the location where the user device (100) is located. The infrared sensor (133) may be either an active infrared sensor that detects changes by emitting infrared rays and blocking the light, or a passive infrared sensor that does not have a light emitter and only detects changes in infrared rays received from the outside. The barometric pressure sensor (134) may measure the barometric pressure of the location where the user device (100) is located. The location sensor (135) may detect the location of the user device (100). For example, the location sensor (135) may be a global positioning system (GPS). The gyroscope sensor (136) may detect angular velocity. The gyro sensor (136) can be used to measure the position of the user device (100) and set the moving direction of the user device (100).

[0205] The communication device (140) may include one or more components that enable the user device (100) to communicate with an external device, such as a server or other electronic device. For example, the communication device (140) may include, but is not limited to, a short-range wireless communicator (141), a mobile communication device (143), etc.

[0206] The short-range wireless communicator (141) may include, but is not limited to, a Bluetooth communication device, a BLE (Bluetooth Low Energy) communication device, a near field communicator, a WLAN (Wi-Fi) communication device, a Zigbee communication device, an Ant+ communication device, a WFD (Wi-Fi Direct) communication device, an UWB (ultra wideband) communication device, an infrared (IrDA, infrared Data Association) communication device, a microwave (uWave) communication device, etc.

[0207] A mobile communication device (143) transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signals may include various types of data, such as voice call signals, video call signals, or text / multimedia message transmission and reception.

[0208] The input / output device (150) may include an input device (151) and an output device (153). The input / output device (150) may be a separate input device (151) and an output device (153), or may be an integrated unit, such as a touchscreen. The input / output device (150) may receive input information from a user and provide output information to the user.

[0209] The input device (151) may refer to a means for obtaining a user's input for controlling the user device (100). For example, the input device (151) may be a key pad, a touch panel (contact electrostatic capacitance type, pressure resistive film type, infrared detection type, surface ultrasonic conduction type, integral tension measurement type, piezoelectric effect type, etc.), a microphone, etc. In addition, the input device (151) may include, but is not limited to, a gaze tracking sensor, a jog wheel, a jog switch, etc. The user's input may be in the form of text, voice, or gesture, but is not limited thereto. The user may input a search command in the form of a search word or sentence through the user interface or microphone of the user device (100). The user may provide a user's input to the user device (100) using a predefined gesture or action (for example, shaking the user device (100).

[0210] The output device (153) can output an audio signal, a video signal, or a vibration signal, and the output device (153) can include a display unit, an audio output device, and a vibration motor. The display unit can display information processed in the user device (100). For example, the display unit can display a user interface for receiving a user's operation. When the display unit and the touchpad are configured as a touch screen in a layered structure, the display unit can be used as an input device in addition to the output device. The display unit can include at least one of a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, and a 3D display. Depending on the implementation form of the user device (100), the user device (100) can include two or more display units. The audio output device can output audio data stored in the memory (110). The audio output device can output an audio signal related to a function performed in the user device (100). Audio output devices may include speakers, buzzers, etc.

[0211] According to one embodiment of the present disclosure, a user device (100) includes a memory (110) including one or more storage media for storing instructions, and at least one processor (120) including a processing circuit. The at least one processor (120) may load and execute instructions or code for a given module that executes one or more instructions to generate a personal knowledge graph based on interrelated user data.

[0212] According to one embodiment, at least one processor (120) of a user device (100) may execute one or more instructions to obtain a first data group clustered from user data related to a first application. The at least one processor (120) may cluster the user data related to the first application based on information about the time of generation to obtain the first data group. The at least one processor (120) may also cluster the user data related to the first application based on information about a specific location or information about a specific user to obtain the first data group. The clustering criteria may vary for each application.

[0213] According to one embodiment, at least one processor (120) may execute one or more instructions to determine a first generation period corresponding to the first data group based on the generation time of the user data included in the first data group. For example, at least one processor (120) may execute one or more instructions to determine a period from the generation time of the earliest generated user data among the user data belonging to the first data group to the generation time of the latest generated user data. As another example, at least one processor (120) may execute one or more instructions to determine a period corresponding to a predetermined time before and after the generation time of the user data belonging to the first data group.

[0214] According to one embodiment, at least one processor (120) may execute one or more instructions to obtain at least one second data group corresponding to a first generation period, clustered from user data related to at least one second application different from a first application, and corresponding to a first data group. At least one processor (120) may extract user data corresponding to a first generation period, corresponding to the first data group, from among user data related to at least one second application. At least one processor (120) may cluster the extracted user data based on information about the generation time to obtain at least one second data group.

[0215] According to one embodiment, at least one processor (120) may execute one or more instructions to determine whether a second generation period corresponding to at least one second data group is outside a first generation period corresponding to a first data group. If, as a result of the determination, the at least one processor (120) determines that the second generation period corresponding to at least one second data group is outside a first generation period corresponding to the first data group, the at least one processor (120) may extend the first generation period based on the second generation period, and repeat the process of acquiring at least one second data group. If, as a result of the determination, the at least one processor (120) determines that the second generation period corresponding to at least one second data group does not exceed the first generation period corresponding to the first data group, the at least one processor (120) may terminate the process of acquiring at least one second data group.

[0216] According to one embodiment, at least one processor (120) may execute one or more instructions to obtain a personal knowledge graph based on user data having mutual correlations from a first data group and at least one second data group. The at least one processor (120) may generate a personal knowledge graph based on the first data group and at least one second data group. The at least one processor (120) may determine the mutual correlations of each node in the generated personal knowledge graph. The at least one processor (120) may determine the mutual correlations of each node based on at least one of the presence or absence of a node connected to a first node based on user data of the first data group, the number of nodes connected to the first node, the presence or absence of a node connected to a second node connected to the first node, and the number of nodes connected to a second node connected to the first node. The at least one processor (120) may remove nodes that are not mutually correlated from the personal knowledge graph according to a predetermined criterion.

[0217] According to one embodiment, at least one processor (120) may execute one or more instructions to determine whether a personal knowledge graph based on correlated user data has uniqueness according to a predetermined criterion. The at least one processor (120) may generate sentences to be input into a large language model or a machine learning-based model that determines the uniqueness of the personal knowledge graph from user data of each node reflected in the personal knowledge graph. The user device (100) may input the generated sentences into the large language model or the machine learning-based model that determines uniqueness to determine whether the personal knowledge graph has uniqueness. As a result of the determination, the at least one processor (120) may store the personal knowledge graph based on correlated user data if the personal knowledge graph has uniqueness, and may not store the personal knowledge graph based on correlated user data if the personal knowledge graph does not have uniqueness.

[0218] According to one embodiment of the present disclosure, at least one processor (120) may execute one or more instructions to store a personal knowledge graph based on interrelated user data in a predetermined space within a user device (100) to build a personalized database. At least one processor (120) may execute one or more instructions to obtain a personal knowledge graph or user data of the personal knowledge graph from the built personalized database, thereby providing a personal knowledge graph-based service.

[0219] According to one embodiment of the present disclosure, a user device (100) includes a memory (110) including one or more storage media for storing instructions, at least one processor (120) including a processing circuit, and an input / output device (130) for outputting a display screen. At least one processor (120) may execute one or more instructions to load and execute commands or codes for a module that provides a journal based on a personal knowledge graph.

[0220] According to one embodiment, at least one processor (120) of a user device (100) may execute one or more instructions to generate a journal using a personal knowledge graph based on interrelated user data. The processor (120) may execute one or more instructions to obtain the text of a journal in sentence form through a journal generation model that utilizes the personal knowledge graph based on the interrelated user data. The processor (120) may execute one or more instructions to obtain the text of the journal corresponding to first user data obtained from the personal knowledge graph and a first image based on the first user data.

[0221] According to one embodiment, at least one processor (120) of a user device (100) may execute one or more instructions to display a first image and text of the journal based on first user data used to generate a journal in a first area of ​​a display screen. The first user data may include a plurality of user data, and the first image may include a plurality of image elements corresponding to the plurality of user data. The first image may include image elements such as a representative image, a photo, or a graphic, such as a thumbnail or a preview image. For example, the first user data may be at least one piece of activity information obtained from a personal knowledge graph based on interrelated activity information. The first image may be represented by image elements such as a representative image, a photo, or a graphic based on at least one piece of activity information. Alternatively, the first image may represent a knowledge graph having at least one piece of activity information as a node. Each node of the knowledge graph may be represented by an image element such as a representative image, a photo, or a graphic based on the activity information.

[0222] According to one embodiment, at least one processor (120) of a user device (100) may execute one or more instructions to obtain a user input for modifying a first image displayed in a first area. The user input may be in the form of a user's touch input, voice, or gesture, but is not limited thereto. The user may input a command for modifying the first image through a user interface or microphone provided in an input / output device (150) of the user device (100). The user may provide the user input to the user device (100) using a predefined gesture or action.

[0223] At least one processor (120) may obtain a user input for modifying any one of a plurality of image elements constituting a first image by executing one or more instructions. For example, at least one processor (120) may obtain a user input for modifying the first image by changing at least one of the type, position, state, and number of image elements constituting the first image. For example, if the first image includes a knowledge graph based on activity information consisting of action information corresponding to a user's action and a series of continuous action information, the user's input for modifying the first image may be to modify the knowledge graph by changing at least one of the type, position, state, and number of nodes of the knowledge graph.

[0224] According to one embodiment, at least one processor (120) of a user device (100) may execute one or more instructions to change the text of a journal based on modification of a first image displayed in a first area of ​​a display screen. The processor (120) may execute one or more instructions to change the text of the journal based on modification of at least one of the type, position, state, and number of image elements constituting the first image. The at least one processor (120) may execute one or more instructions to change first user data used to generate a journal in a personal knowledge graph based on modification of the first image, and further, may also change an existing personal knowledge graph. The at least one processor (120) may execute one or more instructions to obtain the text of the changed journal using a journal generation model that inputs the first user data changed based on modification of the first image.

[0225] At least one processor (120) of a user device (100) according to one embodiment may execute one or more instructions to obtain a user's input for selecting a first image element from among image elements constituting a first image, and change the text of the journal so that text corresponding to the first image element is identified from among the texts of the journal. At least one processor (120) according to one embodiment may execute one or more instructions to obtain a user's input for deleting a first image element from among the image elements constituting the first image, and change the text of the journal so that text corresponding to the first image element is deleted from among the texts of the journal.

[0226] When the first image represents a knowledge graph based on action information and activity information, at least one processor (120) according to one embodiment may execute one or more instructions to obtain a user input for moving a position of a first node of the knowledge graph within a first area, and change a text of a journal based on modification of the knowledge graph according to the changed position of the first node. At least one processor (120) according to one embodiment may execute one or more instructions to obtain a user input for modifying user data related to a first node of the knowledge graph, and change a text of the journal based on modification of the knowledge graph according to the modified user data.

[0227] According to one embodiment of the present disclosure, at least one processor (120) of a user device (100) may execute one or more instructions to display a second image based on second user data that can be used to change the text of a journal in a second area of ​​a display screen. The at least one processor (120) may obtain the second user data from a personal knowledge graph. The second user data may be user data that is related to the first user data but has not been used to create the journal. The second user data may be user data related to text that can be added to the journal. The second user data may be candidate user data that can be used to add or modify the text of the journal. The second user data may be user data excluding the first user data extracted from the personal knowledge graph. The second user data may include a plurality of user data. For example, the second user data may be at least one activity information obtained from a personal knowledge graph based on activity information that is interrelated, and may be candidate activity information that can be used to change the text of the journal. The second image may include a plurality of image elements corresponding to a plurality of user data. The second image may include image elements such as representative images, photos, and graphics, such as thumbnails or preview images. The second image may be expressed as image elements such as images, photos, and graphics based on at least one piece of candidate activity information. Alternatively, the second image may be expressed as image elements corresponding to action information or activity information based on the second user data, and may be images, photos, graphics, etc. based on the action information or activity information.

[0228] According to one embodiment, at least one processor (120) of a user device (100) may obtain a user input for modifying a first image using a second image by executing one or more instructions. The at least one processor (120) may obtain a user input for modifying the first image using any one of a plurality of image elements constituting the second image. The at least one processor (120) may obtain a user input for moving a second image element among the image elements constituting the second image to a first area. For example, the at least one processor (120) may obtain a user input for modifying the first image by changing at least one of the type, position, state, and number of image elements constituting the first image using any one of a plurality of image elements constituting the second image. The at least one processor (120) may change the text of a journal based on the modification of the first image according to the movement of the second image element by executing one or more instructions. At least one processor (120) can change the text of the journal according to a change in at least one of the type, position, state, and number of image elements constituting the first image.

[0229] According to one embodiment, when a first image includes a knowledge graph based on action information and activity information used to create a journal, and a second image includes an image element corresponding to action information or activity information based on second user data that can be used to change the text of the journal, at least one processor (120) of the user device (100) may execute one or more instructions to obtain a user input for modifying the knowledge graph of the first image using any one of a plurality of image elements constituting the second image. For example, the at least one processor (120) may obtain a user input for moving a second image element among the image elements constituting the second image to a first area. In response to the user input for moving the second image element to the first area, the at least one processor (120) may display an edge based on a mutual correlation between the second image element and a node of the knowledge graph of the first image between the second image element and at least one node of the knowledge graph. At least one processor (120) can obtain a user input for positioning a second image element within a first region to become a node of a knowledge graph of the first image. At least one processor (120) can change the text of the journal based on modification of the knowledge graph of the first image due to movement of the second image element. At least one processor (120) can change the text of the journal based on a change in at least one of the type, position, status, and number of nodes of the knowledge graph of the first image. At least one processor (120) can obtain the changed text of the journal using a journal generation model that inputs first user data changed according to the modification of the first image.

[0230] Meanwhile, embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and include both volatile and nonvolatile media, removable and non-removable media. Furthermore, computer-readable media may include computer storage media and communication media. Computer storage media include both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media may typically include computer-readable instructions, data structures, or other data in a modulated data signal, such as program modules.

[0231] Additionally, a computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0232] According to one embodiment, a method according to one embodiment of the present disclosure may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0233] According to one embodiment of the present disclosure, a computer-readable recording medium is provided, which records a program for executing a method for generating a personal knowledge graph in the user device, a method for providing a service using a personal knowledge graph, or a method for providing a journal based on a personal knowledge graph.

[0234] According to one embodiment of the present disclosure, a method for generating a personal knowledge graph is provided. The method for generating a personal knowledge graph may include a step (e.g., S410) of obtaining a first data group clustered from user data associated with a first application. Furthermore, the method for generating a personal knowledge graph may include a step (e.g., S420) of identifying a first generation period corresponding to the first data group based on a generation time of user data included in the first data group. Furthermore, the method for generating a personal knowledge graph may include a step (e.g., S430) of obtaining at least one second data group corresponding to the first generation period, clustered from user data associated with at least one second application different from the first application. Furthermore, the method for generating a personal knowledge graph may include a step (e.g., S440) of obtaining a personal knowledge graph based on user data that is interrelated from the first data group and at least one second data group.

[0235] Additionally, according to one embodiment of the present disclosure, the step of obtaining the first data group (e.g., S410) may obtain the first data group by clustering user data related to the first application based on information about the time of generation.

[0236] Additionally, according to one embodiment of the present disclosure, the step of confirming the first generation period corresponding to the first data group (e.g., S420) may confirm the period from the generation time of the earliest generated user data among the user data belonging to the first data group to the generation time of the latest generated user data.

[0237] Additionally, according to one embodiment of the present disclosure, the step of confirming the first generation period corresponding to the first data group (e.g., S420) may confirm a period corresponding to a predetermined time before and after the generation time of the user data belonging to the first data group.

[0238] Additionally, according to one embodiment of the present disclosure, the step of obtaining at least one second data group (e.g., S430) may include a step of extracting user data corresponding to a first generation period from among user data related to at least one second application (e.g., S610). Additionally, the step of obtaining at least one second data group (e.g., S430) may include a step of obtaining at least one second data group by clustering the extracted user data based on information about the generation time point.

[0239] In addition, the step of obtaining at least one second data group (e.g., S430) may include a step of determining whether a second generation period corresponding to at least one second data group is outside a first generation period corresponding to the first data group (e.g., S630). The step of obtaining at least one second data group (e.g., S430) may include a step of repeating the process of obtaining at least one second data group by extending the first generation period based on the second generation period if the second generation period is outside the first generation period as a result of determining whether it is outside the first generation period, and if the second generation period does not exceed the first generation period, a step of terminating the process of obtaining at least one second data group (e.g., S640).

[0240] Additionally, according to one embodiment of the present disclosure, the step of obtaining a personal knowledge graph (e.g., S440) may include a step of generating a personal knowledge graph based on a first data group and at least one second data group (e.g., S810). Furthermore, the step of obtaining a personal knowledge graph (e.g., S440) may include a step of determining the interrelationship between each node in the generated personal knowledge graph (e.g., S820). Furthermore, the step of obtaining a personal knowledge graph (e.g., S440) may include a step of removing nodes that are not interrelated from the personal knowledge graph according to a predetermined criterion (e.g., S830).

[0241] Additionally, the step of determining the mutual correlation of each node (e.g., S820) may include a step of calculating the mutual correlation based on at least one of the presence or absence of a node connected to the first node based on user data of the first data group, the number of nodes connected to the first node, the presence or absence of a node connected to a second node connected to the first node, and the number of nodes connected to the second node.

[0242] Additionally, according to one embodiment of the present disclosure, the step of acquiring a personal knowledge graph (e.g., S440) may include a step of determining whether the personal knowledge graph has uniqueness according to a predetermined criterion (e.g., S840). Furthermore, the step of acquiring a personal knowledge graph (e.g., S440) may include a step of storing a personal knowledge graph based on correlated user data if the result of determining whether there is uniqueness indicates that there is uniqueness, and a step of not storing a personal knowledge graph based on correlated user data if there is no uniqueness (e.g., S850).

[0243] According to one embodiment of the present disclosure, a non-transitory computer-readable recording medium having recorded thereon a program for executing the method for generating the above-described personal knowledge graph is provided.

[0244] According to one embodiment of the present disclosure, a user device (100) for generating a personal knowledge graph is provided. The user device (100) may include a memory (110) including one or more storage media for storing instructions, and at least one processor (120) including a processing circuit. By executing one or more instructions by the at least one processor (120), the user device (100) may obtain a first data group clustered from user data related to a first application. In addition, by executing one or more instructions by the at least one processor (120), the user device (100) may identify a first generation period corresponding to the first data group based on a generation time of user data included in the first data group. In addition, by executing one or more instructions by at least one processor (120), the user device (100) can obtain at least one second data group corresponding to the first generation period, clustered from user data related to at least one second application different from the first application. In addition, by executing one or more instructions by at least one processor (120), the user device (100) can obtain a personal knowledge graph based on user data that is interrelated from the first data group and the at least one second data group.

[0245] Additionally, according to one embodiment of the present disclosure, one or more instructions may be executed by at least one processor (120), thereby causing the user device (100) to obtain a first data group by clustering user data related to a first application based on information about the time of generation.

[0246] In addition, according to one embodiment of the present disclosure, by executing one or more instructions by at least one processor (120), the user device (100) can check the period from the time of creation of the earliest generated user data among the user data belonging to the first data group to the time of creation of the latest generated user data, thereby checking the first generation period corresponding to the first data group.

[0247] In addition, according to one embodiment of the present disclosure, by executing one or more instructions by at least one processor (120), the user device (100) can check a period corresponding to a predetermined time before and after the time of generation of user data belonging to the first data group, thereby checking a first generation period corresponding to the first data group.

[0248] In addition, according to one embodiment of the present disclosure, by executing one or more instructions by at least one processor (120), the user device (100) can extract user data corresponding to a first generation period corresponding to a first data group among user data related to at least one second application. In addition, by executing one or more instructions by at least one processor (120), the user device (100) can cluster the extracted user data based on information about the generation time to obtain at least one second data group.

[0249] In addition, by executing one or more instructions by at least one processor (120), the user device (100) can determine whether a second generation period corresponding to at least one second data group is outside a first generation period corresponding to a first data group. In addition, by executing one or more instructions by at least one processor (120), the user device (100) can determine whether the second generation period is outside a first generation period, and if the second generation period is outside the first generation period, the process of obtaining at least one second data group can be repeated by extending the first generation period based on the second generation period, and if the second generation period does not exceed the first generation period, the process of obtaining at least one second data group can be terminated.

[0250] In addition, according to one embodiment of the present disclosure, one or more instructions are executed by at least one processor (120), thereby causing the user device (100) to generate a personal knowledge graph based on a first data group and at least one second data group, determine the mutual correlation of each node in the generated personal knowledge graph, and remove nodes that are not mutually correlated from the personal knowledge graph according to a predetermined criterion.

[0251] In addition, by executing one or more instructions by at least one processor (120), the user device (100) can determine the interrelationship of each node in the generated personal knowledge graph based on at least one of the presence or absence of a node connected to a first node based on user data of the first data group, the number of nodes connected to the first node, the presence or absence of a node connected to a second node connected to the first node, and the number of nodes connected to the second node.

[0252] In addition, according to one embodiment of the present disclosure, by executing one or more instructions by at least one processor (120), the user device (100) can determine whether a personal knowledge graph has uniqueness according to a predetermined criterion. In addition, by executing one or more instructions by at least one processor (120), the user device (100) can determine whether there is uniqueness, and if there is uniqueness, the personal knowledge graph based on the user data with mutual correlation can be stored, and if there is no uniqueness, the personal knowledge graph based on the user data with mutual correlation can be not stored.

[0253] Furthermore, according to one embodiment of the present disclosure, by executing one or more instructions by at least one processor (120), the user device (100) can generate sentences from user data of each node reflected in the personal knowledge graph. Furthermore, by executing one or more instructions by at least one processor (120), the user device (100) can input the generated sentences into a large-scale language model or machine learning-based model to determine whether they have specificity according to a predetermined criterion.

[0254] According to one embodiment of the present disclosure, a method for providing a journal based on a personal knowledge graph is provided. The method for providing a journal based on a personal knowledge graph may include a step (e.g., S1110, S1710) of generating a journal using a personal knowledge graph based on interrelated user data. In addition, the method for providing a journal based on a personal knowledge graph may include a step (e.g., S1120, S1720) of displaying a first image and the text of the journal based on the first user data used to generate the journal in a first area of ​​a display screen. In addition, the method for providing a journal based on a personal knowledge graph may include a step (e.g., S1130) of obtaining a user input for modifying the first image displayed in the first area. In addition, the method for providing a journal based on a personal knowledge graph may include a step (e.g., S1140, S1750) of changing the text of the journal based on the modification of the first image.

[0255] Additionally, according to one embodiment of the present disclosure, the step of obtaining a user's input (e.g., S1130) may obtain a user's input for modifying the first image by changing at least one of the type, position, state, and number of image elements constituting the first image.

[0256] Additionally, according to one embodiment of the present disclosure, the step of obtaining user input (e.g., S1130) may obtain user input for selecting a first image element among image elements constituting the first image. Additionally, the step of changing the text of the journal (e.g., S1140) may change the text of the journal so that text corresponding to the first image element among the text of the journal is identified.

[0257] Additionally, according to one embodiment of the present disclosure, the step of obtaining user input (e.g., S1130) may obtain user input for deleting a first image element among the image elements constituting the first image. Additionally, the step of changing the text of the journal (e.g., S1140) may change the text of the journal such that the text corresponding to the first image element among the text of the journal is deleted.

[0258] Additionally, according to one embodiment of the present disclosure, if the first image represents a knowledge graph based on activity information consisting of action information corresponding to the user's action and a series of continuous action information, the step of obtaining the user's input (e.g., S1130) may obtain the user's input of moving the position of the first node of the knowledge graph within the first area. Furthermore, the step of changing the text of the journal (e.g., S1140) may change the text of the journal based on modification of the knowledge graph according to the changed position of the first node.

[0259] Furthermore, according to one embodiment of the present disclosure, if the first image includes a knowledge graph based on activity information consisting of action information corresponding to the user's motion and a series of continuous action information, the step of obtaining user input (e.g., S1130) may obtain user input for modifying user data related to a first node of the knowledge graph. Furthermore, the step of changing the text of the journal (e.g., S1140) may change the text of the journal based on modification of the knowledge graph according to the modified user data.

[0260] Additionally, according to one embodiment of the present disclosure, a method for providing a journal based on a personal knowledge graph may further include a step (e.g., S1730) of displaying a second image based on second user data that can be used to modify the text of the journal in a second area of ​​the display screen. The step (e.g., S1130) of obtaining user input may obtain user input for modifying the first image using the second image (e.g., S1740).

[0261] Additionally, according to one embodiment of the present disclosure, the step of obtaining a user input (e.g., S1740) may obtain a user input for moving a second image element among the image elements constituting the second image to the first area. Additionally, the step of changing the text of the journal (e.g., S1750) may change the text of the journal based on the modification of the first image according to the movement of the second image element.

[0262] In addition, according to one embodiment of the present disclosure, if the first image includes a knowledge graph based on activity information composed of action information corresponding to a user's action and a series of continuous action information, and the second image includes an image element corresponding to the action information or activity information based on second user data, the step of obtaining a user's input (e.g., S1740) may include a step of displaying an edge based on a mutual correlation between the second image element and a node of the knowledge graph between the second image element and at least one node of the knowledge graph in response to a user's input of moving a second image element among the image elements constituting the second image to a first area. In addition, the step of obtaining a user's input (e.g., S1740) may include a step of obtaining a user's input of positioning the second image element within the first area so as to become a node of the knowledge graph. In addition, the step of changing the text of the journal (e.g., S1750) may change the text of the journal based on modification of the knowledge graph according to the movement of the second image element.

[0263] In addition, according to one embodiment of the present disclosure, a computer-readable recording medium having recorded thereon a program for executing a method for providing a journal based on the above personal knowledge graph is provided.

[0264] According to one embodiment of the present disclosure, a user device (100) providing a journal based on a personal knowledge graph is provided. The user device (100) may include a memory (110) including one or more storage media for storing instructions, at least one processor (120) including a processing circuit, and an input / output device (130) for outputting a display screen. By executing one or more instructions by the at least one processor (120), the user device (100) may generate a journal using a personal knowledge graph based on user data that is interrelated. In addition, by executing one or more instructions by the at least one processor (120), the user device (100) may display a first image based on first user data used to generate the journal and text of the journal in a first area of ​​the display screen. Additionally, by executing one or more instructions by at least one processor (120), the user device (100) can obtain a user input for modifying a first image displayed in a first area. Additionally, by executing one or more instructions by at least one processor (120), the user device (100) can change the text of the journal based on the modification of the first image.

[0265] Additionally, according to one embodiment of the present disclosure, one or more instructions may be executed by at least one processor (120), thereby causing the user device (100) to obtain a user input for modifying the first image by changing at least one of the type, position, state, and number of image elements constituting the first image.

[0266] In addition, according to one embodiment of the present disclosure, by executing one or more instructions by at least one processor (120), the user device (100) can obtain a user input for selecting a first image element among image elements constituting the first image. In addition, by executing one or more instructions by at least one processor (120), the user device (100) can change the text of the journal so that text corresponding to the first image element among the text of the journal is identified.

[0267] In addition, according to one embodiment of the present disclosure, by executing one or more instructions by at least one processor (120), the user device (100) can obtain a user input for deleting a first image element among image elements constituting the first image. In addition, by executing one or more instructions by at least one processor (120), the user device (100) can change the text of the journal such that text corresponding to the first image element among the text of the journal is deleted.

[0268] In addition, according to one embodiment of the present disclosure, when the first image includes a knowledge graph based on activity information consisting of action information corresponding to a user's action and a series of continuous action information, one or more instructions may be executed by at least one processor (120) to cause the user device (100) to obtain a user input for moving the position of a first node of the knowledge graph within a first area. In addition, one or more instructions may be executed by at least one processor (120) to cause the user device (100) to change the text of the journal based on modification of the knowledge graph according to the changed position of the first node.

[0269] In addition, according to one embodiment of the present disclosure, when the first image includes a knowledge graph based on activity information consisting of action information corresponding to a user's action and a series of continuous action information, one or more instructions may be executed by at least one processor (120) to enable the user device (100) to obtain a user's input for modifying user data related to a first node of the knowledge graph. In addition, one or more instructions may be executed by at least one processor (120) to enable the user device (100) to change the text of the journal based on modification of the knowledge graph according to the modified user data.

[0270] Additionally, according to one embodiment of the present disclosure, by executing one or more instructions by at least one processor (120), the user device (100) can display a second image based on second user data that can be used to change the text of the journal in a second area of ​​the display screen. Additionally, by executing one or more instructions by at least one processor (120), the user device (100) can obtain a user input for modifying a first image using the second image.

[0271] In addition, according to one embodiment of the present disclosure, by executing one or more instructions by at least one processor (120), the user device (100) can obtain a user input for moving a second image element among image elements constituting a second image to a first area. In addition, by executing one or more instructions by at least one processor (120), the user device (100) can change the text of the journal based on the modification of the first image according to the movement of the second image element.

[0272] In addition, according to one embodiment of the present disclosure, when a first image includes a knowledge graph based on activity information composed of action information corresponding to a user's action and a series of continuous action information, and a second image includes an image element corresponding to the action information or activity information based on second user data, one or more instructions may be executed by at least one processor (120) to cause the user device (100) to, in response to a user input of moving a second image element among the image elements constituting the second image to a first area, display an edge based on a mutual correlation between the second image element and a node of the knowledge graph between the second image element and at least one node of the knowledge graph, and obtain a user input of positioning the second image element within the first area to become a node of the knowledge graph. In addition, one or more instructions may be executed by at least one processor (120) to cause the user device (100) to change the text of the journal based on modification of the knowledge graph according to the movement of the second image element.

[0273] In addition, according to one embodiment of the present disclosure, by executing one or more instructions by at least one processor (120), the user device (100) can display a plurality of tone mode buttons for changing the tone of text in a journal in a second area of ​​a display screen, and obtain a user input for selecting a first tone mode button. In addition, by executing one or more instructions by at least one processor (120), the user device (100) can change the text in the journal according to the tone corresponding to the selected first tone mode button.

[0274] At least one of the devices, units, components, modules, units, etc. represented by blocks or equivalent symbols in the above embodiments may be physically implemented by including analog and / or digital circuits (e.g., logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, etc.). Furthermore, these components may be implemented or driven by software and / or firmware to perform the functions or operations described herein.

[0275] Each embodiment provided in the above description does not exclude that it may be associated with one or more features of other examples or embodiments provided herein, and may also be associated with features of other examples or embodiments not provided herein but consistent with the present disclosure.

[0276] The embodiments of the present disclosure disclosed in this specification and drawings are merely intended to provide specific examples to facilitate easy explanation of the technical content of the present disclosure and to aid understanding of the embodiments, and are not intended to limit the scope of the embodiments of the present disclosure. Therefore, the scope of the various embodiments of the present disclosure should be interpreted to include not only the embodiments disclosed in this specification, but also all modifications or variations derived based on the technical concept of the present disclosure.

Claims

1. A step of creating a journal using a personal knowledge graph based on interrelated user data (S1110, S1710); A step of displaying a first image generated based on the first user data used to generate the journal and the text of the journal in a first area of ​​the display screen (S1120, S1720); A step (S1130) of obtaining a user's input for modifying the first image displayed in the first area; and A step of changing the text of the journal based on the modification of the first image (S1140, S1750); A method for providing a journal based on a personal knowledge graph, including:

2. In paragraph 1, A method for providing a journal based on a personal knowledge graph, further comprising a step of modifying the first image based on the user's input by changing at least one of the type, position, state, and number of image elements constituting the first image.

3. In paragraph 1 or 2, The step of obtaining the user's input (S1130) is as follows: Obtaining the user's input for selecting a first image element among the image elements constituting the first image, The step of changing the text of the above journal (S1140) is: A method for providing a journal based on a personal knowledge graph, wherein the text of the journal is changed so that text corresponding to the first image element can be identified among the texts of the journal.

4. In any one of paragraphs 1 to 3, The step of obtaining the user's input (S1130) is as follows: Obtaining the user's input for deleting the first image element among the image elements constituting the first image, The step of changing the text of the above journal (S1140) is: A method for providing a journal based on a personal knowledge graph, wherein the text of the journal is changed so that the text corresponding to the first image element is deleted from the text of the journal.

5. In any one of paragraphs 1 to 4, Based on the first image including a knowledge graph based on activity information consisting of action information corresponding to the user's actions and a series of continuous action information, The step of obtaining the user's input (S1130) is as follows: Obtaining the user's input for moving the position of the first node of the knowledge graph within the first area, The step of changing the text of the above journal (S1140) is: A method for providing a journal based on a personal knowledge graph, wherein the text of the journal is changed based on modification of the knowledge graph according to the moved position of the first node.

6. In any one of paragraphs 1 to 5, Based on the first image including a knowledge graph based on activity information consisting of action information corresponding to the user's actions and a series of continuous action information, The step of obtaining the user's input (S1130) is as follows: Obtaining user input for modifying user data related to the first node of the above knowledge graph, The step of changing the text of the above journal (S1140) is: A method for providing a journal based on a personal knowledge graph, wherein the text of the journal is changed based on modification of the knowledge graph according to the modified user data.

7. In any one of paragraphs 1 to 6, Further comprising a step (S1730) of displaying a second image based on second user data configured to change the text of the journal in a second area of ​​the display screen; The step of obtaining the user's input (S1130) is as follows: A method for providing a journal based on a personal knowledge graph, wherein the method obtains the user's input for modifying the first image using the second image (S1740).

8. In any one of paragraphs 1 to 7, The step of obtaining the user's input (S1740) is as follows: Obtaining the user's input for moving a second image element among the image elements constituting the second image to the first area, The step of changing the text of the above journal (S1750) is: A method for providing a journal based on a personal knowledge graph, wherein the text of the journal is changed based on modification of the first image according to movement of the second image element.

9. In any one of paragraphs 1 to 8, Based on the first image including a knowledge graph based on activity information consisting of action information corresponding to a user's action and a series of continuous action information, and the second image including an image element corresponding to the action information or activity information based on the second user data, The step of obtaining the user's input (S1740) is as follows: A step of displaying an edge based on a mutual correlation between the second image element and a node of the knowledge graph between the second image element and at least one node of the knowledge graph based on the user's input of moving a second image element among the image elements constituting the second image to the first area; and A step of obtaining a user input for positioning the second image element within the first region so that the second image element becomes a node of the knowledge graph, The step of changing the text of the above journal (S1750) is: A method for providing a journal based on a personal knowledge graph, wherein the text of the journal is changed based on modification of the knowledge graph according to movement of the second image element.

10. A computer-readable recording medium storing instructions that, when executed by at least one processor, cause a device to perform the method of any one of claims 1 to 9.

11. A memory (110) including one or more storage media for storing instructions; At least one processor (120) comprising a processing circuit; and Includes an input / output device (130) that outputs a display screen, By executing the instructions by at least one processor (120), the user device (100) may: A user device (100) that creates a journal using a personal knowledge graph based on user data that is interrelated, displays a first image created based on first user data used to create the journal and text of the journal in a first area of ​​the display screen, obtains a user's input for modifying the first image displayed in the first area, and changes the text of the journal based on modification of the first image.

12. In paragraph 11, By executing the instructions by at least one processor (120), the user device (100) may: A user device (100) that obtains a user's input by selecting a first image element among image elements constituting the first image, and changes the text of the journal so that text corresponding to the first image element among the text of the journal can be identified.

13. In paragraph 11 or 12, Based on the first image including a knowledge graph based on activity information consisting of action information corresponding to the user's actions and a series of continuous action information, By executing the instructions by at least one processor (120), the user device (100) may: A user device (100) that obtains a user's input by moving the position of the first node of the knowledge graph within the first area, and changes the text of the journal based on the modification of the knowledge graph according to the moved position of the first node.

14. In any one of paragraphs 11 to 13, By executing the instructions by at least one processor (120), the user device (100) may: A user device (100) that displays a second image based on second user data configured to change the text of the journal in a second area of ​​the display screen and obtains input from the user for modifying the first image using the second image.

15. In any one of paragraphs 11 to 14, By executing the instructions by at least one processor (120), the user device (100) may: A user device (100) that obtains a user's input by moving a second image element among the image elements constituting the second image to the first area, and changes the text of the journal based on the modification of the first image according to the movement of the second image element.

Images