A cross-device cross-modal user memory graph construction method and system

Abstract

This invention provides a method and system for constructing a cross-device, cross-modal user memory graph. The method includes acquiring multimodal data from multiple devices and applications, uniformly parsing it into standardized memory atomic events with timestamps, calculating causal relationships between adjacent events during real-time processing, linking highly correlated event pairs into instantaneous event chains, performing deep encoding of the event chains using an offline dual-stream temporal semantic embedding model, and inferring candidate graph substructures using a graph neural network. When fusing these substructures into the user's main graph, relationship conflicts are resolved through confidence comparison, retaining highly reliable results, identifying stable subgraphs, and using a multi-head attention graph summarization network to abstract the subgraphs into higher-order memory concept nodes, thereby achieving continuous optimization and updating of the graph.

Description

Technical Field

[0001] This application belongs to the field of graph construction, and in particular relates to a method and system for constructing user memory graphs across devices and modalities. Background Technology

[0002] User behavior data in the digital world is scattered across different devices and applications in various forms, including text, images, voice, and clickstreams, forming a multimodal and heterogeneous data environment. User profiling technology focuses on the statistical analysis and classification of static, single-source data, such as analyzing a user's browsing history to infer interest tags. This method can identify user preferences to some extent, but it cannot determine the temporal and causal relationships of user behavior. For example, a user searching for travel guides on their mobile phone and then booking flights on their computer are often treated as isolated data points in traditional technologies. The underlying chain of "planning a trip" intent is broken, resulting in a one-sided and delayed understanding of the user's state and true intent. Therefore, how to reliably integrate multimodal data across devices and applications and uncover deep temporal relationships reflecting user memory and intent is the primary challenge currently facing technology.

[0003] Methods using knowledge graph technology to build unified user models attempt to organize user behaviors, entities, and relationships into a graph structure to more comprehensively represent user intent. However, most methods rely on offline or batch processing during graph construction, making it difficult to detect and respond instantly to newly generated user behavior flows, leading to lag in user model updates. Existing models, when dealing with conflicts between new and old knowledge, employ overlay strategies or lack reliable confidence assessment mechanisms, making them prone to noise input or loss of important information. Over time, the user memory graph, composed of low-level atomic events, becomes exceptionally large and complex, a phenomenon known as the "graph explosion" problem. The low-level, fine-grained graph structure makes identifying macro-level intent difficult, and the lack of reliable technical means to summarize from complex event subgraphs into higher-order, more generalized memory concepts such as "project planning" and "health management" limits the interpretability of the graph and its application effectiveness in downstream tasks. Summary of the Invention

[0004] To address the problem that existing technologies struggle to detect and respond in real time to newly generated user behavior streams, and lack reliable confidence assessment mechanisms.

[0005] In the first aspect, the present invention proposes a method for constructing a cross-device, cross-modal user memory map, comprising the following steps: Acquire multimodal heterogeneous data from multiple device terminals and applications, and parse the data into standardized memory atomic events carrying timestamps, entities, and action elements; calculate the correlation score of adjacent event pairs for the real-time generated memory atomic event stream, and link the event pairs to construct an instantaneous event chain when the correlation score is higher than a preset threshold; According to a predetermined cycle or batch, the instantaneous event chain is sent to an offline deep inference engine. The engine uses a dual-stream temporal semantic embedding model to extract the semantic features of each entity in the event chain and the temporal context features of the event chain, constructs an initial graph structure containing entity nodes, and then performs relational inference on the initial graph structure through a graph neural network to generate candidate graph substructures. The candidate graph substructures are integrated into the user memory main graph. When a relationship in the candidate graph substructure conflicts with an existing relationship in the user memory main graph, the confidence level of both is calculated based on the evidence source type and the length of the reasoning path, and the relationship with the highest confidence level is retained. The system identifies subgraphs in the updated user memory master graph that meet preset stability and connectivity conditions, and uses a multi-head attention graph summarization network to abstract these subgraphs into higher-order memory concept nodes, thereby updating the user memory master graph.

[0006] Optionally, parsing the data into standardized memory atomic events carrying timestamps, entity, and action elements includes: Extract the URL title text as an entity, "browse" as an action, and the access time as a timestamp from the browser's access log to form a browser atomic event; Extract the destination name as an entity, "navigation" as an action, and the navigation start time as a timestamp from the navigation records of the map application to form a map atomic event.

[0007] Optionally, calculating the association score of adjacent event pairs includes: Calculate the semantic similarity score between entities and actions in adjacent events; Calculate the time factor that decays as the event time interval increases; The association score is obtained by weighting the semantic similarity score with the time factor.

[0008] Optionally, the engine employs a dual-stream temporal semantic embedding model to extract the semantic features of each entity in the event chain and the temporal context features of the event chain, constructing an initial graph structure containing entity nodes, including: The event sequence in the event chain is input into a long short-term memory network to extract temporal context feature vectors that represent global information of the entire event chain. The entity and action text of each event in the event chain are input into the BERT model to extract the deep semantic feature vector of the corresponding event entity; The temporal context feature vector is concatenated with the deep semantic feature vector of each event entity to generate the fused feature representation of the corresponding node in the graph neural network.

[0009] Optionally, the step of performing relational reasoning on the initial graph structure using a graph neural network to generate candidate graph substructures includes: The entities in the event chain are used as input nodes of the graph attention network, and the deep semantic feature vectors extracted from the entities and action text of each event in the dual-stream temporal semantic embedding model are used as the initial features of the corresponding nodes. The feature representation of the node is updated through the graph attention network, and based on the updated node features, the probability score of forming a new type of relationship between entity nodes in the event chain is calculated through a relationship classifier. Entity pairs with probability scores greater than a preset probability threshold are selected, and candidate graph substructures are constructed using the predicted relationship types as edges.

[0010] Optionally, the step of calculating the confidence levels of the evidence source type and the inference path length, and retaining the relationship with the highest confidence level, includes: Set preset weights for different types of evidence sources ; The confidence level Conf(R) of the relationship R is calculated based on the evidence source weights and the confidence factor, which decays with the length of the reasoning path L. The formula is as follows: ; in, The length of the reasoning path. The preset attenuation coefficient; When there is a conflict in a relationship, retain the relationship with the highest confidence level.

[0011] Optionally, the identification of subgraphs in the updated user memory master graph that satisfy preset stability and connectivity conditions includes: The stability condition is set to the existence time of all nodes and edges in the subgraph exceeding a preset duration, and the connectivity condition is set to the node density in the subgraph being greater than a preset density threshold.

[0012] Optionally, the step of using a multi-head attention graph summarization network to abstract the subgraph into higher-order memory concept nodes and updating the user memory master graph includes: The node features and adjacency matrix of the identified subgraph are input into the graph summarization network; The network computes importance weights for each node in the subgraph in different representation subspaces using multiple parallel attention heads; The weighted node features calculated by each attention head are fused to generate a vector representing the summary information of the entire subgraph; Use the vector to create a new higher-order memory concept node, and connect the node to all nodes in the subgraph to update the main graph.

[0013] In another aspect, the present invention also proposes a cross-device, cross-modal user memory graph construction system, comprising the following modules: The module is used to acquire multimodal heterogeneous data from multiple device terminals and applications, and parse the data into standardized memory atomic events carrying timestamps, entities and action elements; for the real-time generated memory atomic event stream, the correlation score of adjacent event pairs is calculated, and when the correlation score is higher than a preset threshold, the event pairs are linked to construct an instantaneous event chain; The generation module is used to send the instantaneous event chain into the offline deep inference engine according to a predetermined period or batch. The engine uses a dual-stream temporal semantic embedding model to extract the semantic features of each entity in the event chain and the temporal context features of the event chain, constructs an initial graph structure containing entity nodes, and then performs relational inference on the initial graph structure through a graph neural network to generate candidate graph substructures. The calculation module is used to integrate the candidate graph substructures into the user memory main graph. When a relationship in the candidate graph substructure conflicts with an existing relationship in the user memory main graph, the confidence level of the two is calculated based on the evidence source type and the inference path length, and the relationship with the highest confidence level is retained. The update module is used to identify subgraphs in the updated user memory master graph that meet preset stability and connectivity conditions, and to use a multi-head attention graph summarization network to abstract the subgraphs into higher-order memory concept nodes, thereby updating the user memory master graph.

[0014] Preferably, parsing the data into standardized memory atomic events carrying timestamps, entities, and action elements includes: Extract the URL title text as an entity, "browse" as an action, and the access time as a timestamp from the browser's access log to form a browser atomic event; Extract the destination name as an entity, "navigation" as an action, and the navigation start time as a timestamp from the navigation records of the map application to form a map atomic event.

[0015] Preferably, the calculation of the correlation score of adjacent event pairs includes: Calculate the semantic similarity score between entities and actions in adjacent events; Calculate the time factor that decays as the event time interval increases; The association score is obtained by weighting the semantic similarity score with the time factor.

[0016] Preferably, the engine employs a dual-stream temporal semantic embedding model to extract the semantic features of each entity in the event chain and the temporal context features of the event chain, constructing an initial graph structure containing entity nodes, including: The event sequence in the event chain is input into a long short-term memory network to extract temporal context feature vectors that represent global information of the entire event chain. The entity and action text of each event in the event chain are input into the BERT model to extract the deep semantic feature vector of the corresponding event entity; The temporal context feature vector is concatenated with the deep semantic feature vector of each event entity to generate the fused feature representation of the corresponding node in the graph neural network.

[0017] Preferably, the step of performing relational reasoning on the initial graph structure using a graph neural network to generate candidate graph substructures includes: The entities in the event chain are used as input nodes of the graph attention network, and the deep semantic feature vectors extracted from the entities and action text of each event in the dual-stream temporal semantic embedding model are used as the initial features of the corresponding nodes. The feature representation of the node is updated through the graph attention network, and based on the updated node features, the probability score of forming a new type of relationship between entity nodes in the event chain is calculated through a relationship classifier. Entity pairs with probability scores greater than a preset probability threshold are selected, and candidate graph substructures are constructed using the predicted relationship types as edges.

[0018] Preferably, the step of calculating the confidence level of the evidence source and the inference path length, and retaining the relationship with the highest confidence level, includes: Set preset weights for different types of evidence sources ; The confidence level Conf(R) of the relationship R is calculated based on the evidence source weights and the confidence factor, which decays with the length of the reasoning path L. The formula is as follows: ; in, The length of the reasoning path. The preset attenuation coefficient; When there is a conflict in a relationship, retain the relationship with the highest confidence level.

[0019] Preferably, the subgraphs in the updated user memory master graph that satisfy preset stability and connectivity conditions include: The stability condition is set to the existence time of all nodes and edges in the subgraph exceeding a preset duration, and the connectivity condition is set to the node density in the subgraph being greater than a preset density threshold.

[0020] Preferably, the step of using a multi-head attention graph summarization network to abstract the subgraph into higher-order memory concept nodes and updating the user memory master graph includes: The node features and adjacency matrix of the identified subgraph are input into the graph summarization network; The network computes importance weights for each node in the subgraph in different representation subspaces using multiple parallel attention heads; The weighted node features calculated by each attention head are fused to generate a vector representing the summary information of the entire subgraph; Use the vector to create a new higher-order memory concept node, and connect the node to all nodes in the subgraph to update the main graph.

[0021] This invention solves the data integration challenge across devices and application scenarios by standardizing multi-source heterogeneous data into unified memory atomic events, enabling coherent recognition of user behavior. By combining a real-time model with an offline deep inference engine, it balances the efficiency of event association construction with the depth of relational reasoning, uncovering temporal and semantic connections. During graph construction, an evidence-based confidence conflict resolution mechanism improves the accuracy and reliability of graph knowledge. Graph summarization networks abstract low-order events into high-order memory concepts, enhancing the structured hierarchy and interpretability of the graph. Attached Figure Description

[0022] Figure 1 A flowchart of the first embodiment; Figure 2 A schematic diagram for constructing a memory atomic event; Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] In the first embodiment, the present invention proposes a method for constructing a cross-device, cross-modal user memory graph, such as... Figure 1 This includes the following steps: S1. Acquire multimodal heterogeneous data from multiple device terminals and applications, and parse the data into standardized memory atomic events carrying timestamps, entities, and action elements; calculate the correlation score of adjacent event pairs for the real-time generated memory atomic event stream, and link the event pairs to construct an instantaneous event chain when the correlation score is higher than a preset threshold. Through agent programs deployed on mobile phones, personal computers, and smart speakers, and by calling system APIs or utilizing accessibility interfaces, and with user authorization, the system collects user call logs, web browsing history, photo metadata, calendar entries, chat logs, and geolocation logs. For unstructured or semi-structured data from different sources, appropriate parsing rules are set to identify and extract key information, and the extracted information is combined into a standardized data structure, namely, a memory of atomic events, such as... Figure 2 ; Causal potential is assessed by calculating the semantic association and temporal proximity between adjacent events. For example, for event A, a user searched for "tomato pasta recipes" and the following event B, "a user opened a fresh food e-commerce application", an association score is calculated based on the semantic relevance and time interval between the two events, such as 0.85. This score is higher than the preset link threshold of 0.7, so events A and B are linked into an instantaneous event chain <event A, event B> containing the two events.

[0025] In an optional embodiment, parsing the data into standardized memory atomic events carrying timestamps, entity, and action elements includes: Extract the URL title text as an entity, "browse" as an action, and the access time as a timestamp from the browser's access log to form a browser atomic event; Extract the destination name as an entity, "navigation" as an action, and the navigation start time as a timestamp from the navigation records of the map application to form a map atomic event.

[0026] The parsing process sets corresponding parsing rules for unstructured or semi-structured data from different sources to identify and extract key information. For example, for a browser access log record stating that a webpage titled "Frontiers in Artificial Intelligence Technology Conference" was accessed at 10:05 AM on October 27, 2023, the access time 2023-10-27 10:05:00 is extracted as the timestamp, the webpage title "Frontiers in Artificial Intelligence Technology Conference" is treated as the entity, and a standardized action tag, namely "browsing," is assigned to this behavior.

[0027] The extracted information is combined into a standardized data structure, namely, a memory atomic event. This structure is a triple, in the format of timestamp, entity, and action. Following this format, the browser log record mentioned above is converted into an atomic event, with the content "2023-10-27 10:05:00, Frontier Technology Conference on Artificial Intelligence, Browsing". Similarly, for a map navigation record with the content "Navigation to XX Conference Center started at 10:30 on October 27, 2023", this record is converted into the atomic event "2023-10-27 10:30:00, XX Conference Center, Navigation". In this way, all raw user data will be unified into a consistent sequence of atomic events that can be processed later.

[0028] In an optional embodiment, calculating the association score of adjacent event pairs includes: Calculate adjacent events and Semantic similarity score between entities and actions ; Calculate the time interval between events Increased and decaying time factor ; The semantic similarity score is weighted and combined with the time factor using the following formula to obtain the association score: ; in, and Preset weights.

[0029] For example, there are two events, event To access the AI ​​conference webpage, the event To navigate to the XX Conference Center, the entity and action text (i.e., AI Conference, browse, XX Conference Center, navigation) are converted into vectors using a word vector model. The word vectors corresponding to the events are then average-pooled to obtain event vectors. The cosine similarity between the vectors of two events is calculated, resulting in a score, such as 0.85. This score is denoted as [missing value]. .

[0030] The word vector model described is a neural network used to learn distributed representations of words. It consists of an input layer, a hidden projection layer, and an output layer, with architectures such as CBOW or Skip-gram. Training data consists of large-scale unlabeled text corpora, such as Baidu Encyclopedia entries and news articles. The training process is unsupervised, adjusting network weights by predicting context words or center words, ensuring that semantically similar words are also positioned close together in the vector space. The loss function is typically negative log-likelihood. The input is a single word from the text representing an event entity or action, and the output is a low-dimensional dense vector representing the semantics of that word, such as a 300-dimensional vector.

[0031] Hypothetical event It happened at 10:05 AM. It happened at 10:30, time interval It is 25 minutes. Time factor. The program is designed as a function that decreases as the time interval increases, such as an exponential decay function. Assuming the attenuation coefficient The value is 0.01, with units of / min. Therefore, the calculated time factor is approximately 0.78. The semantic similarity score and time factor are then weighted according to a preset formula. and Perform a weighted summation to obtain the correlation score. The higher the score, the stronger the causal relationship between the two events.

[0032] In another embodiment, the time factor is used as a weight to weight the semantic similarity score to obtain the association score.

[0033] S2, according to a predetermined cycle or batch, the instantaneous event chain is sent to the offline deep inference engine. The engine uses a dual-stream temporal semantic embedding model to extract the semantic features of each entity in the event chain and the temporal context features of the event chain, constructs an initial graph structure containing entity nodes, and then performs relational inference on the initial graph structure through a graph neural network to generate candidate graph substructures. For example, the offline inference engine is activated every hour or when the instantaneous event chain in the cache pool accumulates to 100. Taking a user journey event chain <search AA attractions, book a flight to AA, book a BB hotel> as an example, the engine first initializes the graph structure of the chain, instantiating the core entities in the chain such as "AA attractions", "flight", and "BB hotel" as graph nodes, and establishing initial temporal connection edges between nodes according to the order in which the events occur, thereby constructing an initial topological subgraph to be inferred.

[0034] A dual-stream temporal semantic embedding model is used to generate high-dimensional feature vectors for each node in the initial topological subgraph. The first temporal stream uses LSTM to process the original event sequence, obtaining the "search-booking" time interval and causal pattern, generating temporal features. The second semantic stream uses a pre-trained language model such as BERT to encode entity text and action descriptions, generating semantic features. The two feature streams are concatenated and attention-weighted to obtain a fused vector, which is then assigned to the corresponding node in the initial topological subgraph as the node feature matrix input to the graph neural network.

[0035] The initial topological subgraph is input into the Graph Attention Network (GAT) for deep relation inference. The GAT layer aggregates neighborhood information through a message-passing mechanism, calculating the probability of potential semantic relationships between nodes; for example, inferring a "travel" relationship from the geographically distant attributes of "attractions" and "hotels." The decoder reconstructs the graph structure based on the inferred high-confidence edges and outputs explicit RDF triples, such as <user, plan, travel plan>, <plan, includes, AA attractions>, and <plan, associated accommodation, BB hotels>. These triples ultimately form a complete and logically clear candidate graph substructure.

[0036] In an optional embodiment, the engine employs a dual-stream temporal semantic embedding model to extract the semantic features of each entity in the event chain and the temporal context features of the event chain, constructing an initial graph structure containing entity nodes, including: The event sequence in the event chain is input into a long short-term memory network to extract temporal context feature vectors that represent global information of the entire event chain. The entity and action text of each event in the event chain are input into the BERT model to extract the deep semantic feature vector of the corresponding event entity; The temporal context feature vector is concatenated with the deep semantic feature vector of each event entity to generate the fused feature representation of the corresponding node in the graph neural network.

[0037] The dual-stream model architecture comprises two parallel processing branches. The temporal processing stream utilizes a Long Short-Term Memory (LSTM) network. For example, consider a chain of events: Event 1 browsing an AI conference, Event 2 navigating the conference center, and Event 3 participating in a technical forum. Each event is transformed into a low-dimensional vector, and these three vectors are sequentially input into the LSTM network. The LSTM network processes each event vector in turn, updating its internal state to detect the sequence and dependencies between events. After processing the last event, the LSTM's hidden state vector serves as a global context feature vector representing the temporal pattern of the entire event chain; for example, it might be a 128-dimensional vector.

[0038] The Long Short-Term Memory (LSTM) network typically consists of an embedding layer, one or more stacked LSTM layers, and an output layer. The training set comprises a large number of event sequence samples, which may be labeled for classification or prediction tasks. The training process uses a backpropagation algorithm over time to optimize the network parameters, aiming to minimize the loss function between the predicted result and the true label, such as cross-entropy loss. The network input is a sequence of event vectors, where each vector represents an event in the event chain; the output is the hidden state vector at the last time step after processing the entire sequence, which is the temporal context feature vector.

[0039] The semantic processing flow employs the BERT model to understand the deep semantics of each entity in the event chain. For each event in the event chain (such as the entities "AI Conference" and the action "Browse" in Event 1), its text content is input into the BERT model. BERT leverages its powerful contextual understanding capabilities to generate a corresponding high-dimensional semantic representation for each input entity. Unlike compressing the entire chain into a single vector, this process extracts the output vector corresponding to a specific entity's token, or pools the text output of that specific event, thereby obtaining an independent deep semantic feature vector for each entity, such as generating a 768-dimensional vector for each entity. Feature fusion is then performed: a global 128-dimensional temporal vector obtained from the temporal flow is concatenated to the 768-dimensional semantic vector of each entity. Assuming there are N entities in the event chain, N 896-dimensional node feature vectors are generated. These vectors contain both the semantic information of their respective entities and share the global temporal context of the same event chain.

[0040] The BERT model, as described, includes a multi-head self-attention mechanism and a feedforward neural network in each layer. This model uses the self-attention mechanism to detect word dependencies in text. The training set consists of massive amounts of unlabeled text data, such as book and web page text. The training process is self-supervised learning, primarily conducted through two pre-training tasks: masked language modeling and next-sentence prediction. When using the pre-trained BERT model, the input is the entity and action text for each event; the output is the deep semantic vector corresponding to each entity.

[0041] In an optional embodiment, the step of performing relational reasoning on the initial graph structure using a graph neural network to generate candidate graph substructures includes: The entities in the event chain are used as input nodes of the graph attention network, and the deep semantic feature vectors extracted from the entities and action text of each event in the dual-stream temporal semantic embedding model are used as the initial features of the corresponding nodes. The feature representation of the node is updated through the graph attention network, and based on the updated node features, the probability score of forming a new type of relationship between entity nodes in the event chain is calculated through a relationship classifier. Entity pairs with probability scores greater than a preset probability threshold are selected, and candidate graph substructures are constructed using the predicted relationship types as edges.

[0042] Construct an initial graph, for example, an event chain containing entities like an AI conference and a conference center. Treat these two entities as nodes in the graph. Based on the chronological order of events in the event chain, establish directed temporal edges between entity nodes of adjacent events; or, based on calculated association scores, establish weighted edges between any pair of entity nodes with scores above a threshold. The initial feature vector of each node is directly derived from the fused feature representation generated by the aforementioned two-stream model: that is, the concatenation of the node's BERT semantic vector and the global LSTM temporal vector. Input the graph with these initial features into a graph attention network (GAT). Calculate the association weight between any two nodes in the graph, i.e., the importance of one node to another. Through multi-layer information propagation, the feature vector of each node aggregates information from its neighboring nodes.

[0043] The graph attention network (GAT) assigns different importance weights to the neighbors of a node, weighted aggregating the features of neighboring nodes to update the features of the central node. The network typically consists of multiple stacked graph attention layers and an output layer, and multi-head attention can be used to enhance the model's expressive power. As a graph encoder, the training objective of the GAT is to generate high-quality feature representations of nodes in the graph that incorporate contextual information about the graph structure. The training set consists of a large amount of graph structure data, such as entities and their relationships in a knowledge graph. The training process can be supervised learning, self-supervised learning, or unsupervised learning. After GAT processing, the model uses the updated node vectors to predict relationships. For example, the vector representations of "AI Conference" and "XX Conference Center" are extracted, concatenated, and fed into a classifier, preferably an MLP. This classifier outputs the probabilities of various possible relationships between the two entities; for example, the probability of a location relationship is 0.95, and the probability of a topic relationship is 0.05. A probability threshold, such as 0.9, is set. Since the probability of a location relationship (0.95) is greater than this threshold, the prediction is accepted. Construct a candidate graph substructure containing two nodes: the Artificial Intelligence Conference and the XX Conference Center, and a directed edge of type Location pointing from the former to the latter.

[0044] S3, the candidate graph substructure is integrated into the user memory main graph, wherein when the relationship in the candidate graph substructure conflicts with the existing relationship in the user memory main graph, the confidence of the two is calculated according to the evidence source type and the inference path length, and the relationship with the highest confidence is retained; Suppose a relation <User, City of Residence, Shanghai> already exists in the main graph. This relation is inferred from the user's IP address and food delivery address from several months ago. The evidence source type is web logs, with a preset weight of 0.8, an inference path length of 5, a preset decay coefficient of 0.9, and a calculated confidence level of 0.525. The inference path length refers to the number of edges in the shortest inference chain connecting the two entities involved in the relation and supporting its validity. Now, a new candidate substructure contains the relation <User, City of Residence, AA>. This relation originates from the text parsing of the user's newly signed rental contract PDF file. The evidence source type is user documents, with a preset weight of 0.9, an inference path length of 1, and a calculated confidence level of 0.9. Since 0.9 is higher than 0.525, the old relation is replaced with <User, City of Residence, AA> in the main graph. Each edge in the user memory master graph stores metadata attributes, which include at least: the source type of the relationship, such as direct observation, inference generation, the initial confidence value of generating the relationship, and the generation time; when updating the graph, the metadata of the new relationship is written synchronously.

[0045] In an optional embodiment, the step of calculating the confidence level of the evidence source type and the inference path length, and retaining the relationship with the highest confidence level, includes: Set preset weights for different types of evidence sources ; The confidence level Conf(R) of the relationship R is calculated based on the evidence source weights and the confidence factor, which decays with the length of the reasoning path L. The formula is as follows: ; in, The length of the reasoning path. The preset attenuation coefficient; When there is a conflict in a relationship, retain the relationship with the highest confidence level.

[0046] Assign basic confidence weights to different types of data sources. For example, data from structured applications such as calendars or maps are considered highly reliable and therefore given higher weights. The weight is 0.95; while data from browser logs and other sources that require parsing has a weight of 0.8. A decay coefficient is also set. , For example, 0.9 is used to penalize relations obtained through long-path reasoning. For instance, a relation stating that meeting A's location is location B, obtained from a user's map navigation history, with the evidence source being a map and the reasoning path length L=1, would have a confidence score of Conf(R)=0.95.

[0047] If another relation, that person C participated in meeting A, is inferred from the known relations that person C is a project team member and that the project team participated in meeting A, then the inference path length L = 2. Assuming the minimum weight of the evidence source is 0.8, the confidence level of this inference relation is Conf(R) = 0.72. When relational conflicts occur, such as two contradictory conclusions based on different evidence—that meeting A's location is location B with a confidence level of 0.95 and meeting A's location is location D with a confidence level of 0.72—the relation with the higher confidence level is chosen to be retained; that is, the relation stating that meeting A's location is location B is retained, and the other relation is discarded.

[0048] S4. Identify subgraphs in the updated user memory master graph that meet preset stability and connectivity conditions, and use a multi-head attention graph summarization network to abstract the subgraphs into higher-order memory concept nodes, and update the user memory master graph.

[0049] A community detection algorithm, such as the Louvain algorithm, is executed on the main graph. For example, with a resolution parameter of 1.0 and a maximum number of iterations of 50, a subgraph containing multiple closely related nodes is identified. These nodes include a series of events, locations, people, and consumption records related to Yunnan travel. The timestamps of all nodes and relationships in this subgraph are examined, and it is found that they have not changed in the past month, satisfying the stability condition. This subgraph is then input into a pre-trained graph summarization network. A multi-head attention mechanism focuses on the core people, key locations, and major activities in the subgraph, aggregating information to generate a new high-order concept node named "2023 Yunnan Trip." This new node is created in the main graph, and it is connected to the user node through an edge named "Owned Memories." A generalized relationship edge is established from the new node to each original node in the atomic graph. Based on this generalized relationship, the atomic nodes are folded and hidden under this high-order node, thereby achieving hierarchical organization and simplification of the graph. After creating and connecting the high-order concept node, the original edges inside the subgraph are deleted to simplify the graph structure. When the user's memory master graph does not exist, the first batch of candidate graph substructures will be directly used as the initial version of the user's memory master graph.

[0050] In an optional embodiment, the identification of subgraphs in the updated user memory master graph that satisfy preset stability and connectivity conditions includes: The stability condition is set to the existence time of all nodes and edges in the subgraph exceeding a preset duration, and the connectivity condition is set to the node density in the subgraph being greater than a preset density threshold.

[0051] Set a time threshold, such as 30 days. Traverse all nodes and edges in the main graph, checking the duration of each element since its creation. Assuming the current date is October 30th, a node representing a project created on September 1st, with a duration exceeding 30 days, is considered a stable node. Conversely, a node representing a temporary meeting created on October 20th, with a duration less than 30 days, is considered an unstable node. All unstable nodes and their connected edges are temporarily removed, resulting in a subgraph containing only stable elements.

[0052] Connectivity filtering is performed on stable subgraphs by setting a node density threshold, such as 0.4. The node density is the sum of the actual number of directed edges *m* in the subgraph and the maximum possible number of directed edges. The ratio is calculated. For each connected component in a stable subgraph, the node density is calculated. For example, a connected component with 4 nodes and 5 edges has a density of 0.417. Since 0.417 is greater than the threshold of 0.4, the subgraph consisting of 4 nodes and 5 edges is successfully identified. Another connected component with 5 nodes and 6 edges has a density of 0.3, which is lower than the threshold of 0.4, so this component will not be selected.

[0053] In an optional embodiment, the step of using a multi-head attention graph summarization network to abstract the subgraph into higher-order memory concept nodes and updating the user memory master graph includes: The node features and adjacency matrix of the identified subgraph are input into the graph summarization network; The network computes importance weights for each node in the subgraph in different representation subspaces using multiple parallel attention heads; The weighted node features calculated by each attention head are fused to generate a vector representing the summary information of the entire subgraph; Use the vector to create a new higher-order memory concept node, and connect the node to all nodes in the subgraph to update the main graph.

[0054] The network model receives a previously identified stable and connected subgraph as input. For example, the input might be a subgraph about attending an AI conference, containing nodes like "AI Conference," "XX Conference Center," "Colleague Zhang San," and the relationships between them. The node feature vectors and adjacency matrices representing node connections are fed into the graph summarization network. The network contains multiple, for example, eight, parallel attention heads. Each attention head independently analyzes the entire subgraph and assigns an importance score to each node. Different heads might focus on different aspects of the subgraph; for example, one head might consider the location node "XX Conference Center" most important, while another head might focus more on the person node "Colleague Zhang San."

[0055] The multi-head attention graph summarization network consists of multiple parallel graph attention heads, each independently learning the importance weights of nodes and performing weighted aggregation of node features. The outputs of each head are integrated, for example through concatenation or averaging, and then passed through a transformation layer to generate a single summary vector. The input is a complete subgraph, including the feature vectors of all nodes and the adjacency matrix; the output is a fixed-dimensional vector representing the core semantics and structure of the entire subgraph, which is used as the feature of newly generated higher-order memory concept nodes.

[0056] Each attention head performs a weighted summation of the feature vectors of all nodes based on its calculated importance score, generating a summary vector for that head. Eight heads will produce eight different summary vectors. These eight summary vectors are then fused, typically by concatenation or averaging, and then subjected to a linear transformation to generate a single vector representing the core concept of the entire subgraph. This vector is the vector representation of higher-order memory. A new node is created in the main graph, named "Higher-Order Memory: Participation in the AI ​​Conference," and its features are set as the summary vector. Simultaneously, new edges are drawn from this new node, connecting to all original nodes in the subgraph, such as "AI Conference," "XX Conference Center," and "Colleague Zhang San." Atomic nodes are folded and hidden beneath this higher-order node, thus constructing a clear semantic connection from higher-order concept nodes to atomic nodes at the data level, achieving a visually hierarchical organization and simplification of the graph.

[0057] In a second embodiment, the present invention also provides a system for constructing cross-device, cross-modal user memory maps, comprising the following modules: The module is used to acquire multimodal heterogeneous data from multiple device terminals and applications, and parse the data into standardized memory atomic events carrying timestamps, entities and action elements; for the real-time generated memory atomic event stream, the correlation score of adjacent event pairs is calculated, and when the correlation score is higher than a preset threshold, the event pairs are linked to construct an instantaneous event chain; The generation module is used to send the instantaneous event chain into the offline deep inference engine according to a predetermined period or batch. The engine uses a dual-stream temporal semantic embedding model to extract the semantic features of each entity in the event chain and the temporal context features of the event chain, constructs an initial graph structure containing entity nodes, and then performs relational inference on the initial graph structure through a graph neural network to generate candidate graph substructures. The calculation module is used to integrate the candidate graph substructures into the user memory main graph. When a relationship in the candidate graph substructure conflicts with an existing relationship in the user memory main graph, the confidence level of the two is calculated based on the evidence source type and the inference path length, and the relationship with the highest confidence level is retained. The update module is used to identify subgraphs in the updated user memory master graph that meet preset stability and connectivity conditions, and to use a multi-head attention graph summarization network to abstract the subgraphs into higher-order memory concept nodes, thereby updating the user memory master graph.

[0058] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0059] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing a cross-device, cross-modal user memory graph, characterized in that, Includes the following steps: Acquire multimodal heterogeneous data from multiple device terminals and applications, and parse the data into standardized memory atomic events carrying timestamps, entities, and action elements; calculate the correlation score of adjacent event pairs for the real-time generated memory atomic event stream, and link the event pairs to construct an instantaneous event chain when the correlation score is higher than a preset threshold; According to a predetermined cycle or batch, the instantaneous event chain is sent to an offline deep inference engine. The engine uses a dual-stream temporal semantic embedding model to extract the semantic features of each entity in the event chain and the temporal context features of the event chain, constructs an initial graph structure containing entity nodes, and then performs relational inference on the initial graph structure through a graph neural network to generate candidate graph substructures. The candidate graph substructures are integrated into the user memory main graph. When a relationship in the candidate graph substructure conflicts with an existing relationship in the user memory main graph, the confidence level of both is calculated based on the evidence source type and the length of the reasoning path, and the relationship with the highest confidence level is retained. The system identifies subgraphs in the updated user memory master graph that meet preset stability and connectivity conditions, and uses a multi-head attention graph summarization network to abstract these subgraphs into higher-order memory concept nodes, thereby updating the user memory master graph.

2. The method according to claim 1, characterized in that, The process of parsing the data into standardized memory atomic events carrying timestamps, entities, and action elements includes: Extract the URL title text as an entity from the browser's access log, "browse" as an action, and the access time as a timestamp to form a browser atomic event; Extract the destination name as an entity, "navigation" as an action, and the navigation start time as a timestamp from the navigation records of the map application to form a map atomic event.

3. The method according to claim 2, characterized in that, The calculation of the association score of adjacent event pairs includes: Calculate the semantic similarity score between entities and actions in adjacent events; Calculate the time factor that decays as the event time interval increases; The association score is obtained by weighting the semantic similarity score with the time factor.

4. The method according to claim 1, characterized in that, The engine employs a dual-stream temporal semantic embedding model to extract the semantic features of each entity in the event chain and the temporal context features of the event chain, constructing an initial graph structure containing entity nodes, including: The event sequence in the event chain is input into a long short-term memory network to extract temporal context feature vectors that represent global information of the entire event chain. The entity and action text of each event in the event chain are input into the BERT model to extract the deep semantic feature vector of the corresponding event entity; The temporal context feature vector is concatenated with the deep semantic feature vector of each event entity to generate the fused feature representation of the corresponding node in the graph neural network.

5. The method according to claim 1, characterized in that, The step of performing relational reasoning on the initial graph structure using a graph neural network to generate candidate graph substructures includes: The entities in the event chain are used as input nodes of the graph attention network, and the deep semantic feature vectors extracted from the entities and action text of each event in the dual-stream temporal semantic embedding model are used as the initial features of the corresponding nodes. The feature representation of the node is updated through the graph attention network, and based on the updated node features, the probability score of forming a new type of relationship between entity nodes in the event chain is calculated through a relationship classifier. Entity pairs with probability scores greater than a preset probability threshold are selected, and candidate graph substructures are constructed using the predicted relationship types as edges.

6. The method according to claim 1, characterized in that, The calculation of the confidence level between the evidence source type and the inference path length, and the retention of the relationship with the highest confidence level, includes: Set preset weights for different types of evidence sources ; The confidence level Conf(R) of the relationship R is calculated based on the evidence source weights and the confidence factor, which decays with the length of the reasoning path L. The formula is as follows: ; in, The length of the reasoning path. The preset attenuation coefficient; When there is a conflict in a relationship, retain the relationship with the highest confidence level.

7. The method according to claim 4, characterized in that, The identified subgraphs in the updated user memory master graph that satisfy preset stability and connectivity conditions include: The stability condition is set to the existence time of all nodes and edges in the subgraph exceeding a preset duration, and the connectivity condition is set to the node density in the subgraph being greater than a preset density threshold.

8. The method according to claim 1, characterized in that, The step of using a multi-head attention graph summarization network to abstract the subgraph into higher-order memory concept nodes and updating the user memory master graph includes: The node features and adjacency matrix of the identified subgraph are input into the graph summarization network; The network computes importance weights for each node in the subgraph in different representation subspaces using multiple parallel attention heads; The weighted node features calculated by each attention head are fused to generate a vector representing the summary information of the entire subgraph; Use the vector to create a new higher-order memory concept node, and connect the node to all nodes in the subgraph to update the main graph.

9. A system for constructing a cross-device, cross-modal user memory map, characterized in that, Includes the following modules: The module is used to acquire multimodal heterogeneous data from multiple device terminals and applications, and parse the data into standardized memory atomic events carrying timestamps, entities and action elements; for the real-time generated memory atomic event stream, the correlation score of adjacent event pairs is calculated, and when the correlation score is higher than a preset threshold, the event pairs are linked to construct an instantaneous event chain; The generation module is used to send the instantaneous event chain into the offline deep inference engine according to a predetermined period or batch. The engine uses a dual-stream temporal semantic embedding model to extract the semantic features of each entity in the event chain and the temporal context features of the event chain, constructs an initial graph structure containing entity nodes, and then performs relational inference on the initial graph structure through a graph neural network to generate candidate graph substructures. The calculation module is used to integrate the candidate graph substructures into the user memory main graph. When a relationship in the candidate graph substructure conflicts with an existing relationship in the user memory main graph, the confidence level of the two is calculated based on the evidence source type and the inference path length, and the relationship with the highest confidence level is retained. The update module is used to identify subgraphs in the updated user memory master graph that meet preset stability and connectivity conditions, and to use a multi-head attention graph summarization network to abstract the subgraphs into higher-order memory concept nodes, thereby updating the user memory master graph.

10. The system according to claim 9, characterized in that, The process of parsing the data into standardized memory atomic events carrying timestamps, entities, and action elements includes: Extract the URL title text as an entity from the browser's access log, "browse" as an action, and the access time as a timestamp to form a browser atomic event; Extract the destination name as an entity, "navigation" as an action, and the navigation start time as a timestamp from the navigation records of the map application to form a map atomic event.

Images