A spatiotemporal causal-based memory system technique

Abstract

This invention discloses a method for designing a memory system based on spatiotemporal causality. The system uniformly encodes multimodal events into vectors that integrate time, space, and semantics. It constructs a directed acyclic causal graph with spatiotemporal annotations through an online causal discovery mechanism, forming a hierarchical evolutionary structure of short-term working memory, intermediate-term episodic memory, and long-term semantic memory. The system can perform cognitive tasks such as forward prediction, reverse causal diagnosis, analogy retrieval, and counterfactual simulation, and achieves end-to-end differentiable optimization through a neural network module. This invention endows intelligent agents with the ability to understand "why it happened" and imagine "how it might evolve," showing broad application prospects in embodied intelligence, complex event understanding, and prediction.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and cognitive computing, specifically to a structured memory system that integrates spatiotemporal information and causal reasoning, and its construction, evolution and retrieval methods, which can be applied to scenarios such as intelligent robots, autonomous driving, smart city monitoring, and complex event understanding and prediction. Background Technology

[0002] Most current mainstream artificial intelligence memory systems[1] use vector indexing[2] or time-series-based retrieval schemes, which can only associate events that "occur sequentially" and lack a deep understanding of the causal logic between events. For example, conventional memory networks can remember the fragment "water accumulated on the ground after a car passed by", but cannot answer counterfactual questions such as "was the water accumulated by the car or by rain?" In addition, existing systems usually do not encode time, space and semantics in a unified way, making it difficult to uncover the real causal structure under specific spatiotemporal constraints, resulting in weak prediction, diagnosis and simulation capabilities. Cognitive science shows that human memory not only depends on temporal proximity, but also organizes experience through causal relationships and forms a hierarchical structure of short-term working memory, medium-term episodic memory and long-term semantic memory. Therefore, there is an urgent need for a memory system that can structurally express spatiotemporal causality, autonomously discover causal chains, hierarchically solidify memory and support counterfactual reasoning. Summary of the Invention

[0003] The purpose of this invention is to provide a memory system and method based on a spatiotemporal causal graph, aiming to solve the problems of weak causal understanding, spatiotemporal fragmentation, lack of hierarchical evolution, and counterfactual reasoning capabilities in existing memory systems. This invention structurally organizes time, space, and causal relationships into a dynamic directed graph, constituting a memory system with spatiotemporal analysis capabilities. This provides intelligent agents with the ability to remember "what happened," understand "why it happened," and "how it might evolve." Its core steps and mechanisms include: unified encoding of multimodal events, construction and dynamic evolution of the spatiotemporal causal graph, causal reasoning and memory retrieval, and machine learning capabilities that support these three parts with adaptive, learnable, and strong generalization abilities. The organization and workflow are described in [reference needed]. Figure 1 .

[0004] 1. Unified coding of multimodal events Raw events from multiple sources, including sensors, text, and video, are uniformly encoded into an event vector e=[e...]. time ⊕e space ⊕e sem ], as a candidate node in the spacetime causal graph.

[0005] 2. Construction and Dynamic Evolution of Spatiotemporal Causal Graphs Graph structure definition: Construct a directed acyclic graph G=(V,E,T,S,C), where nodes represent events, directed edges represent causal effects, and time labels, spatial labels, and causal strength functions are attached.

[0006] Incremental node creation: When a new event arrives, its spatiotemporal semantic similarity with existing nodes is calculated; if it is below a threshold, a new node is created; otherwise, it can be merged into the observation update of an existing node.

[0007] Causal edge discovery (core): Instead of relying on simple temporal correlations, it determines the true causal chain through multiple verification mechanisms. Time-priority filtering and spatial proximity constraints; Online causality testing; Proactive intervention and counterfactual verification (if the system has intervention capabilities); The strength of causal edges is dynamically updated in a Bayesian manner.

[0008] Hierarchical Memory Consolidation and Forgetting: Mimicking the hippocampal-cortical memory transformation, memory is divided into three layers: Short-term working memory (high-resolution detail image); Mid-term plot memory (spatiotemporal clustering compressed into "plot units"); Long-term semantic memory (distilling abstract causal rules from plot).

[0009] Simultaneously, based on retrieval frequency, causal hub degree, and novelty, low-frequency, isolated nodes and edges are subject to decay and forgetting.

[0010] 3. Causal reasoning and memory retrieval Forward prediction: Starting from the current node, perform a random walk along the causal edges to generate a probability distribution of future events.

[0011] Reverse diagnostics: Tracing the causal path in reverse, calculating the attribution score of the parent node, and identifying the main causal chain.

[0012] Analogical retrieval: Using the current local subgraph as a query, a graph matching algorithm is used to find historical events in the graph library that are similar in topology and causal structure, thus realizing case-based reasoning.

[0013] Counterfactual simulation: By deleting specific nodes and their downstream edges in the graph, and combining the conditional generation model to complete the remaining graph structure, the counterfactual results are compared with the actual results to quantify the causal effect and give the system the ability to "hypothesize".

[0014] 4. Adaptive Optimization Based on Machine Learning Differentiable module: By implementing three capabilities—network coding, causal edge prediction, and memory compression—it supports the collaborative optimization of causal discovery and memory compression accuracy.

[0015] Reinforcement learning interface: It allows agents to observe the evolution of causal graphs by intervening in action nodes, maximizing external rewards and updating the graph structure.

[0016] Meta-learning transfer: Using long-accumulated causal patterns as strong priors, it quickly guides graph construction in new environments, enabling spatiotemporal causal cognition with few samples. Attached Figure Description

[0017] Figure 1 Overall Workflow Figure 2 System Architecture Figure 3 Multimodal event coding process Figure 4 Multiple verification mechanism process for causal edge discovery Figure 5 A machine learning-based framework for co-training differentiable modules and causal graphs.

[0018] [1] A Survey on Large Language Model based Autonomous Agents. ByWang, L., Ma, C., Feng, X., et al. 2024 [2] Survey on Vector Database Management Systems. By Pan, J., et al.2023 Detailed Implementation

[0019] The implementation of this invention includes: 1. System architecture; 2. Multimodal information encoding; 3. Spatiotemporal causal graph construction and evolution; 4. Causal reasoning and memory retrieval; 5. Machine learning-based adaptive optimization engine; 6. Storage management and optimization.

[0020] A memory system based on a spatiotemporal causal graph weaves together time, space, and causal relationships in a structured way, enabling intelligent agents to not only remember "what happened," but also understand "why it happened" and "how it might evolve," just like humans. Below is a complete design scheme.

[0021] 1. System Overall Architecture The system is divided into four layers. The machine learning engine provides adaptive optimization services for spatiotemporal causal graph management and memory retrieval, while the graph information management service serves as the foundational service engine for spatiotemporal causal graph management and operation (see [link]). Figure 2 ): Data access layer: Implements multimodal event encoding, including the extraction and fusion of temporal, spatial, and semantic features. Spatiotemporal Causal Graph Management and Evolution Layer: Supports node creation, edge discovery, graph pruning, and hierarchical clustering. Causal Reasoning and Memory Retrieval Layer: Supports causal path search, analogy retrieval, and counterfactual reasoning based on causal graphs. Application Interface Layer: Supports services such as querying, prediction, interpretation, planning, and simulation required for building applications.

[0022] 2. Multimodal event coding Each event entering the system may originate from raw data sources such as sensors, text, and video. This data is first encoded into a unified event vector (see reference). Figure 3 (Sample process): e=[e time ⊕e space ⊕e sem ] e time Time encoding (such as Fourier features of timestamps or learned location encoding).

[0023] e space Spatial coding (latitude, longitude, altitude, or deep spatial features extracted from point clouds / images).

[0024] e sem Semantic features (vector representations of embeddings generated by multimodal large models, such as CLIP and GPT).

[0025] These events will serve as candidate nodes in the spacetime causal graph.

[0026] 3. Construction and Evolution of Spacetime Causal Graphs 3.1 Spacetime Causal Graph A spacetime causal graph is a directed acyclic graph, defined as: G=(V,E,T,S,C) V: Event node, representing an atomic unit of state, behavior, or observation.

[0027] E: Directed edge, representing causal influence or temporal evolution between nodes.

[0028] T: The time stamp function t:V→R assigns an event timestamp or time range.

[0029] S: Spatial label function s:V→R d This assigns the location or spatial embedding of the event.

[0030] C: Causal strength function c:E→[0,1], representing the degree of certainty of the cause regarding the result.

[0031] Key features: The edges are not merely "sequential relationships," but rather genuine causal chains verified through causal discovery (such as intervention and counterfactual reasoning).

[0032] The graph grows over time increments, and evolves dynamically through forgetting and consolidation mechanisms to form a long-term, hierarchical memory structure.

[0033] 3.2 Spatiotemporal causal graph node creation When a new event arrives, calculate its similarity with existing memory nodes: sim(e new ,v_i)=α⋅cos (e sem ,v i sem )+β⋅e x +δ⋅e y x=−γ∥s(e new )−s(v i )∥ y=−λ∣t(e new )−t(v i )∣ If the maximum similarity is lower than the threshold, a new node v new is created; otherwise, the new event may be considered as an "observation update" merged into the existing node to enhance its statistical information.

[0034] 3.3 Causal edge discovery Causal discovery is the core of the system. Instead of relying solely on temporal correlation, directed edges are established through the following mechanisms (refer to Figure 4 for the sample process): Time priority filtering: The edge u→v is considered only when t(u)<t(v) and t(v)−t(u)<τ max .

[0035] Spatial proximity constraint: The spatial distance between s(u) and s(v) must be less than the spatial threshold to ensure the locality of causal relationships.

[0036] Granger causality test (online version): Using the data in the historical window, test whether adding the past values of u can significantly improve the prediction accuracy for v v .

[0037] Intervention and experiment: If the system has intervention capabilities (e.g., a robot performing actions), it can actively intervene and observe the results to obtain data of the intervention distribution, and use JCI and do-calculus to determine the true causality.

[0038] Counterfactual verification: When an abnormality in the occurrence of v is observed, backtrack to check whether the occurrence probability of v is significantly reduced if u does not occur (which needs to be combined with a generative model or retrieve similar historical segments).

[0039] Causal strength update: edge weight c u→v Updated in a Bayesian manner, incorporating the statistical significance of multiple observations.

[0040] 3.4 Hierarchical Memory Consolidation Mimicking hippocampal-cortical memory transformation: Working memory (short term): A high-resolution event graph of the most recent hours, retaining all details.

[0041] Episodic Memory (Intermediate Stage): Closely related nodes are compressed into "episodic units" through spatiotemporal clustering. For example, a set of causal events occurring consecutively at the same location is clustered as Episode_001: Making coffee. The episode retains a micro-causal graph internally, while exposing a macro-level interface externally.

[0042] Semantic memory (long-term): Extracting causal patterns from multiple similar scenarios to form decontextualized abstract rules. For example, the pattern (heating water, →, water boils) becomes knowledge. This is achieved through graph kernel or GNN clustering and concept distillation on the scenario graph.

[0043] Forgetting mechanism: The probability of retaining edges and nodes is related to their retrieval frequency, causal importance (how central they are in multiple causal chains), and novelty. Low-frequency and isolated nodes will be decayed and removed.

[0044] 4. Causal Reasoning and Memory Retrieval 4.1 Graph-based queries Predictive query: "If u happens now, what might happen next?" Starting from node u, perform a forward random walk along the causal edges, based on the transition probability c. u→v The sampling path is weighted by time decay to obtain the distribution of future events.

[0045] Diagnostic query: "Why did v occur?" By tracing the causal path backward from v, calculating the Shapley value or counterfactual attribution score of each parent node, the main causal chain can be found.

[0046] Analogical retrieval: The currently active local subgraph is used as the query, and a graph matching algorithm is used to search for historical events with similar topology and consistent causal structure in the graph database, thus realizing "case-based reasoning".

[0047] 4.2 Counterfactual Simulation Given an observed episode graph G obs The system can perform counterfactual actions: "What would have happened if it hadn't rained?" Delete the "Rain" node and its downstream edges.

[0048] Based on the remaining graph structure, a trained conditional generation model (as shown in the figure diffusion model) is used to complete the new states of missing nodes.

[0049] By comparing counterfactual results with actual results, the causal effect can be quantified.

[0050] This endows the system with the ability to imagine and evaluate hypotheses.

[0051] 5. Machine Learning-Based Adaptive Optimization Engine This invention combines machine learning capabilities to achieve adaptive optimization, providing services for multimodal coding, causal graph construction and evolution, and causal graph memory retrieval. The machine learning framework includes end-to-end modular design and meta-learning and transfer functions (workflow can be found in the documentation). Figure 5 (Example).

[0052] 5.1 End-to-end Differentiable Module Some modules can be implemented using neural networks and trained in conjunction with causal graph structures: Graph Encoder: Uses a temporal graph neural network to generate dynamic representations of nodes and graphs for similarity calculation and prediction.

[0053] Edge predictor: A learned scoring function fθ(vu,vv,Δt,Δs) outputs the probability of the existence of causal edges, guiding online causal discovery.

[0054] Memory Compressor: Variational Autoencoder (VAE) compresses subgraphs into plot representations and maximizes the accuracy of causal edge reconstruction, achieving lossless compression.

[0055] Reinforcement learning interface: In embodied intelligence scenarios, the system can serve as a world model. The agent learns to maximize external rewards by intervening in action nodes, observing the evolution of the causal graph, and updating the graph with new data.

[0056] 5.2 Meta-learning and Transfer Long-accumulated causal patterns (such as "knocking on an egg will cause it to break") constitute a priori graphs. When entering a new environment, the system can load relevant causal patterns as strong priors to quickly guide the construction of new graphs, achieving spatiotemporal causal cognition with few samples.

[0057] 6. Storage and Computing Architecture To support large-scale, real-time operation: Graph storage: Employs a graph data engine combined with a spatiotemporal index.

[0058] Embedded Index: Accelerates approximate search of event vectors with FAISS, quickly linking current perception with historical memory.

[0059] Stream processing: Events first enter the streaming data engine (e.g., a high-performance message queue), are processed by the online learning module, and then written to the graph library.

[0060] Hierarchical caching: Hot working memory resides in memory (e.g., the memory of an AI computing accelerator card), episodic memory resides in system memory, and semantic patterns reside on disk.

Claims

1. A memory system technology based on spatiotemporal causality, characterized in that, A memory system for complex event organization structures is constructed by integrating spatiotemporal information and causal relationships.

2. A memory system technology based on spatiotemporal causality, characterized in that, The main modules include: unified encoding of multimodal events, spatiotemporal causal graph construction and dynamic evolution, causal reasoning and memory retrieval, and machine learning capabilities that support these three parts with adaptive, learnable, and strong generalization capabilities.

3. A memory system technology based on spatiotemporal causality, characterized in that, The system is divided into four layers: data access layer, spatiotemporal causal graph management and evolution layer, causal reasoning and memory retrieval layer, and application interface layer. The machine learning engine provides adaptive optimization services for data access, spatiotemporal causal graph management and memory retrieval, and the graph information management service serves as the basic service engine for spatiotemporal causal graph management and operation.

4. A memory system technology based on spatiotemporal causality, characterized in that, By combining machine learning capabilities to achieve adaptive optimization, it provides services for multimodal coding, causal graph construction and evolution, and causal graph memory retrieval. Its machine learning framework includes end-to-end modular and meta-learning and transfer functions.

5. A memory system technology based on spatiotemporal causality, characterized in that, Directed edges are established through the following steps: time-priority selection; spatial proximity constraint; Granger causality test; intervention and experimentation; counterfactual verification; and causal strength update.

6. A memory system technology based on spatiotemporal causality, characterized in that, A spacetime causal graph is defined using a directed acyclic graph.

7. A memory system technology based on spatiotemporal causality, characterized in that, The edges of a spacetime causal graph are not merely "sequential relationships," but rather genuine causal chains verified through causal discovery (such as intervention and counterfactual reasoning).

8. A memory system technology based on spatiotemporal causality, characterized in that, The spatiotemporal causal graph grows incrementally over time and evolves dynamically through forgetting and solidification mechanisms, forming a long-term, hierarchical memory structure.

9. A memory system technology based on spatiotemporal causality, characterized in that, Multimodal event encoding includes: temporal encoding, spatial encoding, and semantic features.

Images