Embodied intelligent oriented dynamic scene graph real-time construction method, device and medium

Abstract

The application discloses a dynamic scene graph real-time construction method and device for embodied intelligence, and a medium, and relates to the technical field of embodied intelligence. The method comprises the following steps: taking a dynamic scene graph obtained by processing a previous frame of a current frame as a current dynamic scene graph, and taking the first frame as empty; performing multi-modal perception processing on the current frame to obtain instance nodes, attributes and relationship information of the instance nodes; associating the instance nodes with existing nodes in the current dynamic scene graph, and updating according to the association result; wherein, the attributes of the corresponding existing nodes are updated for the instance nodes that are successfully associated, new nodes are created for the instance nodes that are not associated, the existing nodes that disappear are deleted, and directed edges are updated according to the relationship information; and taking the updated dynamic scene graph as the current dynamic scene graph of a next frame. The application can realize real-time structured representation and updating of a dynamic visual scene, and meets the real-time environment understanding requirement of embodied intelligence.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence technology, and in particular to a method, device and medium for real-time construction of dynamic scene maps for embodied intelligence. Background Technology

[0002] Dynamic scene graphs aim to describe people, objects, environment, and their relationships in video footage in the form of structured graphs, providing a computable environmental understanding foundation for high-level applications such as autonomous driving and robot interaction.

[0003] However, real-world scenes are constantly changing; various instances and their relationships appear, disappear, or change over time, requiring the scene graph construction process to closely follow the changing rhythm of the video stream. In existing technologies, most mainstream scene graph construction methods are designed for single static images and cannot be directly applied to video sequences. Furthermore, to process video input, a common approach is to treat each frame as an independent image, running the entire process of object detection, attribute recognition, and relationship extraction frame by frame to generate a corresponding scene graph snapshot. This frame-by-frame full-scale construction method re-identifies and associates all visible instances in each frame, failing to utilize the temporal continuity between adjacent frames, leading to a large amount of redundant computation. As the number of instances and the complexity of relationships in the scene increase, the processing time per frame rises rapidly, and the scene graph update delay becomes significant, making it difficult to maintain synchronization under real-time video stream conditions. Furthermore, the independently constructed scene graphs for each frame are fragmented, lacking means to identify instances that persist across frames, making it difficult to distinguish whether an object is continuously visible from the previous frame or newly entered the frame. This results in frequent fluctuations in the output scene graph structure on the timeline, failing to provide stable and consistent structured information for downstream tasks. These limitations prevent high-level reasoning, task planning, and other modules from obtaining accurate scene understanding in a timely manner, severely restricting the responsiveness of real-time interactive systems. Summary of the Invention

[0004] This invention provides a method, device, and medium for real-time construction of dynamic scene maps for embodied intelligence. The technical problem it aims to solve is: how to provide an effective solution for real-time structured representation and updating of dynamic visual scenes.

[0005] In a first aspect, the present invention provides a method for real-time construction of dynamic scene maps for embodied intelligence, wherein for each video frame in the received video frame sequence, the following steps are performed sequentially: The dynamic scene map obtained from the processing of the previous video frame is used as the current dynamic scene map, wherein when the video frame is the first frame of the video frame sequence, the current dynamic scene map is empty; Multimodal perception processing is performed on the video frames to obtain a set of instance nodes, attribute information of each instance node, and relationship information between nodes; Based on the set of instance nodes and attribute information, the instance nodes are associated with the existing nodes in the current dynamic scene graph. The current dynamic scene graph is updated according to the association results to obtain the updated dynamic scene graph. The updated dynamic scene graph is then used as the current dynamic scene graph corresponding to the next video frame. The step of updating the current dynamic scene graph based on the association results includes: updating the attributes of the corresponding existing nodes in the current dynamic scene graph for successfully associated instance nodes; creating new nodes in the current dynamic scene graph for instance nodes not associated with existing nodes; deleting existing nodes from the current dynamic scene graph for existing nodes not associated with any instance nodes in the video frame; and updating the directed edges between nodes in the current dynamic scene graph based on the relationship information between the nodes.

[0006] Optionally, associating the instance node with existing nodes in the current dynamic scene graph includes: Extract the appearance features of the instance node and the historical appearance features of the existing node; Based on the motion information of the instance node and the historical motion trajectory of the existing node, predict the estimated location region of the existing node in the video frame; Calculate the appearance similarity between the instance node and the existing node in the feature space, and calculate the spatial overlap between the actual position of the instance node and the estimated position region; The appearance similarity and spatial overlap are weighted and fused to obtain the association confidence score. Instance nodes with an association confidence level higher than a preset association threshold are identified as successfully associated with existing node pairs.

[0007] Optionally, updating the attributes of the corresponding existing nodes in the current dynamic scene graph for successfully associated instance nodes includes: Determine the attribute type of the attribute; If the attribute type is a continuously changing first type attribute, then based on the attribute value of the successfully associated instance node, the corresponding attribute value of the existing node is updated by smooth filtering. If the attribute type is a discretely changing second type attribute, then obtain the state sequence of the existing node in a series of consecutive video frames including the video frame, and determine the stability of the state sequence. When the determination result meets the preset stability condition, the second type attribute of the existing node is updated to the attribute value of the successfully associated instance node.

[0008] Optionally, updating the directed edges between nodes in the current dynamic scene graph based on the relationship information between the nodes includes: Determine whether the target relationship in the relationship information already exists in the current dynamic scene graph; If it does not exist, create a directed edge connecting the two successfully associated instance nodes and set an initial confidence level for the directed edge; If it already exists, then the target relationship is determined to be in a continuous observation state in the video frame, and the confidence of the directed edge is increased; For existing directed edges in the current dynamic scene graph that have not been observed again in the video frame, reduce the confidence of the existing directed edges; Delete directed edges with a confidence level lower than a preset extinction threshold from the current dynamic scene graph.

[0009] Optionally, deleting existing nodes that are not associated with any instance node in the video frame from the current dynamic scene graph includes: Maintain a liveness timer for each existing node; When the existing node fails to associate with the corresponding instance node in the video frame for the first time, the survival timer is started to keep track. If, within a series of consecutive video frames before the survival timer reaches a preset extinction time threshold, the existing node is reassociated with the instance node, the survival timer is reset. If the existing node is still not associated with any instance node when the survival timer reaches the preset extinction time threshold, then the existing node and all directed edges connected to it are deleted from the current dynamic scene graph.

[0010] Optionally, the method further includes: During the process of updating the current dynamic scene graph based on the association results, a change log is generated for each node and each directed edge in the current dynamic scene graph. The change log includes the change type, the change time, and the attribute values ​​or confidence levels before and after the change. The change logs generated by each update are sequentially appended to the operation log sequence bound to each node and each directed edge to form their respective historical evolution sequences; In response to receiving a backtracking query request carrying a target time, the target time is parsed and the timestamp corresponding to the target time is determined; In the historical evolution sequence, starting from the end of the sequence and moving towards the beginning, the change time of each change log is compared with the timestamp, and the first change log whose change time is no later than the timestamp is located as the restoration reference point. Starting from the restoration reference point, proceeding forward along the historical evolution sequence, apply the reverse operations recorded in the change log one by one in reverse order until backtracking to the graph state corresponding to the timestamp, and construct a dynamic scene graph snapshot at the target time. The reverse operation is to restore the changed state to the state before the change. The snapshot of the dynamic scene map at the target time is output as a structured query result.

[0011] Optionally, the dynamic scene graph is stored using a multi-version graph data structure. The basic storage unit of the multi-version graph data structure is a key-value pair with a timestamp, used to record the value of node attributes or edge confidence at a certain moment. For the successfully associated instance nodes, the attribute of the corresponding existing node in the current dynamic scene graph is updated. Specifically, in the multi-version graph data structure, a new key-value pair with the current timestamp is appended to the updated attribute of the existing node, and the original key-value pair is retained as a historical version. The step of deleting an existing node from the current dynamic scene graph for an existing node that is not associated with any instance node in the video frame specifically involves writing a deletion marker with the current timestamp for the existing node in the multi-version graph data structure.

[0012] Optionally, the method further includes: Obtain a depth map synchronized with the video frame; Based on the depth map, calculate the point cloud contour of the instance represented by the instance node; The relationship information between the nodes is verified based on the spatial contact state between the point cloud contours. When it is determined from the point cloud contour that two instances have no physical contact in three-dimensional space, the confidence of the directed edge representing the physical contact relationship between the two instances is reduced, or the creation of the directed edge representing the physical contact relationship is prevented.

[0013] Secondly, the present invention also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0014] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0015] This invention provides a method, device, and medium for real-time construction of dynamic scene atlases for embodied intelligence. Through an incremental atlas update architecture with inter-frame iteration, the processing of each video frame directly inherits the dynamic scene atlas constructed in the previous frame as the current dynamic scene atlas. Only the attributes, existence, and directed edges of nodes in the atlas are locally updated based on the perception results of the current frame, rather than a full reconstruction frame by frame. Furthermore, based on a differentiated processing strategy for association results, the attributes of existing instances are refreshed simultaneously, nodes for newly appearing instances are created, and nodes for disappeared instances are deleted. The confidence of directed edges between nodes is dynamically maintained, allowing the atlas structure to continuously evolve with the video stream, accurately representing the dynamic changes of the physical scene. Furthermore, this method transforms unstructured pixel streams into structured dynamic scene atlases in real time with low single-frame update latency, meeting the requirements of embodied intelligent agents for real-time structured representation and updating of dynamic visual scenes, providing a continuous, stable, and low-latency environmental cognition data foundation for upper-level tasks. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram illustrating an application scenario of a real-time construction method for dynamic scene graphs oriented towards embodied intelligence, as provided in an embodiment of the present invention. Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0020] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0023] Please see Figure 1 This invention provides a method for real-time construction of dynamic scene graphs for embodied intelligence. Embodied intelligence refers to intelligent agents capable of perceiving and interacting with the physical environment in real time, such as service robots, industrial robotic arms, and autonomous vehicles. Its core characteristic is that the intelligent agent is embedded in the physical world, continuously acquiring environmental information through sensors and performing physical actions based on the understanding results. Furthermore, a dynamic scene graph (DSG) is a technique for structurally representing visual scenes using graph data structures. Nodes represent instances in the scene (such as people, objects, and environmental elements), and directed edges represent relationships between instances (such as spatial relationships, physical contact relationships, and semantic interaction relationships). Unlike scene graphs that only describe a single static image, dynamic scene graphs emphasize continuous evolution over time. Both nodes and edges have lifecycles, dynamically changing with the appearance, movement, interaction, and disappearance of instances in the physical world. Furthermore, this method processes the continuous video frame sequence input by the image acquisition device frame by frame, transforming the unstructured pixel stream into a structured map that reflects the identity, attributes, and dynamic interaction relationships of instances in the physical world in real time. This provides a low-latency, high-reliability environmental cognition foundation for upper-level behavioral decision-making, task planning, and natural language question answering.

[0024] Furthermore, the core contribution of this invention lies in constructing an incremental graph update architecture based on inter-frame iteration. Traditional methods independently execute complete object detection, attribute recognition, and relation extraction processes for each frame of a video, essentially treating the video as a static sequence of unrelated images. This frame-by-frame full-scale reconstruction approach fails to utilize the strong temporal continuity between adjacent frames, leading to a large amount of redundant computation: the scene content between adjacent frames highly overlaps, and most instances and relations detected and recognized in the previous frame still exist in the subsequent frame with only minor changes, yet the full-scale reconstruction method re-executes the complete perception process for them. Furthermore, the method of this invention allows the processing of each frame to directly inherit the structured knowledge accumulated in the previous frame, only locally updating changed node attributes, node existence, and relation edges, without re-executing full perception and graph reconstruction in each frame. In real dynamic scenes, the instances and relations that change between adjacent frames typically only account for a small portion of the overall scene. Taking a frame rate of 30 fps as an example, the time interval between adjacent frames is only about 33 ms. Within this extremely short time window, the motion and state changes of instances in the physical world are very limited. Most instances remain in similar positions and the same state, and most relationships are maintained. The method of this embodiment only processes these changing parts, and the static parts do not need to be recalculated, thereby controlling the single-frame processing latency to the millisecond level. Specifically, for each video frame in the received video frame sequence, the following processing steps are performed sequentially: S1, obtain the dynamic scene map obtained by processing the previous video frame as the current dynamic scene map, wherein when the video frame is the first frame of the video frame sequence, the current dynamic scene map is empty.

[0025] In practice, the video frame sequence is captured in real time by an image acquisition device associated with the embodied agent and input into the processing pipeline. This image acquisition device can be a head camera of a robot, a vision sensor on the end effector of an industrial robotic arm, or a camera included in the perception module of an autonomous vehicle. The video frames arrive continuously at a fixed frame rate, for example, 30 frames per second, and each frame is an RGB three-channel color image. The image acquisition device is rigidly connected to the embodied agent in physical space, and its spatial pose changes with the movement of the agent.

[0026] Furthermore, the updated dynamic scene graph obtained from the processing of the previous video frame is acquired as the current dynamic scene graph. This step is a crucial step in establishing an inter-frame iterative closed loop. Upon initiation of processing, it is first determined whether the current video frame is the first frame of the video frame sequence. If it is the first frame, the current dynamic scene graph is empty, meaning it does not contain any nodes or edges. Furthermore, if it is not the first frame, the current dynamic scene graph is the updated dynamic scene graph output after the previous video frame has undergone the complete processing flow of this embodiment. This acquisition step ensures that the processing of each frame is based on the scene understanding already constructed in the previous frame, rather than analyzing the entire scene from scratch. The dynamic scene graph is stored in an in-memory graph database. Further, an in-memory graph database refers to a database that completely loads and manages the node table, edge table, and related indexes of the graph data in the system's main memory. Its data organization structure typically adopts an adjacency list or adjacency matrix. The adjacency list maintains an outgoing edge list and an incoming edge list for each node, supporting fast traversal of the node's neighborhood; the adjacency matrix stores the relationship types for all node pairs, facilitating rapid determination of relationship existence. Furthermore, compared to disk-based graph databases, in-memory graph databases reduce read / write latency from milliseconds to microseconds, providing underlying storage guarantees to meet the real-time performance requirement of incremental update latency below 100 ms. Additionally, each node record in the graph database contains the following fields: a unique node identifier, assigned by the system upon node creation and remaining unchanged throughout its lifecycle; a node category label, indicating the semantic type of the instance represented by the node; a set of node attributes, including at least spatial location and status, and potentially auxiliary information such as appearance feature vectors; a node creation timestamp, recording the system time when the node was first created; a node last update timestamp, recording the system time when the node attributes were most recently updated; and a node confidence score, characterizing the reliability of the instance represented by the node in the current scenario. Furthermore, each directed edge record contains the following fields: a unique edge identifier; an edge type label indicating the semantic type of the relation represented by the directed edge; source node identifier and target node identifier, pointing to the subject and object of the relation, respectively; edge confidence, used to characterize the reliability of the relation represented by the directed edge in the physical world; edge creation timestamp; and edge last update timestamp. Furthermore, both node confidence and edge confidence are represented by real values ​​between 0 and 1, with higher values ​​indicating greater certainty of existence. These two types of confidence will be dynamically adjusted in subsequent processing steps as observational evidence accumulates.

[0027] S2, perform multimodal perception processing on the video frame to obtain a set of instance nodes, attribute information of each instance node, and relationship information between nodes.

[0028] In practice, multimodal perception processing is performed on video frames to obtain a set of instance nodes, attribute information of each instance node, and relationship information between nodes. Multimodal perception processing is performed by three collaborative neural network models that share a backbone feature extraction network to save computational resources. Furthermore, the shared backbone feature extraction network can be a convolutional neural network pre-trained on a large-scale image classification dataset, such as a residual network (ResNet), which performs layer-by-layer convolution and pooling operations on the input video frames, outputting multi-scale shared feature maps. These shared feature maps are then fed into the object detection head, attribute recognition head, and relationship detection head, respectively, working in parallel to improve processing efficiency.

[0029] Furthermore, the object detection head generates a large number of candidate regions based on shared feature maps. These candidate regions, based on anchor boxes or feature points, cover potential targets at various locations and scales in the image. For each candidate region, the object detection head performs two parallel predictions: foreground / background classification, determining the probability that the candidate region contains a complete instance; and bounding box coordinate regression, refining the position and size of the candidate region to accurately enclose the instance. Highly overlapping duplicate detections are removed using non-maximum suppression (NMS), ultimately outputting a set of detected instances along with their bounding boxes and class labels. The class label indicates the semantic type of the instance, such as person, tool, table, door, robotic arm end effector, etc. Each instance in the detection result is assigned a temporary instance identifier valid only within the current frame for subsequent association steps.

[0030] Furthermore, for each detected instance, the attribute recognition head extracts Region of Interest (RoI) features from the shared feature map corresponding to the instance's bounding box. RoI feature extraction can be achieved through RoI pooling or RoI alignment operations, mapping bounding box regions of different sizes to a fixed-size feature map. Then, a multi-branch fully connected network is used to predict multiple attributes of the instance in parallel. Position attributes are represented by the two-dimensional coordinates of the instance's bounding box center point in the image coordinate system. If the system is equipped with a depth sensor, the depth value at the corresponding location in the depth map is combined, and the two-dimensional image coordinates are converted to three-dimensional spatial coordinates using camera intrinsics and a pinhole model inverse operation. State attributes have different sets of state values ​​depending on the instance category: for tool instances, the state values ​​are "idle," "picked up," and "in use"; for door instances, the state values ​​are "closed," "half-open," and "fully open"; for personnel instances, the state values ​​are behavioral postures such as "standing," "walking," and "bending over." The prediction of state attributes is achieved through a classifier that takes RoI features as input and outputs the probability distribution of each state in the set of state values. The state prediction result is taken as the one with the highest probability.

[0031] Furthermore, the relation detection head examines every ordered pair of instance nodes in the instance node set output by the target detection head and determines whether a predefined relation type exists between them. Ordered pairs are used instead of unordered pairs because many relations are directional; for example, "A is holding B" and "B is holding A" have different semantics. The relation detection head extracts joint region features from the shared feature map of two instances. These joint region features are constructed as follows: the union of the bounding boxes of the two instances is used as the region of interest (ROI); RoI features are extracted from the shared feature map; and the spatial coordinates and class labels of the two instances are encoded into vectors and concatenated with the RoI features. The fused features are fed into a relation classifier, which outputs the probability of various relation types existing between the ordered pairs. The predefined relation types are divided into three categories: spatial relations, describing the relative spatial position between instances, including "on the left," "on the right," "above," "below," "in front," and "behind"; physical relations, describing physical contact or mechanical interaction between instances, including "holding," "touching," "supporting," and "placing on"; and semantic relations, describing semantic interaction between instances, including "observing," "pointing," and "talking." All outputs with probabilities higher than a preset relationship detection threshold (e.g., 0.5) are retained, forming the relationship information between nodes. Each relationship record contains the following fields: relationship type, temporary identifier of the subject instance, temporary identifier of the object instance, and relationship detection confidence level.

[0032] Furthermore, after multimodal perception processing, three distinct data objects are obtained: a set of instance nodes, containing temporary identifiers and bounding box coordinates of all instances detected in this frame; attribute information of each instance node, including the category, location, and status of each instance node; and relationship information between nodes, including all relationships detected in this frame and the identifiers of the two connected instance nodes, the relationship type, and the relationship detection confidence. These three data objects will be used together in subsequent association and update steps.

[0033] S3, based on the set of instance nodes and attribute information, associate the instance nodes with existing nodes in the current dynamic scene graph, update the current dynamic scene graph according to the association results, obtain the updated dynamic scene graph, and use the updated dynamic scene graph as the current dynamic scene graph corresponding to the next video frame; wherein, the step of updating the current dynamic scene graph according to the association results includes: for successfully associated instance nodes, updating the attributes of the corresponding existing nodes in the current dynamic scene graph; for instance nodes not associated with existing nodes, creating new nodes in the current dynamic scene graph; for existing nodes not associated with any instance nodes in the video frame, deleting the existing nodes from the current dynamic scene graph; and updating the directed edges between nodes in the current dynamic scene graph based on the relationship information between nodes.

[0034] In practice, based on the set of instance nodes and their attribute information, instance nodes are associated with existing nodes in the current dynamic scene graph. The core purpose of association is to establish the identity correspondence between the instances detected in this frame and the continuously tracked physical entities in the graph, thereby determining three key pieces of information for this frame: which instances are previously existing physical entities (associated successfully), which are newly appearing physical entities (not associated with existing nodes), and which existing physical entities disappeared in this frame (existing nodes did not match any instances). The association problem is essentially a data association problem, which is usually solved by Multiple Object Tracking (MOT) technology in the field of computer vision. Its basic processing is as follows: for each existing node in the current dynamic scene graph, maintain its appearance feature queue and motion trajectory for the most recent few frames; for each instance node in the current frame, extract its appearance features and calculate its motion information relative to the previous frame; then calculate an association cost matrix, where the rows of the matrix correspond to instance nodes and the columns correspond to existing nodes, and each matrix element represents the matching confidence that the instance node and the existing node belong to the same physical instance; finally, solve the global optimal matching using the Hungarian Algorithm. The Hungarian algorithm is a classic bipartite graph matching algorithm with polynomial time complexity. Its core idea is to maximize the total weight of the matching by finding augmenting paths, which can maximize the overall association confidence while ensuring the uniqueness of the matching.

[0035] Furthermore, the association results categorize instance nodes and existing nodes into three mutually exclusive categories. The first category consists of successfully associated instance nodes, meaning those instances that found a matching existing node in the current frame; their corresponding existing nodes are considered successfully associated. The second category comprises instance nodes not associated with existing nodes, meaning newly detected instances in the current frame but without a corresponding entity in the current graph. These instances may be newly entered objects or instances that have reappeared after long-term occlusion but have changed too much in appearance to be associated with the original node. The third category consists of existing nodes that exist in the current dynamic scene graph but are not associated with any instance nodes in the current frame; these are physical entities that have disappeared from the view in the current frame, either by leaving the view or by being completely occluded.

[0036] Furthermore, the current dynamic scene map is updated based on the association results to obtain the updated dynamic scene map. The update operation performs four sub-steps for each of the three types of association results and relationship information mentioned above.

[0037] Sub-step 1: For successfully associated instance nodes, update the attributes of the corresponding existing nodes in the current dynamic scene graph. Write the detected attribute values ​​of the successfully associated instance nodes into the corresponding attribute fields of the associated existing nodes, replacing the old values. Update the position attribute to reflect the spatial movement of the physical entity, and update the state attribute to reflect changes in the physical entity's operational state or behavioral posture. At the same time, update the most recent update timestamp of the existing node to the timestamp of the current frame, and raise the node confidence to a high level (e.g., 1.0), because the successful association in this frame provides the strongest evidence that the physical instance represented by the node still exists and is visible.

[0038] Further, sub-step two: For instance nodes not associated with existing nodes, create new nodes in the current dynamic scene graph. Assign a globally unique node identifier to the new node, write its instance node's category, location, status, and other attributes, set the node creation timestamp and last update timestamp to the current frame's timestamp, and assign an initial node confidence level (e.g., 0.5). The initial confidence level is set to a medium level rather than the highest value because isolated detection in a single frame may result in false detections. The new node needs continuous successful association in several subsequent frames to accumulate evidence and increase its confidence. The new node becomes an existing node starting from the next frame and participates in the association processing of subsequent frames.

[0039] Further, sub-step three: For existing nodes that are not associated with any instance nodes in the video frame, delete the existing nodes from the current dynamic scene graph. When an existing node is not associated with any instance node in this frame, it indicates that the physical instance represented by the node may have left the field of view or been completely occluded. The deletion operation removes the node from the node table of the graph database and cascades the deletion of all directed edges with that node as the source or target node, to prevent a large number of non-existent entities and their relationships from remaining in the graph, thus maintaining the timeliness and compactness of the graph.

[0040] Further, sub-step four: Based on the relationship information between nodes, update the directed edges between nodes in the current dynamic scene graph. Match each relationship detected in this frame with the existing directed edges in the current graph, based on the relationship type being the same and the two connected nodes being the same. For successfully matched directed edges, increase their edge confidence; for existing directed edges that are not matched in the relationship information of this frame, decrease their edge confidence; for newly detected relationships in this frame that do not yet exist in the graph, create new directed edges and set initial confidence; for existing directed edges whose edge confidence decays to below a preset extinction threshold, perform a deletion operation.

[0041] Furthermore, after completing all the above update sub-steps, an updated dynamic scene map is obtained, which accurately reflects the physical scene state at the current video frame. This updated dynamic scene map is used as the current dynamic scene map for the next video frame, i.e., as the input to the first step of the next frame processing pipeline, forming an inter-frame iterative closed loop. This closed loop is the structural foundation for the entire system to achieve incremental updates.

[0042] This invention establishes an inter-frame iterative closed loop by acquiring the previous frame's graph as the starting point for processing the current frame, ensuring that each frame's processing is based on historical understanding. The incremental update strategy reduces the computational load of each frame from full perception and full mapping to processing only the changed parts, keeping single-frame processing latency below 100 ms. In contrast, the frame-by-frame full reconstruction scheme requires each frame to independently run a complete object detection, attribute recognition, and relation extraction process, with processing time proportional to the number of instances and relations in the scene, failing to guarantee real-time performance. Furthermore, the dynamic scene graph, as an intermediate representation layer connecting low-level visual perception and high-level symbolic reasoning, provides embodied agents with a structured interface for querying the identity, location, state, and interaction relationships of instances in the physical world. It supports advanced queries such as "finding all currently being operated tools" or "determining who is next to the table," serving as a key cognitive infrastructure for realizing intelligent behavior in the physical world.

[0043] In some preferred embodiments, associating the instance node with existing nodes in the current dynamic scene graph includes: extracting the appearance features of the instance node and the historical appearance features of the existing node; predicting the estimated location region of the existing node in the video frame based on the motion information of the instance node and the historical motion trajectory of the existing node; calculating the appearance similarity between the instance node and the existing node in the feature space, and calculating the spatial overlap between the actual location of the instance node and the estimated location region; weightedly fusing the appearance similarity and the spatial overlap to obtain an association confidence score; and determining that the instance node and the existing node pair with the association confidence score higher than a preset association threshold are successfully associated.

[0044] In practical implementation, accurately associating instances detected in the current frame with historically tracked nodes in dynamic scenes is the prerequisite and foundation for incremental updates. Association errors can lead to serious consequences: identity switching refers to mistaking two different physical instances for the same entity and merging their trajectories, resulting in a chaotic attribute evolution sequence recorded in the graph; identity fragmentation refers to mistaking the same physical instance for different entities in different frames, resulting in a large number of fragmented, short-lived nodes in the graph. Both of these errors severely damage the consistency and reliability of the graph, causing upper-level decisions to be based on incorrect scene understanding. This invention addresses this by explicitly calculating two heterogeneous features—appearance similarity and spatial overlap—and weightedly fusing them to construct a multi-clue comprehensive evaluation mechanism. When a single information source is unreliable due to specific interference factors, another information source provides complementary evidence, thereby significantly improving the association accuracy under difficult conditions such as occlusion, lighting changes, and rapid movement. The specific implementation method is as follows: The appearance features of instance nodes and historical appearance features of existing nodes are extracted. For each instance node, the corresponding image region is cropped from the video frame based on its bounding box. This image region is scaled to a fixed size (e.g., 224×224 pixels) and then input into a pre-trained convolutional neural network as an appearance feature extractor. This appearance feature extractor is pre-trained on a general object re-identification task. The network structure can adopt a backbone network specifically designed for re-identification tasks, enabling it to learn appearance discrimination features robust to changes in illumination, viewpoint, and partial occlusion. Furthermore, the network outputs a fixed-length feature vector (e.g., 128-dimensional or 256-dimensional), which is then L2 normalized and used as the appearance feature of the instance node. L2 normalization scales the feature vector's magnitude to 1, ensuring that subsequent similarity calculations depend only on the vector direction rather than its length, eliminating comparison bias caused by differences in the magnitudes of different feature vectors. Furthermore, the feature vectors of different instances are far apart in the high-dimensional feature space, while the feature vectors of instances of the same instance are close together. This property is the theoretical basis for the feasibility of appearance association.

[0045] Furthermore, for each existing node in the current dynamic scene graph, its historical appearance features are stored in the node's attributes. These historical appearance features can be the exponential moving average (EMA) of the appearance feature vectors of the instances associated with that node in the most recent frames. The recursive operation of the exponential moving average is as follows: the appearance features of the current frame are weighted and mixed with the old moving average in a certain proportion to obtain a new moving average. A typical value for the moving average coefficient α is 0.7. The larger the α value, the more the moving average focuses on recent observations, and the stronger its adaptability to gradual changes in the target appearance (such as brightness and texture gradations caused by slow changes in illumination angle); the smaller the α value, the smoother the moving average but the slower the response. If an existing node failed to be associated with an instance in a previous frame due to occlusion or low detection confidence, its appearance feature moving average is not updated in that frame, maintaining the previous valid value to prevent appearance template degradation due to invalid feature updates.

[0046] Furthermore, based on the motion information of instance nodes and the historical motion trajectories of existing nodes, the estimated location region of existing nodes in the current video frame is predicted. Motion information is obtained by comparing the position of the instance node in the current frame with the positions of associated nodes in previous frames to estimate the motion velocity and direction of the instance. The historical motion trajectory refers to the spatial position sequence of existing nodes in the most recent frames (e.g., the most recent 10 frames), which constitutes a three-dimensional or two-dimensional trajectory curve. Further, based on this trajectory curve, a Kalman filter is used for state estimation and prediction. The Kalman filter is a recursive Bayesian estimation algorithm widely used in navigation, tracking, and control. Its basic idea is to model the state evolution of the physical system as a linear dynamic system, where the system state vector typically includes position and velocity components; the filter alternately performs prediction and update steps at each time step. Further, the prediction step predicts the current state and the uncertainty of the state estimate (represented by the covariance matrix) based on the system state estimate and system motion model from the previous time step. Further, the output of the prediction step is an ellipse of predicted state and prediction error, which represents the estimated location region. The size of the estimated location region is proportional to the magnitude of motion uncertainty and prediction covariance: when the target's trajectory maintains uniform linear motion over a long period, the prediction uncertainty is small, and the estimated location region is compact; when the target experiences a sudden turn or acceleration / deceleration recently, the prediction uncertainty increases, and the estimated location region expands accordingly. In the association step, only the output of the Kalman filter prediction step is used. If the association is successful subsequently, an update step is performed using the observed instance node positions to correct the filter state.

[0047] Furthermore, the appearance similarity between the instance node and existing nodes in the feature space is calculated. Cosine similarity is used as the metric for appearance similarity. Cosine similarity is calculated by dividing the inner product of two vectors by the product of their respective magnitudes. For feature vectors after L2 normalization, since the magnitude is always 1, cosine similarity degenerates into a vector inner product. A higher value indicates that the two vectors are more aligned in direction in the feature space, meaning the two instances are more similar in appearance. Cosine similarity is insensitive to the overall scaling of each feature dimension. This characteristic makes it robust to uniform illumination changes, as uniform illumination changes tend to produce an approximately proportional scaling effect on all feature dimensions. An appearance similarity value of 1 indicates that the two appearance features are completely identical, while a value of 0 indicates that they are orthogonal and have no similarity.

[0048] Further, the spatial overlap between the actual location of the instance node and the estimated location region is calculated. Spatial overlap measures the degree of consistency between the observed location of the instance node and the location predicted by the motion model. One implementation is to calculate the Intersection over Union (IoU) ratio between the instance node's bounding box and the estimated location region. The IoU is defined as the area of ​​the intersection of the two regions divided by the area of ​​their union, ranging from 0 to 1. When the center point of the instance node falls within the estimated location region and the shape of the bounding box matches the estimated region well, the IoU is close to 1; when the two are completely separated in space with no intersection, the IoU is 0. The IoU considers the matching degree of both location and scale dimensions, and can reflect spatial overlap more comprehensively than simply calculating the center point distance. Another option is to calculate the negative exponent of the Mahalanobis distance. The Mahalanobis distance considers the weights of uncertainty in different directions described by the prediction error covariance matrix: in directions with low prediction uncertainty, even small positional deviations will produce large Mahalanobis distances; in directions with high prediction uncertainty, large positional deviations will produce small Mahalanobis distances. The negative exponent of Mahalanobis distance normalizes the spatial overlap to the range of 0 to 1.

[0049] Furthermore, the appearance similarity and spatial overlap are weighted and fused to obtain the association confidence score. The weighted fusion uses a linear weighting method: Association Confidence Score = α × Appearance Similarity + (1-α) × Spatial Overlap, where α is the fusion weight, ranging from 0 to 1. The value of the weight α can be calibrated offline or adaptively adjusted online according to the characteristics of the application scenario: In indoor scenarios with stable lighting conditions and significantly distinguishable target appearances (such as robots operating tools of different colors and shapes), α can be set to a higher value, such as 0.7, to fully leverage the strong discriminative power of appearance features; in fast-moving scenarios or scenarios with a large number of similar-looking targets (such as multiple industrial robots of the same model or uniformly dressed personnel), α can be reduced to 0.3, making the association decision rely more on the spatial constraints predicted by the motion model, because appearance lacks discriminative power in such scenarios. The fused association confidence score integrates the judgments from both appearance and motion heterogeneous information, exhibiting stronger anti-interference capabilities and higher reliability compared to using either feature alone.

[0050] Furthermore, instance nodes with association confidence scores higher than a preset association threshold and existing node pairs are identified as candidate association pairs. The preset association threshold is typically set to 0.6; values ​​below this threshold indicate insufficient matching evidence, suggesting the two nodes are unlikely to belong to the same physical instance. This preset association threshold can be determined through testing on a validation set to achieve a balance between association recall and association precision. In the bipartite graph formed by all candidate association pairs, there may be cases where the association confidence scores between an instance node and multiple existing nodes all exceed the threshold, indicating a matching conflict. In this case, the Hungarian algorithm is used to solve for the globally optimal match. This algorithm finds the globally optimal matching scheme that minimizes the total matching cost or maximizes the total association confidence by searching for augmenting paths on the association cost matrix in polynomial time complexity, ensuring that each instance node is associated with at most one existing node, and each existing node is also associated with at most one instance node. After the Hungarian algorithm solves the problem, the successfully matched instance node and existing node pairs are the final successfully associated pairs. Instance nodes that are not matched by the Hungarian algorithm are classified into the category of instance nodes that are not associated with existing nodes, and existing nodes that are not matched are classified into the category of existing nodes that disappeared in the current frame.

[0051] Through the multi-clue weighted fusion association strategy of this invention, the node association process no longer relies on a single information source. When the appearance degrades due to drastic changes in lighting or partial occlusion, the spatial constraints provided by motion prediction can still maintain correct association; when the motion model makes inaccurate predictions due to sudden changes in target direction or speed, appearance matching can serve as a reliable backup clue. This multi-clue complementary mechanism makes the entire incremental update pipeline highly robust in complex dynamic environments, providing a stable and reliable node identity correspondence for subsequent attribute and relationship updates.

[0052] In some preferred embodiments, updating the attributes of corresponding existing nodes in the current dynamic scene graph for successfully associated instance nodes includes: determining the attribute type of the attribute; if the attribute type is a continuously changing first type attribute, then updating the corresponding attribute value of the existing node by smoothing filtering based on the attribute value of the successfully associated instance node; if the attribute type is a discretely changing second type attribute, then obtaining the state sequence of the existing node in multiple consecutive video frames including the video frame, and determining the stability of the state sequence; when the determination result meets a preset stability condition, updating the second type attribute of the existing node to the attribute value of the successfully associated instance node.

[0053] In practice, the changing patterns of different attributes differ fundamentally in the real physical world. Spatial position is an analog quantity that changes continuously in both time and space, with its value smoothly transitioning between adjacent frames; the difference in position between adjacent frames is a tiny increment. Operational state and interaction state, on the other hand, are discretely changing quantities, whose values ​​should jump from one definite state to another, essentially without intermediate transition values. For example, a tool's state will not have an intermediate state between "idle" and "picked up." If a uniform update strategy is used without distinguishing between these two types, two problems will arise: instant overlay of position may cause nodes to randomly jump in the graph due to perceived noise; instant overlay of state may cause states to frequently switch due to misjudgment in a single frame (resulting in state flickering). Both phenomena will severely impair the usability of the graph in downstream tasks. This invention distinguishes attribute types and employs a differentiated strategy of smooth filtering to update the first type of attribute (continuously changing attributes) and stability judgment to update the second type of attribute (discretely changing attributes), making the evolution of node states in the graph both smooth and reliable. The specific implementation is as follows: Determine the attribute types. Attribute types are pre-classified during the system design phase based on the physical semantics of each attribute. The first type of attribute is continuously changing, with values ​​belonging to a continuous numerical space (such as real-valued coordinate space). It is expected to evolve smoothly over time, with changes between adjacent frames constrained by physical laws. Typical first-type attributes include: the spatial coordinates of the instance's center in the 3D world coordinate system (in the physical world, an instance cannot instantly jump from one position to another); the instance's velocity vector (velocity changes are constrained by inertia and forces); and the instance's orientation angle (or rotation of orientation is a continuous process). The second type of attribute is discretely changing, with values ​​belonging to a predefined finite set of states. Over time, it manifests as transitions from one discrete state to another, with clear event boundaries between these transitions. Typical second-type attributes include: the "idle," "picked up," and "in use" states of tools; the "closed," "half-open," and "fully open" states of doors; and the behavioral postures of people such as "standing," "walking," "bending over," and "raising hands." The classification of attribute types can be customized and expanded according to specific application scenarios.

[0054] Furthermore, when the attribute type is the first type, the corresponding attribute values ​​of existing nodes are updated using smoothing filtering based on the attribute values ​​of successfully associated instance nodes. The core design principle of smoothing filtering is: not to completely trust the perception results of the current single frame, but to weight and fuse them with historical estimates, thereby suppressing random jitter caused by perception noise while maintaining the ability to track the real physical motion trend. One implementation method is to use an exponential moving average filter. The recursive operation of the exponential moving average is: to weight and mix the attribute values ​​of the instance nodes observed in the current frame with the attribute values ​​of existing nodes before the update according to a certain ratio to obtain the updated attribute values. The value of the smoothing coefficient β reflects the trade-off between filtering strength and response speed: the smaller the β value, the greater the filtering strength and the smoother the output, but the more lagging the response to real changes; the larger the β value, the closer the filter follows the observation, but the weaker the noise suppression ability. For spatial location attributes, under the condition of a frame rate of 30fps, the typical range of β values ​​is 0.2 to 0.4. Taking β=0.3 as an example, the filter takes about 3 frames (about 0.1 s) to reach a steady state in response to instantaneous changes. This ensures smooth trajectory while keeping the response delay to normal target movement (such as human walking or robotic arm movement) within an acceptable range for the control system. Smoothing filter updates can also use a Kalman filter as an alternative. The Kalman gain of the Kalman filter is adaptively adjusted based on the ratio of process noise variance to observation noise variance. When the observation noise is large (such as the uncertain output of the detection model under low light conditions), the gain automatically decreases. The filter trusts the motion model's predictions more than the current observations, exhibiting better environmental adaptability compared to an exponential moving average with a fixed β. Regardless of the filter used, the common effect is that when the perception model generates position detection noise due to changes in lighting, motion blur, or partial occlusion, the smoothed position trajectory remains stable between frames, and nodes do not exhibit "jumping" phenomena in the graph, which is beneficial for the upper-level motion planning module to accurately predict object trajectories.

[0055] Furthermore, when the attribute type is the second type, the state sequence of the existing node across multiple consecutive video frames, including the current video frame, is obtained. The state sequence is a list of discrete state labels arranged in temporal order, where each element represents the state detection result of an instance node associated with the existing node in a past frame. For example, for a tool node, its state sequence of the last 5 frames might be "idle, idle, idle, picked up, picked up," recording a state transition from "idle" to "picked up"; it might also be "picked up, picked up, idle, idle, idle," recording a transition from "picked up" to "idle"; or it might be "picked up, picked up, idle, picked up, picked up," with a brief false positive of "idle." Stability is determined for this state sequence. The core objective of this determination is to confirm whether the new state observed in the current frame reflects a stable state reflecting real physical interaction changes, or whether it is an occasional single-frame false positive from the perception model. One stability determination rule is to count the number of consecutive occurrences of the current observed state in the state sequence of the last N frames. N is the observation window length, typically 5 frames. If the number of consecutive occurrences of the current observed state reaches a preset consecutive observation threshold K (e.g., K=3), then the preset stability condition is met, meaning the new state has sufficient multi-frame evidence to support it. If the number of consecutive occurrences does not reach K, then the preset stability condition is not yet met, the evidence is insufficient, and more frames are needed. Another optional determination method is to calculate the proportion of each state in the state sequence. If the proportion of a certain state exceeds a preset proportion threshold (e.g., 80%), then the state is determined to be stable.

[0056] Furthermore, when the judgment result meets the preset stability condition, the second type of attribute of the existing node is updated to the attribute value of the successfully associated instance node, thus formally switching the state to the new state. When the judgment result does not meet the preset stability condition, the current attribute value of the existing node remains unchanged, that is, the original state is maintained, and observation and evidence accumulation continue. This "lagging switch" mechanism sets up an evidence accumulation buffer for discrete state updates: state changes are not triggered by single-frame detection results; only stable observations over multiple frames can drive state switching. For example, the relationship detection model may occasionally misjudge the "picked up" state as "idle," but since this misjudgment only occurs in isolation for 1 to 2 frames in the state sequence, the number of consecutive occurrences of the "idle" state does not reach the threshold of K=3, and the tool state in the graph remains "picked up," avoiding state flashover. Only when the person actually puts down the tool and it is stably detected as "idle" for more than 3 consecutive frames will the graph state smoothly switch to "idle." Similarly, the state update from "idle" to "picked up" also requires more than 3 consecutive frames of "picked up" observation evidence to ensure that the picking up action is actually completed and not a mistake.

[0057] By employing a differentiated update strategy that distinguishes attribute types, this invention introduces smoothing filtering in the evolution of physically continuous attributes. This effectively suppresses jitter caused by perceptual noise while ensuring the accuracy of the trajectory's response to real motion. Furthermore, a delayed evidence accumulation mechanism is introduced in the switching of semantically discrete states. This effectively filters out random jumps caused by single-frame misjudgments while ensuring the timeliness of state responses to real interactive events. This design enables the dynamic scene map to provide high-quality structured data support for both fine-grained manipulation (requiring precise and stable position information) and high-level task planning (requiring accurate and reliable state information).

[0058] In some preferred embodiments, updating the directed edges between nodes in the current dynamic scene graph based on the relationship information between the nodes includes: determining whether the target relationship in the relationship information already exists in the current dynamic scene graph; if it does not exist, creating a directed edge connecting two successfully associated instance nodes and setting an initial confidence level for the directed edge; if it already exists, determining that the target relationship is in a continuously observed state in the video frame and increasing the confidence level of the directed edge; decreasing the confidence level of existing directed edges in the current dynamic scene graph that have not been observed again in the video frame; and deleting directed edges with a confidence level lower than a preset extinction threshold from the current dynamic scene graph.

[0059] In practice, in the real physical world, relationships between instances have a dynamic lifecycle from their creation to their persistence. When a person picks up a tool, the "holding" relationship is established; throughout the operation, the "holding" relationship persists and may move synchronously with the spatial position of the tool; when the person finally puts down the tool, the "holding" relationship terminates and disappears. Relationship detection models independently determine relationships for each frame of the image. Due to limitations in two-dimensional image information, perspective changes, motion blur, occlusion, and the model's own generalization ability, two types of errors may occur: false positives (also known as false detections or false positives), where the relationship does not exist in the physical world but is detected as existing by the model; and temporary false negatives (also known as temporary missed detections), where the relationship does exist in the physical world but is not successfully detected by the model in a particular frame. If the relationship detection result of a single frame is directly equated with the existence or non-existence of the relationship in the physical world, directed edges in the graph will be frequently created and deleted due to these two types of detection errors, resulting in a highly unstable output structure that cannot provide consistent and reliable relationship information for downstream tasks. This invention simulates the lifecycle management of relationships in the physical world by maintaining a continuously changing confidence variable for each directed edge. This mechanism employs a two-way dynamic adjustment mechanism where confidence increases with continuous observation and decreases with unobserved events, ensuring that the creation, maintenance, and termination of directed edges are smooth and consistent with real interactions in the physical world. The specific implementation is as follows: The system determines whether the target relationship in the relation information already exists in the current dynamic scene graph. The target relationship is uniquely identified by three elements: relationship type (e.g., "holding"), the existing node identifier in the graph corresponding to the relationship subject, and the existing node identifier in the graph corresponding to the relationship object. Both the subject and object are existing nodes that have been successfully associated, as determined by the association step. In this embodiment, a matching search is performed on the existing set of directed edges in the graph. If an existing directed edge is found whose edge type label is the same as the relationship type of the target relationship, and whose source node identifier is equal to the subject node identifier of the target relationship and whose target node identifier is equal to the object node identifier of the target relationship, then the target relationship is determined to exist in the current graph. If no existing directed edge satisfying the above three elements is found, the target relationship is determined to be a newly detected relationship in the current frame, and a corresponding directed edge does not yet exist in the graph.

[0060] Furthermore, if the target relation does not exist in the current graph, a directed edge is created connecting the two successfully associated instance nodes, and an initial confidence level is set for this directed edge. The initial confidence level is set to a moderately low value, typically 0.5. This initial value reflects the design principle of "doubt before belief": the relation is marked as "pending confirmation," requiring it to be continuously observed again in subsequent frames to accumulate evidence and increase the confidence level before it reaches the credible range. Further, if the relation does indeed exist in the physical world, it should be continuously detected in subsequent frames, and the directed edge confidence level will gradually increase with each re-observation, reaching a high confidence level close to 1.0 within a few frames. If the relation is merely a false detection result in an isolated single frame, it is highly unlikely to reappear in subsequent frames, and its confidence level will rapidly decrease from the initial value of 0.5 according to the unobserved decay rule, falling below the extinction threshold within 2 to 3 frames and being automatically removed. This "always be suspicious of new edges" mechanism effectively filters out scattered false detections during the perception stage, preventing them from remaining in the map for a long time.

[0061] Furthermore, if the target relation already exists in the current graph, it is confirmed that the target relation has been successfully detected again in the current video frame and is in a continuous observation state. At this time, the confidence of the existing directed edge is increased. The confidence is increased incrementally, with a fixed increment of 0.1 each time, and the upper limit of the confidence is 1.0. When a relation is continuously and stably detected, its directed edge confidence will start from 0.5, and successively pass through 0.6, 0.7, 0.8, 0.9 until 1.0. At a frame rate of 30 fps, if the relation is continuously detected, it takes about 5 frames (about 0.17 s) from creation to reaching the highest confidence. When the confidence reaches the upper limit of 1.0, it means that the relation has been fully verified and has extremely high determinism. At this time, even if it is not detected in one or two frames due to accidental factors (such as brief occlusion or motion blur), since it takes multiple frames for the confidence to decay from 1.0 to the extinction threshold, the directed edge will not be immediately deleted due to brief missed detection, thus providing a fault-tolerant buffer window for continuous tracking of the relation. The mechanism of continuous observation to improve confidence ensures that real relationships can quickly transition from "pending confirmation" to "credible" status, providing high-confidence relationship information to higher-level tasks in a timely manner.

[0062] Furthermore, for existing directed edges in the current dynamic scene graph that are not observed again in the current video frame, the confidence level of these edges is reduced. The criterion for not being observed again is: no target relationship of the same type as the existing directed edge and connecting the same source and target nodes is found in the relationship information of this frame. The magnitude of the confidence reduction is designed to be greater than the magnitude of the increase, forming an asymmetric adjustment. For example, the confidence level is increased by 0.1 each time it is observed again, and decreased by 0.15 or 0.2 each time it is not observed again. This asymmetric design of "fast reduction and slow increase" conforms to the general principle of information reliability accumulation: establishing the conviction that a relationship exists requires multiple positive observations as evidence, while denying the existence of a relationship only requires a small amount of negative evidence. For false relationships created due to false detection that no longer appear, the relatively fast decay rate allows them to disappear quickly within 2 to 3 frames; for real relationships that are no longer detected due to the termination of physical interaction, the relatively fast decay rate also allows them to be removed from the graph within a reasonable delay (such as 5 to 7 frames, about 0.17 s to 0.23 s), thus reflecting the changes in the interaction state of the physical world in a timely manner.

[0063] Furthermore, directed edges with confidence levels below a preset extinction threshold are deleted from the current dynamic scene graph. The extinction threshold is typically set to 0.2, and this threshold is determined based on the application scenario's tolerance for expired relationship information; this invention does not specifically limit this threshold. When the confidence level of a directed edge gradually decreases from a high level to below 0.2 due to not being observed again for several consecutive frames, it can be determined that the relationship has most likely terminated in the physical world. Continuing to retain this edge would inject outdated information into the graph. At this point, the directed edge is deleted from the graph database. The edge deletion operation is a structured representation of the termination of physical interaction, accurately reflecting the physical fact that "at this moment, the interaction relationship no longer exists" at the graph level.

[0064] Furthermore, taking the complete interaction process of a person picking up a tool and then putting it down as an example, the complete evolution path of the directed edge confidence lifecycle is as follows. Before the person touches the tool, there is no "holding" relationship edge connecting the person and the tool in the graph. In the first few frames after the picking-up action, the relationship detection model begins to output the "holding" relationship. If detected in the first frame, a "holding" directed edge is created in the graph, with a confidence level of 0.5. If detected in the second frame, the confidence level increases to 0.6; in the third frame, it increases to 0.7, until it reaches 1.0 after five consecutive frames. Throughout the continuous operation of the tool, the relationship is successfully detected in every frame, and the edge confidence level remains at or near 1.0. Even if a few frames are missed due to rapid movement or brief occlusion, causing the confidence level to drop from 1.0 to 0.85 or 0.7, it is still far above the demise threshold of 0.2, and the edge will not be mistakenly deleted. When the person finally puts down the tool, the relationship detection model no longer outputs the "holding" relationship. As the directed edge remained unobserved for several consecutive frames, its confidence level decreased from 1.0 in increments of 0.15, successively dropping to 0.85, 0.70, 0.55, 0.40, and 0.25. In the 6th frame, it reached 0.1, falling below the demise threshold of 0.2, and the edge was deleted. The "holding" directed edge in the graph disappeared smoothly within approximately 6 frames (about 0.2 s) after the physical interaction ended, accurately reflecting the event of the person putting down the tool.

[0065] Through the directed edge confidence lifecycle management mechanism of this invention, the relation edges in the graph undergo a complete confidence gradient process from creation to destruction, eliminating edge oscillations caused by random fluctuations in single-frame detection results. The output relation structure exhibits high temporal stability and physical consistency. This is particularly crucial for embodied agents that rely on relation information for task planning, safety constraint determination, and operational decisions: stable relation information means continuously reliable decision-making basis, preventing dangerous misjudgments or omissions of relations due to perceptual noise.

[0066] In some preferred embodiments, deleting an existing node from the current dynamic scene graph if it is not associated with any instance node in the video frame includes: maintaining a survival timer for each existing node; starting the survival timer when the existing node fails to be associated with a corresponding instance node for the first time in the video frame; resetting the survival timer if the existing node re-associates with an instance node within a series of consecutive video frames before the survival timer reaches a preset extinction time threshold; and deleting the existing node and all directed edges connected to it from the current dynamic scene graph when the survival timer reaches the preset extinction time threshold.

[0067] In practical implementation, the temporary disappearance of physical instances from the field of view is a frequent phenomenon in real dynamic scenes. The causes include: occlusions in the scene (such as pillars, walls, other objects, or people) temporarily obscuring the target instance from a certain perspective, making it undetectable for several frames; the target instance temporarily moving out of the edge of the camera's field of view and then moving back; and rapid rotation or movement of the image acquisition device itself causing the target to temporarily leave the field of view. If a node is immediately deleted once it is not associated with an instance in a frame, these various temporary disappearances will lead to frequent breaks and reconstructions of node identity. The trajectory of a physical instance before and after passing through an occlusion will be divided into two or more unrelated node segments, and the graph will be filled with a large number of short-lived node fragments with a lifespan of only a few frames, resulting in an extremely unstable output structure. This embodiment of the invention introduces a survival timer mechanism for each existing node. Instead of immediately deleting a node after it disappears from the current frame, a countdown timer is started and an observation waiting window is set. Only when a node cannot be re-observed within the complete window period is a judgment made that the physical instance represented by the node has truly left the scene, and the deletion operation is performed. This delayed deletion mechanism, while ensuring the graph responds promptly to changes in the real-world scene (such as instances actually leaving the scene), effectively filters out false disappearance events caused by brief occlusions, maintaining the spatiotemporal continuity of node identities. The specific implementation is as follows: A liveness timer is maintained for each existing node. The liveness timer is a count variable bound to the node and stored as node metadata in the node record of the graph database. When a new node is first created, its liveness timer is initialized to an inactive state, with a count value of zero. The liveness timer in the inactive state does not automatically increment over time; it only enters the active timer state when the node is first lost, triggering the activation condition. A count value of zero and an inactive state indicate that the node is currently being successfully associated, and the corresponding physical instance is in a normal visible and traceable state.

[0068] Furthermore, when an existing node fails to be associated with a corresponding instance node in the current video frame for the first time, a survival timer is started. After the association step in each frame is completed, the system traverses all existing nodes in the current dynamic scene graph and checks the association status of each node in this frame. For existing nodes that do not match any instance node (i.e., nodes that disappeared in this frame), the current status of their survival timer is checked. If the survival timer was previously inactive, it indicates that the node was previously continuously visible, and this frame is the first lost frame. The system activates the survival timer and starts counting, with the count value changing from 0 to 1. The activation operation marks the node entering the "loss observation period". If the survival timer was previously active, it indicates that the node has lost several frames, and the system increments the survival timer count value by 1. The count value represents the number of consecutive frames that the node has lost.

[0069] Furthermore, within several consecutive video frames before the survival timer count reaches the preset extinction time threshold, if an existing node re-associates with an instance node, the survival timer is reset. The preset extinction time threshold is set according to the tolerance for brief disappearances in the application scenario, with a typical value of 5 frames (corresponding to a time window of approximately 0.17 seconds at 30 fps). This threshold is chosen based on the statistical patterns of brief occlusions in actual physical scenes: pedestrians are typically occluded by pillars for 1 to 3 frames, and robotic arms are typically occluded by themselves for 2 to 4 frames. A tolerance window of 5 frames can cover the vast majority of brief disappearance events. When the survival timer count is less than 5, the node is still considered to have only temporarily left the field of view and will not be deleted. Within this tolerance window, as long as the node successfully re-associates with an instance node in any frame (meaning the physical instance reappears in the field of view), the system immediately resets the survival timer count to zero and restores its state to inactive. The reset operation ensures that the node's identity is seamlessly continued after the brief occlusion ends, and all its historical attributes, associated edges, and other information are fully preserved. For downstream task modules that use graph information, the graph query is not affected at all during the entire occlusion period because the node identifiers remain unchanged and the attributes are updated smoothly.

[0070] Furthermore, if the survival timer count reaches a preset extinction time threshold and the existing node is still not associated with any instance node, the existing node and all directed edges connected to it are deleted from the current dynamic scene graph. When a node's survival timer count reaches 5 frames without successful re-association, the system infers that the physical instance represented by the node has, with a very high probability, truly left the effective observation area. Typical situations include: the instance leaving the room or building and leaving the fixed monitoring range of the camera; the instance being moved into another enclosed space (such as tools being placed in a cabinet); and the image acquisition device undergoing a permanent viewpoint switch. In this case, keeping the node in the graph would consume memory and computing resources, and the spatial location and status information it provides would be outdated and no longer reflect the true state of the current physical scene. The deletion operation removes the node from the node table of the graph database and cascades the deletion of all directed edges with that node as the source or target node. The deletion of the node and its associated edges can synchronously generate corresponding change log records, providing a complete record of the node's extinction time and reason for subsequent historical backtracking queries.

[0071] Furthermore, by introducing a survival timer mechanism, the node deletion behavior of the graph changes from "immediate response to single-frame association loss" to "delayed decision-making based on multi-frame continuous observation evidence." This design enables the graph to maintain the long-term stability of node identity and graph topology when faced with transient occlusion, which is common in real-world scenarios. The survival time threshold of 5 frames strikes a balance between response delay and stability: on the one hand, this threshold is long enough to effectively filter out most transient disappearances of 1 to 3 frames caused by ordinary occlusion; on the other hand, for instances that truly leave the scene, their deletion delay is controlled within 5 frames (approximately 0.17 s), with minimal impact on the timeliness of graph information.

[0072] In some preferred embodiments, the method further includes: during the process of updating the current dynamic scene graph based on the association results, generating a change log for each node and each directed edge in the current dynamic scene graph, the change log including change type, change time, and attribute values ​​or confidence levels before and after the change; sequentially appending the change log generated by each update to the operation log sequence bound to each node and each directed edge to form their respective historical evolution sequences; in response to receiving a backtracking query request carrying a target time, parsing the target time and determining the timestamp corresponding to the target time; in the process of updating the current dynamic scene graph based on the association results, generating a change log for each node and each directed edge in the current dynamic scene graph, the change log including change type, change time, and attribute values ​​or confidence levels before and after the change log; sequentially appending the change log generated by each update to the operation log sequence bound to each node and each directed edge to form their respective historical evolution sequences; in response to receiving a backtracking query request carrying a target time, parsing the target time and determining the timestamp corresponding to the target time; in the process of updating the current dynamic scene graph based on the association results, the method further includes: ... generating a change log for each node and each directed edge in the current dynamic scene graph, the change log including change type, change time, and attribute values ​​or confidence levels before and after the change log; in the process of updating the current dynamic scene graph based on the association results, generating a change log for each node and each directed edge in the current dynamic scene graph, generating a change log for each node and each directed edge in the current dynamic scene graph In the historical evolution sequence, starting from the end of the sequence and moving towards the beginning, the change time of each change log is compared with the timestamp to locate the first change log whose change time is no later than the timestamp, which serves as the restoration benchmark. Starting from the restoration benchmark, the historical evolution sequence is moved forward, and the reverse operations recorded in the change logs before and after the change are applied in reverse order until the graph state corresponding to the timestamp is traced back to, thus constructing a dynamic scene graph snapshot at the target time. The reverse operation is to restore the changed state to the state before the change. The dynamic scene graph snapshot at the target time is output as a structured query result.

[0073] In practical implementation, typical application scenarios of embodied intelligence require not only knowing what is happening in the physical world at the current moment, but also being able to trace back to the entire scene at a past moment. This need for historical backtracking is particularly prominent in the following scenarios: behavioral review, analyzing the execution process of an operation task and tracing the scene state when errors or failures occurred; accident cause analysis, querying the relative positions and interaction relationships of various instances in the scene several seconds before the accident occurred; and operation compliance review, confirming whether an operation was performed according to procedures at the specified time and location. For example, it is necessary to query who picked up a tool five minutes ago, where it was placed at the time, and who else was nearby. If the graph only stores the latest state and overwrites the historical state, such backtracking queries will be unresponsive due to the permanent loss of historical information. This embodiment of the invention generates a structured change log synchronously with each graph update, constructs a complete historical evolution sequence for each node and each directed edge, and provides a backtracking query method based on this sequence to restore the state by any target timestamp. This makes the dynamic scene graph a time-dimensional information infrastructure with both real-time environmental perception capabilities and full-time-domain historical traceability capabilities. The specific implementation method is as follows: During the process of updating the current dynamic scene graph based on the association results, a change log is generated for each node and each directed edge in the current dynamic scene graph. The change log is a persistent record created immediately whenever any state change occurs in the graph. Its data structure is a structured data object, containing at least the following fields: **Change Type Field:** Enumerates the operation category of this change, with values ​​including node creation, node attribute update, node deletion, directed edge creation, directed edge confidence change, and directed edge deletion. **Change Time Field:** Records the precise system timestamp of the change, obtained from the system's high-precision clock, with millisecond-level precision. **Pre-Change Value Field:** Records the complete state of the node attribute value or edge confidence value before the change occurred, in the form of a key-value pair set or a structured object; if the change type is a creation operation (node ​​creation or directed edge creation), the pre-change value field is empty or the record does not exist. **Post-Change Value Field:** Records the new state of the node attribute value or edge confidence value after the change occurred; if the change type is a deletion operation (node ​​deletion or directed edge deletion), the post-change value field is empty or the record does not exist. The associated object identifier field records the globally unique identifier of the changed node or directed edge, serving as the primary key for retrieving the change log by object. The change log generation operation is embedded within the graph update logic and executed synchronously with the update operation, ensuring that every graph state change is faithfully recorded without omission.

[0074] Furthermore, the change logs generated by each update are sequentially appended to the operation log sequences bound to each node and each directed edge, forming their respective historical evolution sequences. The operation log sequence is an ordered list data structure arranged chronologically, stored in order of change time from earliest to latest. Each node and each directed edge has its own independent operation log sequence, which records the first log entry starting from object creation. Subsequently, each change involving the object (attribute update, confidence change, deletion, etc.) is appended to the end of the sequence. Because the operation log sequence is strictly appended sequentially, its physical storage order naturally aligns with the chronological order. This ordered nature allows subsequent timestamp-based queries and location functions to be executed efficiently using the sequence's ordered characteristics. When a node is finally deleted, the last node deletion log entry is appended, and the node's operation log sequence is then sealed, with no new records appended. The entire sequence constitutes a complete lifecycle archive of the node from its birth to its demise.

[0075] Furthermore, in response to receiving a backtracking query request carrying a target time, the system parses the target time and determines its corresponding timestamp. The backtracking query request is initiated by an external query module or an upper-layer application. The target time can be represented as an absolute timestamp (e.g., "October 21, 2024, 14:30:05.250 milliseconds") or as an offset relative to the current time (e.g., "5 minutes ago" or "180 seconds ago"). The parsing module uniformly converts the target time into a system timestamp with the same format and baseline as the changed time field, using it as the query key for subsequent positioning operations. If the target time is later than or equal to the current time, the latest dynamic scene map is directly returned without historical reconstruction.

[0076] Furthermore, in the historical evolution sequence, starting from the end of the sequence and moving towards the beginning, the change time of each change log is compared with the target timestamp to locate the first change log whose change time is no later than the target timestamp, serving as the restoration baseline. Since the operation log sequence is arranged in ascending chronological order, with the latest log record at the end of the sequence, a backward scanning method from back to front allows the target to be located after only a few comparisons. The specific scanning logic is as follows: starting from the last log in the sequence, its change time field is read; if the change time is no later than (i.e., less than or equal to) the target timestamp, the scan stops, and this log is the restoration baseline; if the change time is later than (i.e., greater than) the target timestamp, the process moves forward to the previous log and continues the comparison. The physical meaning of the restoration baseline is: the last state change experienced by the node or directed edge before the target time. The changed value recorded in the baseline log is the accurate state of the object at the target time. The reason for using a backward scanning method is that backtracking queries are mostly for recent historical moments, and the requested timestamp is usually close to the current moment. Starting from the end, the result can be located within a constant number of comparison operations.

[0077] Furthermore, starting from the baseline, the process proceeds backward along the historical evolution sequence, applying the inverse operations recorded in the change log one by one until the graph state corresponding to the target timestamp is reached, thus constructing a dynamic scene graph snapshot at the target time. The inverse operation is based on the correspondence between the values ​​before and after the change recorded in the log, reverting the current graph state to an earlier historical state. The inverse operations for each change type are defined as follows: the inverse operation for the attribute update log is to restore the attribute value from the changed value to the value before the change; the inverse operation for the edge confidence change log is to restore the confidence value from the changed value to the value before the change; the inverse operation for the node creation log is to delete the node and its connected edges, since the node did not exist before the baseline; the inverse operation for the node deletion log is to recreate the node and restore the attributes and connected edges before deletion; the inverse operation for the directed edge creation log is to delete the edge; and the inverse operation for the directed edge deletion log is to recreate the edge and restore the confidence value before deletion. In practice, the system uses the latest state of the dynamic scene graph as the backtracking starting point. It iterates backward through the logs from the end of the sequence, performing the inverse operation only on logs whose change time is later than the target timestamp. Logs earlier than or equal to the target timestamp are already reflected in the baseline state and do not require backtracking. After traversing all logs later than the target timestamp and completing the inverse operation, the current graph state is backtracked to the accurate state at the target time. A snapshot of the state set of all nodes and directed edges in the graph at this point is then saved, thus constructing a snapshot of the dynamic scene graph at the target time.

[0078] Furthermore, a snapshot of the dynamic scene graph at the target time is output as a structured query result. The data structure of this snapshot is completely consistent with that of a regular dynamic scene graph, containing a set of nodes and a set of directed edges. Each node and edge has its attribute value and confidence value at that historical moment. External query modules can use the same query interface as the current graph to perform graph traversal queries and relational queries of arbitrary complexity on this historical snapshot, such as "who was holding tool number T001 5 minutes ago," "which desktop was the tool on at that time," and "who else was in the vicinity of the operating area at that time." The output format can adopt a structured serialized representation of graph data (such as JSON format), facilitating network transmission and consumption by heterogeneous systems.

[0079] Through the change log mechanism and historical evolution sequence management of this invention, the dynamic scene graph possesses the ability to travel to any historical moment and query the complete scene state at that time. This capability provides a solid data foundation for behavior analysis, incident tracing, and compliance review. Regarding storage overhead, the change log only records the incremental portion of each change, rather than saving a complete graph snapshot frame by frame. Storage overhead is proportional to the actual frequency of changes occurring in the scene. In real-world dynamic scenes, most frames involve only a small number of node position updates and a small number of edge confidence adjustments; therefore, the total storage volume of the change log is much smaller than that of the frame-by-frame snapshot scheme.

[0080] In some preferred embodiments, the dynamic scene graph is stored using a multi-version graph data structure. The basic storage unit of the multi-version graph data structure is a key-value pair with a timestamp, used to record the value of node attributes or edge confidence at a certain moment. For successfully associated instance nodes, updating the attributes of the corresponding existing nodes in the current dynamic scene graph specifically involves appending a new key-value pair with the current timestamp to the updated attributes of the existing node in the multi-version graph data structure, while the original key-value pair is retained as a historical version. For existing nodes that are not associated with any instance nodes in the video frame, deleting the existing node from the current dynamic scene graph specifically involves writing a deletion marker with the current timestamp to the existing node in the multi-version graph data structure.

[0081] In practice, during the incremental update of the graph, the attribute values ​​of nodes and the confidence values ​​of edges are continuously modified with each frame. Traditional relational databases or ordinary graph databases typically use in-situ data modification, where new attribute values ​​are directly written to the original storage location, overwriting the old values, which are permanently lost upon writing. This storage method has two key drawbacks: First, it cannot support historical version queries because only the latest version is retained, and all historical states are overwritten, failing to meet the needs of backtracking queries; second, it is not conducive to anomaly recovery. If a program crash or power failure occurs midway through processing a frame, some updates of the current frame may have been written to the storage medium while others have not, leaving the graph in an inconsistent intermediate state of partial updates and partial non-updates. After a system restart, it is difficult to recover to the previous known consistent state. This invention introduces a multi-version graph data structure as the underlying storage engine for the graph. Its basic design principle is: any persisted data version is never modified; all state updates are implemented by appending a new version with a new timestamp; and data deletion is achieved by writing a deletion flag. This append-only storage model naturally supports efficient historical version retrieval and atomic state rollback, fundamentally solving the two aforementioned shortcomings. It forms a solid storage foundation for building time-aware graphs. The specific implementation is as follows: The dynamic scene graph is persistently stored using a multi-version graph data structure. The core abstraction of this structure is that the storage system does not present a single, latest value, but rather an ordered chain of versions with different timestamps. Read operations can specify any timestamp, and the system automatically traverses the version chain, locates and returns the value of the most recent committed version preceding that timestamp. Write operations do not overwrite any existing versions; instead, they append a new version to the head of the version chain. This data structure can be implemented using existing mature key-value storage engines, such as those based on Log-Structured Merge Tree (LSM-Tree) or Sorted String Table (SSTable) based on B+ trees. The basic storage unit of the multi-version graph data structure is a key-value pair with a timestamp. Each key-value pair consists of three components: a key, which uniquely identifies the object to which the stored data belongs and its attribute name. For example, a key format could be "node N1.location.x", representing the x-coordinate attribute of the node identified as N1; a value, which stores the specific numerical value of the attribute at the corresponding write timestamp. The value type can be a floating-point number, integer, string, or serialized structure; and a timestamp, which records the system timestamp when this version was written to the storage system, obtained from the system clock during the write operation. The system maintains a multi-version manager for each key. This manager holds references or pointers to all historical versions under that key, and the versions are linked in descending order of timestamp to form a version chain. When a read operation is performed, the manager traverses down the version chain according to the requested timestamp, finds the first version whose timestamp does not exceed the requested timestamp, and returns its value.

[0082] Furthermore, based on the aforementioned multi-version storage structure, the operation of updating the attributes of corresponding existing nodes in the current dynamic scene graph for successfully associated instance nodes is implemented physically as follows: In the multi-version graph data structure, a new key-value pair with the current timestamp is appended to the updated attribute of the existing node, while the original key-value pair is retained in the version chain as a historical version. For example, if the spatial location of node N1 is updated from coordinates (x1, y1, z1) to (x2, y2, z2), and the current time is T2, the storage system does not locate and modify the original key-value pair written at time T1 (key: "node N1.location", value: (x1, y1, z1), timestamp: T1). Instead, it appends a new key-value pair to the head of the version chain for that key: the key is also "node N1.location", the value is updated to (x2, y2, z2), and the timestamp is the current time T2. After appending, the version chain of the key contains two versions: the head version timestamp T2 (x2, y2, z2) and the next version timestamp T1 (x1, y1, z1). Subsequently, when an external read operation carries a read timestamp greater than or equal to T2, the multi-version manager returns the head version (x2, y2, z2); when the read timestamp is between T1 and T2, the manager skips the head version and returns the version with timestamp T1 (x1, y1, z1). The two versions coexist in physical storage without interfering with each other, together forming a complete evolution record of the attribute value over time. This append-only design provides direct underlying support for historical backtracking queries in the above embodiments: backtracking to the graph state at a certain historical moment essentially involves setting the read timestamp of the read operation to that historical timestamp, and then performing a full database snapshot read with a unified timestamp on the attributes of all nodes and edges in the graph. Multi-version indexes reduce the complexity of historical queries to be comparable to latest value queries, without requiring any reverse calculations or log replay.

[0083] Furthermore, the physical storage implementation of deleting an existing node from the current dynamic scene graph is as follows: In the multi-version graph data structure, a deletion marker with the current timestamp is written to the existing node. The deletion marker is a special type of key-value pair, whose key format can be designed as "node N1.deletion marker", the value is a Boolean true value or a system-predefined deletion sentinel constant, and the timestamp is the time T3 when the deletion operation was performed. The deletion marker is appended to the version chain of the key as a special version. When an external system performs a read operation without a timestamp to retrieve the latest state, the storage system checks whether a version with the key "deletion marker" and the latest timestamp exists before returning the attribute value of the node; if it exists and the value is true, it returns the result "node does not exist" to the caller. When an external system performs a read operation with a historical timestamp: if the read timestamp is less than T3 (earlier than the deletion operation), the deletion marker does not yet exist or has not yet taken effect, and the historical attribute version of the node is returned normally, allowing the caller to see the complete state of the node before deletion; if the read timestamp is greater than or equal to T3 (later than or equal to the deletion operation), the deletion marker takes effect, and "node does not exist" is returned. This logical deletion strategy ensures that after a node is marked for deletion, all its historical attribute data remains intact in the storage system; only queries after the deletion time will show that the data does not exist. This preserves the complete historical truth for application scenarios such as behavior tracing and compliance audits that require tracing the historical state of deleted objects. When it is necessary to reclaim disk space occupied by historical versions that have not been accessed for a long time, old versions with timestamps exceeding the preset retention period (e.g., retaining data from the most recent 30 days) can be physically deleted through a background garbage collection process, but this operation is completely transparent to online read / write operations.

[0084] Furthermore, by introducing a multi-version graph data structure, the storage layer of the dynamic scene graph gains native time dimension support. Write operations, because they only append without modifying existing data, offer lock-free high concurrency performance, making them suitable for real-time update scenarios requiring a large number of attribute writes per frame. Compared to traditional in-place update plus independent log rollback schemes, the timestamp version chain approach reduces the latency of historical queries to the same order of magnitude as current value queries, and the query latency is independent of the historical backtracking depth. This provides an efficient and reliable graph persistence infrastructure for large-scale embodied intelligence systems that need to run continuously for a long time and frequently backtrack to historical states.

[0085] In some preferred embodiments, the method further includes: acquiring a depth map synchronized with the video frame; calculating the point cloud contour of the instance represented by the instance node based on the depth map; verifying the relationship information between the nodes according to the spatial contact state between the point cloud contours; and reducing the confidence of the directed edge representing the physical contact relationship between the two instances, or preventing the creation of the directed edge representing the physical contact relationship, when it is determined from the point cloud contour that two instances have no physical contact in three-dimensional space.

[0086] In specific implementations, in the multimodal perception processing of the above embodiments, the relationship detection model determines whether there is a physical relationship between instances based on two-dimensional image features. However, a two-dimensional image is a perspective projection of the three-dimensional physical world onto the camera's image plane, and this dimensional compression process inevitably loses the depth information of the scene. Under perspective projection, two instances that are far apart in three-dimensional space along the camera's line of sight but are in similar line of sight may appear as adjacent or even overlapping on the image plane, causing the relationship detection model to misjudge them as having physical contact based solely on two-dimensional appearance and position features. Such misjudgments can have serious consequences for the physical interaction operations of embodied intelligent agents: the robot may assume that a tool is currently being held by another person based on an incorrect "holding" relationship, thus abandoning the grasping of the tool and interrupting the normal operation process; or more dangerously, it may mistakenly assume that two objects are physically connected based on an incorrect "touching" relationship, thus performing inappropriate operations, leading to task failure or even equipment damage. This invention introduces a depth map, spatiotemporally synchronized with color video frames, as an independent perceptual channel separate from two-dimensional vision. It reconstructs the partial point cloud contours of each instance in three-dimensional space and performs geometric verification of physical contact relationships in three-dimensional Euclidean space. This corrects the spatial illusions caused by perspective projection in two-dimensional vision, ensuring that the final output physical contact relationships are strictly consistent with the real physical world. The specific implementation is as follows: Acquire a depth map synchronized with the video frames. A depth map is a distance image that is strictly aligned with the color video frames in both timestamps and spatial pixel coordinates. Each pixel stores the straight-line distance from the camera's optical center to the corresponding physical surface point. Depth maps can be directly acquired by depth sensors. Typical depth sensors include those based on structured light and those based on time-of-flight. Structured light sensors project a known coded pattern (such as infrared speckle or stripe patterns) onto the scene, recovering the depth value of each pixel by analyzing the deformation of the projected pattern on the object's surface; this method offers high accuracy at close range. Time-of-flight sensors emit modulated light pulses into the scene, measuring the round-trip time from emission, reflection from the object's surface, to return to the sensor, directly calculating the distance value; this method offers a longer measurement range and a higher frame rate. Depth maps can also be indirectly calculated using binocular stereo vision algorithms: images are simultaneously acquired by two spatially calibrated cameras; a stereo matching algorithm is used to find corresponding pixels between the two images; and the depth value is calculated using triangulation based on the disparity value of the corresponding point, combined with the camera baseline distance and focal length. Regardless of the depth acquisition method used, the resolution and frame rate of the depth map should be consistent with those of the color video frames to ensure that each color pixel has corresponding depth information available. The intrinsic parameters (focal length, optical center coordinates, distortion coefficients) of the depth sensor or stereo camera system, as well as the extrinsic parameters (rotation matrix, translation vector) between the depth sensor or stereo camera system and the color camera, need to be accurately obtained through an offline calibration process. The calibration accuracy directly affects the accuracy of subsequent 3D reconstruction and contact determination.

[0087] Furthermore, based on the depth map, the complete point cloud contour of the instance represented by the instance node is calculated. For each instance node, its detection bounding box or a more refined instance segmentation mask is used to precisely crop out the depth pixel region belonging only to that instance from the depth map. The instance segmentation mask can be obtained in the multimodal perception processing of the above embodiments by adding a foreground and background binary classification segmentation branch to the target detection head. This branch determines whether each pixel within the bounding box belongs to the instance. For each valid pixel in the cropped depth pixel region (i.e., a pixel with a valid depth value and belonging to the instance), backprojection is performed using the inverse operation of the camera pinhole model to convert it from two-dimensional image coordinates and depth values ​​to spatial point coordinates in the three-dimensional camera coordinate system. The backprojection process is as follows: based on the image coordinates of the pixel, the depth value of the pixel, and the focal length and optical center parameters of the camera, the corresponding point position of the pixel in three-dimensional space is calculated through geometric relationships. All valid pixels within the instance mask region are traversed and backprojection is performed one by one. All the generated three-dimensional spatial points are collected to form a partial point cloud of the instance in the current camera view. Due to the self-occlusion effect (the surface of an instance facing away from the camera cannot be observed by the sensor), a single-view point cloud only contains the visible surface portion of the instance and is incomplete. However, for physical contact determination tasks, physical contact between two instances must occur on surfaces visible to each other. Therefore, the single-view partial point cloud already contains all the geometric information required for contact determination. To improve processing efficiency and filter out measurement noise inherent in depth sensors, statistical outlier filtering can be performed on the original point cloud. This involves calculating the average distance from each point to its K nearest neighbors. If the average distance exceeds a preset threshold, the point is identified as an outlier and removed. Then, the 3D convex hull or axis-aligned minimum bounding box of the filtered point cloud is calculated. This convex hull or bounding box is used as a geometric proxy for the instance's contour for subsequent contact determination, reducing the massive computational cost of calculating point-to-point distances between two complete point clouds.

[0088] Furthermore, the relationship information between nodes is verified based on the spatial contact state between point cloud contours. For each relationship marked as a physical contact relationship in the current frame's relationship information (such as "holding", "touching", "supporting"), the point cloud contour geometric proxies (convex hulls or bounding boxes) of the subject instance node and object instance node of that relationship are obtained. The nearest spatial distance between the two point cloud contour geometric proxies is calculated. The nearest spatial distance is defined as: traversing all points on the surface of the subject point cloud or bounding box, calculating the Euclidean distance from each point to all points on the surface of the object point cloud or bounding box, and taking the minimum value among all distances. If the minimum distance is less than a preset physical contact distance threshold, it is determined that the two instances have physical contact in 3D space; if the minimum distance is greater than the threshold, it is determined that there is no physical contact. The setting of the physical contact distance threshold needs to comprehensively consider the depth measurement accuracy of the depth sensor, the calibration error of the camera's intrinsic and extrinsic parameters, and the boundary accuracy of the instance segmentation mask. In a typical configuration, the physical contact distance threshold can be set to 0.5 cm, which is about 2 to 3 times the depth measurement accuracy of commonly used consumer-grade depth sensors at working distances of 1 to 3 meters. This can effectively filter out false judgments caused by measurement errors while ensuring a high detection rate.

[0089] Furthermore, when it is determined from the point cloud contours that two instances have no physical contact in 3D space, the system executes two different processing paths based on the existence status of this physical contact relationship in the current dynamic scene graph. Path 1: If a directed edge representing this physical contact relationship already exists in the current dynamic scene graph (i.e., the relationship has been created and persists in the previous frame), then an additional confidence penalty is applied to the existing directed edge to reduce its confidence level. The reduction is greater than the normal unobserved attenuation, such as a single attenuation of 0.4. This is because not only is the relationship not detected in the current frame (possibly due to missed detection), but there is also direct evidence from 3D spatial geometry indicating that the relationship is physically impossible. The large confidence penalty allows the erroneous directed edge to quickly decay below the extinction threshold and be deleted within 1 to 2 frames, promptly correcting the erroneous relationship already formed in the graph. Path 2: If the relation information output by the multimodal perception in the current frame contains the physical contact relationship (i.e., the two-dimensional relationship detection model misjudges it as having physical contact), then prevent the creation of directed edges representing the physical contact relationship. That is, do not establish this false interaction relationship that is disproven by the depth information in the graph, intercept the error at the information entry point, and prevent it from entering the graph.

[0090] Furthermore, when it is determined from the point cloud contours that two instances are indeed in physical contact in 3D space, this physical contact relationship receives positive verification from an independent depth-sensing channel. At this point, the standard procedure described in the above embodiment is followed: if the relationship does not yet exist in the graph, a new directed edge is created and an initial confidence level is set according to the normal procedure; if it already exists, the confidence level of the directed edge is increased according to the normal procedure. The positive verification result of the depth information can also be used to further slightly increase the confidence level of the relationship edge or mark it as "geometrically verified" to enhance its credibility.

[0091] Through the physical contact relationship verification mechanism that integrates depth information in this invention, false physical contact relationships caused by perspective projection in 2D visual perception are directly filtered out from the 3D spatial geometry. As a physical measurement channel independent of the color camera's spectral response, the depth perception channel's measurement results depend solely on the scene's 3D geometry, unaffected by ambient lighting changes, object surface textures, or 2D appearances. This provides orthogonal evidence at the geometric level, complementary to 2D vision, for physical contact determination. This mechanism ensures extremely high reliability of the physical contact relationships output in dynamic scene maps, providing a solid and reliable environmental cognitive foundation for embodied intelligent agents to perform precise object grasping, handover, placement, and collision avoidance operations. For low-cost systems without depth sensors, this verification step can be an optional module; the rest of the system's map construction process remains unaffected and usable.

[0092] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0093] Please see Figure 2 , Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0094] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0095] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it enables the processor 502 to execute a real-time method for constructing a dynamic scene map oriented towards embodied intelligence.

[0096] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0097] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a method for real-time construction of dynamic scene maps oriented towards embodied intelligence.

[0098] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0099] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of a real-time construction method for dynamic scene maps oriented towards embodied intelligence provided in any of the above method embodiments.

[0100] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0101] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0102] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the steps of a real-time construction method for a dynamic scene map oriented towards embodied intelligence provided in any of the above method embodiments.

[0103] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0104] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0105] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0106] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0107] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0108] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0109] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, this invention is also intended to include these modifications and variations as long as they fall within the scope of the claims of this invention and their equivalents.

[0110] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for real-time construction of dynamic scene graphs for embodied intelligence, characterized in that, For each video frame in the received video frame sequence, perform the following steps in sequence: The dynamic scene map obtained from the processing of the previous video frame is used as the current dynamic scene map, wherein when the video frame is the first frame of the video frame sequence, the current dynamic scene map is empty; Multimodal perception processing is performed on the video frames to obtain a set of instance nodes, attribute information of each instance node, and relationship information between nodes; Based on the set of instance nodes and attribute information, the instance nodes are associated with the existing nodes in the current dynamic scene graph. The current dynamic scene graph is updated according to the association results to obtain the updated dynamic scene graph. The updated dynamic scene graph is then used as the current dynamic scene graph corresponding to the next video frame. The step of updating the current dynamic scene graph based on the association results includes: updating the attributes of the corresponding existing nodes in the current dynamic scene graph for successfully associated instance nodes; creating new nodes in the current dynamic scene graph for instance nodes not associated with existing nodes; deleting existing nodes from the current dynamic scene graph for existing nodes not associated with any instance nodes in the video frame; and updating the directed edges between nodes in the current dynamic scene graph based on the relationship information between the nodes.

2. The method for real-time construction of dynamic scene graphs for embodied intelligence according to claim 1, characterized in that, Associating the instance node with existing nodes in the current dynamic scene graph includes: Extract the appearance features of the instance node and the historical appearance features of the existing node; Based on the motion information of the instance node and the historical motion trajectory of the existing node, predict the estimated location region of the existing node in the video frame; Calculate the appearance similarity between the instance node and the existing node in the feature space, and calculate the spatial overlap between the actual position of the instance node and the estimated position region; The appearance similarity and spatial overlap are weighted and fused to obtain the association confidence score. Instance nodes with an association confidence level higher than a preset association threshold are identified as successfully associated with existing node pairs.

3. The method for real-time construction of dynamic scene graphs for embodied intelligence according to claim 1, characterized in that, For successfully associated instance nodes, updating the attributes of the corresponding existing nodes in the current dynamic scene graph includes: Determine the attribute type of the attribute; If the attribute type is a continuously changing first type attribute, then based on the attribute value of the successfully associated instance node, the corresponding attribute value of the existing node is updated by smooth filtering. If the attribute type is a discretely changing second type attribute, then obtain the state sequence of the existing node in a series of consecutive video frames including the video frame, and determine the stability of the state sequence. When the determination result meets the preset stability condition, the second type attribute of the existing node is updated to the attribute value of the successfully associated instance node.

4. The method for real-time construction of dynamic scene graphs for embodied intelligence according to claim 1, characterized in that, Updating the directed edges between nodes in the current dynamic scene graph based on the relationship information between the nodes includes: Determine whether the target relationship in the relationship information already exists in the current dynamic scene graph; If it does not exist, create a directed edge connecting the two successfully associated instance nodes and set an initial confidence level for the directed edge; If it already exists, then the target relationship is determined to be in a continuous observation state in the video frame, and the confidence of the directed edge is increased; For existing directed edges in the current dynamic scene graph that have not been observed again in the video frame, reduce the confidence of the existing directed edges; Delete directed edges with a confidence level lower than a preset extinction threshold from the current dynamic scene graph.

5. The method for real-time construction of dynamic scene graphs for embodied intelligence according to claim 1, characterized in that, The step of deleting existing nodes that are not associated with any instance node in the video frame from the current dynamic scene graph includes: Maintain a liveness timer for each existing node; When the existing node fails to associate with the corresponding instance node in the video frame for the first time, the survival timer is started to keep track. If, within a series of consecutive video frames before the survival timer reaches a preset extinction time threshold, the existing node is reassociated with the instance node, the survival timer is reset. If the existing node is still not associated with any instance node when the survival timer reaches the preset extinction time threshold, then the existing node and all directed edges connected to it are deleted from the current dynamic scene graph.

6. The method for real-time construction of dynamic scene graphs for embodied intelligence according to claim 1, characterized in that, The method further includes: During the process of updating the current dynamic scene graph based on the association results, a change log is generated for each node and each directed edge in the current dynamic scene graph. The change log includes the change type, the change time, and the attribute values ​​or confidence levels before and after the change. The change logs generated by each update are sequentially appended to the operation log sequence bound to each node and each directed edge to form their respective historical evolution sequences; In response to receiving a backtracking query request carrying a target time, the target time is parsed and the timestamp corresponding to the target time is determined; In the historical evolution sequence, starting from the end of the sequence and moving towards the beginning, the change time of each change log is compared with the timestamp, and the first change log whose change time is no later than the timestamp is located as the restoration reference point. Starting from the restoration reference point, proceeding forward along the historical evolution sequence, apply the reverse operations recorded in the change log one by one in reverse order until backtracking to the graph state corresponding to the timestamp, and construct a dynamic scene graph snapshot at the target time. The reverse operation is to restore the changed state to the state before the change. The snapshot of the dynamic scene map at the target time is output as a structured query result.

7. The method for real-time construction of dynamic scene graphs for embodied intelligence according to any one of claims 1 to 6, characterized in that, The dynamic scene graph is stored using a multi-version graph data structure. The basic storage unit of the multi-version graph data structure is a key-value pair with a timestamp, which is used to record the value of node attributes or edge confidence at a certain moment. For the successfully associated instance nodes, the attribute of the corresponding existing node in the current dynamic scene graph is updated. Specifically, in the multi-version graph data structure, a new key-value pair with the current timestamp is appended to the updated attribute of the existing node, and the original key-value pair is retained as a historical version. The step of deleting an existing node from the current dynamic scene graph for an existing node that is not associated with any instance node in the video frame specifically involves writing a deletion marker with the current timestamp for the existing node in the multi-version graph data structure.

8. The method for real-time construction of dynamic scene graphs for embodied intelligence according to any one of claims 1 to 6, characterized in that, The method further includes: Obtain a depth map synchronized with the video frame; Based on the depth map, calculate the point cloud contour of the instance represented by the instance node; The relationship information between the nodes is verified based on the spatial contact state between the point cloud contours. When it is determined from the point cloud contour that two instances have no physical contact in three-dimensional space, the confidence of the directed edge representing the physical contact relationship between the two instances is reduced, or the creation of the directed edge representing the physical contact relationship is prevented.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-8.

Images