Cross-scene object marking method and device, equipment, medium and product

Abstract

The invention discloses a cross-scene object marking method and device, equipment, a medium and a product. Relates to the technical field of computers, and the method comprises the following steps: determining a multi-modal object instance set and a global semantic anchor point identifier, a context projection identifier and an own state identifier of each object in the multi-modal object instance set according to a multi-modal data sample; according to the multi-modal object instance set and the global semantic anchor point identifier, the context projection identifier and the body state identifier of each object in the multi-modal object instance set, a training data set is generated, and the training data set is used for training the to-be-trained model to obtain the multi-modal model. Robust identification and explainable tracing of the object in different modes, different scenes and different tasks can be realized.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a cross-scene object marking method, apparatus, device, medium and product. Background Technology

[0002] With the development of deep learning and computer vision technologies, multimodal object recognition has become one of the core capabilities of intelligent systems, widely applied in fields such as robot manipulation, autonomous driving, intelligent monitoring, and augmented reality. However, existing technologies still face many challenges in cross-scene, cross-modal, and cross-temporal object recognition and tracking: the representations of objects in different scenes (e.g., kitchen vs. office), different modalities (text description vs. visual image vs. 3D point cloud), and different times (static vs. moving vs. being manipulated) are often isolated and inconsistent, leading to the same object being identified as different entities in different contexts, or different objects being confused due to surface similarity. This has led to the development of various techniques such as multimodal representation learning, cross-modal retrieval, temporal object tracking, and knowledge graph association to enhance the robustness of object recognition, improve cross-scene generalization capabilities, and solve core problems such as identifier drift, semantic confusion, and tracking fragmentation. Existing object recognition and tracking methods mainly fall into the following categories:

[0003] 1. Single-modal object recognition method: This is an image object detection and classification method based on deep convolutional neural networks. It learns the visual feature representation of objects through pre-training on large-scale datasets. Its advantages are high recognition accuracy in a single modality, fast inference speed, and mature engineering application. Its disadvantages are that it relies on labeled data and has limited generalization ability. It is difficult to distinguish objects that look similar but have different meanings using only visual information (such as a red cup vs. a red bowl). It cannot use text descriptions or physical attributes to assist in recognition, and it is sensitive to changes in lighting, occlusion, and viewpoint.

[0004] 2. Multimodal Representation Learning Methods: These methods map different modalities (such as images and text) to a unified semantic space through contrastive learning. A representative example is zero-shot recognition achieved through large-scale image-text contrastive learning. The advantage is that it achieves cross-modal alignment between vision and language, supporting natural language query objects. The disadvantage is that it only considers static image-text pairing relationships, failing to model the contextual differences of objects in different scenarios and tasks. This results in the same object being mapped to the same representation in different contexts (e.g., "Apple" in tech news vs. fruit market), lacking context sensitivity. Furthermore, existing methods are mostly bimodal (image-text) alignment, with insufficient support for the 3D geometric information of embodied scenes.

[0005] 3. Video Object Tracking Methods: Multi-object tracking methods associate the same object across video frames through appearance feature matching and motion prediction, maintaining the continuity of tracking IDs. Their advantages include the ability to handle object tracking in continuous video streams and cope with occlusion and brief disappearances. However, their disadvantages include reliance on assumptions of visual appearance similarity and spatiotemporal continuity, making it difficult to maintain ID consistency during cross-scene transitions (objects reappearing in a new scene after leaving the field of view) or long-term occlusion (objects reappearing after disappearing for several seconds). Furthermore, tracking IDs are merely numerical identifiers, lacking semantic information and unable to be associated with the object's global identity or historical state.

[0006] 4. Knowledge Graph and Entity Linking Method: By constructing object ontologies and knowledge graphs, semantic relationships between objects are established (e.g., "cup is a container", "cup has_part handle"), and detected object instances are linked to entities in the knowledge base. Its advantages include providing rich semantic knowledge and relational reasoning capabilities, supporting object recognition and relational understanding based on prior knowledge. Its disadvantages include that knowledge graphs are typically statically constructed manually, making automatic updates difficult to adapt to new objects or scenarios; and that entity linking relies on name or attribute matching, is sensitive to changes in visual appearance and synonyms, and lacks dynamic modeling of physical states (position, posture, motion).

[0007] 5. Point Cloud 3D Object Detection Methods: These methods utilize point cloud-based 3D object detection, processing 3D point cloud data acquired by LiDAR or RGB-D cameras to achieve object localization and classification within a embodied environment. Their advantages include providing accurate 3D position and pose information, independence from lighting conditions, and suitability for robot operation and autonomous driving scenarios. Disadvantages include processing only geometric information, making it difficult to identify visual attributes such as material and color; the point clouds are sparse and noisy, limiting accuracy in recognizing small objects or complex shapes. Furthermore, they lack association with textual semantics or visual appearance, and cannot utilize multimodal complementary information.

[0008] 6. Retrieval-enhanced object recognition methods: These methods quickly find similar objects in the feature space through vector retrieval (such as the FAISS index), achieving instance-based object recognition. Their advantages include support for open set recognition (not limited to predefined categories) and ease of incremental updates (adding new object instances does not require retraining the model). Disadvantages include reliance on the quality of feature vectors and a lack of explicit modeling of contextual differences. This can lead to retrieval failures due to significant differences in feature vectors for the same object in different scenarios, or mis-retrieving different objects due to similar appearances. Furthermore, retrieval only returns the most similar candidates, lacking multi-dimensional verification and ambiguity resolution mechanisms.

[0009] In summary, existing cross-scene object recognition and tracking technologies are still limited to the paradigms of "single-modal independent modeling" or "shallow feature alignment," lacking a refined characterization and dynamic coordination of the joint evolution of semantic, physical, and task spaces. Furthermore, existing methods struggle to establish a unified identifier with causal consistency among textual, visual, and embodied information, leading to "identifier drift" and "tracking breakage" for the same object during scene switching, modality loss, or long-term occlusion. This, in turn, restricts the robustness and scalability of the system in open environments and long-term tasks. Summary of the Invention

[0010] This invention provides a cross-scenario object marking method, apparatus, device, medium, and product to solve at least one of the above-mentioned problems.

[0011] According to one aspect of the present invention, a cross-scene object tagging method is provided, comprising:

[0012] Based on the multimodal data samples, determine the set of multimodal object instances and the global semantic anchor identifier, context projection identifier, and embodied state identifier of each object in the set of multimodal object instances;

[0013] A training dataset is generated based on the set of multimodal object instances and the global semantic anchor identifier, context projection identifier, and embodied state identifier of each object in the set of multimodal object instances. The training dataset is used to train the model to be trained to obtain a multimodal model.

[0014] According to another aspect of the present invention, a cross-scene object marking device is provided, the cross-scene object marking device comprising:

[0015] The determination module is used to determine the set of multimodal object instances and the global semantic anchor identifier, context projection identifier, and embodied state identifier of each object in the set of multimodal object instances based on the multimodal data samples.

[0016] The generation module is used to generate a training dataset based on the multimodal object instance set and the global semantic anchor identifier, context projection identifier and embodied state identifier of each object in the multimodal object instance set. The training dataset is used to train the model to be trained to obtain a multimodal model.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] At least one processor; and

[0019] A memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the cross-scene object tagging method according to any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the cross-scene object marking method according to any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer program product is provided, which, when executed by a processor, implements the cross-scene object tagging method as described in any of the embodiments of the present invention.

[0023] This invention, through multimodal data samples, determines a set of multimodal object instances and the global semantic anchor identifier, context projection identifier, and embodied state identifier of each object in the set of multimodal object instances; and generates a training dataset based on the set of multimodal object instances and the global semantic anchor identifier, context projection identifier, and embodied state identifier of each object in the set of multimodal object instances, thereby enabling robust identification and interpretable traceability of objects in different modalities, different scenarios, and different tasks.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a cross-scene object marking method in an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of the structure of a cross-scene object marking device according to an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0032] Example 1

[0033] Figure 1 This is a flowchart illustrating a cross-scene object tagging method provided in an embodiment of the present invention. This embodiment is applicable to cross-scene object tagging scenarios. The method can be executed by the cross-scene object tagging device in this embodiment, which can be implemented in software and / or hardware, such as... Figure 1 As shown, the method specifically includes the following steps:

[0034] S110, Based on the multimodal data samples, determine the set of multimodal object instances and the global semantic anchor identifier, context projection identifier, and embodied state identifier of each object in the set of multimodal object instances.

[0035] In this embodiment, the multimodal object instance set includes: a text object instance set, a visual object instance set, an embodied object instance set, a fusion object instance set, and a modality mapping table. The text object instance set includes: a text object identifier, a text object name, a text object type, a coreference list, a semantic role, a semantic feature vector, and contextual text fragments. The visual object instance set includes: a visual object identifier, a bounding box, a visual object type, a visual feature vector, location coordinates, and an image identifier. The embodied object instance set includes: an embodied object identifier, a bounding box, a point cloud cluster, an embodied object type, a six-DOF pose, a physical attribute dictionary, an embodied feature vector, and a sensor timestamp.

[0036] In this embodiment, the multimodal data samples may include: text stream (from dialogue, document, annotation, etc.), image stream (from camera, sensor, etc.), video stream (containing a temporally continuous frame sequence), point cloud stream (from LiDAR, depth camera, etc.), task context description information (including task type, target, scene information), etc.

[0037] In this embodiment, object instances are detected, segmented, and identified from multimodal data samples. Based on the modality type (text / visual / embodied) and scene context of the object, hierarchical annotation is performed to obtain a set of multimodal object instances with clear boundaries, well-defined types, and cross-modal relevance.

[0038] In this embodiment, the method for determining the global semantic anchor identifier of each object in the multimodal object instance set based on the multimodal data samples is as follows: for each object instance in the multimodal object instance set (prioritizing fused objects because they have the most comprehensive information, and processing the corresponding modality set for objects that only appear in a single modality), extract the inherent semantic features that do not change with the scene, time, or observation perspective, as the basis for generating the global semantic anchor identifier GID.

[0039] In this embodiment, the method for determining the context projection identifier of each object in the multimodal object instance set based on multimodal data samples can be as follows: A context projection identifier (CID) is generated for each object instance in a specific context (including scene environment, task background, time stage, agent role, etc.). This captures the semantic role, relationship network, and state differences of the object in that context, enabling the same object (same GID) to be distinguishably represented in different contexts (different CIDs), while maintaining the consistency of global identifiers (through GID association). For example, multi-dimensional contextual features are extracted from the task context and the local environment information where the object is located, and a spatial, semantic, and physical relationship network between objects is constructed as the basis for generating context projection identifiers.

[0040] In this embodiment, the traditional single-modal object ID is upgraded to a three-layer collaborative structure: Global Semantic Anchor ID (constant across scenes), Context Projection ID (scene / task / temporal differences), and Embodied State ID (physical-task evolution). Through hierarchical encoding and mapping tables, the traceability, aggregation, and evolution of GID→CID→EID are achieved.

[0041] Optionally, the multimodal data samples include: text sequences, image sequences, video streams, and point cloud sequences;

[0042] Based on the multimodal data samples, determine the set of multimodal object instances, including:

[0043] The text sequence is processed to obtain a set of text object instances.

[0044] The text object instance set includes: text object identifier, text object name, text object type, coreference list, semantic role, semantic feature vector, and context text fragment.

[0045] In this embodiment, each element in the text object instance set has the following structure: {object_id: unique identifier of the object, object_name: name of the text object, object_type: type of the text object, semantic_role: semantic role, coref_chain: list of coreference chains, context_snippet: context text fragment. The text object types include: person names, place names, items, etc. The semantic roles include: subject, object, and instrumental case, etc.

[0046] In this embodiment, the processing object is a text stream sequence. Natural language processing techniques are used to identify object instances described in the text, specifically: entity recognition and coreference resolution: a pre-trained named entity recognition model (such as a BERT-based NER model or the SpaCy toolkit) is used to extract entities from the text sequence, identifying entity types such as people, places, organizations, and items, generating an initial entity candidate set, entities_raw. Each candidate contains entity text (e.g., "Alice"), entity type (e.g., "PERSON"), and its position in the original text (span_position, the start and end character indices). Subsequently, a neural coreference resolution model (e.g., End-to-End Coreference) is used. The Resolution model identifies different text fragments referring to the same object (e.g., "Alice" in sentence 1 and "she" in sentence 3 refer to the same person entity), and generates a set of deduplicated object instances, entities_text. Each instance contains a unique entity identifier entity_id (a unique ID generated by the system), entity text entity_text (the text mentioned for the first time), entity type entity_type, coref_chain (containing all text fragments referring to the entity and their positions), and context snippet context_snippet (sentence fragments surrounding the entity used for subsequent semantic understanding). Semantic role labeling: Based on entity type and context keywords (verbs and prepositions extracted from context_snippet through dependency parsing), the semantic role (subject, object, instrument, goal, etc.) of the object is labeled. For example, in the sentence "Alice uses a cup to pour water", dependency parsing identifies "Alice" as the subject (nsubj dependency relation) and labels it as the subject, "cup" as the object (dobj dependency relation) and the connecting verb "uses" as the instrument, and "water" as the indirect object and labels it as the object. The semantic role information is recorded in the semantic_role field and is used to distinguish the functional role of the object in different tasks when generating the context identifier (CID) later.

[0047] The image sequence and the video stream are processed to obtain a set of visual object instances.

[0048] The set of visual object instances includes: visual object identifier, bounding box, visual object type, visual feature vector, position coordinates, and image identifier.

[0049] In this embodiment, the structure of each element in the visual object instance set is {object_id: unique identifier of the visual object, bbox: bounding box (x1, y1, x2, y2), mask: segmentation mask (if any), class_label: visual object type, visual_feature: visual feature vector, track_id: tracking identifier (if it is a video), spatial_location: spatial location coordinates, image_id or frame_id: identifier of the image or frame to which it belongs}.

[0050] In this embodiment, the processing objects are the image stream (image_stream) and video stream (video_stream). Visual objects are detected, segmented, and tracked using computer vision technology. Specifically, image object detection and instance segmentation are performed as follows: For static images, object detection models (such as YOLOv8, Faster R-CNN, DETR) are used to locate the bounding box (containing the coordinates of the top left corner x1, y1 and the bottom right corner x2, y2), the class label (identified from a predefined class dictionary such as COCO 80 classes), and the confidence score (the model's confidence in the detection result between 0 and 1) of the objects in the image, generating a preliminary detection result, detections_image. To obtain more accurate object regions, instance segmentation models (such as Mask R-CNN, SOLOv2) are used to generate a pixel-level segmentation mask (a binary mask representing the pixels occupied by the object) for each detection box, generating a segmentation result, segments_image. Each element contains a segmentation mask, a bounding box (bbox), a class label, and an image ID. Video object tracking: For video streams, in addition to detecting objects in each frame, it is also necessary to associate the same object instances across temporal frames to maintain identifier consistency. Multi-object tracking algorithms (such as DeepSORT combining appearance features and motion prediction, and ByteTrack combining high and low confidence detection association) are used to match objects between consecutive frames, generating temporally continuous tracking tracks_video. Each track contains a tracking identifier track_id (the unique ID of the track in the video), a bounding box sequence bbox_sequence (the list of bounding boxes of the object in each frame), a class label class_label (the class of the entire track can be determined through voting or the first frame), and a frame index sequence frame_ids (the frame numbers of all frames in which the object appears). The tracking algorithm performs data association by calculating the intersection area ratio (IoU) of the detection boxes in consecutive frames and the appearance feature similarity (cosine distance of feature vectors extracted using the ReID model). If IoU > 0.5 or feature similarity > 0.7, the objects are considered to be the same.Visual feature extraction: A pre-trained visual encoder (such as ResNet-50 to extract 2048-dimensional features, or ViT to extract 768-dimensional features) is used to extract a deep visual feature vector (visual_feature) for each detected object region (cropped by bounding box or mask). This feature vector encodes the appearance information of the object (color, texture, shape, etc.) and is used for subsequent cross-modal matching and embedding learning. At the same time, the spatial_location (coordinates of the bounding box center point or centroid) of the object in the image coordinate system is recorded for spatial relationship modeling.

[0051] The point cloud sequence is processed to obtain a set of embodied object instances.

[0052] The embodied object instance set includes: embodied object identifier, bounding box, point cloud cluster, embodied object type, six-DOF pose, physical attribute dictionary, embodied feature vector, and sensor timestamp.

[0053] In this embodiment, each element in the embodied object instance set has the following structure: {object_id: embodied object identifier, bbox_3d: 3D bounding box {center, dimensions, rotation}, point_cluster: point cloud cluster (N×3 array), class_label: embodied object type, pose_6d: six-degree-of-freedom pose {position, orientation}, physical_attrs: physical attribute dictionary {mass_estimate, material_type, movable_flag, graspable}, sensor_timestamp: sensor timestamp.

[0054] In this embodiment, the processing object is the pointcloud_stream point cloud sequence (from 3D sensors such as LiDAR and RGB-D cameras). Physical objects in the embodied environment are extracted through 3D target detection and point cloud segmentation technology. Specifically, the point cloud preprocessing is as follows: The original point cloud data is subjected to quality improvement processing, including voxel downsampling (using a 0.05m voxel grid to reduce the point cloud density and improve processing speed), outlier removal (statistical outlier removal) and ground segmentation (RANSAC plane fitting algorithm to detect and remove ground point clouds while retaining obstacle point clouds). The resulting cleaned point cloud, pointcloud_clean, retains the valid object point cloud data. 3D Object Detection: 3D object detection models (such as PointPillars based on columnar voxel representation, VoxelNet based on voxel feature encoding, and PointRCNN based on direct point cloud processing) are used to detect 3D bounding boxes (bbox_3d) in the point cloud. Each 3D bounding box contains a center position (coordinates in the x, y, z world coordinate system), dimensions (length, width, height), and rotation angle (yaw angle around the z-axis, representing the object's orientation). Simultaneously, a class label (e.g., "car", "pedestrian", "bicycle") and a confidence score are output, generating the 3D detection result (detections_3d). Point cloud segmentation and pose estimation: To obtain the precise geometric shape of the object, point cloud segmentation algorithms (such as PointNet++ based on point cloud feature learning and DBSCAN clustering based on density clustering) are used to generate a point cloud cluster (a set of all point cloud points belonging to the object) for each object, accurately extracting the geometric information of the object; 6D pose estimation algorithms (such as PVN3D and DenseFusion) are used to calculate the six-degree-of-freedom pose (pose_6d) of the object in the world coordinate system, including 3-degree-of-freedom position (position(x,y,z)) and 3-degree-of-freedom orientation (represented by quaternion or rotation matrix), and the observation timestamp is recorded for time series tracking.Physical attribute inference: Based on point cloud density (number of points per unit volume), bounding box dimensions (length, width, and height), and category labels (e.g., "wooden table" vs. "metal chair"), the physical attributes of the object are inferred, including mass_estimate (estimated by volume × material density), material_type (e.g., "wood", "metal", "plastic", "glass"), movable_flag (a boolean value indicating whether the object can be moved by the robotic arm; objects with a volume less than 0.5 m³ and a mass less than 5 kg are considered movable), and grabbable (determined by size and shape whether the object can be grabbed by the gripper), generating a physical attribute dictionary physical_attrs.

[0055] Based on the set of text object instances, the set of visual object instances, and the set of embodied object instances, a set of fusion object instances and a modality mapping table are generated.

[0056] The modality mapping table is used to represent the mapping relationship between the fusion object and the modality object. The modality object includes any one of the following: text object, visual object and embodied object. The fusion object instance set includes: fusion object identifier, source modality corresponding to the fusion object and fusion attribute information. The fusion attribute information includes: text object name, object type, visual feature vector, 3-DOF pose, context text fragment and physical attribute dictionary.

[0057] In this embodiment, the Modality_map structure is {fused_id→{text_id, visual_id, embodied_id}}, which records the mapping relationship between each fused object and the modal object (source modal object), and supports subsequent querying of its original data in each modality based on the fused object ID.

[0058] In this embodiment, the processing objects are object instances from three modalities: O_text (text object instance set), O_visual (visual object instance set), and O_embodied (embodied object instance set). A cross-modal matching algorithm is used to establish the correspondence between the same object in different modalities (e.g., a "cup" is mentioned in text, detected in an image, and located in a point cloud), achieving complementary fusion of multimodal information. Specifically: Visual-Text Association: A pre-trained visual language model (such as the CLIP model, which simultaneously encodes images and text into a unified semantic space) is used to calculate the semantic similarity (simi) between the visual_feature of the visual object and the entity_text of the text object. The similarity_score (calculated using cosine similarity) is used to determine the angle between two feature vectors. If the similarity exceeds the threshold threshold_vt=0.75 (this threshold is determined through validation set tuning) and the two are close in time and space (the difference between the time the text mentions the image and the time the image is captured is less than 5 seconds, or the scene described in the text matches the scene in the image), then they are determined to be the same object, an association pair (object_id_visual, object_id_text) is established, and the association confidence is recorded. For example, if the similarity between the text mentioning "the red cup on the table" and the red cup-shaped object detected in the image is 0.82, then an association is established. Visual-embodied association: The 3D bounding box bbox_3d is projected onto the image plane using the camera intrinsic and extrinsic parameter matrices (projection matrix obtained from camera calibration) to obtain the 2D projected box_bbox. The intersection-over-union (IoU) ratio between the projected box and the 2D detection box bbox is calculated. If the IoU exceeds the threshold threshold_ve=0.6, they are determined to be the same object, and an association pair (object_id_visual, object_id_embodied) is established. This method uses geometric consistency constraints to ensure the correct spatial correspondence between visual objects and physical objects. For scenes where RGB-D cameras provide both color and depth images, visual objects and 3D point cloud objects can be directly associated through pixel correspondence.Trimodal fusion: For objects that appear simultaneously in textual, visual, and embodied modalities (e.g., in a robot task, "pick up the cup on the table," where the text description is "cup," the camera sees an image of the cup, and the LiDAR scans a point cloud of the cup), a unified fused object instance `fused_object` is created. This instance aggregates complementary information from each modality to obtain a comprehensive representation of the object: from the textual modality, semantic labels (e.g., the conceptual category of "cup") and task roles (e.g., "target object") are obtained; from the visual modality, visual features (color, texture, etc.) and 2D position are obtained; and from the embodied modality, precise 3D position `pose_3d` and physical attributes `physical_attrs` are obtained. The fusion strategy is: if a certain attribute ( If `class_label` has values ​​in multiple modalities, the final value is determined through a voting mechanism (selecting the value that appears most frequently) or a confidence-weighted approach (selecting the value from the modality with the highest confidence). A fused object set `O_fused` is generated, with each element containing the complete fused attribute `fused_id` (a unique identifier for the newly generated fused object), a list of source modal object IDs `{text_id, visual_id, embodied_id}` (recording which modal objects the fused object originated from, supporting reverse tracing), and the fused attributes `{object_name, class_label, visual_feature, pose_3d, context_snippet, physical_attrs}`.

[0059] A multimodal object instance set is generated based on the text object instance set, the visual object instance set, the embodied object instance set, the fusion object instance set, and the modality mapping table.

[0060] In this embodiment, the multimodal object instance set D_objects = {O_text: set of text object instances, O_visual: set of visual object instances, O_embodied: set of embodied object instances, O_fused: set of fused object instances, Modality_map: modality mapping table}. All object instances have a unique identifier object_id, type label (text / visual / embodied / fused), and core attributes (name / category / feature / location, etc.).

[0061] Optionally, the multimodal data further includes: task context description information;

[0062] Based on the multimodal data samples, determine the global semantic anchor point identifiers for each object in the multimodal object instance set, including:

[0063] The semantic core vector is determined based on the text object name and object type of each object in the multimodal object instance set.

[0064] In this embodiment, core semantic concepts, i.e., the essential semantic attributes of the object, are extracted from the text object name (object_name) and object type (class_label) of each object. Knowledge graph tools (such as WordNet semantic network and ConceptNet commonsense knowledge base) are used to query the object's hypernym (e.g., "cup" → "container" → "vessel" → "artificial object") and essential attributes (e.g., the essential attributes of "cup" include "can hold liquid", "has an opening", and "can be held"), generating a semantic core vector (semantic_core). Specifically, object_name is encoded into an initial vector using a language model such as BERT. The knowledge graph is then queried to obtain 3-5 hypernyms and 5-10 essential attributes. These concepts and attributes are also encoded and concatenated with the initial vector, then dimensionality reduced using a fully connected layer to obtain the semantic_core vector. This vector captures the abstract semantics of the object, making objects of the same type semantically similar in different scenarios.

[0065] Based on the visual feature vector and 3-DOF pose of each object in the multimodal object instance set, determine the morphological invariant feature vector.

[0066] In this embodiment, the shape descriptor and geometric features of each object are extracted from its visual feature vector (visual_feature). These features are relatively stable under different viewpoints, lighting conditions, and backgrounds. Shape descriptor operators (such as HOG histogram of oriented gradients to extract edge and contour direction distribution, and SIFT scale-invariant features to extract keypoint descriptors) and color histograms (normalized RGB color distribution) are used to extract the shape-invariant feature vector (shape_invariant). For embodied objects, additional 3D geometric features (such as the object's length-width-height ratio, the first principal component of PCA in the point cloud, and surface curvature distribution) are extracted. These features are then L2 normalized and concatenated into a shape-invariant feature vector. This vector makes objects with similar appearances (such as cups of different colors but the same shape) appear close in shape space, supporting appearance-based clustering of similar objects.

[0067] The functional feature vector is determined based on the contextual text fragments and task context description information of each object in the multimodal object instance set.

[0068] In this embodiment, the affordance of an object is inferred from the context text snippet and the task context description information task_context, i.e., how the object can be used and what interactive capabilities it possesses. Using a predefined functional ontology library (such as verb frames in WordNet and semantic roles in VerbNet) or a pre-trained functional recognition model, action verbs associated with the object (e.g., "cup" associated with "pour", "drink", "hold") and interaction methods (e.g., "can hold liquid", "can be held", "can be placed") are identified and encoded into a functional feature vector function_role (128-dimensional one-hot or multi-hot encoding, where each bit represents the presence or absence of a function); this vector makes objects with similar functions (e.g., "cup" and "bowl" both have the function of "holding") close in the functional space.

[0069] Based on the semantic core vector, morphological invariant feature vector, and functional feature vector, the intrinsic feature vector of each object in the multimodal object instance set is determined.

[0070] In this embodiment, the object's intrinsic feature vector is obtained by concatenating three types of features: semantic_core, shape_invariant, and function_role. semantic_core is the semantic core vector, shape_invariant is the shape-invariant feature vector, and function_role is the functional feature vector.

[0071] In this embodiment, the intrinsic feature vector `Intrinsic_features` encodes the semantic, morphological, and functional intrinsic attributes of an object. Based on the intrinsic feature vector of each object in the multimodal object instance set, a global semantic anchor identifier for each object is determined.

[0072] In this embodiment, each object instance generates a globally unique semantic anchor identifier (GID). This identifier remains constant in all contexts, all scenarios, and all times, serving as the object's "semantic ID card" to ensure that the same semantic object is always identified as the same entity in different observations, different environments, and different tasks.

[0073] Optionally, after determining the set of multimodal object instances and the global semantic anchor identifier, context projection identifier, and embodied state identifier of each object in the set of multimodal object instances based on the multimodal data samples, the method further includes:

[0074] Based on the global semantic anchor identifier and intrinsic feature vector of each object in the multimodal object instance set, determine the multimodal object instance set and global semantic anchor vector index after adding the global semantic anchor identifier.

[0075] In this embodiment, a bidirectional index mapping is established to support fast retrieval. For example, given a GID, all object instances with that GID can be retrieved; given an object ID, its GID can be retrieved. Specifically: Forward mapping from GID to object: Maintaining a hash table data structure GID_to_Objects, where the key is the GID string and the value is a list of object IDs [object_id_1, object_id_2, ...]; Reverse mapping from object to GID: Maintaining a reverse index hash table Object_to_GID, where the key is the object ID string object_id and the value is the corresponding GID string; This mapping is a one-to-one relationship (each object instance corresponds to only one GID), supporting fast retrieval of its global anchor identifier from an object instance; GID metadata table: Recording metadata information for each GID for easy management and analysis, maintaining a dictionary structure GID_meta. The data key is the GID, and the value is a metadata dictionary containing: semantic_label (the human-readable name of the object category or concept represented by the GID, such as "cup-type object", extracted from the object name corresponding to the cluster center or determined by a voting mechanism), representative_features (the cluster center vector or average feature vector of the GID, used for subsequent similarity calculation between new objects and the GID), creation_timestamp (the time when the GID was first generated), num_instances (the total number of object instances currently belonging to the GID, dynamically updated), and typical_contexts (the types of scenarios in which the GID object often appears, such as ["kitchen", "office", "laboratory"]).

[0076] In this embodiment, the multimodal object instance set D_objects_with_GID, after adding the global semantic anchor identifier, has the following new fields compared to the multimodal object instance set: {GID: global semantic anchor identifier, GID_cluster_ID: category ID, Intrinsic_features: intrinsic feature vector}.

[0077] In this embodiment, a global object mapping can also be obtained: Global_index={GID_to_Objects: a hash table mapping GIDs to a list of objects, Object_to_GID: a reverse hash table mapping objects to GIDs, GID_metadata: a dictionary of GID metadata, GID_vector_index: a GID vector index}. All index data structures support dynamic updates (adding new objects, deleting objects, and updating metadata).

[0078] Based on the set of multimodal object instances after adding global semantic anchors and the context projection identifiers of each object, determine the set of multimodal object instances after adding context projection identifiers, the context index, and the scene relationship graph set.

[0079] The set of multimodal object instances after adding context projection identifiers includes: context projection identifiers, context feature vectors, and a scene relationship graph to which the object belongs.

[0080] In this embodiment, the multimodal object instance set after adding the context projection identifier has the following new fields compared to the multimodal object instance set after adding the global semantic anchor identifier: {CID: context projection identifier, Context_features: context feature vector, Scene_graph_ref: scene relationship graph}.

[0081] In this embodiment, the context index Context_index = {CID_to_GID: a hash table mapping CID to GID, GID_to_CIDs: a reverse index of the GID to CID list, CID_metadata: a dictionary of CID metadata}.

[0082] Based on the embodied state identifiers of each object in the multimodal object instance set and the multimodal object instance set after adding context projection identifiers, determine the multimodal object instance set after adding embodied state identifiers and the hierarchical identifier mapping table.

[0083] In this embodiment, the multimodal object instance set after adding the embodied state identifier has the following new fields compared to the multimodal object instance set after adding the context projection identifier: {EID: embodied state identifier string, Physical_state: physical state dictionary, State_history: historical state list (initially containing 1 state snapshot)}; at the same time, the hierarchical identifier mapping table Hierarchy_map={EID_to_CID: EID to CID mapping dictionary, CID_to_EID: reverse mapping of CID to EID list, CID_to_GID: CID to GID mapping dictionary, GID_to_latest_EID: fast index of GID to latest EID}.

[0084] Based on the hierarchical identifier mapping table, the set of multimodal object instances with added embodied state identifiers, and the set of scene relationship graphs, a spatiotemporal semantic graph is established.

[0085] In this embodiment, objects and their multi-level relationships in the three-layer identification system are integrated into a unified spatio-temporal semantic graph (STSG) data structure. This graph structure simultaneously represents the semantic attributes (concepts, categories, functions), spatial topology (location, adjacency, containment), temporal evolution (state transitions, trajectories), and causal relationships (task dependencies, event triggers) of objects, supporting subsequent complex operations such as cross-modal reasoning, state tracking, and relationship querying.

[0086] In this embodiment, firstly, node attributes are defined and constructed. Each object is treated as a graph node of STSG. The node attributes contain three layers of identification information and multimodal features to form a comprehensive representation of the object. Specifically, the node uses node_id as a unique identifier in the graph, and the node attributes contain complete three layers of identification: GID global semantic anchor identifier, CID context projection identifier, and EID embodied state identifier (if it is a non-embodied object, the EID field is null). The node also contains: semantic layer feature attributes (semantic_features), which record the semantic information of the object: object_name (object name, such as "cup" or "table"), class_label (class label, such as "cup" or "table", from the detection model or ontology library), semantic_embedding (semantic embedding vector, dimension 512 or 768, from the semantic encoder in step 102, encoding the abstract semantic concept of the object), taxonomic_path (classification path, such as ["object", "vessel", "container", "cup"] indicating the position of the object in the ontology hierarchy), and attribute list (key-value pairs such as {color: "red", material: "ceramic", capacity: "300ml"} recording the descriptive attributes of the object). The visual layer feature attribute `visual_features` (this field exists if the object is observed in the visual modality, otherwise it is null) records the object's appearance information: `visual_embedding` (visual embedding vector, dimension 2048 or 768, from the depth features extracted by the visual encoder in step 101, such as ResNet or ViT), `bbox` (format {x1,y1,x2,y2} representing 2D image coordinates or {center,dimensions} representing 3D spatial coordinates), `mask` (binary image or list of polygon vertices, accurately describing the object's outline), `appearance_descriptor` (containing color histogram, such as a 256-bin histogram of each RGB channel, texture features such as Gabor filter response, shape features such as HOG directional gradient or contour moments), and viewpoint information (camera position and orientation, used for multi-view object recognition).The embodied_features attribute (this field exists only for embodied objects) records the object's physical and task information: Physical_state (a dictionary of physical states, including complete physical states such as position, orientation, and velocity as defined in step 104), pose_3d (a precise 6D pose, represented by a combination of position and orientation), bounding_box_3d (a 3D bounding box, represented by the coordinates of 8 vertices or center + size + rotation), and point_cloud_summary (if storing the complete point cloud would occupy a lot of space, it can store representative downsampled points such as 1000 points or statistical features of the point cloud such as centroid, principal orientation, and size). The context metadata attribute `context_metadata` records the context information when an object is observed: `scene_id` (scene ID to which the object belongs), `scene_type` (scene type, such as "kitchen" or "office"), `task_id` (task ID to which the object belongs), `task_type` (task type, such as "grasping" or "navigation"), `agent_id` (agent ID to which the object is observed or manipulated), `timestamp` (time in Unix milliseconds when the node was created or last updated), `observation_modality` (a list such as ["text", "visual", "embodied"] indicating which modalities the object is observed in), and `confidence_score` (0-1 representing the reliability of the node information, a fusion of confidence scores from various modalities). Then, edge relationships are defined and constructed. Define various types of edges in STSG to represent different relational dimensions between objects and construct a complete relational network. Specifically, define and construct spatial edges to represent spatial topological relationships between objects, reusing the spatial edge spatial relationship edges in Scene_graph.Edge attributes include: source_node_id (the starting point of the relationship), target_node_id (the ending point of the relationship), relation_type (a spatial relationship type, enumerated as "on", "in", "near", "above", "below", "left", "right", "front", and "behind", totaling 9 basic spatial relationships), distance (the Euclidean distance between the centroids of the two objects, in meters), relative_position (the 3D vector pointing from the source object to the target object, normalized to indicate direction), spatial_confidence (0-1 indicating the reliability of the spatial relationship determination, based on sensor accuracy and occlusion), and observation_timestamp (the time recorded for the spatial relationship). Spatial edges are directed edges (from source to target), but bidirectional edges can be established to represent symmetrical relationships (e.g., "A near B" simultaneously establishing "B near B"). A); Spatial edges are dynamically updated (when the spatial relationship changes due to changes in the object's position, old edges are deleted and new edges are added, such as when an object moves from "on desktop" to "in drawer", the "on" edge with the desktop is deleted and the "in" edge with the drawer is added). Semantic edges are defined and constructed: representing the semantic relationships between objects, reusing the semantic relationship edges in the Scene_graph constructed in step 103.Edge attributes include: source_node_id, target_node_id, relation_label (semantic relation label with 13 common semantic relations, including "use", "contain", "part_of", "made_of", "support", "belong_to", "produce", "consume", "enable", "cause", "prevent", "similar_to", and "opposite_to"), and relation_strength (0-1 indicating the closeness of the relationship, e.g., "cup contains water" has a strength of 0.9, "cup similar" has a strength of 0.9, and "cup similar" has a strength of 0.9). _to bowl" intensity 0.6), evidence source (the basis for inferring the relationship of the string record, such as "extracted from the text 'drink water from a cup'" or "inferred from the knowledge graph WordNet"), bidirectional bidirectional label (boolean value, True indicates the relationship is symmetric such as similar_to, False indicates the relationship is directed such as use); semantic edges can come from three channels: (1) text parsing (extracting relationship triples from context_snippet or task_context through dependency syntax and semantic role labeling), (2) knowledge base inference (querying external knowledge graphs such as ConceptNet or domain ontology to obtain prior relationships between objects), (3) visual scene graph (using scene graph generation models such as Factorizable Net to generate object relationships from images); semantic edges are relatively stable (do not change with physical state) but can change with task context (such as "cup" has a contain relationship with "water" in the drinking task and has a contain relationship with "sink" in the cleaning task). Temporal Edges Definition and Construction: Connecting the state nodes of the same object at different times, representing the temporal evolution trajectory of the object.Edge attributes include: source_node_id (the node ID at the previous time step), target_node_id (the node ID at the next time step), time_delta (the time interval between the two nodes, in seconds), state_transition_type (the state transition type, enumerated values ​​such as "position_change", "orientation_change", "grasped", "released", "task_update", "velocity_change", "stationary", etc.), and transition_description (a textual description of the state change, such as "from position (1.2, The state transition is recorded as follows: (0.5, 0.8) moves to (1.5, 0.5, 0.8), a distance of 0.3m. The `trigger_event` records events that cause state transitions, such as "robotic arm grasping action," "external force collision," or "natural gravity fall." The `causality` field is an optional field; if the causal relationship of the state change can be identified, `cause_node_id` records the object or agent node ID that caused the change. Temporal edges are directed edges (pointing from an earlier node to a later node) forming a chain structure (all historical states of the same object are linked together by temporal edges). Temporal edges are extracted from the object's `State_history` list (temporal edges are established between two adjacent state snapshots in the list). The `source_node_id` of the edge is the node corresponding to the previous snapshot, and the `target_node_id` is the node corresponding to the next snapshot. Hierarchical edges are defined and constructed by connecting nodes corresponding to different layers of the same object, representing the hierarchical relationship (GID→CID→EID inclusion relationship).Edge attributes include: source_node_id (higher-level identifier node ID, such as the representative node corresponding to GID), target_node_id (lower-level identifier node ID, such as the node corresponding to CID or EID), hierarchy_type (enumeration values ​​"GID_to_CID", "CID_to_EID", "GID_to_EID" indicate the levels traversed), contains (boolean True indicates that the higher-level identifier contains the lower-level identifier), and abstraction_level (integer representing the level difference, 1 for GID→CID, 2 for GID→EID). The CID→EID is 1); hierarchical edges are directed edges (from abstract higher levels to concrete lower levels) and usually do not form cycles (the identifier hierarchy is a directed acyclic graph (DAG) structure); hierarchical edge construction method: create a virtual aggregation node (node_type="GID_aggregate") for each GID, representing the abstraction of all instances of that GID, establish GID_to_CID edges from this aggregation node to all CID nodes, and establish CID_to_EID edges from each CID node to its subordinate EID nodes; hierarchical edges support top-down query (query all instance nodes given a GID) and bottom-up aggregation (trace back to the GID anchor point given an EID node). Next, graph structure integration and metadata construction are performed. The node set and edge set are integrated into a complete STSG graph data structure, recording the graph's metadata information, specifically: Graph data structure definition: STSG adopts a property graph model, and nodes and edges can carry arbitrary key-value pair attributes. The graph structure includes: nodes (node_id as the key and node attribute dictionary as the value), edges (edge ​​type as the key, such as "spatial" as the value, and a list of edges of that type as the value), graph_id (unique graph identifier, UUID format, such as "graph-12345678-abcd-efgh"), and graph_type ("STSG" indicates spatiotemporal semantic graph).Graph metadata records: The global metadata of the graph, `graph_metadata`, includes: `scene_id` (scene ID if the graph corresponds to a single scene, null if it spans multiple scenes), `task_id` (task ID if the graph corresponds to a single task), `spatial_extent` (spatial extent if the graph contains embodied objects, such as the cube boundary `{min_x, max_x, min_y, max_y, min_z, max_z}`), `temporal_range` (time range, recording the earliest and latest timestamps in the graph, such as `{start_time, end_time}`), `num_nodes` (total number of nodes), `num_edges` (total number of edges, categorized by type, such as `{spatial:150, semantic:80, temporal:200, hierarchical:300}`), and `modality_coverage` (a list, such as `["text", "visual", ...`). "embodied" indicates which modalities the objects in the graph come from, creation_timestamp is the graph creation time, last_update_timestamp is the graph last update time, and version is the version number (an integer incrementing each time the graph structure is modified for easy version management).

[0087] In this embodiment, the spatiotemporal semantic graph STSG = {nodes: node dictionary, edges: edge dictionary {spatial_edges spatial relationship edges, semantic_edges semantic relationship edges, temporal_edges temporal sequence edges, hierarchical_edges hierarchical edges}, graph_id graph unique identifier, graph_type graph type label, graph_metadata global metadata}. This graph structure integrates the three-layer identifiers, multimodal features, and multi-level relationships of all objects in D_objects_with_EID, providing a unified data foundation for subsequent stages.

[0088] Optionally, based on the intrinsic feature vector of each object in the multimodal object instance set, a global semantic anchor identifier for each object is determined, including:

[0089] The hash value calculated by normalizing the intrinsic feature vector of each object in the multimodal object instance set is used as the global semantic anchor identifier for each object.

[0090] In this embodiment, Strategy A - Deterministic Hash Generation: The intrinsic feature vector of Intrinsic_features is normalized (L2 normalization makes the vector magnitude 1, eliminating the influence of numerical scale) and then the hash value is calculated. The hash digest hash_digest of the normalized feature vector (converted to a byte sequence) is calculated using the Secure Hash Algorithm SHA256 (which outputs a 256-bit digest). The first 64 bits are truncated as the GID. The GID format is "GID-" prefix + 16-bit hexadecimal string (64 bits of binary are converted to 16 hexadecimal characters), for example, "GID-a3f7c9e2b1d84f06". The advantage of this strategy is that objects with the same intrinsic features generate completely identical GIDs, which facilitates deduplication (e.g., two observations of a "cup" with completely identical features will generate the same GID and be automatically merged). The disadvantage is that small differences in features (e.g., changes in lighting cause slight differences in visual features) can lead to completely different GIDs, resulting in false splits. It is suitable for scenarios that require strict deduplication, such as document object management and knowledge base entity fusion.

[0091] or,

[0092] Based on the snowflake algorithm, a global semantic anchor point identifier is generated for each object.

[0093] In this embodiment, the strategy C-Snowflake algorithm is extended to generate: the improved distributed ID generation algorithm Snowflake is used to generate a 64-bit integer as the GID, ensuring that a globally unique ID can be generated without centralized coordination in the distributed system. The 64-bit structure is divided into: the first bit is the sign bit (fixed to 0 to represent a positive number), bits 2-42 are the timestamp part (41 bits record the millisecond-level timestamp timestamp_ms, the offset relative to the system startup time 2025-01-01 00:00:00, usable for 69 years), bits 43-45 are the modality type part (3 bits distinguish between 8 modalities: text=001, visual=010, embodied=011, fusion=100), bits 46-55 are the object category part (10 bits encode the object category class_id, such as "cup"=0042, supporting 1024 categories), and bits 56-64 are the sequence number part (9 bits are the sequence number that increments within the same millisecond, supporting the generation of 512 IDs per millisecond); the GID generation formula is: GID = (timestamp_ms<<23)|(modality_type) <<20)|(class_id<<9)|sequence; The generated GID format is a 19-digit decimal integer (e.g., "1234567890123456789") or converted to a hexadecimal string (e.g., "GID-112A3F4B5C6D7E8"); The advantages of this strategy are that it is globally unique and conflict-free, the generation time and object category can be deduced from the GID for easy traceability and statistics, and it supports high-concurrency generation (512,000 IDs can be generated per second on a single machine). The disadvantages are that the same object observed at different times will generate different GIDs, which need to be merged by similarity matching later, and the GID has no semantic readability; It is suitable for scenarios that need to generate a large number of object identifiers in real time, such as video stream processing and multi-robot collaborative systems.

[0094] Optional, also includes:

[0095] Based on the text object name and object type of each object in the multimodal object instance set, determine the semantic embedding vector of each object in the multimodal object instance set;

[0096] Based on the semantic embedding vector of each object in the multimodal object instance set, the objects in the multimodal object instance set are clustered to obtain multiple clusters;

[0097] Based on the semantic class identifier corresponding to each cluster and the hash value of the objects in the cluster, the global semantic anchor identifier of each object is determined. The semantic class identifier is used to assist in the semantic alignment or consistency verification of the global semantic anchor identifier of the object instance, and different objects in the same cluster have different global semantic anchor identifiers.

[0098] In this embodiment, strategy B - semantic embedding clustering generation: a pre-trained object semantic encoder (such as CLIP's text encoder or a specially trained object encoder) is used to encode the text object name object_name and the text object type class_label into a semantic embedding vector semantic_embedding (dimension 512 or 768). In the embedding space, a hierarchical clustering algorithm is used to cluster all objects, grouping semantically similar objects into the same cluster. During the clustering process, the minimum cluster size min_cluster_size=3 (to avoid a single noise point forming an independent cluster) and the distance threshold cluster_selection_epsilon=0.15 are set (to control the clustering granularity, this value is optimized through the validation set). Each cluster is assigned a unique cluster center ID as the semantic class identifier (e.g., semantic class ID) for all objects within that cluster, in the format of "GID-cluster-" prefix + cluster number (e.g., "GID-cluster-00042"). Objects with the same semantic class ID are semantically equivalent (e.g., "red cup", "blue cup", "glass" may belong to the same category; "GID-cluster-00123" represents "cup-type object"). The global semantic anchor identifier for each object is obtained by concatenating the semantic class identifier corresponding to each cluster with the hash value of the objects in the cluster. Alternatively, based on the snowflake algorithm, a first identifier for each object is generated, and the first identifier is concatenated with the semantic class identifier corresponding to each cluster to obtain the global semantic anchor identifier for each object.

[0099] The advantages of this strategy are that semantically similar but different objects are automatically grouped into the same category ID to reduce redundancy (which aligns with human cognition that "although they are different colors, they are all cups"). It also supports the rapid assignment of new objects to categories by calculating their distance from existing cluster centers (if the distance is less than the threshold, they are added to the existing cluster; otherwise, a new cluster is created). The disadvantages are that it requires maintaining a mapping table of cluster center vectors, and the classification of objects at the cluster boundaries may be unstable. It is suitable for scenarios that require semantic induction, such as cross-scene object recognition and knowledge graph construction.

[0100] In this embodiment, depending on the application scenario requirements (such as whether GID readability is required, whether strict uniqueness is required, and whether incremental updates are required), the above three GID generation strategies are supported and can be configured to be selected or used in combination.

[0101] In this embodiment, Strategy A (deterministic hashing generation) is used by default as the primary generation method for object instance-level GIDs. By normalizing and hashing the object's intrinsic feature vector, it ensures that each object instance has a consistent and unique global identifier across modalities and time. To enhance the semantic stability and cross-modal alignment capability of GIDs, this embodiment introduces Strategy B (semantic embedding clustering) as an auxiliary mechanism: when an object obtains an instance-level GID, its semantic embedding is mapped to the corresponding semantic cluster and associated with a semantic class ID, providing semantic consistency support (objects of the same type share the same semantic class ID, rather than sharing a GID). The introduction of Strategy B maintains the uniqueness of object instances while possessing good semantic generalization ability, scalability, and noise robustness, making it suitable for unified recognition requirements involving text, vision, and embodied multimodal collaboration. For specific scenarios requiring higher generation throughput or strict uniqueness in a distributed environment (such as financial transaction objects and medical device tracking), strategy C (snowflake algorithm extension) can be switched as needed to generate instance-level GIDs to achieve distributed globally unique identifiers in large-scale, high-concurrency scenarios.

[0102] Optionally, based on the multimodal data samples, determine the context projection identifier for each object in the multimodal object instance set, including:

[0103] Multimodal data samples are identified to obtain scene type vectors, environmental element vectors, and spatial layout feature vectors.

[0104] The scene context vector is determined based on the scene type vector, environmental element vector, and spatial layout feature vector.

[0105] In this embodiment, scene type vectors and environmental element vectors are extracted from image or point cloud data. A scene classification model (such as the Places365 scene recognition model based on ResNet supporting 365 scene categories, or a ViT-based scene classifier) ​​is used to classify the overall image (not a single object region), identifying the scene type (e.g., "kitchen", "laboratory", "outdoor street", "office"), and outputting the scene type label and its confidence score. Simultaneously, an image segmentation model (such as Semantic) is used. Semantic segmentation identifies environmental elements in a scene (such as "ceiling", "walls", "floor", "windows", "furniture", etc.) and generates a list of environmental elements (environment_elements). For embodied scenes, spatial layout features (such as room size calculated from boundary points, obstacle density, and passable area) are extracted from point cloud data. The scene type is one-hot encoded (365-dimensional vectors corresponding to 365 scene categories), the environmental elements are multi-hot encoded (50-dimensional vectors corresponding to 50 common environmental elements), and the spatial layout features (10-dimensional real vectors) are concatenated to obtain the scene context vector scene_context (total dimensions 425).

[0106] Extract the task type, task objective, and constraints from the task context description information.

[0107] The task type, task objective, and constraints are concatenated to obtain the task context vector.

[0108] In this embodiment, the task type and task objective are extracted from the task context description information. A pre-trained text classification model (such as the BERT text classifier) ​​is used to classify the task context description into task types (e.g., "grasping", "navigation", "question answering", "understanding", supporting 20 predefined task types). Named entity recognition (NER) and semantic role labeling (SRL) are used to extract the task objective (e.g., extracting the target object "cup" and target location "table" from "move the cup to the table") and constraints (e.g., "avoid collisions", "stay upright") from the task context description. The task type is encoded using one-hot encoding (20 dimensions), the task objective is encoded using BERT into a vector (768 dimensions), and the constraints are encoded using multi-label classification encoding (32 dimensions) to obtain the task context vector task_context_vec (total dimensions 820).

[0109] The observation timestamp, relative time, and time series label of the object are concatenated to obtain the time series context vector.

[0110] In this embodiment, the time information of the observed object and the task execution stage are recorded. The observation timestamp (Unix timestamp accurate to milliseconds) is extracted, and the relative time (relative_time) (the number of seconds relative to the start time of the task) is calculated. Temporal tags (such as "preparation", "execution", "verification", "completion", etc.) are labeled according to the temporal stages of task execution (defined by the task state machine), using one-hot encoding as a 4-dimensional vector. For video or continuous observation scenarios, the frame number (frame_number) or observation number (observation_index) is recorded. The observation timestamp is normalized (divided by the total task duration to obtain the relative position between 0 and 1), the relative time is normalized, and the temporal tag encoding is concatenated to obtain the temporal context vector (6 dimensions in total).

[0111] The scene context vector, task context vector, and temporal context vector are concatenated to obtain the context feature vector.

[0112] In this embodiment, the context feature vector Context_features=concat(scene_context,task_context_vec,temporal_context) is obtained by concatenation. This vector comprehensively encodes the multi-dimensional context information of the scene, task, time and agent in which the object is located, so that the same object generates different context representations in different contexts.

[0113] Construct a scene relationship diagram.

[0114] The scene relationship graph includes a set of nodes and edges. The set of nodes includes all objects in the scene. The attributes of each node include: object identifier, global semantic anchor identifier, text object name, object type, and object feature vector. The object feature vector includes: semantic feature vector, visual feature vector, and embodied feature vector. The edges include: spatial relationship edges, semantic relationship edges, and physical relationship edges.

[0115] In this embodiment, a spatial, semantic, and physical relationship network between objects in the same scene (same image, point cloud at the same time, same text paragraph) is constructed for consideration of the relational context of objects during subsequent CID generation. Spatial relationship edge construction: For objects in the embodied scene or visual scene, the relative positional relationships between objects are calculated. Iterate through all object pairs (object_i, object_j), and calculate the spatial relationship type (relation_type) (determined by geometric rules): if object_j is above object_i (with a larger y-coordinate) and the distance is less than a threshold, the relationship is "on above"; if the centroid of object_j is within the bounding box of object_i, the relationship is "in inside"; if the distance between the two is less than 0.5m, the relationship is "near"; if object_j is in front of object_i (based on the orientation of object_i), the relationship is "in front of". Similarly, define 8 spatial relationships such as "behind", "left", "right", "above", and "below". Calculate the spatial distance (Euclidean distance or Manhattan distance) and the relative orientation angle (the angle between the orientations of the two objects); generate spatial edges, each with the structure {source_id: the ID of object_i, target_id: ...}. The object_j has an ID, a relation_type (spatial relationship type), a distance (distance value), and a relative_orientation (relative orientation angle). Spatial edges are directed edges (pointing from object_i to object_j) and allow multiple edges between a pair of objects (e.g., satisfying both "near" and "on"). Semantic relationship edge construction: Extracting semantic associations between objects from the text context_snippet.Use dependency parsing tools (such as Stanford NLP and spaCy) to parse the dependency tree of the sentence and extract verb phrases and prepositional phrases involving objects; for example, in the sentence "Alice uses the cup to pour..." In the context of "water", the verb "uses" is identified to connect the subject "Alice" and the object "cup" to form a semantic relationship (Alice, use, cup). Similarly, the verb "pour" is identified to form a relationship with the instrumental case "cup" (cup, instrument_for, pour). Ten common semantic relationship types are defined (e.g., "use", "contain", "support", "belong to", "part of", "made of", "produce", "consume", "interact with", "depend on"). Extracted relation triples are mapped to these types. Semantic relationship edges (semantic_edges) are generated, each with the structure {source_id: subject object ID, target_id: object object ID, relation_label: semantic relationship label, confidence: extraction confidence (probability based on syntactic analysis)}. Semantic edges are also directed and labeled. Physical interaction relationship edge construction: In embodied scenarios, physical contact and interaction between objects are detected. Collision detection algorithms (such as AABB collision detection based on bounding boxes or distance field detection based on point clouds) are used to determine whether physical contact occurs between objects (contact is defined as a distance of less than 0.01m between the surfaces of two objects); a physics simulation engine (such as PyBullet or MuJoCo) or heuristic rules are used to determine support relationships (a support relationship is established if object_j is placed on top of object_i and object_i bears the weight of object_j) and constraint relationships (a constraint relationship is established if two objects are mechanically connected by hinges, rails, etc.); the interaction type (such as "contact", "support", "constrained", "separated") and the interaction force (if measurable, the normal force and friction force are recorded) are recorded; physical edges (physical_edges) are generated, with each edge having the structure {source_id, target_id, interaction_type, contact_points: a list of contact points, interaction_force}.A scene relationship graph, Scene_graph, is constructed by treating all objects as nodes and all relationships as edges. The node set Nodes contains all objects in the scene, and each node's attributes include {object_id, GID, object_name, class_label, features: object feature vector (a concatenation of semantic, visual, and embodied features)}. The edge set Edges is divided into three subsets {spatial_edges: list of spatial relationship edges, semantic_edges: list of semantic relationship edges, physical_edges: list of physical relationship edges}, with the structure of each type of edge as described above. The scene graph data structure uses an adjacency list (each node records its outgoing and incoming edge lists) to facilitate graph traversal.

[0116] Generate the context projection identifier of the object based on the object's global semantic anchor identifier, context feature vector, and local subgraph of the object in the scene relationship graph.

[0117] In this embodiment, a context projection identifier (CID) is generated and an index mapping is established based on the object's global semantic anchor identifier, context feature vector (Context_features), and the object's local subgraph (ego-graph) in the scene relationship graph (Scene_graph). This enables the same global semantic anchor identifier to generate different context projection identifiers in different contexts, and different global semantic anchor identifiers to generate different context projection identifiers in the same context.

[0118] In this embodiment, local subgraph feature extraction: extracting local subgraphs centered on the target object (also known as ego-graph or k-hop neighbor subgraph) from the scene relationship graph Scene_graph. Set the subgraph radius to 2 (to extract neighbor nodes and edges within 2 hops). Starting from the target object (node), use Breadth-First Search (BFS) to traverse all neighbor nodes with a distance ≤ 2, collecting these nodes and their edges to form a local subgraph (local_subgraph). Calculate the structural features of the local subgraph: the global semantic anchor identifier (GID) set `neighbor_GIDs` (sorted after deduplication to ensure order consistency), the edge type distribution (forming a 3D vector by statistically analyzing the proportions of spatial, semantic, and physical edges), the subgraph density (number of edges divided by the maximum possible number of edges), and the degree of the center node (node_degree, in-degree + out-degree). Encode these features into a local subgraph feature vector `local_subgraph_features`: the neighbor GID set is encoded into a fixed-length vector using a hash method (calculating hash values ​​from the sorted GID list), and the edge type distribution, subgraph density, and normalized node degree are concatenated. Hierarchical CID encoding: A hierarchical prefix encoding structure is used for human readability and system traceability, in the format... <gid>@<scene_type><task_type> <timestamp>Specifically: (1) The GID part directly uses the global anchor point identifier of the object (e.g., "GID-cluster-00123"); (2) The scene_type part uses the abbreviation of the scene type (e.g., "kitchen" is abbreviated as "KITCH", "office" is abbreviated as "OFFIC", with a maximum of 5 characters); (3) The task_type part uses the abbreviation of the task type (e.g., "grasping" is abbreviated as "GRASP", "navigation" is abbreviated as "NAVIG", with a maximum of 5 characters); (4) The timestamp part uses the last 10 digits of the Unix timestamp (second-level precision) converted to hexadecimal (8 hexadecimal characters); (5) The CID is obtained by concatenation, for example, "GID-cluster-00123@KITCH_GRASP_5F8A3C2D"; The advantages of this method are that the CID contains GID information, which is convenient for reverse lookup of global anchor points, contains scene and task information, which is convenient for human understanding, and the timestamp supports sorting by time. CID Uniqueness Verification and Conflict Resolution: After generating a CID, check for conflicts with existing CIDs (although the probability is extremely low, it still requires defense). Query the CID index table (built later) to check if the newly generated CID already exists. If it does, compare the corresponding GID with the context features: if the GIDs are the same and the context feature similarity > 0.95 (cosine similarity), it is determined to be a duplicate observation of the same object in a similar context, and the existing CID is reused; if the GIDs are different or the contexts differ significantly, it is determined to be a hash collision (extremely rare), and a random suffix (such as "-001") is appended to the end of the CID until there is no conflict. CID Mapping Table Construction: Establish a bidirectional mapping relationship between CIDs and GIDs, supporting "quickly querying the global anchor GID of an object by CID" and "querying the list of CIDs of an object in all contexts by GID". Forward Mapping from CID to GID: Maintain a hash table CID_to_GID, where the key is the CID string and the value is the corresponding GID string; this mapping is a many-to-one relationship (multiple CIDs can correspond to the same GID because the same object generates different CIDs in different contexts). Example of mapping representation: {"CID-GID-cluster-00123@KITCH_GRASP_5F8A3C2D ":"GID-cluster-00123","CID-GID-cluster-00123@OFFIC_NAVIG_F8A4D3E":"GID-cluster-00123"}.Reverse mapping from GID to CID list: Maintaining a reverse index GID_to_CIDs, where the key is the GID string and the value is a list of all CIDs corresponding to that GID (sorted by generation timestamp); this mapping supports querying the historical context occurrence records of an object; mapping example: {"GID-cluster-00123": ["CID-... 5F8A3C2D","CID-..._5F8A4D3E","CID-..._5F8A5E4F"]}; the list length reflects the frequency of object observation and can be used for statistical analysis. CID metadata table construction: Records complete context information for each CID for easy traceability and analysis. Maintain a dictionary `CID_metadata`, with CID as the key and the following metadata values: GID (corresponding global anchor), Context_features (complete context feature vectors used for subsequent similarity calculations), Scene_graph_id (identifier of the scene graph containing the object), creation_timestamp (CID generation timestamp), observation_count (number of times the CID has been observed, initially 1, incrementing with each observation), and last_seen_timestamp (last observation time). Metadata example: `{"CID-GID-cluster-00123@KITCH_GRASP_5F8A3C2D": {GID:"GID-cluster-00123",Context_features:[0.12,0.34,...],Scene_graph_id:"scene_001",creation_timestamp:1704067200.123,observation_count: 5,}` last_seen_timestamp:1704067800.456}}.

[0119] Optionally, based on the multimodal data samples, determine the embodied state identifier of each object in the multimodal object instance set, including:

[0120] Based on the six-degree-of-freedom attitude and physical attribute dictionary, a physical state dictionary and a historical state list are generated, wherein the historical state list includes multiple state snapshots.

[0121] Based on the physical state dictionary and the historical state list, determine the specific state identifier of each object.

[0122] In this embodiment, objects in embodied scenarios (such as robot operation, AR / VR interaction, and autonomous driving involving the physical world) generate embodied state identifiers (EIDs), recording the object's physical state (position, orientation, velocity, stability, etc.) and task state (task_role, operability, task_status, etc.), supporting the temporal state tracking and dynamic updates of objects, and establishing a complete mapping relationship between GID-CID-EID three-layer identifiers.

[0123] In this embodiment, only objects containing physical information (pose_3d, point_cluster, etc.) in the embodied object instance set O_embodied and the fused object instance set O_fused are processed.

[0124] In this embodiment, the physical state and task state are first initialized. The processing objects are the six-DOF pose (pose_6d) and the physical attribute dictionary (physical_attrs) of the embodied object. The physical state dictionary (Physical_state) is initialized to record the object's complete state at the current moment. Specifically, the position and pose state are recorded: the object's 3D position in the world coordinate system (position=(x,y,z)) is extracted from pose_6d, with the coordinate unit being meters; the world coordinate system typically uses the robot's base or a fixed point in the room as the origin, with the x-axis pointing forward, the y-axis pointing to the left, and the z-axis pointing upward (right-handed coordinate system); simultaneously, the object's 3D orientation (orientation) is extracted and represented using the quaternion (quaternion=(w,x,y,z)) (quaternions avoid gimbal lock issues and provide smooth interpolation). Alternatively, a rotation matrix (3×3 orthogonal matrix) can be used to represent the object; calculate the spatial region_id to which the object belongs, and determine which spatial region the object is located in through a spatial partitioning algorithm (such as recursively dividing the space into cubic grids using an Octree octree, or predefined semantic regions such as "desktop region", "ground region", "shelf region"), which facilitates quick subsequent querying of objects within a certain region; record the position uncertainty (position_uncertainty, a 3×3 position covariance matrix caused by sensor measurement errors) and the orientation uncertainty (orientation_uncertainty, a 3×3 orientation covariance matrix or a 4×4 quaternion covariance) for subsequent Kalman filtering of state estimation.Motion state and stability initialization: Initialize the object's velocity (velocity = (vx, vy, vz) and angular velocity (angular_velocity = (ωx, ωy, ωz)) in meters per second and radians per second, respectively. For static observations (single-frame point cloud or single image), velocity cannot be directly measured; the initial values ​​are set to zero vectors: velocity = (0, 0, 0) and angular_velocity = (0, 0, 0). For continuous observations (video stream or continuous point cloud sequence), velocity can be estimated through the position difference between adjacent frames: velocity = (position_t - position{t-1}) / Δt. Angular velocity is calculated using quaternion differences combined with the chain rule. Record the object's motion pattern (motion_pattern, initial). The initial state is marked as "stationary" and subsequently updated to "uniform", "accelerating", "decelerating", or "oscillating" based on speed. The attitude stability value, stability_score (ranging from 0 to 1 to indicate the stability of the object's attitude), is calculated using a physics simulation engine (such as PyBullet) to simulate whether the object will tip over in the current attitude: if the center of mass is projected within the support surface and the distance from the boundary is greater than the safe distance (e.g., 5cm), then stability_score = 1.0 (stable); if the center of mass is close to the support surface boundary, then it decreases linearly to 0.5 (unstable); if the center of mass is projected outside the support surface, then stability_score = 0.0 (will tip over). Physical attribute records: Extract or infer the physical parameters of the object from physical_attrs, including: mass (kilograms), estimated by volume × material density (e.g., volume of a wooden cup is 0.0002 m³ × density is 600 kg / m³ = 0.12 kg) or a typical value retrieved from the knowledge base; material_type (enumerated values ​​such as "wood", "metal", "plastic", "glass", "ceramic", "fabric"), which affects the coefficient of friction and collision behavior; and collision shape. The shape (used for physics simulation, options include "box", "sphere", "cylinder", "convex hull", and "mesh") and its parameters (such as the length, width, and height of the box, and the radius of the sphere); the friction coefficient (the surface friction coefficient μ is usually between 0.1 and 1.0, 0.15 for metal, 0.4 for wood, and 0.8 for rubber) and the restitution coefficient (the energy retention ratio after a collision is between 0 and 1, 1 for a perfectly elastic collision and 0 for a perfectly inelastic collision).The output of the physical state initialization is Physical_state={position: 3D position vector, position_uncertainty: position covariance matrix, region_id: spatial region identifier, orientation: orientation quaternion, orientation_uncertainty: orientation covariance matrix, stability_score: attitude stability score 0-1, velocity: velocity vector, angular_velocity: angular velocity vector, motion_pattern: motion pattern label, mass: mass (kg), material_type: material type, collision_shape: collider shape and parameters, friction_coefficient: friction coefficient, restitution_coefficient: restitution coefficient, timestamp: physical state record timestamp}. Then, EID generation and temporal state chain initialization are performed. A unique embodied state identifier (EID) is generated for the embodied object, and its state history chain (State_history) is initialized for temporal tracking. Simultaneously, the triggering conditions and transformation constraints for state updates are defined, specifically: EID encoding rule: a hierarchical encoding format is adopted. <cid> E <sequence>Here, CID is the context projection identifier of the object, E is the embodied identifier prefix (the first letter of Embodied), and sequence is the state sequence number of the object in the current context (within the same CID) (incrementing from 0). When the first EID is generated for the object, sequence=0. Subsequently, each time the physical state or task state changes significantly, sequence increments to generate a new EID. The temporal state chain initialization: Creates a state history list, State_history, to record all historical states of the object, supporting state backtracking and trajectory analysis. State_history is a time-ordered list, with each element being a state snapshot, state_snapshot, containing: EID (the embodied identifier of the state), Physical_state (the complete physical state dictionary at that moment), timestamp (the state record timestamp in Unix milliseconds), trigger_event (the event type that triggers the state update, such as "init", "position_change", "grasped", "task_update"), previous_EID (the EID of the previous state, or null if it is the first state), change_sum Mary (a textual description of the state change, such as "position moved from (1.2, 0.5, 0.8) to (1.5, 0.5, 0.8)" for human understanding); during initialization, State_history contains the first state snapshot: State_history = [{EID:"..._E0", Physical_state:{...}, timestamp: current timestamp, trigger_event:"init", previous_EID: null, change_summary:"object initialization"}]; each subsequent state update appends a new snapshot to the end of the list, and the list length reflects the number of state changes the object has undergone. Finally, a hierarchical tagging mapping table is constructed.A mapping relationship is established between the three layers of identifiers: GID, CID, and EID. This supports reverse lookup of identifiers from any layer to other layers and complete object information. Cross-layer synchronization rules are defined to ensure the consistency of the three layers. Specifically, a three-layer mapping relationship, `Hierarchy_map`, is constructed, containing multiple sets of bidirectional mappings: `EID_to_CID` mapping (key is an EID string, value is the corresponding CID string), recording which context each embodied state belongs to; `CID_to_EID` mapping (key is a CID string, value is a list of EIDs corresponding to that CID sorted by sequence), supporting queries of all state evolutions under a given context; `CID_to_GID` mapping (already constructed in step 103, reused here), recording which global object each context belongs to; `GID_to_latest_EID` mapping (key is a GID, value is the latest EID for that GID), supporting quick queries of the latest state of an object. Complete link tracing from EID to CID to GID can be achieved through combined queries. Example mapping: `Hierarchy_map={EID_to_CID:` {"CID-..._E0":"CID-...","CID-..._E1":"CID-..."},CID_to_EID:{"CID-...":["CID-..._E0","CID-..._E1"]} ,CID_to_GID:{"CID-...":"GID-cluster-00123"},GID_to_latest_EID:{"GID-cluster-00123":"CID-..._E1"}}.

[0125] In this embodiment, the initialization of multimodal object identification and the construction of a three-layer system were completed. Specific outputs include: a multimodal object instance set D_objects, enabling the extraction and association of cross-modal object instances; a global semantic anchor identifier (GID), assigning a globally unique identifier to each object; a context projection identifier (CID), generating an identifier for each object in a specific context to capture the object's contextual differences; and an embodied state identifier (EID), generating a state identifier and a state history chain State_history for embodied objects, enabling temporal tracking of objects in physical space. The final output is a spatiotemporal semantic graph STSG, a unified graph structure integrating all objects and relationships, containing complete node attributes (three-layer identifiers + multimodal features) and four types of relationship edges (spatial + semantic + temporal + hierarchical), ensuring the consistency and traceability of object identification across different modalities, scenarios, and times, and realizing the core mechanisms of "semantic anchor maintaining constancy, context projection capturing differences, and embodied state tracking evolution."

[0126] S120: Generate a training dataset based on the set of multimodal object instances and the global semantic anchor identifier, context projection identifier, and embodied state identifier of each object in the set of multimodal object instances.

[0127] In this embodiment, based on the multimodal object instance set and the global semantic anchor identifier, context projection identifier, and embodied state identifier of each object in the multimodal object instance set, an STSG spatiotemporal semantic graph and a multimodal object instance set with added embodied state identifiers are constructed. Training samples are sampled from the STSG spatiotemporal semantic graph and the multimodal object instance set with added embodied state identifiers to construct diverse contrastive learning sample pairs, covering object recognition needs across modalities (text-visual-embodied), across scenes (kitchen-office-outdoor), and across time sequences (states at different times). Training datasets are generated through positive sample pairs (same GID but different CID) and negative sample pairs (different GIDs with similar contexts) for subsequent model training.

[0128] In this embodiment, positive sample pair sampling (same object in different contexts) is first performed. The goal is to construct "sample pairs of the same GID under different CIDs" and train the model to learn semantic constancy (i.e., the core semantic identity of the same object remains consistent even though it appears in different scenes, tasks, and times). Specifically, this involves: cross-scene positive sample sampling: selecting GIDs with ≥2 instances from the GID_to_Objects mapping of Global_index (ensuring that the object appears in at least two different contexts). For each selected GID, obtaining a list of all CIDs corresponding to that GID from the GID_to_CIDs mapping of Context_index; filtering CID pairs with different scene types (scene_type) by parsing the CIDs. For scene type (e.g., "KITCH" vs. "OFFIC" indicates different scenes), for each pair of different scene CIDs (CID_i, CID_j), the corresponding node is queried from STSG (by CID index or by traversing the node's CID attribute), and their respective multimodal features are extracted: text features text_features_i and text_features_j (if the node has a semantic_features.object_name attribute, it is encoded into a 768-dimensional vector using a pre-trained language model, which can be a BERT model), and visual features visual_features_i and visual_ features_j (if the node has the visual_features.visual_embedding attribute, then the visual features are used directly), embodied layer features embodied_features_i and embodied_features_j (if the node has the embodied_features.Physical_state attribute, then the position, dimensions, and orientation fields are extracted and concatenated), and context features context_features_i and context_features_j (scene_type, task_type, timestamp, etc. are extracted from the node's context_metadata and encoded into a context vector); these constitute a cross-scene positive sample pair positive_pair_scene={features_i: multimodal feature dictionary of object i, features_j: multimodal feature dictionary of object j, GID: shared global anchor identifier, pair_type:"cross_scene", label:1 indicates a positive sample}; this process is repeated to sample all GIDs that meet the conditions to generate a cross-scene positive sample set positive_pairs_scene.Cross-temporal positive sample sampling: Select the states of the same object at different times from the State_history state history list of the object (e.g., the "cup" is stationary on the table at time t1, grabbed and moved at time t2, and placed in the sink at time t3), and train the model to learn the object persistence in temporal evolution. For each embodied object (node with non-empty EID), query all the temporal edges of the object from the STSG. Temporal_edges (query the outgoing edges of the temporal type of the node through the adjacency list index), and the source_node and target_node connected by the temporal edge are the temporally adjacent state node pairs; filter the node pairs with the time interval time_delta within a reasonable range (e.g., 5 seconds < time_delta < 300 seconds, too short indicates no significant change in the state, too long indicates it may be different task phases); extract the Physical_state physical states (position, velocity, etc.) and timestamps of the node pairs; form the cross-temporal positive sample pair positive_pair_temporal = {state_i: the state features at time i, state_j: the state features at time j, GID, time_delta: the time interval, state_transition_type: the state transition type, pair_type: "cross_temporal", label: 1}; generate the cross-temporal positive sample set positive_pairs_temporal. Cross-modal positive sample sampling: Select the representations of the same object under different modal observations (e.g., the text description "red ceramic cup", visual image, 3D point cloud of the same "cup"), and train the model to learn cross-modal semantic alignment. Query the source modal objects {text_id, visual_id, embodied_id} corresponding to the fused object fused_id from the modality mapping table Modality_map; extract the features of each modal object (semantic embedding of the text modality, image features of the visual modality, geometric features of the embodied modality); form the cross-modal positive sample pair positive_pair_modal = {text_features, visual_features, embodied_features, GID, pair_type: "cross_modal", label: 1}; generate the cross-modal positive sample set positive_pairs_modal.The output of positive sample pair sampling is a positive sample set: Positive_samples = {positive_pairs_scene: a list of positive samples across scenes, positive_pairs_temporal: a list of positive samples across time sequences, positive_pairs_modal: a list of positive samples across modalities}. Each list contains several sample pair dictionaries, and each sample pair contains features, labels, and meta-information. Then, negative sample pair sampling (similar contexts for different objects) is performed. The goal is to construct "sample pairs with different GIDs but similar contexts or appearances" to train the model to learn object discriminability (i.e., even if two objects appear in similar scenes or with similar appearances, the model can still accurately distinguish their semantic differences). Specifically, this involves Hard Negatives Mining: selecting semantically similar object pairs with different GIDs (e.g., "cup" and "bowl" are both containers but have different GIDs, "red cup" and "blue cup" have similar appearances but belong to different GID clusters). These hard negative sample pairs are most valuable for model training because they are easily confused.The mining strategy is as follows: (1) Screening based on semantic similarity: Use the semantic labels in GID_metadata, such as "cup-like objects" and "bowl-like objects", and calculate the cosine similarity cosine_similarity between semantic labels through a pre-trained language model (such as Sentence-BERT). Screen GID pairs with similarity within the threshold range (e.g., 0.7 < similarity < 0.9. If it is too high, it may indicate synonyms and should be grouped into the same GID; if it is too low, it indicates a large semantic difference and does not constitute a difficult sample); (2) Screening based on appearance similarity: Use the representative feature vectors in GID_metadata (Intrinsic_features from step 102) to calculate the cosine similarity between feature vectors, and screen object pairs with similar appearances but different GIDs (e.g., similar shapes but different materials, similar colors but different sizes); (3) Screening based on context co-occurrence: Query different GID objects that appear in the same scene (same scene_id) or adjacent spatial positions (spatial distance < 2m) in STSG. These objects are similar in context and prone to confusion; Randomly sample object instances from the selected candidate GID pairs (GID_i, GID_j) (randomly select an instance from the GID_to_Objects mapping), and extract their respective multi-modal features features_i and features_j; Form a difficult negative sample pair hard_negative = {features_i, features_j, GID_i, GID_j, similarity_score: similarity score, pair_type: "hard_negative", label: 0 indicates a negative sample}; Generate a set of difficult negative samples hard_negatives. Random negative sample sampling (Random Negatives Sampling): Randomly sample object pairs from different GIDs to ensure a significant GID difference (semantic similarity < 0.The random sampling strategy is as follows: Two different GIDs (GID_i ≠ GID_j) are randomly selected from the GID_to_Objects mapping of the Global_index. Their semantic_labels are checked to ensure their semantic similarity is less than a threshold (e.g., 0.3). An instance is randomly selected from the object list of each GID, and multimodal features are extracted. Random negative sample pairs are constructed: random_negative = {features_i, features_j, GID_i, GID_j, pair_type: "random_negative", label: 0}. A random negative sample set, random_negatives, is generated. Negative sample ratio control: To balance the training data, the positive-to-negative sample ratio is set to 1:2 (i.e., 1 positive sample corresponds to 2 negative samples). Difficult negative samples account for 60% of the negative samples, and random negative samples account for 40% (difficult negative samples have a higher proportion because they have greater training value). The required number of negative samples is calculated based on the total number of positive samples. If there are insufficient difficult negative samples, random negative samples are added to fill the gaps. Finally, the number of positive and negative samples is ensured to meet the set ratio. The output of negative sample pair sampling is a negative sample set `Negative_samples = {hard_negatives: list of hard negative samples, random_negatives: list of random negative samples}`, where each list contains a sample pair dictionary containing information such as features, labels, and similarity. Finally, the training dataset is integrated and partitioned. Positive and negative samples are integrated into a unified training dataset and then divided into training, validation, and test sets. This is a common industry practice and will not be elaborated upon. It generally includes dataset integration, dataset partitioning, and data loader construction, with the output being a dataset object `Training_data`, structured as: `Training_data = { "Train_set": list of training set samples, "Valid_set": list of validation set samples, "Test_set": list of test set samples, "Train_loader": training data loader, "Valid_loader": validation data loader, "Test_loader": test data loader, "dataset_split_info": { "train_samples": N_train, "valid_samples": N_valid, "test_samples": N_test, "num_GIDs": total number of GIDs, "num_positive_pairs": number of positive sample pairs, "num_negative_pairs": number of negative sample pairs, "modality_distribution": {"text": %, "visual": %, "embodied": %}}}.

[0129] In this embodiment, the training dataset Training_data can include a training set, a validation set, a test set, and a corresponding data loader. The data scale reaches hundreds of thousands of samples (assuming the system observes 100,000 object instances, with an average of 10 instances per GID, generating 100,000 pairs of positive samples and 200,000 pairs of negative samples, totaling 300,000 samples), covering diverse samples across scenes, tasks, time series, and modalities. The training dataset is used to train the model to be trained, obtaining a multimodal model.

[0130] Optionally, the model to be trained is trained to obtain a multimodal model, including:

[0131] Obtain the first model.

[0132] The first model is trained based on the training dataset to obtain a context-aware word segmenter.

[0133] In this embodiment, training the first model based on the training dataset to obtain a context-aware word segmenter includes: inputting training samples from the training dataset into the first model to obtain the hidden state and predicted probability output at each token position; determining a first loss function based on the hidden state, predicted probability, and one-hot label of each token position in the training samples; determining a second loss function based on the embedding distance of each token's context; determining a third loss function based on the text descriptions of multiple object instances identified by the same global semantic anchor; and training the first model based on the first loss function, the second loss function, and the third loss function to obtain the context-aware word segmenter.

[0134] The training model is trained based on the training dataset to obtain a multimodal model, wherein the training model includes the context-aware word segmenter.

[0135] In this embodiment, the method for training the model to be trained based on the training dataset to obtain the multimodal model can be as follows: train the first model based on the training dataset to obtain a context-aware word segmenter; generate a second model based on the context-aware word segmenter, wherein the second model includes, from input to output, a context-aware word segmenter, a Transformer encoder, a pooling layer, and a text projection layer; train the second model based on the training dataset to obtain a text encoder; train the third model based on the training dataset to obtain a visual encoder; train the fourth model based on the training dataset to obtain an embodied encoder; and generate the multimodal model based on the text encoder, the visual encoder, and the embodied encoder.

[0136] Optionally, the training dataset includes: training samples, wherein the training samples are masked text sequences, and one-hot labels for each token position in the training samples;

[0137] The first model is trained based on the training dataset to obtain a context-aware word segmenter, including:

[0138] The training samples in the training dataset are input into the first model to obtain the hidden state and predicted probability of each token position.

[0139] In this embodiment, the context-aware word segmenter adopts a multi-layer perceptron (MLP) structure, including an input layer, hidden layers, and an output layer: the input layer receives the original context features (Context_features); Layer Normalization is applied to the input Context_features to improve training stability; the first hidden layer (hidden_layer_1) maps the input to the hidden dimension (hidden_dim=512) through a fully connected layer (Linear(context_dim, hidden_dim)); the activation function uses ReLU (Rectified LinearUnit) to introduce non-linearity; the dropout layer (dropout_layer_1) (dropout_rate=0.1) randomly drops 10% of the neurons to prevent overfitting; the second hidden layer (hidden_layer_2) uses a fully connected layer (Linear(hidden_dim, ...)) to map the input to the hidden dimension (hidden_dim=512). The `hidden_dim` layer maintains its dimension and uses ReLU activation. A Dropout layer, `dropout_layer_2` (with a dropout rate of 0.1), is used. The output layer, `output_layer`, is mapped to the output dimension (`output_dim=768`, consistent with the token embedding dimension) via a fully connected layer, `Linear(hidden_dim, output_dim)`. The forward propagation computation process is as follows: `x = LayerNorm(Context_features)`, `x = ReLU(Linear_1(x))`, `x = Dropout 1(x)`, `x = ReLU(Linear_2(x))`, `x = Dropout_2(x)`, and `context_vector = Linear_out(x)`. The output `context_vector` is a tensor of `batch_size × output_dim`, encoding multi-dimensional context information such as scene type, task type, time stage, and agent identity. The conditional token embedding layer design modifies the traditional token embedding layer to accept context adjustment, dynamically generating context-sensitive token representations.The base token embedding layer is `base_embedding = nn.Embedding(vocab_size, embed_dim)`, where `vocab_size` is the size of the vocabulary of the pre-trained tokenizer, and `embed_dim` is the embedding dimension (e.g., 768). `base_embedding` takes a token ID (integer from 0 to `vocab_size-1`) as input and outputs the corresponding embedding vector (a real vector of dimension `embed_dim`). A context modulation module, `context_modulation`, is introduced to fuse the context vector into the base embedding using gated modulation: `gate = Sigmoid(Linear_gate(base_embed))`, where `Linear_gate` is a learnable fully connected layer. `contextual_embed = base_embed + gate ⊙ context_vec` implements gated fusion, where `⊙` represents element-wise multiplication. This gating mechanism allows the model to adaptively determine how much context modulation each token receives. Given an input token sequence token_ids (a batch_size × seq_len integer tensor, where seq_len is the sequence length) and context features Context_features (a batch_size × context_dim tensor), the process for calculating the conditional embedding is as follows:

[0140] 1) Basic embedding query: base_embeds=base_embedding(token_ids), which yields a tensor of batch_size×seq_len×embed_dim;

[0141] 2) Context encoding: context_vec = Context_encoder(Context_features), resulting in a batch_size × embed_dim tensor;

[0142] 3) Context vector expansion:

[0143] context_vec_expanded=context_vec.unsqueeze(1).expand(batch_size,seq_len,embed_dim) means that the context vector is copied seq_len times so that each token position can be accessed;

[0144] 4) Gated modulation: gate = Sigmoid(Linear_gate(base_embeds)) calculates the batch_size × seq_len × embed_dim gate tensor, contextual_embeds = base_embeds + gate × context_vec_expanded, to obtain the final conditional embedding;

[0145] 5) Positional Encoding Fusion: Positional encoding (position_encoding) is added to the conditional embedding to represent the token's position in the sequence (the Transformer architecture requires explicit positional information). Sinusoidal positional encoding is used: PE(pos, 2i) = sin(pos / 10000^(2i / embed_dim)) and PE(pos, 2i+1) = cos(pos / 10000^(2i / embed_dim)). The positional encoding is added to the conditional embedding: final_embeds = contextual embeddings + position_encoding, resulting in the final token representation containing semantic, contextual, and positional information. The first loss function is determined based on the hidden state output at each token position, the predicted probability, and the one-hot label at each token position in the training samples.

[0146] The second loss function is determined based on the embedding distance of each token's context.

[0147] A third loss function is determined based on the textual descriptions of multiple object instances identified by the same global semantic anchor.

[0148] The first model is trained based on the first loss function, the second loss function, and the third loss function to obtain a context-aware word segmenter.

[0149] In this embodiment, the training objective and loss function are designed, and training is executed. A multi-task learning objective is used to train the conditional tokenizer, combining three loss functions: Masked Language Model Loss, Contextual Discriminative Loss, and GID Alignment Loss. Specifically, the first loss function can be the Masked Language Model Loss function. The Masked Language Model Loss (MLM Loss) function follows the BERT pre-training paradigm, predicting the masked token under contextual conditions, training the model to understand the influence of context on word meaning. Specific implementation examples are as follows: (1) Random masking of input sequence: Randomly select 15% of the tokens from the token sequence for masking, of which 80% are replaced with a special token [MASK], 10% are replaced with random tokens, and 10% remain unchanged (to avoid the model from over-relying on the [MASK] tag); (2) Forward propagation: Input the masked token_ids and Context_features into Contextual_token_embedding to obtain contextual_embeds, and obtain the contextualized_hidden_states (batch_size×seq_len×hidden_dim) through the Transformer encoder (such as BERT's 12-layer Transformer); (3) Predict the masked tokens: predict the token distribution at each position by using the classification head prediction_head (fully connected layer Linear(hidden_dim,vocab_size)+Softmax) predicted_probs(batch_siz×seq_len×vocab_size); (4) Calculate the loss: calculate the cross-entropy loss only for the masked positions: cross_entropy(predicted_probs[masked_positions],ground_truth_tokens[masked_positions]), and denote the loss as L_mlm; use this loss to train the model to predict words based on the context, so that the conditional embedding captures the context semantics.

[0150] The second loss function can be the context discrimination loss function. Context discrimination loss requires that the embeddings of the same token in different contexts be distinguishable, and trains the model to learn context sensitivity. For the embeddings of the same token in different contexts, let it be... Define cosine distance: , the context discrimination loss is: , where is the minimum expected distance (e.g., 0.5). This loss promotes the separation of token embeddings in different contexts in the vector space. A specific implementation example is:

[0151] (1) Construct contrast samples: For the same token (e.g., "Apple"), sample two samples in different contexts (e.g., a technology conference context_tech and a fruit market context_food); (2) Calculate the conditional embedding: embed_tech = Contextual_token_embedding("Apple", context_tech), embed_food = Contextual_token_embedding("Apple", context_food); (3) Calculate the cosine distance: distance = 1 - cosine_similarity(embed_tech, embed_food), the cosine similarity ranges from [-1, 1], and the cosine distance ranges from [0, 2]; (4) Context discrimination loss: L_context_discrim = max(0, margin - distance), where margin is the minimum expected distance (e.g., 0.5). If the embedding distance between two contexts < margin, a loss is generated to promote their separation; this loss ensures distinguishable representations for different contexts.

[0152] The third loss function can be the GID alignment loss function, GID Alignment Loss: It is required that objects with the same GID have similar token embeddings in different contexts (maintaining semantic constancy), and the training model learns global semantic consistency. For two instances with the same GID, let the corresponding sentence-level embeddings be , then it is defined as: , the GID alignment loss is: This loss ensures that the same semantic object maintains similar representation in different contexts. Specific implementation examples are as follows: (1) Construct positive sample pairs: For two object instances with the same GID (e.g., "cup" in kitchen scene1 and office scene2), extract their respective text descriptions text1 and text2 (e.g., "red ceramic cup" and "blue glass cup"), and context1 and context2; (2) Calculate sequence-level embedding: After tokenizing the text, input it into Contextual_token_embedding to obtain the token sequence embeddings, and obtain the sentence-level embedding sent_embed1 and sent_embed2 by average pooling mean_pooling(embeds) or [CLS] token representation; (3) Calculate similarity: similarity=cosine_similarity(sent_embed1,sent_embed2); (4) GID alignment loss: L_gid_align=1-similarity, expecting the similarity between different instances with the same GID to be close to 1 (cosine distance close to 0); This loss ensures that the semantic anchor remains stable in different contexts. The total loss function is L_total = λ1 × L_mlm + λ2 × L_context_discrim + λ3 × L_gid_align. The weight coefficients are optimized based on the validation set performance and set as follows: λ1 = 1.0 (MLM loss maintains a large weight for the main task), λ2 = 0.5 (context discrimination has a moderate weight for the auxiliary task), and λ3 = 0.8 (GID alignment has a high importance and a large weight). During training, the changing trend of each loss term is monitored, and if a certain loss term does not converge, the corresponding weight is adjusted.

[0153] Finally, model training and evaluation are performed, which is a common practice in the industry and will not be elaborated upon.

[0154] In this embodiment, the context-aware word segmenter can dynamically adjust the representation of the token according to the context, and has semantic discrimination ability in the word segmentation stage (that is, the same word generates different token embeddings in different contexts), which solves the problem that traditional word segmenters cannot distinguish polysemous words (such as "Apple" representing a company in a technology context and a fruit in a food context).

[0155] In this embodiment, a context-aware word segmenter is trained using a training dataset generated from a multimodal object instance set and the global semantic anchor identifier, context projection identifier, and embodied state identifier of each object in the multimodal object instance set. This enables the model to learn a representation space that allows for "aggregation of the same object across scenes and separation of different objects within scenes," achieving robust object recognition in different modalities and contexts. Through training data construction and context-aware word segmenter training, the model acquires semantic constancy and context sensitivity.

[0156] Optionally, the model to be trained is trained based on the training dataset to obtain a multimodal model, including:

[0157] A second model is generated based on the context-aware word segmenter.

[0158] The second model, from input to output, includes: a context-aware word segmenter, a Transformer encoder, a pooling layer, and a text projection layer.

[0159] In this embodiment, the context-aware word segmenter is used as the embedding layer of the second model to provide context-sensitive input features for subsequent model training.

[0160] The second model is trained based on the training dataset to obtain the text encoder;

[0161] The third model is trained based on the training dataset to obtain the visual encoder;

[0162] The fourth model is trained based on the training dataset to obtain the embodied encoder;

[0163] A multimodal model is generated based on the text encoder, visual encoder, and embodied encoder.

[0164] In this embodiment, encoders are designed for the three modalities of text, vision, and embodied to map the original input into feature vectors of fixed dimensions. A projection layer is designed to project the features of different modalities onto a unified semantic space, and an updatable anchor vector is learned for each GID. Specifically, the text encoder design uses a context-aware word segmenter as the embedding layer of the text encoder. The specific structure is as follows: The input layer receives the text sequence text_input (a string or a list of token_ids) and context features Context_features; the context-aware token segmenter generates conditional token embeddings (seq_len × embed_dim) through Contextual_token_embedding(token_ids, Context_features); the Transformer encoder (a 12-layer Transformer Encoder, each layer containing a Multi-Head Self-Attention + Feed-Forward Network) encodes the token embeddings to obtain contextualized_hidden_states (seq_len × hidden_dim, hidden_dim=768); the pooling layer averages the sequence dimensions using mean pooling to obtain the sentence-level representation text_features=mean. (contextualized_hidden_states, dim=seq_len), output dimension 768; the text projection layer projects text features onto a unified semantic space (projection_dim=512) through a fully connected layer text_projection=Linear(768,projection_dim), outputting text_embed (a 512-dimensional vector); the text encoder can reuse parameters from a pre-trained BERT (transfer learning) or be initialized from the Contextual_tokenizer in step 202. Visual encoder design: visual features are extracted using a convolutional neural network or a Vision Transformer.The specific structure is as follows: The input layer receives the image visual_input (RGB image tensor H×W×3); the preprocessing layer resizes the image to a fixed size (e.g., 224×224) and normalizes it (subtracting the mean and dividing the standard deviation); the backbone network selects ResNet-50 (outputting a 2048-dimensional feature vector) or ViT-Base (outputting a 768-dimensional [CLS] token representation) to extract global visual features visual_features; the visual projection layer projects onto a unified space (projection_dim=512) through a fully connected layer visual_projection=Linear(2048 or 768, projection_dim), outputting visual_embed (a 512-dimensional vector); the visual encoder initializes with ImageNet pre-trained weights (transfer learning utilizes knowledge from large-scale image classification tasks), and during training, the first few layers of the backbone network can be frozen (fine-tuning) or all layers can be unfrozen (end-to-end training). Embodied encoder design: uses a point cloud neural network to extract 3D geometric features. The specific structure is as follows: The input layer receives the point cloud embodied_input (an N×3 point cloud coordinate matrix, where N is the number of points, e.g., 1024 points) and the physical state (a 12-dimensional vector containing position, orientation, etc.); the point cloud encoder uses PointNet++ (Set Abstraction layer recursive downsampling and feature aggregation) to extract local and global geometric features pointcloud_features (a 1024-dimensional global descriptor); the physical state encoder encodes the physical state into physical_features (a 256-dimensional vector) using MLP (Linear(12,128)+ReLU+Linear(128,256)); the feature fusion layer concatenates pointcloud_features and physical_features to obtain embodied_features=concat(pointcloud_features, The physical_features layer (1280-dimensional vector) projects onto a uniform space (projection_dim=512) through a fully connected layer embodied_projection=Linear(1280, projection_dim), outputting embodied_embed (512-dimensional vector). The embodied encoder can be initialized from ModelNet or ShapeNet pre-trained weights or trained from scratch with random initialization.Multimodal fusion strategy: For the fusion object (which has text, visual and embodied features at the same time), a gating fusion strategy is designed to integrate multimodal embeddings: The gating network (MLP) is used to learn the fusion weights based on the input features: gate=Softmax(MLP(concat(text_embed, visual_embed, embodied_embed))), fused_embed=gate[0]×text_embed+gate[1]×visual_embed+gate[2]×embodied_embed. The gating mechanism allows the model to adaptively select important modalities. In the case of missing modalities, modality_mask indicates which modalities are available. The gating network only calculates and normalizes the weights for the available modalities.

[0165] In this embodiment, object representations of different modalities (text, visual, embodied) and different contexts are mapped to a unified semantic space, while maintaining GID anchor alignment (representations with the same GID are close to the anchor vector) and CID distinguishability (representations with different contexts retain differences), thereby achieving the representation learning goal of "aggregating the same object across scenes and separating different objects within scenes".

[0166] In this embodiment, the loss function and training process of the multimodal model are as follows:

[0167] Contrastive Learning Loss (InfoNCE): InfoNCE loss is used to optimize "aggregation of the same GID and separation of different GIDs" by learning discriminative representations through comparison of positive and negative samples. The calculation method is as follows: Given the anchor sample embedding anchor_embed (batch_size×embed_dim), positive sample embedding positive_embed (batch_size×embed_dim, same GID but different context), negative sample embeddings negative_embeds (batch_size×num_neg×embed_dim, different GIDs), temperature parameter temperature=0.07 (controls the smoothness of the distribution); calculate the positive sample similarity pos_sim=sum(anchor_embed×positive_embed,dim=-1) / temperature is the inner product of anchor and positive divided by temperature (batch_size×1); calculate the negative sample similarity neg_sim=matmul(anchor_embed.unsqueeze(1), negative_ embeds.transpose(1,2)) / temperature is the inner product of the anchor and all negatives (batch_size×num_neg); the exponential function exp_pos_sim=exp(pos_sim) and exp_neg_sim=sum(exp(neg_sim), dim=-1) are applied; the InfoNCE loss L_contrastive= -mean(log(exp_pos_sim / (exp_pos_sim+exp_neg_sim+ε))), where ε=1e-8 to prevent division by zero; this loss expects the similarity of positive samples to be higher than that of negative samples, and optimizes the representation by maximizing the probability of positive samples; the weight coefficient λ2=0.8 (contrastive learning is an important auxiliary objective).

[0168] Triplet Loss (Triplet Loss with Margin): This method uses triplet loss to explicitly constrain the anchor-positive distance to be less than the anchor-negative distance by at least the margin, enabling hard negative sample mining. The calculation is as follows: Given a triplet (anchor_embed, positive_embed, negative_embed), each triplet is a batch_size × embed_dim tensor; the distances pos_dist = ||anchor_embed - positive_embed||² (the squared Euclidean distance between the anchor and positive, batch_size × 1) and neg_dist = ||anchor_embed - negative_embed||² (the squared distance between the anchor and negative, batch_size × 1); the triplet loss L_triplet = mean(max(0, The loss is calculated as pos_dist - neg_dist + margin), where margin is a marginal hyperparameter (e.g., 0.2). The loss is only applied when pos_dist + margin > neg_dist (i.e., only samples that violate the constraints are penalized). This loss optimizes hard samples through hard marginal constraints and complements the InfoNCE loss (InfoNCE is a soft constraint based on probability, while Triplet is a hard constraint based on distance). The weight coefficient λ3 = 0.5 (the triplet loss serves as a supplementary constraint).

[0169] Cross-Modal Alignment Loss: This method requires that the embeddings of different modalities of the same object be similar, and learns a modality-invariant representation. The calculation method is as follows: Given the text embedding (text_embed), visual embedding (visual_embed), and embodied embedding (embodied_embed) of the same GID object (all are batch_size × embed_dim); calculate the text-visual alignment loss_tv = 1 - mean(cosine_similarity(text_embed,visual_embed)), where the cosine similarity ranges from -1 to 1, and the loss ranges from 0 to 2; calculate the visual-embodied alignment loss_ve = 1 - mean. (cosine_similarity(visual_embed,embodied_embed)); calculate text-embodied alignment loss_te=1-mean(cosine_similarity(text_embed,embodied_embed)); cross-modal alignment loss L_cross_modal=(loss_tv+loss_ve+loss_te) / 3 is the average of the three pairs of modal alignment losses (if a pair of modalities is missing in the batch, this term is skipped); this loss ensures the consistency of the representation after multimodal fusion, enabling the model to learn modality-shared semantics rather than modality-specific surface features; weight coefficient λ4=0.6 (cross-modal alignment is of high importance).

[0170] The total loss function L_total=λ1×L_contrastive+λ2×L_triplet+λ3×L_cross_modal, and the weight coefficients {λ1=0.5, λ2=0.2, λ3=0.3} can be determined by tuning on the validation set through grid search.

[0171] In this embodiment, a "context encoder-gated modulation" conditional tokenizer is constructed to dynamically generate heterogeneous token embeddings for the same word in different scenarios / tasks; combined with "contrast / triple / cross-modal" multi-objective loss, instances with the same GID are pulled toward the anchor point and instances with different GIDs are pushed away, realizing cross-scenario aggregation of the same object and separation of different objects within the same scenario.

[0172] Finally, model training and evaluation are performed, which is a common practice in the industry and will not be elaborated upon. During training, the values ​​and gradients of each loss term are monitored. If a certain loss term is too large or the gradient explodes, the weights or learning rate are adjusted. The gradient of the total loss is backpropagated to all parameters of the MultiModal_Encoder (encoder, projection layer, fusion module).

[0173] In this embodiment, the multimodal model can map objects of different modalities and contexts to a unified semantic space while maintaining GID anchor alignment and contextual discriminability, providing a high-quality representation learning foundation for subsequent cross-scene object recognition.

[0174] In this embodiment, the multimodal model possesses the following key capabilities: context sensitivity (generating different representations of the same object in different contexts (scenes / tasks)), semantic constancy (maintaining semantic consistency of the same GID object in different contexts), cross-modal alignment (consistent text / visual / embodied trimodal representations), and the ability to robustly identify and track objects in complex and dynamic environments.

[0175] Optional, also includes:

[0176] Acquire target multimodal data;

[0177] The target multimodal data is input into the multimodal model to obtain the target query vector;

[0178] Query the global semantic anchor point GID vector index to obtain the target GID sequence associated with the target query vector;

[0179] The target GID is determined based on the target multimodal data, the target GID sequence, and the spatiotemporal semantic graph.

[0180] By using a multimodal model, real-time object recognition and continuous tracking can be achieved across modalities, scenarios, and tasks. The specific steps are as follows:

[0181] Multimodal object encoding and matching. Newly observed object instances are encoded into query vectors, and candidate GIDs are quickly located using GID vector indexing, narrowing the search range for subsequent exact matching and achieving fast retrieval with O(logN) time complexity (N is the total number of GIDs). The input is the new observed object O_query (containing multimodal data and context information). Its detailed implementation is as follows:

[0182] First, query object encoding is performed. A multimodal model is used to encode the preprocessed multimodal input into a unified query vector `query_embedding`. Specifically: if `O_query` contains a text description `text_desc` (e.g., "red ceramic cup" or "cup on the table"), text processing is performed: special characters are removed, converted to lowercase, a context-aware token segmenter is used to generate a sequence of integer `token_ids` and an attention mask, and the text length is truncated or padded to a fixed length; if `O_query` contains visual input (RGB image or video frame), image processing is performed: resizing, normalization, etc. The data is then converted to tensor format (C×H×W). If O_query contains embodied_input (point cloud and physical state), 3D data processing is performed: point cloud downsampling, point cloud normalization, physical state extraction, orientation processing, size stitching, etc., and finally converted to tensor format (N×3 point cloud coordinates, 12-dimensional physical state vector). Context features (Context_features) are extracted from the observation environment of O_query (using the context encoding method of step 103), and finally the complete context features (Context_features) are obtained.

[0183] The preprocessed input `Preprocessed_query={text_input,visual_input,embodied_input,Context_features}` is input into the multimodal model for forward propagation. Each modality encoder outputs a vector representation or `None` (if the modality is unavailable), and `fused_embed` is the fused representation (integrated by the gated fusion module). The output of the query object encoding is the query vector `query_embedding`. By default, the fused representation `query_embedding=fused_embed` is used, which is suitable for multimodal fusion object queries. If there is only a single modality input, the corresponding single modality representation is used (e.g., if only visual input is used, then `query_embedding=visual_embed`). If it is necessary to retain the independent information of each modality, `query_embedding= concat([text_embed,visual_embed,embodied_embed])` can be used for concatenation.

[0184] Next, fast vector retrieval and candidate filtering are performed. The constructed GID vector index GID_vector_index is used for efficient nearest neighbor search and filtering of high-quality candidates, such as using FAISS query, which is a common industry practice and will not be elaborated here.

[0185] The final output is a list of candidate GIDs, Candidate_GIDs_enriched. Each candidate contains {GID, similarity_score, confidence, semantic_label, num_instances, typical contexts}. `semantic_label` is the semantic label (e.g., "cup-like object"), `num_instances` is the number of instances (the total number of observed object instances for this GID), and `typical_contexts` is a list of typical contexts (the types of scenarios where this GID frequently appears). This provides high-quality candidates for subsequent precise matching (narrowing the search space from all GIDs to Top-K candidates through fast retrieval).

[0186] The candidate GIDs returned by the fast retrieval are subjected to multi-dimensional precise matching. Fine-grained comparisons are performed using semantic, visual, geometric, contextual, and causal features to determine the best-matching GID or identify it as a new object. First, multi-dimensional feature extraction and similarity calculation are performed. Fine-grained features are extracted from the query object O_query and representative instances of each candidate GID, and similarity scores are calculated dimension by dimension. Specifically: Feature extraction and alignment: Multi-dimensional features are extracted from O_query for precise matching: Semantic features (semantic_features) extract the object name (object_name, if there is a text description, NER is used), class label (classifier prediction is used if there is visual input), and attribute list (such as color, material, size, etc., extracted from text description or inferred from visual features); Visual detail features (visual_detail_features) extract local features (SIFT keypoints, color histogram, texture features Gabor filter response, shape, etc.) in addition to the global visual_embedding. The outline (Hu moment) is used for fine-grained matching; geometric features (if it is a embodied object) include bounding box dimensions (length, width, and height), volume, centroid, principal axes (calculated via PCA), and symmetry score (evaluating the geometric symmetry of the object); context matching features include scene type, task type, spatial location (region ID), and timestamp; encapsulated as a query feature dictionary Query_features={semantic, The functions `visual_detail`, `geometric`, and `context_match` are used to extract corresponding features from each candidate GID's representative instances. For each representative instance, the functions `semantic_features`, `visual_features`, `embodied_features`, and `context_metadata` are extracted. A weighted average of the features from all representative instances of the candidate GID (with higher weight for recent observations) is calculated to obtain the aggregated features `Candidate_aggregated_features`.The matching degree is comprehensively evaluated by calculating similarity scores for query features and candidate features dimension by dimension. This includes semantic similarity calculation, comparing the semantic feature similarity between the query and candidate (such as object name, category matching, attribute matching, etc.), visual similarity calculation, comparing the visual features between the query and candidate (such as global feature matching, local feature matching, color similarity, shape similarity, geometric similarity <size similarity, volume similarity, shape similarity, symmetry similarity, etc.>), and contextual similarity (such as scene matching, task matching, spatial proximity, temporal proximity). Specific similarity algorithms can use industry-standard algorithms, which will not be elaborated here. The output of multi-dimensional feature extraction and similarity calculation is a similarity matrix, Similarity_matrix. The behavior candidate GID column represents the similarity dimension (semantic / visual / geometric / contextual / causal), and each element is the similarity score (between 0 and 1) of that candidate in that dimension. Then, a comprehensive matching decision is made. Multi-dimensional similarity is combined into a single matching score, and candidates are ranked by score. When multiple candidate scores are close, an additional ambiguity resolution strategy is used to determine the best match. This is a common industry practice, such as weighted comprehensive scoring and candidate ranking, ambiguity detection and resolution mechanisms, and final matching decisions, which will not be elaborated upon here. The result is denoted as Matching_result, which includes the GID identifier of the matched GID (or None), confidence_final (final confidence score 0-1), all_candidates (a list of all candidates including their scores, rankings, and dimensional similarity details for subsequent analysis), dimension_scores (matching scores for each dimension {semantic_similarity, visual_similarity, geometric_similarity, context_similarity}), and explanation_text (a readable matching explanation, such as "Based on visual appearance (similarity 0.92) and scene matching (kitchen), identified as a GID-cluster-00123 cup object"), etc.

[0187] The final output is the matching result Matching_result, which achieves high-accuracy object recognition through multi-dimensional precise matching (semantic / visual / geometric / context).

[0188] In this embodiment, the GID vector index (FAISS / HNSW) is first used to filter out the Top-K candidates in milliseconds; then, the semantic, visual, geometric, contextual, and causal similarity of the candidates are calculated in sequence. Combined with ambiguity resolution and spatiotemporal consistency verification, high-precision matching with O(logN) complexity is achieved.

[0189] This embodiment implements a complete process for cross-scene object recognition, specifically including fast GID retrieval and accurate GID matching, which enables continuous identification and tracking of objects in dynamic environments.

[0190] Furthermore, this embodiment proposes a continuous optimization method for the object identification model based on feedback loops. This method dynamically updates the GID anchor space and embedding model parameters by collecting identification feedback from practical applications (explicit user annotations and implicit task execution results), thereby achieving adaptive learning and continuous performance improvement of the system. Specifically:

[0191] Automatic quality assessment and feedback collection mechanism for matching: Automatic quality assessment is performed on each object recognition result, calculating a confidence score (generated as a 0-1 scalar based on comprehensive matching score, candidate score difference, dimensional consistency, etc.), a multi-dimensional consistency index (standard deviation of each dimension's score; the smaller the value, the more consistent the dimensions, supporting the match), a contextual reasonableness index (checking the reasonableness of the matching result in the current scenario and task), and a temporal consistency index (checking tracking stability in continuous observation scenarios). Recognition quality levels are categorized as excellent (overall_quality ≥ 0.9), good (0.75-0.9), average (0.6-0.75), and poor (< 0.6). Simultaneously, explicit user feedback is collected: A GUI interface or API is provided to accept user correct / incorrect / uncertain annotations, encapsulated as structured feedback data {feedback_id, query_object_id, predicted_GID, correct_GID, is_correct, user_confidence, timestamp}; feedback reliability is ensured through majority voting or expert verification mechanisms. Collect implicit behavioral feedback: Monitor the task execution results after object recognition (such as robot grasping success rate, navigation accuracy), infer recognition quality from task success rate (success rate >80% infers correct recognition, success rate <30% infers possible recognition error), and record the reasons for task failure (incorrect object position, size mismatch, object non-existence) to assist in root cause diagnosis. Perform hard sample mining: Identify low-quality matches (overall_quality <0.6), frequently erroneous objects (error rate of a certain GID >30%), new scene challenges (accuracy in zero-sample scenes is significantly lower than in known scenes), and boundary cases (ambiguous samples with a score difference of <0.1 between Top-1 and Top-2), and prioritize their use for model optimization.

[0192] Online fine-tuning method for embedding models: Incremental learning of multimodal embedding models using feedback data: Online training data preparation, converting positive and negative feedback samples into training formats (positive sample pairs, negative sample pairs), oversampling difficult samples (weight × 3) to increase training frequency, mixing 20% ​​of the original training data to prevent catastrophic forgetting, and applying data augmentation (geometric transformation, color jitter, point cloud rotation, text rewriting) to expand the data. Monitoring the actual running performance of the model after fine-tuning (real-time accuracy, average confidence), if the performance degrades (accuracy below 85% for 1 consecutive hour), automatically rolling back to the previous stable version or baseline version; A / B testing is used to verify the effect of controversial updates (50% of traffic uses the old version, 50% uses the new version, and statistical tests are used to determine whether the new version is significantly better).

[0193] In practical applications, depending on specific business needs, the system may have more evaluation and optimization functionalities. These include, for example, evaluation and optimization of the balance of GID anchor point spatial distribution, evaluation and optimization of the discriminative power of CID context encoding, evaluation and optimization of the continuity of EID temporal chains, evaluation and optimization of the relational integrity of STSG graph structures, and evaluation and optimization of modal contribution for multimodal feature fusion, etc., which cannot all be listed here. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding customized developments and modifications based on this invention, all of which fall within the protection scope of the appended claims.

[0194] Furthermore, this embodiment proposes a transparent object recognition decision-making method based on interpretability analysis. By generating human-readable decision explanations, visualizing object evolution trajectories, and analyzing the root causes of failures, it makes the system's decision-making process transparent and traceable, improving user trust and system maintainability. Specifically, it includes a matching decision explanation generation mechanism: generating detailed inference step descriptions for each object recognition. This includes: a hierarchical decision reasoning step, extracting the complete inference chain from the Matching_result: Step 1, Vector Retrieval (description: "10 candidate GIDs were retrieved based on semantic embedding similarity"); Step 2, Multi-dimensional Matching (description: "Candidate GID-cluster-00123 has a semantic similarity of 0.91, a visual similarity of 0.88, a geometric similarity of 0.75, a contextual similarity of 0.82, and a causal similarity of 0.68"); Step 3, Comprehensive Score (description: calculation formula match_score = 0.25 × semantic + 0.30 × visual + 0.20 × geometric). +0.15×context+0.10×causal); Step 4, Ambiguity Resolution (if this occurs, it explains "Top-1 and Top-2 scores are close to 0.89 vs 0.85, ambiguity is resolved through spatiotemporal proximity, the object's location is only 0.3m from the nearest observation point of GID-00123 and 1.2m from GID-00124"); Step 5, Final Decision (it explains "the overall judgment is GID-cluster-00123 cup-type object, confidence level 89%"). Key matching factor hierarchy: sorted by dimension_scores to identify the main matching criteria (e.g., visual appearance similarity 0.92 is the highest), list auxiliary factors (semantic label consistency 0.85, context matching 0.78), label negative factors (low geometric size similarity 0.45 may indicate size differences), and explain the use of special mechanisms (e.g., ambiguity resolution, difficult negative sample mining). Natural language interpretation generation uses template filling to convert structured reasoning into natural language: "The identified object is GID-cluster-00123 (cup-type object), with a confidence level of 89%. The main criteria are: high visual similarity (0.92) and consistent semantic labels (0.85). This object is common in kitchen scenes and is located close to a similar object observed 0.5 seconds ago (0.3m), thus it is determined to be the same object being continuously tracked. Note: Low geometric similarity (0.45), the actual size may differ from typical instances." Avoid using percentages to express confidence levels in technical terms; highlight key information with bolding or color coding.Uncertainty explanations: If the confidence level is <0.7, explicitly state "Note: The identification confidence level is low (65%), manual confirmation is recommended"; if there are candidates with similar scores, state "Candidate GID-00124 has a high score of 0.82, and the two are easily confused (main difference: different colors, red vs. blue)"; if the Top-1 score is moderate, state "It may also be a new object that the system has not seen before (similarity is only 0.68)"; provide follow-up suggestions such as "It is recommended to obtain a clearer image for re-identification" and "It is recommended to manually label the sample for model improvement". Failure Case Analysis and Improvement Mechanism: Provides tools to help analyze and identify the root causes of failure cases: failure case collection and classification, screening of prediction error samples where predicted_GID ≠ true_GID, classification by error type (intra-class confusion, such as misclassifying a cup as a bowl, inter-class confusion, such as misclassifying a cup as a chair, misclassification of new objects, missed detection of new objects), and classification by severity (high confidence error, confidence > 0.8 but the most severe prediction error, medium confidence error, low confidence error), prioritizing the analysis of high confidence errors (model is confident but incorrect) and high-frequency errors (the same error occurs repeatedly). Error root cause diagnosis: For each failure case, diagnose the root cause: Feature quality diagnosis (check for image blur (high blur_score), insufficient lighting (low brightness), and sparse point cloud (low point_density). If the feature quality is poor, mark root_cause="low quality input" and suggest "improving data acquisition conditions"); Modality missing diagnosis (if there is only text and no visual data, and the GID mainly relies on visual differentiation, mark root_cause="key modality missing" and suggest "supplementing visual observations"); Similar object confusion diagnosis (if the anchor similarity between true_GID and predicted_GID is >0.85, mark root_cause="similar objects are difficult to distinguish" and suggest "enhancing discriminative feature training"); New scene challenge diagnosis (if scene_type is a zero-sample scene, mark root_cause="new scene distribution offset" and suggest "collecting training data for this scene"); Model defect diagnosis (if this type of error occurs frequently, such as all red objects being misclassified, mark root_cause="model is not sensitive to color features" and suggest "targeted enhancement training").Error pattern clustering and systemic problem identification: Feature vectors (true_GID, predicted_GID, scene_type, error type, scores of each dimension, input feature quality) are extracted from failed cases. K-means or HDBSCAN clustering is used. For each cluster, common features are analyzed to identify systemic problems (e.g., cluster 1 pattern "poor robustness to outdoor lighting" - all cases are outdoor scenes with large lighting variations and low visual scores; cluster 2 pattern "difficulty in fine-grained differentiation of red containers" - all cases are red cups vs. red bowls with high visual scores and low geometric scores). Clusters are sorted by size to prioritize the resolution of problems with a large impact. Improvement suggestions are generated based on failure case analysis, resulting in tiered improvement recommendations: Data level (e.g., if the failure rate is high in outdoor scenes, suggest "adding 500+ outdoor scene samples"; if red objects are frequently confused, suggest "adding data augmentation to enhance color contrast"; if blurred images fail frequently, suggest "filtering low-quality input or training a deblurring module"); Model level (e.g., if the visual encoder scores low in certain scenarios, suggest "fine-tuning the visual encoder backbone"; if geometric similarity discrimination is poor, suggest "enhancing geometric feature extraction, such as adding curvature symmetry features"; if generalization in new scenarios is poor, suggest "meta-learning or domain adaptation methods"); Process level (e.g., if there are many misjudgments in low-confidence cases, suggest "lowering the decision threshold or adding a manual verification step"; if multimodal fusion does not play a role, suggest "adjusting fusion weights or improving the gating mechanism"). The core value of this interpretable analysis method lies in: transparent decision-making, by generating human-readable explanations of the reasoning steps (from vector retrieval → multi-dimensional matching → comprehensive scoring → ambiguity resolution → final decision), users can clearly understand "why the system identifies the object as this GID," and the basis and contribution of each reasoning step can be traced, avoiding "black box" decision-making; diagnosable problems, through root cause diagnosis of failure cases (five categories: feature quality, modality loss, similar object confusion, new scene challenges, and model defects) and error pattern clustering (identifying systemic problems such as "poor robustness to outdoor lighting" and "difficulty in fine-grained differentiation of red containers"), scattered identification errors are summarized into structured problem patterns, accurately locating the weak links in the system; and guideable improvement, based on failure case analysis, generating specific improvement suggestions at the data, model, and process levels, forming a closed-loop optimization mechanism of "evaluation → diagnosis → improvement → re-evaluation," enabling the system to learn from failures and continuously evolve. This method is particularly suitable for high-risk application scenarios, which have extremely low tolerance for identification errors and require that system decisions be interpretable, traceable, and auditable. Domain experts can understand the system's judgment logic and make final decisions based on their professional knowledge, thereby rapidly improving the system's robustness in specific scenarios.

[0195] This embodiment can migrate the three-layer system from "objects" to entities such as "events / behaviors / video clips" by replacing the identifier generation rules and feature extraction module, and reuse STSG and multimodal fusion engine to achieve rapid domain adaptation in zero-sample scenarios.

[0196] Furthermore, this embodiment also proposes a method for generating RAG search terms based on the evaluation results of the object identification system. This method analyzes the diagnostic results of failed cases, specifically generates knowledge retrieval keywords, and supplements missing or erroneous domain knowledge from external knowledge bases, enhancing the system's professionalism and accuracy. Specifically, the implementation is as follows: Search term generation rules: For GID layer defects (such as poor anchor point representativeness and ambiguous semantic definitions): core concept words (such as "medical device" and "surgical tool") are extracted from the semantic_label or representative_features of the problem GID, and search terms (such as "medical device + standard classification" and "surgical tool + identification features") are generated by combining the application domain_type. Complete object classification standards or discriminant features are obtained by searching a professional ontology library. For CID layer defects (such as missing contextual features and inaccurate scene coding): search terms are generated from the scene type (scene_type) and task class... Semantic tags are extracted from the task_type (e.g., "operating room scene lacks lighting feature encoding" → extract "operating room + lighting standard"), generating professional terminology search terms (e.g., "operating room environment + lighting requirements" "medical scene + light source configuration specifications"), and the standard environmental feature definition is obtained by searching the scene knowledge base; Addressing EID layer deficiencies (e.g., incomplete state transition rules, missing physical constraints): state machine design keywords are extracted from the definitions of Physical_state and Task_state (e.g., "robotic arm grasping state transition"), generating search terms (e.g., "robot grasping + state machine model" "manipulator + grasp"). For FSM (Focusing on Multimodal Learning), retrieve standard state transition specifications and physical constraints from the robotics knowledge base; for multimodal fusion defects (such as low modal alignment and unreasonable fusion weights): extract problem modal pairs from the modal alignment evaluation results (e.g., "text-embodied alignment is only 0.45" → extract "text description + 3D geometry + alignment method"), generate search terms (e.g., "language grounding in 3D" "text-to-shape alignment methods"), and retrieve advanced cross-modal alignment algorithms from the multimodal learning literature library; for STSG graph defects (such as sparse relation edges and broken causal chains): extract graph structure keywords from relation types and problem descriptions (e.g., "spatial relation edges are missing" → extract "spatial relation + inference rules"), generate search terms (e.g., "spatial relationship ontology" "scene graph generation methods"), and retrieve standard relation extraction and inference rules from the knowledge graph construction method library.

[0197] Search Result Application Strategy: Retrieved knowledge base content is used as candidate supplementary knowledge. The most relevant knowledge fragments are selected through semantic similarity matching (using Sentence-BERT to calculate the similarity between the query and the search result, with a threshold ≥ 0.8). The search results are then integrated into the optimization process: For GID layer optimization: Retrieved object classification criteria or discriminant features are used as prior knowledge for anchor point re-clustering, guiding semantic similarity calculation and cluster center initialization. For CID layer optimization: Retrieved scene feature definitions are added to the input layer of the context encoder, expanding the Context_features dimension or replacing inaccurate scene encodings. For EID layer optimization: Retrieved state transition specifications are used as expansion rules for transition_triggers, supplementing missing triggering conditions and constraints. For multimodal fusion optimization: Retrieved alignment algorithms are used as new fusion strategy candidates, replacing existing gating fusion mechanisms or adjusting modality weight calculation methods. For STSG graph optimization: Retrieved relation inference rules are used as input to the graph completion algorithm, automatically inferring missing relation edges (e.g., inferring "above" relation from "on" and "support" relations). The object identification optimization process can be enhanced by RAG (Retrieval-Augmented Generation), which utilizes external professional knowledge bases to supplement the domain knowledge that the system may not have, thereby improving the professionalism and accuracy of optimization measures. It is particularly suitable for object recognition scenarios in professional fields (such as medical diagnosis, industrial inspection, and autonomous driving).

[0198] Furthermore, to further improve the adaptability, robustness, and scalability of the object identification system in this embodiment, the following improvements can be made based on existing technological innovations:

[0199] Improvement 1: Dynamic Threshold Adaptive Mechanism. The original fixed thresholds (such as anchor similarity threshold threshold_merge≥0.92, context similarity threshold threshold_context≥0.8) are replaced with a dynamic threshold mechanism. This mechanism automatically adjusts the judgment criteria based on application scenario characteristics (such as the number of object categories, scenario complexity, and real-time requirements) and the statistical distribution of historical recognition data, improving the system's scenario adaptability. For example, for high real-time scenarios (such as autonomous driving), the number of search candidates K can be reduced from 10 to 5 to accelerate inference; for high-accuracy scenarios (such as medical diagnosis), the matching confidence threshold can be increased from 0.85 to 0.95 to ensure reliability.

[0200] Improvement 2: Incremental Learning and Experience Replay Mechanism: Establish a long-term memory for recognition feedback, storing the (query_features, matched_GID, feedback, performance_metrics) data pairs from each recognition into an experience replay buffer. Through experience replay technology, historical difficult samples and current feedback samples are mixed and sampled during online learning. Meta-learning algorithms are used to continuously optimize the model's rapid adaptability, enabling the system to learn from historical failure cases and avoid repeating errors, thereby improving the recognition success rate and robustness under long-term operation.

[0201] Improvement 3: Cross-Domain Portable Architecture: The "GID-CID-EID" three-layer identifier system is extended to non-object recognition scenarios (such as video understanding, event detection, and behavior analysis). Adaptation can be achieved quickly by replacing the identifier generation rules and feature extraction modules. For example, in video understanding, GID represents the event type (e.g., "goal" or "foul"), CID represents the context of the event (e.g., "match phase" or "on-field situation"), and EID represents the temporal evolution of the event (e.g., the action chain "pass → shot → goal"). By reusing the STSG graph structure and multimodal fusion framework, a cross-domain spatiotemporal event tracking system can be quickly built, improving the system's scalability and universality in different AI applications, intelligent monitoring, and sports analysis scenarios.

[0202] Improvement 4: Neural-Symbol Hybrid Reasoning: Building upon the neural network-based representation learning of this invention, a symbolic reasoning module can be introduced to enhance interpretability and controllability. For example, semantic and causal edges in the STSG graph can be converted into first-order logic rules (e.g., "on(X,Y)∧fragile(X)→handle_carefully(X)"). Symbolic deduction can then be performed using a logic reasoning engine (e.g., Prolog, Answer Set Programming). The deduction results can be used as constraints to guide the neural network's recognition decisions (e.g., automatically adjusting the grasping strategy when the item is identified as fragile). This achieves complementary advantages between neural perception and symbolic reasoning, improving the system's performance in complex tasks requiring explicit reasoning (e.g., robot task planning, industrial fault diagnosis).

[0203] It should be noted that the following alternative solutions can also be adopted:

[0204] Semantic-Hash GID uses a pre-trained large model to perform a single SHA-3 hash on the vector concatenated from text, vision, and 3D geometry, directly using it as the global ID. Advantages: extremely simple implementation, no clustering, distributed and conflict-free; Disadvantages: extremely sensitive to noise, even slight changes in perspective generate new IDs, making merging impossible and leading to severe fragmentation.

[0205] Incremental Entity Linking (IELTS) treats each observation as a candidate entity and links it to the global knowledge graph in real time. If the confidence level falls below a threshold, a new node is created. Advantages: High interpretability; RDF / OWL toolchains can be directly reused. Disadvantages: Depends on human-made ontologies and schemas, resulting in high update latency; visual-3D features are difficult to integrate into the graph, easily leading to "visual islands."

[0206] Contrastive learning combined with a single-center memory bank maintains only one momentum-updated center vector for each object. New observations with a cosine distance <τ are merged; otherwise, a new center is created. Advantages: constant memory usage and O(1) retrieval; Disadvantages: irreversible center drift and inability to split after erroneous merging; lack of explicit alignment for "multimodal centers" and low cross-modal recall.

[0207] An end-to-end Transformer unified ID generator (ID-Transformer) directly tokenizes text, images, and point clouds and inputs them into the Transformer. The decoder outputs a 64-bit discrete ID, trained using differentiable hashing. Advantages: Completely end-to-end, gradients are propagable; Disadvantages: Requires a massive number of labeled "same object" triples, training is unstable; gradient approximation in the discrete hash layer leads to performance loss. Zero-shot CLIP+ReID tracking (CLIP-ReID Baseline) uses a CLIP visual encoder to obtain global features, then cascades a person / object ReID network for temporal association, tracking the ID as the feature's nearest neighbor. Advantages: Deployable with zero samples, code is readily available; Disadvantages: Lacks explicit semantic anchors, appearance changes after long-term occlusion cause ID switching; 3D geometry and text are only used for initialization, unable to provide loopback correction.

[0208] This embodiment differs from traditional methods that use only a single ID or static embedding to represent objects. For the first time, it decomposes object identification into a three-layer collaborative structure of "Semantic Anchor (GID) - Context Projection (CID) - Embodied State (EID)". Among them, GID provides global uniqueness across scenarios, CID captures contextual differences in semantic-causal space through hierarchical encoding, and EID records spatiotemporal evolution in physical-task space through state chain. The three are dynamically related, which fundamentally solves the problems of "identity drift of the same object in different scenarios" and "semantic confusion of different objects", allowing object identification to be traced across modalities and aggregated across scenarios.

[0209] This embodiment introduces context-aware tokenizers and embeddings to address the issue of "semantic discriminative power in representation learning." Unlike traditional tokenizers that map the same words to fixed codes and traditional embeddings that focus on local semantic features, this embodiment constructs "context-conditional tokenization." The tokenizer layer can generate adjustment vectors through a context encoder, dynamically adjusting the token embedding based on the scene type (e.g., "technology conference," "fruit market") and task role (e.g., "discussion object," "purchase target"). This allows "Apple" to be mapped as a "company entity" in a technology context and as a "fruit category" in a food context. By modeling the temporal evolution of objects in the event stream (e.g., the causal path of "cup" from "stationary on the table" → "grabbed" → "moved to the sink"), representation learning naturally possesses semantic constancy, context sensitivity, and causal attribution.

[0210] Existing technologies mostly employ single-modal static representations (such as plain text embeddings that cannot perceive physical changes) or simple multimodal splicing (such as CLIPs that only align text and images but do not model task roles), lacking explicit dynamic modeling of the physical space and task state. This embodiment uses a "Spatiotemporal Semantic Graph (STSG)" data structure to simultaneously embed objects into the semantic space, embodied space, and task space, and achieves dynamic updates and collaborative reasoning across the three spaces, significantly improving the coherence and accuracy of cross-modal and cross-task object tracking.

[0211] This embodiment extracts object instances from multimodal data (text descriptions, image sequences, video streams, point cloud data, etc.) and generates a three-layered structured identifier for each object: "Global Semantic Anchor (GID) - Context Projection Identifier (CID) - Embodied State Identifier (EID)". By clarifying the generation rules, boundary definitions and mapping relationships of each layer of identifiers, a unified object representation is provided for subsequent cross-scene learning and recognition, avoiding the problems of "different modal objects cannot be associated" and "the same object is identified as different entities in different scenes" in traditional methods.

[0212] The technical solution of this embodiment determines a set of multimodal object instances and the global semantic anchor identifier, context projection identifier, and embodied state identifier of each object in the set of multimodal object instances based on multimodal data samples; and generates a training dataset based on the set of multimodal object instances and the global semantic anchor identifier, context projection identifier, and embodied state identifier of each object in the set of multimodal object instances. The training dataset is used to train the model to be trained to obtain a multimodal model that can achieve robust recognition and interpretable traceability of objects in different modalities, different scenarios, and different tasks.

[0213] Example 2

[0214] Figure 2 This is a schematic diagram of a cross-scene object marking device provided in an embodiment of the present invention. This embodiment is applicable to cross-scene object marking scenarios. The device can be implemented using software and / or hardware methods and can be integrated into any device that provides cross-scene object marking functionality, such as… Figure 2 As shown, the cross-scene object marking device specifically includes: a determination module 210 and a generation module 220.

[0215] The determination module is used to determine the set of multimodal object instances and the global semantic anchor identifier, context projection identifier, and embodied state identifier of each object in the set of multimodal object instances based on the multimodal data samples.

[0216] The generation module is used to generate a training dataset based on the multimodal object instance set and the global semantic anchor identifier, context projection identifier and embodied state identifier of each object in the multimodal object instance set. The training dataset is used to train the model to be trained to obtain a multimodal model.

[0217] The above-described products can perform the methods provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects for performing the methods.

[0218] Example 3

[0219] Figure 3 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0220] like Figure 3 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0221] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0222] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as object tagging methods across scenes.

[0223] In some embodiments, the cross-scene object tagging method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the cross-scene object tagging method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the cross-scene object tagging method by any other suitable means (e.g., by means of firmware).

[0224] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0225] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0226] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0227] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0228] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0229] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0230] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0231] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the cross-scene object marking method according to any embodiment of the invention.

[0232] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0233] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.< / sequence> < / cid> < / timestamp> < / gid>

Claims

1. A cross-scene object tagging method, characterized in that, include: Based on the multimodal data samples, determine the set of multimodal object instances and the global semantic anchor identifier, context projection identifier, and embodied state identifier of each object in the set of multimodal object instances; A training dataset is generated based on the set of multimodal object instances and the global semantic anchor identifier, context projection identifier, and embodied state identifier of each object in the set of multimodal object instances. The training dataset is used to train the model to be trained to obtain the multimodal model.

2. The method according to claim 1, characterized in that, The multimodal data samples include: text sequences, image sequences, video streams, and point cloud sequences; Based on the multimodal data samples, determine the set of multimodal object instances, including: The text sequence is processed to obtain a set of text object instances, wherein the set of text object instances includes: text object identifier, text object name, text object type, coreference list, semantic role, semantic feature vector, and context text fragment; The image sequence and the video stream are processed to obtain a set of visual object instances, wherein the set of visual object instances includes: visual object identifier, bounding box, visual object type, visual feature vector, position coordinates and image identifier; The point cloud sequence is processed to obtain a set of embodied object instances, wherein the set of embodied object instances includes: embodied object identifier, bounding box, point cloud cluster, embodied object type, six-degree-of-freedom pose, physical attribute dictionary, embodied feature vector and sensor timestamp; Based on the set of text object instances, the set of visual object instances, and the set of embodied object instances, a set of fusion object instances and a modality mapping table are generated. The modality mapping table is used to represent the mapping relationship between fusion objects and modal objects. The modal objects include any one of text objects, visual objects, and embodied objects. The set of fusion object instances includes: fusion object identifier, source modality corresponding to the fusion object, and fusion attribute information. The fusion attribute information includes: text object name, object type, visual feature vector, 3-DOF pose, context text fragment, and physical attribute dictionary. A multimodal object instance set is generated based on the text object instance set, the visual object instance set, the embodied object instance set, the fusion object instance set, and the modality mapping table.

3. The method according to claim 2, characterized in that, The multimodal data also includes: task context description information; Based on the multimodal data samples, determine the global semantic anchor point identifiers for each object in the multimodal object instance set, including: The semantic core vector is determined based on the text object name and object type of each object in the multimodal object instance set; Based on the visual feature vector and 3-DOF pose of each object in the multimodal object instance set, determine the morphological invariant feature vector; The functional feature vector is determined based on the contextual text fragments and task context description information of each object in the multimodal object instance set; Based on the semantic core vector, morphological invariant feature vector, and functional feature vector, determine the intrinsic feature vector of each object in the multimodal object instance set; Based on the intrinsic feature vector of each object in the multimodal object instance set, determine the global semantic anchor identifier of each object.

4. The method according to claim 3, characterized in that, Based on the intrinsic feature vector of each object in the multimodal object instance set, determine the global semantic anchor identifier for each object, including: The hash value calculated by normalizing the intrinsic feature vector of each object in the multimodal object instance set is used as the global semantic anchor identifier for each object. or, Based on the snowflake algorithm, a global semantic anchor point identifier is generated for each object.

5. The method according to claim 4, characterized in that, Also includes: Based on the text object name and object type of each object in the multimodal object instance set, determine the semantic embedding vector of each object in the multimodal object instance set; Based on the semantic embedding vector of each object in the multimodal object instance set, the objects in the multimodal object instance set are clustered to obtain multiple clusters; Based on the semantic class identifier corresponding to each cluster and the hash value of the objects in the cluster, a global semantic anchor identifier for each object is determined. The semantic class identifier is used to assist in the semantic alignment or consistency verification of the global semantic anchor identifier of the object instance, and different objects within the same cluster have different global semantic anchor identifiers.

6. The method according to claim 3, characterized in that, Based on the multimodal data samples, determine the context projection identifier of each object in the multimodal object instance set, including: Multimodal data samples are identified to obtain scene type vectors, environmental element vectors, and spatial layout feature vectors; The scene context vector is determined based on the scene type vector, environmental element vector, and spatial layout feature vector. Extract the task type, task objective, and constraints from the task context description information; The task type, task objective, and constraints are concatenated to obtain the task context vector; The observation timestamp, relative time, and time series label of the object are concatenated to obtain the time series context vector; The scene context vector, task context vector, and temporal context vector are concatenated to obtain the context feature vector; Construct a scene relationship graph, wherein the scene relationship graph includes: a set of nodes and edges. The set of nodes includes all objects in the scene. The attributes of each node include: object identifier, global semantic anchor identifier, text object name, object type and object feature vector. The object feature vector includes: semantic feature vector, visual feature vector and embodied feature vector. The edges include: spatial relationship edges, semantic relationship edges and physical relationship edges. Generate the context projection identifier of the object based on the object's global semantic anchor identifier, context feature vector, and local subgraph of the object in the scene relationship graph.

7. The method according to claim 3, characterized in that, Based on the multimodal data samples, determine the embodied state identifier of each object in the multimodal object instance set, including: Based on the six-degree-of-freedom attitude and physical attribute dictionary, a physical state dictionary and a historical state list are generated, wherein the historical state list includes: multiple state snapshots; Based on the physical state dictionary and the historical state list, determine the specific state identifier of each object.

8. The method according to claim 3, characterized in that, After determining the set of multimodal object instances and the global semantic anchor identifier, context projection identifier, and embodied state identifier of each object in the set of multimodal object instances based on the multimodal data samples, the following is also included: Based on the global semantic anchor identifier and intrinsic feature vector of each object in the multimodal object instance set, determine the multimodal object instance set and global semantic anchor vector index after adding the global semantic anchor identifier; Based on the set of multimodal object instances after adding global semantic anchor points and the context projection identifiers of each object, determine the set of multimodal object instances after adding context projection identifiers, the context index, and the scene relationship graph set. The set of multimodal object instances after adding context projection identifiers includes: context projection identifiers, context feature vectors, and the scene relationship graph to which the object belongs. Based on the embodied state identifiers of each object in the multimodal object instance set and the multimodal object instance set after adding context projection identifiers, determine the multimodal object instance set after adding embodied state identifiers and the hierarchical identifier mapping table; Based on the hierarchical identifier mapping table, the set of multimodal object instances with added embodied state identifiers, and the set of scene relationship graphs, a spatiotemporal semantic graph is established.

9. The method according to claim 1, characterized in that, The model to be trained is trained to obtain a multimodal model, including: Obtain the first model; The first model is trained based on the training dataset to obtain a context-aware word segmenter; The training model is trained based on the training dataset to obtain a multimodal model, wherein the training model includes the context-aware word segmenter.

10. The method according to claim 9, characterized in that, The training dataset includes: training samples, wherein the training samples are masked text sequences, and one-hot tags for each token position in the training samples; The first model is trained based on the training dataset to obtain a context-aware word segmenter, including: Input the training samples from the training dataset into the first model to obtain the hidden state and predicted probability of each token position; The first loss function is determined based on the hidden state, predicted probability, and one-hot label of each token position in the training samples. The second loss function is determined based on the embedding distance of each token's context. The third loss function is determined based on the textual descriptions of multiple object instances identified by the same global semantic anchor. The first model is trained based on the first loss function, the second loss function, and the third loss function to obtain a context-aware word segmenter.

11. The method according to claim 9, characterized in that, The model to be trained is trained based on the training dataset to obtain a multimodal model, including: Based on the context-aware word segmenter, a second model is generated, wherein the second model includes, from input to output, the following components in sequence: a context-aware word segmenter, a Transformer encoder, a pooling layer, and a text projection layer; The second model is trained based on the training dataset to obtain the text encoder; The third model is trained based on the training dataset to obtain the visual encoder; The fourth model is trained based on the training dataset to obtain the embodied encoder; A multimodal model is generated based on the text encoder, visual encoder, and embodied encoder.

12. The method according to claim 1, characterized in that, Also includes: Acquire target multimodal data; The target multimodal data is input into the multimodal model to obtain the target query vector; Query the global semantic anchor point GID vector index to obtain the target GID sequence associated with the target query vector; The target GID is determined based on the target multimodal data, the target GID sequence, and the spatiotemporal semantic graph.

13. A cross-scene object marking device, characterized in that, include: The determination module is used to determine the set of multimodal object instances and the global semantic anchor identifier, context projection identifier, and embodied state identifier of each object in the set of multimodal object instances based on the multimodal data samples. The generation module is used to generate a training dataset based on the multimodal object instance set and the global semantic anchor identifier, context projection identifier and embodied state identifier of each object in the multimodal object instance set. The training dataset is used to train the model to be trained to obtain a multimodal model.

14. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the cross-scene object tagging method according to any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the cross-scene object marking method according to any one of claims 1-12.

16. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the cross-scene object tagging method according to any one of claims 1-12.

Images