Robot scene understanding and operation control method and system based on visual large model

Abstract

The present application relates to the technical field of robots, and in particular to a robot scene understanding and operation control method and system based on a visual large model. Scene visual data is collected, time and space are aligned and fused to construct a unified scene representation, a pre-trained visual large model is input to extract target object visual features and generate a scene semantic graph with target objects as nodes and semantic relationships as edges; attribute reasoning is performed on each node to determine the category, geometric parameters, physical properties and operation state, and the scene understanding structured graph is obtained by backfilling; a task description is encoded into a task semantic vector, candidate nodes and associated edges are retrieved in the structured graph to construct a task-oriented operation subgraph; hierarchical planning is performed in combination with robot kinematic constraints to generate a hierarchical operation sequence, and a closed-loop position-force hybrid control is used to generate a motion control instruction sequence to drive the robot to complete the scene operation task. The method improves the accuracy of scene understanding and the robustness and adaptability of operation control.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a method and system for robot scene understanding and operation control based on a large visual model. Background Technology

[0002] As robotics extends into unstructured environments, the deep integration of scene understanding and operational control has become a core technological challenge for realizing intelligent service robots. Current technologies typically employ a phased pipeline architecture: the first stage relies on traditional computer vision methods or convolutional neural network-based detection and segmentation models to extract static target object categories and spatial coordinates from visual sensor data, constructing a sparse semantic map containing geometric information; the second stage, based on predefined task templates or state machines, maps task instructions into discrete action sequences through symbolic reasoning or probabilistic graphical models; the third stage uses inverse kinematics or planning algorithms (such as rapidly expanding random trees) to generate continuous joint trajectories and achieves physical interaction through position control or impedance control. This hierarchical architecture achieves good results in structured industrial environments, but faces inherent limitations when dealing with complex scenarios such as homes and healthcare.

[0003] One of the core flaws of existing conventional approaches lies in the semantic gap between scene representation and task representation. Traditional semantic maps only record object categories and locations, lacking functional relationships, spatial topological relationships, and implicit task context information (such as the supporting relationship of "the cup needs to be placed on the tray"). The operation planning module has to rely on hard-coded rules or manually preset interfaces for information bridging, making it difficult for scene understanding results to directly support flexible task reasoning, especially when objects are stacked, occluded, or placed in atypical ways, significantly reducing planning efficiency and robustness.

[0004] Another drawback lies in the fragmentation and static nature of the generated operation sequence. Conventional methods treat perception, planning, and control as independent modules, and the discrete steps generated during the task planning phase lack closed-loop feedback on the real-time state of the object (such as deformation of the grasping surface and changes in friction). When the object's properties or environmental constraints change slightly during the operation (e.g., the target object is accidentally moved), the system needs to re-trigger a complete global replanning, rather than making local adaptive adjustments. Furthermore, the decoupling design of force / position control in the hierarchical control architecture often ignores the temporal dependencies and pre-constraints between operation steps, making it difficult to effectively suppress the impact or slippage at the moment of contact between the end effector and the object, thus limiting the reliability and accuracy of the operation task. Summary of the Invention

[0005] The embodiments of the present invention provide a method and system for robot scene understanding and operation control based on a large visual model, which can solve the problems in the prior art.

[0006] A first aspect of this invention provides a robot scene understanding and operation control method based on a large visual model, comprising:

[0007] Visual data of robot operation scenarios are collected, and a unified scene representation containing geometric and appearance features is constructed through spatiotemporal alignment and fusion. The unified scene representation is input into a pre-trained large visual model to extract visual features of target objects, encode the spatial topological relationships, semantic associations and task contexts between target objects, and generate a scene semantic graph with target objects as nodes and semantic relationships as edges.

[0008] Attribute reasoning is performed on each node in the scene semantic graph to determine the category, geometric parameters, physical attributes and operation state of each target object. The reasoning results are then backfilled into the corresponding nodes to obtain a structured graph of scene understanding.

[0009] Obtain a preset operation task description and encode it as a task semantic vector. Retrieve candidate nodes and associated edges with semantic matching in the scene understanding structured graph and construct a task-oriented operation subgraph that includes operation objects, path constraints, and time sequence constraints.

[0010] Based on the task-oriented operation subgraph, combined with robot kinematic constraints and end effector boundary hierarchical planning, a hierarchical operation sequence consisting of atomic operation steps is generated. According to the execution order and prerequisite dependencies of the hierarchical operation sequence, a motion control command sequence adapted to the robot joints and end effector is generated using closed-loop position-force hybrid control to drive the robot to complete the scene operation task.

[0011] The unified representation of the scene is input into a pre-trained large-scale visual model to extract visual features of target objects, encode the spatial topological relationships, semantic associations, and task context between target objects, and generate a scene semantic graph with target objects as nodes and semantic relationships as edges, including:

[0012] The scene is uniformly represented by dividing it into multiple local scene blocks with overlapping boundaries according to a spatial grid. Target detection and feature extraction are performed on each block to obtain the local visual features and bounding box coordinates of the target objects in each block.

[0013] For target objects detected in overlapping areas of adjacent blocks, cross-block consistency matching is performed based on the bounding box intersection-union ratio and local visual feature cosine similarity to establish a cross-block target object correspondence table.

[0014] Based on the cross-block target object correspondence table, the local visual features of the same target object in each block are stitched together and linearly projected to obtain the global visual feature representation vector of each target object.

[0015] Using the global visual feature representation vector of each target object as node input, the spatial topological attention weight matrix and semantic association attention weight matrix between target objects are learned in parallel.

[0016] The spatial topology attention weight matrix and the semantic association attention weight matrix are weighted and fused according to learnable fusion coefficients, and channel modulation is performed with the task context semantic guidance vector to generate a comprehensive relationship strength matrix between target object pairs.

[0017] The connection relationships corresponding to object pairs whose comprehensive relationship strength matrix values ​​meet the threshold are determined as semantic relationship edges, and a scene semantic graph is constructed with the target object as the node and the semantic relationship as the edge.

[0018] The spatial topology attention weight matrix and the semantic association attention weight matrix are weighted and fused according to learnable fusion coefficients, and then channel-modulated with the task context semantic guidance vector to generate a comprehensive relationship strength matrix between target object pairs, including:

[0019] Obtain the task type label of the current operation task, input the task type label into the pre-trained task semantic encoder to generate task semantic context features, and retrieve the basic semantic guidance vector with the smallest cosine distance to the task semantic context features from the pre-built task semantic guidance vector library;

[0020] The basic semantic guidance vector is transformed element-wise through a learnable gating network consisting of fully connected layers and nonlinear activation functions to generate a channel-level modulation coefficient vector in which each element takes values ​​within a preset range.

[0021] A weighted linear combination of each element of the spatial topology attention weight matrix and the corresponding element of the semantic association attention weight matrix is ​​performed using learnable scalar fusion coefficients to generate a fused attention weight matrix.

[0022] The fusion attention weight matrix is ​​broadcast multiplied along the feature channel dimension with the channel-level modulation coefficient vector, so that the features of different channels obtain differentiated task semantic modulation intensity, and a comprehensive relation strength matrix modulated by task semantic is obtained.

[0023] The element values ​​of each row in the comprehensive relation strength matrix modulated by the task semantics are sorted in descending order. Only the element values ​​at the first preset number of sorted positions in each row are kept unchanged. The element values ​​exceeding the preset number are forcibly set to zero. The non-zero element values ​​retained in each row are normalized to generate a sparse comprehensive relation strength matrix.

[0024] Attribute reasoning is performed on each node in the scene semantic graph to determine the category, geometric parameters, physical properties, and operational state of each target object. The reasoning results are then backfilled into the corresponding nodes to obtain a structured scene understanding graph, including:

[0025] Using the visual features associated with each target object node in the scene semantic graph and the semantic relationship edge features of adjacent nodes as joint inputs, classification reasoning, geometric reasoning, physical reasoning and state reasoning are executed in parallel through a multi-branch attribute reasoning network.

[0026] In the classification reasoning branch, the joint input is mapped to a predefined object category label space, and the fine-grained category label and functional category affiliation of the target object are output.

[0027] In the geometric reasoning branch, the joint input is decoded into the three-dimensional bounding box size parameters, principal axis direction vector and surface curvature distribution parameters of the target object, forming a geometric morphology parameter set;

[0028] In the physics reasoning branch, the joint input is mapped to a preset set of material categories and physical property space, and the material category label, mass range identifier and surface friction coefficient level of the target object are output.

[0029] In the state reasoning branch, the joint input is fused with the operation state of the target object in the previous time step, and the current switch state label, content filling state label and spatial pose stability evaluation value of the target object are output.

[0030] The category labels, geometric morphology parameter groups, physical attribute description values, and operation status label groups output by each inference branch are used as attribute fields and backfilled into the attribute fields of the corresponding target object nodes in the scene semantic graph to obtain a structured graph of scene understanding.

[0031] In the geometric reasoning branch, the joint input is decoded into the three-dimensional bounding box size parameters, principal axis direction vectors, and surface curvature distribution parameters of the target object, forming a geometric morphology parameter set, including:

[0032] The visual features of the target object are separated from the joint input. The visual features are then upsampled layer by layer and mapped to a normalized coordinate system with the centroid of the target object as the origin through a pre-trained 3D shape prior decoder, generating a dense 3D point cloud representation, in which each point carries spatial coordinates and local geometric confidence.

[0033] The spatial coordinates of each point in the dense 3D point cloud representation are constructed using a covariance matrix and eigenvalue decomposition. Three orthogonal eigenvectors are extracted as candidate principal axis directions. The projection span of the point cloud along each candidate principal axis direction is used as a candidate size parameter. The three orthogonal directions that maximize the variance of the projection span are selected as principal axis direction vectors, and the corresponding projection span is used as the 3D bounding box size parameter.

[0034] Based on the Poisson disk sampling strategy, multiple local neighborhood point sets are uniformly selected in the surface region represented by the dense three-dimensional point cloud. Local quadratic surface fitting is performed on each local neighborhood point set, and the Gaussian curvature value and average curvature value at each sampling point are calculated. The mean and standard deviation of the Gaussian curvature value and average curvature value of all sampling points are statistically calculated to form the surface curvature distribution parameters.

[0035] The three-dimensional bounding box size parameters, the principal axis direction vector, the mean and standard deviation of Gaussian curvature, and the mean and standard deviation of average curvature are combined into a geometric morphological parameter set.

[0036] Obtain a preset operation task description and encode it as a task semantic vector. Retrieve semantically matching candidate nodes and associated edges in the scene understanding structured graph. Construct a task-oriented operation subgraph containing operation objects, path constraints, and time sequence constraints, including:

[0037] The preset operation task description is segmented and contextual semantically encoded, and the encoding result is compressed into a fixed-dimensional task semantic vector through pooling.

[0038] Traverse the target object nodes in the structured graph of the scene understanding, calculate the cosine similarity between the attribute field concatenation vector of each node and the task semantic vector, and mark the nodes whose cosine similarity meets the preset threshold as candidate operation object nodes.

[0039] Starting with each candidate operation object node, a restricted breadth-first traversal is performed along the direction of the semantic relationship edges in the structured graph of the scene understanding. Within the traversal depth range, intermediate nodes and corresponding semantic relationship edges along the path are collected to construct a subset of candidate operation paths.

[0040] Based on the coordinate distribution of the candidate operation object nodes in the three-dimensional space of the scene and the spatial Euclidean length of the paths in each candidate operation path subset, the operation space path constraint parameters are determined.

[0041] Based on the logical preconditions of each subtask operation in the preset operation task description, determine the partial order constraints of the operation timing between the corresponding operation object nodes of different subtasks.

[0042] The candidate operation object node set, the candidate operation path subset, the operation space path constraint parameters, and the operation temporal partial order constraint are combined into a task-oriented operation subgraph.

[0043] Based on the task-oriented operation subgraph, and combined with robot kinematic constraints and end effector boundary hierarchical planning, a hierarchical operation sequence consisting of atomic operation steps is generated. Each step includes the operation object, operation type, target pose, and prerequisite dependencies, including:

[0044] The task-oriented operation subgraph is topologically sorted, and each operation object node is arranged into a linear execution order according to the operation timing partial order constraint, forming a coarse-grained subtask sequence.

[0045] For each subtask in the coarse-grained subtask sequence, according to its operation object type and target state requirements, it is decomposed into at least one atomic operation step among approach action, grasp action, move action and release action, and a corresponding operation type label is assigned to each atomic operation step.

[0046] For each atomic operation step, the geometric morphological parameters and physical properties of the operation object are used as constraints. Inverse kinematics solution is performed in the reachable workspace of the robot end effector to obtain the target pose parameters that satisfy joint limit and collision avoidance constraints.

[0047] Based on the topological sorting results of the coarse-grained subtask sequence and the execution order of each atomic operation step within its respective subtask, establish the pre-dependencies between atomic operation steps and mark the pre-dependency condition identifiers of each atomic operation step.

[0048] The operation object identifier, operation type label, target pose parameter and prerequisite dependency condition identifier of each atomic operation step are encapsulated into hierarchical operation step data records, and arranged in the execution order to generate the hierarchical operation sequence.

[0049] A second aspect of this invention provides a robot scene understanding and operation control system based on a large visual model, comprising:

[0050] The semantic graph construction unit is used to collect visual data of the robot's operation scene, and construct a unified scene representation containing geometric and appearance features through spatiotemporal alignment and fusion. The unified scene representation is input into a pre-trained large visual model to extract visual features of target objects, encode the spatial topological relationships, semantic associations and task context between target objects, and generate a scene semantic graph with target objects as nodes and semantic relationships as edges.

[0051] The attribute reasoning unit is used to perform attribute reasoning on each node in the scene semantic graph, determine the category, geometric parameters, physical attributes and operation status of each target object, and fill the reasoning results back into the corresponding node to obtain a scene understanding structured graph.

[0052] The operation subgraph unit is used to obtain a preset operation task description and encode it into a task semantic vector, retrieve candidate nodes and associated edges with semantic matching in the scene understanding structured graph, and construct a task-oriented operation subgraph containing operation objects, path constraints and time sequence constraints.

[0053] The operation sequence unit is used to generate a hierarchical operation sequence consisting of atomic operation steps based on the task-oriented operation subgraph, combined with robot kinematic constraints and end effector boundary hierarchical planning. According to the execution order and prerequisite dependencies of the hierarchical operation sequence, a motion control command sequence adapted to the robot joints and end effector is generated by closed-loop position-force hybrid control to drive the robot to complete the scene operation task.

[0054] A third aspect of the present invention provides an electronic device, comprising:

[0055] processor;

[0056] Memory used to store processor-executable instructions;

[0057] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0058] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0059] Scene representation and semantic graph construction based on a large visual model compress high-dimensional visual data into a structured graph with target objects as nodes and semantic relationships as edges, significantly enhancing the robot's global understanding of complex task scenarios. The unified scene representation integrates spatiotemporally aligned geometric and appearance features, enabling the large visual model to accurately extract object visual attributes while encoding spatial topology, semantic relationships, and task context, overcoming the problems of low object recognition rates and difficult relationship reasoning in unstructured environments encountered by traditional methods. The scene semantic graph dynamically preserves the potential interaction logic between objects, providing a starting point with clear causal links for subsequent reasoning.

[0060] By inferring object categories, geometric parameters, physical properties, and operational states, the robot fills in the graph nodes, forming a structured graph for scene understanding. This process endows the robot with a detailed understanding of each object in the scene, including its material, deformability, and current operational state (such as whether it is occluded or in motion), thereby avoiding misoperations or planning failures due to missing information. The structured graph enables the robot to understand the environment at a semantic level, supporting efficient and accurate matching during subsequent retrieval and subgraph construction, significantly improving the robustness of task adaptation.

[0061] A semantic matching mechanism between task semantic vectors and structured graphs accurately identifies the operation object, path constraints, and temporal constraints, constructing a task-oriented operation subgraph. This eliminates the computational waste of global search, allowing the robot to focus only on sub-regions and relationships highly relevant to the current task. Hierarchical planning integrating robot kinematic constraints and end-effector boundaries decomposes complex tasks into atomic operation steps, clearly defining the execution order and prerequisite dependencies, generating a traceable and verifiable sequence. Closed-loop position-force hybrid control ensures real-time compensation for environmental uncertainties (such as minute object offsets and surface friction changes) during operation, giving motion commands both compliance and precision, ultimately driving the robot to complete tasks stably, safely, and efficiently in dynamic scenarios. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating a method for robot scene understanding and operation control based on a large visual model. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0065] Figure 1 This is a flowchart illustrating the robot scene understanding and operation control method based on a large visual model according to an embodiment of the present invention. The robot scene understanding and operation control method based on a large visual model includes:

[0066] Visual data of robot operation scenarios are collected, and a unified scene representation containing geometric and appearance features is constructed through spatiotemporal alignment and fusion. The unified scene representation is input into a pre-trained large visual model to extract visual features of target objects, encode the spatial topological relationships, semantic associations and task contexts between target objects, and generate a scene semantic graph with target objects as nodes and semantic relationships as edges.

[0067] Attribute reasoning is performed on each node in the scene semantic graph to determine the category, geometric parameters, physical attributes and operation state of each target object. The reasoning results are then backfilled into the corresponding nodes to obtain a structured graph of scene understanding.

[0068] Obtain a preset operation task description and encode it as a task semantic vector. Retrieve candidate nodes and associated edges with semantic matching in the scene understanding structured graph and construct a task-oriented operation subgraph that includes operation objects, path constraints, and time sequence constraints.

[0069] Based on the task-oriented operation subgraph, combined with robot kinematic constraints and end effector boundary hierarchical planning, a hierarchical operation sequence consisting of atomic operation steps is generated. According to the execution order and prerequisite dependencies of the hierarchical operation sequence, a motion control command sequence adapted to the robot joints and end effector is generated using closed-loop position-force hybrid control to drive the robot to complete the scene operation task.

[0070] The unified representation of the scene is input into a pre-trained large-scale visual model to extract visual features of target objects, encode the spatial topological relationships, semantic associations, and task context between target objects, and generate a scene semantic graph with target objects as nodes and semantic relationships as edges, including:

[0071] The scene is uniformly represented by dividing it into multiple local scene blocks with overlapping boundaries according to a spatial grid. Target detection and feature extraction are performed on each block to obtain the local visual features and bounding box coordinates of the target objects in each block.

[0072] For target objects detected in overlapping areas of adjacent blocks, cross-block consistency matching is performed based on the bounding box intersection-union ratio and local visual feature cosine similarity to establish a cross-block target object correspondence table.

[0073] Based on the cross-block target object correspondence table, the local visual features of the same target object in each block are stitched together and linearly projected to obtain the global visual feature representation vector of each target object.

[0074] Using the global visual feature representation vector of each target object as node input, the spatial topological attention weight matrix and semantic association attention weight matrix between target objects are learned in parallel.

[0075] The spatial topology attention weight matrix and the semantic association attention weight matrix are weighted and fused according to learnable fusion coefficients, and channel modulation is performed with the task context semantic guidance vector to generate a comprehensive relationship strength matrix between target object pairs.

[0076] The connection relationships corresponding to object pairs whose comprehensive relationship strength matrix values ​​meet the threshold are determined as semantic relationship edges, and a scene semantic graph is constructed with the target object as the node and the semantic relationship as the edge.

[0077] After inputting the unified scene representation into the pre-trained large-scale visual model, it is first divided into blocks according to a spatial grid. Specifically, the spatial range of the unified scene representation is uniformly divided into several local scene blocks, with a certain proportion of overlapping boundaries maintained between adjacent blocks. The width of the overlapping area is usually set to 10% to 30% of the block's side length to ensure that target objects located at the block's edge do not lose complete local contextual information due to the segmentation. For each local scene block, it is fed into the feature extraction branch of the pre-trained large-scale visual model to perform object detection and feature extraction operations, outputting the local visual feature vectors of all target objects within the block and their corresponding bounding box coordinates. The local visual feature vectors retain the object's texture, color, shape, and other appearance information, while the bounding box coordinates record the object's position and size in the current block's coordinate system, providing a localization basis for subsequent cross-block matching.

[0078] Because overlapping boundaries exist between adjacent blocks, the same target object may be detected in two adjacent blocks, resulting in duplicate detection. To address this, cross-block consistency matching is performed on target objects detected within overlapping regions. The matching process utilizes two metrics: one is the bounding box intersection-union ratio (IU), which is the ratio of the intersection area to the union area of ​​two candidate bounding boxes, denoted as IU. ,when When the threshold is exceeded, the two detection results are considered to be highly overlapping in spatial location; the second is the cosine similarity of local visual features, denoted as... By calculating two local visual feature vectors and cosine similarity To measure consistency at the appearance level. Only when and Only when both blocks meet their respective threshold conditions are the corresponding detection results in the two blocks determined to be the same target object, and an association record is established in the cross-block target object correspondence table. This correspondence table uses the globally unique identifier of the target object as the primary key and records the local detection index of the object in each block, providing an index basis for subsequent feature fusion.

[0079] Based on the cross-block target object correspondence table, the local visual feature vectors detected in each block for each target object are concatenated to form a concatenated feature vector, which is then processed by a learnable linear projection matrix. Mapping it to a unified feature space yields the global visual feature representation vector of the target object. The purpose of linear projection is to eliminate distribution biases introduced by local context differences during feature extraction from different blocks, while compressing the high-dimensional features after stitching to a fixed dimension, facilitating unified processing during subsequent graph structure modeling. For target objects detected only within a single block, their local visual feature vectors are directly obtained through linear projection to obtain the global visual feature representation vector, without performing a stitching operation.

[0080] Representation vectors of global visual features of each target object Two types of attention weight matrices are learned in parallel as node inputs. Spatial topological attention weight matrix. Using the target object's 3D position coordinates, relative distance, and azimuth as primary inputs, a multi-head attention mechanism is employed to model the spatial proximity and orientational dependencies between objects. This results in higher attention weights being assigned to spatially adjacent objects or those with specific orientational relationships. (Semantic association attention weight matrix) Then the vector is represented by global visual features. As input, a separate multi-head attention mechanism is used to capture the semantic-level correlation between objects, such as the support relationship between "cup" and "tabletop", and the compositional relationship between "bottle cap" and "bottle body". The two attention mechanisms share the low-level feature extraction capabilities of the pre-trained visual model, but each has independent query, key, and value projection parameters, thus enabling them to focus on modeling relationships in the spatial and semantic dimensions respectively.

[0081] Spatial topological attention weight matrix Attention weight matrix related to semantics Based on learnable fusion coefficient Weighted fusion is performed to obtain the fused attention matrix. ,in Automatic optimization during training enables the model to dynamically adjust the relative weights of spatial and semantic relationships based on different task scenarios. Building upon this, a task-context semantic guidance vector is introduced. This vector, encoded from the task description text by a language encoder, carries semantic intent information for the current task. The attention matrix will then be fused. With task context semantic guidance vector Channel modulation is performed, specifically by: Affine transformation is used to generate channel-by-channel scaling factors and offsets. Scaling and offset are applied to each channel dimension separately to generate the final comprehensive relation strength matrix. It maintains semantic consistency with the current task. The introduction of the channel modulation mechanism allows the same scene to generate different distributions of relationship strength in different task contexts. For example, in a grasping task, it will highlight the contact relationship between the object and the supporting surface, while in a liquid pouring task, it will strengthen the spatial alignment relationship between the container and the target area.

[0082] Based on the comprehensive relationship strength matrix For each object pair in the matrix, the relationship strength value is compared with a preset threshold. When comparing, the relationship strength value of a pair of objects satisfies When the connection between the two object pairs is defined as a semantic relation edge, the weight of the edge is set to a value. The edge type label is determined by the semantic relation category corresponding to the channel with the strongest relation strength, such as predefined relation types like "support," "containment," "adjacent," and "functional dependency." Threshold The threshold setting needs to strike a balance between the sparsity of the graph and the completeness of relation coverage. Too low a threshold will introduce a large number of noisy edges, while too high a threshold may miss important semantic relationships. In practical applications, it can be adaptively adjusted according to the complexity of the scenario. For example, in dense scenes with a large number of target objects, the threshold can be appropriately increased to maintain the resolvability of the graph structure. All target objects are treated as nodes, and the connections between object pairs that meet the threshold conditions are treated as semantic relationship edges. Finally, a scene semantic graph is constructed with target objects as nodes and semantic relationships as edges, providing a complete graph structure foundation for the generation of subsequent structured graphs for scene understanding.

[0083] The spatial topology attention weight matrix and the semantic association attention weight matrix are weighted and fused according to learnable fusion coefficients, and then channel-modulated with the task context semantic guidance vector to generate a comprehensive relationship strength matrix between target object pairs, including:

[0084] Obtain the task type label of the current operation task, input the task type label into the pre-trained task semantic encoder to generate task semantic context features, and retrieve the basic semantic guidance vector with the smallest cosine distance to the task semantic context features from the pre-built task semantic guidance vector library;

[0085] The basic semantic guidance vector is transformed element-wise through a learnable gating network consisting of fully connected layers and nonlinear activation functions to generate a channel-level modulation coefficient vector in which each element takes values ​​within a preset range.

[0086] A weighted linear combination of each element of the spatial topology attention weight matrix and the corresponding element of the semantic association attention weight matrix is ​​performed using learnable scalar fusion coefficients to generate a fused attention weight matrix.

[0087] The fusion attention weight matrix is ​​broadcast multiplied along the feature channel dimension with the channel-level modulation coefficient vector, so that the features of different channels obtain differentiated task semantic modulation intensity, and a comprehensive relation strength matrix modulated by task semantic is obtained.

[0088] The element values ​​of each row in the comprehensive relation strength matrix modulated by the task semantics are sorted in descending order. Only the element values ​​at the first preset number of sorted positions in each row are kept unchanged. The element values ​​exceeding the preset number are forcibly set to zero. The non-zero element values ​​retained in each row are normalized to generate a sparse comprehensive relation strength matrix.

[0089] Obtain the task type label for the current operation task. This label describes the high-level semantic category of the task in natural language or structured enumeration form, such as "grab," "place," "assemble," or "push." ​​Input the task type label into a pre-trained task semantic encoder. This encoder, based on a Transformer structure and pre-trained on a large-scale task description corpus, can map discrete task type labels into dense vectors in a continuous high-dimensional semantic feature space, denoted as the task semantic context feature vector. Retrieve relevant semantic guidance vectors from the pre-built task semantic guidance vector library. The basic semantic guidance vector with the minimum cosine distance. The task semantic guidance vector library consists of prototype vectors for multiple typical operational tasks, with one basic guidance vector corresponding to each task type. The vectors in the library are pre-determined and stored offline through clustering or manual annotation. Calculation occurs during retrieval. The vector with the highest cosine similarity (i.e., the smallest cosine distance) to each vector in the library is selected as the basic semantic guide vector. This anchors the semantic information of the current task to the known task prototype space, thus avoiding feature drift caused by the diversity of task descriptions.

[0090] Guide the basic semantic vector The input is a learnable gated network consisting of fully connected layers and a nonlinear activation function, which performs an element-wise nonlinear transformation. This gated network contains at least one fully connected layer and a sigmoid activation function, which compresses the element values ​​to a minimum. The interval ensures the output channel-level modulation coefficient vector. Each element falls within a preset range Inner. Vector The dimension of the modulated coefficient vector is aligned with the number of feature channels in the fusion attention weight matrix, where each element represents the modulation intensity of the corresponding channel: a value close to 1 indicates that the channel's features are highly relevant to the current task and should be fully preserved; a value close to 0 indicates that the channel contributes little to the current task and should be suppressed. Through parameterized learning of the gating network, the modulation coefficient vector can adaptively learn the importance distribution of each channel under different task types during training, without the need for manual specification of channel weights.

[0091] Learnable scalar fusion coefficients Spatial topological attention weight matrix Attention weight matrix of each element and its semantic association The corresponding elements are combined using a weighted linear combination to generate a fused attention weight matrix. The calculation method is as follows .in As a scalar, it is automatically updated through backpropagation during model training, requiring no manual adjustment. When When the value approaches 1, the fusion result reflects spatial topological relationships more effectively, making it suitable for tasks sensitive to the relative positions of objects; when... When the value approaches 0, the fusion result reflects semantic relationships more effectively, making it suitable for tasks that require understanding the functional attributes and semantic categories of objects. This weighted linear combination method maintains the structure of the two types of attention matrices while achieving flexible dynamic balance through a single learnable parameter, resulting in extremely low computational cost.

[0092] Fusion attention weight matrix Along the feature channel dimension and the channel-level modulation coefficient vector Perform broadcast multiplication, such that The eigenvalues ​​of different channels are multiplied by respectively The modulation coefficient at the corresponding position. Broadcast mechanisms allow... Automatically copy objects in the same dimension. The same task semantic modulation is applied to the corresponding channel features of each pair of objects. After this operation, channel features highly relevant to the current task are amplified, while irrelevant channel features are compressed, thereby generating a comprehensive relation strength matrix modulated by task semantics. Each element of the matrix Reflecting object pairs under the semantic guidance of the current task The strength of the relationships between objects integrates both spatial topology and semantic association information, and has been enhanced by task relevance differentiation, enabling it to more accurately capture the relationships between objects that have a substantial impact on the execution of the current task.

[0093] The comprehensive relation strength matrix modulated by task semantics The elements in each row of the matrix are sorted in descending order. Each row of the matrix corresponds to a target object node, and the elements in each column of the row represent the strength of the relationship between that object and all other objects. After sorting in descending order, only the elements in the first position of each row are retained. Bit( The element values ​​(for a preset number) remain unchanged, and will exceed the previous number. The element value of each bit is forced to zero. Preset quantity. The value is determined based on the number of objects in the scene and the complexity of the task, typically ranging from 3 to 10. This ensures that each object node maintains effective connections only with a few of the most relevant objects, preventing the graph structure from becoming too dense and causing excessive computational burden for subsequent graph inference, as well as preventing key relationships from being overwhelmed by noisy relationships. After the zeroing operation is completed, the non-zero element values ​​retained in each row are normalized so that the sum of the non-zero elements in each row is 1. This transforms the relationship strength into a probability distribution form, facilitating message passing and aggregation calculations in the subsequent graph neural network.

[0094] After the above sparsification and normalization processes, the final sparsed comprehensive relation strength matrix is ​​generated. This matrix significantly reduces the edge density of the graph structure while preserving the most task-relevant relationships between objects. The core value of sparsification lies in: firstly, by forcibly preserving the edges of each row... The strongest relation edges ensure that the semantic graph structure focuses on object pairs truly important to the current task, filtering out weak relation interference introduced by noise or distant irrelevant objects. Furthermore, normalization makes the outgoing edge weights of each node comparable, eliminating absolute strength deviations caused by differences in scene complexity. This provides a clear and semantically prominent relational foundation for the construction of subsequent task-oriented operation subgraphs. In real-world robot operation scenarios, such as multi-object grasping tasks on a desktop, the target object is often surrounded by dozens of interfering objects. By using a sparse comprehensive relation strength matrix, key neighbor nodes such as supports and occluders directly related to the target object can be quickly located, significantly improving the efficiency and robustness of subsequent operation planning.

[0095] Attribute reasoning is performed on each node in the scene semantic graph to determine the category, geometric parameters, physical properties, and operational state of each target object. The reasoning results are then backfilled into the corresponding nodes to obtain a structured scene understanding graph, including:

[0096] Using the visual features associated with each target object node in the scene semantic graph and the semantic relationship edge features of adjacent nodes as joint inputs, classification reasoning, geometric reasoning, physical reasoning and state reasoning are executed in parallel through a multi-branch attribute reasoning network.

[0097] In the classification reasoning branch, the joint input is mapped to a predefined object category label space, and the fine-grained category label and functional category affiliation of the target object are output.

[0098] In the geometric reasoning branch, the joint input is decoded into the three-dimensional bounding box size parameters, principal axis direction vector and surface curvature distribution parameters of the target object, forming a geometric morphology parameter set;

[0099] In the physics reasoning branch, the joint input is mapped to a preset set of material categories and physical property space, and the material category label, mass range identifier and surface friction coefficient level of the target object are output.

[0100] In the state reasoning branch, the joint input is fused with the operation state of the target object in the previous time step, and the current switch state label, content filling state label and spatial pose stability evaluation value of the target object are output.

[0101] The category labels, geometric morphology parameter groups, physical attribute description values, and operation status label groups output by each inference branch are used as attribute fields and backfilled into the attribute fields of the corresponding target object nodes in the scene semantic graph to obtain a structured graph of scene understanding.

[0102] The visual features carried by each target object node in the scene semantic graph include local and global appearance information extracted by a pre-trained large visual model, while the semantic relationship edges between nodes record the spatial topological relationships and semantic association strength between object pairs. During the attribute reasoning stage, the visual feature vector associated with each target object node is concatenated with the semantic relationship edge features of its neighboring nodes to form a joint input representation. This joint input carries both the object's own appearance information and its structural context in the scene graph, providing a unified information source for subsequent multi-branch parallel reasoning. The multi-branch attribute reasoning network consists of four parallel sub-networks: classification reasoning branch, geometric reasoning branch, physical reasoning branch, and state reasoning branch. These four branches share the same joint input and each completes reasoning for its corresponding attribute dimension through an independent set of parameters, thus avoiding gradient interference between different attribute tasks. Simultaneously, the shared input preserves implicit cross-attribute association information.

[0103] The classification inference branch maps the joint input to a predefined object category label space via a multilayer perceptron. This predefined label space comprises two levels: a fine-grained category system and a functional category system. The fine-grained category system differentiates objects based on their shape and appearance, such as distinguishing between a "straight-handled cup" and a "lidded cup." The functional category system categorizes objects according to their role in the task, such as "container," "tool," and "fixed support." The output of the classification inference branch is the fine-grained category label of the target object and its corresponding functional category. Both provide semantic anchors for the subsequent construction of the task-oriented operation subgraph. During training, the classification inference branch uses cross-entropy loss to supervise the fine-grained category predictions and introduces hierarchical consistency constraints to ensure logical consistency between the fine-grained category predictions and functional category attributions at the category system level.

[0104] The geometric inference branch decodes the joint input into a set of parameters describing the 3D morphology of the target object. Specifically, the decoding output consists of three parts: 3D bounding box size parameters, principal axis direction vectors, and surface curvature distribution parameters, which together constitute the geometric morphology parameter set. The 3D bounding box size parameters are represented by three scalars—length, width, and height—in the object's local coordinate system and are used for collision detection and grasp point estimation in subsequent motion planning. The principal axis direction vector describes the object's main extension direction in 3D space. For objects with obvious axisymmetric structures (such as bottles and pipes), the principal axis direction vector directly determines the approach direction strategy of the end effector. The surface curvature distribution parameters are encoded by the curvature statistical characteristics of the object's visible surface, represented by discretized intervals of mean curvature, maximum principal curvature, and minimum principal curvature, and are used to evaluate the geometric adaptability of the grasping contact surface. The loss function of the geometric inference branch uses a smoothed L1 loss for the size parameters, a cosine distance loss for the principal axis direction vector, and a classification cross-entropy loss for the curvature distribution parameters. The three losses are weighted and summed before joint optimization.

[0105] The physics inference branch maps the joint input to a predefined set of material categories and a physical property space, outputting the target object's material category label, mass range identifier, and surface friction coefficient level. The predefined material category set covers common material types such as metal, plastic, wood, glass, fabric, and rubber. Each material type corresponds to a specific gloss distribution, texture spectrum, and edge reflection characteristics. The branch network learns these visual cues to complete material recognition. The mass range identifier divides the object's mass into several discrete ranges (e.g., light, medium, and heavy), outputting range identifiers rather than continuous numerical values. This reduces the reliance on precise mass labeling data while providing sufficient coarse-grained information for the end effector's force control strategy. The surface friction coefficient level is also output using a discrete level identifier. The level division is determined based on the typical friction coefficient range of the object's material and surface treatment process. High friction levels correspond to rough or soft surfaces, while low friction levels correspond to smooth or hard surfaces. This information directly affects the clamping force parameter setting in the subsequent closed-loop position-force hybrid control.

[0106] The state reasoning branch, building upon the static attribute reasoning of the other three branches, introduces temporal information fusion. It temporally fuses the joint input with the target object's previous operational state feature vector. This temporal fusion employs a gated recurrent unit structure, using the current frame's joint input as the current input and the previous operational state feature vector as the hidden state. Through selective memory mechanisms of update and reset gates, historical state information is organically integrated with current observation information. The output of the state reasoning branch includes three types of state descriptions: switch state labels describing the object's open / closed, locked / unlocked, or other binary or multi-valued discrete states; content filling state labels describing the filling degree of container-like objects, such as empty / full / half-full; and a spatial pose stability evaluation value, a continuous scalar ranging from 0 to 1, reflecting the target object's current pose deviation from a stable equilibrium state. Values ​​close to 1 indicate high stability, while values ​​close to 0 indicate instability or a risk of tipping over. Spatial pose stability assessment value is of great safety significance for robot operation decision-making. When the value is lower than the preset safety threshold, the operation sequence planning module will prioritize the execution of stabilization operation steps before proceeding with subsequent grasping or movement operations.

[0107] After completing parallel inference across four branches, the category labels, geometric morphology parameter sets, physical attribute description values, and operation state label sets output by each branch are integrated into a structured attribute field set. Based on the unique identifier index of the target object node, each attribute field is backfilled into the corresponding attribute field position in the scene semantic graph. The backfilling operation employs an incremental update strategy: for newly entered object nodes, all attribute fields are fully written; for object nodes already existing in the graph, only attribute fields with confidence higher than historical records are updated, retaining attribute values ​​with higher confidence from historical inference, thus maintaining the continuous stability of graph attributes in dynamic scenes. After attribute backfilling, each node in the scene semantic graph no longer carries only a visual feature vector, but also contains a structured description across four dimensions: semantic category, geometric morphology, physical attributes, and operation state, forming a structured graph for scene understanding. This provides a complete and interpretable scene knowledge representation for subsequent task-oriented operation subgraph retrieval and operation sequence planning.

[0108] In the geometric reasoning branch, the joint input is decoded into the three-dimensional bounding box size parameters, principal axis direction vectors, and surface curvature distribution parameters of the target object, forming a geometric morphology parameter set, including:

[0109] The visual features of the target object are separated from the joint input. The visual features are then upsampled layer by layer and mapped to a normalized coordinate system with the centroid of the target object as the origin through a pre-trained 3D shape prior decoder, generating a dense 3D point cloud representation, in which each point carries spatial coordinates and local geometric confidence.

[0110] The spatial coordinates of each point in the dense 3D point cloud representation are constructed using a covariance matrix and eigenvalue decomposition. Three orthogonal eigenvectors are extracted as candidate principal axis directions. The projection span of the point cloud along each candidate principal axis direction is used as a candidate size parameter. The three orthogonal directions that maximize the variance of the projection span are selected as principal axis direction vectors, and the corresponding projection span is used as the 3D bounding box size parameter.

[0111] Based on the Poisson disk sampling strategy, multiple local neighborhood point sets are uniformly selected in the surface region represented by the dense three-dimensional point cloud. Local quadratic surface fitting is performed on each local neighborhood point set, and the Gaussian curvature value and average curvature value at each sampling point are calculated. The mean and standard deviation of the Gaussian curvature value and average curvature value of all sampling points are statistically calculated to form the surface curvature distribution parameters.

[0112] The three-dimensional bounding box size parameters, the principal axis direction vector, the mean and standard deviation of Gaussian curvature, and the mean and standard deviation of average curvature are combined into a geometric morphological parameter set.

[0113] The input to the geometric inference branch is a joint input tensor, which is composed of the visual features of the target object and the scene context features. After entering the geometric inference branch, the visual features belonging to the target object itself are first separated from the joint input. This separation operation is achieved through feature channel index slicing, ensuring that subsequent geometric decoding only operates on the feature dimensions directly related to the target object, avoiding interference from scene background features on geometric parameter estimation.

[0114] The separated visual features are then fed into a pre-trained 3D shape prior decoder. This decoder consists of multiple deconvolutional layers and residual upsampling modules stacked alternately, enabling it to upsample compact high-dimensional visual features layer by layer, gradually restoring spatial resolution and supplementing local detail information. The decoding process establishes a normalized coordinate system with the centroid of the target object as the origin. The spatial coordinates of all output points are expressed in this coordinate system, thus eliminating the influence of changes in the object's position and pose in the scene on geometric parameter estimation. The decoder ultimately outputs a dense 3D point cloud representation, where each point carries 3D spatial coordinates. In addition, a local geometric confidence scalar is included, which is given by the sigmoid activation output of the last layer of the decoder and reflects the geometric reliability of the point in the canonical coordinate system. Points with lower confidence will be assigned smaller weights in subsequent covariance analysis and curvature calculation to reduce the impact of noise points on the accuracy of parameter estimation.

[0115] After obtaining a dense 3D point cloud representation, a weighted covariance matrix is ​​constructed for the spatial coordinates of all points in the point cloud, with the weights representing the local geometric confidence of each point. Let the point cloud contain... The point, the first The coordinates of the points are Confidence level is The weighted centroid is The weighted covariance matrix is For the covariance matrix Eigenvalue decomposition yields three eigenvalues. and the corresponding three unit orthogonal eigenvectors These three feature vectors are the candidate principal axis directions, corresponding to the three orthogonal directions with the largest, second largest, and smallest variances of the point cloud distribution, respectively. Projecting the entire point cloud along each candidate principal axis direction, and calculating the difference between the maximum and minimum projection values ​​in each direction, gives the projection span for that direction, denoted as […]. Since the eigenvalue decompositions have been arranged in descending order of variance, the three orthogonal directions that maximize the variance of the projected span are... The corresponding 3D bounding box size parameters are The principal axis direction vector group is The above method can adaptively determine the minimum bounding box of an object, and it is also well applicable to irregularly shaped objects (such as curved pipes and irregularly shaped workpieces).

[0116] The calculation of surface curvature distribution parameters is based on a Poisson disk sampling strategy, which uniformly selects a local neighborhood point set in the surface region of a dense 3D point cloud. Poisson disk sampling ensures that the selected sampling points are uniformly distributed on the object surface by maintaining a minimum distance constraint between sampling points, thus avoiding excessive concentration of sampling points in a certain local area and resulting in curvature statistics bias. Let the sampling result be... Each sampling point ( ), with a fixed radius Searching for its local neighborhood point set in a point cloud The number of neighboring points must be no less than a preset lower limit (usually 8 to 15 points). If the number of neighboring points is insufficient, the search radius should be appropriately expanded until the requirement is met.

[0117] For each local neighborhood point set The local quadratic surface is fitted using the least squares method. Specifically, the sampling points are... A local coordinate system is established for the origin, where the direction of the normal vector is determined by principal component analysis of the neighborhood point set. After projecting the neighborhood points onto the tangent plane, a quadratic polynomial surface is fitted. ,in These are local coordinates within the tangent plane. The height is the normal vector direction, and the coefficient is... The solution is obtained by least squares. Based on the coefficients of the fitted surface, a shape operator (Weingarten mapping matrix) is constructed to calculate the Gaussian curvature value at the sampling point. With mean curvature value Gaussian curvature reflects the intrinsic curvature of a surface, while mean curvature reflects the extrinsic curvature; together, they describe the local geometry of an object's surface. For planar regions, both Gaussian curvature and mean curvature are close to zero; for spherical regions, Gaussian curvature is positive and mean curvature is constant; for saddle-shaped regions, Gaussian curvature is negative.

[0118] After obtaining all Gaussian curvature values ​​at each sampling point With mean curvature value Then, their mean and standard deviation were calculated separately. Mean Gaussian curvature Gaussian curvature standard deviation Mean curvature Mean curvature standard deviation These four statistics together constitute the surface curvature distribution parameters, which can describe the overall curvature distribution characteristics of the target object's surface in a compact form. The mean reflects the overall curvature of the surface, and the standard deviation reflects the dispersion of the curvature distribution. The combination of the two can distinguish between objects with uniformly curved surfaces (such as cylinders) and objects with drastic changes in surface curvature (such as complex curved parts).

[0119] Finally, the 3D bounding box size parameters Principal axis direction vector group Mean Gaussian curvature Gaussian curvature standard deviation Mean curvature and mean curvature standard deviation The parameters are combined to form a set of geometric parameters for the target object. This set of parameters is stored in a structured format and will later be populated into the attribute fields of the corresponding nodes in the scene semantic graph. This provides precise geometric basis for gripping point calculation, collision detection, and end effector posture optimization during the operation planning phase. For example, when gripping a slender rod-shaped object, the principal axis direction vector directly determines the opening and closing direction of the gripper; when operating a curved workpiece with a large curvature, the curvature distribution parameters can guide the selection of contact points and the estimation of suction force for the suction cup end effector, thereby improving the success rate of the operation.

[0120] Obtain a preset operation task description and encode it as a task semantic vector. Retrieve semantically matching candidate nodes and associated edges in the scene understanding structured graph. Construct a task-oriented operation subgraph containing operation objects, path constraints, and time sequence constraints, including:

[0121] The preset operation task description is segmented and contextual semantically encoded, and the encoding result is compressed into a fixed-dimensional task semantic vector through pooling.

[0122] Traverse the target object nodes in the structured graph of the scene understanding, calculate the cosine similarity between the attribute field concatenation vector of each node and the task semantic vector, and mark the nodes whose cosine similarity meets the preset threshold as candidate operation object nodes.

[0123] Starting with each candidate operation object node, a restricted breadth-first traversal is performed along the direction of the semantic relationship edges in the structured graph of the scene understanding. Within the traversal depth range, intermediate nodes and corresponding semantic relationship edges along the path are collected to construct a subset of candidate operation paths.

[0124] Based on the coordinate distribution of the candidate operation object nodes in the three-dimensional space of the scene and the spatial Euclidean length of the paths in each candidate operation path subset, the operation space path constraint parameters are determined.

[0125] Based on the logical preconditions of each subtask operation in the preset operation task description, determine the partial order constraints of the operation timing between the corresponding operation object nodes of different subtasks.

[0126] The candidate operation object node set, the candidate operation path subset, the operation space path constraint parameters, and the operation temporal partial order constraint are combined into a task-oriented operation subgraph.

[0127] When segmenting the pre-defined task description, a sub-word unit-based segmentation strategy is adopted to decompose the natural language description into a sequence of lexical units, preserving semantic integrity while covering low-frequency words and specialized operational terms. After segmentation, the lexical sequence is input into a contextual semantic encoder based on the Transformer architecture. A multi-layer self-attention mechanism captures long-range dependencies between lexical units, obtaining the contextual semantic representation corresponding to each lexical unit. To address the issue of inconsistent task description lengths, average pooling is performed on all lexical representations output by the encoder, compressing the variable-length sequence into a dense vector of fixed dimensions, which serves as the task semantic vector for subsequent graph retrieval and subgraph construction. This vector carries multi-dimensional information in the semantic space, such as operation type, operation object category preference, and task goal, providing a unified semantic anchor for subsequent node retrieval.

[0128] When traversing all target object nodes in the structured graph of scene understanding, for each node, its attribute fields—including object category label, geometric dimension parameters, physical material description, current operation state, etc.—are concatenated into a unified attribute string, and then converted into a node attribute vector with the same dimension as the task semantic vector through the same encoding and pooling process. Calculate the node attribute vector. With task semantic vector Cosine similarity between The calculation method is as follows .Will With preset search threshold Compare and satisfy The node is marked as a candidate operation object node and added to the candidate node set. The threshold setting needs to strike a balance between recall and precision. A threshold that is too low will introduce a large number of semantically irrelevant nodes, increasing the computational burden of subsequent path searching; a threshold that is too high may miss semantically related nodes with significant differences in representation. In practical applications, The threshold can be dynamically adjusted according to the task type. For example, for grasping tasks, the threshold can be appropriately increased to ensure the accuracy of the operation object, while for scene inspection tasks, it can be appropriately relaxed to cover a wider range of related nodes.

[0129] With candidate node set Each node in the graph serves as the starting node, and a restricted breadth-first traversal is performed along the directions of the semantic relationship edges in the scene understanding structured graph. A maximum traversal depth is set during the traversal. The search is conducted only along edges that satisfy specific semantic relationship constraints (such as predefined relationship types like "support," "containment," "adjacency," and "functional dependency"), excluding edge types irrelevant to the semantics of the current task. This ensures consistency in path semantics while controlling the size of the search space. In depth... Within the scope, collect all intermediate nodes and corresponding semantic relationship edges traversed on the traversal path. Record each complete path starting from the candidate operation object node and ending at a leaf node or the upper limit of the traversal depth as a candidate operation path. All candidate paths form a subset of candidate operation paths. In the case of multiple candidate starting nodes, a breadth-first traversal is performed independently for each node, and the respective path subsets are merged. After removing duplicate paths, a complete subset of candidate operation paths is formed.

[0130] Based on the coordinate distribution of candidate operation object nodes in the scene's 3D space, calculate the candidate node set. The spatial envelope range is defined, with the maximum span of the three-dimensional coordinates of each node serving as the boundary reference for the operation space. This applies to a subset of candidate operation paths. For each path in the algorithm, the spatial Euclidean length of the path is obtained by successively accumulating the Euclidean distances of adjacent nodes in three-dimensional space. After sorting all candidate paths by their spatial Euclidean lengths, and considering the task type's requirements for the operational scope, the upper limit of the path length is determined. Path length lower limit The length constraint parameters are the core components of the operation space path constraint parameters. Candidate paths that exceed the length constraint range are filtered out, and paths that satisfy the constraints are retained for subsequent subgraph assembly. The operation space path constraint parameters also include end effector reachability constraints, that is, the three-dimensional coordinates of each node on the path must fall within the workspace envelope of the robot's end effector; paths that do not meet the reachability requirements are also eliminated.

[0131] Based on the logical preconditions between subtasks in the predefined task description, partial order constraints are extracted. Specifically, dependency parsing is performed on the task description to identify temporal connectors (such as "first...then...", "after completing...", "under the premise of...", etc.) and conditional clause structures, from which the preconditions between subtasks are parsed. Each subtask is then mapped to its corresponding candidate operation object node, constructing a set of partial order constraints between nodes. Each of the partial order constraints Indicates a node Operations on the corresponding object must be performed on the node. Operations on the corresponding object are completed before they begin. Partial order constraints are stored in the form of a directed acyclic graph, facilitating topological sorting during the subsequent hierarchical operation sequence planning phase and ensuring the logical validity of the operation execution order. For subtasks whose temporal relationships are not explicitly specified in the task description, their operation order is assumed to have no mandatory constraints, allowing for flexible adjustment of the execution order during the motion planning phase based on the end effector position and path cost.

[0132] Set of candidate operation object nodes A subset of candidate operation paths that satisfy spatial path constraints Operation space path constraint parameters (including upper and lower limits of path length) , and end effector reachability boundary) and operation timing partial order constraint set This is combined into a task-oriented operation subgraph. This subgraph uses candidate operation object nodes as its core, intermediate nodes and semantic relationship edges involved in the candidate operation paths as its skeleton, and adds spatial and temporal constraint attributes to form a structurally complete, semantically clear, and constraint-executable operation subgraph representation. Compared to a complete scene understanding structured graph, the task-oriented operation subgraph significantly reduces its size, retaining only nodes and edges directly related to the current operation task. This reduces the search complexity in the subsequent operation sequence planning stage. Simultaneously, through explicit encoding of path and temporal constraints, it provides structured task prior information for the generation of hierarchical operation sequences, ensuring that the planning results meet the needs of actual robot operations in both spatial feasibility and logical validity dimensions.

[0133] Based on the task-oriented operation subgraph, and combined with robot kinematic constraints and end effector boundary hierarchical planning, a hierarchical operation sequence consisting of atomic operation steps is generated. Each step includes the operation object, operation type, target pose, and prerequisite dependencies, including:

[0134] The task-oriented operation subgraph is topologically sorted, and each operation object node is arranged into a linear execution order according to the operation timing partial order constraint, forming a coarse-grained subtask sequence.

[0135] For each subtask in the coarse-grained subtask sequence, according to its operation object type and target state requirements, it is decomposed into at least one atomic operation step among approach action, grasp action, move action and release action, and a corresponding operation type label is assigned to each atomic operation step.

[0136] For each atomic operation step, the geometric morphological parameters and physical properties of the operation object are used as constraints. Inverse kinematics solution is performed in the reachable workspace of the robot end effector to obtain the target pose parameters that satisfy joint limit and collision avoidance constraints.

[0137] Based on the topological sorting results of the coarse-grained subtask sequence and the execution order of each atomic operation step within its respective subtask, establish the pre-dependencies between atomic operation steps and mark the pre-dependency condition identifiers of each atomic operation step.

[0138] The operation object identifier, operation type label, target pose parameter and prerequisite dependency condition identifier of each atomic operation step are encapsulated into hierarchical operation step data records, and arranged in the execution order to generate the hierarchical operation sequence.

[0139] After obtaining the task-oriented operation subgraph, it needs to be transformed into a hierarchical sequence of operations that the robot can directly execute. This process starts with the topology of the operation subgraph, gradually refines it to the atomic operation level, and attaches complete execution attributes to each atomic operation step, ultimately forming a structured set of operation instructions.

[0140] When performing topological sorting on the task-oriented operation subgraph, the node preorder relationships recorded in the operation temporal partial order constraint set are used as the sorting basis. Specifically, each operation object node in the operation subgraph is regarded as a vertex of a directed acyclic graph, and the partial order constraint relationship is regarded as a directed edge. The Kahn algorithm is used to perform topological sorting on the nodes in the graph. During the sorting process, a queue of nodes with an in-degree of zero is maintained. Each time, the node with the smallest in-degree is taken from the queue and added to the linear sequence, and the in-degree of its successor node is decremented by one. If the in-degree of a successor node drops to zero, it is enqueued. When there are multiple nodes with an in-degree of zero in the operation subgraph, priority sorting is performed according to the spatial position relationship of the operation object corresponding to each node in the scene understanding structured graph. Operation object nodes that are closer to the current position of the robot's end effector in space are processed first to reduce the path cost of subsequent motion planning. After the topological sorting is completed, a linear execution order of all operation object nodes arranged according to temporal constraints is obtained, forming a coarse-grained subtask sequence. Each subtask corresponds to an operation object node and its associated target state requirements.

[0141] After the coarse-grained subtask sequence is determined, atomic operations are decomposed for each subtask. The decomposition strategy is jointly determined based on the type of the operation object and the target state requirements of the subtask. The type of operation object is derived from the attribute inference results of the corresponding node in the scene understanding structured graph, including different categories such as rigid objects, flexible objects, and container objects; the target state requirements are derived from the state annotations of the nodes in the task-oriented operation subgraph, describing the position, posture, or functional state that the operation object should be in after the operation is completed. For typical grasp-place type subtasks, they are decomposed into four atomic operation steps: approach action, grasp action, move action, and release action; for subtasks that only need to adjust the object posture, they are decomposed into approach action, grasp action, rotation adjustment action, and release action; for push type operations, they only contain two atomic steps: approach action and push action. Each atomic operation step is assigned a corresponding operation type label. The label system adopts a predefined set of operation primitives, including types such as APPROACH, GRASP, MOVE, RELEASE, PUSH, and ROTATE, to ensure that the label semantics are clear and correspond to the instruction parsing interface of the motion control layer.

[0142] When solving for the target pose parameters for each atomic operation step, the geometric parameters and physical properties of the manipulated object are used as the core constraints. The geometric parameters are derived from the bounding box size, principal axis direction, and centroid coordinates stored in the corresponding nodes of the scene understanding structured graph; physical properties include object mass estimation, surface friction coefficient, and fragility markers. For approach actions, the target pose is set as a pre-grasping pose with the end effector offset by a certain safe distance along the principal axis direction of the manipulated object. This safe distance is determined jointly by the geometry of the end effector and the bounding box size of the manipulated object. For grasping actions, candidate grasping poses are enumerated within the reachable workspace of the end effector. These candidate poses are generated from a set of stable grasping points on the surface of the manipulated object. The selection of stable grasping points comprehensively considers curvature distribution, normal vector direction, and relative positional relationship with the object's centroid. For each candidate grasping pose, inverse kinematics is performed to convert the end effector's target pose into the target angle configuration of each joint of the robot. The inverse kinematics solution employs a numerical iterative method based on the pseudo-inverse of the Jacobian matrix, simultaneously applying joint constraint and collision avoidance constraints during the iteration process. Joint constraint requires that the angles of each joint not exceed their physical travel range; collision avoidance constraints are achieved by maintaining a minimum safe distance between the robot links and obstacles in the scene, with obstacle information extracted from the geometric parameters of non-operational object nodes in the scene understanding structured graph. When the inverse kinematics solution for a candidate grasping pose fails or violates constraints, the solution switches to the next candidate pose until a valid target pose parameter satisfying all constraints is obtained. For movement actions, the target pose is directly given by the target state annotation of the operation object node in the task-oriented operation subgraph, and is also converted into joint angle configuration through inverse kinematics solution. The target pose for the release action is the target pose of the movement action superimposed with the end effector opening action, requiring no additional inverse kinematics solution.

[0143] The establishment of prerequisite dependencies is divided into two levels. At the subtask level, based on the topological sorting result, the first atomic operation step of a subsequent subtask depends on the completion of the last atomic operation step of a preceding subtask. Within a subtask, each atomic operation step depends on the previous atomic operation step in the decomposition sequence, i.e., each atomic operation step uses its preceding atomic operation step in the decomposition sequence as its prerequisite dependency. For parallel executable atomic operation steps, when the operation objects corresponding to two steps do not interfere spatially and the end effector trajector trajector trajector does not conflict, they can be marked as having no mutual dependency, supporting parallel scheduling by the subsequent control layer. The prerequisite dependency condition identifier of each atomic operation step is recorded in the form of a unique set of step numbers. An empty set indicates that the step has no prerequisite dependencies and can be used as the starting step of the execution sequence.

[0144] After calculating and labeling the above parameters, the operation object identifier, operation type label, target pose parameters, and pre-dependency identifiers for each atomic operation step are encapsulated into hierarchical operation step data records. The operation object identifier corresponds to the unique node number in the scene understanding structured graph, ensuring that subsequent execution stages can trace back to the complete node attributes in the graph. The target pose parameters are stored in the form of homogeneous transformation matrices, including the position vector and rotation matrix of the end effector in the robot's base coordinate system. All data records of atomic operation steps are arranged in the execution order determined by the pre-dependencies, forming the final hierarchical operation sequence. This sequence retains both the coarse-grained structural information of the sub-task level and the fine-grained execution parameters of the atomic operation level, providing complete and unambiguous operation planning input for the subsequent closed-loop position-force hybrid control stage.

[0145] A second aspect of this invention provides a robot scene understanding and operation control system based on a large visual model, comprising:

[0146] The semantic graph construction unit is used to collect visual data of the robot's operation scene, and construct a unified scene representation containing geometric and appearance features through spatiotemporal alignment and fusion. The unified scene representation is input into a pre-trained large visual model to extract visual features of target objects, encode the spatial topological relationships, semantic associations and task context between target objects, and generate a scene semantic graph with target objects as nodes and semantic relationships as edges.

[0147] The attribute reasoning unit is used to perform attribute reasoning on each node in the scene semantic graph, determine the category, geometric parameters, physical attributes and operation status of each target object, and fill the reasoning results back into the corresponding node to obtain a scene understanding structured graph.

[0148] The operation subgraph unit is used to obtain a preset operation task description and encode it into a task semantic vector, retrieve candidate nodes and associated edges with semantic matching in the scene understanding structured graph, and construct a task-oriented operation subgraph containing operation objects, path constraints and time sequence constraints.

[0149] The operation sequence unit is used to generate a hierarchical operation sequence consisting of atomic operation steps based on the task-oriented operation subgraph, combined with robot kinematic constraints and end effector boundary hierarchical planning. According to the execution order and prerequisite dependencies of the hierarchical operation sequence, a motion control command sequence adapted to the robot joints and end effector is generated by closed-loop position-force hybrid control to drive the robot to complete the scene operation task.

[0150] A third aspect of the present invention provides an electronic device, comprising:

[0151] processor;

[0152] Memory used to store processor-executable instructions;

[0153] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0154] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0155] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot scene understanding and operation control method based on a large visual model, characterized in that, include: Visual data of robot operation scenarios are collected, and a unified scene representation containing geometric and appearance features is constructed through spatiotemporal alignment and fusion. The unified scene representation is input into a pre-trained large visual model to extract visual features of target objects, encode the spatial topological relationships, semantic associations and task contexts between target objects, and generate a scene semantic graph with target objects as nodes and semantic relationships as edges. Attribute reasoning is performed on each node in the scene semantic graph to determine the category, geometric parameters, physical attributes and operation state of each target object. The reasoning results are then backfilled into the corresponding nodes to obtain a structured graph of scene understanding. Obtain a preset operation task description and encode it as a task semantic vector. Retrieve candidate nodes and associated edges with semantic matching in the scene understanding structured graph and construct a task-oriented operation subgraph that includes operation objects, path constraints, and time sequence constraints. Based on the task-oriented operation subgraph, combined with robot kinematic constraints and end effector boundary hierarchical planning, a hierarchical operation sequence consisting of atomic operation steps is generated. According to the execution order and prerequisite dependencies of the hierarchical operation sequence, a motion control command sequence adapted to the robot joints and end effector is generated using closed-loop position-force hybrid control to drive the robot to complete the scene operation task.

2. The method according to claim 1, characterized in that, The unified representation of the scene is input into a pre-trained large-scale visual model to extract visual features of target objects, encode the spatial topological relationships, semantic associations, and task context between target objects, and generate a scene semantic graph with target objects as nodes and semantic relationships as edges, including: The scene is uniformly represented by dividing it into multiple local scene blocks with overlapping boundaries according to a spatial grid. Target detection and feature extraction are performed on each block to obtain the local visual features and bounding box coordinates of the target objects in each block. For target objects detected in overlapping areas of adjacent blocks, cross-block consistency matching is performed based on the bounding box intersection-union ratio and local visual feature cosine similarity to establish a cross-block target object correspondence table. Based on the cross-block target object correspondence table, the local visual features of the same target object in each block are stitched together and linearly projected to obtain the global visual feature representation vector of each target object. Using the global visual feature representation vector of each target object as node input, the spatial topological attention weight matrix and semantic association attention weight matrix between target objects are learned in parallel. The spatial topology attention weight matrix and the semantic association attention weight matrix are weighted and fused according to learnable fusion coefficients, and channel modulation is performed with the task context semantic guidance vector to generate a comprehensive relationship strength matrix between target object pairs. The connection relationships corresponding to object pairs whose comprehensive relationship strength matrix values ​​meet the threshold are determined as semantic relationship edges, and a scene semantic graph is constructed with the target object as the node and the semantic relationship as the edge.

3. The method according to claim 2, characterized in that, The spatial topology attention weight matrix and the semantic association attention weight matrix are weighted and fused according to learnable fusion coefficients, and then channel-modulated with the task context semantic guidance vector to generate a comprehensive relationship strength matrix between target object pairs, including: Obtain the task type label of the current operation task, input the task type label into the pre-trained task semantic encoder to generate task semantic context features, and retrieve the basic semantic guidance vector with the smallest cosine distance to the task semantic context features from the pre-built task semantic guidance vector library; The basic semantic guidance vector is transformed element-wise through a learnable gating network consisting of fully connected layers and nonlinear activation functions to generate a channel-level modulation coefficient vector in which each element takes values ​​within a preset range. A weighted linear combination of each element of the spatial topology attention weight matrix and the corresponding element of the semantic association attention weight matrix is ​​performed using learnable scalar fusion coefficients to generate a fused attention weight matrix. The fusion attention weight matrix is ​​broadcast multiplied along the feature channel dimension with the channel-level modulation coefficient vector, so that the features of different channels obtain differentiated task semantic modulation intensity, and a comprehensive relation strength matrix modulated by task semantic is obtained. The element values ​​of each row in the comprehensive relation strength matrix modulated by the task semantics are sorted in descending order. Only the element values ​​at the first preset number of sorted positions in each row are kept unchanged. The element values ​​exceeding the preset number are forcibly set to zero. The non-zero element values ​​retained in each row are normalized to generate a sparse comprehensive relation strength matrix.

4. The method according to claim 1, characterized in that, Attribute reasoning is performed on each node in the scene semantic graph to determine the category, geometric parameters, physical properties, and operational state of each target object. The reasoning results are then backfilled into the corresponding nodes to obtain a structured scene understanding graph, including: Using the visual features associated with each target object node in the scene semantic graph and the semantic relationship edge features of adjacent nodes as joint inputs, classification reasoning, geometric reasoning, physical reasoning and state reasoning are executed in parallel through a multi-branch attribute reasoning network. In the classification reasoning branch, the joint input is mapped to a predefined object category label space, and the fine-grained category label and functional category affiliation of the target object are output. In the geometric reasoning branch, the joint input is decoded into the three-dimensional bounding box size parameters, principal axis direction vector and surface curvature distribution parameters of the target object, forming a geometric morphology parameter set; In the physics reasoning branch, the joint input is mapped to a preset set of material categories and physical property space, and the material category label, mass range identifier and surface friction coefficient level of the target object are output. In the state reasoning branch, the joint input is fused with the operation state of the target object in the previous time step, and the current switch state label, content filling state label and spatial pose stability evaluation value of the target object are output. The category labels, geometric morphology parameter groups, physical attribute description values, and operation status label groups output by each inference branch are used as attribute fields and backfilled into the attribute fields of the corresponding target object nodes in the scene semantic graph to obtain a structured graph of scene understanding.

5. The method according to claim 4, characterized in that, In the geometric reasoning branch, the joint input is decoded into the three-dimensional bounding box size parameters, principal axis direction vectors, and surface curvature distribution parameters of the target object, forming a geometric morphology parameter set, including: The visual features of the target object are separated from the joint input. The visual features are then upsampled layer by layer and mapped to a normalized coordinate system with the centroid of the target object as the origin through a pre-trained 3D shape prior decoder, generating a dense 3D point cloud representation, in which each point carries spatial coordinates and local geometric confidence. The spatial coordinates of each point in the dense 3D point cloud representation are constructed using a covariance matrix and eigenvalue decomposition. Three orthogonal eigenvectors are extracted as candidate principal axis directions. The projection span of the point cloud along each candidate principal axis direction is used as a candidate size parameter. The three orthogonal directions that maximize the variance of the projection span are selected as principal axis direction vectors, and the corresponding projection span is used as the 3D bounding box size parameter. Based on the Poisson disk sampling strategy, multiple local neighborhood point sets are uniformly selected in the surface region represented by the dense three-dimensional point cloud. Local quadratic surface fitting is performed on each local neighborhood point set, and the Gaussian curvature value and average curvature value at each sampling point are calculated. The mean and standard deviation of the Gaussian curvature value and average curvature value of all sampling points are statistically calculated to form the surface curvature distribution parameters. The three-dimensional bounding box size parameters, the principal axis direction vector, the mean and standard deviation of Gaussian curvature, and the mean and standard deviation of average curvature are combined into a geometric morphological parameter set.

6. The method according to claim 1, characterized in that, Obtain a preset operation task description and encode it as a task semantic vector. Retrieve semantically matching candidate nodes and associated edges in the scene understanding structured graph. Construct a task-oriented operation subgraph containing operation objects, path constraints, and time sequence constraints, including: The preset operation task description is segmented and contextual semantically encoded, and the encoding result is compressed into a fixed-dimensional task semantic vector through pooling. Traverse the target object nodes in the structured graph of the scene understanding, calculate the cosine similarity between the attribute field concatenation vector of each node and the task semantic vector, and mark the nodes whose cosine similarity meets the preset threshold as candidate operation object nodes. Starting with each candidate operation object node, a restricted breadth-first traversal is performed along the direction of the semantic relationship edges in the structured graph of the scene understanding. Within the traversal depth range, intermediate nodes and corresponding semantic relationship edges along the path are collected to construct a subset of candidate operation paths. Based on the coordinate distribution of the candidate operation object nodes in the three-dimensional space of the scene and the spatial Euclidean length of the paths in each candidate operation path subset, the operation space path constraint parameters are determined. Based on the logical preconditions of each subtask operation in the preset operation task description, determine the partial order constraints of the operation timing between the corresponding operation object nodes of different subtasks. The candidate operation object node set, the candidate operation path subset, the operation space path constraint parameters, and the operation temporal partial order constraint are combined into a task-oriented operation subgraph.

7. The method according to claim 1, characterized in that, Based on the task-oriented operation subgraph, and combined with robot kinematic constraints and end effector boundary hierarchical planning, a hierarchical operation sequence consisting of atomic operation steps is generated. Each step includes the operation object, operation type, target pose, and prerequisite dependencies, including: The task-oriented operation subgraph is topologically sorted, and each operation object node is arranged into a linear execution order according to the operation timing partial order constraint, forming a coarse-grained subtask sequence. For each subtask in the coarse-grained subtask sequence, according to its operation object type and target state requirements, it is decomposed into at least one atomic operation step among approach action, grasp action, move action and release action, and a corresponding operation type label is assigned to each atomic operation step. For each atomic operation step, the geometric morphological parameters and physical properties of the operation object are used as constraints. Inverse kinematics solution is performed in the reachable workspace of the robot end effector to obtain the target pose parameters that satisfy joint limit and collision avoidance constraints. Based on the topological sorting results of the coarse-grained subtask sequence and the execution order of each atomic operation step within its respective subtask, establish the pre-dependencies between atomic operation steps and mark the pre-dependency condition identifiers of each atomic operation step. The operation object identifier, operation type label, target pose parameter and prerequisite dependency condition identifier of each atomic operation step are encapsulated into hierarchical operation step data records, and arranged in the execution order to generate the hierarchical operation sequence.

8. A robot scene understanding and operation control system based on a large visual model, used to implement the method as described in any one of claims 1-7, characterized in that, include: The semantic graph construction unit is used to collect visual data of the robot's operation scene, and construct a unified scene representation containing geometric and appearance features through spatiotemporal alignment and fusion. The unified scene representation is input into a pre-trained large visual model to extract visual features of target objects, encode the spatial topological relationships, semantic associations and task context between target objects, and generate a scene semantic graph with target objects as nodes and semantic relationships as edges. The attribute reasoning unit is used to perform attribute reasoning on each node in the scene semantic graph, determine the category, geometric parameters, physical attributes and operation status of each target object, and fill the reasoning results back into the corresponding node to obtain a scene understanding structured graph. The operation subgraph unit is used to obtain a preset operation task description and encode it into a task semantic vector, retrieve candidate nodes and associated edges with semantic matching in the scene understanding structured graph, and construct a task-oriented operation subgraph containing operation objects, path constraints and time sequence constraints. The operation sequence unit is used to generate a hierarchical operation sequence consisting of atomic operation steps based on the task-oriented operation subgraph, combined with robot kinematic constraints and end effector boundary hierarchical planning. According to the execution order and prerequisite dependencies of the hierarchical operation sequence, a motion control command sequence adapted to the robot joints and end effector is generated by closed-loop position-force hybrid control to drive the robot to complete the scene operation task.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images