Robot action planning method and device, terminal and medium

Abstract

The application provides a robot action planning method, device, terminal and medium, and the method comprises the following steps: acquiring a user instruction, a scene perception image set, sensor information and a safety threshold table; generating an action draft according to the user instruction and the scene perception image set; performing virtual simulation on the scene perception image set and the sensor information according to a dynamic virtual twin method to obtain virtual twin state information; processing the action draft and the virtual twin state information to determine a risk vector of the current iteration, generating an iterated action draft based on the risk vector of the current iteration and the safety threshold table, and obtaining iterated virtual twin state information; and determining an iterated risk vector according to the iterated action draft and the iterated virtual twin state information, and obtaining a robot action instruction. The method realizes accurate physical consequence prediction for a local and dynamically changing interactive scene without relying on a pre-established perfect global physical model.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing technology, specifically to a robot motion planning method, device, terminal, and medium. Background Technology

[0002] Robot motion planning and control is a core topic in robotics, and its development reflects an evolution from precise modeling to data-driven approaches, and then to deep cognitive integration. In its early stages, this field primarily relied on deterministic methods based on geometric and dynamic models. Researchers built precise mathematical models of the robot and its environment, using algorithms such as Rapid Expanding Random Trees (RRTs) or optimization techniques to plan collision-free, dynamically feasible trajectories. These methods achieved great success in structured, predictable environments (such as industrial assembly lines), but their limitations lie in the extremely high precision requirements of the models and their severely insufficient adaptability and generalization capabilities when facing unstructured, dynamically changing environments and interacting with unknown objects.

[0003] To overcome the rigidity of traditional methods, research has gradually shifted towards data-driven approaches and machine learning. The rise of reinforcement learning (RL) and imitation learning (IL) has enabled robots to acquire complex skills through trial-and-error interactions with their environment or by learning from human examples. These methods reduce reliance on precise physical models and have shown potential in handling complex contact tasks such as grasping and assembly. However, learning-based methods often require massive amounts of interaction data, and the training process is lengthy and "black box," making it difficult to guarantee the safety and interpretability of their strategies. This is especially true in critical safety applications where the cost of "trial and error" is unacceptable.

[0004] In recent years, breakthroughs in Large Language Models (LLM) and Multimodal Large Models (VLM) have ushered in a new paradigm revolution in the field of robotics. These models endow robots with new capabilities in semantic understanding, common-sense reasoning, and long-term task planning, enabling them to parse complex natural language instructions and generate high-level action strategies, and are considered key to building the "brain" of robots. However, this also brings new challenges: a "semantic-physical gap" exists between the abstract intentions generated by the large model and the robot's underlying physical execution. The action plans generated by the model often lack a nuanced causal understanding of the physical world, such as the transmission of forces, changes in friction, and the deformation and breakage limits of objects. Directly executing these plans can easily lead to task failure or even physical damage. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a robot motion planning method, which aims to accurately predict the physical consequences of local, dynamically changing interactive scenarios without relying on a pre-established, perfect global physical model.

[0006] In a first aspect, embodiments of this application provide a robot motion planning method, including:

[0007] Acquire user commands, scene-aware image sets, sensor information, and safety threshold tables;

[0008] Generate an action draft based on the user instructions and the scene-aware image set;

[0009] The scene perception image set and the sensor information are processed according to the dynamic virtual twin method to obtain virtual twin state information;

[0010] The action draft and the virtual twin state information are processed by a pre-trained physical causal prediction network model to determine the risk vector of the current iteration. If the risk vector of the current iteration does not meet the safety threshold table, the action draft for the next iteration is generated based on the risk vector of the current iteration. The virtual twin state information for the next iteration is obtained based on the scene perception image set, the sensor information, and the risk vector of the current iteration. The risk vector for the next iteration is determined based on the action draft and the virtual twin state information of the next iteration. The action draft corresponding to the risk vector that meets the safety threshold table is used as the robot action instruction.

[0011] Optionally, the step of processing the scene-aware image set and the sensor information according to the dynamic virtual twin method to obtain virtual twin state information includes:

[0012] Based on the scene-perceived image set and the sensor information, interactive dynamic information is obtained;

[0013] Objects are identified in the scene-perceived image set using an object detection model to obtain images of the target objects;

[0014] Based on the target object image and the scene perception image set, physical attribute prediction information is obtained;

[0015] The interactive dynamic information and the physical attribute prediction information are packaged together to obtain the virtual twin state information.

[0016] Optionally, obtaining the virtual twin state information for the next iteration based on the scene-aware image set, the sensor information, and the risk vector of the current iteration includes:

[0017] Based on the scene-perceived image set and the sensor information, interactive dynamic information is obtained;

[0018] Objects are identified in the scene-perceived image set using an object detection model to obtain images of the target objects;

[0019] Check whether the risk vector in the current iteration satisfies the high-quality sensitivity condition;

[0020] If the risk vector of the current iteration meets the high-quality sensitivity condition, a trial action is performed on the target object, and the trial acquisition information of the target object is obtained; the image of the target object is input into a pre-trained physical attribute prediction model to obtain centroid coordinate information and shape description information; based on the trial acquisition information of the target object and the scene perception image set, physical attribute prediction information is obtained; the interactive dynamic information and the physical attribute prediction information are packaged to obtain the virtual twin state information for the next iteration.

[0021] If the risk vector of the current iteration does not meet the high-quality sensitivity condition, physical attribute prediction information is obtained based on the target object image and the scene perception image set; the interaction dynamic information and the physical attribute prediction information are packaged together to obtain the virtual twin state information for the next iteration.

[0022] Optionally, the action draft and the virtual twin state information are processed by a pre-trained physical causal prediction network model to determine the risk vector for the current iteration, including:

[0023] Node attribute information, node state information, and edge type information are obtained based on the virtual twin state information;

[0024] Based on the node attribute information, the node state information, and the action draft, the initial node hidden state and the global action vector are obtained;

[0025] The initial hidden edge state is obtained based on the node attribute information, the node state information, and the edge type information;

[0026] The initial node hidden state, the global action vector, the initial edge hidden state, and the edge type information are input into a processor composed of N stacked graph network blocks for iteration to obtain the final node hidden state;

[0027] Input all the hidden states of the final nodes into the pre-trained task decoder for decoding to obtain the risk vector of the current iteration.

[0028] Optionally, obtaining the initial hidden state of the node and the global action vector based on the node attribute information, the node state information, and the action draft includes:

[0029] The node attribute information is input into the node attribute encoder for encoding to obtain the initial node attribute latent vector.

[0030] The node state information is input into the node state encoder for encoding to obtain the initial node state hidden vector;

[0031] Based on the initial node attribute hidden vector information and the initial node state hidden vector information, the initial node hidden state is obtained;

[0032] The motion draft is input into the motion encoder for encoding to obtain the global motion vector.

[0033] Optionally, obtaining the initial hidden edge state based on the node attribute information, the node state information, and the edge type information includes:

[0034] Based on the node attribute information and the node state information, the original geometric feature information of the edge is obtained;

[0035] The original geometric feature information of the edge is input into the edge encoder to obtain the initial hidden edge state.

[0036] Optionally, the step of inputting the initial node hidden state, the global action vector, the initial edge hidden state, and the edge type information into a processor composed of N stacked graph network blocks for iteration to obtain the final node hidden state includes:

[0037] Based on the initial node hidden state and the initial edge hidden state, the current updated edge hidden state is obtained;

[0038] The hidden state of the current iteration node is obtained based on the current updated edge hidden state, the global action vector, and the edge type information;

[0039] Based on the current iteration node hidden state, the current update edge hidden state, and the global action vector, the next iteration node hidden state is obtained, and after N iterations, the node hidden state of the Nth iteration is taken as the final node hidden state.

[0040] Optionally, obtaining the hidden state of the current iteration node based on the current updated edge hidden state, the global action vector, and the edge type information includes:

[0041] Detect whether the edge type information is a contact edge;

[0042] If the edge type information is a contact edge, then the current updated edge hidden state is input into the contact interactive encoder for encoding to obtain the current contact edge hidden state;

[0043] If the edge type information is not contact edge information, then the current updated edge hidden state is input into the neighbor interactive encoder for encoding to obtain the current neighbor edge hidden state;

[0044] Based on the current hidden state of the contact edge and / or the current hidden state of the neighboring edge, node aggregation information is obtained;

[0045] The hidden state of the current iteration node is obtained based on the initial node hidden state, the global action vector, and the node aggregation information.

[0046] Secondly, embodiments of this application provide a robot motion planning device, comprising:

[0047] The information acquisition module is used to acquire user commands, scene-aware image sets, sensor information, and safety threshold tables;

[0048] The action draft determination module is used to generate an action draft based on the user instruction and the scene-aware image set;

[0049] The twin information determination module is used to process the scene perception image set and the sensor information according to the dynamic virtual twin method to obtain virtual twin state information;

[0050] The action instruction determination module is used to process the action draft and the virtual twin state information through a pre-trained physical causal prediction network model to determine the risk vector of the current iteration. If the risk vector of the current iteration does not meet the safety threshold table, the module generates the action draft for the next iteration based on the risk vector of the current iteration, and obtains the virtual twin state information for the next iteration based on the scene perception image set, the sensor information, and the risk vector of the current iteration. The module then determines the risk vector for the next iteration based on the action draft and the virtual twin state information of the next iteration, until the risk vector meets the safety threshold table. After that, the action draft corresponding to the risk vector that meets the safety threshold table is used as the robot action instruction.

[0051] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the robot motion planning method as described in any one of the first aspects above.

[0052] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the robot motion planning method as described in any one of the first aspects above.

[0053] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the robot motion planning method described in any one of the first aspects.

[0054] In this embodiment, user instructions, a scene-aware image set, sensor information, and a safety threshold table are acquired. An action draft is generated based on the user instructions and the scene-aware image set. The scene-aware image set and the sensor information are processed using a dynamic virtual twin method to obtain virtual twin state information. A pre-trained physical causal prediction network model is used to process the action draft and the virtual twin state information to determine the risk vector for the current iteration. If the risk vector for the current iteration does not conform to the safety threshold table, an action draft for the next iteration is generated based on the risk vector for the current iteration. The virtual twin state information for the next iteration is obtained based on the scene-aware image set, the sensor information, and the risk vector for the current iteration. The risk vector for the next iteration is determined based on the action draft and the virtual twin state information for the next iteration. This process continues until the risk vector conforms to the safety threshold table, at which point the action draft corresponding to the risk vector conforming to the safety threshold table is used as the robot's action instruction. This achieves accurate prediction of physical consequences for locally changing, dynamically alternating interactive scenarios without relying on a pre-established, perfect global physical model. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating a robot motion planning method provided in an embodiment of this application;

[0056] Figure 2 This is a schematic diagram of the system framework of the robot motion planning method provided in this application;

[0057] Figure 3 This is a schematic diagram of the data processing of the physical causal prediction network model of the robot motion planning method provided in this application;

[0058] Figure 4 This is a flowchart illustrating the task completion process of the robot motion planning method provided in this application;

[0059] Figure 5 This is the core verification loop flowchart of the robot motion planning method provided in this application;

[0060] Figure 6 This is a schematic diagram of the robot motion planning device provided in the embodiments of this application;

[0061] Figure 7 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0062] Figure 1 This illustration shows a schematic flowchart of a robot motion planning method provided in an embodiment of this application. It is provided as an example and not as a limitation. This method can be applied to the aforementioned robot motion planning device, or it can be a method for users or operators to operate and make judgments on the robot motion planning device. Figure 1 As shown, the method may include:

[0063] S10: Acquire user commands, scene-aware image set, sensor information, and safety threshold table;

[0064] To achieve accurate prediction of physical consequences in locally changing, dynamically altered interactive scenarios without relying on a pre-established, perfect global physical model, the robot motion planning device acquires user commands, a set of scene-aware images, sensor information, and a safety threshold table. The flowchart of the robot motion planning method is as follows: Figure 2 As shown.

[0065] The user commands are natural language instructions given by the user, such as "hand me the vase" or "please move that glass vase on the table to the empty space next to it." The scene-aware image set consists of RGB-D images continuously captured from different perspectives by RGB-D cameras mounted on the robot's wrist or head; RGB images (scene-aware image set) are used for visual feature extraction, and depth images (scene-aware image set) are used for 3D geometric information acquisition. The scene-aware image set can include color image sets and depth image sets. Sensor information is the sensing information acquired by force / tactile sensors located on the robot's gripper, providing contact information for the robot. The sensor information also includes the sensing information acquired by a six-axis force / torque sensor located at the robot's end effector.

[0066] The safety threshold table is pre-stored in the robot's memory or downloaded from a cloud knowledge base. The safety threshold table is a mapping data structure (such as a hash table or JSON object) that defines the physical limits allowed for different object categories or materials during interaction. Specifically, it includes, but is not limited to: (1) the maximum allowable stress threshold, setting a lower threshold (such as 5N) for fragile objects (such as a glass) and a higher threshold (such as 50N) for rigid objects (such as a metal block). (2) the minimum stability probability threshold, for example, set to 99%, meaning that the predicted probability of the object tipping over or slipping must not exceed 1%. (3) the maximum allowable slip probability threshold, for example, set to 5%. During runtime, the decision controller queries the object's attributes (such as material) based on the "target object ID" in the action draft, and then indexes the corresponding specific threshold parameters from the safety threshold table for subsequent risk vector comparison. The safety threshold table can be {If the target object is fragile: 5.0 stress threshold = 5.0; otherwise: stress threshold = 20.0; success probability threshold = 0.99; slip probability threshold = 0.1}.

[0067] S20, Generate an action draft based on the user instruction and the scene-aware image set;

[0068] After acquiring user instructions and a set of scene-aware images, the robot motion planning device generates a draft motion based on the user instructions and the set of scene-aware images.

[0069] As one implementation, generating an action draft based on the user instruction and the scene-aware image set may include:

[0070] A1, Select a scene-aware image from the scene-aware image set;

[0071] Select a scene-aware image from the scene-aware image set (containing continuous time-series image frames). The specific selection criteria include: (1) Time timeliness criterion: prioritize the image frame with the latest timestamp to ensure that the delay between the perceived scene state and the current physical world state is minimized (e.g., select the frame at the current time t, rather than at time t-1). (2) Data validity criterion: select frames with a depth information missing rate lower than a preset ratio (e.g., 5%) to ensure the quality of subsequent 3D reconstruction. (3) Field of view coverage criterion: if there are multiple camera perspectives, select the image whose field of view center is closest to the target operation area based on the approximate location involved in the user instruction.

[0072] A2, process the scene perception image to obtain object detection information;

[0073] The scene-aware images are processed to obtain object detection information, specifically using object detection algorithms or instance segmentation algorithms based on deep convolutional neural networks (CNNs). Specific processing methods (algorithms) may include, but are not limited to: (1) inputting the selected RGB images into a pre-trained YOLO series model or Faster R-CNN model; (2) the model outputs the two-dimensional bounding box coordinates, category labels, and confidence scores of all identified objects in the scene through feature extraction and bounding box regression; and (3) using Mask R-CNN or SAM to further extract pixel-level semantic segmentation masks of the objects. These output data collectively constitute the object detection information, providing structured visual semantic input for the subsequent VLA model to understand 'what objects are where'.

[0074] A3, generate scene semantic information from the selected scene-aware images and the object detection information;

[0075] A4. Input the user instructions and the scene semantic information into the pre-trained visual language action model to obtain the action draft.

[0076] Scene-aware images are used to identify objects, understand scene layouts, and interpret references in human commands (e.g., "that red cup").

[0077] Object detection information may include object detection boxes, semantic segmentation maps, or scene maps. Object detection information describes "what objects are where" and the spatial relationships between them.

[0078] Among them, the Visual Language Action Model (VLA model) can be RT-2 or a similar multimodal embodied intelligence model.

[0079] The action draft is a JSON-formatted list, where each element represents a high-level sub-action. This format is strictly defined.

[0080] An example of an action draft is as follows: Action ID: "step_01"; Action Name: "Approach"; Target Object ID: "Vase_01"; Parameters: { "Offset": [-0.1, 0, 0]}. Action ID: "step_02"; Action Name: "Grab"; Target Object ID: "Vase_01"; Parameters: {"Grab Strategy": "Cylindrical Power Grab"; "Default Force (Newtons)": 10.0. The output action draft clearly separates "what to do" (semantic planning) from "how to do it" (physical execution parameters), the latter being handled by the core validation module for subsequent optimization.

[0081] S30, The scene perception image set and the sensor information are processed according to the dynamic virtual twin method to obtain virtual twin state information;

[0082] After acquiring the scene-aware image set and the sensor information, the robot motion planning device processes the scene-aware image set and the sensor information according to the dynamic virtual twin method to obtain virtual twin state information. The dynamic virtual twin method is implemented through a Physical Attribute Regression Network (PPR-Net).

[0083] That is, after acquiring the scene perception image set and the sensor information, the robot motion planning device processes the scene perception image set and the sensor information according to the dynamic virtual twin method to obtain virtual twin state information.

[0084] As one implementation method, the scene-aware image set and the sensor information are processed according to the dynamic virtual twin method to obtain virtual twin state information, specifically including:

[0085] B1. Based on the scene-perceived image set and the sensor information, interactive dynamic information is obtained;

[0086] After acquiring a scene-aware image set and sensor information, the robot motion planning device obtains interactive dynamic information based on the scene-aware image set and sensor information. The interactive dynamic information may include target object dynamic information, manipulator dynamic information, environmental object state information, and edge type information.

[0087] The target's dynamic information includes its precise attitude and dynamic state. For example, the precise attitude is its position: [x, y, z], and its orientation / attitude: [qx, qy, qz, qw]. The dynamic state is its linear velocity: [0,0,0], and its angular velocity: [0,0,0] (because it is stationary).

[0088] The dynamic information of the robotic arm includes the state of the robotic gripper, including the precise position and posture of the robotic finger joints.

[0089] The state information of the robot's own nodes (such as knuckles) can be determined through forward kinematics and proprioception. The robot motion planning device reads the precise angles of each joint encoder through the robot's real-time controller. Then, using the robot's kinematic model (established using the Denavit-Hartenberg (DH) parametric method), the position and orientation of the end effector (gripper, knuckles) in the world coordinate system can be accurately calculated from the joint angles. Finally, by differentiating the position and orientation of two consecutive frames, the linear velocity and angular velocity can be calculated.

[0090] The robot motion planning device stores the DH parameter table for each link of the robot (including link length). Linkage torsion angle Linkage offset and joint angle The robot motion planning device's real-time controller reads the joint angles from the encoders of each joint. Then, the homogeneous transformation matrix between each adjacent link is calculated. And multiply all transformation matrices from the base to the end effector in sequence (i.e. This allows for the precise calculation of the pose matrix of the end effector in the world coordinate system. Furthermore, by analyzing the current moment... and the previous moment The pose matrix is ​​used to perform difference operations to calculate the linear velocity vector and angular velocity vector of the end effector. This method is a standard kinematics solution method in the field of robot kinematics.

[0091] Environmental object state information can include support surface state information and other object state information. Support surface (tabletop) state information includes the position, normal vector, and friction coefficient of the support surface.

[0092] The state information of object nodes in the environment (such as vases and tables) (environmental object state information) can be determined through 3D object detection, pose estimation, and tracking algorithms. First, objects are identified and segmented from the image. Then, combining the RGB and depth images, a pose estimation algorithm (such as a variant of PoseCNN or a point cloud registration-based method) is used to calculate the object's 6D pose (position + attitude) in the camera coordinate system, which is then transformed to the world coordinate system. Finally, to obtain velocity information, the system runs a tracking algorithm (such as a Kalman filter or the simpler SORT / DeepSORT algorithm). This algorithm associates the same object in consecutive frames and estimates its linear and angular velocities through smoothing and filtering.

[0093] The edge type information includes contact edge type and / or adjacent edge type. The steps to obtain the edge type information may include: obtaining an explicit 3D mesh model of the interactive object based on the scene perception image set and sensor information; determining whether two objects corresponding to the explicit 3D mesh model of the interactive object are in contact based on a collision detection algorithm; if the two objects corresponding to the explicit 3D mesh model of the interactive object are in contact, then setting the edge type between the two objects (nodes) to contact edge type; if the two objects corresponding to the explicit 3D mesh model of the interactive object are not in contact, then setting the edge type between the two objects (nodes) to adjacent edge type.

[0094] B2, using an object detection model to identify objects in the scene-perceived image set to obtain the target object image;

[0095] After acquiring a scene-aware image set, the robot motion planning device performs object recognition on the scene-aware image set using an object detection model to obtain a target object image; the target object image is the image of the target object; for example, when the target object is a vase, the target object image is the image of the vase.

[0096] That is, the target object is located from the RGB image using a standard object detection or segmentation model (such as YOLO, SAM).

[0097] B3. Based on the target object image and the scene perception image set, obtain physical attribute prediction information;

[0098] After obtaining the target object image, the robot motion planning device derives physical property prediction information based on the target object image and the scene perception image set. This physical property prediction information includes the target object's mass range, estimated friction coefficient, centroid coordinates, fragility score, and shape description. For example, when the target object is a vase, the physical property prediction information is: {Mass: 0.8 kg; Friction coefficient: 0.4; Centroid: [offset_x, offset_y, offset_z]; Fragility score: 0.95}. The physical property prediction model is a Property Estimation CNN (PE-CNN).

[0099] As one implementation method, obtaining physical property prediction information based on the target object image and the scene-aware image set includes: inputting the target object image into a physical property prediction model to obtain the target object's mass range, estimated friction coefficient, and fragility score; constructing a 3D object model based on the scene-aware image set to obtain a 3D network model; calculating the geometric center and shape parameters of the 3D network model to obtain centroid coordinate information and shape description information; and merging the target object's mass range, estimated friction coefficient, fragility score, centroid coordinate information, and shape description information to obtain the physical property prediction information. That is, after the 3D mesh model of the target object is generated, the system runs a standard computational geometry algorithm. This algorithm assumes uniform object density and then calculates the geometric center of the 3D shape. This calculated geometric center serves as the initial estimate of the centroid coordinate information. Once the 3D model (mesh or point cloud) of the object is determined, the system extracts various parameters used to describe its shape. The most commonly used shape description information is the dimensions of the axis-aligned bounding box, namely the length, width, and height of the object [size_x, size_y, size_z].

[0100] That is, the image patch of the object (i.e. the image of the target object) is input into a specially trained, small convolutional neural network for physical property estimation (PE-CNN) to obtain an attribute vector containing multiple fields, such as {mass range: [0.1, 0.3] kg, friction coefficient estimate: 0.6, brittleness score: 0.8}.

[0101] When training on the physical property prediction information, the CNN was trained offline on a large dataset containing images of objects of various materials and their corresponding real-world physical properties. This dataset could be generated manually or automatically in a simulated environment. The model learned implicit associations between visual appearance (such as texture, material, transparency, and deformation) and physical properties. For example, it learned that metallic objects are generally heavier, glass objects are more fragile, and rubber objects have greater friction.

[0102] Physical property prediction information can be obtained through a three-stage progressive physical property prediction method.

[0103] Phase 1: Category matching based on prior knowledge base;

[0104] 1. Objective: To quickly assign a reasonable initial physical property baseline based on the category of the target object in the scene.

[0105] 2. Core Components: Object Recognition Network and Physical Attribute Prior Database. The object recognition network is a pre-trained deep learning model capable of outputting object category labels, such as a fine-tuned ResNet, EfficientNet, or CLIP model. The physical attribute prior database is a structured, scalable database (e.g., JSON, XML, or SQL database) storing a large number of typical physical attributes for common object categories.

[0106] 3. Specific Implementation Steps

[0107] a. Input reception: Receive RGB image slices and / or 3D point cloud fragments of the target object provided by the upstream sensing module.

[0108] b. Category Recognition: Input the image into the object recognition network to obtain one or more category labels with the highest confidence, for example, {"Category": "Ceramic Mug", "Confidence": 0.98}.

[0109] c. Database query: Use the category label with the highest confidence as the primary key to query the prior database of physical attributes.

[0110] d. Data structure: Each entry in the database is a structured object containing the statistical distribution information of the physical parameters of objects in that category, including at least the mean and standard deviation.

[0111] Database entry example:

[0112] {"Category ID": "ceramic_mug_01";"Description": "Standard ceramic mug";"Attributes": {"Weight (kg)": ["Mean": 0.35, "Standard Deviation": 0.12],"Static Friction Coefficient": ["Mean": 0.7, "Standard Deviation": 0.15],"Center of Gravity Offset (m)": {"Mean": [0, 0, -0.01], "Standard Deviation": [0, 0, 0.005]},"Fragility Score": {"Mean": 0.9, "Standard Deviation": 0.05}}

[0113] e. Attribute Assignment: The retrieved mean is used as the initial physical attribute of the object and assigned to the corresponding digital model in the dynamic virtual twin. The standard deviation (std_dev) of the object is also recorded as an initial measure of parameter uncertainty in subsequent steps.

[0114] Phase Two: Fine-tuning of parameters based on visual appearance;

[0115] 1. Objective: To personalize the category averages provided in Phase 1 based on the individual visual characteristics of objects (such as size, shape, and texture) to obtain more accurate non-contact predictions.

[0116] 2. Core Component: Physical Attribute Regression Network (PPR-Net), which is a specially designed multi-head output deep learning network.

[0117] 3. Specific Implementation Steps (Specific steps are deep learning training)

[0118] a. Input Preparation: Prepare the input data for PPR-Net, including:

[0119] RGB image slices of the target object.

[0120] Optionally, a 3D mesh or point cloud of the object (for graph convolution or PointNet layers).

[0121] The category ID obtained in Phase 1 (serves as a conditional input to activate category-related knowledge).

[0122] b. Network inference: Feed the data into PPR-Net for forward propagation.

[0123] Network Structure: PPR-Net internally employs a CNN backbone (such as MobileNet) to extract 2D visual features, and optionally fuses a GCN or PointNet branch to process 3D information. Multiple parallel regression heads branch off at the network's ends.

[0124] Output definition: Each regression head specifically predicts a corrected value for a physical parameter, which can be a scaling factor or an offset.

[0125] Output example: {"Mass scaling factor": 1.3; "Friction offset": -0.05; "Brittle offset": 0.02}

[0126] c. Parameter Update: Apply the corrected values ​​output by PPR-Net to the baseline parameters of Phase 1.

[0127] Updated formula:

[0128] Corrected mass = Base mass × Mass scaling factor

[0129] Corrected friction coefficient = Base friction coefficient + Friction offset.

[0130] The updated parameters will replace the properties of objects in the dynamic virtual twin.

[0131] d. The Physical Attribute Regression Network (PPR-Net) is fully trained offline. Its training set contains a large amount of visual data of objects and their true physical parameters obtained through physical measurements. The loss function is designed to minimize the difference between the parameters after applying the network's predicted corrections and the true parameters.

[0132] B4. Package the interactive dynamic information and the physical attribute prediction information to obtain virtual twin state information.

[0133] After obtaining the interactive dynamic information and the physical attribute prediction information, the robot motion planning device packages the target object dynamic information, the manipulator dynamic information, and the physical attribute prediction information to obtain virtual twin state information (that is, to construct a graph structure data object to describe the physical state of the current scene). ).

[0134] When packaging the interactive dynamic information and the physical attribute prediction information, the graph structure data is divided into node features and edge features.

[0135] Among them, node feature vectors The correspondence, (That is: physical properties combined with dynamic states). Among them... (The “node dynamic state part” in the interactive dynamic information) corresponds to: (1) When the node is a target object, it is the target object dynamic information, and the specific data is the target object’s precise posture ( (1) and the target object's dynamic state (linear velocity, angular velocity). (2) When the node is a robot, it represents the dynamic information of the manipulator, specifically the precise position and orientation of the manipulator's finger joints. (3) When the node is an environmental object, it represents the state information of the environmental object, specifically the position of the supporting surface, normal vector, etc. (Physical property prediction information section) corresponds to: the physical property prediction information, specifically the inherent properties such as mass, friction coefficient, and size obtained in the previous "three-stage prediction". The feature vector set of all nodes. The node feature matrix V constitutes the virtual twin state information.

[0136] Among them, the edge feature vector This includes edge type information in the interactive dynamic information, specifically contact edge type or adjacent edge type; and the geometric relationship (such as distance) between the two objects corresponding to the explicit 3D mesh model in the original text. It also includes the feature vectors of all edges. The set is located in the edge feature matrix E, which constitutes the virtual twin state information.

[0137] Through this packaging rule, scattered sensor data and prediction data are organized into computer-readable, structured graph data, which can then be directly processed by the subsequent Physical Causal Prediction Network (PICP-Net).

[0138] Ultimately, the high-precision parameters identified through physical means will be used as the final values ​​to update the dynamic virtual twin.

[0139] Meanwhile, the system uses the identification result (i.e., [visual features] -> [real physical parameters]) as a high-quality training sample, which can be used for incremental learning or online fine-tuning of the Physical Attribute Regression Network (PPR-Net) in the future, so that the visual prediction capability of the entire system can continue to evolve.

[0140] S40, the action draft and the virtual twin state information are processed by a pre-trained physical causal prediction network model to determine the risk vector of the current iteration. If the risk vector of the current iteration does not meet the safety threshold table, the action draft for the next iteration is generated based on the risk vector of the current iteration, and the virtual twin state information for the next iteration is obtained based on the scene perception image set, the sensor information, and the risk vector of the current iteration. The risk vector for the next iteration is determined based on the action draft and the virtual twin state information for the next iteration, until the risk vector meets the safety threshold table. The action draft corresponding to the risk vector that meets the safety threshold table is used as the robot action instruction.

[0141] After obtaining the motion draft and virtual twin state information, the robot motion planning device processes the motion draft and virtual twin state information through a pre-trained physical causal prediction network model to determine the risk vector of the current iteration. If the risk vector of the current iteration does not meet the safety threshold table, the robot generates the motion draft for the next iteration based on the risk vector of the current iteration, and obtains the virtual twin state information for the next iteration based on the scene perception image set, the sensor information, and the risk vector of the current iteration. The robot then determines the risk vector for the next iteration based on the motion draft for the next iteration and the real-time virtual twin state information, until the risk vector meets the safety threshold table. Once the risk vector meets the safety threshold table, the motion draft corresponding to the risk vector that meets the safety threshold table is used as the robot motion instruction.

[0142] As one implementation method, specifically as follows: Figure 3 As shown, the risk vector for the current iteration is determined by processing the action draft and the virtual twin state information through a pre-trained physical causal prediction network model, which may include:

[0143] C1, obtain node attribute information, node state information, and edge type information based on the virtual twin state information;

[0144] After obtaining the virtual twin state information, the robot motion planning device derives node attribute information, node state information, and edge type information based on the virtual twin state information. The nodes include the robot gripper and the object to be interacted with (the target object). The node attribute information includes the target object's mass range, estimated friction coefficient, centroid coordinates, and fragility score. Node attribute information may include the target object's mass range, estimated friction coefficient, centroid coordinates, fragility score, and shape descriptor. It may also include the target object's friction coefficient estimate, high-precision estimated mass, centroid coordinates, and fragility score. Node attribute information may further include the target object's shape descriptor (e.g., bounding box dimensions) and static or quasi-static attributes. The node state information includes the target object's dynamic information and the robot gripper's dynamic information. Specifically, the node state information includes the target object's and robot gripper's position, orientation, linear velocity, angular velocity, and rotational rate components.

[0145] Node state information includes 13 dimensions: position, orientation, linear velocity, and angular velocity. Position is represented as a 3-dimensional vector [x, y, z]. Orientation is typically represented by quaternions [q]. x , q y , q z , q w The vector [] represents a 4-dimensional vector to avoid gimbal lock issues. The linear velocity is [v] x , v y , v z ], a 3D vector. Angular velocity is [ω x ,ω y ,ω z ], a 3-dimensional vector.

[0146] Node attribute information consists of the inherent physical properties of an object that typically do not change over time.

[0147] Physical properties include mass (a scalar), friction coefficient, center of mass position ([offset_x, offset_y, offset_z]), stiffness / fragility score, and shape descriptor. Mass is a scalar; the friction coefficient is a scalar (or divided into static and kinetic friction); the center of mass position is the offset relative to the object's geometric center. The stiffness / fragility score is a scalar used to guide stress prediction. Shape descriptors include, for example, the object's bounding box dimensions, or more complex shape embedding vectors.

[0148] C2, based on the node attribute information, the node state information, and the action draft, obtain the initial node hidden state and the global action vector;

[0149] After obtaining node attribute information, node state information, and motion draft, the robot motion planning device determines the initial hidden state of the nodes based on these information. and global action vector .

[0150] As one implementation method, specifically as follows: Figure 3 As shown, based on the node attribute information, the node state information, and the action draft, the initial node hidden state and global action vector are obtained, specifically including:

[0151] D1, Input the node attribute information into the node attribute encoder for encoding to obtain the initial node attribute latent vector;

[0152] After obtaining the node attribute information, the robot motion planning device inputs the node attribute information into the node attribute encoder for encoding to obtain the initial node attribute latent vector.

[0153] D2, input the node state information into the node state encoder for encoding to obtain the initial node state hidden vector;

[0154] After obtaining the node state information, the robot motion planning device inputs the node state information into the node state encoder for encoding to obtain the initial node state latent vector.

[0155] D3. Based on the initial node attribute hidden vector information and the initial node state hidden vector information, the initial node hidden state is obtained;

[0156] After obtaining the initial node attribute latent vector information and the initial node state latent vector information, the robot motion planning device obtains the initial node latent state based on the initial node attribute latent vector information and the initial node state latent vector information. ;

[0157] That is, after obtaining the node attribute information, node status information, and the action draft, the node attribute information, node status information, and the action draft are input into the encoder for encoding.

[0158] In other words, a multilayer perceptron (MLP) is used to map the original feature vector of each node to a higher-dimensional hidden state representation. .

[0159] This process occurs inside the encoder module, and the goal is to provide each node with... Generate an information-rich and well-structured initial hidden state vector.

[0160] First, perform layered input, for each node... Its original feature vector is logically divided into two parts: a vector describing the inherent properties of the object, such as... and a vector describing the current dynamic state of an object, such as .

[0161] Next, parallel encoding is performed: two different but structurally similar MLPs (Multilayer Perceptrons) are used to process the two parts of information in parallel. (1) Attribute encoder (MLP_attributes), which will encode the two parts of information in parallel. Encode into a meaningful latent attribute vector (2) State Encoder (MLP_state): Encodes the instantaneous motion state of an object into a latent state vector. .

[0162] Finally, the two latent vectors obtained from parallel processing are concatenated along the dimension to form the final, complete initial latent state of the node. :

[0163] For example, It is 64-dimensional. It's also 64-dimensional, so the final... It's a 128-dimensional vector. By concatenating the elements, all information can be retained in the simplest and most direct way. In this 128-dimensional vector, the first 64 dimensions always represent "attributes," and the last 64 dimensions always represent "states." This fixed structure allows the downstream processor (GNB) to more easily establish the rules governing how "attributes" influence changes in "states" during learning.

[0164] D4. Input the motion draft into the motion encoder for encoding to obtain the global motion vector.

[0165] After obtaining the motion draft, the robot motion planning device inputs the motion draft into the motion encoder for encoding to obtain the global motion vector. .

[0166] In other words, the motion draft is encoded by the motion encoder in the encoder to obtain the global motion vector.

[0167] For example, the action draft is {"Action Name": "Grab"; "Target Object ID": "cup_01"; "Parameters": {"Approach Speed": 0.15,"Grab Force": 10.0,"Lift Acceleration": 0.5,"Grab Vector": [0, 1, 0] / / Grab Direction}. During vectorization, all numeric parameters in the JSON object are first extracted and flattened into a single raw action vector, denoted as […]. .

[0168] like Encoding: The input is fed into the action encoder (MLP_action). The MLP_action encoder outputs a high-dimensional, continuous global latent action vector. (That is, the global action vector) (for example, a 128-dimensional vector). Global action vector The robot's current "intention" is entirely dependent on the following factors: action type, action target, action dynamics parameters, force / torque, velocity / acceleration, direction, action geometry parameters, and target position / orientation, and is indirectly related to user commands and scene semantic information. Specifically, the action type is defined as "is it grasping, pushing, or placing? Different action types are encoded into different vectors." The action target is defined as "on which object is the action applied?". The force / torque parameter is defined as "how much force is intended?". The velocity / acceleration parameter is defined as "at what speed is intended to be moved?". The direction parameter is defined as "from which direction is the force applied or the movement performed?". The target position / or orientation parameter is defined as "at what precise location does the gripper intend to move?".

[0169] A common problem with standard GNNs is that the input features of nodes are often flattened into a single vector. However, physically, an object's properties (such as mass) and state (such as velocity) are two different types of information. Furthermore, standard decoders typically only make an overall graph-level prediction (e.g., ...). Figure 3 (The categories shown).

[0170] Hierarchical encoding can be used, where the encoder processes the physical properties and dynamic states of a node separately. This can be done using two different encoders, which are then concatenated when the data enters the PICP-Net encoder. This helps the network better distinguish between constants and variables. Hierarchical encoding contributes to learning more robust physical representations.

[0171] C3. Based on the node attribute information, the node state information, and the edge type information, the initial hidden edge state is obtained;

[0172] After obtaining node attribute information and node state information, the robot motion planning device determines the initial hidden edge state based on the node attribute information, the node state information, and the edge type information. .

[0173] As one implementation method, such as Figure 3 As shown, the initial hidden edge state is obtained based on the node attribute information, the node state information, and the edge type information, and may include:

[0174] E1, based on the node attribute information and the node state information, obtain the original geometric feature information of the edge;

[0175] After obtaining node attribute information and node state information, the robot motion planning device obtains the original geometric feature information of the edge based on the node attribute information and node state information. That is, based on the node attribute information and node state information, it calculates the geometric features such as the relative position, distance, and whether they are in contact between two nodes, forming an original feature vector of the edge.

[0176] E2, input the original geometric feature information of the edge and the edge type information into the edge encoder to obtain the initial hidden edge state.

[0177] After obtaining the original geometric feature information of the edges, the robot motion planning device inputs the original geometric feature information and the edge type information into the edge encoder to obtain the initial hidden edge state. That is, the original geometric feature vector of the edge and the original feature vector of the edge type are input into a dedicated edge encoder, and its output is the initial hidden edge state. In other words, the edge type and other geometric features together form a complete original feature vector, which is then input into a dedicated edge encoder, and its output is the initial hidden edge state.

[0178] C4, input the initial node hidden state, the global action vector, the initial edge hidden state and the edge type information into a processor composed of N graph network blocks for iteration to obtain the final node hidden state;

[0179] After obtaining the initial hidden state of nodes, the global motion vector, and the initial hidden state of edges, the robot motion planning device inputs these information, along with the edge type information, into a processor composed of N stacked graph network blocks for iterative processing to obtain the final hidden state of nodes. N can be 10 or 20.

[0180] As one implementation method, such as Figure 3 and Figure 4As shown, the initial node hidden state, the global action vector, the initial edge hidden state, and the edge type information are input into a processor composed of N stacked graph network blocks for iteration to obtain the final node hidden state, specifically including:

[0181] F1, based on the initial node hidden state and the initial edge hidden state, obtain the current updated edge hidden state;

[0182] After obtaining the initial node hidden state and the initial edge hidden state, the robot motion planning device obtains the current updated edge hidden state based on the initial node hidden state and the initial edge hidden state.

[0183] That is, after obtaining the initial hidden state of the nodes and the initial hidden state of the edges, the robot motion planning device updates the edges. When updating the edges, for each edge... An update function It will depend on the hidden state of the two connected nodes. , and the current edge state This is used to calculate the new edge states. This simulates the calculation of the interaction forces between objects. The current hidden edge states are updated. The specific formula is as follows: ,in For the physical state (including position, velocity, and material properties) of a robot part (such as a finger) at the previous tiny time step; The physical state (including position, velocity, and material properties) of the object being touched (such as a cup) at the previous tiny time step. (Current edge state) is the contact state between the two at the previous moment (e.g., the current contact force magnitude, friction coefficient, and whether slippage occurred). is the first weight matrix, and represents the physical interaction laws learned by the network (e.g., Coulomb's law of friction). This is the activation function used to fit complex physical contact nonlinear characteristics; This updates the bias vector of the network for each edge. The hidden states of the edges are updated beforehand. The specific formula is based on the current state of the robot's finger and the object, as well as their previous contact, and how the contact force or connection between them will change in the next moment.

[0184] F2, based on the current updated edge hidden state, the global action vector, and the edge type information, obtain the hidden state of the current iteration node;

[0185] After obtaining the current updated edge hidden state and the global motion vector, the robot motion planning device obtains the hidden state of the current iteration node based on the current updated edge hidden state, the global motion vector, and the edge type information.

[0186] As one implementation method, such as Figure 3 and Figure 4 As shown, the hidden state of the current iteration node is obtained based on the current updated edge hidden state, the global action vector, and the edge type information, specifically including:

[0187] G1, detect whether the edge type information is a contact edge;

[0188] After obtaining the edge type information, the robot motion planning device detects whether the edge type information is a contact edge;

[0189] G2, if the edge type information is a contact edge, then the current updated edge hidden state is input into the contact interactive encoder for encoding to obtain the current contact edge hidden state;

[0190] When the robot motion planning device determines that the edge type corresponding to the current updated edge hidden state is a contact edge, it inputs the current updated edge hidden state into the contact interaction encoder for encoding to obtain the current contact edge hidden state.

[0191] G3, if the edge type information is not a contact edge, then the current updated edge hidden state is input into the neighbor interactive encoder for encoding to obtain the current neighbor edge hidden state;

[0192] When the robot motion planning device determines that the edge type corresponding to the current updated edge hidden state is not a contact edge, that is, a neighboring edge, it inputs the current updated edge hidden state into the neighboring interactive encoder for encoding to obtain the current neighboring edge hidden state.

[0193] G4, based on the current hidden state of the contact edge and / or the current hidden state of the neighboring edge, obtain node aggregation information;

[0194] After obtaining the current hidden state of the contact edge and / or the current hidden state of the neighboring edge, the robot motion planning device obtains node aggregation information based on the current hidden state of the contact edge and / or the current hidden state of the neighboring edge.

[0195] That is, for each node This aggregates information about all edges connected to it. This simulates the calculation of the net force acting on an object. Node aggregation information. . (Contact force / interaction term) directly corresponds to the current updated edge hidden state calculated in step F1. Specifically, at the physical simulation level, the updated hidden edge states Nodes were encoded and nodes The interaction relationship between them; for contact edges, the hidden state vector is physically represented as a node. Apply to node The contact force vector (containing the magnitude and direction of the force) on the edge; for "nearby edges," it is characterized as the potential field force or spatial concern. Used in the formula. Symbols are used to intuitively express the physical property of "force". Nodes aggregate information. Specifically, in the current iteration step In the middle, node This aggregated feature, combining the influence of all neighboring nodes, represents the net external force or total physical influence acting on the object (node) at the current moment. This aggregated information serves as input to the next step, the "node update" formula, used to calculate the object's acceleration and motion state at the next moment (i.e., simulation). (The process).

[0196] G5, based on the initial node hidden state, the global action vector, and the node aggregation information, obtain the current iteration node hidden state;

[0197] After obtaining the initial node hidden state, the global motion vector, and the node aggregation information, the robot motion planning device obtains the current iteration node hidden state based on the initial node hidden state, the global motion vector, and the node aggregation information.

[0198] Many standard GNN models (such as GCN and GraphSAGE) treat all edges equally, using the same message passing and update functions. This is not feasible in the physical world because "contact" and "proximity" are two completely different relationships with distinct physical laws. This is addressed by introducing conditional computation based on edge type. In the edge update step of the Graph Network Block (GNB), the network selects different computational paths based on the edge type ("contact edge" or "proximity edge"). A dedicated MLP sub-network is designed for each edge type. When processing a "contact edge," a more complex, parameter-rich ContactNet sub-network is invoked, which specializes in learning contact mechanics (such as friction and elasticity). When processing a "proximity edge," a simpler ProximityNet is invoked, which may only learn some weak interactions or not pass information at all.

[0199] By introducing edge-type-based conditional computation, the network selects different computation paths based on the edge type ("contact edge" or "nearby edge") during the edge update step of the Graph Network Block (GNB). This reduces the learning difficulty of the model, enabling it to more accurately capture the essence of different physical interactions and avoids fitting all cases with a fuzzy set of parameters.

[0200] As one implementation method, obtaining the hidden state of the current iteration node based on the initial node hidden state, the global action vector, and the node aggregation information may include:

[0201] ;

[0202] in, This represents the hidden state of the current iteration node; Aggregate node information for node i; This represents the hidden state of node v in the previous iteration. This is the global action vector; For update functions; This is the second weight matrix; Update the network bias vector for each node.

[0203] Standard GNNs are typically used to predict the natural evolution of a system, but they lack a definite, continuous external intervention input. However, in the task at hand, the robot's actions are a core external factor driving the changes throughout the entire physical system. This is achieved by encoding the global action vector... This is injected as a global condition into the node update step of each Graph Network Block (GNB) of the processor. Specifically, it refers to the standard node update function. Based on this, modify it to This means that at each discrete time step of the simulation, the network "always remembers" the actions the robot is performing and calculates the future state of the nodes accordingly.

[0204] By encoding the global action vector As a global condition, this is injected into the node update step of every graph network block (GNB) in the processor. This transforms the pre-trained Physical Causal Prediction Network model (PICP-Net) from a passive observer into a causal prediction engine capable of actively inferring "what will happen under... action". It enables conditional prediction and enhances task relevance, ensuring that the physical processes of model learning and prediction are directly related to the robot's current task intent, rather than aimlessly simulating all possibilities. This improves the effectiveness and practicality of the predictions.

[0205] In achieving conditional prediction, the model's predictions are closely linked to the input actions. We can ask, "What will happen if I push with a force of 5N?" and "What will happen if I push with a force of 10N?", and get two different answers.

[0206] F3, based on the current iteration node hidden state, the current update edge hidden state, and the global action vector, obtains the next iteration node hidden state, and after N iterations, takes the node hidden state of the Nth iteration as the final node hidden state.

[0207] After obtaining the hidden state of the current iteration node, the hidden state of the current update edge, and the global motion vector, the robot motion planning device obtains the hidden state of the next iteration node based on the hidden state of the current iteration node, the hidden state of the current update edge, and the global motion vector, and after N iterations, takes the hidden state of the node in the Nth iteration as the final hidden state of the node.

[0208] As one implementation method, obtaining the hidden state of the node for the next iteration based on the hidden state of the current iteration node, the hidden state of the current updated edge, and the global action vector may include:

[0209] ;

[0210] in, This represents the hidden state of the current iteration node; Aggregate node information for node i; This represents the hidden state of node v in the previous iteration. This is the global action vector; For update functions.

[0211] That is, the hidden state of the node in the Nth iteration. This is the hidden state of the final node.

[0212] By stacking N GNB graph network blocks, the network can simulate the physical evolution process over a relatively long period (e.g., 500ms), ultimately obtaining a final node hidden state containing future state information. .

[0213] C5 inputs all the hidden states of the final nodes into the pre-trained task decoder for decoding to obtain the risk vector of the current iteration.

[0214] After obtaining the hidden states of the final nodes, the robot motion planning device inputs all the hidden states of the final nodes into the pre-trained task decoder for decoding to obtain the risk vector of the current iteration.

[0215] As one implementation, the task decoder includes a success probability head, a physical quantity prediction head, a state prediction head, and a failure mode classification head; the risk vector includes: success probability, key physical quantities, final state, and one-hot encoding of the most likely failure mode; all the hidden states of the final nodes are input into the pre-trained task decoder for decoding to obtain the risk vector for the current iteration, specifically including:

[0216] Input all the hidden states of the final node into the success probability header, physical quantity prediction header, state prediction header and failure mode classification header respectively to obtain the one-hot encoding of success probability, key physical quantity, final state and most likely failure mode.

[0217] The task decoder is an MLP set that receives the final graph state (of all nodes) output by the processor. And decode it into the structured risk vector we need.

[0218] That is, after obtaining the hidden state of the final node, all of them are... By inputting the pre-trained success probability header, the pre-trained physical quantity prediction header, the pre-trained state prediction header, and the pre-trained failure mode classification header respectively, one-hot encodings of success probability, key physical quantities, final state, and most likely failure mode are obtained.

[0219] The success probability header consists of a graph-level pooling operation that aggregates information from all nodes, and then outputs a success probability between 0 and 1 using an MLP and a sigmoid function. .

[0220] The physical quantity prediction head is used to predict key physical quantities, such as the maximum stress value (predicted_max_stress), for specific nodes (such as target objects) by direct regression using MLP.

[0221] The state classification header is also for a specific node, using MLP to classify and predict its final state, such as whether slippage (is_slip) or tilting (is_stable) has occurred.

[0222] The failure mode classification head is another graph-level MLP that classifies the entire final graph state and outputs the one-hot vector of the most likely failure mode, such as [sliding state, stress failure state, collision state,...].

[0223] A common problem with standard GNNs is that the input features of nodes are often flattened into a single vector. However, physically, an object's properties (such as mass) and state (such as velocity) are two different types of information. Furthermore, standard decoders typically only make an overall graph-level prediction (e.g., ...). Figure 3 (The categories shown).

[0224] A multi-head task decoder can be used: the decoder is not a single output module, but a parallel collection of multiple dedicated "decoding heads". Each head is trained to solve a specific subtask (predicting stress, predicting slippage, predicting success rate, etc.).

[0225] By using a multi-head decoder to decompose a complex prediction problem into several simpler sub-problems, the training difficulty is reduced, and the output structured risk vector provides downstream decision controllers with a much richer and more interpretable basis for decision-making than a single "success / failure" signal.

[0226] When training the Physical Causal Prediction Network (PICP-Net), the training of the Physical Causal Prediction Network (PICP-Net) is carried out entirely in an offline virtual environment.

[0227] The dataset can be obtained sequentially through scene sampling, action sampling, running truth simulation, label extraction, and data storage.

[0228] Scene sampling involves procedurally generating millions of different interactive scenes within a dynamic virtual twin. Each scene contains randomly placed objects with different physical properties (randomized mass, friction, shape, fragility). Action sampling involves randomly sampling thousands of different action proposals within each scene. Action parameters (such as grasping force, velocity, and direction) are randomized within reasonable ranges. Truth-based simulation involves running a high-fidelity dynamic virtual twin for each scene-action pair, recording the complete physical evolution process.

[0229] Label extraction involves precisely calculating the truth labels corresponding to the risk vector from the simulation results. The truth labels corresponding to the risk vector include: success labels, physical quantity labels, state labels, and failure mode labels. Specifically, the success label checks whether the task was ultimately accomplished (e.g., whether the object was picked up stably). The physical quantity label records the physical quantities experienced by the target object during the simulation. The state label analyzes whether the object ultimately tipped over or slipped. The failure mode label determines the true cause of failure based on the simulation results.

[0230] Data storage involves storing each set (initial state, action proposal, baseline risk vector, etc.) as a data point in the training dataset.

[0231] After obtaining the training dataset, training can be performed by calculating the loss function (Loss Function & Training). A multi-task learning framework can be adopted, with the loss function... It is a weighted sum of the losses of each decoder head.

[0232] ;

[0233] Among them, the success probability head loss function A binary cross-entropy loss function is used. (Physical quantity prediction head loss function) State prediction head prediction function Mean Squared Error (MSE) loss is used. Failure Mode Classification Head Loss Function Cross-entropy loss is employed. These are hyperparameter weights used to balance the importance of different tasks.

[0234] The entire PICP-Net can be trained using a standard deep learning optimizer (such as Adam) until the loss converges on the validation set.

[0235] Based on the above technical solution, a concrete and implementable PICP-Net is defined. It effectively learns physical interactions through a GNN architecture, outputs rich risk information through a multi-task decoder, and is trained using large-scale offline simulation data.

[0236] As one implementation, the risk vector includes the success probability, key physical quantities, final state, and one-hot encoding of the most likely failure mode; the step of generating a draft action for the next iteration based on the risk vector of the current iteration if the risk vector of the current iteration does not conform to the safety threshold table may include:

[0237] H1 compares the success probability, key physical quantities, and final state with the safety threshold table to generate a decision signal.

[0238] H2, if the decision signal is a correction signal, then look up the correction mapping table according to the one-hot encoding of the most likely failure mode, and generate a correction instruction according to the risk vector;

[0239] H3, based on the correction instructions and the current action draft, generates the action draft for the next iteration.

[0240] After obtaining the risk vector, the robot motion planning device must intelligently formulate a strategy to correct motion parameters based on the nature of the risk when a risk is detected. It manages the entire "proposal-verification-correction" cycle, including initiation, iteration, and termination upon success or reaching the maximum number of attempts. The Safety Threshold Table can be dynamically adjusted based on the fragility of the interacting object.

[0241] For example, when the risk vector is {"Success Probability": 0.998; "Stability Score": 0.99; "Predicted Maximum Stress": 4.6; "Failure Mode Vector": [0.01, 0.02, 0.95, ...], / / corresponding to [Slippage, Stress, Collision, ...]; "Slippage Prediction": 0.01}.

[0242] The safety threshold table is as follows: {If the target object is fragile: stress threshold = 5.0; otherwise: stress threshold = 20.0; success probability threshold = 0.99; slippage probability threshold = 0.1}

[0243] After comparing the risk vector with the safety threshold table, a decision signal is generated. The step of comparing the risk vector with the safety threshold table is as follows: {If (the success probability of the risk vector > the success probability threshold) and (the predicted maximum stress of the risk vector < the stress threshold) and (the predicted slip value of the risk vector < the slip probability threshold), then the decision signal is "Approve"; otherwise, the decision signal is "Reject and Optimize". The correction signal is "Decision Signal = Reject and Optimize".

[0244] When the decision signal is a "reject and optimize" signal, that is, when the decision signal is a correction signal, a specific correction instruction is generated.

[0245] Specific correction instructions are generated. In the preferred embodiment, a "failure mode-correction strategy" mapping table based on expert knowledge is used.

[0246] Based on the failure mode vector (risk vector) and other risk indicators, the system queries a predefined, structured mapping table (as shown in Table 1). This table maps the most likely failure mode (the category with the highest probability in the vector) to one or more corrective actions.

[0247] Table 1

[0248]

[0249] When generating a correction instruction, the system first selects a priority 1 correction strategy corresponding to the detected failure mode. This strategy is instantiated as a specific correction instruction object, for example: {"Parameter to be modified": "Grasp force"; "Modification type": "Scaling value", / / can be 'Scaling value', 'Set value', 'Increase value'; "Scaling factor": 1.2}. If the priority 1 strategy still fails in subsequent iterations, the system can choose to try a priority 2 strategy, or combine multiple strategies.

[0250] As one implementation, the risk vector includes the success probability, key physical quantities, final state, and one-hot encoding of the most likely failure mode; the step of generating a draft action for the next iteration based on the risk vector of the current iteration if the risk vector of the current iteration does not conform to the safety threshold table may include:

[0251] H1 compares the success probability, key physical quantities, and final state with the safety threshold table to generate a decision signal.

[0252] H2, if the decision signal is a correction signal, then the optimal correction command is automatically calculated by using the gradient descent method by calculating the gradient of the input action parameters of the risk vector.

[0253] H3, based on the correction instructions and the current action draft, generates the action draft for the next iteration.

[0254] That is, in another embodiment of the present invention, since PICP-Net is differentiable, the optimal correction command can be automatically calculated by gradient descent using the gradient descent method by calculating the gradient of the risk vector with respect to the input action parameters, without the need to look up a table.

[0255] After processing the action draft and the virtual twin state information through a pre-trained physical causal prediction network model to determine the risk vector for the current iteration, the process may include:

[0256] If the risk vector of the current iteration conforms to the safety threshold table, then the draft action corresponding to the risk vector that conforms to the safety threshold table will be used as the robot action instruction.

[0257] The risk vector includes the success probability, key physical quantities, final state, and one-hot encoding of the most likely failure mode; if the risk vector of the current iteration conforms to the safety threshold table, the draft action is used as the robot's action instruction, including:

[0258] The success probability, key physical quantities, and final state are compared with the safety threshold table to generate decision signals.

[0259] If the decision signal is an approval signal, then the draft action corresponding to the risk vector that conforms to the safety threshold table will be used as the robot's action instruction.

[0260] As one implementation method, the virtual twin state information for the next iteration is obtained based on the scene-aware image set, the sensor information, and the risk vector of the current iteration, specifically including:

[0261] K1 obtains interactive dynamic information based on the scene-perceived image set and the sensor information;

[0262] After acquiring a scene-aware image set and sensor information, the robot motion planning device obtains interactive dynamic information based on the scene-aware image set and sensor information. The interactive dynamic information may include target object dynamic information, manipulator dynamic information, and environmental object state information.

[0263] The target's dynamic information includes its precise attitude and dynamic state. For example, the precise attitude is its position: [x, y, z], and its orientation / attitude: [qx, qy, qz, qw]. The dynamic state is its linear velocity: [0,0,0], and its angular velocity: [0,0,0] (because it is stationary).

[0264] The dynamic information of the robotic arm includes the state of the robotic gripper, including the precise position and posture of the robotic finger joints.

[0265] The state information of the robot's own nodes (such as knuckles) can be determined through forward kinematics and proprioception. The robot's motion planning device reads the precise angles of each joint encoder through the robot's real-time controller. Then, using the robot's kinematic model (a mathematical formula describing its geometry), the position and orientation of the end effector (gripper, knuckles) in the world coordinate system can be accurately calculated from the joint angles. Finally, by differentiating the position and orientation of two consecutive frames, the linear velocity and angular velocity can be calculated.

[0266] Environmental object state information can include support surface state information and other object state information. Support surface (tabletop) state information includes the position, normal vector, and friction coefficient of the support surface.

[0267] The state information of object nodes in the environment (such as vases and tables) (environmental object state information) can be determined through 3D object detection, pose estimation, and tracking algorithms. First, objects are identified and segmented from the image. Then, combining the RGB and depth images, a pose estimation algorithm (such as a variant of PoseCNN or a point cloud registration-based method) is used to calculate the object's 6D pose (position + attitude) in the camera coordinate system, which is then transformed to the world coordinate system. Finally, to obtain velocity information, the system runs a tracking algorithm (such as a Kalman filter or the simpler SORT / DeepSORT algorithm). This algorithm associates the same object in consecutive frames and estimates its linear and angular velocities through smoothing and filtering.

[0268] K2 uses an object detection model to identify objects in the scene-perceived image set to obtain the target object image;

[0269] After acquiring a scene-aware image set, the robot motion planning device performs object recognition on the scene-aware image set using an object detection model to obtain a target object image; the target object image is the image of the target object; for example, when the target object is a vase, the target object image is the image of the vase.

[0270] In other words, the target object is located from the RGB image using a standard object detection or segmentation model (such as the YOLO model or the SAM model).

[0271] K3 checks whether the risk vector in the current iteration satisfies the high-quality sensitivity condition;

[0272] After obtaining the risk vector for the current iteration, the robot motion planning device checks whether the risk vector for the current iteration meets the high-quality sensitivity condition.

[0273] K4, if the risk vector of the current iteration satisfies the high-quality sensitivity condition, perform a trial action on the target object and obtain the target object trial collection information; based on the target object trial collection information and the scene perception image set, obtain physical attribute prediction information; package the interaction dynamic information and the physical attribute prediction information to obtain the virtual twin state information for the next iteration;

[0274] After determining that the risk vector of the current iteration meets the high-quality sensitivity condition, the robot motion planning device performs a trial action on the target object and acquires the target object trial acquisition information; based on the target object trial acquisition information and the scene perception image set, it obtains physical attribute prediction information; and packages the interaction dynamic information and the physical attribute prediction information to obtain the virtual twin state information for the next iteration.

[0275] In other words, after the robot motion planning device determines that the success rate in the current iteration's risk vector is highly sensitive to the object's mass parameters, it triggers step three, which is to trigger the mass identification subroutine.

[0276] As one implementation method, a probing action is performed on the target object, and probing information of the target object is acquired; based on the probing information of the target object and the scene-aware image set, physical attribute prediction information is obtained; the interactive dynamic information and the physical attribute prediction information are packaged to obtain the virtual twin state information for the next iteration, specifically including:

[0277] L1 performs a probing action on the target object and acquires probing information about the target object;

[0278] After determining that the success rate in the current iteration's risk vector is highly sensitive to the object's mass parameters, the robot motion planning device performs a trial action on the target object and acquires trial data collection information of the target object.

[0279] As one implementation method, detecting whether the risk vector of the current iteration satisfies the high-quality sensitivity condition includes:

[0280] Obtain the mass range of the target object;

[0281] Based on the target object's mass range, the maximum and minimum estimated masses are obtained;

[0282] Replace the target object mass range in the next iteration's action draft with the maximum estimated mass and the minimum estimated mass, respectively, to obtain the first mass-sensitive action draft and the second mass-sensitive action draft.

[0283] The first quality-sensitive action draft and the virtual twin state information are processed by a pre-trained physical causal prediction network model to obtain the first quality-sensitive success rate.

[0284] The second quality-sensitive action draft and the virtual twin state information are processed by a pre-trained physical causal prediction network model to obtain the second quality-sensitive success rate.

[0285] The sensitivity score is obtained by subtracting the second quality sensitivity success rate from the first quality sensitivity success rate.

[0286] Detect whether the sensitivity score is greater than the sensitivity decision threshold;

[0287] If the sensitivity score is less than or equal to the sensitivity decision threshold, then the risk vector of the current iteration satisfies the high-quality sensitivity condition.

[0288] If the sensitivity score is greater than the sensitivity decision threshold, then the risk vector of the current iteration does not meet the high-quality sensitivity condition.

[0289] The reference steps C1 to C5 are as follows: to obtain the first quality-sensitive success rate and the second quality-sensitive success rate.

[0290] Obtain the current estimated mass of the target object and its uncertainty. This uncertainty can be expressed as the standard deviation.

[0291] The decision controller creates two temporary, hypothetical virtual twin state copies in memory. The higher-order state (maximum estimated mass): In this copy, all parameters are identical to the current state, except that the object's mass is set to the maximum estimated mass. The lower-order state (minimum estimated mass): In this copy, all parameters are identical to the current state, except that the object's mass is set to the minimum estimated mass.

[0292] The decision controller uses the current action draft to again invoke the pre-trained Physical Causal Prediction Network (PICP-Net) to predict the two perturbation states, obtaining the first quality-sensitive success rate. Second quality-sensitive success rate Sensitivity score =abs( - ).

[0293] As one implementation method, detecting whether the risk vector of the current iteration satisfies the high-quality sensitivity condition may further include:

[0294] The quality gradient is obtained by processing the action draft and the virtual twin state information through a pre-trained physical causal prediction network model.

[0295] The absolute value of the quality gradient is used to obtain the sensitivity score;

[0296] Detect whether the sensitivity score is greater than the sensitivity decision threshold;

[0297] If the sensitivity score is less than or equal to the sensitivity decision threshold, then the risk vector of the current iteration satisfies the high-quality sensitivity condition.

[0298] If the sensitivity score is greater than the sensitivity decision threshold, then the risk vector of the current iteration does not meet the high-quality sensitivity condition.

[0299] The decision controller invokes a pre-trained Physical Causal Prediction Network (PICP-Net), requesting predictions and, simultaneously, computation of the backpropagation gradient. PICP-Net returns the baseline success rate in a single forward + backpropagation pass. The gradient of success rate with respect to quality .

[0300] The sensitivity score is directly the quality gradient. The absolute value;

[0301] The decision controller will also assign this sensitivity score. Compare with a sensitivity decision threshold and combine with baseline success rate. Whether or not it is in the "danger zone" will determine whether to ultimately trigger a physical probe.

[0302] In other words, after determining that the success rate in the current iteration's risk vector is highly sensitive to the object's mass parameters, the robot performs one or more "probing actions" and acquires the minute displacement of the target object caused by these actions. The robot's motion planning device then obtains this probing information via a camera or sensor. This is a standardized, extremely subtle sequence of interactions. The probing action can be a gentle push, i.e., using a fingertip to gently push the object (target) with a known, minute force. The probing information is acquired by tracking the object's displacement and acceleration through vision (camera) and recording the contact force through a force sensor on the fingertip. Alternatively, a small, incremental force can be applied to the object's surface, and the minute deformation can be perceived through a force sensor and vision (or a fingertip position sensor), thereby estimating its stiffness or fragility.

[0303] L2, based on the target object probe information and the scene perception image set, obtain physical attribute prediction information;

[0304] After obtaining the target object exploration and collection information and the scene perception image set, the robot motion planning device obtains physical property prediction information based on the target object exploration and collection information and the scene perception image set; among which, the high-precision target object physical information includes high-precision target object predicted mass, high-precision target object friction coefficient, high-precision target object fragility score, high-precision centroid coordinate information and high-precision shape description information.

[0305] Based on the target object exploration and acquisition information and the scene perception image set, physical property prediction information is obtained, including: constructing a 3D model of the target object based on the scene perception image set; calculating the geometric center and shape parameters of the target object's 3D network model to obtain centroid coordinate information and shape description information; correcting the centroid coordinate information and shape description information based on the target object exploration and acquisition information to obtain high-precision centroid coordinate information and high-precision shape description information; obtaining accurate target object predicted mass, high-precision target object friction coefficient, and high-precision target object fragility score based on the target object exploration and acquisition information; merging the high-precision target object predicted mass, high-precision target object friction coefficient, high-precision target object fragility score, high-precision centroid coordinate information, and high-precision shape description information to obtain physical property prediction information.

[0306] The inertial mass of an object can be identified from the [force, displacement, acceleration] data.

[0307] A small, incremental force can be applied to the surface of an object, and the minute deformation of the object can be detected by force sensors and vision (or a fingertip position sensor), thereby estimating its stiffness or fragility.

[0308] Phase 3: Online parameter identification based on proactive interaction;

[0309] 1. Objective: When a task requires extremely high accuracy for a specific physical parameter, obtain a high-precision true value for that parameter by planning and executing minimally invasive physical interactions, and use it as the final calibration.

[0310] 2. The core components are a parameter identification controller and sensors; the parameter identification controller is a software module responsible for planning and executing the sequence of actions for the "physical experiment". The sensors are a six-axis force / torque sensor at the robot's end effector and an external RGB-D camera.

[0311] 3. Specific Implementation Steps

[0312] a. Triggering condition: Triggered by the [E] decision controller. For example, the mass identification subroutine is triggered when the predictions of the Physical Causal Prediction Network Model (PICP-Net) show that the success rate of the task is highly sensitive to the mass parameter of the object.

[0313] b. Motion planning and execution (taking the identification of "mass" and "static friction coefficient" as an example):

[0314] Step A: Identify the static friction coefficient

[0315] 1. Planning: The parameter recognition controller plans a "slow tilt" motion. If the object is on a movable plane (such as a tray), the robot grasps the plane; if the object is directly on a fixed plane, the robot attempts to push it with its fingertips. Taking tilting as an example, a trajectory is planned that tilts the plane at a constant low angular velocity ω.

[0316] 2. Execution and Perception: The robot performs a tilting motion. Simultaneously, the vision system tracks the object at a high frequency (>100Hz) and accurately detects the critical tilt angle of the plane when the object begins to slide. .

[0317] 3. Calculation: Based on the physical formulas The static friction coefficient with high precision was calculated.

[0318] Step B: Identify quality m

[0319] 1. Planning: Given... After (coefficient of kinetic friction) It can be approximated as equal to The parameter identification controller plans a "controlled push" motion. It plans a trajectory so that the robot's fingertip pushes the object horizontally with a constant, known force Fpush (achieved through closed-loop force control via a force sensor).

[0320] 2. Execution and Perception: The robot performs a pushing motion. Simultaneously, the vision system tracks the object's motion at high frequency and calculates its acceleration 'a' under the force.

[0321] 3. Calculation: According to Newton's second law ,Right now (where the normal force N = mg), solve the equation The mass m of the object can then be solved by inverse problem.

[0322] In existing technologies, these parameters are typically set manually offline for objects in a simulation environment. This patent proposes a complete, online, non-contact to contact, multimodal fusion system and method to determine these key physical properties of unknown objects encountered by a robot in real time. This greatly enhances the system's adaptability and versatility.

[0323] L4, the interactive dynamic information and the physical attribute prediction information are packaged together to obtain virtual twin state information.

[0324] After obtaining the interactive dynamic information and the physical attribute prediction information, the robot motion planning device packages the target object dynamic information, the manipulator dynamic information, and the physical attribute prediction information to obtain virtual twin state information.

[0325] The high-precision parameters identified through physical means will ultimately be used as the final values ​​to update the virtual twin.

[0326] Meanwhile, the system uses the identification result (i.e., [visual features] -> [real physical parameters]) as a high-quality training sample, which can be used for incremental learning or online fine-tuning of the Physical Attribute Regression Network (PPR-Net) in the future, so that the visual prediction capability of the entire system can continue to evolve.

[0327] K5, if the risk vector of the current iteration does not meet the high-quality sensitivity condition, obtain the physical attribute prediction information based on the target object image and the scene perception image set; package the interaction dynamic information and the physical attribute prediction information to obtain the virtual twin state information for the next iteration.

[0328] After determining that the risk vector of the current iteration does not meet the high-quality sensitivity condition, the robot motion planning device obtains physical attribute prediction information based on the target object image and the scene perception image set; the interaction dynamic information and the physical attribute prediction information are packaged together to obtain the virtual twin state information for the next iteration.

[0329] That is, after determining that the success rate in the risk vector of the current iteration is not highly sensitive to the mass parameter of the object, the robot motion planning device triggers step one or step two, executing steps B3 and B4; that is, after determining that the success rate in the risk vector of the current iteration is not highly sensitive to the mass parameter of the object, the robot motion planning device obtains physical attribute prediction information based on the target object image and the scene perception image set; the interactive dynamic information and the physical attribute prediction information are packaged to obtain the virtual twin state information for the next iteration.

[0330] When training on the physical property prediction information, the CNN was trained offline on a large dataset containing images of objects of various materials and their corresponding real-world physical properties. This dataset could be generated manually or automatically in a simulated environment. The model learned implicit associations between visual appearance (such as texture, material, transparency, and deformation) and physical properties. For example, it learned that metallic objects are generally heavier, glass objects are more fragile, and rubber objects have greater friction.

[0331] Physical property prediction information can be obtained through a three-stage progressive physical property prediction method.

[0332] Phase 1: Category matching based on prior knowledge base;

[0333] 1. Objective: To quickly assign a reasonable initial physical property baseline based on the category of the target object in the scene.

[0334] 2. Core Components: Object Recognition Network and Physical Attribute Prior Database. The object recognition network is a pre-trained deep learning model capable of outputting object category labels, such as a fine-tuned ResNet, EfficientNet, or CLIP model. The physical attribute prior database is a structured, scalable database (e.g., JSON, XML, or SQL database) storing a large number of typical physical attributes for common object categories.

[0335] 3. Specific Implementation Steps

[0336] a. Input reception: Receive RGB image slices and / or 3D point cloud fragments of the target object provided by the upstream sensing module.

[0337] b. Category Recognition: Input the image into the object recognition network to obtain one or more category labels with the highest confidence, for example, {"Category": "Ceramic Mug", "Confidence": 0.98}.

[0338] c. Database query: Use the category label with the highest confidence as the primary key to query the prior database of physical attributes.

[0339] d. Data structure: Each entry in the database is a structured object containing the statistical distribution information of the physical parameters of objects in that category, including at least the mean and standard deviation.

[0340] Database entry example:

[0341] {"Category ID": "ceramic_mug_01";"Description": "Standard ceramic mug";"Attributes": {"Weight (kg)": ["Mean": 0.35, "Standard Deviation": 0.12],"Static Friction Coefficient": ["Mean": 0.7, "Standard Deviation": 0.15],"Center of Gravity Offset (m)": {"Mean": [0, 0, -0.01], "Standard Deviation": [0, 0, 0.005]},"Fragility Score": {"Mean": 0.9, "Standard Deviation": 0.05}}

[0342] e. Attribute Assignment: The retrieved mean is used as the initial physical attribute of the object and assigned to the corresponding digital model in the virtual twin. The standard deviation (std_dev) of the object is also recorded as an initial measure of parameter uncertainty in subsequent steps.

[0343] Phase Two: Fine-tuning of parameters based on visual appearance;

[0344] 1. Objective: To personalize the category averages provided in Phase 1 based on the individual visual characteristics of objects (such as size, shape, and texture) to obtain more accurate non-contact predictions.

[0345] 2. Core Component: Physical Attribute Regression Network (PPR-Net), which is a specially designed multi-head output deep learning network.

[0346] 3. Specific Implementation Steps (Specific steps are deep learning training)

[0347] a. Input Preparation: Prepare the input data for the Physical Attribute Regression Network (PPR-Net), including:

[0348] RGB image slices of the target object.

[0349] Optionally, a 3D mesh or point cloud of the object (for graph convolution or PointNet layers).

[0350] The category ID obtained in Phase 1 (serves as a conditional input to activate category-related knowledge).

[0351] b. Network Inference: The data is fed into the Physical Attribute Regression Network (PPR-Net) for forward propagation.

[0352] Network Structure: The Physical Attribute Regression Network (PPR-Net) internally employs a CNN backbone (such as MobileNet) to extract 2D visual features, and optionally fuses a GCN or PointNet branch to process 3D information. Multiple parallel regression heads branch off at the network's ends.

[0353] Output definition: Each regression head specifically predicts a corrected value for a physical parameter, which can be a scaling factor or an offset.

[0354] Output example: {"Mass scaling factor": 1.3; "Friction offset": -0.05; "Brittle offset": 0.02}

[0355] c. Parameter Update: Apply the corrected values ​​output by the Physical Attribute Regression Network (PPR-Net) to the baseline parameters of Phase 1.

[0356] Updated formula:

[0357] Corrected mass = Reference mass × Mass scaling factor

[0358] Corrected friction coefficient = Base friction coefficient + Friction offset.

[0359] The updated parameters will replace the properties of objects in the virtual twin.

[0360] d. The Physical Attribute Regression Network (PPR-Net) is fully trained offline. Its training set contains a large amount of visual data of objects and their true physical parameters obtained through physical measurements. The loss function is designed to minimize the difference between the parameters after applying the network's predicted corrections and the true parameters.

[0361] The step of determining the risk vector for the next iteration based on the draft action for the next iteration and the virtual twin state information for the next iteration has a maximum number of iterations of 5 or 10.

[0362] When the risk assessment logic returns an annotation signal, the loop terminates immediately and returns the final optimized action proposal. When the current number of attempts reaches the maximum number of attempts, if no safe action is found, the loop terminates forcibly and reports a validation failure to the upper layer.

[0363] Figure 5 A flowchart illustrating the task completion process of the robot motion planning method. Specific embodiments of the robot motion planning method of this application are as follows:

[0364] Scenario setting: The user's instruction is "Please move the glass vase on the table to the empty space next to it." The robot has decomposed this task into three high-level action drafts using the VLA model (the VLA model is not within the scope of this patent discussion): [A: Approach the vase, B: Grab the vase, C: Move and place the vase].

[0365] The steps for handling and grabbing the vase are as follows.

[0366] First, a main function will be defined to execute the safety action plan. It will receive natural language instructions from the user as input and ultimately drive the robot to execute a safety-verified sequence of actions.

[0367] Step 1: Verify and optimize the action function startup - First attempt:

[0368] 1.1 Initialization Action Proposal

[0369] When the main process calls to validate and optimize the action function, the initial action draft passed in is generated by the VLA model. This draft is "semantically correct" but "physically ambiguous." It might be:

[0370] {"Action Name": "Grab",

[0371] "Target Object ID": "vase_01",

[0372] "Grasp Strategy": "Cylinder-type gripping", / / VLA (Visual Language Model / Visual Analysis Module) visually determines that the object is a cylinder;

[0373] Initial parameters: {

[0374] "Approach speed": 0.15, / / m / s, normal speed

[0375] "Grasp Force": 10.0, / / Newtons, the default force

[0376] "Increased acceleration": 0.5 / / m / s 2 Standard acceleration

[0377] This initial draft will be named the Current Action Proposal.

[0378] The current attempt count counter is set to 1.

[0379] 1.2 Obtaining High-Fidelity World State

[0380] The decision controller [E] sends a status query request to the virtual twin [C]. The virtual twin [C] (updated in real time by the perception module [A]) returns a highly detailed "world snapshot" including the following information:

[0381] Vase status:

[0382] Precise attitude: Position: [x,y,z], Direction, Attitude: [qx,qy,qz,qw].

[0383] Dynamic state: linear velocity: [0,0,0], angular velocity: [0,0,0] (because it is stationary).

[0384] Estimated physical properties: {Mass: 0.8kg, Coefficient of friction: 0.4, Center of mass: [offset_x, offset_y, offset_z], Brittleness score: 0.95}. The brittleness score is estimated by the system based on visual perception (transparency, texture) or previous slight contact, and it directly affects the subsequent stress threshold.

[0385] Robot hand state: including the precise position and orientation of each phalanx.

[0386] Support surface (tabletop) status: position, normal vector, and coefficient of friction.

[0387] 1.3 Using a physical causal prediction network model for prediction

[0388] The action draft and the virtual twin state information are packaged together and sent to the physical causal prediction network model. The core task of the physical causal prediction network model is to answer the question: "If a robotic gripper in a virtual twin state executes the action draft, what is most likely to happen in the next 500 milliseconds?"

[0389] The physical causal prediction network model performs an ultra-fast forward propagation (inference) through its deep neural network, and its output is not a simple "yes / no", but a structured risk vector.

[0390] The risk vector can be {

[0391] "Success probability": 0.35, / / Extremely low success rate

[0392] Stability rating: 0.98, / / Stability is not an issue; the vase won't tip over.

[0393] "Predicted maximum stress": 12.5, / / Predicted maximum stress value (unit: Pascal)

[0394] "Failure Mode": "Structural high-stress failure", / / Failure mode: Structural damage due to high stress

[0395] "Slipping prediction value": 0.02 / / Prediction will not slip}

[0396] 1.4 Decision and Analysis

[0397] The risk vector is then compared with a safety threshold table, for example, a safety threshold table that states "fragile object stress threshold = 5.0 Pascals (Pa)".

[0398] The controller detected that the predicted maximum stress (12.5) was much greater than the stress threshold (5.0). At the same time, the success rate was extremely low, and the failure mode clearly pointed to "high stress failure".

[0399] Conclusion: First attempt failed. Performing this action directly will cause the vase to break.

[0400] The second step involves entering a feedback and optimization loop – the second attempt (Attempt 2), as detailed below. Figure 6 As shown.

[0401] 2.1 Generate correction instructions

[0402] Then, reasoning is performed based on the failure mode:

[0403] Reasoning process: "The failure is due to excessive stress. Stress is mainly determined by the gripping force. Therefore, the most direct correction method is to significantly reduce the gripping force." Thus, it generates a clear correction instruction:

[0404] It can be described as:

[0405] {"Parameters to be modified": "Grabbing power",

[0406] "Modification Type": "Setting Value"

[0407] "New value": 3.0 / / Significantly reduce the strength from 10.0 Newtons to a more conservative value}

[0408] 2.2 Apply corrections and update action proposals

[0409] The controller updated the action proposal based on the correction instructions, generating version v2:

[0410] / / Current action proposal v2

[0411] {"Initial parameters": {

[0412] Approach speed: 0.15,

[0413] "Catching power": 3.0, / / <--- Fixed

[0414] "Increase acceleration": 0.5}

[0415] The current attempt counter has been increased to 2.

[0416] 2.3 Second Prediction

[0417] The process returns to 1.2, where the system retrieves the latest world state again (in case of external changes) and sends the new action proposal v2 to the physics causal prediction network model. This time, PICP-Net's prediction result might be as follows:

[0418] Success probability: 0.40, / / The success rate is still not high.

[0419] Stability Score: 0.75, / / Stability decreased

[0420] "Predicted maximum stress": 4.5, / / Stress value is safe

[0421] Failure Mode: "Low Friction Slippage", / / New failure mode: slippage due to insufficient friction.

[0422] "Slippage Prediction": 0.85 / / The prediction has a high risk of slippage.

[0423] 2.4 Second Decision and Analysis

[0424] Analysis: The controller observed that the stress problem was resolved (4.5 < 5.0). However, a new and equally fatal problem emerged: slippage.

[0425] Conclusion: The second attempt failed. Performing this action directly will cause the vase to slip from your hands.

[0426] The third step, further intelligent correction - the third attempt (Attempt 3), specifically as follows: Figure 6 As shown.

[0427] 3.1 Generate more complex correction instructions

[0428] The decision controller re-analyzed the reasons for the failure:

[0429] Reasoning: "The failure was due to slippage. Slippage occurs because the gripping force is relatively small compared to the object's weight and surface friction. Simply increasing the gripping force will lead back to the same mistake as the first failure (crushing). So, what methods can prevent slippage? The answer is: increase the contact area or reduce the acceleration during lifting. Therefore, the latter can be tried first, as it seems safer."

[0430] This generates a new correction instruction:

[0431] {"Parameters to be modified": "Increase acceleration",

[0432] "Modification type": "Scaling value",

[0433] "Scaling factor": 0.2 / / Reduces acceleration to 20% of its original value}

[0434] 3.2 Application Correction

[0435] A new motion suggestion, v3, has been introduced. Its gripping force is a safe value of 3.0N, and the lifting acceleration has been significantly reduced to 0.1m / s². The current number of attempts has been increased to 3.

[0436] 3.3 Third Prediction

[0437] The physical causal prediction network model makes predictions for v3 and returns:

[0438] Probability of success: 0.998

[0439] Stability score: 0.99

[0440] "Predicted maximum stress": 4.6,

[0441] Failure Mode: None

[0442] "Sliding prediction value": 0.01}

[0443] 3.4 Final Decision: Approved

[0444] Analysis: All indicators are safe. The success rate is extremely high and stable, the stress is within the threshold, and there is no risk of slippage.

[0445] Conclusion: Validation passed.

[0446] 3.5 Execution

[0447] The verification and optimization of the action function returns (TRUE, final action v3 (final action draft)). The main process receives this final optimized and safe action instruction and sends it to the underlying motion controller. The robot then safely grasps and lifts the glass vase in an extremely gentle and slow manner.

[0448] In summary, the process involves acquiring user commands, a set of scene-aware images, sensor information, and a safety threshold table; generating a draft action based on the user commands and the scene-aware image set; processing the scene-aware image set and the sensor information using a dynamic virtual twin method to obtain virtual twin state information; processing the draft action and the virtual twin state information using a pre-trained physical causal prediction network model to determine the risk vector for the current iteration; if the risk vector for the current iteration does not conform to the safety threshold table, generating a draft action for the next iteration based on the risk vector for the current iteration, and obtaining the virtual twin state information for the next iteration based on the scene-aware image set, the sensor information, and the risk vector for the current iteration; and determining the risk vector for the next iteration based on the draft action and the virtual twin state information for the next iteration, until the risk vector conforms to the safety threshold table, at which point the draft action corresponding to the risk vector conforming to the safety threshold table is used as the robot's action command. This achieves accurate prediction of physical consequences for locally changing, dynamically altered interactive scenarios without relying on a pre-established, perfect global physical model.

[0449] This invention deeply integrates physical causal prediction and virtual twin technology into the robot decision-making process, bringing multi-dimensional benefits and improving the safety, success rate, generalization ability and interaction efficiency of embodied intelligent systems in the real world.

[0450] For those consistent with the above, please refer to Figure 6 , Figure 6 This application provides a schematic diagram of the structure of a robot motion planning device according to an embodiment. Figure 6 As shown, the device includes:

[0451] The information acquisition module 601 is used to acquire user commands, scene perception image sets, sensor information, and safety threshold tables;

[0452] The action draft determination module 602 is used to generate an action draft based on the user instruction and the scene-aware image set;

[0453] The twin information determination module 603 is used to process the scene perception image set and the sensor information according to the dynamic virtual twin method to obtain virtual twin state information;

[0454] The action instruction determination module 604 is used to process the action draft and the virtual twin state information through a pre-trained physical causal prediction network model to determine the risk vector of the current iteration. If the risk vector of the current iteration does not meet the safety threshold table, the module generates the action draft for the next iteration based on the risk vector of the current iteration, and obtains the virtual twin state information for the next iteration based on the scene perception image set, the sensor information, and the risk vector of the current iteration. The module also determines the risk vector for the next iteration based on the action draft and the virtual twin state information of the next iteration, until the risk vector meets the safety threshold table. After that, the action draft corresponding to the risk vector that meets the safety threshold table is used as the robot action instruction.

[0455] like Figure 7 As shown, this application embodiment also provides a terminal device 2, which includes: at least one processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor. The processor 20 and the memory 21 are connected. When the processor 20 executes the computer program 22, it implements the steps in the robot motion planning method embodiment.

[0456] This application also provides a computer storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the robot motion planning methods described in the above method embodiments.

[0457] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program that causes a computer to perform some or all of the steps of any of the robot motion planning methods described in the above method embodiments.

[0458] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include at least: any entity or device capable of carrying computer program code to a device / terminal equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable storage media cannot be electrical carrier signals or telecommunication signals.

Claims

1. A method of robot motion planning, characterized by ,comprising: obtaining user instructions, scene perception image sets, sensor information, and a safety threshold table; generating an action draft according to the user instructions and the scene perception image sets; processing the scene perception image sets and the sensor information according to a dynamic virtual twin method to obtain virtual twin state information; processing the action draft and the virtual twin state information through a pre-trained physical causal prediction network model to determine a risk vector of the current iteration, and if the risk vector of the current iteration does not meet the safety threshold table, generating an action draft of the next iteration according to the risk vector of the current iteration, and obtaining virtual twin state information of the next iteration based on the scene perception image sets, the sensor information, and the risk vector of the current iteration; and determining a risk vector of the next iteration according to the action draft of the next iteration and the virtual twin state information of the next iteration, until the risk vector meets the safety threshold table, and taking the action draft corresponding to the risk vector that meets the safety threshold table as a robot action instruction; processing the action draft and the virtual twin state information through a pre-trained physical causal prediction network model to determine a risk vector of the current iteration, comprising: obtaining node attribute information, node state information, and edge type information according to the virtual twin state information; obtaining initial node hidden state and global action vector according to the node attribute information, the node state information, and the action draft; obtaining initial edge hidden state according to the node attribute information, the node state information, and the edge type information; inputting the initial node hidden state, the global action vector, the initial edge hidden state, and the edge type information into a processor stacked by N graph network blocks for iteration to obtain final node hidden state; inputting all final node hidden states into a pre-trained task decoder for decoding to obtain the risk vector of the current iteration.

2. The robotic action planning method of claim 1, wherein the processing of the scene perception image sets and the sensor information according to the dynamic virtual twin method to obtain virtual twin state information, comprising: obtaining interaction dynamic information according to the scene perception image sets and the sensor information; performing object recognition on the scene perception image sets through an object detection model to obtain target object images; obtaining physical property estimation information according to the target object images and the scene perception image sets; packing the interaction dynamic information and the physical property estimation information to obtain virtual twin state information.

3. The robotic action planning method of claim 1, wherein the obtaining of virtual twin state information of the next iteration based on the scene perception image sets, the sensor information, and the risk vector of the current iteration, comprising: obtaining interaction dynamic information according to the scene perception image sets and the sensor information; performing object recognition on the scene perception image sets through an object detection model to obtain target object images; detecting whether the risk vector of the current iteration meets the quality high sensitivity condition; If the risk vector of the current iteration meets the high-quality sensitive condition, a tentative action is performed on the target object, and tentative collection information of the target object is obtained; physical attribute estimation information is obtained according to the tentative collection information of the target object and the set of scene perception images; and the interaction dynamic information and the physical attribute estimation information are packaged to obtain virtual twin state information for the next iteration; If the risk vector of the current iteration does not meet the high-quality sensitive condition, physical attribute estimation information is obtained according to the target object image and the set of scene perception images; and the interaction dynamic information and the physical attribute estimation information are packaged to obtain virtual twin state information for the next iteration.

4. The robotic action planning method of claim 1, wherein The obtaining of the initial node hidden state and the global action vector according to the node attribute information, the node state information and the action draft comprises: encoding the node attribute information in a node attribute encoder to obtain an initial node attribute hidden vector; encoding the node state information in a node state encoder to obtain an initial node state hidden vector; obtaining an initial node hidden state according to the initial node attribute hidden vector information and the initial node state hidden vector information; encoding the action draft in an action encoder to obtain a global action vector.

5. The robotic action planning method of claim 1, wherein The iteration of the initial node hidden state, the global action vector, the initial edge hidden state and the edge type information in the processor stacked by N graph network blocks to obtain a final node hidden state comprises: obtaining a current updated edge hidden state according to the initial node hidden state and the initial edge hidden state; obtaining a current iteration node hidden state according to the current updated edge hidden state, the global action vector and the edge type information; obtaining a next iteration node hidden state according to the current iteration node hidden state, the current updated edge hidden state and the global action vector, and taking the node hidden state of the Nth iteration as the final node hidden state after N iterations.

6. The method of claim 5, wherein The obtaining of the current iteration node hidden state according to the current updated edge hidden state, the global action vector and the edge type information comprises: detecting whether the edge type information is a contact edge; if the edge type information is a contact edge, encoding the current updated edge hidden state in a contact interaction encoder to obtain a current contact edge hidden state; if the edge type information is not a contact edge, encoding the current updated edge hidden state in a proximity interaction encoder to obtain a current proximity edge hidden state; obtaining node aggregation information according to the current contact edge hidden state and / or the current proximity edge hidden state; obtaining a current iteration node hidden state according to the initial node hidden state, the global action vector and the node aggregation information.

7. A robot motion planning apparatus, characterized by The method comprises: an information acquisition module is configured to acquire a user instruction, a set of scene perception images, sensor information and a safety threshold table; an action draft determination module is configured to generate an action draft according to the user instruction and the set of scene perception images; The twin information determination module is configured to process the set of scene perception images and the sensor information according to a dynamic virtual twin method to obtain virtual twin state information. The action instruction determination module is configured to obtain node attribute information, node state information and edge type information according to the virtual twin state information; obtain an initial node hidden state and a global action vector according to the node attribute information, the node state information and the action draft; obtain an initial edge hidden state according to the node attribute information, the node state information and the edge type information; input the initial node hidden state, the global action vector, the initial edge hidden state and the edge type information into a processor stacked by N graph network blocks for iteration to obtain a final node hidden state; input all the final node hidden states into a pre-trained task decoder for decoding to obtain a risk vector of the current iteration, and if the risk vector of the current iteration does not meet a safety threshold table, generate an action draft for the next iteration according to the risk vector of the current iteration, and obtain virtual twin state information for the next iteration based on the set of scene perception images, the sensor information and the risk vector of the current iteration; and determine the risk vector of the next iteration according to the action draft for the next iteration and the virtual twin state information for the next iteration, until the risk vector meets the safety threshold table, and take the action draft corresponding to the risk vector meeting the safety threshold table as the robot action instruction.

8. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that The processor implements the robot action planning method according to any one of claims 1 to 6 when executing the computer program. 9.A computer readable storage medium, storing a computer program, characterized in that The computer program implements the robot action planning method according to any one of claims 1 to 6 when executed by the processor.

Images