Robot action planning method and device, terminal and medium

Abstract

The invention provides a robot action planning method and device, a terminal and a medium. The method comprises the steps that a user instruction, a scene perception image set, sensor information and a safety threshold table are acquired; generating an action draft according to the user instruction and the scene perception image set; performing virtual simulation on the scene perception image set and the sensor information according to a dynamic virtual twinning method to obtain virtual twinning state information; performing processing through the action draft and the virtual twinning state information, determining a risk vector of current iteration, generating an iterative action draft based on the risk vector of the current iteration and a safety threshold table, and obtaining iterative virtual twinning state information; and determining an iterative risk vector according to the iterative action draft and the iterative virtual twin state information, and obtaining a robot action instruction. On the premise of not depending on a pre-established and perfect global physical model, accurate physical consequence prediction is carried out on locally and dynamically changing interaction scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent manufacturing, and particularly relates to a robot action planning method and device, a terminal and a medium. BACKGROUND

[0002] Robot action planning and control is the core issue of robotics, and its development process reflects the evolution trend from precise modeling to data-driven and then to deep cognitive fusion. In the early stage, this field mainly relied on deterministic methods based on geometric and dynamic models. Researchers established precise mathematical models for robots and environments, and used methods such as Rapidly-exploring Random Tree (RRT) or optimization algorithms to plan collision-free and dynamically feasible motion trajectories. Such methods have achieved great success in structured and predictable environments (such as industrial assembly lines), but their limitations lie in the high precision requirements for models, and their adaptability and generalization ability are severely insufficient when facing unstructured and dynamic environments and interacting with unknown objects.

[0003] To overcome the rigidity of traditional methods, the research field gradually shifted to data-driven and machine learning. The rise of reinforcement learning (RL) and imitation learning (IL) enables robots to acquire complex skills through trial-and-error interactions with the environment or learning from human examples. These methods reduce the dependence on precise physical models and show potential in handling complex contact tasks such as grasping and assembly. However, learning-based methods often require massive interaction data, and the training process is lengthy and "black box", making it difficult to guarantee the safety and explainability of the strategy, especially in critical safety applications, where the cost of "trial and error" is unacceptable.

[0004] In recent years, with the breakthrough progress of large language models (LLM) and multi-modal large models (VLM), the field of robotics has ushered in a new paradigm revolution. These models have given robots new abilities in semantic understanding, common sense reasoning, and long-term task planning, enabling them to parse complex natural language instructions and generate high-level action strategies, and are considered a key to building a robot "brain". However, this has also brought new challenges: there is a "semantic-physical gap" between the abstract intentions generated by large models and the underlying physical execution of robots. The action planning generated by the model often lacks a detailed causal understanding of the physical world, such as the transmission of force, changes in friction, and the deformation and breaking limits of objects. Directly executing these plans can easily lead to task failure or even physical damage. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a robot action planning method, aiming to accurately predict the physical consequences of a local and dynamically changing interactive scene without relying on a pre-established perfect global physical model.

[0006] In a first aspect, the embodiments of the present application provide a robot action planning method, comprising: obtaining a user instruction, a scene perception image set, sensor information, and a safety threshold table; generating an action draft according to the user instruction and the scene perception image set; processing the scene perception image set and the sensor information according to a dynamic virtual twin method to obtain virtual twin state information; processing the action draft and the virtual twin state information through a pre-trained physical causal prediction network model to determine a risk vector of the current iteration, and if the risk vector of the current iteration does not meet the safety threshold table, generating an action draft of the next iteration according to the risk vector of the current iteration, and obtaining virtual twin state information of the next iteration based on the scene perception image set, the sensor information, and the risk vector of the current iteration; and determining a risk vector of the next iteration according to the action draft of the next iteration and the virtual twin state information of the next iteration, until the risk vector meets the safety threshold table, and taking the action draft corresponding to the risk vector meeting the safety threshold table as a robot action instruction.

[0007] Optionally, the processing of the scene perception image set and the sensor information according to the dynamic virtual twin method to obtain virtual twin state information comprises: obtaining interactive dynamic information according to the scene perception image set and the sensor information; performing object recognition on the scene perception image set through an object detection model to obtain a target object image; obtaining physical property estimation information according to the target object image and the scene perception image set; packing the interactive dynamic information and the physical property estimation information to obtain virtual twin state information.

[0008] Optionally, the obtaining of virtual twin state information of the next iteration based on the scene perception image set, the sensor information, and the risk vector of the current iteration comprises: obtaining interactive dynamic information according to the scene perception image set and the sensor information; performing object recognition on the scene perception image set through an object detection model to obtain a target object image; detecting whether the risk vector of the current iteration meets a high-quality sensitive condition; if the risk vector of the current iteration meets the high-quality sensitive condition, performing a tentative action on the target object and obtaining target object tentative collection information; inputting the target object image into a pre-trained physical property estimation model to obtain centroid coordinate information and shape description information; obtaining physical property estimation information according to the target object tentative collection information and the set of scene perception images; and packaging the interaction dynamic information and the physical estimation information to obtain virtual twin state information for the next iteration; if the risk vector of the current iteration does not meet the high-quality sensitive condition, obtaining physical property estimation information according to the target object image and the set of scene perception images; and packaging the interaction dynamic information and the physical estimation information to obtain virtual twin state information for the next iteration.

[0009] Optionally, the action draft and the virtual twin state information are processed by a pre-trained physical causal prediction network model to determine the risk vector of the current iteration, including: obtaining node attribute information, node state information and edge type information according to the virtual twin state information; obtaining initial node hidden state and global action vector according to the node attribute information, the node state information and the action draft; obtaining initial edge hidden state according to the node attribute information, the node state information and the edge type information; inputting the initial node hidden state, the global action vector, the initial edge hidden state and the edge type information into a processor stacked by N graph network blocks for iteration to obtain final node hidden state; inputting all the final node hidden states into a pre-trained task decoder for decoding to obtain the risk vector of the current iteration.

[0010] Optionally, the obtaining initial node hidden state and global action vector according to the node attribute information, the node state information and the action draft includes: inputting the node attribute information into a node attribute encoder for encoding to obtain initial node attribute hidden vector; inputting the node state information into a node state encoder for encoding to obtain initial node state hidden vector; obtaining initial node hidden state according to the initial node attribute hidden vector information and the initial node state hidden vector information; inputting the action draft into an action encoder for encoding to obtain global action vector.

[0011] Optionally, the obtaining initial edge hidden state according to the node attribute information, the node state information and the edge type information includes: According to the node attribute information and the node state information, edge original geometric feature information is obtained; The edge original geometric feature information is input into an edge encoder to obtain an initial edge hidden state.

[0012] Optionally, the initial node hidden state, the global action vector, the initial edge hidden state and the edge type information are iteratively input into a processor stacked by N graph network blocks to obtain a final node hidden state, including: According to the initial node hidden state and the initial edge hidden state, a current updated edge hidden state is obtained. According to the current updated edge hidden state, the global action vector and the edge type information, a current iteration node hidden state is obtained. According to the current iteration node hidden state, the current updated edge hidden state and the global action vector, a next iteration node hidden state is obtained, and after N iterations, the node hidden state of the Nth iteration is taken as the final node hidden state.

[0013] Optionally, according to the current updated edge hidden state, the global action vector and the edge type information, the current iteration node hidden state is obtained, including: Detecting whether the edge type information is a contact edge; If the edge type information is a contact edge, the current updated edge hidden state is input into a contact interaction encoder for encoding to obtain a current contact edge hidden state; If the edge type information is not contact edge information, the current updated edge hidden state is input into a proximity interaction encoder for encoding to obtain a current proximity edge hidden state; According to the current contact edge hidden state and / or the current proximity edge hidden state, node aggregation information is obtained. According to the initial node hidden state, the global action vector and the node aggregation information, a current iteration node hidden state is obtained.

[0014] In a second aspect, an embodiment of the present application provides a robot action planning device, including: An information acquisition module is configured to acquire a user instruction, a scene perception image set, sensor information and a safety threshold table; An action draft determination module is configured to generate an action draft according to the user instruction and the scene perception image set; A twin information determination module is configured to process the scene perception image set and the sensor information according to a dynamic virtual twin method to obtain virtual twin state information; The action instruction determination module is configured to determine a risk vector of a current iteration by processing the action draft and the virtual twin state information through a pre-trained physical causal prediction network model, generate an action draft of a next iteration according to the risk vector of the current iteration if the risk vector of the current iteration does not meet the safety threshold table, and obtain virtual twin state information of the next iteration based on the set of scene perception images, the sensor information, and the risk vector of the current iteration; and determine a risk vector of the next iteration according to the action draft of the next iteration and the virtual twin state information of the next iteration, until the risk vector meets the safety threshold table, and then take the action draft corresponding to the risk vector meeting the safety threshold table as the robot action instruction.

[0015] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the robot action planning method according to any one of the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executable by a processor to implement the robot action planning method according to any one of the first aspect.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, causes the terminal device to perform the robot action planning method according to any one of the first aspect.

[0018] In the embodiment of the present application, the user instruction, the set of scene perception images, the sensor information, and the safety threshold table are acquired; an action draft is generated according to the user instruction and the set of scene perception images; the set of scene perception images and the sensor information are processed according to a dynamic virtual twin method to obtain virtual twin state information; a risk vector of a current iteration is determined by processing the action draft and the virtual twin state information through a pre-trained physical causal prediction network model, an action draft of a next iteration is generated according to the risk vector of the current iteration if the risk vector of the current iteration does not meet the safety threshold table, and virtual twin state information of the next iteration is obtained based on the set of scene perception images, the sensor information, and the risk vector of the current iteration; and a risk vector of the next iteration is determined according to the action draft of the next iteration and the virtual twin state information of the next iteration, until the risk vector meets the safety threshold table, and then the action draft corresponding to the risk vector meeting the safety threshold table is taken as the robot action instruction. The embodiment realizes accurate physical consequence prediction for a local and dynamically changing interactive scene without relying on a pre-established perfect global physical model. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of a robot action planning method provided by an embodiment of the present application; Figure 2 is a system framework diagram of a robot action planning method provided by the present application; Figure 3 is a data processing diagram of a physical causal prediction network model of a robot action planning method provided by the present application; Figure 4 is a flowchart of a robot action planning method provided by the present application; Figure 5 is a core verification cycle flowchart of a robot action planning method provided by the present application; Figure 6 is a structural diagram of a robot action planning device provided by an embodiment of the present application; Figure 7 is a structural diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0020] Figure 1 A schematic flowchart of a robot action planning method provided by an embodiment of the present application is shown, which can be applied to the robot action planning device described above, or can be a method for a user or an operator to operate and judge on the robot action planning device. As shown in Figure 1 , the method can include: S10, obtaining a user instruction, a scene perception image set, sensor information, and a safety threshold table; In order to achieve accurate physical consequence prediction for a local and dynamically changing interactive scene without relying on a pre-established perfect global physical model, the robot action planning device obtains a user instruction, a scene perception image set, sensor information, and a safety threshold table. The flowchart of the robot action planning method is shown in Figure 2 .

[0021] The user instruction is a natural language instruction given by a user, for example, the user instruction is "hand me the vase", for example, the user instruction is "please move the glass vase on the table to the empty space next to it." The scene perception image set is a set of RGB-D images continuously captured by an RGB-D camera installed on the wrist or head of the robot from different perspectives; the RGB image (scene perception image set) is used for visual feature extraction, and the depth (Depth) image (scene perception image set) is used for three-dimensional geometric information acquisition. The scene perception image set can include a color image set and a depth image set. The sensor information is sensing information obtained by a force / tactile sensor located on the robot hand, which can provide contact information for the robot. The sensor information is sensing information obtained by a six-axis force / torque sensor arranged at the end of the robot.

[0022] The safety threshold table is pre-stored in the memory of the robot or downloaded from a cloud knowledge base. The safety threshold table is a mapping relationship data structure (such as a hash table or a JSON object) that defines the physical limit values allowed during interaction of different object categories or materials. Specifically, but not limited to: (1) the maximum allowable stress threshold, which is set to a lower threshold (such as 5N) for fragile objects (such as glass cups); and a higher threshold (such as 50N) for rigid objects (such as metal blocks). (2) The minimum stable probability threshold is set to 99%, i.e., the predicted object tipping or sliding probability should not be higher than 1%. (3) The maximum allowable slip probability threshold is set to 5%. In runtime, the decision controller queries the attributes (such as material) of the "target object ID" in the action draft, and then indexes the corresponding specific threshold parameters from the safety threshold table for subsequent risk vector comparison. Wherein, the safety threshold table can be {if the target object is fragile: 5.0 stress threshold = 5.0; otherwise: stress threshold = 20.0; success probability threshold = 0.99; slip probability threshold = 0.1}.

[0023] S20, generating an action draft according to the user instruction and the scene perception image set; The robot action planning device generates an action draft according to the user instruction and the scene perception image set after obtaining the user instruction and the scene perception image set.

[0024] As an implementation, generating an action draft according to the user instruction and the scene perception image set can include: A1, screening a scene perception image from the scene perception image set; A1, filtering a scene-aware image from a set of scene-aware images (containing consecutive time-series image frames), specific filtering criteria include: (1) time-effectiveness criterion, prefer to select the image frame with the latest timestamp to ensure the delay between the perceived scene state and the current physical world state is minimized (e.g. select the frame at current time t, not at time t-1). (2) data-validity criterion, filter out the frame with depth information missing rate lower than a pre-set proportion (e.g. 5%) to ensure the quality of subsequent 3D reconstruction. (3) field-of-view coverage criterion, if there are multiple camera perspectives, according to the approximate orientation involved in the user instruction, select the image whose field-of-view center is closest to the target operation area.

[0025] A2, processing the scene-aware image to obtain object detection information; Processing the scene-aware image to obtain object detection information, specifically using a target detection algorithm or instance segmentation algorithm based on deep convolutional neural network (CNN). Specific processing methods (algorithms) can include but are not limited to: (1) input the filtered RGB image into a pre-trained YOLO series model or Faster R-CNN model. (2) the model outputs the two-dimensional bounding box coordinates, class labels and confidence of all recognized objects in the scene through feature extraction and bounding box regression. (3) use Mask R-CNN or SAM to further extract pixel-level semantic segmentation mask of the object. These output data collectively constitute the object detection information, providing structured visual semantic input for the subsequent VLA model to understand 'what objects are where'.

[0026] A3, generating scene semantic information from the filtered scene-aware image and the object detection information; A4, inputting the user instruction and the scene semantic information into a pre-trained visual language action model to obtain an action draft.

[0027] Wherein, the scene-aware image is used to identify objects, understand scene layout and the referential relationship in human instructions (e.g. "that red cup").

[0028] Wherein, the object detection information may include target detection box, semantic segmentation map or scene graph. The object detection information describes "what objects are where" and the spatial relationship between them.

[0029] Wherein, the visual language action model (VLA model) can be RT-2 or similar multi-modal embodied intelligent model.

[0030] The action draft is a JSON-formatted list, where each element represents a high-level sub-action. The format is strictly defined.

[0031] An example of the action draft is as follows: "action ID": "step_01"; "action name": "approach"; "target object ID": "vase_01"; "parameters": { "offset": [-0.1, 0, 0]}. "action ID": "step_02"; "action name": "grasp"; "target object ID": "vase_01"; "parameters": { "grasp strategy": "cylinder power grasp"; "default force (Newton)": 10.0. The output action draft can clearly separate "what to do" (semantic planning) from "how to do it" (physical execution parameters), which will be verified by the core verification module for subsequent optimization.

[0032] S30, processing the scene perception image set and the sensor information according to a dynamic virtual twin method to obtain virtual twin state information; After obtaining the scene perception image set and the sensor information, the robot action planning device processes the scene perception image set and the sensor information according to a dynamic virtual twin method to obtain virtual twin state information. The dynamic virtual twin method is realized by a physical property regression network (PPR-Net).

[0033] That is, after obtaining the scene perception image set and the sensor information, the robot action planning device processes the scene perception image set and the sensor information according to a dynamic virtual twin method to obtain virtual twin state information.

[0034] As an implementation manner, processing the scene perception image set and the sensor information according to a dynamic virtual twin method to obtain virtual twin state information specifically includes: B1, obtaining interaction dynamic information according to the scene perception image set and the sensor information; After obtaining the scene perception image set and the sensor information, the robot action planning device obtains interaction dynamic information according to the scene perception image set and the sensor information. The interaction dynamic information can include target object dynamic information, manipulator dynamic information, environment object state information, and edge type information. Among them, the target object dynamic information includes target object accurate pose and target object dynamics state. For example, the target object accurate pose is position: [x, y, z], direction / pose: [qx, qy, qz, qw]. The target object dynamics state is linear velocity: [0, 0, 0], angular velocity: [0, 0, 0] (because it is in a static state).

[0035] Among them, the manipulator dynamic information includes manipulator claw state, including the accurate position and pose of the manipulator finger joint.

[0036] The state information of the nodes of the robot itself (e.g. the fingers) can be determined through forward kinematics and proprioception. The robot motion planning device reads the accurate angles of the joint encoders through the real-time controller of the robot. Then, using the kinematic model of the robot (established using the Denavit-Hartenberg (D-H) parameter method), the position and pose of the end effector (hand, finger) in the world coordinate system can be accurately calculated from the joint angles. Finally, by differentiating the position and pose of two consecutive frames, the linear and angular velocities can be calculated.

[0037] The robot motion planning device stores the D-H parameter table of each link of the robot (including the link length , the link twist angle , the link offset , and the joint angle ). After the real-time controller of the robot motion planning device reads the joint angles of the joint encoders , the homogeneous transformation matrix between each adjacent link is calculated , and all the transformation matrices from the base to the end effector are multiplied in turn (i.e. ), so as to accurately calculate the pose matrix of the end effector in the world coordinate system. Further, by differentiating the pose matrix at the current time and the pose matrix at the previous time , the linear and angular velocity vectors of the end effector are calculated. This method is a standard kinematic solution method in the field of robot kinematics.

[0038] The state information of the environment objects can include support surface state information and other object state information, and the support surface (table top) state information includes the position, normal vector and friction coefficient of the support surface.

[0039] The state information of the object nodes in the environment (e.g. vase, table) (environment object state information) can be determined through three-dimensional object detection, pose estimation and tracking algorithms. First, the object is identified and segmented from the image. Then, using an RGB image and a depth image, a pose estimation algorithm (such as a variant of PoseCNN or a point cloud registration-based method) is used to calculate the 6D pose (position + pose) of the object in the camera coordinate system, and then converted to the world coordinate system. Finally, in order to obtain the velocity information, the system runs a tracking algorithm (such as a Kalman Filter or a simpler SORT / DeepSORT algorithm). This algorithm associates the same object in consecutive frames and estimates its linear and angular velocities through smoothing and filtering.

[0040] The edge type information includes a contact edge type and / or a proximity edge type. The step of obtaining the edge type information can include: obtaining an explicit three-dimensional mesh model of the interactive object according to the set of scene perception images and sensor information; determining whether two objects corresponding to the explicit three-dimensional mesh model of the interactive object are in contact according to a collision detection algorithm; if the two objects corresponding to the explicit three-dimensional mesh model of the interactive object are in contact, setting the type of an edge between the two objects (nodes) as the contact edge type; and if the two objects corresponding to the explicit three-dimensional mesh model of the interactive object are not in contact, setting the type of the edge between the two objects (nodes) as the proximity edge type.

[0041] B2, performing object recognition on the set of scene perception images by an object detection model to obtain a target object image; After obtaining the set of scene perception images, the robot action planning device performs object recognition on the set of scene perception images by an object detection model to obtain a target object image. The target object image is an image of a target object. For example, when the target object is a vase, the target object image is an image of the vase.

[0042] That is, a standard object detection or segmentation model (such as YOLO, SAM) is used to locate the target object from an RGB image.

[0043] B3, obtaining physical property estimation information according to the target object image and the set of scene perception images; After obtaining the target object image, the robot action planning device obtains physical property estimation information according to the target object image and the set of scene perception images. The physical property estimation information includes a target object mass range, a target object friction coefficient estimate, a target object center of mass coordinate, a target object fragility score, and shape description information. For example, when the target object is a vase, the physical property estimation information is {mass: 0.8 kg; friction coefficient: 0.4; center of mass: [offset_x, offset_y, offset_z]; fragility score: 0.95}. The physical property estimation model is a physical property estimation convolutional neural network (Property Estimation CNN, PE-CNN).

[0044] As an implementation, the physical property estimation information is obtained according to the target object image and the set of scene perception images, including: inputting the target object image into a physical property estimation model to obtain a target object mass range, a target object friction coefficient estimation value and a target object fragility score; performing object three-dimensional construction on the target object according to the set of scene perception images to obtain a target object three-dimensional network model; performing geometric center and shape parameter calculation on the target object three-dimensional network model to obtain centroid coordinate information and shape description information; and merging the target object mass range, the target object friction coefficient estimation value, the target object fragility score, the centroid coordinate information and the shape description information to obtain the physical property estimation information. That is, when the three-dimensional mesh model of the target object is generated, the system will run a standard computational geometry algorithm. The algorithm will assume that the object density is uniform, and then calculate the geometric center of the three-dimensional shape. The calculated geometric center is used as the initial estimate of the centroid coordinate information. Once the three-dimensional model (mesh or point cloud) of the object is determined, the system will extract various parameters used to describe its shape. The most commonly used shape description information is the size of the axis-aligned bounding box, that is, the length, width and height [size_x, size_y, size_z] of the object.

[0045] That is, the image patch (that is, the target object image) of the object is input into a specially trained, small physical property estimation convolutional neural network (Property Estimation CNN, PE-CNN) to obtain an attribute vector containing multiple fields, such as {mass range: [0.1, 0.3] kg, friction coefficient estimation value: 0.6, fragility score: 0.8}.

[0046] When training the physical property estimation information, the CNN is trained in an offline state on a large data set containing images of objects of various materials and their corresponding real physical properties. This data set can be measured manually or automatically generated in a simulated environment. The model learns the implicit relationship from visual appearance (such as texture, material, transparency, deformation) to physical properties. For example, it can learn that metal objects are usually heavier, glass objects are more fragile, and rubber objects have greater friction.

[0047] The physical property estimation information can be obtained by a three-stage progressive physical property estimation method.

[0048] Stage 1: category matching based on prior knowledge base; 1. Objective: quickly assign a reasonable, category-based initial physical property baseline to the target object in the scene.

[0049] 2. Core components: object recognition network and physical property prior database. The object recognition network is a pre-trained deep learning model that can output object class labels, such as a fine-tuned ResNet, EfficientNet, or CLIP model. The physical property prior database is a structured and extensible database (e.g., JSON, XML, or SQL database) that stores typical physical properties for a large number of common object classes.

[0050] 3. Specific implementation steps a. Input reception: receive RGB image slices and / or three-dimensional point cloud segments of the target object provided by the upstream perception module.

[0051] b. Class recognition: input the image into the object recognition network to obtain one or more class labels with the highest confidence, such as {"class": "ceramic mug", "confidence": 0.98}.

[0052] c. Database query: use the class label with the highest confidence as the primary key to query the physical property prior database.

[0053] d. Data structure: each entry in the database is a structured object that contains statistical distribution information for each physical parameter of the object, including at least the mean and standard deviation.

[0054] Database entry example: {"class ID": "ceramic_mug_01"; "description": "standard ceramic mug"; "properties": {"mass (kg)": ["mean": 0.35, "std_dev": 0.12], "static friction coefficient": ["mean": 0.7, "std_dev": 0.15], "center of mass offset (m)": {"mean": [0, 0, -0.01], "std_dev": [0, 0, 0.005]}, "brittleness score": {"mean": 0.9, "std_dev": 0.05}} e. Property assignment: assign the mean value obtained from the query as the initial physical property of the object to the corresponding digital model in the dynamic virtual twin. The standard deviation (std_dev) of the object is also recorded as an initial measure of parameter uncertainty for subsequent steps.

[0055] Phase two: parameter fine-tuning based on visual appearance; 1. Objective: based on the individual visual features of the object (such as size, shape, texture), individualize the class average provided in phase one to obtain more accurate non-contact estimates.

[0056] 2. Core component: Physical Property Regression Network (PPR-Net), which is a specially designed deep learning network with multiple output heads.

[0057] 3. Specific implementation steps (specific steps are deep learning training) a. Input preparation: Prepare the input data for PPR-Net, including: RGB image slices of the target object.

[0058] Optionally, three-dimensional mesh or point cloud of the object (for graph convolution or PointNet layers).

[0059] Category ID obtained in stage one (as a conditional input to activate class-related knowledge).

[0060] b. Network inference: Send the data into PPR-Net for forward propagation.

[0061] Network structure: PPR-Net can use a CNN backbone (such as MobileNet) to extract two-dimensional visual features, and optionally fuse a GCN or PointNet branch to process three-dimensional information. The end of the network branches into multiple parallel regression heads.

[0062] Output definition: Each regression head is specifically designed to predict a correction value for a physical parameter, which can be a scale factor or an offset.

[0063] Output example: {"mass scale factor": 1.3; "friction offset": -0.05; "brittleness offset": 0.02} c. Parameter update: Apply the correction values output by PPR-Net to the baseline parameters in stage one.

[0064] Update formula: Corrected mass = baseline mass × mass scale factor Corrected friction coefficient = baseline friction coefficient + friction offset.

[0065] The updated parameters will replace the properties of the object in the dynamic virtual twin.

[0066] d. The Physical Property Regression Network (PPR-Net) is fully trained in the offline stage. Its training set contains a large amount of visual data of objects and their real physical parameters obtained through physical measurement. The loss function is designed to minimize the difference between the parameters after applying the correction values predicted by the network and the real parameters.

[0067] B4, packing the interaction dynamic information and the physical estimation information to obtain virtual twin state information.

[0068] The robot action planning device packs the target object dynamic information, the robot dynamic information, and the physical estimation information to obtain virtual twin state information (i.e., constructing a graph structure data object for describing the physical state of the current scene ).

[0069] When packing the interaction dynamic information and the physical estimation information, the graph structure data is divided into node features and edge features.

[0070] The node feature vector corresponds to the interaction dynamic information and the physical estimation information. (i.e., the physical attribute spliced dynamic state). Wherein, The (interaction dynamic information "node dynamic state part") corresponds to: (1) when the node is a target object, the target object dynamic information, and the specific data is the target object accurate pose ) and the target object dynamics state (linear velocity, angular velocity). (2) When the node is a robot, the robot dynamic information, and the specific data is the accurate position and pose of the robot finger joint. (3) When the node is an environmental object, the environmental object state information, and the specific data is the position, normal vector, etc. of the support surface. Wherein, The (physical estimation information part) corresponds to the physical estimation information, and the specific data is the mass, friction coefficient, size, etc. inherent attribute obtained in the previous "three-stage estimation". The node feature vector set of all nodes constitutes the node feature matrix V of the virtual twin state information.

[0071] The edge feature vector is the edge type information in the interaction dynamic information, and the specific data is the contact edge type or the adjacent edge type; and the geometric relationship (such as distance) between the two objects corresponding to the original explicit three-dimensional mesh model. The edge feature vector set of all edges constitutes the edge feature matrix E of the virtual twin state information.

[0072] Through this packing rule, the scattered sensor data and estimation data are organized into computer-readable, structured graph data, which can be directly processed by the subsequent physical causal prediction network (PICP-Net).

[0073] Finally, the high-precision parameters identified through physics will be updated to the dynamic virtual twin as the final value.

[0074] Meanwhile, the system takes the result of this recognition (i.e., [visual feature] -> [real physical parameter]) as a high-quality training sample, which can be used for future incremental learning or online fine-tuning of the physical property regression network (PPR-Net), so that the visual estimation ability of the entire system continues to evolve.

[0075] S40, determining a risk vector of the current iteration by processing the action draft and the virtual twin state information through the pre-trained physical causal prediction network model, if the risk vector of the current iteration does not meet the safety threshold table, generating an action draft of the next iteration according to the risk vector of the current iteration, and obtaining virtual twin state information of the next iteration based on the set of scene perception images, the sensor information and the risk vector of the current iteration; and determining the risk vector of the next iteration according to the action draft of the next iteration and the virtual twin state information of the next iteration, until the risk vector meets the safety threshold table, and taking the action draft corresponding to the risk vector meeting the safety threshold table as the robot action instruction.

[0076] After the robot action planning device obtains the action draft and the virtual twin state information, it determines a risk vector of the current iteration by processing the action draft and the virtual twin state information through the pre-trained physical causal prediction network model, if the risk vector of the current iteration does not meet the safety threshold table, generates an action draft of the next iteration according to the risk vector of the current iteration, and obtains virtual twin state information of the next iteration based on the set of scene perception images, the sensor information and the risk vector of the current iteration; and determines the risk vector of the next iteration according to the action draft of the next iteration and the real-time virtual twin state information, until the risk vector meets the safety threshold table, and takes the action draft corresponding to the risk vector meeting the safety threshold table as the robot action instruction.

[0077] As an implementation manner, as shown in Figure 3 determining a risk vector of the current iteration by processing the action draft and the virtual twin state information through the pre-trained physical causal prediction network model can include: C1, obtaining node attribute information, node state information and edge type information according to the virtual twin state information; The robot action planning device obtains node attribute information, node state information and edge type information according to the virtual twin state information after obtaining the virtual twin state information; wherein the nodes include: a robot gripper and an object to be interacted (target object); the node attribute information includes a target object mass range, a target object friction coefficient estimate value, a target object centroid coordinate and a target object fragility score. The node attribute information can include the target object mass range, the target object friction coefficient estimate value, the target object centroid coordinate, the target object fragility score and the shape descriptor. The friction coefficient estimate value of the target object, the estimated mass of the high-precision target object, the centroid coordinate of the high-precision target object and the fragility score of the high-precision target object. The node attribute information can also include the shape descriptor (such as the bounding box size) of the target object, the static or quasi-static attribute of the target object; the node state information includes the dynamic information of the target object and the dynamic information of the robot. The node state information specifically includes the position, pose, linear velocity, angular velocity and rotation rate component of the target object and the robot.

[0078] The node state information includes 13-dimensional states such as position (Position), pose / orientation (Orientation), linear velocity (Linear Velocity) and angular velocity (Angular Velocity). Among them, the position is [x, y, z], a 3-dimensional vector. The pose / orientation is usually represented by a quaternion [q x , q y , q z , q w ], a 4-dimensional vector, to avoid gimbal lock. The linear velocity is [v x , v y , v z ], a 3-dimensional vector. The angular velocity is [ω x , ω y , ω z ], a 3-dimensional vector.

[0079] The node attribute information is the physical property of the object that is inherent and usually does not change with time.

[0080] The physical attribute information includes mass (Mass) (scalar), friction coefficient (Friction Coefficient), center of mass position (Center of Mass): [offset_x, offset_y, offset_z], stiffness / fragility score and shape descriptor. Among them, the mass is a scalar; the friction coefficient is a scalar (or divided into static friction and dynamic friction); the center of mass position is the position offset relative to the geometric center of the object. The stiffness / fragility score is a scalar, which is used to guide stress prediction. The shape descriptor, such as the bounding box size of the object, or a more complex shape embedding vector.

[0081] C2, obtaining an initial node hidden state and a global action vector according to the node attribute information, the node state information, and the action draft; The robot action planning device obtains an initial node hidden state and a global action vector according to the node attribute information, the node state information, and the action draft after obtaining the node attribute information, the node state information, and the action draft. .

[0082] As an implementation, as shown in particular Figure 3 obtaining an initial node hidden state and a global action vector according to the node attribute information, the node state information, and the action draft, specifically includes: D1, inputting the node attribute information into a node attribute encoder for encoding to obtain an initial node attribute hidden vector; The robot action planning device inputs the node attribute information into a node attribute encoder for encoding to obtain an initial node attribute hidden vector after obtaining the node attribute information. D2, inputting the node state information into a node state encoder for encoding to obtain an initial node state hidden vector; The robot action planning device inputs the node state information into a node state encoder for encoding to obtain an initial node state hidden vector after obtaining the node state information. D3, obtaining an initial node hidden state according to the initial node attribute hidden vector information and the initial node state hidden vector information; The robot action planning device obtains an initial node hidden state according to the initial node attribute hidden vector information and the initial node state hidden vector information after obtaining the initial node attribute hidden vector information and the initial node state hidden vector information. ; That is, after obtaining the node attribute information, the node state information, and the action draft, inputting the node attribute information, the node state information, and the action draft into an encoder for encoding; That is, mapping the original feature vector of each node to a higher-dimensional hidden state representation through a multi-layer perception (MLP) .

[0083] This process occurs inside the encoder module, and the goal is to generate an initial hidden state vector for each node which is informative and structurally clear.

[0084] First, hierarchical input is performed, and for each node ​, its original feature vector is logically divided into two parts, the vector describing the inherent properties of the object such as and the vector describing the current dynamic state of the object such as .

[0085] Again, parallel encoding is performed: two different but structurally similar MLPs (Multi-Layer Perceptron) are used to process the two parts of information in parallel. (1) Attribute encoder (MLP_attributes), which encodes into a meaningful attribute hidden vector ; (2) State encoder (MLP_state): which encodes the instantaneous motion state of the object into a state hidden vector .

[0086] Finally, splicing is performed: the two hidden vectors obtained by parallel processing are spliced together along the dimensions to form the final, complete node initial hidden state : For example, is 64-dimensional, is also 64-dimensional, then the final is a 128-dimensional vector. By splicing, all information can be preserved in the simplest and most direct way; in this 128-dimensional vector, the first 64 dimensions always represent "attributes" and the last 64 dimensions always represent "state". This fixed structure makes it easier for downstream processors (GNB) to learn the rules of how "attributes" affect "state" changes.

[0087] D4, inputting the action draft into the action encoder to encode and obtain a global action vector.

[0088] The robot action planning device inputs the action draft into the action encoder to encode and obtain a global action vector .

[0089] That is, the action draft is encoded by the action encoder in the encoder to obtain a global action vector.

[0090] For example, the action draft is {"action name": "grab"; "target object ID": "cup_01"; "parameters": {"approach speed": 0.15, "grab force": 10.0, "lift acceleration": 0.5, "grab vector": [0, 1, 0] / / direction of grabbing}}. During vectorization processing, first extract all numerical parameters in the JSON object and flatten them into a raw action vector, denoted as .

[0091] For example, Encoding: The input is fed into an action encoder (MLP_action). The MLP_action encoder outputs a high-dimensional, continuous global action latent vector (i.e., the global action vector) (e.g., a 128-dimensional vector). The global action vector is completely and only dependent on the robot’s current “intention”, which is directly related to the following factors: action type, action target, action dynamics parameters, force / torque, velocity / acceleration, direction, action geometric parameters, and target position / posture, and indirectly related to user instructions and scene semantic information. Among them, the action type is “is it grasping, pushing, or placing? Different action types will be encoded into different vectors”. The action target is “on which object is the action applied?”. The force / torque is “how much force is intended to be used?”. The velocity / acceleration is “how fast is the movement intended to be?”. The direction is “from which direction is the force applied or movement performed?”. The target position / posture is “to which precise position is the hand intended to move?”.

[0092] Problems that standard GNNs are prone to: the input features of nodes are usually flattened into a single vector. Physically, the properties (such as mass) and states (such as velocity) of objects are two different types of information. At the same time, the standard decoder usually only makes a global graph-level prediction (such as Figure 3 classification shown in FIG. 1).

[0093] Hierarchical encoding can be performed, in which the physical properties and dynamic states of nodes are processed separately by the encoder, which can be encoded by two different encoders and then spliced when entering the encoder of the PICP-Net. This helps the network to better distinguish which are constants and which are variables. Hierarchical encoding helps to learn more robust physical representations.

[0094] C3, obtaining an initial edge latent state according to the node attribute information, the node state information, and the edge type information; The robot action planning device obtains an initial edge latent state according to the node attribute information, the node state information, and the edge type information after obtaining the node attribute information and the node state information .

[0095] As an implementation manner, as shown in Figure 3 obtaining an initial edge latent state according to the node attribute information, the node state information, and the edge type information can include: E1, obtaining edge original geometric feature information according to the node attribute information and the node state information; The robot action planning device obtains the edge original geometric feature information according to the node attribute information and the node state information after obtaining the node attribute information and the node state information. That is, the relative position, distance, and whether to contact between two nodes are calculated as geometric features according to the node attribute information and the node state information to form an original feature vector of an edge.

[0096] E2, inputting the edge original geometric feature information and the edge type information into an edge encoder to obtain an initial edge hidden state.

[0097] The robot action planning device inputs the edge original geometric feature information and the edge type information into an edge encoder to obtain an initial edge hidden state after obtaining the edge original geometric feature information. That is, the original geometric feature vector of an edge and the original feature vector of the edge type are input into a special edge encoder, and the output is an initial edge hidden state. That is, the edge type and other geometric features together form a complete original feature vector which is input into a special edge encoder, and the output is an initial edge hidden state.

[0098] C4, inputting the initial node hidden state, the global action vector, the initial edge hidden state, and the edge type information into a processor stacked by N graph network blocks for iteration to obtain a final node hidden state; The robot action planning device inputs the initial node hidden state, the global action vector, the initial edge hidden state, and the edge type information into a processor stacked by N graph network blocks for iteration to obtain a final node hidden state after obtaining the initial node hidden state, the global action vector, and the initial edge hidden state. N can be 10 or 20.

[0099] As an implementation mode, as shown in Figure 3 and Figure 4 the initial node hidden state, the global action vector, the initial edge hidden state, and the edge type information are input into a processor stacked by N graph network blocks for iteration to obtain a final node hidden state, which specifically includes: F1, obtaining a current updated edge hidden state according to the initial node hidden state and the initial edge hidden state; The robot action planning device obtains a current updated edge hidden state according to the initial node hidden state and the initial edge hidden state after obtaining the initial node hidden state and the initial edge hidden state. That is, the robot action planning device updates the edge after obtaining the initial node hidden state and the initial edge hidden state. When updating the edge, for each edge , an update function The new edge state is calculated based on the hidden states of the two connected nodes , and the current edge state . This simulates the calculation of the interaction force between objects. The current updated edge hidden state is calculated as follows: where is the physical state (including position, velocity, material properties) of the robot component (e.g., finger) at the previous micro-time step; is the physical state (including position, velocity, material properties) of the contacted object (e.g., cup) at the previous micro-time step; is the contact state (e.g., current contact force magnitude, friction coefficient, whether slipping occurs) of the two at the previous time step; is the first weight matrix, which is the physical interaction law (e.g., Coulomb's law of friction) learned by the network; is the activation function, which is used to fit the complex physical contact nonlinear characteristics; is the bias vector of the edge update network. The previous updated edge hidden state is calculated based on the current states of the robot finger and the object, as well as their previous contact situation, and how the contact force or connection between them will change at the next time step.

[0100] F2, obtaining the current iteration node hidden state according to the current updated edge hidden state, the global action vector, and the edge type information; The robot action planning device obtains the current iteration node hidden state according to the current updated edge hidden state, the global action vector, and the edge type information after obtaining the current updated edge hidden state and the global action vector; As an implementation mode, as shown in Figure 3 and Figure 4 , obtaining the current iteration node hidden state according to the current updated edge hidden state, the global action vector, and the edge type information specifically includes: G1, detecting whether the edge type information is a contact edge; The robot action planning device detects whether the edge type information is a contact edge after obtaining the edge type information; G2, if the edge type information is a contact edge, inputting the current updated edge hidden state into a contact interaction encoder to encode and obtaining a current contact edge hidden state; The robot action planning device inputs the current updated edge hidden state into a contact interaction encoder to encode and obtains a current contact edge hidden state when it is judged that the edge type corresponding to the current updated edge hidden state is a contact edge.

[0101] G3, if the edge type information is not a contact edge, input the current updated edge hidden state into a neighbor interaction encoder to encode and obtain a current neighbor edge hidden state; The robot motion planning device judges whether the edge type corresponding to the current updated edge hidden state is a contact edge, i.e., a neighbor edge, and if so, inputs the current updated edge hidden state into a neighbor interaction encoder to encode and obtain a current neighbor edge hidden state.

[0102] G4, according to the current contact edge hidden state and / or the current neighbor edge hidden state, obtain node aggregation information; The robot motion planning device obtains node aggregation information according to the current contact edge hidden state and / or the current neighbor edge hidden state.

[0103] That is, for each node , aggregate the information of all edges connected thereto. This simulates the calculation of the resultant force on the object. The node aggregation information . (contact force / interaction term) directly corresponds to the current updated edge hidden state calculated in step F1 , specifically, the updated edge hidden state encodes the interaction relationship between node and node ; for a contact edge, the hidden state vector is physically represented as a contact force vector (including the size and direction of the force) exerted by node on node ; for a "neighbor edge", it is represented as a potential field force or spatial attention. The symbol is used in the formula to intuitively express its "force" physical property. The node aggregation information , specifically, in the current iteration step , node collects the comprehensive features after the influence of all neighbor nodes, representing the resultant external force or total physical influence on the object (node) at the current time; this aggregation information will be input into the next "node update" formula to calculate the acceleration and motion state of the object at the next time (i.e., the process of simulation ).

[0104] G5, according to the initial node hidden state, the global action vector, and the node aggregation information, obtain a current iteration node hidden state; The robot action planning device obtains the current iteration node hidden state according to the initial node hidden state, the global action vector and the node aggregation information.

[0105] Many standard GNN models (such as GCN, GraphSAGE) treat all edges equally, using the same message passing and update functions. This is not true in the physical world, because "contact" and "proximity" are two very different kinds of relationships with completely different physical laws. By introducing edge type-based conditioning computation, in the edge update step of the graph network block (GNB), the network will choose different computation paths according to the type of the edge ("contact edge" or "proximity edge"). A special MLP subnetwork is designed for each type of edge. When processing a "contact edge", the system will call a more complex and parameter-rich ContactNet subnetwork, which is specifically designed to learn contact mechanics (such as friction, elastic force). When processing "proximity edges", a simpler ProximityNet is called, which may only learn some weak interactions or not pass information.

[0106] By introducing edge type-based conditioning computation, in the edge update step of the graph network block (GNB), the network will choose different computation paths according to the type of the edge ("contact edge" or "proximity edge"), reducing the learning difficulty of the model and enabling it to more accurately capture the essence of different physical interactions, avoiding the use of a set of ambiguous parameters to fit all cases.

[0107] As an implementation, obtaining the current iteration node hidden state according to the initial node hidden state, the global action vector and the node aggregation information can include: ; wherein, is the current iteration node hidden state; is the node aggregation information of node i; is the node hidden state of the last iteration of node v; is the global action vector; is the update function; is the second weight matrix; is the bias vector of the node update network.

[0108] Standard GNNs are usually used to predict the natural evolution of a system, and they lack a clear and continuous external intervention input. In the task of the application, the action of the robot is a core external factor that drives the change of the entire physical system throughout. By introducing the encoded global action vector As a global condition, the global action vector is injected into the node update step of each GNB in the Processor. Specifically, the standard node update function is modified to This means that at each discrete time step of the simulation, the network is "always aware" of the action being applied by the robot and computes the future state of the nodes accordingly.

[0109] The encoded global action vector is injected as a global condition into the node update step of each GNB in the Processor. This makes the pre-trained PICP-Net from a passive observer into an active causal prediction engine that can reason "what will happen under... action". It enables conditional prediction and enhances task relevance, ensuring that the model learns and predicts the physical process that is directly related to the current task intention of the robot, rather than aimlessly simulating all possibilities. This improves the effectiveness and practicality of the prediction.

[0110] In terms of achieving the effect of conditional prediction: the model's prediction results are closely related to the input action. We can ask "what if I push with 5N force?" and "what if I push with 10N force?" and get two different answers.

[0111] F3, obtaining the next iteration node hidden state according to the current iteration node hidden state, the current update edge hidden state and the global action vector, and taking the Nth iteration node hidden state as the final node hidden state after N iterations.

[0112] The robot action planning device obtains the next iteration node hidden state according to the current iteration node hidden state, the current update edge hidden state and the global action vector after obtaining the current iteration node hidden state, the current update edge hidden state and the global action vector, and takes the Nth iteration node hidden state as the final node hidden state after N iterations.

[0113] As an implementation, obtaining the next iteration node hidden state according to the current iteration node hidden state, the current update edge hidden state and the global action vector can include: ; Wherein, is the current iteration node hidden state; is the node aggregation information of node i; is the node hidden state of the last iteration of node v; is the global action vector; For update functions.

[0114] That is, the hidden state of the node in the Nth iteration. This is the hidden state of the final node.

[0115] By stacking N GNB graph network blocks, the network can simulate the physical evolution process over a relatively long period (e.g., 500ms), ultimately obtaining a final node hidden state containing future state information. .

[0116] C5 inputs all the hidden states of the final nodes into the pre-trained task decoder for decoding to obtain the risk vector of the current iteration.

[0117] After obtaining the hidden states of the final nodes, the robot motion planning device inputs all the hidden states of the final nodes into the pre-trained task decoder for decoding to obtain the risk vector of the current iteration.

[0118] As one implementation, the task decoder includes a success probability head, a physical quantity prediction head, a state prediction head, and a failure mode classification head; the risk vector includes: success probability, key physical quantities, final state, and one-hot encoding of the most likely failure mode; all the hidden states of the final nodes are input into the pre-trained task decoder for decoding to obtain the risk vector for the current iteration, specifically including: Input all the hidden states of the final node into the success probability header, physical quantity prediction header, state prediction header and failure mode classification header respectively to obtain the one-hot encoding of success probability, key physical quantity, final state and most likely failure mode.

[0119] The task decoder is an MLP set that receives the final graph state (of all nodes) output by the processor. And decode it into the structured risk vector we need.

[0120] That is, after obtaining the hidden state of the final node, all of them... By inputting the pre-trained success probability header, the pre-trained physical quantity prediction header, the pre-trained state prediction header, and the pre-trained failure mode classification header respectively, one-hot encodings of success probability, key physical quantities, final state, and most likely failure mode are obtained.

[0121] The success probability header consists of a graph-level pooling operation that aggregates information from all nodes, and then outputs a success probability between 0 and 1 using an MLP and a sigmoid function. .

[0122] The physical quantity predictor head is a MLP that directly regresses the key physical quantity of a specific node (e.g. the target object), such as the maximum stress value (predicted_max_stress).

[0123] The state classifier head is another MLP that classifies the final state of the same specific node, predicting whether it will slip (is_slip) or fall (is_stable).

[0124] The failure mode classifier head is a graph-level MLP that classifies the final state of the entire graph, outputting a one-hot vector of the most likely failure mode, such as [slip, stress failure, collision,...].

[0125] The problem with standard GNNs: the input features of nodes are usually flattened into a single vector. Physically, the properties of objects (such as mass) and their states (such as velocity) are two different types of information. At the same time, standard decoders usually only make a general graph-level prediction (such as Figure 3 the classification shown).

[0126] The multi-head task decoder: the decoder is not a single output module, but a parallel collection of multiple dedicated "decoder heads". Each head is trained to solve a specific sub-task (predict stress, predict slip, predict success rate, etc.).

[0127] By decomposing a complex prediction problem into multiple simpler sub-problems through a multi-head decoder, not only does it reduce the difficulty of training, but the structured risk vector output provides the downstream decision controller with much richer and more interpretable decision-making basis than the single "success / failure" signal.

[0128] When training the physical causal prediction network model (PICP-Net), the training of the physical causal prediction network model (PICP-Net) is completely carried out in an offline virtual environment.

[0129] The dataset can be obtained by sequentially sampling scenes, sampling actions, running truth simulations, extracting labels, and storing data.

[0130] Among them, scene sampling is to programmatically generate millions of different interaction scenarios in a dynamic virtual twin. Each scenario contains randomly placed objects with different physical properties (randomized mass, friction, shape, fragility). Action sampling is to randomly sample thousands of different action_proposals in each scenario. Action parameters (such as grabbing force, speed, direction) are randomized within a reasonable range. Running the true value simulation is to run a high-fidelity dynamic virtual twin for each "scene-action" pair, and record the complete physical evolution process.

[0131] Label extraction is to accurately calculate the true value label corresponding to the risk vector from the simulation results. The true value label corresponding to the risk vector includes: success label, physical quantity label, state label and failure mode label. Among them, the success label is whether the final check task is achieved (such as whether the object is stably picked up). The physical quantity label is to record the physical quantity borne by the target object during the simulation process. The state label is to analyze whether the object finally tips over or slides. The failure mode label is to determine the real failure reason according to the simulation results.

[0132] Data storage is to store each group (initial state, action proposal, benchmark risk vector…) as a data point in the training data set.

[0133] After obtaining the training data set, training can be performed through the calculation of the loss function (Loss Function&Training). A multi-task learning (Multi-task Learning) framework can be used, and the loss function is the weighted sum of the loss of each decoder head.

[0134] ; Among them, the success probability head loss function uses binary cross-entropy loss (Binary Cross-EntropyLoss). The physical quantity prediction head loss function , the state prediction head prediction function : uses mean squared error loss (Mean Squared Error, MSE). The failure mode classification head loss function uses cross-entropy loss (Cross-Entropy Loss). is a hyperparameter weight used to balance the importance of different tasks.

[0135] The entire PICP-Net can be trained using a standard deep learning optimizer (such as Adam) until the loss on the validation set converges.

[0136] By the above technical solution, a specific and implementable PICP-Net is defined. It effectively learns physical interaction through the GNN architecture, outputs rich risk information through the multi-task decoder, and is trained through large-scale offline simulation data.

[0137] As an implementation, the risk vector includes success probability, key physical quantity, final state, and one-hot encoding of the most likely failure mode; and if the risk vector of the current iteration does not meet the safety threshold table, the action draft of the next iteration is generated according to the risk vector of the current iteration, which can include: H1, respectively, the success probability, the key physical quantity and the final state are compared with the safety threshold table to generate a decision signal; H2, if the decision signal is a correction signal, the one-hot encoding of the most likely failure mode is queried from the correction mapping table, and a correction instruction is generated according to the risk vector; H3, according to the correction instruction and the current action draft, the action draft of the next iteration is generated.

[0138] After obtaining the risk vector, the robot action planning device must be able to intelligently formulate a strategy for correcting action parameters according to the nature of the risk when detecting the risk. And it is responsible for managing the entire "proposal-check-correction" loop, including starting, iteration, and terminating after success or reaching the maximum number of attempts. The safety threshold table (SafetyThreshold Table) can be dynamically adjusted according to the fragility of the interactive object.

[0139] For example: when the risk vector is {"success probability": 0.998; "stability score": 0.99; "predicted maximum stress": 4.6; "failure mode vector": [0.01, 0.02, 0.95,...], / / corresponding to [slip, stress, collision,...]; "slip prediction value": 0.01}.

[0140] The safety threshold table is {if the target object is fragile: stress threshold = 5.0; otherwise: stress threshold = 20.0; success probability threshold = 0.99; slip probability threshold = 0.1} After threshold comparison of the risk vector and the safety threshold table, a decision signal is generated. The step of threshold comparison of the risk vector and the safety threshold table is {if (the success probability of the risk vector > the success probability threshold) and (the predicted maximum stress of the risk vector < the stress threshold) and (the slip prediction value of the risk vector < the slip probability threshold), then the decision signal = "approved"} {otherwise, the decision signal = "rejected and optimized"}. The correction signal is "the decision signal = "rejected and optimized" When the decision signal is the "reject and refine" signal, i.e., the decision signal is the revise signal, a specific revise instruction is generated.

[0141] The specific revise instruction is generated, and a preferred embodiment uses an expert-knowledge-based "failure mode-refine strategy" mapping table.

[0142] According to the failure mode vector (risk vector) and other risk indicators, the system queries a pre-defined, structured mapping table (e.g., as shown in Table 1). This table maps the most likely failure mode (the category with the highest probability in the vector) to one or more revise actions.

[0143] Table 1 In the generation of the revise instruction, a priority 1 revise strategy corresponding to the detected failure mode is first selected. This strategy is instantiated as a specific revise instruction object (Refinement Instruction Object), for example: {"parameter to be modified": "grasping force"; "modification type": "scale value", / / can be'scale value','set value', 'increase value'; "scale factor": 1.2}. If the priority 1 strategy still fails in subsequent iterations, the system can choose to try a priority 2 strategy, or combine multiple strategies.

[0144] As an implementation, the risk vector includes a success probability, a key physical quantity, a final state, and a one-hot encoding of the most likely failure mode; and if the risk vector of the current iteration does not meet the safety threshold table, generating a draft action for the next iteration according to the risk vector of the current iteration can include: H1, performing threshold comparisons between the success probability, the key physical quantity, and the final state, respectively, and the safety threshold table to generate a decision signal; H2, if the decision signal is a revise signal, automatically calculating the optimal revise instruction using gradient descent method by calculating the gradient of the input action parameter of the risk vector; H3, generating a draft action for the next iteration according to the revise instruction and the current draft action.

[0145] That is, in another embodiment of the present application, since the PICP-Net is differentiable, the optimal revise instruction can be automatically calculated using gradient descent method by calculating the gradient of the input action parameter of the risk vector without table lookup.

[0146] After the action draft and the virtual twin state information are processed by the pre-trained physical causal prediction network model to determine the risk vector of the current iteration, the following can be included: If the risk vector of the current iteration meets the safety threshold table, the action draft corresponding to the risk vector meeting the safety threshold table is taken as the robot action instruction.

[0147] The risk vector includes a success probability, a key physical quantity, a final state, and a one-hot encoding of the most likely failure mode; and if the risk vector of the current iteration meets the safety threshold table, the action draft is taken as the robot action instruction, including: Threshold comparison of the success probability, the key physical quantity, and the final state with the safety threshold table respectively to generate a decision signal; If the decision signal is an approval signal, the action draft corresponding to the risk vector meeting the safety threshold table is taken as the robot action instruction.

[0148] As an implementation, the virtual twin state information of the next iteration is obtained based on the set of scene perception images, the sensor information, and the risk vector of the current iteration, and specifically includes: K1, interaction dynamic information is obtained according to the set of scene perception images and the sensor information; After the robot action planning device obtains the set of scene perception images and the sensor information, the interaction dynamic information is obtained according to the set of scene perception images and the sensor information. The interaction dynamic information can include target object dynamic information, robot dynamic information, and environment object state information. Among them, the target object dynamic information includes target object accurate pose and target object dynamics state. For example, the target object accurate pose is position: [x, y, z], direction / pose: [qx, qy, qz, qw]. The target object dynamics state is linear velocity: [0, 0, 0], angular velocity: [0, 0, 0] (because it is in a static state).

[0149] Among them, the robot dynamic information includes robot gripper state, including the accurate position and pose of the robot finger joint.

[0150] The state information of the nodes of the robot itself (such as the finger joints) can be determined through forward kinematics and proprioception. The robot action planning device reads the accurate angles of each joint encoder through the real-time controller of the robot. Then, by using the kinematics model of the robot (a mathematical formula describing its geometric structure), the position and pose of the end effector (gripper, finger joint) in the world coordinate system can be accurately calculated from the joint angles. Finally, by differentiating the position and pose of two consecutive frames, the linear velocity and angular velocity can be calculated.

[0151] The environment object state information can include support surface state information and other object state information. The support surface (table surface) state information includes the position, normal vector, and friction coefficient of the support surface.

[0152] The state information of the object nodes in the environment (such as a vase, a table) (environmental object state information) can be determined by a three-dimensional object detection, pose estimation and tracking algorithm. First, the object is identified and segmented from the image. Then, combining the RGB image and the depth image, a pose estimation algorithm (such as a variant of PoseCNN or a point cloud registration-based method) is used to calculate the 6D pose (position + attitude) of the object in the camera coordinate system, and then converted to the world coordinate system. Finally, in order to obtain the speed information, the system will run a tracking algorithm (such as Kalman Filter or a simpler SORT / DeepSORT algorithm). The algorithm will associate the same object in consecutive frames, and estimate its linear and angular velocity through smoothing and filtering.

[0153] K2, performing object recognition on the set of scene perception images through an object detection model to obtain target object images; The robot action planning device, after obtaining the set of scene perception images, performs object recognition on the set of scene perception images through an object detection model to obtain target object images; the target object images are images of target objects; for example, when the target object is a vase, the target object images are images of the vase.

[0154] That is, through a standard object detection or segmentation model (such as a YOLO model, a SAM model), the target object is located from the RGB image.

[0155] K3, detecting whether the risk vector of the current iteration meets the quality high sensitivity condition; The robot action planning device, after obtaining the risk vector of the current iteration, detects whether the risk vector of the current iteration meets the quality high sensitivity condition.

[0156] K4, if the risk vector of the current iteration meets the quality high sensitivity condition, performing a tentative action on the target object and obtaining target object tentative collection information; obtaining physical property estimation information according to the target object tentative collection information and the set of scene perception images; packaging the interaction dynamic information and the physical estimation information to obtain virtual twin state information for the next iteration; The robot action planning device, after judging that the risk vector of the current iteration meets the quality high sensitivity condition, performs a tentative action on the target object and obtains target object tentative collection information; obtains physical property estimation information according to the target object tentative collection information and the set of scene perception images; packages the interaction dynamic information and the physical estimation information to obtain virtual twin state information for the next iteration; That is, the robot motion planning device triggers the steps of phase three, i.e., triggers the quality recognition subprogram, after judging that the success rate in the risk vector of the current iteration is highly sensitive to the mass parameter of the object.

[0157] As an implementation form, a tentative action is performed on the target object, and tentative collection information of the target object is obtained; physical attribute estimation information is obtained according to the tentative collection information of the target object and the set of scene perception images; the interaction dynamic information and the physical estimation information are packaged to obtain virtual twin state information of the next iteration, specifically including: L1, performing a tentative action on the target object, and obtaining tentative collection information of the target object; The robot motion planning device performs a tentative action on the target object and obtains tentative collection information of the target object after judging that the success rate in the risk vector of the current iteration is highly sensitive to the mass parameter of the object. As an implementation form, the detection of whether the risk vector of the current iteration meets the high-sensitivity condition of quality includes: Obtaining a mass range of the target object; According to the mass range of the target object, obtaining a maximum estimated mass and a minimum estimated mass; Replacing the mass range of the target object in the action draft of the next iteration with the maximum estimated mass and the minimum estimated mass respectively to obtain a first quality-sensitive action draft and a second quality-sensitive action draft; Processing the first quality-sensitive action draft and the virtual twin state information through a pre-trained physical causal prediction network model to obtain a first quality-sensitive success rate; Processing the second quality-sensitive action draft and the virtual twin state information through a pre-trained physical causal prediction network model to obtain a second quality-sensitive success rate; Subtracting the second quality-sensitive success rate from the first quality-sensitive success rate to obtain a sensitivity score; Detecting whether the sensitivity score is greater than a sensitivity decision threshold; If the sensitivity score is less than or equal to the sensitivity decision threshold, the risk vector of the current iteration meets the high-sensitivity condition of quality. If the sensitivity score is greater than the sensitivity decision threshold, the risk vector of the current iteration does not meet the high-sensitivity condition of quality.

[0158] Wherein, the reference steps C1 to C5 for obtaining the first quality-sensitive success rate and the second quality-sensitive success rate.

[0159] Obtaining the estimated value of the mass of the target object and its uncertainty. This uncertainty can be represented as a standard deviation.

[0160] The decision controller creates two temporary, hypothetical virtual twin state copies in memory. The high state (max estimated mass): in this copy, all parameters are identical to the current state, except the mass of the object is set to the max estimated mass. The low state (min estimated mass): in this copy, all parameters are identical to the current state, except the mass of the object is set to the min estimated mass.

[0161] The decision controller uses the current action draft to re-call the pre-trained physics causal prediction network model (PICP-Net) to make predictions on both perturbed states, obtaining the first mass-sensitive success rate and the second mass-sensitive success rate . The sensitivity score = abs( - ).

[0162] As an implementation, the detecting whether the risk vector of the current iteration satisfies the mass high sensitivity condition can further include: processing the action draft and the virtual twin state information by the pre-trained physics causal prediction network model to obtain a mass gradient; taking an absolute value of the mass gradient to obtain a sensitivity score; detecting whether the sensitivity score is greater than a sensitivity decision threshold; if the sensitivity score is less than or equal to the sensitivity decision threshold, the risk vector of the current iteration satisfies the mass high sensitivity condition; if the sensitivity score is greater than the sensitivity decision threshold, the risk vector of the current iteration does not satisfy the mass high sensitivity condition.

[0163] The decision controller calls the pre-trained physics causal prediction network model (PICP-Net) to make predictions while also requesting to calculate backpropagation gradients. The PICP-Net returns the baseline success rate and the gradient of the success rate with respect to the mass in one forward + backpropagation.

[0164] The sensitivity score is directly the absolute value of the mass gradient . The decision controller also compares this sensitivity score with the sensitivity decision threshold, and combines whether the baseline success rate is in the "danger zone" to make a final decision on whether to trigger the physical exploration.

[0165] That is, after judging that the success rate in the risk vector of the current iteration is highly sensitive to the mass parameter of the object, the robot performs one or more "tentative actions" and obtains the slight displacement of the target object due to the execution of the tentative action by the target object, and then the robot motion planning device obtains the target object tentative collection information through the camera or sensor. This is a standardized and extremely slight interaction sequence. The target object performing a tentative action can be a light pushing action, that is, using the fingertips to gently push the object (target object) with a known, slight force. The target object tentative collection information is obtained by tracking the displacement and acceleration of the object through vision (camera) and recording the contact force through the force sensor (sensor) of the fingertips to achieve data collection of the target object tentative collection information. The surface of the object can be pressed with a slight and incremental force, and the slight deformation of the object can be perceived through the force sensor and vision (or fingertip position sensor) to estimate the stiffness or fragility of the object.

[0166] L2, obtaining physical attribute estimation information according to the target object tentative collection information and the set of scene perception images; After obtaining the target object tentative collection information and the set of scene perception images, the robot motion planning device obtains physical attribute estimation information according to the target object tentative collection information and the set of scene perception images; wherein the high-precision target object physical information includes high-precision target object estimated mass, high-precision target object friction coefficient, high-precision target object fragility score, high-precision center of mass coordinate information, and high-precision shape description information; Obtaining physical attribute estimation information according to the target object tentative collection information and the set of scene perception images includes: performing object three-dimensional construction on the target object according to the set of scene perception images to obtain a target object three-dimensional network model; performing geometric center and shape parameter calculation on the target object three-dimensional network model to obtain center of mass coordinate information and shape description information; correcting the center of mass coordinate information and shape description information according to the target object tentative collection information to obtain high-precision center of mass coordinate information and high-precision shape description information; obtaining high-precision target object estimated mass, high-precision target object friction coefficient, and high-precision target object fragility score according to the target object tentative collection information; and merging high-precision target object estimated mass, high-precision target object friction coefficient, high-precision target object fragility score, high-precision center of mass coordinate information, and high-precision shape description information to obtain physical attribute estimation information.

[0167] The inertial mass of the object can be inversely identified from [force, displacement, acceleration] data.

[0168] The surface of the object can be pressed with a slight and incremental force, and the slight deformation of the object can be perceived through the force sensor and vision (or fingertip position sensor) to estimate the stiffness or fragility of the object.

[0169] Phase 3: Online parameter identification based on active interaction; 1. Goal: When the task requires high precision of a certain physical parameter, obtain the high-precision true value of the parameter by planning and executing minimally invasive physical interaction, and use it as the final calibration.

[0170] 2. Core components are parameter identification controller and sensors; the parameter identification controller is a software module responsible for planning and executing the sequence of "physical experiment" actions. The sensors are the six-axis force / torque sensor at the end of the robot and the external RGB-D camera.

[0171] 3. Specific implementation steps a. Triggering condition: triggered by the [E] decision controller. For example, when the prediction of the physical causal prediction network model (PICP-Net) shows that the success rate of the task is highly sensitive to the mass parameter of the object, trigger the mass identification subprogram.

[0172] b. Action planning and execution (taking the identification of "mass" and "static friction coefficient" as an example): Step A: Identify the static friction coefficient 1. Planning: The parameter identification controller plans a "slow tilting" action. If the object is on a movable plane (such as a tray), the robot will grab the plane; if the object is directly on a fixed plane, the robot will try to push it with the fingertips. Taking tilting as an example, plan a trajectory that tilts the plane at a constant low angular velocity ω.

[0173] 2. Execution and perception: the robot executes the tilting action. At the same time, the vision system tracks the object at a high frequency (>100Hz) and accurately detects the critical tilt angle of the plane when the object starts to slide .

[0174] 3. Calculation: calculate the high-precision static friction coefficient according to the physical formula .

[0175] Step B: Identify the mass m 1. Planning: After knowing , the parameter identification controller plans a "controlled flat push" action. Plan a trajectory that allows the robot fingertips to horizontally push the object with a constant, known force Fpush (achieved through closed-loop force control of the force sensor).

[0176] ​​2. Execution vs. Perception: The robot performs a push action. Meanwhile, the vision system tracks the object's motion at high frequency and computes its acceleration a under the force.

[0177] 3. Computation: According to Newton's second law i.e. (where the normal force N = mg), solve the equation i.e. the mass m of the object can be solved inversely.

[0178] In the prior art, these parameters usually need to be manually set for the object in the simulation environment offline. The present patent proposes a complete, online, from non-contact to contact, multi-modal fusion system and method to determine these key physical properties in real time for unknown objects encountered by the robot. This greatly enhances the adaptability and versatility of the system.

[0179] L4, pack the interaction dynamic information and the physical estimation information to obtain virtual twin state information.

[0180] The robot action planning device packs the target object dynamic information, the manipulator dynamic information, and the physical estimation information to obtain virtual twin state information after obtaining the interaction dynamic information and the physical estimation information.

[0181] Finally, the high-precision parameters recognized through physics are updated to the virtual twin as final values.

[0182] At the same time, the system takes the result of this recognition (i.e., [visual feature] -> [real physical parameter]) as a high-quality training sample, which can be used for incremental learning or online fine-tuning of the physical property regression network (PPR-Net) in the future, so that the vision estimation capability of the entire system continues to evolve.

[0183] K5, if the risk vector of the current iteration does not meet the quality high sensitivity condition, obtain physical property estimation information according to the target object image and the set of scene perception images; pack the interaction dynamic information and the physical estimation information to obtain virtual twin state information for the next iteration.

[0184] The robot action planning device obtains physical property estimation information according to the target object image and the set of scene perception images if the risk vector of the current iteration does not meet the quality high sensitivity condition; packs the interaction dynamic information and the physical estimation information to obtain virtual twin state information for the next iteration.

[0185] That is, the robot motion planning device triggers the steps of stage one or stage two to perform steps B3 and B4 after determining that the success rate in the risk vector of the current iteration is not highly sensitive to the mass parameter of the object; that is, the robot motion planning device obtains physical property estimation information according to the target object image and the set of scene perception images after determining that the success rate in the risk vector of the current iteration is not highly sensitive to the mass parameter of the object; and packs the interaction dynamic information and the physical property estimation information to obtain virtual twin state information for the next iteration.

[0186] When training the physical property estimation information, the CNN is trained in an offline state on a large data set containing images of objects of various materials and their corresponding real physical properties. This data set can be measured manually or generated automatically in a simulated environment. The model learns the implicit correlation from visual appearance (such as texture, material, transparency, deformation) to physical properties. For example, it can learn that metal objects are usually heavier, glass objects are more fragile, and rubber objects have greater friction.

[0187] The physical property estimation information can be obtained by a three-stage progressive physical property estimation method.

[0188] Stage one: category matching based on prior knowledge base; 1. Objective: quickly assign a reasonable, category-based initial physical property baseline to the target object in the scene.

[0189] 2. Core components: object recognition network and physical property prior database. The object recognition network is a pre-trained deep learning model that can output object category labels, such as a fine-tuned ResNet, EfficientNet, or CLIP model. The physical property prior database is a structured, extensible database (e.g., JSON, XML, or SQL database) that stores a large number of typical physical properties of common object categories.

[0190] 3. Specific implementation steps a. Input reception: receive the RGB image slice and / or three-dimensional point cloud segment of the target object provided by the upstream perception module.

[0191] b. Category recognition: input the image into the object recognition network to obtain one or more highest-confidence category labels, such as {"category": "ceramic mug", "confidence": 0.98}.

[0192] c. Database query: use the highest-confidence category label as the primary key to query the physical property prior database.

[0193] d. Data structure: Each entry in the database is a structured object containing statistical distribution information of physical parameters of the category object, at least including mean and std_dev.

[0194] Database entry example: {"category ID": "ceramic mug 01"; "description": "standard ceramic mug"; "attributes": {"mass (kg)": ["mean": 0.35, "std_dev": 0.12], "static friction coefficient": ["mean": 0.7, "std_dev": 0.15], "center of mass offset (m)": {"mean": [0, 0, -0.01], "std_dev": [0, 0, 0.005]}, "brittleness score": {"mean": 0.9, "std_dev": 0.05}} e. Attribute assignment: The mean value obtained from the query is assigned as the initial physical attribute of the virtual twin to the corresponding digital model. The standard deviation (std_dev) of the object is also recorded as an initial measure of parameter uncertainty for subsequent steps.

[0195] Stage two: parameter fine-tuning based on visual appearance; 1. Objective: Based on the individual visual features of the object (such as size, shape, texture), individualize the category average provided in stage one to obtain more accurate non-contact estimation.

[0196] 2. Core component: Physical property regression network (PPR-Net), a specially designed multi-output deep learning network.

[0197] 3. Specific implementation steps (specific steps are deep learning training) a. Input preparation: Prepare the input data for the physical property regression network (PPR-Net), including: RGB image slices of the target object.

[0198] Optionally, three-dimensional mesh or point cloud of the object (for graph convolution or PointNet layer).

[0199] Category ID obtained in stage one (as a conditional input to activate category-related knowledge).

[0200] b. Network inference: Send the data into the physical property regression network (PPR-Net) for forward propagation.

[0201] Network Architecture: The PPR-Net can internally adopt a CNN backbone (e.g. MobileNet) to extract 2D visual features, and optionally fuse a GCN or PointNet branch to handle 3D information. The network ends with multiple parallel Regression Heads.

[0202] Output Definition: Each Regression Head is specialized to predict a correction value for one physical parameter, which can be a scale factor or an offset.

[0203] Output Example: {"mass scale factor": 1.3; "friction offset": -0.05; "brittleness offset": 0.02} c. Parameter Update: Apply the correction values output by the PPR-Net to the baseline parameters from Stage 1.

[0204] Update Formula: corrected mass = baseline mass × mass scale factor corrected friction coefficient = baseline friction coefficient + friction offset.

[0205] The updated parameters will replace the properties of the object in the virtual twin.

[0206] d. The PPR-Net is fully trained offline. Its training set contains a large number of objects' visual data and their real physical parameters obtained through physical measurement. The loss function is designed to minimize the difference between the parameters after applying the correction values predicted by the network and the real parameters.

[0207] wherein the step of determining the risk vector for the next iteration according to the action draft for the next iteration and the virtual twin state information for the next iteration has a maximum iteration number of 5 or 10.

[0208] When the risk assessment logic returns the annotation signal, the loop immediately terminates and returns the final optimized action proposal. When the current number of attempts reaches the maximum number of attempts, if a safe action is still not found, the loop is forced to terminate and reports a check failure to the upper layer.

[0209] Figure 5 Flowchart of the task completion process for the robot action planning method. The specific implementation of the robot action planning method of the present application is as follows: Scenario setup: The user's instruction is "Please move the glass vase on the table to the empty spot next to it." The robot has broken this task down into three high-level action drafts via a VLA model (VLA model not discussed in this patent): [A: Approach vase, B: Grasp vase, C: Move and place vase].

[0210] Wherein, how to handle the step of grasping the vase is as follows.

[0211] First, a main function to execute the safety action plan will be defined, which receives the user's natural language instruction as input and ultimately drives the robot to execute a safety-checked action sequence.

[0212] Step 1, Verify and optimize action function launch - first attempt: 1.1 Initialize action proposal When the main process calls the verify and optimize action function, the initial action draft passed in is generated by the VLA model. This draft is "semantically correct" but "physically ambiguous". It could be: {"action name": "grasp", "target object ID": "vase_01", "grasp strategy": "cylindrical grip grasp", / / VLA (Visual Language Model / Visual Analysis Module) determines from vision that the object is a cylinder; "initial parameters": { "approach speed": 0.15, / / meters / second, regular speed "grasp force": 10.0, / / Newton, general default force "lift acceleration": 0.5 / / meters / second 2 , standard acceleration} This initial draft will be named the current action proposal.

[0213] The current attempt count is set to 1.

[0214] 1.2 Get high-fidelity world state The decision controller [E] sends a state query request to the virtual twin [C]. The virtual twin [C] (updated in real-time by the perception module [A]) returns an extremely detailed "world snapshot" including the following information: Vase state: Exact pose: position: [x, y, z], orientation, pose: [qx, qy, qz, qw].

[0215] Dynamics state: linear velocity: [0, 0, 0], angular velocity: [0, 0, 0] (because it is stationary).

[0216] Estimated physical properties: {mass: 0.8 kg, friction coefficient: 0.4, center of mass: [offset_x, offset_y, offset_z], fragility score: 0.95}. The fragility score is estimated by the system based on visual cues (transparency, texture) or previous light contact, and it directly influences the stress threshold for the next step.

[0217] Robot gripper state: includes the precise position and pose of each joint.

[0218] Support surface (tabletop) state: position, normal vector, and friction coefficient.

[0219] 1.3 Invoke the physical causal prediction network model for prediction The action draft and virtual twin state information are packaged together and sent to the physical causal prediction network model. The core task of the physical causal prediction network model is to answer the question: "If a robot gripper in a virtual twin state executes the action draft, what is the most likely outcome within the next 500 milliseconds?" The physical causal prediction network model performs a super-fast forward pass (inference) through its deep neural network, and instead of a simple "yes / no", it outputs a structured risk vector.

[0220] The risk vector can be { "success probability": 0.35, / / very low success rate "stability score": 0.98, / / stability is fine, vase won't fall "predicted maximum stress": 12.5, / / predicted maximum stress value (in Pascals) "failure mode": "structural high stress failure", / / failure mode: structure damaged due to high stress "slip prediction": 0.02 / / no slip predicted} 1.4 Decision and Analysis The risk vector is then compared to a safety threshold table, for example, the safety threshold table is "fragile object stress threshold = 5.0 Pascals (Pa)".

[0221] The controller finds that the predicted maximum stress (12.5) is much greater than the stress threshold (5.0). At the same time, the success probability is very low, and the failure mode clearly points to "high stress damage".

[0222] Conclusion: the first attempt fails. Directly executing this action will result in the vase being crushed.

[0223] Second step, enter the feedback and optimization loop - Attempt 2, as shown in Figure 6 .

[0224] 2.1 Generate the correction instruction Then, infer from the failure mode: Inference process: "The cause of failure is excessive stress. Stress is mainly determined by the gripping force. Therefore, the most direct correction method is to significantly reduce the gripping force." So it generates a clear correction instruction: Can be described as: {"Parameter to be modified": "Gripping force", "Modification type": "Set value", "New value": 3.0 / / Reduce the force from 10.0 Newton to a more conservative value} 2.2 Apply the correction and update the action proposal The controller updates the action proposal according to the correction instruction, generating version v2: / / Current action proposal v2 {"Initial parameters": { "Approach speed": 0.15, "Gripping force": 3.0, / / <--- Corrected "Lifting acceleration": 0.5} The current attempt count is increased to 2 2.3 Second prediction The process returns to 1.2, and the system again obtains the latest world state (in case there are changes in the outside world) and sends the new action proposal v2 to the physical causal prediction network model. This time, the PICP-Net prediction result may be as follows: {"Success probability": 0.40, / / Success rate is still not high "Stability score": 0.75, / / Stability decreases "Predicted maximum stress": 4.5, / / Stress value is safe "Failure mode": "Low friction sliding", / / New failure mode: slipping due to insufficient friction "Slip prediction value": 0.85 / / Predicted high risk of slipping} 2.4 Second decision and analysis Analysis: The controller sees that the stress problem is solved (4.5<5.0). But a new, equally deadly problem has emerged: slipping.

[0225] Conclusion: Second attempt failed. Direct execution of this action will result in the vase slipping from the hand.

[0226] Third step, further intelligent correction - Third attempt (Attempt 3), as shown in Figure 6 .

[0227] 3.1 Generating a more complex correction instruction The decision controller analyzes the failure reason again: Reasoning process: "The failure reason is slipping. The slipping is due to the grip being too small relative to the object's weight and the surface friction. Simply increasing the grip force will lead back to the first failure (crushing). So, what else can prevent slipping? The answer is: increase the contact area or reduce the acceleration during lifting. So, let's try the latter first, as it seems safer." A new correction instruction is generated: {"Parameter to modify": "Lifting acceleration", "Modification type": "Scale value", "Scaling factor": 0.2 / / Reduce acceleration to 20% of original} 3.2 Applying the correction A new action proposal v3 is born, with a safe grip force of 3.0 N and a significantly reduced lifting acceleration of 0.1 m / s2. The current attempt count is increased to 3.

[0228] 3.3 Third prediction The physical causal prediction network model predicts for v3, returning: {"Success probability": 0.998, "Stability score": 0.99, "Predicted maximum stress": 4.6, "Failure mode": "None", "Slip prediction value": 0.01} 3.4 Final decision: Pass Analysis: All indicators are safe. The success rate is extremely high, stable, and the stress is within the threshold, with no risk of slipping.

[0229] Conclusion: Check passed 3.5 Execution The verify and optimize action function returns (TRUE, final action v3 (final action draft)). The main flow receives this final optimized and safe action instruction and sends it to the underlying action controller. The robot then safely and gently grabs and lifts the glass vase in an extremely gentle and slow manner.

[0230] In summary, the user instruction, the scene perception image set, the sensor information and the safety threshold table are acquired, the action draft is generated according to the user instruction and the scene perception image set, the scene perception image set and the sensor information are processed according to a dynamic virtual twin method to obtain virtual twin state information, the action draft and the virtual twin state information are processed through a pre-trained physical causal prediction network model to determine a risk vector of the current iteration, if the risk vector of the current iteration does not conform to the safety threshold table, an action draft of the next iteration is generated according to the risk vector of the current iteration, and virtual twin state information of the next iteration is obtained based on the scene perception image set, the sensor information and the risk vector of the current iteration, and the risk vector of the next iteration is determined according to the action draft of the next iteration and the virtual twin state information of the next iteration, until the risk vector conforms to the safety threshold table, and the action draft corresponding to the risk vector conforming to the safety threshold table is taken as the robot action instruction. The precise physical consequence prediction for a local and dynamically changing interactive scene is realized without relying on a pre-established perfect global physical model.

[0231] The physical causal prediction and virtual twin technology are deeply integrated into the robot decision-making process, which brings multi-dimensional beneficial effects, and improves the safety, success rate, generalization ability and interaction efficiency of the embodied intelligent system in the real world.

[0232] Consistent with the above, please refer to Figure 6 , Figure 6 A structural schematic diagram of a robot action planning device is provided for the embodiments of the present application. As shown in Figure 6 , the device comprises: The information acquisition module 601 is configured to acquire a user instruction, a scene perception image set, sensor information and a safety threshold table. The action draft determination module 602 is configured to generate an action draft according to the user instruction and the scene perception image set. The twin information determination module 603 is configured to process the scene perception image set and the sensor information according to a dynamic virtual twin method to obtain virtual twin state information. The action instruction determination module 604 is configured to process the action draft and the virtual twin state information by using the pre-trained physical causal prediction network model to determine a risk vector of the current iteration, generate an action draft of the next iteration according to the risk vector of the current iteration if the risk vector of the current iteration does not meet the safety threshold table, and obtain virtual twin state information of the next iteration based on the scene perception image set, the sensor information, and the risk vector of the current iteration; and determine a risk vector of the next iteration according to the action draft of the next iteration and the virtual twin state information of the next iteration, until the risk vector meets the safety threshold table, and then take the action draft corresponding to the risk vector meeting the safety threshold table as the robot action instruction.

[0233] As shown in Figure 7 The terminal device 2 provided by the embodiment of the present application includes at least one processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20. The processor 20 and the memory 21 are connected. When the processor 20 executes the computer program 22, the steps in the robot action planning method embodiment are implemented.

[0234] The embodiment of the present application also provides a computer storage medium. The computer storage medium stores a computer program for electronic data exchange. The computer program causes a computer to execute part or all of the steps of any one of the robot action planning methods described in the above method embodiments.

[0235] The embodiment of the present application also provides a computer program product. The computer program product includes a non-transitory computer-readable storage medium storing a computer program. The computer program causes a computer to execute part or all of the steps of any one of the robot action planning methods described in the above method embodiments.

[0236] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer readable storage medium at least includes any entity or device capable of carrying the computer program code to the device / terminal equipment, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable storage medium can not be an electrical carrier signal and a telecommunication signal.

Claims

1. A robot motion planning method, characterized by, The method comprises the following steps: acquiring user instructions, a scene perception image set, sensor information, and a safety threshold table; generating an action draft according to the user instructions and the scene perception image set; processing the scene perception image set and the sensor information according to a dynamic virtual twin method to obtain virtual twin state information; processing the action draft and the virtual twin state information through a pre-trained physical causal prediction network model to determine a risk vector of the current iteration, and if the risk vector of the current iteration does not meet the safety threshold table, generating an action draft of the next iteration according to the risk vector of the current iteration, and obtaining virtual twin state information of the next iteration based on the scene perception image set, the sensor information, and the risk vector of the current iteration; and determining the risk vector of the next iteration according to the action draft of the next iteration and the virtual twin state information of the next iteration, until the risk vector meets the safety threshold table, and taking the action draft corresponding to the risk vector meeting the safety threshold table as a robot action instruction.

2. The robot motion planning method of claim 1, wherein, The processing of the scene perception image set and the sensor information according to the dynamic virtual twin method to obtain virtual twin state information comprises the following steps: obtaining interaction dynamic information according to the scene perception image set and the sensor information; performing object recognition on the scene perception image set through an object detection model to obtain a target object image; obtaining physical property estimation information according to the target object image and the scene perception image set; packing the interaction dynamic information and the physical property estimation information to obtain virtual twin state information.

3. The robotic action planning method of claim 1, wherein, The obtaining of virtual twin state information of the next iteration based on the scene perception image set, the sensor information, and the risk vector of the current iteration comprises the following steps: obtaining interaction dynamic information according to the scene perception image set and the sensor information; performing object recognition on the scene perception image set through an object detection model to obtain a target object image; detecting whether the risk vector of the current iteration meets a quality-high sensitivity condition; if the risk vector of the current iteration meets the quality-high sensitivity condition, performing a tentative action on the target object, obtaining tentative collection information of the target object, obtaining physical property estimation information according to the tentative collection information of the target object and the scene perception image set, and packing the interaction dynamic information and the physical property estimation information to obtain virtual twin state information of the next iteration; if the risk vector of the current iteration does not meet the quality-high sensitivity condition, obtaining physical property estimation information according to the target object image and the scene perception image set, and packing the interaction dynamic information and the physical property estimation information to obtain virtual twin state information of the next iteration.

4. The robot motion planning method according to any one of claims 1 to 3, characterized in that, The processing of the action draft and the virtual twin state information through the pre-trained physical causal prediction network model to determine the risk vector of the current iteration comprises the following steps: obtaining node attribute information, node state information, and edge type information according to the virtual twin state information; obtaining initial node hidden state and global action vector according to the node attribute information, the node state information, and the action draft; obtaining an initial edge hidden state according to the node attribute information, the node state information and the edge type information; inputting the initial node hidden state, the global action vector, the initial edge hidden state and the edge type information into a processor stacked by N graph network blocks for iteration to obtain a final node hidden state; inputting all the final node hidden states into a pre-trained task decoder for decoding to obtain a risk vector of the current iteration.

5. The robot motion planning method of claim 4, wherein, The method for obtaining the initial node hidden state and the global action vector according to the node attribute information, the node state information and the action draft comprises: inputting the node attribute information into a node attribute encoder for encoding to obtain an initial node attribute hidden vector; inputting the node state information into a node state encoder for encoding to obtain an initial node state hidden vector; obtaining the initial node hidden state according to the initial node attribute hidden vector information and the initial node state hidden vector information; inputting the action draft into an action encoder for encoding to obtain the global action vector.

6. The robot motion planning method of claim 4, wherein, The method for obtaining the final node hidden state by inputting the initial node hidden state, the global action vector, the initial edge hidden state and the edge type information into the processor stacked by N graph network blocks for iteration comprises: obtaining a current updated edge hidden state according to the initial node hidden state and the initial edge hidden state; obtaining a current iteration node hidden state according to the current updated edge hidden state, the global action vector and the edge type information; obtaining a next iteration node hidden state according to the current iteration node hidden state, the current updated edge hidden state and the global action vector, and taking the node hidden state of the Nth iteration as the final node hidden state after N iterations.

7. The robotic action planning method of claim 6, wherein, The method for obtaining the current iteration node hidden state according to the current updated edge hidden state, the global action vector and the edge type information comprises: detecting whether the edge type information is a contact edge; if the edge type information is a contact edge, inputting the current updated edge hidden state into a contact interaction encoder for encoding to obtain a current contact edge hidden state; if the edge type information is not a contact edge, inputting the current updated edge hidden state into a proximity interaction encoder for encoding to obtain a current proximity edge hidden state; obtaining node aggregation information according to the current contact edge hidden state and / or the current proximity edge hidden state; obtaining the current iteration node hidden state according to the initial node hidden state, the global action vector and the node aggregation information.

8. A robot motion planning apparatus characterized by comprising: The method comprises: an information acquisition module is configured to acquire a user instruction, a scene perception image set, sensor information and a safety threshold table; an action draft determination module is configured to generate an action draft according to the user instruction and the scene perception image set; a twin information determination module is configured to obtain virtual twin state information by processing the scene perception image set and the sensor information according to a dynamic virtual twin method. The action instruction determination module is configured to process the action draft and the virtual twin state information by using a pre-trained physical causal prediction network model to determine a risk vector of a current iteration, and if the risk vector of the current iteration does not meet a safety threshold table, generate an action draft of a next iteration according to the risk vector of the current iteration, and obtain virtual twin state information of the next iteration based on the set of scene perception images, the sensor information, and the risk vector of the current iteration. And determine the risk vector of the next iteration according to the action draft of the next iteration and the virtual twin state information of the next iteration, until the risk vector meets the safety threshold table, and take the action draft corresponding to the risk vector meeting the safety threshold table as the robot action instruction.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the robot action planning method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the robot action planning method according to any one of claims 1 to 7.

Images