Flexible arm support kinematics resolving method and device based on graph network simulator

Abstract

The invention relates to a flexible arm support kinematics calculation method and device based on a graph network simulator. The graph network simulator comprises an encoder, a processor and a decoder, the flexible arm support kinematics resolving method comprises the following steps: receiving a flexible arm support motion trail sequence of a current frame through the encoder, and extracting and encoding node information and side information of the flexible arm support motion trail sequence to obtain node features and side features, the node information of the motion trail sequence of the flexible arm support comprises the coordinate position, the node type and the joint angle value of each node; iteratively updating the node features and the edge features through the processor to obtain updated node features; the updated node features are decoded by a decoder into acceleration information of the node, and a node position of the next frame is predicted based on the node information of the current frame and the acceleration information. According to the invention, the precision and physical consistency of motion prediction can be significantly improved.

Description

Technical Field

[0001] This disclosure relates to the field of kinematics and dynamics modeling of flexible robotic arms, and more specifically, to a method and apparatus for kinematics calculation of flexible robotic arms based on a graph network simulator. Background Technology

[0002] Traditional control methods for flexible lever robotic arms are typically based on ideal deterministic physical models and rely on real-time feedback to improve control accuracy. However, for ultra-long flexible booms, the physical model is difficult to determine, and end-effector data is difficult to collect. Therefore, traditional methods have many limitations in controlling ultra-long flexible booms. With the rapid development of artificial intelligence and machine learning technologies, data-driven real-time simulation methods have emerged.

[0003] Sanchez-Gonzalez et al. proposed a Graph Network Simulator (GNS) that represents particles in a physical system as nodes in a graph and computes dynamics through a learned message-passing mechanism. The encoder embeds the input state into a latent graph, the processor updates the graph state through multi-step message passing, and the decoder extracts dynamic information from the final latent graph. The GNS framework can directly simulate physical systems from data through learning, reducing reliance on traditional simulators. GNS can simulate the interactions of various complex physical systems, including fluids, rigid bodies, and deformable materials. By learning the dynamics of physical systems, GNS can improve simulation efficiency while maintaining high accuracy. However, existing graph network-based simulations are highly dependent on the quality and diversity of training data, making effective transfer difficult in data-scarce real-world scenarios, and errors accumulate during long-term simulations.

[0004] To address the error problem in long-term physical series predictions, Raissi et al. proposed a deep learning framework called Physical Information Neural Network (PINN). This framework integrates physical laws into the neural network training process, directly processes differential terms in partial differential equations, and encodes physical laws as constraints for the network. This ensures that the network's predictions satisfy both observational data and physical laws. This innovative integration of physical laws and deep learning not only improves the model's generalization ability in small-data scenarios but also successfully applies it to classic problems in multiple fields, promoting the development of data-driven scientific computing and providing new data-driven tools for scientific computing and engineering modeling. Summary of the Invention

[0005] To address the aforementioned issues, this disclosure proposes a method and apparatus for kinematic calculation of flexible booms based on a graph network simulator, as well as a computing system and a computer-readable storage medium.

[0006] According to one aspect of this disclosure, a kinematics calculation method for a flexible boom based on a graph network simulator is provided. The graph network simulator includes an encoder, a processor, and a decoder. The kinematics calculation method for the flexible boom includes: receiving a flexible boom motion trajectory sequence of the current frame through the encoder, and extracting and encoding node information and edge information of the flexible boom motion trajectory sequence to obtain node features and edge features, wherein the node information of the flexible boom motion trajectory sequence includes the coordinate position, node type, and joint angle value of each node; iteratively updating the node features and edge features through the processor to obtain updated node features; decoding the updated node features into node acceleration information through the decoder, and predicting the node position of the next frame based on the node information of the current frame and the acceleration information.

[0007] Optionally, the steps of extracting and encoding node information and edge information of the flexible boom motion trajectory sequence to obtain node features and edge features include: for each node, determining the distance from the node to the boom boundary based on the coordinate position, determining the node's velocity through the coordinate difference between adjacent frames, mapping the node type to a type vector, and merging the distance, velocity, type vector, and joint angle value to constitute node motion information; obtaining the ID of each node, and mapping the ID to a physical feature vector through a pre-trained physical parameter embedding layer to constitute node physical information; constructing edges of the graph based on the spatial proximity relationship between nodes, determining the node connection relationship, and calculating the vector and magnitude of each edge to obtain edge information; for each node, merging the node motion information and node physical information and inputting them into a multilayer perceptron to encode the merging result as node features; and inputting the edge information into another multilayer perceptron to encode the edge information as edge features.

[0008] Optionally, geometric constraint loss and physical constraint loss are introduced during the training process of the graph network simulator. The geometric constraint loss includes node position prediction loss and distance preservation loss between adjacent nodes. The physical constraint loss includes node bending moment consistency loss. The physical constraint loss is calculated based on a pre-trained physical parameter embedding layer. The physical parameter embedding layer learns the mapping relationship between node ID and physical parameters through sample data, and uses the mapped node physical parameters as fixed values ​​when calculating the physical constraint loss.

[0009] Optionally, the kinematics solution method for the flexible boom further includes: pre-training the graph network simulator using simulation data with randomized physical parameters generated by a simulation physics engine; enabling the physical parameter embedding layer and simultaneously combining output loss, geometric constraint loss, and physical constraint loss, and using real sample data to adjust the parameters of the pre-trained graph network simulator, wherein the physical constraint loss of the real sample data is calculated separately to suppress parameter updates of the physical parameter embedding layer during the parameter adjustment process.

[0010] Optionally, the kinematics calculation method for the flexible boom further includes: generating an initial node trajectory based on an ideal rigid link using a virtual trajectory frame generator and joint angle values; predicting node information for the next frame based on the initial node trajectory using the graph network simulator; and using the predicted end node coordinates as the positive kinematics calculation pose of the flexible boom in response to the predicted end node velocity being lower than a predetermined threshold.

[0011] Optionally, the flexible boom kinematics calculation method further includes: determining an initialized virtual target pose based on the desired end-effector target pose; calculating the virtual target pose using the standard inverse kinematics algorithm of a rigid boom to determine candidate joint angle values; determining the actual end-effector pose of the flexible boom under the candidate joint angle values ​​based on the candidate joint angle values; calculating the difference between the actual end-effector pose and the desired end-effector target pose; updating the virtual target pose in response to the difference being greater than a predetermined difference threshold, and returning to the step of calculating the virtual target pose using the standard inverse kinematics algorithm of a rigid boom; and determining the candidate joint angle value corresponding to the difference as the inverse kinematics calculation result of the flexible boom in response to the difference being less than or equal to the predetermined difference threshold.

[0012] Optionally, the physical parameter embedding layer calculates the stiffness parameters of each node by the relationship between the bending moment and the stiffness of each node, and maps the stiffness parameters to the node ID. The stiffness parameters include the bending stiffness and equivalent mass of the node.

[0013] According to another aspect of this disclosure, a kinematics calculation device for a flexible boom based on a graph network simulator is provided. The graph network simulator includes an encoder, a processor, and a decoder. The kinematics calculation device for the flexible boom includes: a feature acquisition unit configured to receive a flexible boom motion trajectory sequence of the current frame through the encoder, and extract and encode node information and edge information of the flexible boom motion trajectory sequence to obtain node features and edge features, wherein the node information of the flexible boom motion trajectory sequence includes the coordinate position, node type, and joint angle value of each node; a feature update unit configured to iteratively update the node features and edge features through the processor to obtain updated node features; and a prediction unit configured to decode the updated node features into node acceleration information through the decoder, and predict the node position of the next frame based on the node information of the current frame and the acceleration information.

[0014] Optionally, the feature acquisition unit is further configured to: for each node, determine the distance from the node to the boom boundary based on the coordinate position, determine the node's velocity through the coordinate difference between adjacent frames, map the node type to a type vector, and merge the distance, velocity, type vector, and joint angle value to form node motion information; obtain the ID of each node, and map the ID to a physical feature vector through a pre-trained physical parameter embedding layer to form node physical information; construct the edges of the graph based on the spatial proximity relationship between nodes, determine the node connection relationship, and calculate the vector and magnitude of each edge to obtain edge information; for each node, merge the node motion information and node physical information and input them into a multilayer perceptron to encode the merging result as node features; input the edge information into another multilayer perceptron to encode the edge information as edge features.

[0015] Optionally, geometric constraint loss and physical constraint loss are introduced during the training process of the graph network simulator. The geometric constraint loss includes node position prediction loss and distance preservation loss between adjacent nodes. The physical constraint loss includes node bending moment consistency loss. The physical constraint loss is calculated based on a pre-trained physical parameter embedding layer. The physical parameter embedding layer learns the mapping relationship between node ID and physical parameters through sample data, and uses the mapped node physical parameters as fixed values ​​when calculating the physical constraint loss.

[0016] Optionally, the prediction unit is further configured to: pre-train the graph network simulator using simulation data with randomized physical parameters generated by the simulation physics engine; enable the physical parameter embedding layer and simultaneously combine the output loss, geometric constraint loss, and physical constraint loss to adjust the parameters of the pre-trained graph network simulator using real sample data, wherein the physical constraint loss of the real sample data is calculated separately to suppress parameter updates of the physical parameter embedding layer during the parameter adjustment process.

[0017] Optionally, the prediction unit is further configured to: generate an initial node trajectory based on an ideal rigid link using a virtual trajectory frame generator and joint angle values; predict node information for the next frame using the graph network simulator based on the initial node trajectory; and, in response to the predicted end node velocity being lower than a predetermined threshold, use the predicted end node coordinates as the positive kinematics pose of the flexible boom.

[0018] Optionally, the prediction unit is further configured to: determine an initialized virtual target pose based on the desired end-effector pose; solve the virtual target pose using the standard inverse kinematics algorithm of the rigid boom to determine candidate joint angle values; determine the actual end-effector pose of the flexible boom under the candidate joint angle values ​​based on the candidate joint angle values; calculate the difference between the actual end-effector pose and the desired end-effector pose; update the virtual target pose in response to the difference being greater than a predetermined difference threshold, and return to the step of solving the virtual target pose using the standard inverse kinematics algorithm of the rigid boom; and determine the candidate joint angle value corresponding to the difference as the inverse kinematics solution result of the flexible boom in response to the difference being less than or equal to the predetermined difference threshold.

[0019] Optionally, the physical parameter embedding layer calculates the stiffness parameters of each node by the relationship between the bending moment and the stiffness of each node, and maps the stiffness parameters to the node ID. The stiffness parameters include the bending stiffness and equivalent mass of the node.

[0020] According to another aspect of this disclosure, a computing system is provided that includes at least one computing device and at least one storage device for storing instructions, wherein, when the instructions are executed by the at least one computing device, they cause the at least one computing device to perform the flexible boom kinematics calculation method based on a graph network simulator as described above.

[0021] According to another aspect of this disclosure, a computer-readable storage medium is provided for storing instructions, wherein when the instructions are executed by at least one computing device, the at least one computing device causes the at least one computing device to perform the flexible boom kinematics calculation method based on a graph network simulator as described above.

[0022] By adopting this disclosure, the topological relationships of flexible bodies can be effectively modeled through graph networks, and the concept of Physical Information Neural Network (PINN) is introduced to incorporate geometric and physical laws as hard constraints into the training, significantly improving the accuracy and physical consistency of motion prediction. The proposed small-sample virtual-real transfer training strategy utilizes a large amount of readily available simulation data for pre-training, requiring only a small amount of real data for fine-tuning, greatly reducing the dependence on expensive real datasets. Based on the trained simulator, a forward / inverse kinematics solution method suitable for ultra-long flexible booms is proposed. Forward kinematics can accurately simulate the static pose after flexible deformation. Inverse kinematics effectively solves the inherent complex nonlinear problems of inverse kinematics of flexible bodies through an iterative strategy of "virtual-real mapping". Furthermore, the precise analytical model does not depend on a specific boom and is applicable to ultra-long flexible booms of different configurations and materials, exhibiting good generalization ability. Attached Figure Description

[0023] The above and / or other objects and advantages of this disclosure will become clearer from the following description of embodiments in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating a kinematics calculation method for a flexible boom based on a graph network simulator according to an exemplary embodiment of the present disclosure; Figure 2 This is a schematic diagram illustrating the physical information constraints of a graph network simulator according to an embodiment of the present disclosure; Figure 3 The training process of a virtual-to-real migration model based on small sample data according to an embodiment of the present disclosure is illustrated. Figure 4 The flow chart for calculating the forward kinematics according to an embodiment of the present disclosure is shown. Figure 5 The inverse kinematics solution process according to an embodiment of the present disclosure is shown; Figure 6 A block diagram of a flexible boom kinematics calculation apparatus based on a graph network simulator according to an exemplary embodiment of the present disclosure is shown. Figure 7 This is a block diagram illustrating a computing system including at least one computing device and at least one storage device of storage instructions according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0024] The following description, in conjunction with the accompanying drawings, provides specific embodiments to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, upon understanding this disclosure, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be altered as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0025] To address the issues of insufficient accuracy and slow solution speed of traditional kinematic models for ultra-long flexible booms due to their high flexibility and multi-physics coupling in actual operations, this disclosure proposes a kinematics solution method for flexible booms based on a graph network simulator. This method discretizes the continuous flexible motion of the ultra-long flexible boom as a particle system with interacting relationships, constructs a graph structure representation of the flexible boom, and learns the spatiotemporal evolution laws between particles based on a graph network simulator, achieving high-precision modeling and prediction of the kinematic behavior of the flexible boom. This method designs a virtual-to-real transfer training process under small sample conditions. It utilizes a large amount of simulation data with uncertain model parameters for pre-training of the graph network model, and then performs transfer fine-tuning using a small amount of real data, effectively reducing the dependence on the amount of real data. Furthermore, it introduces a Physics-Informed Neural Network (PINN) to learn the explicit and implicit physical parameters of the flexible boom from the sample data, and applies physical consistency constraints to the prediction results, improving the model's prediction accuracy. A framework for solving the forward and inverse kinematics of an ultra-long flexible boom based on a graph network simulator is constructed, enabling rapid and stable solutions for the end-effector pose of the flexible boom. This disclosure significantly improves computational efficiency while maintaining high solution accuracy, and is suitable for real-time control and intelligent operation of ultra-long flexible booms (hundred-meter level). It has significant engineering application value for improving the safety, stability, and operational accuracy of ultra-long flexible booms.

[0026] Figure 1This is a flowchart illustrating a kinematics calculation method for a flexible boom based on a graph network simulator according to an exemplary embodiment of the present disclosure. Based on a graph network simulator, geometric information such as link length and sampling point position is introduced for constraints. Simultaneously, implicit expressions of physical parameters are learned through real samples. A physical information neural network is used to apply physical consistency constraints to physical information such as link curvature and motion damping, resulting in a graph network simulator for high-precision motion prediction of the flexible boom. This graph network simulator includes an encoder, a processor, and a decoder. The encoder receives the motion trajectory sequence of the flexible boom, extracts node and edge information, encodes the node features and edge features respectively through two multilayer perceptrons, and sends the results to the processor. The processor receives the node and edge features from the encoder, updates the node features through a message passing network, and sends them to the decoder. The decoder receives the node features from the processor, decodes them through a multilayer perceptron, obtains the predicted node motion information for the next frame, and outputs it.

[0027] like Figure 1 As shown, in step S101, the flexible boom motion trajectory sequence of the current frame is received by the encoder, and the node information and edge information of the flexible boom motion trajectory sequence are extracted and encoded to obtain node features and edge features. The node information of the flexible boom motion trajectory sequence includes the coordinate position, node type and joint angle value of each node.

[0028] In the example, for each node, the distance from the node to the boom boundary is determined based on its coordinate position, the node's velocity is determined by the coordinate difference between adjacent frames, the node type is mapped to a type vector, and the distance, velocity, type vector, and joint angle value are merged to form node motion information; the ID of each node is obtained, and the ID is mapped to a physical feature vector through a pre-trained physical parameter embedding layer to form node physical information; the edges of the graph are constructed based on the spatial proximity relationship between nodes, the node connection relationship is determined, and the vector and magnitude of each edge are calculated to obtain edge information; for each node, the node motion information and node physical information are merged and input into a multilayer perceptron to encode the merged result as node features; the edge information is input into another multilayer perceptron to encode the edge information as edge features. For example, the encoder receives an N-frame sequence of flexible boom motion trajectories. The trajectory contains the coordinates of each sampling point, node type label, and joint angle value. It calculates the distance from the sampling point to the boundary using the coordinates; calculates the velocity information of the sampling point using coordinate difference; maps the node type to a multi-dimensional vector; merges the boundary distance, node velocity, type vector, and joint angle to obtain node motion information; obtains the ID of each sampling point in the trajectory, calculates the physical feature vector corresponding to each ID through an embedding layer, and obtains the node physical information; obtains the node connection relationship through the distance between sampling points in the trajectory, constructs edges in the graph, calculates the edge vector based on the head and tail of the edge, calculates the edge magnitude, and merges them to obtain edge information; merges the node physical information and node motion information, calculates node features through a multilayer perceptron; and calculates edge features through another multilayer perceptron for edge connection information. In this example, geometric constraint loss and physical constraint loss are introduced during the training of the graph network simulator. Geometric constraint loss includes node position prediction loss and distance preservation loss between adjacent nodes. Physical constraint loss includes node bending moment consistency loss. The physical constraint loss is calculated based on a pre-trained physical parameter embedding layer. This layer learns the mapping relationship between node IDs and physical parameters through sample data and uses the mapped node physical parameters as fixed values ​​when calculating the physical constraint loss. In this example, the physical parameter embedding layer calculates the stiffness parameters of each node based on the relationship between the bending moment and stiffness of each node, and maps these stiffness parameters to the node ID. The stiffness parameters of each node include the node's bending stiffness and equivalent mass.

[0029] Furthermore, in the example, the graph network simulator is pre-trained using simulation data with randomized physical parameters generated by a simulation physics engine. A physical parameter embedding layer is enabled, and output loss, geometric constraint loss, and physical constraint loss are combined simultaneously. Real sample data is used to adjust the parameters of the pre-trained graph network simulator. Specifically, the physical constraint loss of the real sample data is calculated separately to suppress parameter updates in the physical parameter embedding layer during parameter adjustment. The graph network simulator predicts the motion of each node by predicting acceleration to obtain the node's position in the next frame. The output loss, geometric constraint loss, and physical constraint loss are losses calculated based on the node's acceleration (force) and position, ensuring that the predicted node's geometric, physical, and acceleration characteristics match the node characteristics of the real data. In the initial graph network simulator, each node only has feature information such as velocity, position, and type encoding. The physical embedding layer estimates a physical parameter for each node based on its ID, representing information such as stiffness at that location. This additional information is also input into the graph network simulator, which can better guide the simulator in training and predicting the motion of each point. The output loss can be based on the difference between the model's predicted future state of the node (e.g., position, velocity) and the actual future state. During the training of the graph network simulator, the geometric loss of the prediction results is calculated to ensure that the simulator's prediction results meet geometric constraints. The geometric loss can include the loss of node position and the loss of distance between adjacent nodes. Furthermore, the physical loss of the prediction results can be calculated to ensure that the simulator's prediction results meet physical constraints. The physical loss can include the loss of bending moment experienced by the node. The calculation method for physical constraints is consistent and involves two steps during training: First, physical parameter estimation: estimating the physical parameters (e.g., stiffness parameters) for each node from real samples; second, physical information constraints: after obtaining a rough estimate of the physical parameters for each node, evaluating whether the predicted node motion result matches the physical parameters learned from the samples, and adjusting the network accordingly. The results of the first step can guide the training in the second step, but the network may change its embedding layers during learning, affecting the physical parameters estimated in the first step. Therefore, during training, it is necessary to calculate the loss for both parameter estimation and information constraints simultaneously. The first step "holds back" the physical parameters, preventing them from changing significantly, and the second step uses these relatively stable parameters as a benchmark to guide node motion.

[0030] This disclosure presents an innovative kinematics solution method for flexible booms based on a graph network simulator. It innovatively combines a small-sample data virtual-to-real-data transfer model training process, pre-training the graph network model using a large amount of virtual data with model uncertainty, and then adding a small amount of real data for fine-tuning and physical constraint optimization. Furthermore, to improve prediction accuracy, physical information constraints are introduced into the graph network simulator. Implicit expressions of physical parameters are learned through finite samples, and a physical information neural network is used to strictly constrain geometric information such as link length and node position, as well as physical properties such as material bending deformation and elastic damping.

[0031] In step S102, the processor iteratively updates the node features and edge features to obtain updated node features. For example, the processor receives node features and edge features from the encoder, passes messages through M message passing networks, aggregates messages after each transmission, and updates the node features.

[0032] In step S103, the updated node features are decoded into node acceleration information by the decoder, and the node position of the next frame is predicted based on the node information of the current frame and the acceleration information. For example, the decoder receives node features from the processor, decodes them into node acceleration information by a multilayer perceptron, and calculates and outputs the predicted node position of the (N+1)th frame based on the calculated node position and node velocity of the Nth frame.

[0033] Furthermore, in the example, an initial node trajectory based on an ideal rigid link is generated using a virtual trajectory frame generator, based on joint angle values. A graph network simulator is then used to predict the node information for the next frame based on the initial node trajectory. In response to the predicted end-effector velocity being below a predetermined threshold, the predicted end-effector coordinates are used as the pose for the forward kinematics calculation of the flexible boom. In the forward kinematics calculation, a forward kinematics process based on a graph network simulator is proposed, predicting the dynamic deformation process of the robotic arm until it reaches a stable state by inputting joint motion commands.

[0034] In the example, based on the desired end-effector pose, an initial virtual target pose is determined; the virtual target pose is solved using the standard inverse kinematics algorithm of the rigid boom to determine candidate joint angle values; based on the candidate joint angle values, the actual end-effector pose of the flexible boom under the candidate joint angle values ​​is determined; the difference between the actual end-effector pose and the desired end-effector pose is calculated; in response to the difference being greater than a predetermined difference threshold, the virtual target pose is updated, and the step of solving the virtual target pose using the standard inverse kinematics algorithm of the rigid boom is returned; in response to the difference being less than or equal to the predetermined difference threshold, the candidate joint angle value corresponding to the difference is determined as the inverse kinematics solution result of the flexible boom. In the inverse kinematics solution, the joint output is iteratively adjusted in combination with the forward kinematics results to achieve high-precision compensation for bending deformation.

[0035] The kinematics calculation method for flexible booms based on graph network simulators according to exemplary embodiments of this disclosure aims to solve the kinematic prediction and control challenges of ultra-long flexible booms in complex operations. By combining graph network technology with physical constraint optimization, a highly efficient, robust, and accurate kinematics calculation method is achieved. Furthermore, this disclosure can be applied not only to ultra-long flexible booms but also to other complex flexible mechanical systems.

[0036] Figure 2 This is a schematic diagram illustrating the physical information constraints of a graph network simulator infrastructure according to an embodiment of the present disclosure. Figure 3 The training process for a virtual-to-real migration model using small sample data according to an embodiment of this disclosure is illustrated. Figure 4 The flow chart for calculating the forward kinematics according to an embodiment of the present disclosure is shown. Figure 5 The inverse kinematics solution process according to an embodiment of the present disclosure is shown.

[0037] like Figure 2 As shown, this disclosure proposes a graph network simulator that incorporates physical information constraints. Based on the graph network simulator, geometric information such as link length and sampling point position are introduced for constraints. Simultaneously, implicit expressions of physical parameters are learned through real samples. A physical information neural network is used to constrain physical information such as link curvature and motion damping, training a graph network simulator with higher prediction accuracy to achieve high-precision motion prediction of ultra-long flexible booms. The simulator includes an encoder, a processor, and a decoder. The encoder receives the motion trajectory sequence of the flexible boom, extracts node and edge information, and encodes the node and edge features using two multilayer perceptrons respectively, sending the results to the processor. The processor receives the node and edge features from the encoder, updates the node features through a message passing network, and sends them to the decoder. The decoder receives the node features from the processor, decodes them using a multilayer perceptron, obtains the predicted node motion information for the next frame, and outputs it.

[0038] In the example, the prediction steps of the graph network simulator include the following: Step 1: The encoder receives an N-frame sequence of flexible boom motion trajectories. The trajectory contains the coordinate position, node type label, and joint angle value of each sampling point. The distance from the sampling point to the boundary is calculated using the coordinate position; the velocity information of the sampling point is calculated using coordinate difference; the node type is mapped to a multi-dimensional vector; and the boundary distance, node velocity, type vector, and joint angle are merged to obtain the node motion information. Step 2: The ID of each sampling point in the trajectory is obtained. The physical feature vector corresponding to each ID is calculated through an embedding layer to obtain the node physical information. Step 3: The connection relationship between nodes is obtained through the distance between sampling points in the trajectory. Edges are constructed in the graph. The edge vector is calculated based on the head and tail of the edge, and the magnitude of the edge is calculated. The edge information is then merged to obtain the edge information. Step 4: The node physical information and node motion information are merged. The node features are calculated using a multilayer perceptron; the edge connection information is calculated using another multilayer perceptron to obtain the edge features. Step 5: The processor receives the node features and edge features from the encoder. Messages are passed through M message passing networks. After each pass, the messages are aggregated, and the node features are updated. Step 6: The decoder receives the node features from the processor, decodes them into node acceleration information through a multilayer perceptron, and calculates and outputs the node position prediction for the N+1th frame based on the node position and node velocity calculated in Step 1.

[0039] This disclosure proposes a geometric information constraint method based on the aforementioned graph network simulator. During the training process of the graph network simulator, the geometric loss of the prediction results is calculated to ensure that the simulator's prediction results meet the geometric constraints. The geometric loss may include the loss of node position and the loss of distance between adjacent nodes.

[0040] Specifically, the sampling points on the flexible links of the robotic arm conform to geometric constraints, and the graph network simulator ultimately calculates the particle positions. The sampling point locations should correspond to those of the actual sample data. Consistent, and the distance between adjacent points in the simulation results The distance between two adjacent points in the actual sample data should be... If consistent, then the geometric constraint loss of the graph network simulator... for:

[0041]

[0042]

[0043] This disclosure proposes a physical information constraint method based on the aforementioned graph network simulator. Before training the simulator, an embedding layer for physical parameters is pre-trained. The stiffness parameters of each node are calculated by analyzing the relationship between the bending moment and stiffness of each node in the sample, and a mapping relationship is established between the stiffness parameters and the node ID. Specifically: for points in the trajectory sample... The following limitations apply:

[0044]

[0045] Calculated points curvature for:

[0046] point bending moment for:

[0047] In the formula, For point The bending stiffness of the section about the neutral axis.

[0048] point static moment With inertial torque They are respectively:

[0049]

[0050] In the formula, This represents the total number of sampling points. For point Equivalent quality at the location, It is the acceleration due to gravity. For point The acceleration at that point.

[0051] point bending moment It should be subjected to static torque With inertial torque Composition, namely:

[0052]

[0053] Then, when training the embedding layer, the loss function is:

[0054] The trained embedding layer maps the bending stiffness and equivalent mass of the sampling points based on their IDs.

[0055] During the training of the graph network simulator, the physical loss of the prediction results is calculated to ensure that the simulator's predictions meet physical constraints. The physical loss may include the loss due to the bending moment experienced by the nodes. The calculation formula for this loss is consistent with the loss calculation formula for training the embedding layer, but the physical parameters of the nodes are changed during the calculation. and These are fixed values ​​obtained through embedding layer mapping.

[0056] This disclosure proposes a graph network simulator model training method with virtual-real transfer using small sample data. First, it pre-trains with a large amount of simulation data to learn the general motion pattern of the robotic arm. Then, it uses a small amount of real sample data to correct the physical features and complete the final training. High-precision simulation that conforms to the real physical world can be achieved using only a small amount of real data.

[0057] The pre-training process includes: using a simulation physics engine to generate simulation data with randomized physical parameters as data for the pre-training stage; not using a physical parameter embedding layer to map physical information; and only inputting node and edge information to train the graph network simulator model. During training, only the decoder output results are processed. Acceleration with real sample data Calculate the loss and output the loss. for:

[0058] The final training process includes: using a small amount of real sampled data as the data for the final training phase, and calculating the output loss during training. Geometric constraint loss Simultaneously, considering its physical constraints, calculate the physical loss of its prediction results. ; and by calculating the physical loss of real samples separately. Suppress the updating of physical parameters of each node in the embedding layer. The total loss in the final training phase is:

[0059] According to the embodiments of this disclosure, the forward kinematics calculation method for an ultra-long flexible boom based on a graph network simulator predicts the end-effector pose of the ultra-long flexible boom. The forward kinematics calculation process is as follows: Figure 4As shown. The forward kinematics solution algorithm requires the joint angles of the robotic arm as input and outputs the end-effector pose. The graph network simulator takes the sampling point trajectory as input and outputs the predicted data for the next frame. Therefore, the forward kinematics solution method based on the graph network simulator needs to convert the joint angle input into sampling point trajectory data and extract the final pose reached by the end-effector from the predicted sampling point trajectory data. The forward kinematics solution method based on the graph network simulator uses an ideal rigid link as a reference, generates a completely rigid initial pose based on the input joint angles, and generates the initial sampling point trajectory based on the forward kinematics process of the rigid robotic arm through a virtual trajectory frame generator. After supplementing the sampling point type and joint angle, the generated initial trajectory is simulated and calculated by the graph network simulator, and the end-effector velocity of each frame is calculated. When the end-effector velocity is lower than a certain threshold, the robotic arm in the simulator is considered to have basically reached a stationary state, and the current sampling point coordinates of the robotic arm are output. The end-effector sampling point coordinate information is extracted as the forward kinematics solution result.

[0060] According to the embodiments of this disclosure, the inverse kinematics calculation method for an ultra-long flexible boom based on graph network simulator forward kinematics calculation obtains a relatively accurate joint angle output by adjusting the virtual target pose and gradually correcting it. The inverse kinematics calculation process is as follows: Figure 6 As shown, this method is a "virtual-real mapping" method. Its purpose is to solve for a virtual target pose of a rigid arm, and then to obtain a joint angle in this pose that enables the flexible arm to move to a specified pose. When the virtual target pose is almost no longer updated, it indicates that the difference between the estimated end-effector pose and the target pose is small. At this point, the calculated joint angle value allows the flexible arm to essentially reach the target pose. Specifically, the target pose is used as the initial value of the virtual target pose. First, a set of joint angles corresponding to this virtual target pose is calculated using the standard inverse kinematics of the rigid arm. This set of joint angles, on an ideal rigid linkage manipulator, enables the end-effector to reach the virtual target pose. Next, using this set of joint angles as the output joint angle values, forward kinematics are calculated to obtain a set of end-effector poses. A certain error exists between this end-effector pose and the target pose. This error is used to update and correct the virtual pose, making the actual pose corresponding to the joint angle value calculated from the virtual target pose close to the target pose. Each update requires calculating the error between the end pose estimated by forward kinematics and the target pose to evaluate the current update state. When the pose error is less than or equal to a predetermined threshold, the joint angle value at this time is output as the result of the inverse kinematics solution.

[0061] Figure 6 A block diagram of a flexible boom kinematics calculation apparatus based on a graph network simulator according to an exemplary embodiment of the present disclosure is shown. The graph network simulator includes an encoder, a processor, and a decoder.

[0062] like Figure 6 As shown, the flexible boom kinematics calculation device 600 based on a graph network simulator according to an exemplary embodiment of the present disclosure includes: a feature acquisition unit 601, configured to receive the flexible boom motion trajectory sequence of the current frame through an encoder, and extract and encode the node information and edge information of the flexible boom motion trajectory sequence to obtain node features and edge features, wherein the node information of the flexible boom motion trajectory sequence includes the coordinate position, node type and joint angle value of each node; a feature update unit 602, configured to iteratively update the node features and edge features through a processor to obtain updated node features; and a prediction unit 603, configured to decode the updated node features into node acceleration information through a decoder, and predict the node position of the next frame based on the node information and acceleration information of the current frame.

[0063] In the example, the feature acquisition unit 601 is further configured to: for each node, determine the distance from the node to the boom boundary based on the coordinate position, determine the node's velocity through the coordinate difference between adjacent frames, map the node type to a type vector, and merge the distance, velocity, type vector, and joint angle value to form node motion information; obtain the ID of each node, and map the ID to a physical feature vector through a pre-trained physical parameter embedding layer to form node physical information; construct the edges of the graph based on the spatial proximity relationship between nodes, determine the node connection relationship, and calculate the vector and magnitude of each edge to obtain edge information; for each node, merge the node motion information and node physical information and input them into a multilayer perceptron to encode the merged result as node features; input the edge information into another multilayer perceptron to encode the edge information as edge features.

[0064] In the example, geometric constraint loss and physical constraint loss are introduced during the training of the graph network simulator. The geometric constraint loss includes node position prediction loss and distance preservation loss between adjacent nodes. The physical constraint loss includes node bending moment consistency loss. The physical constraint loss is calculated based on a pre-trained physical parameter embedding layer. The physical parameter embedding layer learns the mapping relationship between node ID and physical parameters through sample data and uses the mapped node physical parameters as fixed values ​​when calculating the physical constraint loss.

[0065] In the example, prediction unit 603 is further configured to: pre-train a graph network simulator using simulation data with randomized physical parameters generated by a simulation physics engine; enable a physical parameter embedding layer and simultaneously combine output loss, geometric constraint loss, and physical constraint loss to adjust the parameters of the pre-trained graph network simulator using real sample data, wherein the physical constraint loss of the real sample data is calculated separately to suppress parameter updates of the physical parameter embedding layer during the parameter adjustment process.

[0066] In the example, the prediction unit 603 is further configured to: generate an initial node trajectory based on an ideal rigid link using a virtual trajectory frame generator and joint angle values; predict node information for the next frame using a graph network simulator based on the initial node trajectory; and, in response to the predicted end node velocity being lower than a predetermined threshold, use the predicted end node coordinates as the positive kinematics pose of the flexible boom.

[0067] In the example, the prediction unit 603 is further configured to: determine an initialized virtual target pose based on the desired end-effector pose; solve the virtual target pose using the standard inverse kinematics algorithm of the rigid boom to determine candidate joint angle values; determine the actual end-effector pose of the flexible boom under the candidate joint angle values ​​based on the candidate joint angle values; calculate the difference between the actual end-effector pose and the desired end-effector pose; update the virtual target pose in response to the difference being greater than a predetermined difference threshold, and return to the step of solving the virtual target pose using the standard inverse kinematics algorithm of the rigid boom; and determine the candidate joint angle value corresponding to the difference as the inverse kinematics solution result of the flexible boom in response to the difference being less than or equal to the predetermined difference threshold.

[0068] In the example, the physical parameter embedding layer calculates the stiffness parameters of each node by the relationship between the bending moment and the stiffness of each node, and maps the stiffness parameters to the node ID. The stiffness parameters include the bending stiffness and equivalent mass of the node.

[0069] The above combination Figures 1 to 5 The specific operations shown are respectively by Figure 6 The corresponding unit in the flexible boom kinematics calculation device 600 based on the graph network simulator shown here will execute the calculation. The specific operational details will not be elaborated here.

[0070] Figure 7 This is a block diagram illustrating a computing system including at least one computing device and at least one storage device of storage instructions according to an exemplary embodiment of the present disclosure.

[0071] like Figure 7 As shown, the computing system 700 provided according to an exemplary embodiment of the present invention includes a computing device 701 and a storage device 702. The storage device 702 stores computer-executable instructions. When the computer-executable instructions are executed by the computing device 701, the flexible boom kinematics calculation method based on a graph network simulator described in any of the foregoing embodiments is executed.

[0072] The computing device 701 can be deployed in a server or client, or on a node device in a distributed network environment. Furthermore, the computing device 701 can be a PC, tablet, personal digital assistant, smartphone, web application, or other device capable of executing the aforementioned set of instructions. Here, the computing device is not necessarily a single computing device; it can be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. The computing device can also be part of an integrated control system or system manager, or can be configured to interconnect locally or remotely (e.g., via wireless transmission) through an interface. In the computing device, the processor includes a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor also includes analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0073] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores instructions, which, when executed by at least one computing device, cause the at least one computing device to perform the kinematics calculation of a flexible boom based on a graph network simulator as described in any of the preceding embodiments. The computer-readable storage medium includes magnetic media such as floppy disks and magnetic tapes, optical media (including optical disc (CD) ROMs and DVD ROMs), magneto-optical media such as flexible optical discs, hardware devices such as ROMs and RAMs designed for storing and executing program commands, and flash memory. The instructions may include language code executable by a computer using an interpreter and machine language code generated by a compiler.

[0074] By adopting this disclosure, the topological relationships of flexible bodies can be effectively modeled through graph networks, and the concept of physical information neural networks is introduced, incorporating geometric and physical laws as hard constraints into the training, which significantly improves the accuracy and physical consistency of motion prediction. The proposed small-sample virtual-real transfer training strategy utilizes a large amount of readily available simulation data for pre-training, requiring only a small amount of real data for fine-tuning, greatly reducing the dependence on expensive real datasets. Based on the trained simulator, a forward / inverse kinematics solution method suitable for ultra-long flexible booms is proposed. Forward kinematics can accurately simulate the static pose after flexible deformation. Inverse kinematics effectively solves the inherent complex nonlinear problem of inverse kinematics of flexible bodies through the iterative strategy of "virtual-real mapping". Furthermore, it does not depend on a specific boom's precise analytical model, is applicable to ultra-long flexible booms of different configurations and materials, and has good generalization ability.

[0075] The processes, methods, or algorithms disclosed herein can be transmitted to, or implemented by, a processing device, controller, or computer, which may include any existing programmable electronic control unit or a dedicated electronic control unit. Similarly, the processes, methods, or algorithms can be stored in various forms as data and instructions executable by a controller or computer, including but not limited to information permanently stored on non-writable storage media (such as ROM devices) and information variablely stored on writable storage media (such as floppy disks, magnetic tapes, CDs, RAM devices, and other magnetic and optical media). The processes, methods, or algorithms can also be implemented in a software executable object. Optionally, the processes, methods, or algorithms can be implemented wholly or partially using suitable hardware components (such as ASICs, FPGAs, state machines, controllers, or other hardware components or devices) or a combination of hardware components, software components, and firmware components.

[0076] Although this disclosure includes specific examples, it will be apparent to those skilled in the art that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered merely for descriptive purposes and not for limiting purposes. The description of features or aspects in each example is to be considered applicable to similar features or aspects in other examples. Suitable results may be obtained if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner and / or if components in the described system, architecture, apparatus, or circuit are replaced or supplemented with other components or their equivalents. Therefore, the scope of this disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents shall be construed as included in this disclosure.

Claims

1. A method for kinematic calculation of a flexible boom based on a graph network simulator, characterized in that, The graph network simulator includes an encoder, a processor, and a decoder, and the kinematics calculation method for the flexible boom includes: The encoder receives the flexible boom motion trajectory sequence of the current frame and extracts and encodes the node information and edge information of the flexible boom motion trajectory sequence to obtain node features and edge features. The node information of the flexible boom motion trajectory sequence includes the coordinate position, node type and joint angle value of each node. The processor iteratively updates the node features and edge features to obtain updated node features. The decoder decodes the updated node features into node acceleration information, and predicts the node position of the next frame based on the node information of the current frame and the acceleration information.

2. The kinematics calculation method for a flexible boom according to claim 1, characterized in that, The steps for extracting and encoding node and edge information from the flexible boom motion trajectory sequence to obtain node and edge features include: For each node, the distance from the node to the boom boundary is determined based on the coordinate position, the node velocity is determined by the coordinate difference between adjacent frames, the node type is mapped to a type vector, and the distance, velocity, type vector and joint angle value are merged to form the node motion information; Obtain the ID of each node, and map the ID to a physical feature vector through a pre-trained physical parameter embedding layer to form the node's physical information; The edges of the graph are constructed based on the spatial proximity relationship between nodes, the node connection relationship is determined, and the vector and magnitude of each edge are calculated to obtain edge information. For each node, the node motion information and node physical information are merged and input into a multilayer perceptron to encode the merged result as node features; The edge information is input into another multilayer perceptron to encode the edge information into edge features.

3. The kinematics calculation method for a flexible boom according to claim 1, characterized in that, In the training process of the graph network simulator, geometric constraint loss and physical constraint loss are introduced. The geometric constraint loss includes node position prediction loss and distance preservation loss between adjacent nodes. The physical constraint loss includes node bending moment consistency loss. The physical constraint loss is calculated based on a pre-trained physical parameter embedding layer. The physical parameter embedding layer learns the mapping relationship between node ID and physical parameters through sample data, and uses the mapped node physical parameters as fixed values ​​when calculating the physical constraint loss.

4. The kinematics calculation method for a flexible boom according to claim 3, characterized in that, The kinematics calculation method for the flexible boom also includes: The graph network simulator is pre-trained using simulation data with randomized physical parameters generated by a simulation physics engine. The physical parameter embedding layer is enabled and combined with the output loss, geometric constraint loss and physical constraint loss. The parameters of the pre-trained graph network simulator are adjusted using real sample data. The physical constraint loss of the real sample data is calculated separately to suppress parameter updates of the physical parameter embedding layer during the parameter adjustment process.

5. The kinematics calculation method for a flexible boom according to claim 1, characterized in that, The kinematics calculation method for the flexible boom also includes: An initial nodal trajectory is generated based on an ideal rigid link using a virtual trajectory frame generator and joint angle values. The graph network simulator predicts the node information of the next frame based on the initial node trajectory. In response to the predicted end-node velocity being lower than a predetermined threshold, the predicted end-node coordinates are used as the positive kinematics pose of the flexible boom.

6. The kinematics calculation method for a flexible boom according to claim 5, characterized in that, The kinematics calculation method for the flexible boom also includes: Based on the desired end-target pose, determine the initial virtual target pose; The pose of the virtual target is calculated using the standard inverse kinematics algorithm of the rigid boom to determine candidate values ​​for joint angles; Based on the candidate joint angle values, the actual end pose of the flexible boom under the candidate joint angle values ​​is determined; Calculate the difference between the actual end-effector pose and the desired end-effector target pose; In response to the difference being greater than a predetermined difference threshold, the virtual target pose is updated, and the step of solving the virtual target pose using the standard inverse kinematics algorithm of the rigid boom is returned; In response to the difference being less than or equal to a predetermined difference threshold, the candidate value of the joint angle corresponding to the difference is determined as the inverse kinematics solution result of the flexible boom.

7. The kinematics calculation method for a flexible boom according to claim 2, characterized in that, The physical parameter embedding layer calculates the stiffness parameters of each node by relating the bending moment of each node to the node stiffness, and maps the stiffness parameters to the node ID. The stiffness parameters include the bending stiffness and equivalent mass of the node.

8. A kinematics calculation device for a flexible boom based on a graph network simulator, characterized in that, The graph network simulator includes an encoder, a processor, and a decoder, and the flexible boom kinematics calculation device includes: The feature acquisition unit is configured to receive the flexible boom motion trajectory sequence of the current frame through an encoder, and extract and encode the node information and edge information of the flexible boom motion trajectory sequence to obtain node features and edge features. The node information of the flexible boom motion trajectory sequence includes the coordinate position, node type and joint angle value of each node. The feature update unit is configured to iteratively update the node features and edge features through the processor to obtain updated node features; The prediction unit is configured to decode the updated node features into node acceleration information using a decoder, and predict the node position in the next frame based on the node information of the current frame and the acceleration information.

9. A computing system comprising at least one computing device and at least one storage device for storing instructions, characterized in that, When the instruction is executed by the at least one computing device, it causes the at least one computing device to perform the kinematics calculation method for a flexible boom based on a graph network simulator according to any one of claims 1 to 7.

10. A computer-readable storage medium for storing instructions, characterized in that, When the instruction is executed by at least one computing device, it causes the at least one computing device to perform the kinematics calculation method for a flexible boom based on a graph network simulator according to any one of claims 1 to 7.

Images