Digital twin mechanical arm control method and system for discrete manufacturing

Abstract

A digital twin mechanical arm control method and system for discrete manufacturing, based on digital twin technology, a virtual mechanical arm is constructed for a physical mechanical arm, and the physical mechanical arm and the virtual mechanical arm interact bidirectionally through uplink and downlink, wherein the control method comprises the following steps: receiving real-time collected multi-modal perception data of the physical mechanical arm through the uplink; performing physical consistency verification on the multi-modal perception data based on a physical information neural network, and when the data is normal, real-time identification of the real physical parameters of the physical mechanical arm is performed through online learning of the physical information neural network; performing multi-modal causal logic verification on the multi-modal perception data based on a space-time graph neural network; the control method and system can deeply integrate physical mechanism and deep learning features, and have a digital twin interaction safety method with virtual-real parameter synchronization and countermeasures exercise capability, to ensure high robustness and high reliability of the discrete manufacturing system in uncertain environment.

Description

Technical Field

[0001] This invention relates to the field of robotic arm control technology, specifically to a digital twin robotic arm control method and system for discrete manufacturing. Background Technology

[0002] Digital twin technology has become a key driving force for the intelligent transformation of discrete manufacturing, with its core lying in building a closed-loop interaction system between physical entities and virtual models. Especially in scenarios such as precision aerospace assembly and automated automotive welding, six-degree-of-freedom industrial robotic arms, as key execution units, rely on high-frequency closed-loop interactions between physical entities and virtual models for operation. By sensing the real-time status of physical equipment through the uplink and utilizing downlink feedback control commands, precise mapping of the manufacturing process can be achieved. An efficient and reliable digital twin system must be built upon the absolute trustworthiness and security of this two-way data flow of "perception-decision-execution."

[0003] However, ensuring the smooth operation of a virtual-physical interactive system in complex industrial environments faces severe security challenges and a crisis of trust. Attackers may inject false data into the digital twin through man-in-the-middle attacks or sensor malfunction injections, such as replay attacks or data tampering, even if the data is within the specified range but violates the laws of physics. This can cause the digital twin to misjudge the physical state, leading to decision failures. Secondly, during long-term operation, the physical parameters of a robotic arm, such as joint friction coefficients, link damping, and transmission clearances, will change over time due to equipment aging and lubrication conditions. If the digital twin cannot perceive and synchronize these subtle changes in physical properties in real time, a gap between the virtual and physical worlds will emerge, causing overshoot, oscillations, or even collisions when control commands are executed on aging physical equipment. Existing control command generation is mostly based on algorithm planning under ideal conditions, without fully considering unstructured interference such as network random latency and instantaneous load disturbances. Once subjected to malicious network attacks or extreme operating conditions, seemingly compliant commands may instantly transform into abnormal commands that damage the equipment.

[0004] Current general solutions have several limitations: First, existing uplink detection methods largely rely on fixed rigid body dynamics models to calculate residuals, which cannot adapt to parameter drift caused by equipment aging and are prone to generating a large number of false alarms. While pure data-driven methods do not require precise modeling, they lack physical interpretability and are susceptible to adversarial examples that appear statistically normal but violate causal logic. Second, existing fusion technologies only focus on numerical alignment, ignoring the physical causal topological relationships between sensors. This makes it difficult for the system to identify covert semantic-level attacks that only tamper with a single modality. Third, existing downlink verification methods mostly use static thresholds or simple collision detection under ideal simulation environments, which cannot assess robustness in real-world scenarios such as network jitter and malicious disturbances. When dangerous commands are detected, only passive strategies such as interception or emergency stop are adopted, lacking an active mechanism for automatic correction to safe commands, leading to frequent downtime and seriously affecting production line continuity and overall equipment efficiency.

[0005] In summary, facing increasingly complex industrial security threats and the demands of equipment lifecycle management, existing technologies have significant shortcomings in areas such as adaptive migration of physical parameters, cross-modal causal verification, and adversarial command defense and correction. Therefore, it is necessary to propose a digital twin robotic arm control method and system for discrete manufacturing. Summary of the Invention

[0006] The purpose of this invention is to propose a digital twin robotic arm control method and system for discrete manufacturing. This digital twin interactive safety method can deeply integrate physical mechanisms and deep learning features, and has the ability to synchronize virtual and real parameters and conduct adversarial drills, so as to ensure the high robustness and high reliability of discrete manufacturing systems in uncertain environments.

[0007] The technical solution adopted in this invention is: a digital twin robotic arm control method for discrete manufacturing, which constructs a virtual robotic arm for a physical robotic arm based on digital twin technology, and the physical robotic arm and the virtual robotic arm interact bidirectionally through uplink and downlink. The control method includes the following steps:

[0008] The system receives multimodal sensing data from the physical robotic arm in real time via an uplink; it performs physical consistency verification on the multimodal sensing data based on Physical Information Neural Networks (PINNs), and identifies the actual physical parameters of the physical robotic arm in real time through online learning of the PINNs when the data is normal. Simultaneously, multimodal causal logic verification is performed on the multimodal sensing data based on the spatiotemporal graph neural network ST-GNN; based on the results of the physical consistency verification and causal logic verification, the credibility of the uplink data is determined, and the credible real physical parameters are identified. Write to the dynamic physical parameter pool;

[0009] When there are nominal control commands to be issued At that time, the actual physical parameters of the dynamic physical parameter pool are retrieved. The physical properties of the Unity sandbox are reconstructed to obtain a high-fidelity simulation environment; the nominal control commands are then executed within this high-fidelity simulation environment. Conduct adversarial robustness drills and calculate cumulative risk scores. If the cumulative risk score Exceeding the safety threshold Then, based on the control barrier function CBF, the nominal control command is applied. Optimize and modify to generate security control commands. ; to the security control command Or cumulative risk score Not exceeding the safety threshold The nominal control command The command is sent to the physical robotic arm via a downlink for execution.

[0010] As a preferred embodiment, the steps of performing physical consistency verification on the multimodal sensing data based on Physical Information Neural Networks (PINNs), and identifying the real physical parameters of the physical robotic arm in real time through online learning of the Physical Information Neural Networks (PINNs) when the data is normal, include:

[0011] A physical information neural network (PINNs) model is constructed, whose loss function includes physical constraint terms based on the Lagrange dynamics equations;

[0012] The joint state data from the multimodal sensing data Inputting the physical information into the PINNs neural network model yields the predicted torque. ;

[0013] Calculate the predicted torque The measured torque is obtained by converting the measured motor current from the multimodal sensing data. Physical consistency residuals between ;

[0014] According to statistics Criteria for calculating the first physical threshold First physical threshold A dynamic threshold used to distinguish between model fitting error and significant physical anomalies;

[0015] If the physical consistency residual Exceeding the first physical threshold If so, it is judged as a physical anomaly and the data is intercepted;

[0016] If the physical consistency residual Not exceeding the first physical threshold Then, using the aforementioned physical consistency residual The learnable physical parameters in the PINNs physical information neural network model are updated via backpropagation, and the updated parameter values ​​are used as the true physical parameters. .

[0017] As a preferred embodiment, the learnable physical parameters include the joint friction coefficient and the link load mass parameter.

[0018] As a preferred embodiment, the steps for performing multimodal causal logic verification on the multimodal sensing data based on the spatiotemporal graph neural network ST-GNN include:

[0019] A graph structure is constructed, and the multimodal sensing data is mapped to predefined graph structure nodes, where the graph structure nodes represent different sensors and the edges represent the physical causal relationships between the sensors;

[0020] Calculate the causal attention coefficient between nodes using the graph attention mechanism. ;

[0021] Take the causal attention coefficient at the current moment Compared with the baseline attention distribution under normal operating conditions The difference is used as the logical anomaly score. ;

[0022] Logical thresholds are set based on the statistical distribution characteristics of historical operating data under normal operating conditions. ;

[0023] If the logic is abnormal, the score is correct. Exceeding the logical threshold If the score is not met, it is determined to be a logical anomaly; if the logical anomaly score is not met, it is determined to be a logical anomaly. Not exceeding the logical threshold If so, it is considered logically normal.

[0024] As a preferred option, the multimodal sensing data includes joint states from the encoder. Motor current from the driver Contact force from the end force sensor With torque, and spatial coordinates of the end effector from the vision system Furthermore, all data is time-synchronized before entering the Physical Information Neural Network (PINNs).

[0025] As a preferred approach, when determining the reliability of uplink data, if the physical consistency check and the multimodal causal logic check do not report any anomalies, the data is considered reliable; if either check reports an anomaly, the data is considered unreliable and is blocked.

[0026] As a preferred option, the nominal control commands are processed in a high-fidelity simulation environment. The steps for conducting adversarial robustness exercises include:

[0027] The nominal control command The Adversarial RL agent integrated in the Unity sandbox acts as the attacker, while the object under test is the attacker.

[0028] The tested object follows the nominal control command In motion, the attacker simulates applying at least one of the following interferences: network latency, torque noise, and virtual external force to induce instability in the tested object;

[0029] Real-time monitoring of the collision status of the virtual robotic arm in the high-fidelity simulation environment. Resultant torque of joints ;

[0030] Based on the collision state and joint resultant torque The cumulative risk score is calculated based on the degree to which the value is exceeded. .

[0031] As a preferred embodiment, the security threshold Calibrated in the following manner:

[0032] Run a baseline trajectory in the Unity sandbox and apply environmental noise, recording the baseline risk peak. And multiply by the safety margin factor To obtain the safety threshold .

[0033] As a preferred embodiment, the nominal control command is based on the control barrier function CBF. The steps for optimization and correction include:

[0034] Based on the dynamic model of the physical robotic arm and the real physical parameters migrated from the dynamic physical parameter pool. Constructing control barrier functions ;

[0035] To minimize correction instructions With the nominal control command The deviation is the target, and the control barrier function is used as the control barrier function. The defined safety conditions are used as constraints to construct a quadratic programming model, OSQP.

[0036] Solving the quadratic programming model OSQP yields the nominal control command. The security control command with the smallest Euclidean distance .

[0037] A digital twin robotic arm control system for discrete manufacturing includes:

[0038] The physical entity layer includes a physical robotic arm equipped with sensors and a physical controller; the sensors are used to collect multimodal perception data of the physical robotic arm and transmit the multimodal perception data to the edge computing layer; the physical controller is used to receive instructions and control the physical robotic arm according to the instructions.

[0039] The edge computing layer includes an edge gateway and an edge server; the edge gateway is used to synchronize the multimodal sensing data transmitted by the physical entity layer in time and upload it to the edge server; the edge server is used to preprocess the synchronized multimodal sensing data and transmit it to the digital twin interaction layer.

[0040] The digital twin interaction layer includes an uplink inference engine with built-in physical information neural networks (PINNs) and spatiotemporal graph neural networks (ST-GNN), a memory area with a dynamic physical parameter pool, and a downlink defense engine with built-in Unity sandbox and CBF corrector. The digital twin interaction layer is used to receive multimodal perception data uploaded by the edge computing layer and execute the aforementioned digital twin robotic arm control method for discrete manufacturing.

[0041] Compared with the prior art, the beneficial effects of the present invention are:

[0042] To address the problems of existing discrete manufacturing digital twin systems, such as their inability to adapt to aging and wear of robotic arms due to fixed physical model parameters, difficulty in identifying hidden semantic attacks due to a lack of multimodal causal verification, and insufficient anti-interference robustness of downlink control commands due to a lack of adversarial verification, this invention provides a digital twin interactive safety method for discrete manufacturing robotic arms based on dynamic migration of physical parameters and virtual-real adversarial drills. This method utilizes a dual-stream perception architecture of Physical Information Neural Networks (PINNs) and Spatiotemporal Graph Neural Networks (ST-GNNs) to identify time-varying friction and dynamic parameters of the equipment in real time and verify logical consistency. It innovatively constructs a virtual-real parameter migration channel, injecting the identified real physical parameters into the downlink simulation environment in real time to reconstruct a high-fidelity sandbox. Based on this, an adversarial reinforcement learning agent is used to conduct extreme stress drills on commands, and a control barrier function (CBF) is used to actively project and correct high-risk commands. This invention ensures the physical authenticity and command immunity of the digital twin system throughout the entire equipment lifecycle, meeting the complex requirements of continuous and safe production in uncertain environments in high-end discrete manufacturing. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is an architecture diagram of the digital twin robotic arm control system in this invention;

[0045] Figure 2 This is a flowchart of the uplink in the control method of the present invention;

[0046] Figure 3 This is a schematic diagram of the downlink in the control method of the present invention. Detailed Implementation

[0047] The present invention will now be described in detail through exemplary embodiments. However, it should be understood that, without further description, elements, structures, and features in one embodiment may be advantageously incorporated into other embodiments.

[0048] It should be noted that, unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "a," "an," or "the," etc., used in the specification and claims of this patent application do not express a limitation on quantity, but rather indicate the presence of at least one; the terms "first," "second," and "third," as used herein, should not be considered as a limitation on the order of components, but are merely for distinguishing different components; the terms "comprising," "including," etc., indicate that the elements or objects preceding "comprising" or "including" encompass the elements or objects listed following "comprising" or "including" and their equivalents, but do not exclude other elements or objects having the same function.

[0049] To more clearly describe the control method and system for the digital twin robotic arm in discrete manufacturing, in conjunction with the appendix... Figure 1-3 This embodiment is described as follows:

[0050] like Figure 1 As shown, a digital twin robotic arm control system for discrete manufacturing selects a six-degree-of-freedom industrial robotic arm in a precision assembly scenario of discrete manufacturing as the physical entity object, and constructs a digital twin interactive safety system based on dynamic migration of physical parameters and virtual-real adversarial exercises; by constructing a high-precision time synchronization network and an innovative virtual-real parameter coupling channel, it achieves deep cleaning of uplink sensing data and immune enhancement of downlink control commands; the control system includes a physical entity layer, an edge computing layer, and a digital twin interaction layer.

[0051] The physical entity layer includes a physical robotic arm equipped with sensors and a physical controller;

[0052] Sensors are used to acquire multimodal sensing data from the physical robotic arm (multimodal sensing data includes joint states from the encoder). Motor current from the driver Contact force from the end force sensor With torque, and spatial coordinates of the end effector from the vision system The multimodal sensing data is transmitted to the edge computing layer; the sensors include internal and external sensors. The internal sensor group is based on the EtherCAT real-time bus and acquires joint angle vectors at a high frequency from the slave interface of the robotic arm controller at a 1ms cycle. Joint angular velocity Motor drive current An external sensor array is installed at the end flange of the robotic arm, consisting of a six-dimensional torque sensor, to collect the end contact torque. An RGB-D depth camera is deployed above the workspace side, transmitting environmental point clouds and image streams via the GigE Vision protocol, and calculating the spatial coordinates of the robotic arm's end effector in real time. ,in, This indicates the rotational angle position of the six joints of the robotic arm at the current moment, in radians (rad). The real-time current value for driving the joint motor of the robotic arm is expressed in amperes (A) and is positively correlated with the joint output torque. The contact force and torque are measured by a torque sensor installed at the flange. It is the three-dimensional position vector of the robotic arm end effector in the world coordinate system, calculated by the external vision system.

[0053] The physical controller is used to receive instructions and control the physical robotic arm according to the instructions.

[0054] The edge computing layer includes edge gateways and edge servers. Edge gateways are used to synchronize the multimodal sensing data transmitted from the physical entity layer and upload it to the edge servers. For time synchronization, the edge gateway is deployed as the master clock node for IEEE 1588 PTP (Precise Time Protocol), while the physical controller, vision acquisition card, and force acquisition card of the physical robotic arm are configured as slave nodes. A unified global timestamp is applied to all heterogeneous sensor data via the PTP protocol to ensure the time synchronization error between visual frames and current frames. This is a prerequisite for ST-GNN to correctly learn the causal topology that current changes cause visual displacement. Edge servers are used to preprocess the synchronized multimodal sensing data and transmit it to the digital twin interaction layer.

[0055] The digital twin interaction layer is used to receive multimodal perception data uploaded from the edge computing layer and execute a digital twin robotic arm control method for discrete manufacturing. The digital twin interaction layer includes an uplink inference engine with built-in physical information neural networks PINNs and spatiotemporal graph neural networks ST-GNN, a memory area with a dynamic physical parameter pool, and a downlink defense engine with built-in Unity sandbox and CBF corrector.

[0056] I. PINNs (Physical Information Neural Network) Model

[0057] Construct a fully connected deep neural network whose input layer receives the motion state of the robotic arm. Output layer predicts joint torque Embed the Lagrange dynamic residual term into the network's loss function. In the network, a learnable friction coefficient vector is defined using `torch.nn.Parameter`. and connecting rod load mass parameters . Defined as the physical loss function of PINNs networks, the calculation formula is:

[0058]

[0059] in, For network to predict torque, and They represent The first and second derivatives with respect to time. The inertia matrix, For the Coriolis force and centrifugal force terms, This is the gravity term.

[0060] This model is not only used to detect anomalies that violate the laws of physics, but also acts as an online system identifier to extract the true physical parameters of devices in real time.

[0061] II. ST-GNN (Spatiotemporal Graph Neural Network) Model

[0062] Constructing a sensor graph structure . It includes 6 articulated motor nodes, 1 visual observation node, and 1 force sensing node. Two types of edges are defined: kinematic connection edges based on DH parameters (e.g., joint 1 connecting joint 2) and causal observation edges based on physical responses (e.g., joint current nodes pointing to end force nodes). A graph attention mechanism is used to aggregate neighborhood features to identify cross-modal logical inconsistencies.

[0063] III. Unity Competitive Practice Sandbox

[0064] Based on the Unity 3D engine, a rigid body model and mesh collider corresponding to the physical entity are loaded, and an Adversarial RL (Adversarial Reinforcement Learning) agent is integrated into the sandbox. This agent is trained as the "red team" and has the ability to apply network latency, torque noise, and virtual external force pushing to the robotic arm within the simulation step size, which is used to test the robustness of the commands.

[0065] IV. CBF (Control Barrier Function) Modifier

[0066] Based on the kinematic and dynamic constraints of the robotic arm, a control obstacle function is constructed. Deploy the OSQP (OperatorSplitting Quadratic Program) solver to solve quadratic programming problems in milliseconds, projecting risky instructions onto a safe set. Define the set of safe states for the robotic arm, satisfying This indicates that the system is in a safe zone.

[0067] V. Virtual-Real Migration Coupling Channel (Dynamic Parameter Pool)

[0068] A dynamic physical parameter pool is allocated in memory as a coupling medium for virtual-real interaction, and the real physical parameter set of the PINNs model convergence is extracted in real time through a bidirectional interface. This is injected into the PhysicalMaterial interface of the Unity simulation engine to achieve real-time updating and synchronization of the physical properties of the simulation environment as the entities age. It is the set of parameters identified by PINNs uplink. The actual joint friction coefficient identified at the current moment. This represents the equivalent mass of the links and loads identified at the current moment. PhysicMaterial is an API class in the Unity 3D engine that controls the physical properties of rigid body surfaces. By modifying its property values, the dynamic response characteristics of the robotic arm in the simulation sandbox can be changed.

[0069] like Figure 2 and Figure 3 As shown, a digital twin robotic arm control method for discrete manufacturing constructs a virtual robotic arm based on digital twin technology. The physical and virtual robotic arms interact bidirectionally through uplink and downlink. The control method includes the following steps:

[0070] The system receives real-time multimodal sensing data from the physical robotic arm via the uplink; it performs physical consistency verification on the multimodal sensing data based on the Physical Information Neural Network (PINNs), and identifies the true physical parameters of the physical robotic arm in real time through online learning of the PINNs when the data is normal. Simultaneously, multimodal causal logic verification is performed on multimodal sensing data based on the spatiotemporal graph neural network ST-GNN; based on the results of physical consistency verification and causal logic verification, the credibility of uplink data is determined, and credible real physical parameters are identified. Write to the dynamic physical parameter pool;

[0071] When there are nominal control commands to be issued At that time, retrieve the actual physical parameters from the dynamic physical parameter pool. The physical properties of the Unity sandbox are reconstructed to obtain a high-fidelity simulation environment; nominal control commands are then processed within this high-fidelity simulation environment. Conduct adversarial robustness drills and calculate cumulative risk scores. If the cumulative risk score Exceeding the safety threshold Then, the nominal control command is based on the control barrier function CBF. Optimize and modify to generate security control commands. ; transfer security control commands Or cumulative risk score Not exceeding the safety threshold nominal control commands The command is sent to the physical robotic arm via a downlink for execution.

[0072] The specific steps of the control method are as follows:

[0073] Step 1: Multimodal causal logic verification based on ST-GNN

[0074] A graph structure is constructed, and multimodal sensing data is mapped to predefined graph structure nodes. Graph structure nodes represent different sensors, and edges represent the physical causal relationships between sensors. The causal attention coefficients between nodes are calculated through a graph attention mechanism. Take the causal attention coefficient at the current moment. Compared with the baseline attention distribution under normal operating conditions The difference is used as the logical anomaly score. ; Set logical thresholds based on the statistical distribution characteristics of historical operating data under normal operating conditions. If the logic is abnormal, a score will be given. Exceeding the logical threshold If it is not found, it is judged as a logical anomaly; if the logical anomaly score is not found, it is considered a logical anomaly. Not exceeding the logical threshold If so, it is considered logically normal.

[0075] This step serves as a defense mechanism, designed to identify covert attacks that are numerically compliant but logically broken. Such attacks often meticulously forge data from a single sensor to meet threshold ranges, but sever the inherent physical causal connections between the sensors. The specific steps are as follows:

[0076] Data encapsulation and graph mapping: The system first encapsulates the synchronized multimodal data into a feature vector at time t. Subsequently, based on the graph structure defined in Part 1... This maps each component in the feature vector to its corresponding graph node.

[0077] In the eigenvector middle, This represents the rotational angle position of the six joints of the robotic arm at the current moment, in radians. It is a joint angle vector that is collected at high frequency from the slave interface of the robotic arm controller through the internal sensor group. The joint angular velocity of the robotic arm is also data collected at high frequencies by the internal sensor array. This represents the real-time current value of the motor that drives the joint of the robotic arm, measured in amperes (A), and is positively correlated with the joint output torque. This represents the contact force and torque measured by a torque sensor installed at the end flange of the robotic arm. This represents the three-dimensional position vector of the robotic arm's end effector in the world coordinate system, calculated by the RGB-D depth camera. .

[0078] Causal strength inference: The ST-GNN model's startup graph attention mechanism does not focus on the isolated numerical value of nodes, but rather calculates the node's... Its neighboring nodes Attention coefficient between :

[0079]

[0080] This indicates the strength of the causal dependency between two physical quantities. For example, under normal operating conditions, an increase in motor current will inevitably cause a displacement in the visual image; in this case, the causal relationship between the two... It should remain at a high level. Among them, It is a non-linear activation function that allows small negative gradients to pass through, preventing neurons from "dying". , For nodes in the graph and nodes eigenvectors, This is a learnable linear transformation matrix used to map input features to a high-dimensional latent space. This represents the weight vector of the attention mechanism. Let be its transpose vector, used to concatenate two transformed eigenvectors into a long vector. For nodes The set of neighboring nodes, The neighbor node index variable represents the node. One of the neighboring nodes, Representing neighbor nodes The feature vector at the current moment.

[0081] The subsequent model calculates the difference between the current causal edge weight distribution and the baseline distribution to generate a logical anomaly score. :

[0082]

[0083] in, For the current moment The calculated attention coefficient represents the actual causal dependency strength between nodes. This is the baseline attention distribution obtained by training with historical data under normal operating conditions. Representing graph structure set of all edges Quantity, Indicates the graph Each connection node defined in and nodes The edges are traversed and summed.

[0084] Compare the calculated logic anomaly score with the logic threshold. In comparison, if If this occurs, it is considered an anomaly. This indicates that the previously stable physical causal chain (edge ​​weights) between sensors has broken or shifted drastically. Among these, the logical threshold... It is a dynamic benchmark set based on the statistical distribution characteristics of the system's historical operating data under normal operating conditions.

[0085] Multimodal causal logic verification based on ST-GNN can be used to detect the following two classic anomalies:

[0086] I. Visual Replay Attack: The attacker locks the visual view. (The image freezes), but at this moment the motor current... The display shows the robotic arm accelerating. In the graph structure, the value of the "current node" changes drastically, while the value of the "vision node" remains unchanged, causing the attention coefficient of the causal edge between the two to drop sharply, triggering a logic alarm.

[0087] II. Force sensor failure or spoofing: The robotic arm performs a contact task, and the current... The increase (indicating that the motor is outputting torque), but the force sensor reading... The value remains zero. This "force without feedback" state leads to anomalies in the "current-force" edge weights.

[0088] Step 2: Physical consistency verification and parameter extraction based on PINNs

[0089] A Physical Information Neural Network (PINNs) model is constructed, whose loss function includes physical constraint terms based on the Lagrange dynamics equations; joint state data from multimodal sensing data is incorporated. Inputting physical information into a PINNs neural network model yields the predicted torque. ; Calculate the predicted torque The measured torque is obtained by converting the measured motor current from multimodal sensing data. Physical consistency residuals between According to statistics Criteria for calculating the first physical threshold First physical threshold A dynamic threshold used to distinguish between model fitting error and significant physical anomalies;

[0090] If physical consistency residual Exceeding the first physical threshold If the physical consistency residual is not found, it is considered a physical anomaly and the data is intercepted; if the physical consistency residual is not found, it is considered a physical anomaly and the data is intercepted. Not exceeding the first physical threshold Then, using physical consistency residuals The learnable physical parameters in the PINNs (Physical Information Neural Network) model are updated via backpropagation, and the updated parameter values ​​are used as the true physical parameters. .

[0091] This step is not only used for anomaly detection, but also a crucial step in realizing the parameter migration of this invention. The PINNs network both detects attacks that violate the laws of dynamics and extracts the actual physical parameters of the device in real time to reflect the current aging state of the device. The specific steps are as follows:

[0092] The specific steps are as follows: [Regulate joint position] Input to PINNs network, output predicted torque Predicted torque The Lagrangian dynamic constraints defined in Part 1 must be satisfied. The system converts this predicted value with the measured torque obtained by converting the measured motor current. Perform a comparison and calculate the physical consistency residual. :

[0093]

[0094] like The price surged instantly, exceeding the first physical threshold. This indicates that the current data combination violates Newton's second law or the law of conservation of energy. The system determines this to be a non-physical data injection attack or a serious mechanical failure, and directly discards the data frame. The first physical threshold... It is based on statistics The parameters calculated by the criteria can tolerate the small fitting errors inherent in the PINNs model and the sensor noise floor, while accurately intercepting anomalous attacks that significantly deviate from Newton's second law or the law of conservation of energy.

[0095] when When within the normal fluctuation range, the PINNs network enters online learning mode. It uses the backpropagation gradient of the current residual to fine-tune the learnable parameters inside the network. (Coefficient of friction) and (Load mass). The physical meaning of this reaction is: as the robotic arm heats up during operation or lubricant is consumed, the actual frictional force will drift. PINNs force the internal [mechanism / mechanism] to continuously minimize prediction errors. The parameters gradually approach the current true friction coefficient, a process that enables a real-time digital profile of the aging state of a physical entity.

[0096] PINN-based physical consistency checks can be used to detect the following two classic anomalies:

[0097] I. Data Injection Attack: The attacker injects a set of fake joint angles into the controller. (The robotic arm is shown performing complex movements), but corresponding current data is not being falsified simultaneously. (The current display is stationary). At this point, the theoretical torque calculated according to the formula will be very large, while the measured torque will be very small, resulting in a residual between the two. It surged instantly and was immediately intercepted.

[0098] II. Sudden Mechanical Failure: If the robotic arm's drivetrain suddenly breaks (motor idles), the motor current... Very small, but the acceleration fed back by the encoder However, it is greatly affected by gravity. This cannot be currently... and The mutations explained by the parameters can also lead to residuals exceeding limits.

[0099] In the first and second steps, the system starts two threads in parallel: ST-GNN causal logic verification and PINNs physical parameter identification. While eliminating false injected data, it extracts the aging characteristic parameters of physical entities in real time, providing high-fidelity environmental data for downlink simulation.

[0100] Step 3: Comprehensive Judgment and Parameter Injection

[0101] To determine the reliability of upstream data, if neither the physical consistency check nor the multimodal causal logic check reports anomalies, the data is considered reliable, and the reliable physical parameters will be identified. Write to the dynamic physical parameter pool; if any check reports an anomaly, the data is deemed untrustworthy and intercepted.

[0102] The system scores based on logic. and physical residuals Make the final decision and execute the parameter migration operation. Define exception flags. :

[0103]

[0104] like (Abnormal): If the current frame data is determined to be untrustworthy, under attack, or seriously faulty, the data will be discarded directly without model update and an alarm will be triggered.

[0105] like (Normal): The current data is determined to be reliable and reflects the true state of the device. At this time, the virtual-physical migration coupling mechanism is triggered: extract the parameter values ​​of PINNs convergence at the current moment and update the real physical parameter set. Then, using the dynamic physics parameter pool constructed in the first part, Send to the downlink simulation engine in real time.

[0106] In the downlink, real physical parameters mined from the uplink are used to implement a three-tiered defense against the control commands to be issued: "environmental reconstruction - adversarial drills - dynamic correction." This process is deployed between the digital twin's control layer and the physical execution layer. When the digital twin's decision algorithm generates a nominal control command... When a pre-planned, unverified instruction is about to be issued, the system automatically intercepts the instruction and triggers the following processing steps in sequence:

[0107] Step 4: High-fidelity reconstruction of the simulation environment based on parameter transfer

[0108] This is a crucial step in achieving virtual-real evolution in this invention. Before rehearsing the instructions, it is essential to ensure that the physical properties of the training ground (Unity sandbox) are completely consistent with the current real state of the robotic arm; otherwise, the training results will lose their reference value.

[0109] The downlink engine first accesses the dynamic physical parameter pool in memory, reading the latest parameter set identified and written in real time by the PINNs model in Part 2. Using Unity 3D's scripting interface, it iterates through each rigid body component of the robotic arm model in the sandbox. Then, it calls the PhysicMaterial API to forcibly update the corresponding dynamic / static friction, damping coefficient, and other attribute values ​​in the simulation model. The value in [the code]. At this point, the current joint angle of the physical robotic arm [is set]. and angular velocity The values ​​are assigned to the sandbox model to ensure that the starting line of the pre-show is synchronized with the real world. At this point, the Unity sandbox has evolved from an ideal environment into a digital twin that accurately replicates the aging state of physical entities.

[0110] Step 5: Adversarial Robustness Exercises Based on Adversarial RL

[0111] Nominal control commands As the object under test, the Adversarial RL agent integrated in the Unity sandbox acts as the attacker; the object under test follows the nominal control commands. The attacker simulates the application of at least one of the following interferences—network latency, torque noise, and virtual external forces—to induce instability in the tested object; the collision status of the virtual robotic arm in the high-fidelity simulation environment is monitored in real time. Resultant torque of joints Based on collision state and joint resultant torque The degree to which the limit is exceeded is used to calculate the cumulative risk score. .

[0112] In the reconstructed high-fidelity environment, the system does not simply replay commands, but introduces a "red-blue team game" mechanism to test the commands' anti-interference capabilities under extreme conditions. The nominal control commands to be issued... The target is set as the "Blue Team"; the Adversarial RL agent integrated in the sandbox is set as the attacker, i.e., the "Red Team".

[0113] During the confrontation, the system activated real-time physics simulation. The Blue Team drove the virtual robotic arm along... During the movement, the red agent, based on its current state, moves within the action space. The attack strategy is selected to attempt to induce instability in the robotic arm.

[0114] The system monitors the dynamic response during the exercise in real time and calculates the cumulative risk score. :

[0115]

[0116] in, This is a collision indication function. If the virtual robotic arm interferes with itself or collides with the environment, the value is 1; otherwise, it is 0. The resultant torque on the virtual joints during the exercise (resultant torque of the joints, including the disturbances applied by the red team). This is the rated peak torque of the motor; , This is the risk weighting coefficient.

[0117] To determine whether the above risk scores are acceptable, the system needs to predetermine a safety threshold. This threshold is not a fixed value, but rather derived based on benchmark calibration. During system initialization or idle periods, a set of standard, fault-free trajectories are run in the Unity sandbox, without initiating red-team attacks, and only subjecting natural ambient noise conforming to a Gaussian distribution. The peak risk score during this benchmark run is recorded. And introduce a safety margin factor Calculation . This represents the upper limit of dynamic fluctuations that the robotic arm can withstand under non-destructive normal operating conditions.

[0118] The scores from the current combat exercise With the calibrated safety threshold Compare:

[0119] like (Robust Instruction): This indicates that even under the malicious interference of the Red Team and the current aging parameters, the risk caused by this instruction does not exceed the safety boundary of normal operating conditions, and is therefore deemed safe and can be directly issued.

[0120] like (Vulnerable Instruction): This indicates that the Red team's attack successfully induced a collision or severe torque overload in the robotic arm. This instruction is highly likely to cause an accident in extreme environments and is therefore deemed unsafe, requiring the next step of the CBF correction process.

[0121] Step 6: CBF-based dynamic projection correction of instructions

[0122] If nominal control command Cumulative risk score Not exceeding the safety threshold Then the nominal control command The control command is sent directly to the physical robotic arm via the downlink for execution; if the nominal control command... Cumulative risk score Exceeding the safety threshold Then the nominal control command will be... Revised to security control command Then, the security control command is transmitted via the downlink. The command is then sent to the physical robotic arm for execution.

[0123] Dynamic model of the physical robotic arm and real physical parameters transferred from the dynamic physical parameter pool. Constructing control barrier functions To minimize correction instructions With nominal control commands The deviation is the target, and the control barrier function is used to control the obstacle. The defined safety conditions are used as constraints to construct a quadratic programming model OSQP; solving the quadratic programming model OSQP yields the result corresponding to the nominal control command. Safety control command with minimum Euclidean distance .

[0124] When the RL exercise determines that a command is vulnerable, the system activates the CBF corrector, which projects the dangerous command to a safe area based on optimal control theory, generating the final safety control command. .

[0125] Based on the dynamic model of the robotic arm Construct the control barrier function defined in the first part. Special note: The drift vector field here... and control input matrix The same real parameters were also used, which were migrated from PINNs. This means that the safety boundary will also be automatically adjusted according to the degree of equipment aging.

[0126] Establish a quadratic programming (OSQP) model. The goal is to find a safe control instruction. This makes it consistent with the original safety control commands. The Euclidean distance is minimized, which ensures task continuity while satisfying the CBF safety constraint:

[0127]

[0128]

[0129] in, To optimize variables (correction instructions); , representing the barrier function Lie derivative along the direction of the system's natural dynamics; , representing the barrier function Lie derivative along the direction of control input; For class The function coefficients (decay rate) determine the rate at which the system moves away from the boundary of the unsafe set.

[0130] Finally, the OSQP solver outputs the optimal solution, which is the corrected safety control command. The edge gateway transmits the revised security control commands via the EtherCAT bus. The data is written into the physical controller to drive the robotic arm to perform actions. Through this step, the system achieves a leap from passive interception to proactive self-healing, ensuring the continuity of the production process.

[0131] The parts not described in detail in the above embodiments are existing technologies.

[0132] It should be noted that although the present invention has been described through the above embodiments, the present invention may have many other embodiments. Without departing from the spirit and scope of the present invention, those skilled in the art can obviously make various corresponding changes and modifications to the present invention, but all such changes and modifications should fall within the scope of protection of the appended claims and their equivalents.

Claims

1. A control method for a digital twin robotic arm for discrete manufacturing, characterized in that: A virtual robotic arm is constructed based on digital twin technology for a physical robotic arm. The physical and virtual robotic arms interact bidirectionally via uplink and downlink. The control method includes the following steps: The system receives multimodal sensing data from the physical robotic arm in real time via an uplink; it performs physical consistency verification on the multimodal sensing data based on Physical Information Neural Networks (PINNs), and identifies the actual physical parameters of the physical robotic arm in real time through online learning of the PINNs when the data is normal. Simultaneously, multimodal causal logic verification is performed on the multimodal sensing data based on the spatiotemporal graph neural network ST-GNN; based on the results of the physical consistency verification and causal logic verification, the credibility of the uplink data is determined, and the credible real physical parameters are identified. Write to the dynamic physical parameter pool; When there are nominal control commands to be issued At that time, the actual physical parameters of the dynamic physical parameter pool are retrieved. The physical properties of the Unity sandbox are reconstructed to obtain a high-fidelity simulation environment; the nominal control commands are then executed within this high-fidelity simulation environment. Conduct adversarial robustness drills and calculate cumulative risk scores. If the cumulative risk score Exceeding the safety threshold Then, based on the control barrier function CBF, the nominal control command is applied. Optimize and modify to generate security control commands. ; to the security control command Or cumulative risk score Not exceeding the safety threshold The nominal control command The command is sent to the physical robotic arm via a downlink for execution.

2. The digital twin robotic arm control method for discrete manufacturing according to claim 1, characterized in that, The steps of performing physical consistency verification on the multimodal sensing data based on Physical Information Neural Networks (PINNs), and identifying the real physical parameters of the physical robotic arm in real time through online learning of the PINNs when the data is normal, include: A physical information neural network (PINNs) model is constructed, whose loss function includes physical constraint terms based on the Lagrange dynamics equations; The joint state data from the multimodal sensing data Inputting the physical information into the PINNs neural network model yields the predicted torque. ; Calculate the predicted torque The measured torque is obtained by converting the measured motor current from the multimodal sensing data. Physical consistency residuals between ; According to statistics Criteria for calculating the first physical threshold First physical threshold A dynamic threshold used to distinguish between model fitting error and significant physical anomalies; If the physical consistency residual Exceeding the first physical threshold If so, it is judged as a physical anomaly and the data is intercepted; If the physical consistency residual Not exceeding the first physical threshold Then, using the aforementioned physical consistency residual The learnable physical parameters in the PINNs physical information neural network model are updated via backpropagation, and the updated parameter values ​​are used as the true physical parameters. .

3. The digital twin robotic arm control method for discrete manufacturing according to claim 2, characterized in that: The learnable physical parameters include the joint friction coefficient and the link load mass parameter.

4. The digital twin robotic arm control method for discrete manufacturing according to claim 1, characterized in that, The steps for performing multimodal causal logic verification on the multimodal sensing data based on the spatiotemporal graph neural network ST-GNN include: A graph structure is constructed, and the multimodal sensing data is mapped to predefined graph structure nodes, where the graph structure nodes represent different sensors and the edges represent the physical causal relationships between the sensors; Calculate the causal attention coefficient between nodes using the graph attention mechanism. ; Take the causal attention coefficient at the current moment Compared with the baseline attention distribution under normal operating conditions The difference is used as the logical anomaly score. ; Logical thresholds are set based on the statistical distribution characteristics of historical operating data under normal operating conditions. ; If the logic is abnormal, the score is correct. Exceeding the logical threshold If the score is not met, it is determined to be a logical anomaly; if the logical anomaly score is not met, it is determined to be a logical anomaly. Not exceeding the logical threshold If so, it is considered logically normal.

5. The digital twin robotic arm control method for discrete manufacturing according to claim 1, characterized in that: Multimodal sensing data includes joint states from the encoder. Motor current from the driver Contact force from the end force sensor With torque, and spatial coordinates of the end effector from the vision system Furthermore, all data is time-synchronized before entering the Physical Information Neural Network (PINNs).

6. The digital twin robotic arm control method for discrete manufacturing according to claim 1, characterized in that: When determining the trustworthiness of uplink data, if the physical consistency check and the multimodal causal logic check do not report any anomalies, the data is considered trustworthy; if either check reports an anomaly, the data is considered untrustworthy and is blocked.

7. The digital twin robotic arm control method for discrete manufacturing according to claim 1, characterized in that, The nominal control command in a high-fidelity simulation environment The steps for conducting adversarial robustness exercises include: The nominal control command The Adversarial RL agent integrated in the Unity sandbox acts as the attacker, while the object under test is the attacker. The tested object follows the nominal control command The attacker simulates applying at least one of the following interferences—network latency, torque noise, and virtual external force—to induce instability in the object under test. Real-time monitoring of the collision status of the virtual robotic arm in the high-fidelity simulation environment. Resultant torque of joints ; Based on the collision state and joint resultant torque The cumulative risk score is calculated based on the degree to which the value is exceeded. .

8. The digital twin robotic arm control method for discrete manufacturing according to claim 1, characterized in that, The security threshold Calibrated in the following manner: Run a baseline trajectory in the Unity sandbox and apply environmental noise, recording the baseline risk peak. And multiply by the safety margin factor To obtain the safety threshold .

9. The digital twin robotic arm control method for discrete manufacturing according to claim 1, characterized in that: The nominal control command is based on the control barrier function CBF. The steps for optimization and correction include: Based on the dynamic model of the physical robotic arm and the real physical parameters migrated from the dynamic physical parameter pool. Constructing control barrier functions ; To minimize correction instructions With the nominal control command The deviation is the target, and the control barrier function is used as the control barrier function. The defined safety conditions are used as constraints to construct a quadratic programming model, OSQP. Solving the quadratic programming model OSQP yields the nominal control command. The security control command with the smallest Euclidean distance .

10. A digital twin robotic arm control system for discrete manufacturing, characterized in that, include: The physical entity layer includes a physical robotic arm equipped with sensors and a physical controller; the sensors are used to collect multimodal perception data of the physical robotic arm in real time and transmit the multimodal perception data to the edge computing layer; the physical controller is used to receive instructions and control the physical robotic arm according to the instructions. The edge computing layer includes an edge gateway and an edge server; the edge gateway is used to synchronize the multimodal sensing data transmitted by the physical entity layer in time and upload it to the edge server; Edge servers are used to preprocess the synchronized multimodal sensing data and transmit it to the digital twin interaction layer; The digital twin interaction layer includes an uplink inference engine with built-in physical information neural networks (PINNs) and spatiotemporal graph neural networks (ST-GNN), a memory area with a dynamic physical parameter pool, and a downlink defense engine with built-in Unity sandbox and CBF corrector; the digital twin interaction layer is used to receive multimodal perception data uploaded by the edge computing layer and execute the digital twin robotic arm control method as described in any one of claims 1-9.

Images