A Method for Optimizing the Operational Parameters of Embossed Intelligent Robots Based on Digital Twins

Abstract

This invention belongs to the field of embodied intelligent robot technology and discloses a method for optimizing the operational parameters of embodied intelligent robots based on digital twins. Addressing the problem of discrepancies and disconnects between simulation and reality in terms of physical parameters, this invention constructs a three-dimensional twin representation and uses photometric error gradients to generate virtual force injection into a differentiable engine to achieve state synchronization. The simulation is reconstructed into a differentiable computational graph, and physical parameters are identified through automatic differentiation via backpropagation. Then, using the aligned parameters, a nested dual-loop control architecture is employed. The outer loop outputs the target pose, while the inner loop estimates the contact force based on inverse dynamics residual inversion, executing sensorless compliant impedance control. This invention effectively improves parameter identification accuracy and achieves high-frequency closed-loop execution, making it widely applicable to tasks such as flexible assembly.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligent robot technology, and in particular to a method for optimizing the operational parameters of embodied intelligent robots based on digital twins. Background Technology

[0002] Embodied intelligent systems aim to learn and optimize closed-loop control strategies for perception, decision-making, and execution through continuous interaction between intelligent agents and complex physical environments. Due to the high-dimensional nonlinear characteristics and unpredictable dynamic disturbances of the physical world, large-scale trial-and-error learning directly on real physical hardware faces extremely high time costs and hardware degradation risks. Therefore, concurrent training in digital twin simulators and transferring converged control strategies to physical robots for deployment has become a widely adopted R&D paradigm in the industry.

[0003] However, the operation of digital twin simulators is highly dependent on a series of hidden physical and dynamic parameters, including the robot's control gain, joint damping, mass distribution of the manipulated object, and friction coefficients between contact surfaces. In existing technologies, these parameters are typically preset manually by engineers based on experience or obtained through offline static calibration. Such static modeling cannot capture the nonlinear dynamic characteristics that evolve with temperature, wear, or load changes in real physical interactions. No matter how sophisticated the digital twin system is designed, there is always a systematic deviation between its underlying mathematical model and the actual physical and dynamic evolution. This deviation is significantly amplified when performing contact-rich tasks, causing the task parameters trained in the simulation environment to severely degrade in real-world task performance. To bridge this deviation, existing solutions have introduced gradient-free heuristic search methods such as simulated annealing, genetic algorithms, or Bayesian optimization to identify physical parameters. However, these methods treat the simulation engine as an impenetrable black box, unable to obtain gradient information of physical state changes on underlying parameters. The parameter search process exhibits high degree of blindness and randomness in high-dimensional space, with slow convergence speed and a high tendency to get trapped in local optima.

[0004] Meanwhile, existing digital twin systems suffer from temporal and spatial disconnects between robot strategy training and physical deployment. During actual operation, digital twin systems typically function only as visual monitoring terminals, unable to perform high-frequency real-time state synchronization based on real-world visual feedback and end-effector contact torque. When unmodeled disturbances occur in the real environment, the state within the simulator quickly deviates from reality. Furthermore, current mainstream imitation learning and reinforcement learning-based control strategies are position-driven, with the policy network outputting only the desired coordinates of the end effector, lacking explicit force perception and compliant control capabilities. To compensate for this deficiency, the industry standard practice is to add a six-dimensional torque sensor to the robot's end effector. However, this not only increases the system's hardware cost and mechanical design complexity but also, due to the lack of a dual-loop control architecture that integrates short-term online fine-tuning of physical parameters with long-term global strategy evolution, the robot still cannot perform flexible parameter adjustments within millisecond timescales when facing work objects with unknown materials or geometries. The lack of a physical identification mechanism leads to distortion of the digital twin base, making it unable to provide reliable model constraints for the control algorithm. The lack of a high-frequency dual-loop feedback execution architecture also prevents the underlying parameters from obtaining continuous closed-loop correction data in dynamic operations. These two factors mutually restrict the optimization level of the embodied intelligent robot's operational parameters in contact-rich tasks. Summary of the Invention

[0005] The purpose of this invention is to provide a method for optimizing the operation parameters of an embodied intelligent robot based on digital twins. This method aims to solve the problems in the prior art where static modeling of digital twin systems cannot capture the nonlinear dynamic characteristics of real physics, resulting in a systematic deviation between simulation and the real world, and the lack of a high-frequency dual-loop feedback execution architecture, which prevents the underlying parameters from obtaining continuous closed-loop correction data in dynamic operations.

[0006] To achieve the above objectives, this invention provides a method for optimizing the operational parameters of an embodied intelligent robot based on digital twins, comprising the following steps: A three-dimensional digital twin representation of the work scene is constructed. Virtual visual force is generated based on the photometric error gradient between real camera images and twin rendered images. The virtual visual force is then injected into the dynamic equation of a differentiable physics engine to achieve real-time state synchronization between physical space and twin space. The multi-rigid-body dynamics simulation process is reconstructed into a fully differentiable computational graph to obtain the forward simulation trajectory based on the robot's actual execution trajectory and control commands. The gradient of the physical loss function with respect to the physical parameters to be identified is calculated through backpropagation using automatic differentiation technology. The high-precision aligned physical parameters are obtained through iterative optimization. Using the aligned physical parameters, an adaptive adjustment and execution control of the operation parameters are performed through a nested dual-loop control architecture. The low-frequency outer loop infers and outputs the target pose based on multimodal observation features, while the high-frequency inner loop calculates the inverse dynamic residual torque based on the aligned physical parameters, estimates the end contact force through the generalized pseudo-inverse inversion of the end effector kinematic Jacobian matrix, and performs sensorless compliant impedance control based on the estimated end contact force.

[0007] Through the above technical solution, the present invention can replace the blind random search in the traditional gradient-free heuristic algorithm with a deterministic gradient descent path, significantly improve the convergence speed and estimation accuracy of physical parameter identification, realize the closed-loop state synchronization of high-frequency operation in physical space and twin space, and enable the robot to have millisecond-level adaptive flexible control capability under unknown disturbances.

[0008] As a preferred embodiment of the present invention, the construction of the three-dimensional digital twin representation of the work scene includes: A multi-view image sequence covering the workspace is acquired to generate a sparse point cloud, and a three-dimensional Gaussian particle set is initialized based on the sparse point cloud. The parameters of each Gaussian particle include a three-dimensional position mean vector, a covariance matrix represented by a combination of a rotation quaternion and a three-dimensional scaling vector, an opacity parameter, and spherical harmonic function coefficients. A subset of Gaussian particles belonging to the object to be manipulated is extracted using a 3D instance segmentation network. The surface contours of these particles are then mapped onto the surface of a rigid body collision mesh within the differentiable physics engine using a differentiable moving cube algorithm. This establishes a mathematical mapping from the rigid body state driven by the physics engine to the Gaussian particle pose update in the visual rendering layer. This achieves a deep binding between the purely visual representation of Gaussian particles and the physically dynamic entities.

[0009] As a preferred embodiment of the present invention, the method of generating virtual visual power based on the photometric error gradient between a real camera image and a twin-rendered image includes: The twin image is synthesized in real time from a virtual perspective using a twin rendering pipeline, and the photometric error loss function between the real camera image and the twin image is calculated. The photometric error loss function is composed of a weighted combination of pixel-level absolute value norm distance and structural similarity index; The backpropagation mechanism of the computation graph is invoked to calculate the photometric error loss function with respect to the mean vector of the associated Gaussian particle three-dimensional positions. The partial derivatives are used to generate the corresponding discrete virtual visual power. ,in This refers to the damping gain parameter; The discrete virtual visual forces of all associated particles are aggregated into a resultant lateral force acting on the object's center of mass and a resultant torque relative to the center of mass, which are then injected as an external additional torque into the dynamic equation of the next time step.

[0010] As a preferred embodiment of the present invention, the reconstructing of the multi-rigid-body dynamics simulation process into a fully differentiable computational graph includes: Construct the positive dynamic equations of the rigid body that include joint friction terms: in, For joint generalized coordinates, The inertia matrix, The matrix of Coriolis force and centrifugal force. The vector of the gravity term. For driving torque, Let Jacobian matrix be the point of contact in the collision. For contact impulse vector, The term for the differentiable joint friction torque is constructed by combining the Coulomb friction coefficient and the viscous friction coefficient to be identified with a smooth approximation function. An implicit integrator is used to discretize the continuous dynamic equations, transforming the contact constraints into a linear complementary problem that satisfies the non-penetration condition and the Coulomb friction cone constraint. The iterative solution process of the linear complementary problem is encapsulated into a differentiable function through analytical differentiation using implicit function theorems.

[0011] As a preferred embodiment of the present invention, the step of calculating the gradient of the physical loss function with respect to the physical parameters to be identified through backpropagation using automatic differentiation techniques, and obtaining the high-precision aligned physical parameters through iterative optimization, includes: Obtain the actual observation trajectory within a fixed time window It uses real control commands to drive a differentiable engine to perform continuous forward dynamic simulation to obtain the simulation prediction trajectory sequence. ; Construct a physical loss function consisting of a weighted sum of position trajectory error and velocity trajectory error terms. : The physical loss function is calculated by backpropagation along the time axis with respect to the physical parameters to be identified, including friction, mass, center of gravity offset, and damping parameters. The total derivative of the physical loss function is used to update the gradient descent parameters using an adaptive learning rate optimizer until the physical loss function converges to the residual threshold and the aligned physical parameters are output.

[0012] As a preferred embodiment of the present invention, the low-frequency outer loop infers and outputs the target pose based on multimodal observation features, including: In each decision cycle, the environmental observation features of the current physical operation step are extracted. The environmental observation features are composed of digital twin visual representation vectors, the angle states of each joint of the robot body, and the aligned physical parameters. The historical observation sequence containing causal self-attention masks is input into the deep neural network policy model for forward inference, outputting the target 3D position reference of the end effector in the next time step. With target attitude quaternion reference .

[0013] As a preferred embodiment of the present invention, a first-stage time interpolator is provided between the low-frequency outer ring and the high-frequency inner ring, and the operation processing logic of the time interpolator includes: Within the time interval between two adjacent outer loop decisions, a smooth transition trajectory is constructed using the target pose of the two most recent cycles as the starting and ending points. The mechanism employs polynomial interpolation to construct position continuity for the target's 3D position reference, and spherical linear interpolation on a unit quaternion manifold to perform attitude transition for the target's attitude quaternion reference. This allows the high-frequency inner loop to query the corresponding smooth target pose and its first and second time derivatives from the time interpolator in each servo cycle. This mechanism effectively eliminates the reference signal discontinuity problem caused by cross-frequency data synchronization.

[0014] As a preferred embodiment of the present invention, the high-frequency inner loop performs sensorless compliant impedance control, which first requires calculating the six-dimensional pose deviation vector. The calculation steps include: Obtain the true end position calculated from the forward kinematics. With true pose quaternion ; Position deviation is calculated using vector subtraction. ; Calculate the attitude error quaternion The three-dimensional rotation error vector is extracted by taking the logarithm of the attitude error quaternion. ; The positional deviation is concatenated longitudinally with the three-dimensional rotation error vector to form the six-dimensional pose deviation vector containing six degrees of freedom. .

[0015] As a preferred embodiment of the present invention, the step of calculating the inverse dynamic residual torque based on the aligned physical parameters of the high-frequency inner loop and estimating the end contact force through the generalized pseudo-inverse inversion of the end effector kinematic Jacobian matrix includes: Read the measured joint drive torque of the current servo output And based on the aligned physical parameters, calculate the theoretical internal consumption torque including theoretical friction compensation. ; The inverse dynamic residual torque is obtained by subtracting the measured joint driving torque from the theoretical internal consumption torque. ; Using the Jacobian matrix of the end effector kinematics Generalized pseudo-inverse back projection calculation of software-driven end contact force estimation : Substituting the estimated end contact force into the dynamic equation of virtual impedance behavior It calculates and outputs a target driving torque command with directional compliance, wherein , , These are the positive definite virtual inertial mass, virtual damping, and virtual stiffness matrices in the six degrees of freedom of space, respectively.

[0016] As a preferred embodiment of the present invention, the sensorless compliant impedance control based on the estimated end contact force further includes a safety protection mechanism based on anti-collision monitoring logic: Real-time calculation of the L2 norm of the estimated end contact force ; When the L2 norm is determined to exceed the preset safety force threshold scalar, an emergency interruption flag is thrown, the current target pose is overwritten with the safety buffer position of the previous cycle, and a bounce back is executed to terminate torque accumulation and avoid hardware overload.

[0017] Compared with the prior art, the present invention has the following beneficial effects: This invention transforms the impenetrable simulation engine black box in traditional system identification into a differentiable operator link that supports backpropagation by constructing a fully differentiable physical dynamics computation graph. This allows the analytical gradient of the trajectory error relative to the hidden physical parameters to be directly calculated, thus replacing the blind random search process in gradient-free methods with a deterministic gradient descent path. This results in a substantial improvement in both the convergence speed and estimation accuracy of physical parameter identification. Consequently, the aligned digital twin system can approximate the real environment with high precision at the level of contact physics and dynamics properties, making the robot operation parameters trained in the twin environment reliable for deployment in the real world.

[0018] This invention integrates 3D Gaussian scene representation and differentiable rasterization rendering technology into a digital twin system. It utilizes the pixel-level photometric error gradient between real camera images and twin-rendered images to generate virtual visual forces, which are then injected into the dynamic equations of the physics engine. This achieves closed-loop state synchronization between physical and twin spaces at a frequency of 60 Hz, eliminating the unidirectional disconnect between simulation and reality found in traditional solutions. At the control execution level, this invention uses precisely calibrated dynamic model parameters from a differentiable physics engine. Through inverse dynamic residual torque calculation and pseudo-inverse projection of the Jacobian matrix, it achieves pure software estimation of the end-effector contact force. This endows the robot with compliant impedance control capabilities without relying on external torque sensor hardware. Furthermore, a nested double-loop architecture of growth and metabolism loops coordinates the frequency difference between global strategy planning and local force compensation, enabling the robot to adaptively adjust its operational parameters within millisecond timescales when facing unknown environmental disturbances. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the method flow of a preferred embodiment of the present invention.

[0020] 100. Digital twin vision synchronization module; 200. Differentiable physical parameter identification module; 300. Multimodal dual-loop control module. Detailed Implementation

[0021] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0022] like Figure 1As shown, this invention provides a method for optimizing the operational parameters of an embodied intelligent robot based on digital twins. This method establishes a unified closed-loop optimization framework from reality to simulation and back again. The technical concept of this framework lies in transforming the traditional unidirectional, open-loop embodied intelligent training and deployment process into a multimodal data-driven bidirectional feedback system with a built-in differentiable computing engine at its core. The system injects temporal observation data from the physical world into the end-to-end differentiable digital twin system, and uses automatic differentiation technology to identify and calibrate the underlying physical parameters in reverse. Based on this, the system uses the aligned high-fidelity digital twin environment to forward deduce the operational strategy. In the real physical operation stage, relying on 3D Gaussian visual rendering and inverse dynamics calculation, a purely software-driven dual-loop control structure is constructed to achieve real-time flexible optimization and execution of operational parameters. The system embodiment of this invention is logically decoupled into three core execution modules: a digital twin visual synchronization module 100, a differentiable physical parameter identification module 200, and a multimodal dual-loop control module 300. The digital twin vision synchronization module 100 is responsible for the 3D Gaussian scene representation of the working environment and high-frequency state synchronization based on photometric feedback virtual force; the differentiable physical parameter identification module 200 is responsible for reconstructing the rigid body dynamics simulation process into a fully differentiable computational graph and completing the gradient optimization identification of implicit physical parameters; the multimodal dual-loop control module 300 is responsible for integrating multimodal perception representation and sensorless analytical Jacobian observation, realizing policy inference and compliant servo control with a nested growth loop and metabolism loop architecture. Each of the three modules undertakes different technical tasks and is tightly coupled and collaborative through data flow, which will be described in detail below.

[0023] Before proceeding with the mathematical descriptions of the subsequent modules, it is necessary to establish a unified convention for the core mathematical notation system used throughout the text. In this specification, the symbols... and its derivative It is always used exclusively to represent the generalized coordinates and generalized velocity tensors of robot joints; the attitude quaternions of rigid bodies or end effectors are uniformly represented by the symbol... This is to avoid ambiguity with joint coordinates. Furthermore, when there are Jacobian matrices with different physical contexts between modules, [the following is used]... Indicates the point of contact during the collision, Jacobi, The Jacobian kinematics of the end effector is represented by the Jacobian, and the dimensions and physical meanings of the two will be given in their respective contexts.

[0024] The digital twin visual synchronization module 100 is responsible for establishing a 3D scene representation of the embodied intelligent robot's working environment and ensuring that the physical space and the twin space remain synchronized in geometric and visual dimensions during strategy execution and deployment. This module's operational logic consists of three phased steps.

[0025] During the cold start phase of robot operation, the digital twin vision synchronization module 100 performs embodied 3D Gaussian initialization and reconstruction of the work scene. Specifically, a multi-view color camera mounted on the robot's head or workbench acquires a multi-view image frame sequence covering the entire workspace, and the intrinsic and extrinsic parameter poses of the camera are calibrated using the structure-of-motion (COMO) algorithm, while simultaneously generating a sparse 3D point cloud. In this embodiment, the COMO algorithm can utilize an open-source implementation scheme for feature point matching and bundle adjustment to obtain robust camera parameter estimation and a sparse spatial structure. Based on this sparse point cloud, a set of 3D Gaussian particles is initialized. For any Gaussian particle in the 3D Gaussian particle set... Its software data structure needs to explicitly define and store the following parameters: three-dimensional position mean vector The covariance matrix is ​​used to describe the spatial location of the particle in the world coordinate system. To ensure the positive semidefiniteness of the covariance matrix, it is decomposed into a rotation quaternion in the code implementation. and a three-dimensional scaling vector The composite representation, i.e. ,in This is the mapping function from quaternions to rotation matrices. This represents the element-wise multiplication operation of vectors, i.e. The result is with Vectors of the same dimension, where each component is the square of the corresponding component; opacity parameter. These parameters control the particle's contribution weight to the rendered image; and the spherical harmonic function coefficients characterize the viewpoint-dependent color appearance, allowing the same particle to exhibit near-realistic lighting variations under different viewing angles. After initialization, these parameters will serve as optimizable tensors in the forward derivation and backward differentiation of the subsequent computational graph.

[0026] To endow the Gaussian particles represented purely by vision with physical rigid body interaction characteristics, the digital twin visual synchronization module 100 further performs a deep binding operation between the visual particles and the physical dynamics engine entities. The system uses a 3D instance segmentation network to separate Gaussian particles belonging to different operational objects in the scene according to semantic categories. For example, the particle subsets belonging to the desktop, the workpiece to be grasped, and obstacles are extracted independently. The 3D instance segmentation network can be implemented using point cloud or voxel-level segmentation models with real-time inference capabilities. Its purpose is to explicitly label objects with independent degrees of freedom of motion in the scene through particle clustering. After extracting the Gaussian particle subset belonging to the movable workpiece, the surface contours are mapped to the rigid body collision mesh surface within the differentiable physics engine using a voxelization algorithm and a differentiable moving cube algorithm. The so-called differentiable moving cube algorithm introduces differentiable vertex interpolation operations on the basis of traditional isosurface extraction methods, allowing gradient transfer in the transformation process from the Gaussian particle density field to the triangular mesh. Through the above mapping, a corresponding collider instance is created for each operable object within the differentiable physics engine. Let the state vector of a rigid body in the physics engine be... ,in For three-dimensional centroid coordinates, If we consider the attitude quaternion, then the mean coordinate update of all Gaussian particles associated with the rigid body follows the rigid body transformation equation: in This represents the initial position of the particle in the object's local coordinate system. This mathematical mapping ensures that the rigid body motion driven by the physics engine can synchronously drive the pose update of particles in the visual rendering layer, thereby achieving deep coupling between the visual scene representation and the state of the physics simulation engine at the computational graph level.

[0027] During the real-time operation phase, the digital twin vision synchronization module 100 establishes a closed-loop photometric error correction mechanism with a period of 60 Hz. Let... Real-time environmental observation images captured by the real-time camera are Digital twin systems are based on The object state output by the physics engine at any given time Update the pose of all Gaussian particles and render the corresponding twin image from the same virtual perspective as a real camera using a differentiable rasterizer renderer. The differentiable rasterizer is a core component of 3D Gaussian splashing technology. Its basic principle involves projecting 3D Gaussian particles onto a 2D imaging plane, then performing transparency blending according to depth order to generate pixel-level color output. All operations in the entire rendering pipeline are defined as differentiable operators. A pixel-level photometric error loss function is constructed within the automatic differential computation graph. The loss function is a weighted combination of absolute value norm distance and structural similarity index: Among them, weight parameters In engineering practice, the typical value is... The structural similarity index is used to balance the contribution ratio between pixel-wise differences and structurally perceived differences. It measures the perceptual consistency between two images from three dimensions: brightness, contrast, and structure. Compared with a simple pixel-wise distance metric, it has a stronger ability to discern local structural perturbations.

[0028] After the photometric error loss function is constructed, the digital twin visual synchronization module 100 executes the crucial backpropagation logic step, calling the backpropagation mechanism of the computation graph to determine the spatial position of the loss function with respect to the Gaussian particle. Calculate the partial derivative to generate the feedback correction vector, i.e., the virtual visual power: in The damping gain parameter, used to control the correction strength, has a physical meaning similar to the gain coefficient in a proportional controller. An excessively large value will cause oscillation, while an excessively small value will result in tracking lag. In practical deployments... The value needs to be adjusted comprehensively based on camera resolution, scene scale, and the time step of the physics engine. A feasible range of values ​​is... to Magnitude. The resulting discrete virtual visual power. By summing over all related particles Obtain the resultant planar force acting on the object's center of mass, and sum it using the cross product. The resultant torque relative to the center of mass is obtained, and the two together constitute the external additional torque. Injected into the next timing step of the physics engine In the dynamic equations, this mechanism is similar to a visual spring in a mechanically equivalent sense, continuously forcing the twin object that deviates from the real-world state back to its aligned position. This ensures that even when external disturbances cause unexpected displacement of the object, the digital twin system can still autonomously complete state correction within tens of milliseconds, achieving the ability to continuously track the dynamic evolution of the real environment.

[0029] It should be noted that the aforementioned virtual force correction mechanism based on pixel-level photometric gradients may attenuate to the point where it cannot effectively drive correction when the manipulated object undergoes large-scale displacement, resulting in a severe lack of pixel overlap between the rendered and real images. To enhance the robustness of this mechanism under large deviation conditions, the digital twin visual synchronization module 100 introduces a multi-resolution image pyramid strategy as an aid in its engineering implementation: before calculating the photometric error, the system downsamples the real image and the twin rendered image to multiple resolution levels to form a pyramid. Gradients are calculated starting from the coarsest level to drive coarse-grained pose correction, gradually transitioning to sub-pixel alignment at fine resolution levels. When the photometric gradient at the coarsest level still cannot provide an effective correction direction, the system can further backtrack to a geometric constraint loss term based on sparse feature point matching, using identifiable corner points or edge features in the scene to establish correspondences and re-lock the pose estimation of the object in the twin space through a global search. This multi-level fault-tolerant strategy ensures that the virtual visual force mechanism maintains convergence across a continuous range of conditions from small perturbations to large deviations.

[0030] At the engineering implementation level, the digital twin vision synchronization module 100 runs on an edge graphics processor computing node, receiving real-time multimodal sensor data tensors pushed by cameras at the work site, with a typical dimension of [missing information]. The system generates color image frames. Within each synchronization cycle, the twin rendering pipeline synthesizes viewpoint-consistent predicted frames in real time and uses parallel computing kernel functions to synchronously calculate the full-frame photometric error matrix. When the photometric error exceeds a preset tolerance threshold, it indicates a deviation in the physical state of the environment, such as external forces causing the manipulated object to slide outside the robot's control commands. In this case, the algorithm activates the virtual correction force calculation process. If the photometric error is within the tolerance range, the vision lock is considered normal, and the partial derivative calculation step is skipped to avoid consuming graphics processor computing power. The model state is updated solely based on the rigid volume step-by-step function at the physics engine's underlying level. This conditional judgment logic effectively reduces the peak occupancy of computing resources while ensuring synchronization accuracy.

[0031] During the cold start phase, the software needs to evaluate the peak signal-to-noise ratio (PSNR) between the rendered initial virtual scene image and the real image. When the PSNR is below 30 decibels, the spatial reconstruction accuracy is deemed insufficient. The system prevents the state machine from flowing downstream and issues an interrupt alarm through the human-machine interface, prompting the operator to supplement the multi-view image sequence or check the lighting conditions. When the PSNR meets the standard, the system is allowed to enter standby mode and persistently saves the static digital twin parameter file of the workspace, including the weight vector of Gaussian particles, geometric metadata, and the loaded differentiable engine runtime context object.

[0032] The differentiable physical parameter identification module 200 constitutes the core support of the technical solution of this invention. Its main function is to solve the calibration problem of environmental parameters in the physics engine, including the accurate identification of hidden parameters such as friction, mass, and damping. In this module, the entire rigid body dynamics simulation process is completely reconstructed into a series of operators that can transmit gradients. The implicit physical parameter vector to be identified and optimized by the differentiable physical parameter identification module 200 is set as follows: The parameters included cover the static and dynamic friction coefficients of the working surface, the mass of the workpiece and its center of gravity offset, the nonlinear viscous damping inside each joint of the robot, and the motor control gain. In traditional practices, these parameters are often manually preset by engineers based on experience or obtained through offline static calibration. However, such static modeling cannot capture the nonlinear dynamic characteristics that evolve with temperature, wear, or load changes in real physical interactions.

[0033] The differentiable physics parameter identification module 200 constructs a complete multi-rigid-body dynamics model including joint friction terms and a contact calculation model based on a linear complementarity problem at the forward dynamics level. In the twin physics engine, the robot's generalized coordinate tensor is defined as... The generalized velocity tensor is The sequence of externally applied driving torques is as follows According to the multi-rigid-body dynamics equations, the system's motion can be described as follows: in The system inertia matrix is ​​determined by the combination of the mass of each link, the inertia tensor, and the kinematic Jacobian. The matrix characterizing the Coriolis force and centrifugal force effects; It is the gravity term vector; The joint friction torque term is mathematically represented using the Coulomb viscous friction model. ,in and These are the independent Coulomb friction coefficient vector and viscous friction coefficient vector for each joint, respectively. This represents element-wise multiplication, and both are contained within the parameter vector to be identified. In particular, it should be noted that due to the sign function... It is not differentiable at zero, and in the implementation of differentiable computational graphs, it is replaced by a differentiable smooth approximation function, such as the hyperbolic tangent function. ,in To control the positive coefficients for the approximate steepness, this smooth approximation is also applied element-wise to each component of the velocity vector. Let be the Jacobian matrix of the object at the collision detection contact point. The number of contact degrees of freedom; This represents the unknown contact impulse vector generated at the corresponding contact point. The joint friction torque term is explicitly incorporated into the forward dynamic equation and participates in the subsequent automatic differentiation calculation, thereby ensuring that the gradient link of the loss function with respect to joint damping and friction-related parameters is not broken, allowing these parameters to be effectively identified through backpropagation.

[0034] To address the most challenging contact constraint handling problem in embodied intelligent operations, the differentiable physical parameter identification module 200 employs an implicit Euler integrator to discretize the continuous dynamic equations and transforms the contact problem into a linear complementarity problem for solution. Let a small time step be assumed. The contact point velocity is Based on a first-order Taylor expansion, it can be expressed as a function of the contact impulse. Affine function form: Where the mapping matrix It contains composite information about the system's mass distribution and contact geometry, and the bias term vector. It includes a free motion prediction component determined by external forces, gravity, frictional torque, and current velocity. To conform to physical reality, the contact impulse must satisfy the non-penetration condition and the Coulomb friction cone constraint. The system needs to solve a tensor solution that satisfies the following complementary constraints. and : Subscript and These represent the contact normal and tangential components, respectively. Let be the Coulomb coefficient of friction. The first condition requires that the normal contact force is non-negative and that the normal contact force and the normal separation velocity are complementary and zero, that is, the force is zero when there is no contact and there is no penetration when there is contact; the second condition confines the tangential friction force within a friction cone with the normal force as its radius, and when the object is in a sliding state, the friction force is obtained exactly on the boundary of the cone.

[0035] At the code implementation level, the differentiable physical parameter identification module 200 employs either the predictive-corrected principal-dual interior-point method or the projective Gauss-Seidel iterative method to quickly solve the aforementioned inequality constraint equations. To enable gradient propagation during this solution process, the entire iterative process is encapsulated as a custom differentiable function within the automatic differentiation framework. This custom differentiable function performs standard numerical solutions to the complementarity problem during forward computation, while during backpropagation, it analytically calculates the Jacobian matrix of the equation output with respect to the input parameter matrix using implicit function theorems. Specifically, for implicit equations satisfying the complementarity conditions... According to the implicit function theorem, we can obtain: This analytical gradient expression avoids stepwise expansion and differentiation of the iterative solution process itself, thus ensuring both computational efficiency and maintaining the numerical accuracy of the gradient. Through this technique, the entire forward integration process, from external driving input to contact force output and then to object state update, including all differentiable operators such as joint friction terms, forms a complete and unbroken backward propagation chain within the automatic differentiation framework.

[0036] In the specific operation of physical parameter calibration, the differentiable physical parameter identification module 200 drives the robot to perform a series of specifically designed or randomly generated exploratory physical actions in safe mode, such as pushing an object on a workbench, lifting and lowering a workpiece with controlled force, and other non-destructive action sequences. The system records a fixed time window. The high-frequency time-series state data sequence derived from real sensors is defined as the real observation trajectory. Simultaneously, force the differentiable twin engine to be configured in The initial physical state is completely identical to the real state, based on the control commands recorded by the real system. Drive the differentiable engine to perform continuous The forward dynamic simulation of the step acquires the parameters based on the current guess. Generated simulation prediction trajectory sequence .

[0037] Differentiable physical parameter identification module 200 constructs a comprehensive loss function for physical parameter optimization. This function mainly consists of position trajectory error and velocity trajectory error terms: in and To balance the empirically normalized scale weights of different dimensions, the position error and velocity error are made to be on the same order of magnitude, preventing either from dominating the gradient calculation. In practical engineering, the position weight is usually set to the unit order of magnitude, while the velocity weight is scaled accordingly based on the resolution and velocity order of the joint encoder.

[0038] On the computation graph containing all the aforementioned temporal physical state evolution operators, the differentiable physical parameter identification module 200 triggers an automatic differentiation engine based on time backpropagation to calculate the comprehensive loss function. Relative to the attribute parameter vector to be identified Total derivative: The calculation path for the total derivative spans the entire time-series simulation unfolding process, from the final time step... Starting from the state error, the gradient signal is progressively transmitted backward along the time axis, traversing the integrator operator, contact force solver, joint friction calculation node, and collision detection module at each step, finally reaching the underlying physical parameter node to be identified. This is due to the joint friction torque term... It has been explicitly modeled in the forward dynamic equations as about The differentiable function allows the backpropagation process to naturally pass the gradient to the Coulomb friction coefficient vector through this node. , viscous friction coefficient vector In addition to other damping parameters, this ensures that joint friction characteristics are identified during gradient optimization. Compared to traditional gradient-free optimization methods, this end-to-end gradient propagation essentially transforms blind random search into deterministic descent along the gradient direction of the loss function, resulting in an order-of-magnitude improvement in convergence efficiency.

[0039] Then, an adaptive learning rate optimizer is used to update the gradient descent parameters, and the update law is expressed as: in This represents the iteration round number. In this embodiment, an adaptive moment estimation optimizer with a weight decay term is preferably used. This optimizer exhibits robust convergence characteristics in high-dimensional parameter spaces with sparse gradients or high noise. The iterative process runs continuously in the background of a workstation or edge server, and the loss function is evaluated... The decay decreases to the preset convergence threshold. Updates stop when the time is right. In this embodiment, the upper limit threshold for iterations is set to 500 rounds, and the convergence residual threshold is... Set as In each round, the system performs forward simulation to calculate the current temporal loss scalar. If the loss value is lower than the convergence threshold, the loop is exited directly; otherwise, the framework's backpropagation interface is called to extract the gradient tensor and perform incremental parameter updates. If the iteration exceeds the limit and convergence is not achieved due to singular data or parameters entering the ill-conditioned region, the system will output a diagnostic log and reset the parameters to conservative safe values ​​to ensure that the downstream control module does not receive significantly distorted physical model parameters.

[0040] When the iteration converges normally, the output is the optimal solution. This means that the digital twin system has achieved high-precision alignment with the current real-world environment at the level of contact physical and dynamic properties. This alignment explicitly covers inertial parameters, contact surface friction coefficients, and internal joint friction and damping parameters. Compared to traditional system identification methods based on gradient-free heuristics such as simulated annealing, genetic algorithms, or Bayesian optimization, the differentiable dynamic computational graph scheme of this invention significantly reduces the absolute error of physical parameter estimation and greatly improves the convergence speed. The aligned optimal parameter vector... The parameters will be fixed and instantly transmitted into the memory pool of the downstream multimodal dual-loop control module 300, overriding the default configuration parameters, and providing accurate model support for inverse dynamics compensation in the control law.

[0041] The multimodal dual-loop control module 300 defines the complete control layer software logic architecture of the robot from perception to servo execution after parameter calibration and actual task deployment. To overcome the vulnerability of pure position control strategies in contact-rich tasks during imitation learning, and to significantly reduce the hardware cost of external force sensors, the multimodal dual-loop control module 300 adopts a nested growth loop and metabolism loop dual-loop control structure.

[0042] The outer loop control section of the multimodal dual-loop control module 300, referred to as the growth loop, operates at a frequency of approximately 10 to 20 Hz and is responsible for high-order multimodal policy inference. The growth loop establishes a deep neural network policy model equipped with a causal self-attention mask; in this embodiment, it is implemented using a sequence modeling architecture based on a self-attention mechanism. This policy model extracts environmental observation features for the current physical operation step within each decision cycle. The input data is composed of the following multi-source information: the representation vectors extracted by the encoding network from the global scene view and local hand-eye view images rendered by the digital twin vision synchronization module 100, and the angle states of each joint of the robot body. And the optimal physical context parameters for online convergence are identified in the differentiable physical parameter identification module 200. By explicitly including physical parameters as part of the policy network input, the model gains prior knowledge of the current workpiece's quality, friction characteristics, etc., enabling zero-shot or few-shot generalization transfer across different work objects. Multimodal observation feature sequences The input is fed into the policy network for forward reasoning, where Given the historical observation window length, the causal self-attention mask ensures that the network is at all times. The output depends only on It accesses observations from previous moments without accessing future data, thus ensuring causal consistency during online deployment. The output tensor of the policy network is defined as the expected macroscopic target pose of the robot at the next time step, including the three-dimensional position reference of the end effector in Cartesian coordinates. With attitude quaternion reference Since the network weights have been optimized through large-scale training in early high-fidelity twin environments via imitation learning, their output trajectories are task-appropriate on a global scale.

[0043] Since the decision cycle of the growth loop is approximately 10 to 20 Hz, while the servo cycle of the downstream metabolic loop is 500 to 1000 Hz, their update frequencies differ by tens of times. If the metabolic loop directly uses the target pose output by the growth loop in each high-frequency servo cycle without any smoothing, the reference trajectory will exhibit a step change. The first and second time derivatives of the deviation vector in the impedance control law will experience a pulse-like surge at the step transition, triggering a transient impact on the joint torque. In practical hardware, this can easily trigger the overload protection of the servo driver. To address this, the multimodal dual-loop control module 300 sets a time interpolator based on cubic splines or fifth-order polynomials between the growth loop output and the metabolic loop input: whenever the growth loop at time... Release new target pose At this time, the interpolator uses the target pose of the previous decision cycle as the starting point and the current new target as the ending point, constructing a smooth transition trajectory in which both position and velocity are continuous within the time interval between two adjacent decisions. For the position component... Polynomial interpolation can be directly used to ensure the continuity of position and its derivatives; for attitude components... Since the quaternion space is not Euclidean space, the interpolator uses a spherical linear interpolation method to smoothly transition on the unit quaternion manifold, ensuring that the attitude trajectory does not deviate from the legal rotation group constraints. The metabolic loop queries the interpolator for the target pose and its first and second time derivatives at the current moment during each high-frequency servo cycle, using these as the smoothing input signal for the impedance control law. This design eliminates the reference signal discontinuity problem caused by cross-frequency data synchronization at its source.

[0044] The inner loop control section of the multimodal dual-loop control module 300 is called the metabolic loop, operating at a frequency of 500 to 1000 Hz. It is responsible for sensorless compliant impedance control based on inverse reconstruction of dynamic equations. The target pose output by the growth loop cannot be directly used in hazardous work scenarios involving rigid contact, because simple position tracking control would generate a huge impact force at the moment of contact, which could easily cause overload damage to the robotic arm or breakage of the manipulated object. The core task of the metabolic loop is to convert pose commands into compliant joint drive torque commands and send them to the servo motors.

[0045] The metabolic loop models the contact relationship between the robot's end effector and its environment as a virtual impedance dynamic system. To accurately calculate the end effector pose deviation, the position and attitude components need to be processed separately. Let the current true position of the end effector be... The current true pose quaternion is From the current joint angle, by positive kinematics The solution is obtained; the corresponding reference quantity is the output of the interpolator. and Positional deviation can be obtained directly through vector subtraction. Attitude deviations cannot be simply calculated by subtracting quaternions, because the algebraic difference of quaternions has no geometric meaning on rotation groups. This invention employs a combination of quaternion error multiplication and logarithmic mapping to extract the three-dimensional rotation error vector: first, the attitude error quaternion is calculated... ,in This is quaternion multiplication. To find the conjugate of the reference quaternion; then take the logarithmic mapping of the error quaternion. This mapping projects the unit quaternion onto its tangent space to obtain a rotational error vector equivalent to the three-dimensional angular velocity. The position deviation and attitude deviation are then concatenated longitudinally to define a six-dimensional pose deviation vector. The desired impedance behavior is described in this six-dimensional bias space as follows: in These are all positive definite diagonal matrices set by software, representing the desired virtual inertial mass, virtual damping coefficient, and virtual stiffness coefficient in the six degrees of freedom of space, respectively. By adjusting the stiffness parameter values ​​in the contact direction, a compliant contact effect can be achieved—reducing the stiffness value in the direction where contact and collision are expected allows the end effector to produce a certain amount of compliant offset rather than rigid resistance when subjected to external forces; while maintaining high stiffness in the direction of free motion to ensure position tracking accuracy. It represents the six-dimensional physical contact and interaction force between the end tool and the environment, with the upper three components corresponding to linear forces and the lower three components corresponding to stress moments.

[0046] Without the addition of a six-dimensional torque sensor at the end, the multimodal dual-loop control module 300 estimates the contact force using an analytical algorithm. This constitutes a key breakthrough in the control logic layer of this invention. The current loop sensor at the bottom layer of the robot servo motor can read the nominal measurement value of the current actual output joint drive torque at an extremely high frequency. Its mathematical structure is as follows: The first three terms on the right-hand side of the equation represent the internal torque required to overcome the robot arm's own inertia, Coriolis force, and gravity; the fourth term... The fifth term refers to the nonlinear dissipation torque generated by internal friction in the joint motor. For the end contact force via the end effector kinematics Jacobi The equivalent torque contribution mapped to joint space. The Jacobian matrix used here. The standard end effector kinematics Jacobian has a fixed number of six rows (corresponding to the six-dimensional force spinor of the end effector), which is the same as the collision point Jacobian used in the differentiable physical parameter identification module 200. They differ in both physical meaning and dimension. Thanks to the differentiable physical parameter identification module 200, all hidden dynamic parameters have been identified in the preliminary steps. The precise identification and feedback to the controller framework enable the control algorithm in the metabolic loop to calculate the theoretical internal consumption torque in the current state with extreme accuracy. The terms marked with a superscript cap indicate that their calculations rely on parameters derived from a precisely calibrated dynamic model. Of particular note is the term concerning joint friction torque. The Coulomb friction coefficient and viscous friction coefficient are precisely calibrated by the differentiable physical parameter identification module 200 during the gradient optimization process, along with the inertial parameters and contact friction parameters, forming a tight data causal chain. By subtracting the theoretical model torque from the measured torque, the residual joint torque perturbation vector caused entirely by the contact reaction of the external environment can be obtained: The residual moment vector contains the projected components of the end-effector contact force transmitted to each joint via the kinematic chain. This is achieved using the current end-effector kinematic Jacobian matrix. The generalized pseudo-inverse operator is obtained through singular value decomposition, and the software estimate of the end contact moment is calculated by back-projection. in This represents the generalized pseudoinverse operation based on singular value decomposition. Because... Its pseudo-inverse is ,and The output dimension after multiplication is The defined dimensions are strictly consistent with those of the six-dimensional contact force. For a redundant degree-of-freedom manipulator, the pseudo-inverse operation provides the optimal solution in the least squares sense, and by setting a truncation threshold for singular values, numerical divergence can be effectively avoided when approaching kinematic singular configurations. The estimated contact force obtained from the above calculations... Substituting the readings from the real force sensor into the aforementioned desired impedance control law, the corrected joint target driving torque command sequence is calculated. It sends data to the joint servo driver in real time at a communication bus cycle of over 500 Hz.

[0047] The effectiveness of this sensorless force estimation scheme directly depends on the accuracy of the dynamic model parameters. In traditional schemes, due to significant identification errors in inertial and friction parameters, the difference between the theoretical and actual torques calculated by the model is mixed with a large amount of modeling error noise, making it impossible for the residual torque to accurately reflect the true level of external contact force, and the force estimation results often lack practical value. However, within the technical framework of this invention, the differentiable physical parameter identification module 200 has aligned all dynamic parameters, including joint friction, to extremely high accuracy through end-to-end differentiable gradient descent optimization. The theoretical torque calculated by the model can approximate the actual joint torque without external force with a very small error, thus the external contact force signal in the residual component can be cleanly separated. This logical chain constitutes the technical basis for deep collaboration among the three modules.

[0048] Regarding safety assurance during task execution, the multimodal dual-loop control module 300 is equipped with safety anti-collision monitoring logic with the highest interrupt privileges. When the estimated external contact force L2 norm... Exceeding the preset safety force threshold scalar If the controller determines that an abnormal hard collision has occurred or the system is stuck, the system immediately raises a hardware-level emergency interrupt flag, overwrites the current end effector pose target with the safe buffer position of the previous cycle, and performs a bounce protection back to terminate subsequent torque accumulation, thus avoiding injury to the robotic arm or personnel. Regarding task completion determination, when the Euclidean distance between the current pose and the target pose of the end effector is less than the success margin threshold and the speed of each joint is lower than the rest margin threshold, the system determines that the workpiece has smoothly reached the specified target and reached a stationary state as required by the task, sends a task completion flag to the host computer, and smoothly exits the main control loop.

[0049] From the perspective of system deployment and computing resource allocation, the three modules mentioned above operate collaboratively across different hardware computing layers. The digital twin visual synchronization module 100 is deployed on an edge graphics processor node, processing the real video stream and updating the twin particle positions in real time at a high frequency of 60 Hz. Its core algorithm logic is essentially based on partial derivative calculations of pixel photometric errors and virtual force driving. The differentiable physics parameter identification module 200 runs on a cloud or workstation graphics processor cluster, processing real historical joint time-series trajectories at an offline trigger or an extremely low frequency of approximately one Hz and outputting global physics parameters. Its core algorithm logic is essentially transforming a linear complementarity problem into a differentiable operator and then performing gradient descent optimization. The growth loop in the multimodal dual-loop control module 300 runs on the embedded main control neural processing unit, processing twin representation features and pose parameters with a mid-to-low frequency decision cycle of 10-20 Hz and outputting the downstream target pose. Its core algorithm logic is essentially autoregressive timing generation based on a self-attention mechanism. The metabolism loop runs on the real-time system motion controller, processing pose tracking errors and inverse dynamic residual torque with an ultra-high frequency servo cycle exceeding 500 Hz and outputting motor commands. Its core algorithm logic is essentially analytical reconstruction of contact force and compliance impedance compensation under sensorless conditions. The modules exchange status and parameters via a high-speed data bus. Cross-frequency data exchange points are equipped with the aforementioned interpolators or a zero-order hold plus low-pass filtering synchronization mechanism to ensure signal continuity and causal consistency. The optimal physical parameters output by the differentiable physical parameter identification module 200 are... It is a quasi-static quantity that only changes when physical calibration is completed or when an incremental update is triggered. Its update frequency is much lower than the servo cycle of the control loop. Therefore, it can be regarded as a constant on the time scale of the metabolic loop without introducing additional discontinuities.

[0050] In a specific application scenario, the method described in this invention can be applied to high-precision shaft-hole assembly operations. The robot needs to insert a metal pin with sub-millimeter tolerances into the corresponding hole. During the cold start phase, the digital twin vision synchronization module 100 reconstructs a 3D Gaussian scene of the worktable, pin, and hole parts through multi-view image acquisition, and binds the subset of Gaussian particles corresponding to the pin to the rigid body collider within the physics engine. During the exploration and calibration phase, the differentiable physics parameter identification module 200 drives the robotic arm to perform pushing and light-touching actions on the pin. Gradient descent optimization is performed based on the error between the recorded joint trajectory and the simulated trajectory to accurately identify the pin's mass distribution, the friction coefficient of the hole wall, and the damping parameters of each joint. During the actual insertion phase, the growth loop of the multimodal dual-loop control module 300 makes decisions and plans based on the visual features of the twin rendering and the identified physical parameters, outputting a target pose sequence that guides the pin to gradually approach the hole and align with its posture. The metabolic loop detects the contact force generated by the inverse dynamic residual torque at the moment the pin contacts the hole wall, automatically reducing the virtual stiffness parameter in the insertion direction and increasing lateral compliance. This allows the pin to slide in along the hole axis under the guidance and constraint of the hole wall, preventing excessive lateral force from causing jamming or damage to the part due to rigid position control. Throughout the process, the digital twin vision synchronization module 100 continuously monitors the scene status at a frequency of 60 Hz. If the pin deflects or slips unexpectedly, it immediately uses virtual visual force to pull the pin state in the twin space back to align with the real state, ensuring that the growth loop's decisions are always based on accurate environmental perception.

[0051] In another application scenario, the method described in this invention is also applicable to friction gripping operations on objects of unknown materials. When the robot faces an object it has never encountered before, the differentiable physical parameter identification module 200 can quickly identify the object's mass and surface friction characteristics within hundreds of iterations. Based on this, the metabolic loop of the multimodal dual-loop control module 300 adjusts the gripping force of the grippers within a millisecond timescale to prevent the object from slipping due to insufficient force or deforming and damaging the object due to excessive force.

[0052] In another application scenario, the method described in this invention can be applied to the insertion and removal operations of flexible cables. The dynamic characteristics of flexible objects are far more complex than those of rigid bodies, and their deformation behavior is highly sensitive to the coefficient of friction and damping parameters. The differentiable physics engine in the differentiable physics parameter identification module 200 can model the flexible body by discretizing it into multiple rigid segments and connecting them with constrained joints. The bending stiffness and torsional damping parameters between each segment are also identified through gradient optimization, thereby constructing a simulation model in a twin environment that highly matches the mechanical behavior of real cables. The metabolic loop of the multimodal dual-loop control module 300 senses the changes in contact force between the cable and the connector in real time during the cable insertion and removal process. It adaptively adjusts the end motion speed and force direction according to the impedance model to ensure a smooth insertion and removal process without damaging the connector.

[0053] It should be noted that the operating frequency and specific parameter values ​​of the above modules are typical configurations in the preferred embodiment. Those skilled in the art can adaptively adjust these parameters based on the computing power, communication bandwidth, and operational accuracy requirements of the actual robot hardware platform, without departing from the core technical concept of this invention. For example, the synchronization frequency of the digital twin visual synchronization module 100 can fluctuate within the range of 30 to 120 Hz depending on the camera frame rate and the graphics processor's computing power; the convergence threshold of the differentiable physical parameter identification module 200 can be adjusted according to the task accuracy requirements. to Within a selectable range, the operating frequency of the metabolic loop in the multimodal dual-loop control module 300 depends on the communication bus specifications of the servo driver, and the control effect described in this invention can be achieved within the range of 250 to 2000 Hz. Furthermore, the specific architecture of the policy network is not limited to a model based on a self-attention mechanism; using other deep network architectures with temporal modeling capabilities is also an equivalent implementation within the scope of this invention. Similarly, the underlying tensor computation framework of the differentiable physics engine is not limited to a specific automatic differentiation library; any computational framework that supports custom gradient operators and backpropagation can be used to implement the technical solution described in this invention.

[0054] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A digital-twin-based embodied intelligent robot job parameter optimization method, characterized in that, Includes the following steps: A three-dimensional digital twin representation of the work scene is constructed. Virtual visual force is generated based on the photometric error gradient between real camera images and twin rendered images. The virtual visual force is then injected into the dynamic equation of a differentiable physics engine to achieve real-time state synchronization between physical space and twin space. The multi-rigid-body dynamics simulation process is reconstructed into a fully differentiable computational graph to obtain the forward simulation trajectory based on the robot's actual execution trajectory and control commands. The gradient of the physical loss function with respect to the physical parameters to be identified is calculated through backpropagation using automatic differentiation technology. The high-precision aligned physical parameters are obtained through iterative optimization. Using the aligned physical parameters, an adaptive adjustment and execution control of the operation parameters are performed through a nested dual-loop control architecture. The low-frequency outer loop infers and outputs the target pose based on multimodal observation features, while the high-frequency inner loop calculates the inverse dynamic residual torque based on the aligned physical parameters, estimates the end contact force through the generalized pseudo-inverse inversion of the end effector kinematic Jacobian matrix, and performs sensorless compliant impedance control based on the estimated end contact force.

2. The method according to claim 1, characterized in that, The construction of the three-dimensional digital twin representation of the work scene includes: A multi-view image sequence covering the workspace is acquired to generate a sparse point cloud, and a three-dimensional Gaussian particle set is initialized based on the sparse point cloud. The parameters of each Gaussian particle include a three-dimensional position mean vector, a covariance matrix represented by a combination of a rotation quaternion and a three-dimensional scaling vector, an opacity parameter, and spherical harmonic function coefficients. A subset of Gaussian particles belonging to the object to be manipulated is extracted by a 3D instance segmentation network, and its surface contour is mapped to the surface of the rigid body collision mesh in the differentiable physics engine using a differentiable moving cube algorithm. This establishes a mathematical mapping from the rigid body state driven by the physics engine to the Gaussian particle pose update of the visual rendering layer.

3. The method according to claim 2, characterized in that, The method of generating virtual visual power based on the photometric error gradient between real camera images and twin-rendered images includes: The twin image is synthesized in real time from a virtual perspective using a twin rendering pipeline, and the photometric error loss function between the real camera image and the twin image is calculated. The photometric error loss function is composed of a weighted combination of pixel-level absolute value norm distance and structural similarity index; The backpropagation mechanism of the computation graph is invoked to calculate the photometric error loss function with respect to the mean vector of the associated Gaussian particle three-dimensional positions. The partial derivatives are used to generate the corresponding discrete virtual visual power. ,in This refers to the damping gain parameter; The discrete virtual visual forces of all associated particles are aggregated into a resultant lateral force acting on the object's center of mass and a resultant torque relative to the center of mass, which are then injected as an external additional torque into the dynamic equation of the next time step.

4. The method according to claim 1, characterized in that, The process of reconstructing the multi-rigid-body dynamics simulation into a fully differentiable computational graph includes: constructing the rigid-body forward dynamic equations that include joint friction terms. in, For joint generalized coordinates, The inertia matrix, The matrix of Coriolis force and centrifugal force. The vector of the gravity term. For driving torque, Let Jacobian matrix be the point of contact in the collision. For contact impulse vector, The term of the joint friction torque is constructed by combining the Coulomb friction coefficient and the viscous friction coefficient to be identified with a smooth approximation function. An implicit integrator is used to discretize the continuous dynamic equation, and the contact constraint is transformed into a linear complementary problem that satisfies the non-penetration condition and the Coulomb friction cone constraint. The iterative solution process of the linear complementary problem is encapsulated into a differentiable function through analytical differentiation using implicit function theorems.

5. The method according to claim 4, characterized in that, The process of calculating the gradient of the physical loss function with respect to the physical parameters to be identified through backpropagation using automatic differentiation techniques, and obtaining highly accurate aligned physical parameters through iterative optimization, includes: acquiring the actual observation trajectory within a fixed time window. It uses real control commands to drive a differentiable engine to perform continuous forward dynamic simulation to obtain the simulation prediction trajectory sequence. Construct a physical loss function consisting of a weighted average of position trajectory error and velocity trajectory error terms. : The physical loss function is calculated by backpropagation along the time axis with respect to the physical parameters to be identified, including friction, mass, center of gravity offset, and damping parameters. The total derivative of the physical loss function is used to update the gradient descent parameters using an adaptive learning rate optimizer until the physical loss function converges to the residual threshold and the aligned physical parameters are output.

6. The method according to claim 1, characterized in that, The low-frequency outer loop infers and outputs the target pose based on multimodal observation features, including: In each decision cycle, the environmental observation features of the current physical operation step are extracted. The environmental observation features are composed of digital twin visual representation vectors, the angle states of each joint of the robot body, and the aligned physical parameters. The historical observation sequence containing causal self-attention masks is input into the deep neural network policy model for forward inference, outputting the target 3D position reference of the end effector in the next time step. With target attitude quaternion reference .

7. The method according to claim 6, characterized in that, A time interpolator is provided between the low-frequency outer loop and the high-frequency inner loop. The operation logic of the time interpolator includes: Within the time interval between two adjacent outer loop decisions, a smooth transition trajectory is constructed using the target pose of the two most recent cycles as the starting and ending points. Polynomial interpolation is used to construct position continuity for the target's three-dimensional position reference, and spherical linear interpolation is used to perform attitude transition on the unit quaternion manifold for the target's attitude quaternion reference, so that the high-frequency inner loop queries the corresponding smooth target pose and its first and second time derivatives from the time interpolator in each servo cycle.

8. The method according to claim 7, characterized in that, The high-frequency inner loop performs sensorless compliant impedance control, which first requires calculating the six-dimensional pose deviation vector. The calculation steps include: Obtain the true end position calculated from the forward kinematics. With true pose quaternion ; Position deviation is calculated using vector subtraction. ; Calculate the attitude error quaternion The three-dimensional rotation error vector is extracted by taking the logarithm of the attitude error quaternion. ; The positional deviation is concatenated longitudinally with the three-dimensional rotation error vector to form the six-dimensional pose deviation vector containing six degrees of freedom. .

9. The method according to claim 8, characterized in that, The steps of calculating the inverse dynamic residual torque based on the aligned physical parameters of the high-frequency inner loop and estimating the end-effector contact force through the generalized pseudo-inverse inversion of the end-effector kinematic Jacobian matrix include: reading the measured joint drive torque output by the current servo. And based on the aligned physical parameters, calculate the theoretical internal consumption torque including theoretical friction compensation. The inverse dynamic residual torque is obtained by subtracting the measured joint driving torque from the theoretical internal consumption torque. ; Utilizing the Jacobian matrix of the end effector kinematics Generalized pseudo-inverse back projection calculation of software-driven end contact force estimation : Substituting the estimated end contact force into the dynamic equation of virtual impedance behavior It calculates and outputs a target driving torque command with directional compliance, wherein , , These are the positive definite virtual inertial mass, virtual damping, and virtual stiffness matrices in the six degrees of freedom of space, respectively.

10. The method according to claim 9, characterized in that, The sensorless compliant impedance control based on the estimated end contact force also includes a safety protection mechanism based on anti-collision monitoring logic: Real-time calculation of the L2 norm of the estimated end contact force ; When the L2 norm is determined to exceed the preset safety force threshold scalar, an emergency interruption flag is thrown, the current target pose is overwritten with the safety buffer position of the previous cycle, and a bounce back is executed to terminate torque accumulation and avoid hardware overload.

Images