A bionic robot control system based on digital human motion mapping

Abstract

This invention relates to a bionic robot control system based on digital human motion mapping, specifically in the field of robot control technology. It solves the problem that bionic robots cannot perceive physical properties by relying solely on vision. By analyzing the microscopic color changes of the digital human's fingertips and the deformation of object textures, it achieves accurate prediction of contact force and material properties, enabling the robot to establish an appropriate mechanical model before contact. The system dynamically adjusts the compliance and damping of the robotic arm according to the environmental stiffness and friction characteristics, achieving soft gripping of hard objects and stable gripping of slippery objects. This effectively overcomes the lag of traditional feedback control. At the same time, by combining timing feedforward and energy monitoring mechanisms, it provides collision protection while ensuring zero-delay synchronization of actions, significantly improving the robot's operational stability and safety in complex environments when dealing with fragile and flexible objects.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and more specifically, to a bionic robot control system based on digital human motion mapping. Background Technology

[0002] With the rapid development of intelligent manufacturing and service robot technology, the deep integration of digital human technology and physical robots has become a key path to improve the operational flexibility and intelligence of robots. Under this technological architecture, high-precision optical capture or inertial sensing devices are used to collect motion data from real humans and map it onto bionic robots, enabling the robots to inherit human operational experience and motion logic. This bionic control system has broad application prospects in high-end services, home care, and precision industrial assembly. Especially in home service scenarios, robots often face complex operational tasks in unstructured environments, such as grasping thin-walled glass cups, handling fragile items like raw eggs, or arranging flexible objects like silk fabrics and folding clothing. These tasks not only require robots to have precise spatial trajectory tracking capabilities but also, at the moment of contact with an object, to control the pressure and friction of their fingertips with extreme finesse according to the material properties and deformation characteristics of the object, just like a human, to achieve safe and stable anthropomorphic operation.

[0003] However, a significant technical bottleneck exists in existing robot control technologies based on digital human motion mapping: a severe information gap exists between visual motion data and physical force control. Mainstream motion capture technologies primarily focus on acquiring kinematic characteristics such as the spatial position, rotation angle, and speed of key points on the human skeleton. This data only describes the geometric shape of the movement but cannot record the dynamic information such as contact force, torque, and microscopic deformation of the object applied by the operator during interaction. When this motion data, containing only geometric information, is directly mapped to a physical robot, the robot can only mechanically reproduce the movement trajectory without perceiving the physical properties of the object. Although existing solutions typically install six-dimensional torque sensors or tactile skin on the robot's end effector to adjust the grasping force based on force feedback signals after contact, this passive feedback-based control mechanism suffers from significant signal lag. When handling fragile or soft materials, if the robot's instantaneous speed upon contact with the object is too high or the preset stiffness is too great, irreversible physical damage or slippage often occurs before the sensor feedback signal intervenes and triggers adjustment. This lack of a feedforward control mechanism based on visual features to predict physical properties makes it difficult for existing bionic robots to achieve true visual fusion and adaptive compliant control when dealing with objects of diverse materials and physical properties, greatly limiting their operational capabilities and safety in real-world complex scenarios. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing a bionic robot control system based on digital human motion mapping, thereby solving the problems mentioned in the background art.

[0005] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: a bionic robot control system based on digital human motion mapping, comprising: a visual micro-feature extraction module, a contact dynamics parameter inference module, a force impedance control generation module, and a time-bounded feedforward execution module, wherein;

[0006] Visual micro-feature extraction module: Receives the video stream of the digital human operation, performs temporal filtering and spatial magnification on the pixel data of the contact area between the human hand and the operation object, extracts the pixel light intensity change features representing contact pressure and the texture gradient features representing the deformation of the object, and generates a spatiotemporal visual micro-feature tensor containing color shift and geometric deformation components after time synchronization alignment.

[0007] Contact dynamics parameter inference module: The spatiotemporal visual micro-feature tensor mentioned above is obtained and input into a pre-trained physical constraint generative adversarial network. The generator infers the corresponding physical properties and outputs a predicted physical contact parameter vector containing the friction coefficient, object stiffness coefficient and normal contact force. The adversarial network contains a loss function based on physical law constraints to verify the physical consistency of the output parameters.

[0008] Force impedance control generation module: The inverse dynamics algorithm is used to map the predicted physical contact parameter vector to the dynamic model of the robot end effector. The target contact stiffness is calculated based on the object stiffness coefficient, and the target contact damping is obtained by combining the friction coefficient. The adaptive impedance control command containing dynamic stiffness matrix and damping matrix parameters is generated.

[0009] The time-bound feedforward execution module establishes a preset duration feedforward instruction buffer to store the control sequence. During execution, it collects actual force feedback data from the end sensor, calculates and compares the deviation through energy state divergence, and prioritizes outputting the preset duration feedforward instruction buffer instruction to drive the joint actuator, thereby achieving precise and efficient control.

[0010] In a preferred embodiment, the specific operation of extracting pixel light intensity change features characterizing contact pressure in the visual micro-feature extraction module is as follows:

[0011] First, the digital human operation video stream is decomposed into a multi-level Laplacian pyramid structure, and low-frequency image components containing color distribution information are selected from the Laplacian pyramid structure as the processing objects.

[0012] Then, the low-frequency image components are converted from the time domain to the frequency domain using the short-time Fourier transform algorithm, and a preset frequency band covering the frequency of muscle tremors and the frequency of blood volume changes is defined in the frequency domain.

[0013] Next, using the nonlinear amplification gain coefficient that changes dynamically with frequency, an exponential enhancement process is performed on the chroma channel signal components within the preset frequency band, making skin color changes that are below the preset threshold and distributed within a specific physiological frequency band more visible.

[0014] Finally, a skin region mask covering the digital human hand area is constructed, and the enhanced chroma channel signal components are multiplied element-wise with the skin region mask to filter out background noise and output pixel light intensity variation characteristics.

[0015] In a preferred embodiment, the specific operation of extracting texture gradient features characterizing object deformation in the visual micro-feature extraction module is as follows:

[0016] First, high-frequency image components containing texture details are selected from the Laplacian pyramid structure, and the instantaneous displacement velocity of pixels is calculated in the contact area between the digital human hand and the object being manipulated to construct a dense optical flow vector field.

[0017] Subsequently, divergence calculations are performed on the dense optical flow vector field to quantify the degree of compressive or tensile deformation of the object's surface by calculating the flux sources and sinks of the vector field.

[0018] In this process, the magnitude of the image brightness gradient is calculated to characterize the saliency of the texture, and the dense optical flow vector field is weighted according to the magnitude of the image brightness gradient to highlight the texture edge region.

[0019] Finally, the second-order spatial rate of change of the dense optical flow vector field is calculated and multiplied by the smoothing regularization coefficient. The weighted divergence calculation result is then superimposed with the second-order spatial rate of change to generate texture gradient features.

[0020] In a preferred embodiment, the specific operation of deriving and predicting the physical contact parameter vector in the contact dynamics parameter inference module is as follows:

[0021] First, a spatiotemporal encoder composed of a three-dimensional convolutional neural network is used to perform multi-level convolution processing on the spatiotemporal visual micro-feature tensor. In the time dimension, the temporal lag correlation between pixel light intensity change features and texture gradient features is extracted, and the spatiotemporal visual micro-feature tensor is encoded into a low-dimensional physical latent feature vector.

[0022] The physical latent feature vector is then input into the generator model in the pre-trained physical constraint generative adversarial network. The physical latent feature vector is reconstructed using a fully connected deconvolutional layer, and the reconstructed features are mapped to a preset physical value range through a nonlinear activation function, thereby synthesizing a predicted physical contact parameter vector containing normal contact force, object stiffness coefficient and friction coefficient.

[0023] In a preferred embodiment, the specific operation of verifying physical consistency using a loss function based on physical law constraints in the contact dynamics parameter inference module is as follows:

[0024] First, a contact mechanics constitutive residual potential energy equation containing a nonlinear elastic mapping operator and a Coulomb friction cone constraint term is constructed, which serves as a constraint condition for network training.

[0025] Subsequently, the Mahalanobis distance between the normal contact force and the elastic force was calculated using the contact mechanics constitutive residual potential energy equation as the elastic constitutive residual. The elastic force was derived based on the object stiffness coefficient and the texture gradient features processed by the nonlinear elastic mapping operator.

[0026] Simultaneously, the exponential function value of the ratio of the estimated tangential shear force to the maximum static friction force is calculated as a friction constraint penalty, wherein the maximum static friction force is the product of the friction coefficient and the normal contact force.

[0027] Finally, by minimizing the weighted sum consisting of the elastic constitutive residual and the friction constraint penalty, the generator model is forced to output a vector of predicted physical contact parameters that conforms to the laws of contact mechanics.

[0028] In a preferred embodiment, the specific operation for generating the dynamic stiffness matrix parameters and dynamic damping matrix parameters in the force impedance control generation module is as follows:

[0029] First, the baseline stiffness matrix of the robot in the non-contact free space state is set, and the body stiffness coefficient is extracted from the predicted physical contact parameter vector.

[0030] Subsequently, the stiffness adjustment amount is calculated using the exponential decay function logic, so that the values ​​of the diagonal elements in the generated dynamic stiffness matrix parameters show a nonlinear decreasing trend as the stiffness coefficient of the object increases, thereby constructing a complementary mapping between environmental stiffness and robot flexibility.

[0031] Next, the virtual inertia matrix of the robot's end effector is set, and the friction coefficient is extracted from the predicted physical contact parameter vector;

[0032] Finally, the damping gain factor based on the friction coefficient is calculated using the inverse hyperbolic tangent function, and the damping gain factor is applied to the critical damping term synthesized from the dynamic stiffness matrix parameters and the virtual inertia matrix, thereby generating dynamic damping matrix parameters with high damping characteristics under low friction coefficient conditions.

[0033] In a preferred embodiment, the specific operation of generating adaptive impedance control commands in the force impedance control generation module is as follows:

[0034] First, obtain the reference motion state generated by the digital human motion mapping and the actual motion state of the robot's end effector, and calculate the position deviation vector and velocity deviation vector between the two;

[0035] Subsequently, the position deviation vector is weighted using the dynamic stiffness matrix parameters, the velocity deviation vector is weighted using the dynamic damping matrix parameters, and the inertial force term is calculated in conjunction with the virtual inertia matrix.

[0036] Simultaneously, the normal contact force in the predicted physical contact parameter vector is introduced into the control loop as a feedforward force term.

[0037] Finally, the inertial force term, the weighted position deviation vector and velocity deviation vector, and the feedforward force term are mapped to the joint space using the robot's Jacobian transpose matrix, and a robot dynamics compensation term including gravity, Coriolis force, and centrifugal force is superimposed to synthesize an adaptive impedance control command for driving the robot's joint actuators.

[0038] In a preferred embodiment, the specific operation of establishing the feedforward instruction buffer in the time-bound feedforward execution module is as follows:

[0039] First, a shared memory space is allocated for storing data, and adaptive impedance control instructions within a preset time horizon are extracted based on the current system time.

[0040] Then, each frame of adaptive impedance control instruction is marked with an absolute timestamp to construct a control sequence containing future time steps and store it in a shared memory space to form a feedforward instruction buffer.

[0041] Simultaneously, clock synchronization calibration logic is executed, and the time axis of the data source that generates adaptive impedance control commands is aligned with the time axis of the robot controller using the network time protocol, ensuring that the command frames in the feedforward command buffer accurately correspond to the spatial positions that the robot will reach in the future.

[0042] In a preferred embodiment, the specific operation for comparing deviations in the time-fedforward execution module is as follows:

[0043] The energy state divergence operator is used for calculation. First, a sliding monitoring window is set, and the normal contact force corresponding to the current moment is extracted from the feedforward instruction buffer as the predicted value.

[0044] Then, the deviation vector between the actual force feedback data of the robot's end sensor and the normal contact force in the direction of the contact normal vector is calculated, and the weighted quadratic energy of the deviation vector is calculated using a preset force perception weighted metric matrix.

[0045] Simultaneously, the square of the rate of change of the actual force feedback data over time is calculated as a rigid collision detection item;

[0046] Finally, the weighted sum of the weighted quadratic energy and the rigid collision detection term within the sliding monitoring window is integrated to generate the energy state divergence, which characterizes the degree of deviation between real physical interaction and digital human experience prediction.

[0047] In a preferred embodiment, the specific operation of outputting the instructions to drive the robot joint actuator in the time-fedforward execution module is as follows:

[0048] First, a confidence-gated logic is constructed, and a feedforward confidence factor based on the energy state divergence is calculated using an S-shaped confidence decay function, so that the value of the feedforward confidence factor decreases smoothly as the energy state divergence increases.

[0049] Subsequently, the instruction fusion operation is performed. On the one hand, the feedforward confidence factor is used to weight the adaptive impedance control instruction in the feedforward instruction buffer after look-ahead time step compensation to generate feedforward components. On the other hand, the conservative admittance compensation instruction calculated based on the current actual force feedback data is weighted using weights complementary to the feedforward confidence factor to generate feedback components.

[0050] Finally, the feedforward and feedback components are linearly superimposed to synthesize the final driving torque and send it to the servo driver, so as to achieve zero-delay feedforward control under low deviation and smoothly switch to feedback control under high deviation.

[0051] The beneficial effects of this invention are: it solves the problem that bionic robots cannot perceive physical properties by relying solely on vision. By analyzing the microscopic color changes of the digital human's fingertips and the deformation of object textures, it achieves accurate prediction of contact force and material properties, enabling the robot to establish an appropriate mechanical model before contact. The system dynamically adjusts the compliance and damping of the robotic arm according to the environmental stiffness and friction characteristics, achieving soft gripping of hard objects and stable gripping of slippery objects, effectively overcoming the lag of traditional feedback control. At the same time, by combining timing feedforward and energy monitoring mechanisms, it provides collision protection while ensuring zero-delay synchronization of actions, significantly improving the stability and safety of the robot's operation on fragile and flexible objects in complex environments. Attached Figure Description

[0052] Figure 1 This is a block diagram of the system structure of the present invention;

[0053] Figure 2 This is a logic diagram of the visual micro-feature extraction module of the present invention;

[0054] Figure 3 This is a logic diagram of the contact dynamics parameter reasoning module of the present invention;

[0055] Figure 4 This is a logic diagram of the force impedance control generation module of the present invention;

[0056] Figure 5 This is a logic diagram of the time-fedforward execution module of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0058] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0059] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0060] This embodiment provides, for example Figures 1-5 The diagram illustrates a bionic robot control system based on digital human motion mapping, comprising: a visual micro-feature extraction module, a contact dynamics parameter inference module, a force impedance control generation module, and a time-fedforward execution module, wherein;

[0061] The visual micro-feature extraction module is configured to receive the digital human operation video stream, perform temporal filtering and spatial magnification processing on the pixel data of the contact area between the digital human's hand and the operation object in the digital human operation video stream, extract the pixel light intensity change features representing contact pressure and the texture gradient features representing object deformation, and synchronize the pixel light intensity change features and texture gradient features in the time dimension to generate a spatiotemporal visual micro-feature tensor containing color offset components and geometric deformation components.

[0062] The contact dynamics parameter inference module is configured to acquire the spatiotemporal visual micro-feature tensor generated by the visual micro-feature extraction module, input the spatiotemporal visual micro-feature tensor into a pre-trained physical constraint generative adversarial network, use the generator model in the pre-trained physical constraint generative adversarial network to infer the physical properties corresponding to the spatiotemporal visual micro-feature tensor, and output a predicted physical contact parameter vector containing the friction coefficient, object stiffness coefficient and normal contact force. The pre-trained physical constraint generative adversarial network contains a loss function based on physical law constraints to verify the physical consistency of the output parameters.

[0063] The force impedance control generation module is configured to obtain the predicted physical contact parameter vector output by the contact dynamics parameter inference module, use the inverse dynamics algorithm to map the predicted physical contact parameter vector to the robot's end effector dynamics model, calculate the target contact stiffness based on the object stiffness coefficient, and calculate the target contact damping based on the friction coefficient, and generate an adaptive impedance control command containing dynamic stiffness matrix parameters and dynamic damping matrix parameters.

[0064] The time-bound feedforward execution module is configured to acquire the adaptive impedance control command generated by the force impedance control generation module, establish a feedforward command buffer with a preset time length, store the adaptive impedance control command as a control sequence for future time steps in the feedforward command buffer, and collect the actual force feedback data of the robot end effector in real time when the robot performs an action. The actual force feedback data is compared with the command value in the feedforward command buffer. When the comparison deviation is less than a preset allowable threshold, the adaptive impedance control command in the feedforward command buffer is output first to drive the robot joint actuator.

[0065] In a preferred embodiment of the present invention, the visual micro-feature extraction module serves as the perception core, responsible for extracting invisible microscopic physical clues from the macroscopic video stream.

[0066] In the visual micro-feature extraction module, pixel light intensity variation features characterizing contact pressure are extracted. The specific operation is as follows:

[0067] First, the digital human operation video stream is decomposed into a multi-level Laplacian pyramid structure, and low-frequency image components containing color distribution information are selected from the Laplacian pyramid structure as the processing objects.

[0068] Specifically, the system receives high frame rate video streams of digital human operation. In order to separate macroscopic motion from microscopic changes, the system analyzes the video stream of the digital human's operation. Each frame of the image is decomposed using a Laplacian pyramid. In this process, the top layer of the pyramid mainly contains large-scale color change information, namely low-frequency image components. These components can reflect the overall shift in skin hue caused by changes in blood volume, without being affected by local texture noise.

[0069] Then, the low-frequency image components are converted from the time domain to the frequency domain using the short-time Fourier transform algorithm, and a preset frequency band covering the frequency of muscle tremors and the frequency of blood volume changes is defined in the frequency domain.

[0070] In this step, the system applies the short-time Fourier transform operator. Processing low-frequency image components. The system's preset frequency band is defined by a lower cutoff frequency. With upper limit cutoff frequency The defined frequency range is typically set between 0.5Hz and 4Hz. This range precisely covers the inherent frequencies of muscle tremors and capillary blood filling changes when the human body grasps something forcefully, thus effectively isolating low-frequency noise caused by ambient light drift and high-frequency noise generated by imaging equipment.

[0071] Next, using the nonlinear amplification gain coefficient that changes dynamically with frequency, exponential enhancement processing is performed on the chroma channel signal components within the preset frequency band, making the skin color more visible when the signal-to-noise ratio is below the preset threshold and the distribution is within a specific physiological frequency band.

[0072] To quantify this change, which is difficult to detect with the naked eye, this implementation method uses a blood flow state response equation for calculation:

[0073] ;

[0074] This formula is used to calculate pixel light intensity variation characteristics. In simple terms, it acts as a frequency domain microscope, converting video images from the "time domain" to the "frequency domain," finding specific frequencies (corresponding to the frequencies of muscle tremors and blood flow pulses), amplifying the signals at these frequencies, and then converting them back to the "time domain," retaining only the skin area. Thus, the previously invisible, subtle process of fingers turning white or red is transformed into a numerically significant signal after processing by this formula, which can be recognized by a computer as a "pressure value."

[0075] In this formula: Represents the final output in time and spatial coordinates Pixel light intensity variation characteristics at the location Its physical meaning is the distribution trend of normal pressure at various points on the fingertips; This represents the chroma channel intensity signal of the input video frame at the corresponding coordinates, used to remove luminance interference and focus on color changes; This represents the short-time Fourier transform operator mentioned above; Represents the angular frequency variable; and These represent the lower and upper cutoff frequencies mentioned above. This represents the nonlinear amplification gain coefficient, which is not a constant but rather a frequency-dependent gain factor. The function is used to exponentially amplify weak signals within a preset frequency band, thereby making the latent blood flow obstruction explicit at the data level. Represents the complex exponential basis function, used to reconstruct time-domain signals through inverse transform; Represents the skin region mask, which is a binary matrix (0 or 1).

[0076] In this embodiment, the selection of the preset frequency band is based on human physiological characteristics. Considering the frequency characteristics of blood volume pulse wave (BVP) and muscle tremors, the lower cutoff frequency is... The preferred setting is 0.5Hz, with an upper limit cutoff frequency. The preferred setting is 4Hz. This range effectively covers the physiological signal frequencies of most operators while filtering out extremely low-frequency body sway and high-frequency image sensor thermal noise. Furthermore, the nonlinear amplification gain coefficient... The value is typically set between 10 and 100. Preferably, an adaptive gain strategy that attenuates with spatial frequency is employed; for example, a larger gain is used for lower spatial frequencies. Value (e.g.) For higher spatial frequencies, smaller values ​​are used. Value (e.g.) This is to prevent the introduction of excessive jagged edges or artifacts while amplifying physiological signals.

[0077] It is worth noting that the above values ​​are only preferred parameters for this embodiment. Technicians can adjust the above parameters within a reasonable range according to the sampling frame rate (FPS) of the camera and the lighting conditions of the actual application scenario, without departing from the technical scope of this invention.

[0078] In this embodiment, the formula solves the problem that weak physiological signals are easily drowned out by background noise, and realizes the conversion from visual color to physical quantities of contact pressure.

[0079] Finally, a skin region mask covering the digital human hand area is constructed, and the enhanced chroma channel signal components are multiplied element-wise with the skin region mask to filter out background noise and output pixel light intensity variation characteristics.

[0080] In the formula This refers to a mask for the skin area. This represents the Hadamard product operator. Through this operation, the system retains only the calculation results for the fingertip skin area, setting the background pixel values ​​of non-contact areas to zero, thus ensuring the accuracy of the output pixel intensity variation characteristics. It is pure and depends only on the operator's applied force.

[0081] In the visual micro-feature extraction module, the specific operation for extracting texture gradient features that characterize object deformation is as follows:

[0082] First, high-frequency image components containing texture details are selected from the Laplacian pyramid structure, and the instantaneous displacement velocity of pixels is calculated in the contact area between the digital human hand and the object being manipulated to construct a dense optical flow vector field.

[0083] Unlike extracting pressure features, this system selects the high-frequency image components from the bottom layer of the pyramid because these components preserve the texture details of the object's surface. The system calculates the motion vector for each pixel within the contact area, generating a dense optical flow vector field. This vector field describes the instantaneous displacement velocity of the surface texture of an object over time.

[0084] Subsequently, divergence calculations are performed on the dense optical flow vector field to quantify the degree of compressive or tensile deformation of the object's surface by calculating the flux sources and sinks of the vector field.

[0085] In this step, the system introduces the divergence operator. Its physical meaning is as follows: if the divergence is negative, it means that the texture converges to that point, corresponding to the compressive indentation (compression deformation) of the object's surface; if the divergence is positive, it means that the texture diffuses, corresponding to the tensile deformation of the object's surface.

[0086] In this process, the magnitude of the image brightness gradient is calculated to characterize the saliency of the texture, and the dense optical flow vector field is weighted according to the magnitude of the image brightness gradient to highlight the texture edge region. Finally, the second-order spatial rate of change of the dense optical flow vector field is calculated and multiplied by the smoothing regularization coefficient. The weighted divergence operation result is superimposed with the second-order spatial rate of change to generate texture gradient features.

[0087] To accurately characterize this microscopic deformation, this implementation method uses the local texture deformation potential energy equation for calculation:

[0088] ;

[0089] In this implementation, the formula solves the problem that the simple optical flow method cannot distinguish between overall motion and local deformation. By combining divergence and regularization terms, it accurately extracts the microscopic deformation information of an object caused by forces. Finally, the system extracts the pixel light intensity variation characteristics. Texture gradient features Combined, a spatiotemporal visual micro-feature tensor is generated. This is for use by subsequent modules.

[0090] In this formula: Represents the final generated texture gradient features It is used to characterize the microscopic geometric deformation topology of an object under stress. Represents a dense optical flow vector field; Represents the divergence operator; A vector representing the image brightness gradient. This represents the modulo operation; the two combined are used to characterize the texture richness of an object's surface. This represents the texture saliency weighting function, which adjusts the weight of the optical flow vector according to the texture richness, giving higher weight to texture edges and ignoring misjudgments of smooth regions; This represents the smoothing regularization coefficient, used to adjust the strength of the regularization term; This represents the Laplacian operator, used to extract the second-order spatial rate of change of the optical flow field to suppress outlier noise.

[0091] In this embodiment, the texture saliency weighting function The preferred activation function is the sigmoid-like function, which can be specifically set as follows: ,in, For the brightness gradient magnitude, This is a gradient threshold that is adaptively adjusted based on the ambient light intensity (e.g., 1.5 times the average gradient of the image). The slope coefficient ensures that the system collects optical flow data only in regions with significant texture features (high gradient regions), effectively suppressing computational noise on smooth surfaces.

[0092] In addition, smoothing regularization coefficients The value is typically set between 0.1 and 0.5. When dealing with objects with rich and rough surface textures (such as fabrics), a smaller value is preferred. A value (e.g., 0.15) is preferred to preserve more details of local deformation; a larger value is preferred when dealing with objects with relatively smooth or highly reflective surfaces (such as ceramic cups). A value (e.g., 0.45) is used to suppress optical flow mismatch caused by reflection by using a second-order smoothing term, thereby ensuring the robustness of the output features.

[0093] In a further embodiment of the present invention, the contact dynamics parameter reasoning module serves as the cognitive center of the system, responsible for converting the visual signals extracted from the preceding steps into physical instructions that the robot can understand.

[0094] In the contact dynamics parameter inference module, the physical contact parameter vector is derived and predicted. The specific operation is as follows:

[0095] First, a spatiotemporal encoder composed of a three-dimensional convolutional neural network is used to perform multi-level convolution processing on the spatiotemporal visual micro-feature tensor. In the time dimension, the temporal lag correlation between pixel light intensity change features and texture gradient features is extracted, and the spatiotemporal visual micro-feature tensor is encoded into a low-dimensional physical latent feature vector.

[0096] Specifically, this module receives spatiotemporal visual micro-feature tensors from the preceding module. Because the physical contact process has a causal temporal sequence (e.g., a fingertip first contacts an object's surface, causing deformation, and then blood is compressed, leading to a color change), the module internally deploys a spatiotemporal encoder composed of a 3D convolutional neural network (3D-CNN). This encoder deeply mines the pixel light intensity variation features by sliding convolutions along the time axis. , and texture gradient features The time lag correlation between them compresses and maps high-dimensional tensor data into a low-dimensional physical latent feature vector. This vector encodes the "mechanical timing logic" of the action in the latent space.

[0097] The physical latent feature vector is then input into the generator model in the pre-trained physical constraint generative adversarial network. The physical latent feature vector is reconstructed using a fully connected deconvolutional layer, and the reconstructed features are mapped to a preset physical value range through a nonlinear activation function, thereby synthesizing a predicted physical contact parameter vector containing normal contact force, object stiffness coefficient and friction coefficient.

[0098] In this step, the system utilizes a pre-trained physical constraint generative adversarial network (PC-GAN) generator model. To perform cross-modal mapping. The generator receives the physical latent feature vector. The physical parameters are finally output by upsampling and reconstruction through the internal fully connected deconvolutional layer.

[0099] This implementation method uses the following generator-based physical parameter assumption synthesis logic formula for derivation:

[0100] ;

[0101] This formula is used to generate the predicted physical contact parameter vector. It is a decoder. It decodes the encoded features of video that are incomprehensible to humans. It decompresses and maps data into specific physical values ​​(force, stiffness, friction), serving as a bridge between the "visual micro-feature extraction module" and the "force impedance control generation module." Without it, visual features cannot be converted into control commands.

[0102] In this formula: This represents the vector of predicted physical contact parameters for the output. This represents the derived normal contact force, which is the vertical force that the robot's end effector needs to apply. The derived stiffness coefficient of an object is used to characterize the ability of an object (such as tofu vs. walnut) to resist deformation. The coefficient of friction, derived from the model, is used to characterize the roughness and anti-slip ability of the contact surface. Representative generator model The mapping function; The physical latent feature vector represents the input.

[0103] In this embodiment, to ensure that the parameters output by the generator model are adapted to the actual operational capabilities of the home service robot, the system presets a normalized mapping range for the physical parameters. Specifically:

[0104] Regarding normal contact force Its mapping range is set to [0 N, 20 N]. This range covers typical force values ​​required for everything from lightly touching tofu to firmly gripping a glass full of water. (Regarding the object stiffness coefficient...) Its mapping range is set to [100 N / m, 5000 N / m]. 100 N / m corresponds to highly flexible objects (such as sponges), and 5000 N / m corresponds to highly rigid objects (such as ceramics). The robot controller will dynamically adjust the impedance characteristics of the end effector based on this stiffness value. (Regarding the coefficient of friction...) Its mapping range is set to [0.1, 1.0]. The generator limits the output to between 0 and 1 through the final Sigmoid activation layer, directly corresponding to the static friction coefficient range of most common household materials (such as plastic, glass, and fabric). By limiting the above value range, this invention ensures that the parameters derived by artificial intelligence are always within the physical boundaries of robot hardware safety.

[0105] In this embodiment, the formula addresses the unsteadiness problem of directly regressing physical parameters from pure visual data. By leveraging the manifold mapping capability of the generative model, it synthesizes dynamic parameters with physical meaning.

[0106] In the contact dynamics parameter inference module, the specific operation of verifying physical consistency using a loss function based on physical law constraints is as follows:

[0107] First, a contact mechanics constitutive residual potential energy equation containing a nonlinear elastic mapping operator and a Coulomb friction cone constraint term is constructed, which serves as a constraint condition for network training.

[0108] To ensure that the generated parameters do not violate Newtonian mechanics principles (e.g., to avoid the error of "large deformation corresponding to small force"), this implementation method constructs a contact mechanics constitutive residual potential energy equation as the core constraint during the training phase:

[0109] ;

[0110] This formula is used for calculation. (Physical loss value), serving as a feedback signal for training the generative adversarial network (GAN);

[0111] Part 1 (integral term on the left): Hooke's Law constraint. It checks whether the "calculated force" and the "observed deformation" match. If the network says "this is a piece of iron (high stiffness)," but the video shows it "flat (high deformation)," this part of the loss will skyrocket.

[0112] Part Two (Right-hand exponent): Coulomb's Law of Friction constraint. It checks whether the object will "slip". If the network calculates "very low coefficient of friction (very slippery)" but the tangential force is large (very strong grip), physically the object should fly away. If the object doesn't fly away, it means the network calculation is wrong, and this part of the loss will skyrocket.

[0113] Subsequently, the Mahalanobis distance between the normal contact force and the elastic force was calculated using the contact mechanics constitutive residual potential energy equation as the elastic constitutive residual. The elastic force was derived based on the object stiffness coefficient and the texture gradient features processed by the nonlinear elastic mapping operator.

[0114] In the first part of the above formula: This represents the value of the loss function based on physical law constraints; , and These represent the normal contact force, object stiffness coefficient, and friction coefficient output by the generator, respectively. This represents a nonlinear elastic mapping operator, which is used to map visual texture gradient features. Mapped to the equivalent physical compression (unit: meters); This represents the area integral operation over the contact region, used to calculate the total equivalent deformation. Representing the Mahalanobis Distance Norm, it uses the covariance matrix to consider the correlation between different physical dimensions and is used to quantify the constitutive residual of Hooke's Law, that is, the deviation between the calculated force and the "theoretical elastic force" derived based on stiffness and deformation.

[0115] Simultaneously, the exponential function value of the ratio of the estimated tangential shear force to the product of the maximum static friction force is calculated as a friction constraint penalty;

[0116] In the second part of the above formula: This represents the friction constraint penalty factor, used to adjust the weight of the friction constraint in the total loss. Represents the estimated value of tangential shear force This value is derived from the input motion speed sequence; It represents a small quantity with numerical stability, preventing the denominator from being zero; The term represents the Coulomb friction cone constraint term, whose physical meaning is to check whether the current force state satisfies Coulomb's friction law. When the calculated ratio of tangential force to normal force exceeds the friction coefficient... When slippage occurs, this item will generate an exponentially increasing penalty value.

[0117] Finally, by minimizing the weighted sum consisting of the elastic constitutive residual and the friction constraint penalty, the generator model is forced to output a vector of predicted physical contact parameters that conforms to the laws of contact mechanics.

[0118] The core purpose of this formula in this implementation is as a "physics checker." It does not require real force sensor truth values ​​for supervision, but rather utilizes the self-consistency of the physical formula itself to constrain the network. This is achieved by minimizing... Forced generator model The learned parameter mapping relationship strictly follows Hooke's law and Coulomb's law of friction, thus ensuring the output predicted physical contact parameter vector. It is physically real and believable.

[0119] In this embodiment, the parameter settings in the formula are crucial for the convergence of the network.

[0120] Friction constraint penalty factor The optimal setting is between 5.0 and 10.0. Since object slippage (violation of friction constraints) is a fatal error that leads to task failure in robot grasping tasks, this item is given a high weight, forcing the generator to prioritize the output of physical parameters that satisfy the non-slip condition.

[0121] For small quantities of numerical stability Usually set to This is to prevent division by zero errors from occurring during the non-contact phase when the contact force is close to zero.

[0122] For the covariance matrix in the Mahalanobis distance norm Its diagonal elements are initialized based on the reciprocal of the variance of a preset range of physical parameters. For example, the normalization coefficient for the force dimension is set to... The normalization coefficient for the dimension corresponding to stiffness is set to This eliminates the gradient imbalance problem caused by different physical quantities, ensuring that the network can simultaneously optimize the prediction accuracy of force and stiffness.

[0123] In a further embodiment of the present invention, the force impedance control generation module inherits the reasoning results of the previous module and is responsible for converting abstract physical properties into dynamic impedance control laws that can be executed by the robot entity, thereby realizing a closed loop of "perception-decision-execution".

[0124] In the force impedance control generation module, dynamic stiffness matrix parameters are generated. With dynamic damping matrix parameters The specific operation is as follows:

[0125] First, the baseline stiffness matrix of the robot in the non-contact free space state is set, and the body stiffness coefficient is extracted from the predicted physical contact parameter vector.

[0126] Specifically, the system pre-sets a baseline stiffness matrix, representing the robot's nominal high stiffness in free space (non-contact) to ensure rapid positioning. Simultaneously, the module acquires the predicted physical contact parameter vector output from the previous stage. And the stiffness coefficient of the object is analyzed from it. .

[0127] Subsequently, the stiffness adjustment amount is calculated using the exponential decay function logic, so that the values ​​of the diagonal elements in the generated dynamic stiffness matrix parameters show a nonlinear decreasing trend as the stiffness coefficient of the object increases, thereby constructing a complementary mapping between environmental stiffness and robot flexibility.

[0128] To avoid contact instability, this implementation constructs an exponential stiffness fitting equation:

[0129] ;

[0130] In this formula: The parameters representing the output dynamic stiffness matrix This is The diagonal matrix determines the robot's ability to resist displacement deviation when subjected to external forces. This represents the aforementioned reference stiffness matrix; Represents the natural exponential function, used to implement nonlinear mappings; This represents the reference environmental stiffness threshold, used to normalize the input stiffness. Represents the stiffness decay sensitivity tensor, used to adjust the rate at which stiffness decays as the environment hardens in different degrees of freedom directions; This represents a numerical stability factor to prevent the denominator from being zero. This represents the minimum safe stiffness matrix, preventing the robot from losing its attitude maintenance ability due to excessively low stiffness.

[0131] In this implementation, the formula addresses the problem of rigid impact in hard-contact scenarios. Specifically, in layman's terms, it achieves a biomimetic compliant strategy of "weakening against strong impacts and strengthening against weak impacts." This is particularly relevant when gripping a sponge. The formula calculates robot stiffness close to the benchmark value (very stiff), ensuring accurate gripping; this is especially important when gripping a glass. The formula calculates that the robot's stiffness decreases rapidly (softens), ensuring that the cup will not break due to excessive hardness at the moment of contact.

[0132] In this embodiment, the selection of each parameter aims to balance operational accuracy and contact safety.

[0133] For the reference stiffness matrix It is preferable to set its diagonal translation component to 2000 N / m to 3000 N / m to ensure that the robot maintains trajectory tracking accuracy during high-speed movement in free space.

[0134] For the minimum safety stiffness matrix The preferred setting is 50 N / m to 100 N / m. This lower limit is set based on the robot's dynamic gravity compensation requirements, ensuring that even when operating on extremely hard objects (such as steel plates), the robot retains the minimum torque stiffness required to maintain its end-effector posture, preventing the robotic arm from sagging or becoming unstable at singular points due to excessive stiffness.

[0135] For stiffness decay sensitivity tensor It is recommended to set a higher sensitivity value (e.g., 3.0) in the contact normal direction and a lower sensitivity value (e.g., 1.0) in the tangential direction to achieve an anisotropic compliant control strategy.

[0136] Next, the virtual inertia matrix of the robot's end effector is set, and the friction coefficient is extracted from the predicted physical contact parameter vector;

[0137] In this step, the system loads a preset virtual inertia matrix. This is typically set as the equivalent inertia of the robot's end effector. Simultaneously, it is derived from the predicted physical contact parameter vector. Extracting the friction coefficient .

[0138] Finally, the damping gain factor based on the friction coefficient is calculated using the inverse hyperbolic tangent function, and the damping gain factor is applied to the critical damping term synthesized from the dynamic stiffness matrix parameters and the virtual inertia matrix, thereby generating dynamic damping matrix parameters with high damping characteristics under low friction coefficient conditions.

[0139] To maintain gripping stability on low-friction surfaces, this implementation constructs a friction-compensated damping modulation equation:

[0140] ;

[0141] In this formula: The parameters represent the output dynamic damping matrix; The critical damping term, representing the standard, ensures that the system itself does not overshoot; Represents the Hadamard product operation; Represents the identity matrix; This represents the damping gain weighting matrix, used to amplify the damping compensation ratio under low friction conditions; Represents the hyperbolic tangent function, which is used here as a smooth implementation of the inverse hyperbolic tangent logic, restricting the adjustment factor to a bounded range; Represents the benchmark for safe friction coefficient, referring to the minimum friction value required for stable gripping without additional damping compensation; It represents a tiny quantity that indicates numerical stability.

[0142] In this embodiment, the formula automatically adjusts the robot's "viscosity" based on the "slipperyness" of the environment. When it detects an extremely slippery object surface, it automatically injects high damping to suppress vibrations. In simpler terms, "walk slowly on slippery surfaces." When the robot senses a very slippery object surface (low coefficient of friction), it automatically increases the "viscosity" (damping) of its joints to prevent the object from slipping due to excessively fast movements or vibrations. When the object surface is very rough (high coefficient of friction), it reduces damping to make movements more agile.

[0143] In this embodiment, the modulation logic of the damping parameters is designed to handle frictional uncertainties in unstructured environments.

[0144] For virtual inertia matrix It is preferably set to 1.0 to 1.5 times the sum of the physical mass of the robot end effector and the average gripping load mass (e.g., the diagonal element is set to 2.0 kg) to provide a stable inertial reference for impedance control.

[0145] Based on the safety friction coefficient benchmark The preferred setting is 0.6. This value corresponds to the common coefficient of friction in rubber-plastic contact. When the system identifies the actual coefficient of friction through visual reasoning... When the value is below 0.6 (e.g., when picking up a wet ceramic cup), the inverse hyperbolic tangent term will output a positive value, and the driving damping gain will increase significantly.

[0146] For the damping gain weighting matrix Its diagonal element is preferably set to 2.0 to 4.0. This means that under extremely low friction conditions (such as...), The system generates commands with damping that is more than three times the critical damping (i.e., overdamped state), thereby effectively suppressing end-effector micro-slippage caused by motor torque fluctuations through high viscosity characteristics, ensuring absolute safety in grasping fragile items.

[0147] In the force-sensing impedance control generation module, adaptive impedance control commands are generated. The specific operation is as follows:

[0148] First, obtain the reference motion state generated by the digital human motion mapping and the actual motion state of the robot's end effector, and calculate the position deviation vector and velocity deviation vector between the two;

[0149] The system reads the reference motion state (including reference position) in real time. Reference speed Reference acceleration ) and the actual motion state of the robot's end effector (including its actual position) Actual speed The difference between the two is then calculated to obtain the position deviation vector. With velocity deviation vector .

[0150] Subsequently, the position deviation vector is weighted using the dynamic stiffness matrix parameters, the velocity deviation vector is weighted using the dynamic damping matrix parameters, and the inertial force term is calculated in conjunction with the virtual inertia matrix.

[0151] This step is the core part of constructing the impedance control law: ;

[0152] Simultaneously, the normal contact force in the predicted physical contact parameter vector is introduced into the control loop as a feedforward force term.

[0153] To overcome feedback delay, the system directly uses the predicted physical contact parameter vector. Normal contact force in Along the contact normal vector Direction applied as a feedforward force term .

[0154] Finally, the mechanical quantities calculated in Cartesian space are mapped to joint space using the robot's Jacobian transpose matrix, and robot dynamics compensation terms including gravity, Coriolis force and centrifugal force are superimposed to synthesize adaptive impedance control commands for driving the robot's joint actuators.

[0155] This implementation method uses a full-state inverse dynamic impedance control law for final synthesis:

[0156] ;

[0157] In this formula: The adaptive impedance control command, representing the final output, is physically represented as a joint torque vector. The Jacobian transpose matrix represents the robot and is used to map the end effector force to the motor torque of each joint. These represent the robot's joint angle vector and joint angular velocity vector, respectively. Represents robot dynamics compensation terms It includes gravity, Coriolis force and centrifugal force, ensuring that impedance control is based on dynamic decoupling.

[0158] This formula integrates environmental perception and dynamics models, enabling the robot to adjust its "hardness" and "speed" before contacting objects, thus achieving precise physical execution of the prediction results from the previous level.

[0159] In this embodiment, the setting of the control law parameters needs to take into account both the system's response bandwidth and stability.

[0160] For virtual inertia matrix The preferred setting is a diagonal matrix, with diagonal element values ​​ranging from 1.0 kg to 5.0 kg. This range is based on the end-effector payload capacity of a typical collaborative robotic arm. A smaller virtual inertia can improve the tracking ability of high-frequency micro-movements of the digital human, but too small an inertia may introduce high-frequency noise oscillations, so it needs to be used in conjunction with a low-pass filter.

[0161] For robot dynamics compensation terms Its calculation relies on the offline identified robot link mass and center of mass parameters. In this embodiment, the Newton-Euler recursive algorithm is preferably used for real-time calculation to ensure that the gravity compensation error is less than 0.5 Nm, thereby ensuring the transparency and sensing sensitivity of the impedance control law under low contact force (< 2N) conditions.

[0162] In another embodiment of the present invention, the time-fedforward execution module, as the final execution link of the control system, is responsible for converting the instructions generated by the previous level, which contain "environmental cognition", into physical drive signals with time foresight and safety redundancy.

[0163] In the time-bounded feedforward execution module, a feedforward instruction buffer is established. The specific operation is as follows:

[0164] First, a shared memory space is allocated for storing data, and adaptive impedance control instructions within a preset time horizon are extracted based on the current system time.

[0165] Specifically, the module does not process one instruction at a time, but instead allocates a high-speed shared memory segment as a feedforward instruction buffer. The system uses the current time Based on the baseline, a preset time horizon is truncated backward. Internal adaptive impedance control command sequence.

[0166] Then, each frame of adaptive impedance control instruction is marked with an absolute timestamp to construct a control sequence containing future time steps and store it in a shared memory space to form a feedforward instruction buffer.

[0167] The system arranges the captured instruction frames in chronological order and marks them with absolute timestamps to form a control sequence for future time steps. This means that the buffer stores the actions and torque information that the robot is "about to" execute.

[0168] Simultaneously, clock synchronization calibration logic is executed, and the time axis of the data source that generates adaptive impedance control commands is aligned with the time axis of the robot controller using the network time protocol, ensuring that the command frames in the feedforward command buffer accurately correspond to the spatial positions that the robot will reach in the future.

[0169] The system utilizes Network Time Protocol (PTP) to eliminate transmission jitter and ensure the feedforward command buffer. The first in The frame command corresponds precisely to the robot's The spatial location of the arrival time is determined, thereby achieving microsecond-level synchronization between the digital human's motion flow and the physical robot's execution flow.

[0170] In the time-fedforward execution module, the specific operation for comparing deviations is as follows:

[0171] The energy state divergence operator is used for calculation. First, a sliding monitoring window is set, and the normal contact force corresponding to the current moment is extracted from the feedforward instruction buffer as the predicted value.

[0172] The system is set to a length of 1000. A sliding monitoring window (e.g., 10ms - 50ms). Simultaneously, from the feedforward instruction buffer... Backtracking extraction and current time Corresponding normal contact force As a benchmark for prediction.

[0173] Then, the deviation vector between the actual force feedback data of the robot's end sensor and the normal contact force in the direction of the contact normal vector is calculated, and the weighted quadratic energy of the deviation vector is calculated using a preset force perception weighted metric matrix.

[0174] The system collects real-time force feedback data from the robot's end effector. And calculate its relationship with the predicted normal contact force. Contact normal vector Differences in direction.

[0175] Simultaneously, the square of the rate of change of the actual force feedback data over time is calculated as a rigid collision detection item;

[0176] Finally, the weighted sum of the weighted quadratic energy and the rigid collision detection term within the sliding monitoring window is integrated to generate the energy state divergence, which characterizes the degree of deviation between real physical interaction and digital human experience prediction.

[0177] To comprehensively assess the accuracy of the prediction, this implementation method uses the energy state divergence equation for calculation:

[0178] ;

[0179] In this formula: Representing the calculated energy state divergence, this scalar value characterizes the degree of deviation between "real physical interaction" and "digital human experience prediction"; Represents the length of the sliding monitoring window; This represents the actual force feedback data collected; The normal contact force represents the historical moment extracted from the buffer. Represents the contact normal vector; This represents the force-weighted metric matrix, which is a positive definite symmetric matrix used to amplify the error weight of the normal contact force while ignoring the interference of the tangential friction force. Represents the force variable rate penalty factor; The squared modulus of the time derivative of the actual force feedback data is the rigid collision detection term. If the robot suddenly collides with a hard object, the force value will change drastically, causing this term to surge.

[0180] In this implementation, the formula not only possesses extremely high noise suppression capabilities but also keenly detects "unexpected collisions" caused by prediction errors by detecting force variability. In simpler terms, its core function is to continuously compare the "actual force felt by the robot" with the "theoretical force predicted by the system" within a very short time window. If the difference is small, it indicates accurate prediction, and the robot can move confidently (feedforward); if the difference suddenly increases, it indicates an unexpected collision or prediction failure, and the system must immediately become alert and switch back to conservative mode.

[0181] In this embodiment, the calculation parameters for energy state divergence need to be adapted based on the signal-to-noise ratio of the end sensor and the mission cycle.

[0182] Regarding the length of the sliding monitoring window The preferred setting is 20 ms. This window length covers approximately 20 control cycles (assuming a system frequency of 1 kHz), which is sufficient to smooth out high-frequency electromagnetic noise from the sensor while ensuring that the response delay to unexpected collisions does not exceed 0.02 seconds.

[0183] For force-weighted metric matrix An anisotropic weighted strategy is preferred. Specifically, the weighting coefficient for the contact normal direction is set to 1.0, while the weighting coefficient for the tangential direction is set to 0.2. This setting makes the system highly sensitive to the prediction error of the contact depth, while maintaining a certain tolerance for the tangential force error caused by the deviation in the estimation of the friction coefficient, thus preventing frequent interruptions of feedforward control due to slight slippage.

[0184] Penalty factor for force variation The preferred setting is 0.005s. This coefficient is used to balance the amplitude error and the rate of change error of the force. It ensures that when the actual rate of change of the force exceeds 200N / s (typical rigid collision characteristics), the second term dominates the integral, rapidly increases the divergence value, and triggers the safety backoff mechanism.

[0185] In the time-fedforward execution module, the specific operation of outputting instructions to drive the robot joint actuators is as follows:

[0186] First, a confidence-gated logic is constructed, and a feedforward confidence factor based on the energy state divergence is calculated using an S-shaped confidence decay function, so that the value of the feedforward confidence factor decreases smoothly as the energy state divergence increases.

[0187] Based on the calculated energy state divergence The system needs to decide the extent to which it trusts the feedforward instructions. This implementation uses a feedforward confidence gating equation:

[0188] ;

[0189] In this formula: This represents the feedforward confidence factor, with a value range of (0,1). This represents the gating sensitivity coefficient, which determines the steepness of the switching. This represents the preset allowable threshold, i.e., the critical point of energy deviation.

[0190] This formula implements a "soft switch" function: when the deviation is less than a threshold, Approaching 1; when the deviation increases significantly, It rapidly approaches 0.

[0191] In this embodiment, the setting of the confidence gating parameter is directly related to the security and robustness of the system in unstructured environments.

[0192] For the preset allowable threshold Its value is typically set to 0.1 J (normalized energy unit). This threshold is based on the statistical analysis of residual energy during normal free space motion and soft contact operations of the robot, ensuring that the system will not switch erroneously due to background white noise of the sensor (usually < 0.02 J), and will only trigger intervention when a significant mismatch in the dynamic model is detected.

[0193] Regarding the gating sensitivity coefficient The preferred setting is 20.0. This coefficient is chosen to construct a 'soft-switching' characteristic with a steep falling edge. When the energy state divergence... Exceeding the threshold At approximately 20%, the confidence factor It will rapidly decay to below 0.1, thereby cutting off the feedforward path within milliseconds and preventing erroneous predicted torque from further exacerbating collision damage, achieving a protective mechanism similar to a neural reflex arc.

[0194] Subsequently, the instruction fusion operation is performed. On the one hand, the feedforward confidence factor is used to weight the adaptive impedance control instruction in the feedforward instruction buffer after look-ahead time step compensation to generate feedforward components. On the other hand, the conservative admittance compensation instruction calculated based on the current actual force feedback data is weighted using weights complementary to the feedforward confidence factor to generate feedback components.

[0195] System calculates conservative admittance compensation instructions This is a safety control strategy based on pure sensor feedback. Dynamic mixing is then executed.

[0196] Finally, the feedforward and feedback components are linearly superimposed to synthesize the final driving torque and send it to the servo driver, so as to achieve zero-delay feedforward control under low deviation and smoothly switch to feedback control under high deviation.

[0197] This implementation method employs a dynamic hybridization strategy to synthesize the final driving torque. :

[0198] ;

[0199] In this formula: This represents the final driving torque that is ultimately sent to the servo driver. Represents the feedforward instruction buffer The data extracted from the data were obtained after a look-ahead time step. Compensated adaptive impedance control command ; This represents a conservative admittance compensation instruction.

[0200] In this embodiment, the look-ahead time step The setting is key to achieving zero-latency biomimetic control.

[0201] Preferably, the The value is set at 10 ms. This value is calculated based on the calibration of the end-to-end delay of the control system, specifically including: EtherCAT bus communication delay (approximately 1 ms), servo driver current loop response delay (approximately 0.6 ms), and electromechanical inertia time constant of the mechanical transmission system (approximately 8 ms).

[0202] By prefetching control commands 10 ms ahead of the current time from the feedforward command buffer. The distribution process effectively compensates for the inherent lag in the physical components mentioned above. This means that the robot's joint torques are established in advance the instant the digital human's movements change, thus exhibiting a 'zero-latency' following effect that is completely synchronized with the digital human on a macroscopic level.

[0203] Regarding the conservative admittance compensation instruction Its internal parameters are usually set to high damping mode (e.g., damping ratio). This ensures that kinetic energy can be quickly dissipated when the system returns to feedback mode, achieving a soft landing.

[0204] The function of this formula in this embodiment is: when (Low deviation) The robot executes "future" instructions, so that the motor has already established a precise preload torque in the millisecond that physical contact occurs, completely eliminating the phase lag caused by the sensor circuit and realizing zero-delay feedforward control; when an accidental collision (high deviation) occurs, the system instantly and smoothly switches to feedback control to ensure physical safety.

[0205] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0206] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0207] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0208] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0209] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A biomimetic robot control system based on digital human motion mapping, characterized by, The method comprises the following steps: A visual micro-feature extraction module, a contact dynamics parameter inference module, a force sense impedance control generation module, and a time boundary feedforward execution module are included, wherein: The visual micro-feature extraction module receives a digital human operation video stream, performs time domain filtering and spatial amplification on the pixel data of the contact area between the human hand and the operation object, extracts pixel light intensity change features representing contact pressure and texture gradient features representing object deformation, and generates a space-time visual micro-feature tensor containing color shift and geometric deformation components after time synchronization alignment; The contact dynamics parameter inference module obtains the space-time visual micro-feature tensor and inputs it into a pre-trained physically constrained generative adversarial network, uses a generator to deduce corresponding physical properties, and outputs a predicted physical contact parameter vector containing a friction coefficient, an object stiffness coefficient, and a normal contact force; The adversarial network contains a loss function based on physical law constraints to verify the physical consistency of the output parameters; The force sense impedance control generation module uses inverse dynamics algorithm to map the predicted physical contact parameter vector to the dynamics model of the robot end effector, calculates the target contact stiffness according to the object stiffness coefficient, and obtains the target contact damping combined with the friction coefficient, to generate adaptive impedance control instructions containing dynamic stiffness matrix and damping matrix parameters; The time boundary feedforward execution module establishes a preset time length feedforward instruction buffer to store the control sequence, collects actual force feedback data from the end sensor during execution, calculates and compares the divergence of the energy state, and outputs the preset time length feedforward instruction buffer instruction to drive the joint actuator when the deviation is less than the threshold, to realize precise and efficient control; In the visual micro-feature extraction module, the specific operation of extracting pixel light intensity change features representing contact pressure is as follows: First, the digital human operation video stream is decomposed into a multi-level Laplacian pyramid structure, and the low-frequency image component containing color distribution information is selected from the Laplacian pyramid structure as the processing object; Then, the short-time Fourier transform algorithm is used to convert the low-frequency image component from the time domain to the frequency domain, and a preset frequency band interval covering the muscle tremor frequency and the blood volume change frequency is defined in the frequency domain; Next, a nonlinear amplification gain coefficient that changes dynamically with frequency is used to perform exponential enhancement processing on the chroma channel signal component within the preset frequency band interval, so that the skin color change with a signal-to-noise ratio below the preset threshold and distributed in a specific physiological frequency band interval is highlighted; Finally, a skin region mask covering the digital human hand region is constructed, and the enhanced chroma channel signal component is multiplied element by element with the skin region mask to filter out background noise and output the pixel light intensity change features; In the visual micro-feature extraction module, the specific operation of extracting texture gradient features representing object deformation is as follows: First, select the high-frequency image component containing texture details from the Laplacian pyramid structure, and calculate the instantaneous displacement velocity of the pixel points in the contact area between the digital human hand and the operation object to construct a dense optical flow vector field; Then, perform divergence operation on the dense optical flow vector field to quantify the compression or stretching deformation degree of the object surface by calculating the flux source and sink of the vector field; In the process, the modulus of the image brightness gradient is calculated to represent the texture saliency, and the dense optical flow vector field is weighted according to the modulus of the image brightness gradient to highlight the texture edge region; Finally, the second-order spatial variation rate of the dense optical flow vector field is calculated and multiplied by a smoothing regularization coefficient, and the weighted divergence operation result is superimposed with the second-order spatial variation rate, thereby generating the texture gradient feature.

2. The biomimetic robot control system based on digital human motion mapping of claim 1, wherein: In the contact dynamics parameter inference module, the specific operation of inferring the predicted physical contact parameter vector is: Firstly, the spatio-temporal visual micro-feature tensor is processed by multi-level convolution using a spatio-temporal encoder composed of a three-dimensional convolutional neural network, the time lag correlation between the pixel light intensity change feature and the texture gradient feature is extracted in the time dimension, and the spatio-temporal visual micro-feature tensor is encoded into a low-dimensional physical hidden feature vector; Then, the physical hidden feature vector is input into the generator model of the pre-trained physical constraint generative adversarial network, the physical hidden feature vector is reconstructed by a fully connected deconvolution layer, and the reconstructed feature is mapped to a preset physical value interval by a nonlinear activation function, thereby synthesizing the predicted physical contact parameter vector containing the normal contact force, the object stiffness coefficient and the friction coefficient.

3. The biomimetic robot control system based on digital human motion mapping of claim 2, wherein: In the contact dynamics parameter inference module, the specific operation of verifying the physical consistency using the loss function based on the physical law constraint is: Firstly, a contact mechanics constitutive residual potential energy equation containing a nonlinear elastic mapping operator and a Coulomb friction cone constraint term is constructed as a constraint condition for network training; Then, the Mahalanobis distance between the normal contact force and the elastic force is calculated using the contact mechanics constitutive residual potential energy equation as the elastic constitutive residual, wherein the elastic force is derived based on the object stiffness coefficient and the texture gradient feature processed by the nonlinear elastic mapping operator; At the same time, the exponential function value of the ratio of the tangential shear force estimate to the maximum static friction force is calculated as the friction constraint penalty, and the maximum static friction force is the product of the friction coefficient and the normal contact force; Finally, by minimizing the weighted sum of the elastic constitutive residual and the friction constraint penalty, the generator model is forced to output the predicted physical contact parameter vector that conforms to the contact mechanics law.

4. The biomimetic robot control system based on digital human motion mapping of claim 3, wherein: In the force sense impedance control generation module, the specific operation of generating the dynamic stiffness matrix parameter and the dynamic damping matrix parameter is: Firstly, the reference stiffness matrix of the robot in the non-contact free space state is set, and the object stiffness coefficient is extracted from the predicted physical contact parameter vector; Then, the stiffness adjustment amount is calculated using an exponential decay function logic, so that the diagonal element values in the generated dynamic stiffness matrix parameter show a nonlinear decreasing trend as the object stiffness coefficient increases, thereby constructing a complementary mapping of the environmental stiffness and the robot flexibility; Next, the virtual inertia matrix of the robot end effector is set, and the friction coefficient is extracted from the predicted physical contact parameter vector; Finally, the damping gain factor based on the friction coefficient is calculated using the inverse hyperbolic tangent function logic, and the damping gain factor is applied to the critical damping term synthesized by the dynamic stiffness matrix parameter and the virtual inertia matrix, thereby generating the dynamic damping matrix parameter with high damping characteristics in the low friction coefficient state.

5. The biomimetic robot control system based on digital human motion mapping of claim 4, wherein: In the force impedance control generation module, the specific operation of generating adaptive impedance control instructions is: First, the reference motion state generated by the digital human action mapping and the actual motion state of the robot end effector are obtained, and the position deviation vector and the velocity deviation vector between the two are calculated; Then, the position deviation vector is weighted using the dynamic stiffness matrix parameter, the velocity deviation vector is weighted using the dynamic damping matrix parameter, and the inertial force term is calculated in combination with the virtual inertia matrix; At the same time, the normal contact force in the predicted physical contact parameter vector is introduced as a feedforward force term into the control loop; Finally, the inertial force term, the weighted position deviation vector and the velocity deviation vector, and the feedforward force term are mapped to the joint space using the Jacobian transpose matrix of the robot, and the robot dynamics compensation term including gravity, Coriolis force and centrifugal force is superimposed, thereby synthesizing adaptive impedance control instructions for driving the robot joint actuator.

6. The biomimetic robot control system based on digital human motion mapping of claim 5, wherein: In the time boundary feedforward execution module, the specific operation of establishing a feedforward instruction buffer is: First, a shared memory space for storing data is opened, and the adaptive impedance control instructions within a preset time horizon are intercepted backward from the current system time; Then, an absolute time stamp is marked for each frame of adaptive impedance control instructions, thereby constructing a control sequence containing future time steps and storing it in the shared memory space to form a feedforward instruction buffer; At the same time, the clock synchronization calibration logic is executed, and the data source time axis of the adaptive impedance control instruction generation is aligned with the time axis of the robot controller using the network time protocol, ensuring that the instruction frames in the feedforward instruction buffer accurately correspond to the spatial positions that the robot will reach in the future.

7. The biomimetic robot control system based on digital human motion mapping of claim 6, wherein: In the time boundary feedforward execution module, the specific operation of comparing the deviation is: An energy state divergence operator is used for calculation. First, a sliding monitoring window is set, and the normal contact force corresponding to the current time is extracted from the feedforward instruction buffer as a predicted value; Then, the deviation vector of the actual force feedback data of the robot end sensor and the normal contact force in the contact normal vector direction is calculated, and the weighted quadratic form energy of the deviation vector is calculated using the preset force weighting metric matrix; At the same time, the rate of change of the actual force feedback data over time is calculated as a rigid collision detection term; Finally, the weighted sum of the weighted quadratic form energy and the rigid collision detection term in the sliding monitoring window is integrated to generate an energy state divergence representing the degree of deviation between real physical interaction and digital human experience prediction.

8. The biomimetic robot control system based on digital human motion mapping of claim 7, wherein: In the time boundary feedforward execution module, the specific operation of outputting the instructions for driving the robot joint actuator is: First, the confidence gating logic is constructed, and the feedforward confidence factor based on the energy state divergence is calculated using the S-type confidence decay function, so that the value of the feedforward confidence factor decreases smoothly with the increase of the energy state divergence; Subsequently, an instruction fusion operation is performed, on one hand, the adaptive impedance control instruction in the feedforward instruction buffer is weighted by the feedforward confidence factor to generate a feedforward component after being compensated by the look-ahead time step, on the other hand, the conservative admittance compensation instruction calculated based on the current actual force feedback data is weighted by a weight complementary to the feedforward confidence factor to generate a feedback component; Finally, the feedforward component and the feedback component are linearly superimposed to synthesize a final driving torque and sent to the servo driver, so as to realize zero-delay feedforward control in a low deviation state and smoothly switch to feedback control in a high deviation state.

Images