Virtual-real synchronous digital twin bionic hand bidirectional control method

Abstract

The invention relates to the field of bionic robot control, in particular to a virtual-real synchronous digital twin bionic hand two-way control method, which realizes multi-mode cooperative control and dynamic optimization by constructing two-way mapping of a physical entity and a virtual model. According to the method, the dynamic modeling precision is improved by adopting an improved D-H parameter method, the system response delay is controlled within 50ms in combination with a Unity 3D virtual simulation and multi-thread communication technology, three control modes of wearable gloves, machine vision and a digital interface are supported, the grabbing success rate reaches 98%, and the method is suitable for the fields of medical rehabilitation, industrial automation and the like.

Description

Technical Field

[0001] The present invention relates to the field of bionic robot control, and in particular to a bidirectional control method of a virtual-real synchronized digital twin bionic hand. Background Art

[0002] With the rapid development of artificial intelligence and robotics, bionic manipulators, as an important carrier of human-computer interaction, have shown great potential in the fields of medical rehabilitation, industrial collaboration, and life assistance. However, traditional bionic hands generally have problems such as insufficient flexibility, low control accuracy, and poor adaptability. The core reason is that the separate design of the mechanical structure and the control system makes it difficult to achieve dynamic perception and real-time feedback. In addition, existing bionic hands mostly rely on pre-programmed or simple sensor signals, and cannot accurately simulate the complex movements and coordination capabilities of the human hand. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a bidirectional control method for a digital twin bionic hand with virtual and real synchronization, which effectively solves the problems mentioned in the above background technology.

[0004] In order to solve the above problems, the technical solution adopted by the present invention is:

[0005] The bidirectional control method of a virtual-real synchronized digital twin bionic hand comprises the following steps:

[0006] S1. Construct a bidirectional mapping model between the physical entity of the bionic hand and the virtual digital twin:

[0007] The kinematic model of the bionic hand was established based on the improved DH parameter method, and the joint motion space was verified through MATLAB simulation;

[0008] Construct a virtual bionic hand model that is consistent with the physical entity structure in the Unity 3D environment, and establish a real-time data communication channel between the two;

[0009] S2. Establish real-time synchronous control of physical entities and virtual models:

[0010] Collect motion data of physical entities through multi-source sensors;

[0011] Map physical entity data to the virtual model in real time, and drive the virtual model to move synchronously;

[0012] Reversely regulate the actuators of the physical entity through the control instructions of the virtual model

[0013] S3. Establish multimodal human-computer interaction control:

[0014] Support wearable glove control mode: collect hand joint angles through potentiometers, and convert them into PWM signals to drive the bionic hand through the STM32 main controller;

[0015] Support machine vision gesture recognition control mode: extract the coordinates of key points of the hand through OpenCV and transmit control instructions via TCP protocol;

[0016] Support direct control of the digital twin interface: adjust virtual model parameters through the front-end interface developed by Qt and drive the physical entity synchronously;

[0017] S4. Dynamic parameter optimization and fault prediction:

[0018] Monte Carlo simulation based on virtual model to optimize the grasping path;

[0019] Through the analysis of virtual and real data deviation, mechanical structure abnormality warning can be achieved.

[0020] The improved DH parameter method in step S1 is specifically:

[0021] The DH modeling method improved by Craig is adopted, and the origin of the connecting rod coordinate system is set at the head end of the connecting rod;

[0022] A four-parameter model including joint rotation, joint offset, link length and link torsion angle is established, and the position and posture of the end effector are calculated through the homogeneous transformation matrix.

[0023] The implementation of the real-time data communication channel in step S2 includes:

[0024] Use C# scripts to write slider monitoring functions in Unity to convert virtual model joint angles into physical entity control instructions;

[0025] The virtual and real data synchronization is achieved through a multi-threaded serial communication protocol, where the baud rate of the serial communication protocol is ≥115200bps and the data refresh rate is ≥30Hz.

[0026] The control mode of the wearable gloves in step S3 is specifically:

[0027] The bending angles of five fingers are collected by Alps RK09 potentiometer and converted into digital signals by the ADC module of STM32F103RBT6.

[0028] The PWM control instructions are transmitted to the bionic hand slave controller through the ESP8266 wireless module, driving the DS3120 servo to achieve multi-joint linkage.

[0029] The machine vision gesture recognition control mode in step S3 includes:

[0030] Extract the coordinates of 21 key points of the hand based on OpenCV and MediaPi pe library;

[0031] Generate control instructions by calculating the curvature of the finger;

[0032] The TCP / IP protocol is used to realize full-duplex communication between the PC and the NodeMCU controller.

[0033] The dynamic parameter optimization in step S4 is specifically as follows:

[0034] Preset the physical properties of the grasping target in the virtual model;

[0035] The grasping path is generated by random sampling using the Monte Carlo method, and the path with the least force on the end effector is selected as the optimal solution.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] The present invention realizes multimodal collaborative control and dynamic optimization by constructing a bidirectional mapping between physical entities and virtual models. It also adopts an improved DH parameter method to improve the accuracy of kinematic modeling. It combines Unity 3D virtual simulation and multi-threaded communication technology to control the system response delay within 50ms. It supports three control modes: wearable gloves, machine vision, and digital interface. The grasping success rate reaches 98%, and it is suitable for medical rehabilitation, industrial automation and other fields. DETAILED DESCRIPTION

[0038] The following are specific embodiments of the present invention, and further describe the technical solutions of the present invention, but the present invention is not limited to these embodiments.

[0039] The bidirectional control method of a virtual-real synchronized digital twin bionic hand comprises the following steps:

[0040] S1. Construct a bidirectional mapping model between the physical entity of the bionic hand and the virtual digital twin:

[0041] The kinematic model of the bionic hand was established based on the improved DH parameter method, and the joint motion space was verified through MATLAB simulation;

[0042] Construct a virtual bionic hand model that is consistent with the physical entity structure in the Unity 3D environment, and establish a real-time data communication channel between the two;

[0043] S2. Establish real-time synchronous control of physical entities and virtual models:

[0044] Collect motion data of physical entities through multi-source sensors;

[0045] Map physical entity data to the virtual model in real time, and drive the virtual model to move synchronously;

[0046] Reversely regulate the actuators of the physical entity through the control instructions of the virtual model

[0047] S3. Establish multimodal human-computer interaction control:

[0048] Support wearable glove control mode: collect hand joint angles through potentiometers, and convert them into PWM signals to drive the bionic hand through the STM32 main controller;

[0049] Support machine vision gesture recognition control mode: extract the coordinates of key points of the hand through OpenCV and transmit control instructions via TCP protocol;

[0050] Support direct control of the digital twin interface: adjust virtual model parameters through the front-end interface developed by Qt and drive the physical entity synchronously;

[0051] S4. Dynamic parameter optimization and fault prediction:

[0052] Monte Carlo simulation based on virtual model to optimize the grasping path;

[0053] Through the analysis of virtual and real data deviation, mechanical structure abnormality warning can be achieved.

[0054] The improved DH parameter method in step S1 is specifically:

[0055] The DH modeling method improved by Craig is adopted, and the origin of the connecting rod coordinate system is set at the head end of the connecting rod;

[0056] A four-parameter model including joint rotation, joint offset, link length and link torsion angle is established, and the position and posture of the end effector are calculated through the homogeneous transformation matrix.

[0057] The implementation of the real-time data communication channel in step S2 includes:

[0058] Use C# scripts to write slider monitoring functions in Unity to convert virtual model joint angles into physical entity control instructions;

[0059] The virtual and real data synchronization is achieved through a multi-threaded serial communication protocol, where the baud rate of the serial communication protocol is ≥115200bps and the data refresh rate is ≥30Hz.

[0060] The control mode of the wearable gloves in step S3 is specifically:

[0061] The bending angles of five fingers are collected by Alps RK09 potentiometer and converted into digital signals by the ADC module of STM32F103RBT6.

[0062] The PWM control instructions are transmitted to the bionic hand slave controller through the ESP8266 wireless module, driving the DS3120 servo to achieve multi-joint linkage.

[0063] The machine vision gesture recognition control mode in step S3 includes:

[0064] Extract the coordinates of 21 key points of the hand based on OpenCV and MediaPi pe library;

[0065] Generate control instructions by calculating the curvature of the finger;

[0066] The TCP / IP protocol is used to realize full-duplex communication between the PC and the NodeMCU controller.

[0067] The dynamic parameter optimization in step S4 is specifically as follows:

[0068] Preset the physical properties of the grasping target in the virtual model;

[0069] The grasping path is generated by random sampling using the Monte Carlo method, and the path with the least force on the end effector is selected as the optimal solution.

[0070] Grasping action virtual and real synchronous control:

[0071] 1. The operator wears gloves and bends his index finger. The output voltage change of the potentiometer is converted into a digital signal by ADC.

[0072] 2. The STM32 controller generates PWM instructions to drive the index finger servo of the bionic hand to rotate;

[0073] 3. At the same time, the Unity virtual model receives angle data through the serial port and updates the virtual finger posture;

[0074] 4. If the virtual model detects a collision interference, it sends adjustment instructions to the physical entity in the opposite direction to reduce the servo torque.

[0075] Remote gesture recognition control:

[0076] 1. The camera captures the "fist" gesture, and OpenCV extracts the coordinates of key points;

[0077] 2. Send instructions to the NodeMCU controller through the TCP protocol to drive the five fingers to bend synchronously;

[0078] 3. The virtual model displays the grasping trajectory in real time and optimizes the path through the Monte Carlo method to reduce end jitter.

[0079] The present invention realizes multimodal collaborative control and dynamic optimization by constructing a bidirectional mapping between physical entities and virtual models. It also adopts an improved DH parameter method to improve the accuracy of kinematic modeling. It combines Unity 3D virtual simulation and multi-threaded communication technology to control the system response delay within 50ms. It supports three control modes: wearable gloves, machine vision, and digital interface. The grasping success rate reaches 98%, and it is suitable for medical rehabilitation, industrial automation and other fields.

[0080] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways.

Claims

1. A bidirectional control method for a digital twin bionic hand with virtual and real synchronization, characterized in that , including the following steps: S1. Construct a bidirectional mapping model between the physical entity of the bionic hand and the virtual digital twin: The kinematic model of the bionic hand was established based on the improved DH parameter method, and the joint motion space was verified through MATLAB simulation; Build a virtual bionic hand model that is consistent with the physical entity structure in the Unity 3D environment, and establish a real-time data communication channel between the two; S2. Establish real-time synchronous control of physical entities and virtual models: Collect motion data of physical entities through multi-source sensors; Map physical entity data to the virtual model in real time, and drive the virtual model to move synchronously; Reversely regulate the actuators of the physical entity through the control instructions of the virtual model S3. Establish multimodal human-computer interaction control: Support wearable glove control mode: collect hand joint angles through potentiometers, and convert them into PWM signals to drive the bionic hand through the STM32 main controller; Support machine vision gesture recognition control mode: extract the coordinates of key points of the hand through OpenCV and transmit control instructions via TCP protocol; Support direct control of the digital twin interface: adjust virtual model parameters through the front-end interface developed by Qt and drive the physical entity synchronously; S4. Dynamic parameter optimization and fault prediction: Monte Carlo simulation based on virtual model to optimize the grasping path; Through the analysis of virtual and real data deviation, mechanical structure abnormality warning can be achieved.

2. The bidirectional control method of the virtual-real synchronized digital twin bionic hand according to claim 1, characterized in that: The improved DH parameter method in step S1 is specifically: The DH modeling method improved by Craig is adopted, and the origin of the connecting rod coordinate system is set at the head end of the connecting rod; A four-parameter model including joint rotation, joint offset, link length and link torsion angle is established, and the position and posture of the end effector are calculated through the homogeneous transformation matrix.

3. The bidirectional control method of the virtual-real synchronized digital twin bionic hand according to claim 1, characterized in that: The implementation of the real-time data communication channel in step S2 includes: Use C# scripts to write slider monitoring functions in Unity to convert virtual model joint angles into physical entity control instructions; The virtual and real data synchronization is achieved through a multi-threaded serial communication protocol, where the baud rate of the serial communication protocol is ≥115200bps and the data refresh rate is ≥30Hz.

4. The bidirectional control method of the virtual-real synchronized digital twin bionic hand according to claim 1, characterized in that: The control mode of the wearable gloves in step S3 is specifically: The bending angles of the five fingers are collected using an Alps RK09 potentiometer and converted into digital signals by the ADC module of the STM32F103RBT6. The PWM control instructions are transmitted to the bionic hand slave controller through the ESP8266 wireless module, driving the DS3120 servo to achieve multi-joint linkage.

5. The virtual-real synchronous bidirectional control method of the digital twin bionic hand according to claim 1, characterized in that: The machine vision gesture recognition control mode in step S3 includes: Extract the coordinates of 21 key points of the hand based on OpenCV and MediaPipe libraries; Generate control instructions by calculating the curvature of the finger; The TCP / IP protocol is used to realize full-duplex communication between the PC and the NodeMCU controller.

6. The virtual-real synchronous bidirectional control method of the digital twin bionic hand according to claim 5 is characterized in that: The dynamic parameter optimization in step S4 is specifically as follows: Preset the physical properties of the grasping target in the virtual model; The grasping path is generated by random sampling using the Monte Carlo method, and the path with the least force on the end effector is selected as the optimal solution.