Operation method of bionic operation robot based on digital human

Abstract

The invention provides an operation method of a bionic operation robot based on a digital human. The invention discloses a bionic operation robot operation method based on a digital human. The bionic operation robot operation method comprises the following steps: S1, constructing a digital twin virtual human system and a bionic operation robot; s2, the digital twin virtual human system interacts with the bionic operation robot; and S3, a control room is constructed, and remote control over the bionic operation robot is achieved. The operation method of the bionic operation robot based on the digital human has the advantages that the calculated amount for establishing a human body digital twinning system can be greatly reduced, the quality of reconstructed twinning bodies is improved, the quality of the bionic robot is improved, and the bionic robot can replace manual work to enter a dangerous environment for work and rescue.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a bionic operating robot operating method based on digital human. Background Art

[0002] With the continuous advancement of technology, robots are used in all walks of life, effectively improving production efficiency.

[0003] However, most existing bionic operating robots have poor bionic effects and cannot provide physical feedback to the operator, which means that the operator can only rely on the bionic robot to perform a single action.

[0004] Therefore, it is necessary to provide a bionic operating robot operating method based on digital human to solve the above technical problems. Summary of the Invention

[0005] The technical problem solved by the present invention is to provide a bionic operating robot operation method based on digital humans that can greatly reduce the computational complexity of establishing a human digital twin system, improve the quality of the reconstructed twin, and thus improve the quality of the bionic robot, and can replace human labor in entering dangerous environments for work and rescue.

[0006] In order to solve the above technical problems, the present invention provides a method for operating a bionic operating robot based on a digital human, comprising the following steps: S1: Build a digital twin virtual human system and bionic operating robot; S2: Interact the digital twin virtual human system with the bionic operating robot; S3: Build a control room to realize remote control of the bionic operating robot.

[0007] Preferably, the construction of the digital twin virtual human system in S1 includes: construction of biological joints, construction of force interaction with the environment, and construction of passive structures and physiological actuators.

[0008] Preferably, in S2, the interaction between the digital twin virtual human system and the bionic operating robot includes: the mechanical influence of the interaction force on the human body, the update of joint data and the update of the virtual human state.

[0009] Preferably, the control room includes an operating room body, a master hand data glove and a console, and the console includes a console shell, control buttons, an emergency stop device, a voice system, a display system and a communication system.

[0010] Preferably, the communication system is used to connect the bionic operating robot terminal and the bionic teleoperation system; cooperate with the master hand data glove to transmit the information collected by the perception module of the robot and the bionic teleoperation system to the control center for data analysis and processing and distribution of control instructions; at the same time, the control center sends the data results to the intelligent software analysis system platform for display; the instructions issued by the control center can also be transmitted to the robot for execution through it.

[0011] Preferably, the voice system is used to collect audio from the robot's environment and play it to the operator in real time, so that the operator can better understand the operation of the equipment in the robot's environment. Specifically, the control personnel wears the AR / VR device and uses the walking mode to control the first-person perspective of the operator and the operator's own posture; The specific bionic operation method includes the following steps: T1: The human body heading angle and motion frequency sensor worn by the controller's legs detects the human body's heading angle and synchronously controls the robot's heading angle. The controller's stepping frequency controls the robot's travel speed, and the proportional coefficient of the travel speed is adjustable. This is the leg system of the bionic operating robot. T2: The horizontal and pitch posture sensors of the human head, installed on the VR glasses worn by the operator, synchronously follow the pan / tilt perspective of the 3D panoramic camera on the robot to ensure synchronization with the operator's perspective, forming the eye system of the bionic operating robot. T3: The upper arm, forearm, wrist and data glove sensors worn by the operator's right arm collect real-time shoulder height data to synchronously control the height of the robot's mechanical arm lifting platform, which serves as the waist system of the bionic operating robot; T4: The upper arm, forearm, wrist and data glove sensors collect 2-3 degrees of freedom of the human shoulder joint, such as horizontal, pitch, rotation and other posture data in real time. The forearm sensor mainly collects the posture of the elbow flexion and extension, the wrist sensor collects the posture of the wrist flexion and extension, and the data glove worn on the human hand collects the posture of the five fingers of the human hand with 6 degrees of freedom as well as force and tactile information for synchronous control of the single-arm dexterous hand to perform operations. It is the arm system of the bionic operation robot.

[0012] Compared with related technologies, the bionic operating robot operating method based on digital human provided by the present invention has the following beneficial effects: The present invention provides a method for operating a bionic operating robot based on a digital human, and a method for parameterizing the construction of a digital twin virtual human musculoskeletal system, which can greatly reduce the amount of calculation required to establish a digital twin system of the human body, improve the quality of the reconstructed twin, and thus improve the quality of the bionic robot. At the same time, through remote operation of the robot by an operator in a control room, the feedback signal of the bionic robot is used to act on the operator in the control room, thereby enabling the operator to better control the bionic robot, and thus utilizing the bionic robot to replace people in entering places that cannot be directly entered, such as fires and floods, to perform work, thereby ensuring work efficiency while better protecting personal safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A schematic diagram of a 3-DOF internal coordinate system model of the shoulder joint established using the ellipsoidal surface modeling method of the bionic manipulation robot operation method based on the digital human provided by the present invention; Figure 2 The principle of external interaction force between the lower limb hip, knee and ankle of the bionic operation robot operation method based on digital human provided by the present invention Figure 3 The principle and analysis diagram of the muscle mechanics model of the bionic operating robot based on digital human provided by the present invention; Figure 4 The muscle-tendon dynamics equations for the bionic operating robot operation method based on digital human provided by the present invention; Figure 5 A differential equation describing the degree of muscle activation in the bionic operating robot operating method based on digital human provided by the present invention; Figure 6 A schematic diagram of the structure of the knee joint gastrocnemius (calf) muscle path wrapped around a sleeve fixed on the tibia in the bionic operating robot operating method based on digital human provided by the present invention; Figure 7 A schematic diagram of the human upper limb external force interaction model for the bionic operating robot operation method based on the digital human provided by the present invention; Figure 8 Schematic diagram of the elbow joint tendon actuator model for the bionic operating robot operation method based on digital human provided by the present invention; Figure 9 A block diagram of the principle of a virtual human-synchronized bionic manipulation robot based on the bionic manipulation robot operation method of a digital human provided by the present invention; Figure 10 A system communication principle diagram of the bionic operating robot operation method based on digital human provided by the present invention; Figure 11 A schematic diagram of the force feedback principle between the master hand and the slave hand on the robot in the bionic manipulation robot manipulation method based on a digital human provided by the present invention; Figure 12 This is a working principle diagram of the visual system of the bionic operating robot based on the bionic operating robot operating method of the digital human provided by the present invention. DETAILED DESCRIPTION

[0014] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0015] Please refer to Figure 1-12 The bionic operation robot operation method based on digital human includes the following steps: 1. Construction of digital twin virtual human system and bionic operating robot: The method of constructing a virtual human through anthropometric parameters and heart rate, body fat percentage, muscle / movement force or other means should include three components: skeleton, muscles and organs.

[0016] 1.1 Biological joints Biological joints are different from artificially designed mechanical joints. It is difficult to simulate the unique effects of rotational, slider, universal, plane, ball and sleeve joints in mechanical structures. Therefore, using the joint-rigid body modeling method in mechanical design will greatly simplify or change the actual biomechanical structure.

[0017] Let's take the shoulder joint as an example to illustrate how to construct a biological joint. The shoulder joint is typically treated as an ellipsoidal chest surface, primarily involving rotational and translational motion. With three degrees of freedom, it can be modeled using six generalized coordinates and three constraint equations, while with four degrees of freedom, nine generalized coordinates and five constraint equations are required. When constructing a digital twin virtual human, any internal coordinate system can be modeled using one to six generalized coordinates. Traditional rigid body-joint modeling typically uses one to three generalized coordinates.

[0018] Using ellipsoid joints for modeling allows parameterizing the relative motion of moving bodies and attaching the motion of moving bodies relative to a reference body to an ellipsoidal surface. The motion of this surface is parameterized by two rotational coordinates (similar to latitude and longitude on Earth) and three three-dimensional rotations about the moving body, with the three rotation axes being perpendicular to the ellipsoid normal.

[0019] like Figure 1 This is a 3-DOF internal coordinate system model of the shoulder joint constructed using the ellipsoid modeling method. The scapula is a sub-rigid body constrained to the thorax ellipsoid relative to the thorax, capable of rotation about a Z-axis orthogonal to the ellipsoid. Similarly, other joints, such as the knee, can also be constrained using ellipsoids. The red muscle curves allow the tibia to perform translational (x, y) and rotational motion relative to the femur. The figure above shows the effect of rotating the knee joint by θ (positive extension) of 0°, 60°, and 120° in the generalized coordinate system.

[0020] 1.2 Force Interaction with the Environment (Forward Dynamics / Kinematics) There are many ways to model force. This paper uses a time vector function of force and action points, composed of force and pressure center data. Therefore, the external contact force at the end of a joint is a key data source for inferring the internal muscle force of the human body. Taking the human foot's contact with the ground as an example, an elastic foundation model is used to simplify the calculation of deformation and force.

[0021] The present invention provides a surface rolling constraint consisting of four constraint equations, including no surface penetration; two no-slip constraints at contact points on the surface; and a no-twist constraint about the surface normal at the experimentally measured contact points. Taking the hip, knee, and ankle joints of a single lower limb as an example, the lengths of the thigh and calf are L1 and L2, respectively. The corresponding hip joint is a 5-DOF ellipsoid, the knee joint is a 3-DOF ellipsoid, and the ankle joint is a 5-DOF ellipsoid. Muscles acting on these joints generate torques and forces along the gluteus maximus, respectively, which are projected onto the sagittal plane for analysis.

[0022] like Figure 2 It can be seen that the resultant force of the foot constraint on the ground is This function is obtained through forward kinematics analysis of the human body, so that we can know the external and internal interaction forces of the digital twin virtual person under known joint and bone dynamics. By extending from one leg, we can deduce the forward dynamics of the entire human body.

[0023] 1.3 Passive Structure (Skeleton) and Physiological Actuators (Virtual Muscles) (Joint Dynamics / Kinematics) The basic muscle mechanics model is defined as a combination of springs, dampers, sleeves, and ligaments, such as Figure 3 As shown, all these mechanical elements include the calculation of position and velocity. The actuator can be directly understood as the output of force. However, unlike the passive force, the output of force is a control calculation value, such as the current of the motor to generate torque. Figure 3 (a) Schematic diagram of a tendon actuator, which is a single muscle in series with an elastic tendon. F is the scalar tension, L is the length, the superscripts denote the muscle M, T the tendon, and MT the complete tendon actuator. α is the myofiber angle, which represents the average orientation of the muscle fibers in the muscle relative to the line of action and changes as a function of the muscle layer (LM) to maintain a constant muscle width, w. Phenomenological curves describe the force-generating capacity of the muscle contractile unit (CE, red) and the passive elastic unit (PE, green) as a function of normalized myofiber length (b) and shortening velocity (c). The fiber length is normalized by the optimal fiber length, which is the length at which the muscle can isometrically generate the maximum active force (zero velocity). Its velocity is normalized by the optimal fiber length and the maximum shortening rate, Vmax.

[0024] First, assume that muscle FM and tendon FT are in equilibrium at all times (1), and muscle activation a (0 < a ≤ 1) and muscle fiber length LM are two state variables of the actuator, thus establishing the muscle-tendon dynamics equation. The specific equation is as follows Figure 4 As shown in the formula, where kT and LST represent the tendon stiffness and relaxation length characteristics, fl and fv represent the physiological force-length and force-velocity relationships of the contractile element, and fPE represents the force-length curve of the passive element without normalized input.

[0025] The muscle itself is triggered by electrical signals to move. It can be understood that the degree of muscle activation is the strength of the electrical signal. Here, x is used to represent the muscle activation control signal, and Figure 5 The differential equation shown is used to describe the degree of muscle activation, where X can be represented as the control signal, 0 < x ≤1.

[0026] Muscles apply tension to bones along their path. Tensile paths (e.g., ropes) can be traversed through a specified path by attaching points to bones and wrapped around various geometric objects, such as cylinders, ellipsoids, and tori. Tension is applied at every point along the line segment connecting to adjacent points on the path. Therefore, all intermediate (through-hole) points exert two forces (two adjacent points), while the origin (first) and insertion (last) points of the path exert one force. In the case of wrapping, intermediate points are projected onto a ground reference line, with the number of points proportional to the ground reference line, and forces are applied to these points in the same manner.

[0027] Ligaments and muscles apply calculated forces to bones along the same paths, but unlike muscles, the tension generated by ligaments depends only on the path length and velocity.

[0028] 2. Interacting the digital twin virtual human system with the bionic operating robot: 2.1 Mechanical effects of interaction forces on the human body When the bionic manipulation robot interacts with the outside world, it can obtain the human multi-body kinematics data and the force data of the end through the sensors at the end of the limbs and the human body image. It is necessary to update the internal joint force and torque of the human body to synchronize it with the virtual human, so as to obtain the muscle force and bone stress of the virtual human. Figure 7 As shown in the figure, the upper limbs of the human body are simplified into three rigid bodies, where the shoulder joint is a 6-DOF ellipsoid, the elbow joint is a 2-DOF ellipsoid, and the wrist joint is a 4-DOF ellipsoid. When the external force is known, it can be calculated by inverse kinematics / dynamics. And update the digital twin virtual person.

[0029] 2.2 Joint data update After obtaining the joint motion and force data inferred from the external world through 5.1, it is necessary to analyze specific joints to determine the magnitude of the final force acting on the muscles, thereby determining the muscle state z of the digital twin virtual human. In addition to analyzing the virtual human's internal forces, the interaction forces of other limb extremities with the external world, such as the pressure or friction of the foot on the ground, can also be obtained through the sections in Chapter 4.

[0030] like Figure 8 As shown in the figure below, the human elbow joint is an ellipsoidal joint with two degrees of freedom (DOF). The forearm moves along a circle with radius R, with two degrees of freedom (θ1) and (θ2). Its motion is controlled by two muscles (the biceps and triceps). The corresponding mechanical analog structure of the muscles is also shown in the figure below. The joint's resultant force is decomposed into a vector F and a scalar M. Given the muscle fiber angles α1 and α2, the current muscle state z (represented by the excitability) and the muscle's tension can be determined.

[0031] The muscle state can be obtained by the following formula, where MT represents the corresponding simulation model of the tendon actuator.

[0032]

[0033] From this point on, the loop has been closed, inferring the musculoskeletal state of the joints by acquiring the force vectors (direction and magnitude) at the extremities and the state of the joints (angle and velocity). This data is updated in real time and generalized to the entire human body, keeping the digital twin virtual human system synchronized with the real physical system.

[0034] 2.3 Virtual Human Status Update Through external detection equipment, including dynamometers, limb end motion sensors, image sensors, etc., the data of the interaction between the bionic operating robot and the outside world is obtained. Combined with the digital twin system and the virtual human constructed by the joints, the internal movement and force details of the virtual human are generated. These details, combined with the muscle movement model, can obtain the muscle movement state of the corresponding joints of the virtual human, including the degree of excitement, etc., completing the mapping from the physical world to the digital twin system. However, the digital twin system is not a simple mapping relationship. It also needs to be combined with a control algorithm to allow the virtual human to execute the movements and physiological states of the bionic operating robot in synchronization, including Google movement, joint movement (angle, angular velocity), joint torque, muscle state, and even further deduce the interaction force of the virtual human's other limbs with the outside world, such as the pressure and friction of the sole of the foot on the ground, etc. Its update method is as follows: Figure 9 shown.

[0035] 3. Build a control room to realize remote control of the bionic operating robot: The operating room is mainly composed of three parts: the operating room body, the main hand data glove and the console.

[0036] The console includes console housing, control buttons (knobs), emergency stop device, voice system, display system and communication system.

[0037] The communication system primarily connects the robot terminal and the bionic teleoperation system. Working in conjunction with the master hand data glove, it transmits various information collected by the robot and bionic teleoperation system's perception modules (forearm and finger position, force, and tactile modules) (such as captured images, collected temperature, video data, 3D panoramic video information, forearm and finger position, finger command force, and tactile information) to the control center for data analysis and processing, as well as the distribution of control commands. The control center also sends the data results to the intelligent software analysis system platform for display. Commands from the control center can also be transmitted to the robot through it for execution.

[0038] like Figure 11 As shown, Figure 11 A schematic diagram of force feedback between the master hand and the slave hand on the robot. This allows the operator in the control room to intuitively feel the tactile sensation of the slave hand, enabling better control of the slave hand.

[0039] Display system: The display system includes the robot's visual acquisition system, specifically Figure 12 shown.

[0040] Voice system: The voice system mainly collects audio from the robot's environment and plays it to the operator in real time, so that the operator can better understand the operation of the equipment in the robot's environment.

[0041] After the controller puts on the AR / VR device, the controller can use the "walking" mode to control the first-person perspective and the controller's own posture (the specific implementation method of bionic operation: 1. The human body heading angle and motion frequency sensor worn by the controller's legs detects the human body's own heading angle and synchronously controls the robot's heading angle, and the frequency of the controller's stepping on the spot controls the robot's travel speed, and the proportional coefficient of the travel speed is adjustable, which can be called the "leg" system of the bionic operation robot; 2. The human head horizontal and pitch posture sensor set on the VR glasses worn by the controller synchronously controls the pan-tilt perspective of the 3D panoramic camera on the robot in the far field to ensure that it is synchronized with the controller's perspective, which can be called the "eye" system of the bionic operation robot; 3. The controller The upper arm, forearm, wrist and data glove sensors worn on the right arm collect the height data of the human shoulder in real time for synchronous control of the lifting height of the robot's mechanical arm lifting platform, which can be called the "waist" system of the bionic operating robot; 4. The upper arm, forearm, wrist and data glove sensors collect the posture data of 2~3 degrees of freedom of the human shoulder joint, such as horizontal, pitch, rotation, etc. in real time. The forearm sensor mainly collects the posture data of the flexion and extension of the elbow joint. The wrist sensor collects the posture data of the flexion and extension of the wrist joint. The data glove worn on the human hand collects the posture data of 6 degrees of freedom of the five fingers of the human hand as well as force and tactile information for synchronous control of the single-arm dexterous hand for operation. It can be called the "arm" system of the bionic operating robot;) The remote-field control robot realizes roaming and bionic teleoperation functions at the substation site.

[0042] Compared with related technologies, the bionic operating robot operating method based on digital human provided by the present invention has the following beneficial effects: The present invention provides a method for operating a bionic operating robot based on a digital human, and a method for parameterizing the construction of a digital twin virtual human musculoskeletal system, which can greatly reduce the amount of calculation required to establish a digital twin system of the human body, improve the quality of the reconstructed twin, and thus improve the quality of the bionic robot. At the same time, through remote operation of the robot by an operator in a control room, the feedback signal of the bionic robot is used to act on the operator in the control room, thereby enabling the operator to better control the bionic robot, and thus utilizing the bionic robot to replace people in entering places that cannot be directly entered, such as fires and floods, to perform work, thereby ensuring work efficiency while better protecting personal safety.

[0043] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for operating a bionic operating robot based on a digital human, characterized in that: The following steps are involved: S1: Build a digital twin virtual human system and bionic operating robot; S2: Interact the digital twin virtual human system with the bionic operating robot; S3: Build a control room to realize remote control of the bionic operating robot.

2. The bionic operating robot operating method based on digital human according to claim 1, characterized in that: The construction of the digital twin virtual human system in S1 includes: the construction of biological joints, the construction of force interaction with the environment, and the construction of passive structures and physiological actuators.

3. The bionic operating robot operating method based on digital human according to claim 1, characterized in that: In S2, the interaction between the digital twin virtual human system and the bionic operating robot includes: the mechanical influence of the interaction force on the human body, the update of joint data and the update of the virtual human state.

4. The bionic operating robot operating method based on digital human according to claim 1, characterized in that: The control room includes an operating room body, a master hand data glove and a console. The console includes a console shell, control buttons, an emergency stop device, a voice system, a display system and a communication system.

5. The bionic operating robot operating method based on digital human according to claim 4 is characterized in that: The communication system is used to connect the bionic operating robot terminal and the bionic teleoperation system; in conjunction with the master hand data glove, the information collected by the perception module of the robot and the bionic teleoperation system is transmitted to the control center for data analysis and processing, as well as the distribution of control instructions; at the same time, the control center sends the data results to the intelligent software analysis system platform for display; the instructions issued by the control center can also be transmitted to the robot for execution through it.

6. The method for operating a bionic operating robot based on a digital human according to claim 4, characterized in that: The voice system is used to collect audio from the robot's environment and play it to the operator in real time, so that the operator can better understand the operation of the equipment in the robot's environment. Specifically, the operator wears the AR / VR device and uses the walking mode to control the operator's first-person perspective and the operator's own posture; The specific bionic operation method includes the following steps: T1: The human body heading angle and motion frequency sensor worn by the controller's legs detects the human body's heading angle and synchronously controls the robot's heading angle. The controller's stepping frequency controls the robot's travel speed, and the proportional coefficient of the travel speed is adjustable. This is the leg system of the bionic operating robot. T2: The horizontal and pitch posture sensors of the human head, installed on the VR glasses worn by the operator, synchronously follow the pan / tilt perspective of the 3D panoramic camera on the robot to ensure synchronization with the operator's perspective, forming the eye system of the bionic operating robot. T3: The upper arm, forearm, wrist and data glove sensors worn by the operator's right arm collect real-time shoulder height data to synchronously control the height of the robot's mechanical arm lifting platform, which serves as the waist system of the bionic operating robot; T4: The upper arm, forearm, wrist and data glove sensors collect 2-3 degrees of freedom of the human shoulder joint, such as horizontal, pitch, rotation and other posture data in real time. The forearm sensor mainly collects the posture of the elbow flexion and extension, the wrist sensor collects the posture of the wrist flexion and extension, and the data glove worn on the human hand collects the posture of the five fingers of the human hand with 6 degrees of freedom as well as force and tactile information for synchronous control of the single-arm dexterous hand to perform operations. It is the arm system of the bionic operation robot.

Images