Multi-form haptic feedback system based on cooperation of double mechanical arms

Abstract

The invention discloses a multi-form force tactile feedback system based on cooperation of double mechanical arms. The tail ends of the double mechanical arms are respectively provided with a touch screen and a mechanical operation piece panel, a user wears a helmet display to obtain a real cockpit view field in the using process, the system predicts user hand motion interaction data and interaction types in real time and decides an execution arm reaching an interaction target point, and the double mechanical arms cooperatively move to send a target control to the hand position of the user. And multi-form haptic feedback matched with the view field is provided for the user. According to the invention, multiple types of driving environments can be provided for users through the virtual reality technology, and the system flexibility is improved; different forms of force tactile feedback control panels are carried at the tail end of a double-mechanical-arm system, and two interactive control types of a touch screen and a mechanical operation piece are provided for a user to meet the simulation training requirement of an intelligent cockpit. The efficient cooperation of the double-mechanical-arm feedback system in the working space effectively improves the operation efficiency of a user in the large-range working space.

Description

Technical Field

[0001] The present invention belongs to the field of virtual reality technology, and in particular to the field of information-based virtual cockpits. The present invention is a multi-form force tactile feedback system based on the collaboration of two robotic arms. Background Art

[0002] Virtual cockpits are an important tool for pilot training. The lack of force-tactile interaction hinders further improvement in force immersion and interactivity, leading to operational errors and reduced mission efficiency. Researchers have developed several force-tactile interaction methods. For example, wearable devices and electrotactile devices are used to provide force-tactile feedback in virtual cockpits. However, wearable devices can be quite restrictive. Companies such as Force Dimension and Sensable have developed desktop force-tactile interaction devices that provide force-tactile feedback by driving parallel robots or small serial robots, but these devices can only achieve small-scale interaction. These methods struggle to simulate the large, multi-directional control panels in virtual cockpits, which often feature a wide variety of control types. The Royal Netherlands Aerospace Center and Nanjing University of Aeronautics and Astronautics have both conducted research in the development of semi-virtual cockpits. These cockpits retain physical control panels and provide users with realistic force-tactile feedback, but they cannot simulate different cockpit types and lack flexibility. The TOPT force haptic solution developed in the United States uses a servo-mechanical system equipped with various controls. This simplifies the control panel and operating mechanism, and allows for the simulation of different control panels by changing the software. However, this mechanical system is still large, and the controls only move in two dimensions, preventing three-dimensional interaction. While this approach can simulate a large range of virtual cockpit control panels, it fails to leverage the advantages of virtual cockpits, such as low cost, small size, and high flexibility.

[0003] To address these issues, the State Key Laboratory of Virtual Reality at Beihang University has designed a servo-based active tactile feedback system. Users can access multiple simulated operational scenarios by wearing a helmet-mounted display (HMD) compatible with the cockpit. The force tactile feedback mechanism consists of a six-degree-of-freedom robotic arm with slide rails, the end of which carries the interactive controls required for the currently rendered scene. The operating process is divided into a pre-interaction servo motion phase and a post-interaction active tactile feedback phase based on the interaction between the mechanism and the user's hand. This system addresses the bulky and inflexible nature of traditional virtual cockpits, enabling flexible interaction within a wide range of workspaces with a lightweight system. At the same time, the force tactile rendering computer provides users with force tactile feedback that matches the visual scene, enhancing the user experience.

[0004] The development of modern flight technology places high demands on piloting, leading to increasingly complex cockpit interactions. To reduce operational complexity and cognitive burden on pilots, cockpit design is trending towards information-based and intelligent interfaces. Interactions combining touchscreens and mechanical controls are a trend in current cockpit development. For example, the US F-35 fighter jet's cockpit, which combines touchscreens and mechanical controls, transforms some piloting operations into touch-based operations, improving operational convenience. Information-based cockpits are a crucial tool for pilot training on the ground. For the intelligent transformation of modern aircraft cockpits combining touchscreens and mechanical controls, conventional force haptic feedback mechanisms based on single-arm servo motion are unable to meet these requirements. Consequently, there is an urgent need to develop a multi-modal force haptic feedback system based on the collaboration of two robotic arms. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention proposes a multi-modal force haptic feedback system based on the collaboration of two robotic arms. By equipping the ends of the dual robotic arms with different types of force haptic feedback control panels, the system provides users with two types of interactive controls: a touch screen and mechanical controls, meeting the requirements of intelligent cockpit simulation training. The present invention also relates to a method for operating the multi-modal force haptic feedback system based on the collaboration of two robotic arms.

[0006] The present invention solves its technical problems by adopting a technical solution: a multi-modal force tactile feedback system based on dual-manipulator collaboration, comprising a computer, a motion tracking sensor, a dual-manipulator force tactile feedback mechanism, and its controller. The computer comprises a motion prediction computer and motion planning computers 1 and 2 for the two manipulators. The motion prediction computer is used to calculate hand pose, collision detection, predict hand motion trajectories before contact with a target control, and generate real-time force tactile responses after contact. The motion planning computer is used to read the manipulator state and control motion. Motion planning computer 1 is equipped with a dual-manipulator collaborative motion planning algorithm. It uses the motion prediction computer results and a rendered scene as input. It considers collisions between the two manipulators and the environment, as well as collisions between the two manipulators, to avoid contact between the hands and the interactive device. The three computers are connected by a network cable and communicate via the Ethernet protocol for data exchange and computational synchronization. The motion tracking sensor is connected to the motion prediction computer to collect hand motion data, including palm position and orientation data, as well as finger joint angle data, as input to the motion prediction computer. The dual-arm force haptic feedback mechanism consists of two robotic arms, each equipped with a touchscreen panel and a mechanical control panel, and two supporting slide rails. The user is presented with a realistic cockpit view by wearing a helmet-mounted display. The dual-arm force haptic feedback mechanism serves as an interactive device positioned directly in front of the user. The dual-arm collaborative workspace covers the cockpit operating space, with the touchscreen panel and mechanical control panel used to simulate the touchscreen and mechanical controls found in a real cockpit, respectively. The touchscreen display and mechanical control types are matched to the user's field of view and are flexibly adjustable based on the aircraft model and rendering scene. A motion prediction computer acquires data from motion tracking sensors to predict the user's hand movement intentions in real time and predict interaction data. This interaction data includes the user's hand movement direction, interaction target point, and interaction time. This computer then determines the type of interaction when the user's hand reaches the interaction target point in the virtual scene, which includes touchscreen panel interaction and mechanical control panel interaction. Motion planning computer 1 plans the coordinated movement of the two robotic arms and synchronizes the results to motion planning computer 2 to control the movement of the two robotic arms. When the user's hand has not reached the target point for interaction with the virtual scene, the robotic arm equipped with the target control panel follows the servo movement of the user's hand; when it reaches the target point for interaction with the virtual scene, the target control interacts with the hand to provide the user with corresponding tactile feedback. During the use process, the two robotic arms coordinate and cooperate in movement. Motion planning computer 1 obtains the interaction data and interaction type from the motion prediction computer, decides the execution arm to reach the interaction target point and the priority of the two robotic arms' movement, plans the coordinated movement of the two robotic arms based on the secondary priority avoidance, and then sends the motion planning results to motion planning computer 2. The two motion planning computers generate movement instructions for the two robotic arms so that the robotic arm equipped with the target control panel reaches the interaction target point and interacts with the user without collision within the predicted time.The robotic arms 1 and 2 equipped with touch screen panels and mechanical control panels receive control signals from controllers 1 and 2 respectively. The controllers 1 and 2 are connected to motion planning computers 1 and 2 respectively, receive control instructions from the computers, output torque signals to drive the motors of the robotic arms and slide rails, and feed back motion information to the computers.

[0007] The operating method of a multi-modal force tactile feedback system for an information-based virtual cockpit based on dual-manipulator collaboration includes a dual-manipulator collaborative servo motion phase before the user's hand contacts the target control, and a force tactile feedback phase after contact. A motion tracking sensor collects the user's hand motion data in real time, transmits it to a motion prediction computer to calculate the hand pose, and renders the virtual environment. Collision detection is performed based on the positions of the virtual hand and virtual controls in the virtual environment. When the hand is not in contact with the target control, the dual-manipulator force tactile feedback mechanism is in the dual-manipulator collaborative servo motion phase. Based on the interaction data and interaction type obtained by the motion prediction computer and the rendered virtual scene, the dual-manipulator force tactile feedback mechanism determines which manipulator moves to the interaction target point. The manipulator tracks the hand's motion. When the hand reaches the interaction target point and contacts the target control, the dual-manipulator force tactile feedback mechanism enters the active force tactile feedback phase.

[0008] (1) Dual-arm collaborative servo motion stage

[0009] 1) Single robotic arm provides force tactile feedback

[0010] During simulated communication and cruise phase flight parameter adjustment, the pilot can operate with one hand. In this scenario, only one of the dual-arm force tactile feedback mechanisms, which carries the target control panel, needs to interact with the user, providing a force tactile feedback scenario for the single arm. The motion prediction computer receives the hand motion prediction data, predicts the hand motion trajectory, determines the type of the interactive target control, and determines the moment t when the hand interacts with the target control. c , interaction position p c and interaction direction q c , and sends the data to the motion planning computer 1 equipped with the dual-manipulator collaborative motion algorithm. The motion planning computer 1 decides the execution arm that moves to the interactive target point, taking into account the obstacle avoidance between the dual manipulators and the environment and the obstacle avoidance between the dual manipulators, calculates the obstacle avoidance strategy and motion path of the dual manipulators, and sends it to the motion planning computer 2. The motion planning computers 1 and 2 respectively calculate the collision-free coordinated motion trajectory and control instructions of the manipulators 1 and 2, and send them to the controllers 1 and 2 to drive the dual manipulator force tactile feedback mechanism to move, so that the manipulator carrying the target control panel can move in t without collision. c Send the corresponding target control to the interactive target point p at any time c As the hand moves, the predicted hand movement trajectory and interaction time t c , interaction target point p cand interaction direction q c Continuously updating requires real-time judgment of the interaction type to determine the robotic arm that performs servo motion, and recalculating and updating the dual robotic arm obstacle avoidance strategy, motion path, motion trajectory, and control instructions until the prediction t c 、p c ,q c Converges to the true value.

[0011] 2) Dual robotic arms provide force tactile feedback

[0012] When flying in adverse weather conditions or other complex scenarios, the pilot needs to use both hands to maintain stable control of the aircraft. One hand operates the joystick to adjust the aircraft's attitude, and the other hand operates the buttons on the touch screen control panel to respond to emergencies. In order to provide corresponding force tactile feedback to both hands at the same time, the two robotic arms need to be driven simultaneously and sent to the target operation positions of both hands respectively. The motion prediction computer receives the user's hand motion data and calculates the time t when the user's hands interact with the corresponding target controls. c1 , t c2 , position p c1 、p c2 , direction q c1 ,q c2 The interaction data is collected and the interaction type between the user's hand and the interaction target point in the virtual scene is determined, and the interaction data is sent to the motion planning computer 1. The motion planning computer 1 decides the execution arms that move to the interaction target points respectively, and considers the obstacle avoidance between the dual robotic arms and the environment as well as the obstacle avoidance between the dual robotic arms, calculates the obstacle avoidance strategy and motion path of the dual robotic arms, and sends them to the motion planning computer 2. The motion planning computers 1 and 2 respectively calculate the collision-free coordinated motion trajectory and control instructions of the robotic arms 1 and 2, and send them to the controllers 1 and 2 to drive the force tactile feedback mechanism of the dual robotic arms to move, so that the dual robotic arms can move the target control at t without collision. c1 , t c2 As the hand moves, the predicted hand movement trajectory and interaction time t c1 , t c2 , interaction position p c1 、p c2 and interaction direction q c1 ,q c2 Continuous updating requires real-time re-judgment of the interaction type between the user's hand and the interactive target point in the virtual scene and the corresponding servo motion robotic arm, and recalculation and update of the dual robotic arm obstacle avoidance strategy, motion path, motion trajectory and control instructions until they converge to the true value.

[0013] (2) Force tactile feedback stage

[0014] When the user's hand reaches the target interaction point to perform an operation, the dual-arm force tactile feedback mechanism delivers the target control to the desired position in the appropriate direction, providing the user with force tactile feedback that matches their visual perception. A motion prediction computer, combined with rendered virtual scene calculations, performs real-time collision response and force tactile generation, driving the dual-arm force tactile feedback mechanism to output force, providing active force tactile feedback. This provides tactile feedback for operating the touchscreen panel and mechanical controls, and provides displacement resistance when the control reaches its limit.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] 1. Dual robotic arms equipped with touch screens and mechanical control panels provide users with a variety of driving operations to meet the development needs of intelligent driving.

[0017] 2. Considering two types of dual-arm mission scenarios in aircraft piloting tasks, the execution system comprehensively considers user behavioral intentions, dual-arm avoidance, and interaction with the environment. Planning strategies such as dual-arm collaborative task execution decision-making and dual-arm trajectory conflict resolution are designed to achieve real-time interaction between the dual-arm and the user in a shared space.

[0018] 3. This invention utilizes a multi-modal force tactile feedback system based on dual-arm collaboration. Its application process includes a dual-arm collaborative servo motion phase before contact between the hand and the target control, and a force tactile feedback phase after contact. The dual-arm collaboration enables collision-free movement within a shared space, enabling flexible operation across a wide range of workspaces and effectively improving operational efficiency. The dual-arm force tactile feedback mechanism provides users with different forms of force tactile feedback based on the type of end-mounted control, giving them immediate, authentic sensory feedback, enhancing the user experience, and improving training effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the structure of the multi-form force tactile feedback system based on dual-manipulator collaboration of the present invention;

[0020] Figure 2 It is a schematic structural diagram of the dual-manipulator force tactile feedback mechanism of the present invention;

[0021] Figure 3 This is a flow chart of the working method of the multi-form force tactile feedback system based on dual robotic arm collaboration of the present invention; DETAILED DESCRIPTION

[0022] like Figure 1 、 2 As shown, a multi-form force tactile feedback system based on dual robotic arm collaboration is composed of a computer 1, a motion tracking sensor 2, a dual robotic arm force tactile feedback mechanism 3 and a controller 4.

[0023] Computer 1 consists of three computers: PC1, PC2, and PC3, which are motion prediction computer, motion planning computer 1, and motion planning computer 2 respectively. The motion prediction computer PC1 is used for hand posture calculation, collision detection, hand motion trajectory prediction before the hand contacts the target control, and real-time collision response and force tactile generation when the hand performs operation. It provides corresponding tactile feedback when the user interacts with the robotic arm 301 equipped with a touch panel 305, and provides corresponding rotation and operation limit force feedback when the user interacts with the robotic arm 302 equipped with a mechanical control panel 306; the motion planning computers PC2 and PC3 respectively realize the state reading and motion control of the robotic arms 301 and 302, among which the motion planning computer PC2 is equipped with a dual-robotic arm collaborative motion algorithm, and takes the interaction data results calculated by the motion prediction computer PC1 and the user's helmet display scene as input to determine the interaction type between the user's hand and the interaction target point in the virtual scene, decide the execution arm that moves to the interaction target point, consider the priority avoidance of the dual robotic arms and the collision between the dual robotic arms and the environment to avoid motion planning of the dual robotic arm force tactile feedback mechanism before the hand contacts the target control, and send the results to the motion planning computer PC3. The motion planning computers 1 and 2 generate dual robotic arm motion instructions. The three computers PC1, PC2, and PC3 are connected by a network cable and communicate through the Ethernet protocol to exchange data and synchronize calculations.

[0024] The motion tracking sensor 2 is connected to the motion prediction computer PC1 to collect hand motion data, including the position and direction data of the palm, and the angle data of the finger joints, etc. as input to the motion prediction computer.

[0025] The dual-arm force tactile feedback mechanism 3 consists of two six-degree-of-freedom robotic arms 301 and 302, each equipped with a touch panel 305 and a mechanical control panel 306 at the end, and each equipped with a slide rail 303 and 304. The flexibility of the control is adjustable according to the virtual scene. The dual-arm force tactile feedback mechanism 3 is placed directly in front of the user, and its workspace intersects with the user's hand movement; the end is equipped with a target control to interact with the user's hand at a certain point in the workspace.

[0026] The robotic arm 301 and the slide rail 303 are connected to the controller C1, and the robotic arm 302 and the slide rail 304 are connected to the controller C2. The two controllers are respectively connected to the motion planning computers PC2 and PC3, receive control instructions from the computers PC2 and PC3, output torque signals to drive the motors of the robotic arms 301, 302 and the slide rails 303, 304, and feed back the motion information to the computers PC2 and PC3.

[0027] like Figure 3As shown, a multi-form force tactile feedback system working method based on dual-manipulator collaboration includes a dual-manipulator collaborative servo motion stage before the hand contacts the target control and a force tactile feedback stage after the contact. The dual-manipulator force tactile feedback mechanism performs corresponding operations in each working stage according to different task types. The specific steps are as follows, where (1) and (2) are the dual-manipulator force tactile feedback mechanism working stage judgment steps, (3), (4), (5) and (7) are dual-manipulator collaborative servo motion stage steps, and (6) and (7) are force tactile feedback stage steps.

[0028] (1) The motion tracking sensor 2 collects the user's hand motion data in real time, sends it to the motion prediction computer PC1, and transfers it to (2);

[0029] (2) The motion prediction computer PC1 calculates the hand posture and performs collision detection based on the positions of the virtual hand and the virtual control in the virtual environment; when the hand is not in contact with the target control, the dual-manipulator force tactile feedback mechanism 3 is in the coordinated servo motion stage and enters (3); when the hand is in contact with the target control, the hand reaches the real interactive target point to operate the target control, and the dual-manipulator force tactile feedback mechanism 3 is in the active force tactile feedback stage and enters (6);

[0030] (3) The motion prediction computer PC1 fits the hand motion curve based on the previous and current hand motion data, and predicts the future motion trajectory of the hand to determine the moment t when the hand interacts with the target control. c , interaction position p c , interaction direction q c The interaction data and interaction type are sent to the motion planning computer PC2; the motion prediction computer PC1 receives the user's single hand motion data at the same time, determines that the current force tactile feedback is provided by a single robotic arm, and transfers to (5); the motion prediction computer PC1 receives the user's two-hand motion data at the same time, determines that the current force tactile feedback is provided by two robotic arms, and transfers to (4);

[0031] (4) The motion planning computer PC2 decides to move to the corresponding interactive target point p c1 、p c2 The execution arm, motion planning computer PC3 comprehensively processes the interaction target point p of the dual manipulators c1 、p c2 and interaction time t c1 , t c2 The constraints generate relative priorities, which are transferred to (5);

[0032] (5) The motion planning computer PC2 calculates the predicted t c 、p c , and q cThe user's helmet displays a virtual scene to determine the interaction type between the user's hand and the interaction target point in the virtual scene, decide the execution arm to reach the interaction target point, and perform dual-manipulator force tactile feedback system 3 to plan the dual-manipulator coordinated motion taking into account the dual-manipulator priority avoidance and the avoidance of collision between the dual-manipulator and the environment. The data is then transmitted to the motion planning computer PC3. The two motion planning computers generate motion instructions, which are sent to controllers C1 and C2 and transferred to (7);

[0033] (6) When the user's hand reaches the real interactive target point and contacts the target control, the dual-arm force tactile feedback mechanism 3 moves the target control to the target position of the hand operation in the appropriate direction, providing the user with force tactile feedback that matches the vision. The motion prediction computer PC1 combines the virtual scene to perform real-time collision response and force tactile generation, driving the dual-arm force tactile feedback mechanism 3 to output force, providing active force tactile feedback, and then proceeding to (7);

[0034] (7) The controllers C1 and C2 drive the motors to generate torque according to the input control instructions, driving the robotic arms 301 and 301 and the corresponding slide rails 303 and 304 in the dual-arm force tactile feedback mechanism 3 to perform corresponding movements.

Claims

1. A multi-form force tactile feedback system based on dual robotic arm collaboration, characterized in that: The multi-form force tactile feedback system consists of a computer, a motion tracking sensor, a dual-manipulator force tactile feedback mechanism and a controller. The dual-manipulator force tactile feedback mechanism is equipped with two interactive panels: a touch screen and a mechanical operating part.

2. A multi-form force tactile feedback system based on dual robotic arm collaboration as claimed in claim 1, characterized in that: The computer is composed of a motion prediction computer, a motion planning computer 1, and a motion planning computer 2. The three computers are connected by a network cable and communicate through the Ethernet protocol to perform data exchange and synchronous calculation; The motion tracking sensor is connected to the motion prediction computer and collects the user's hand motion data as input to the motion prediction computer, including the position and direction of the palm and the angles of the finger joints; The dual-arm force tactile feedback mechanism, serving as an interactive device, is located directly in front of the user and consists of a robotic arm 1, a robotic arm 2, and two slide rails supporting the robotic arms. The collaborative workspace of the dual robotic arms covers the cockpit operating space. A touch screen panel is installed at the end of robotic arm 1, and a mechanical control panel is installed at the end of robotic arm 2, respectively used to simulate the touch screen and mechanical control components in a real cockpit. The controller receives control instructions from the motion planning computer, and includes controller 1 and controller 2. Controller 1 is connected to motion planning computer 1, and controller 2 is connected to motion planning computer 2. They output torque information to robot arm 1 and robot arm 2 respectively, drive the motor movement of the two robot arms and the slide rail, and feed back the motor movement information to the two motion planning computers.

3. A multi-form force tactile feedback system based on dual robotic arm collaboration as claimed in claim 2, characterized in that: The motion prediction computer uses the user's hand motion data collected by the motion tracking sensor to perform interaction prediction and generate interaction data, the interaction data including the user's hand motion direction, interaction target point, interaction time, and determines the type of interaction between the user's hand and the interaction target point in the virtual scene, the interaction type including touch screen panel interaction and mechanical control panel interaction; The motion planning computer 1 receives the transmission data from the motion prediction computer, including interaction data and interaction type, plans the coordinated movement of the dual robotic arms, and transmits the motion planning results to the motion planning computer 2, so that the motion planning computer 1 and the motion planning computer 2 simultaneously generate dual robotic arm motion instructions, and respectively drive the dual robotic arms to move to the interaction target point without collision through the controller 1 and the controller 2, and use the touch screen panel and the mechanical control panel to interact with the user's hands to provide the user with corresponding tactile feedback.

Images