Flexible robotic arm remote operation human-computer interaction device and method based on mixed reality

Abstract

The present invention discloses a flexible robotic arm remote-operation human-machine interaction device and method based on mixed reality, the remote-operation human-machine interaction device comprises: a mixed reality device and a control module, the mixed reality device is connected to the control module, the control module is in communication connection with the flexible robotic arm, the control module performs SLAM real-time positioning and mapping, and presents the mapping result and the position and posture of the flexible robotic arm in the operation scene to the operator through the mixed reality device, the mixed reality device collects the operator's dynamics, identifies the control mode selected by the operator and the human control instructions issued, the control module receives and parses the instructions, and performs motion control and path-finding and obstacle avoidance calculations on the flexible robotic arm, converts the human control instructions into flexible robotic arm control instructions, thereby performing motion control on the flexible robotic arm. The present invention can improve the real-time, accuracy, safety and user-friendliness of the flexible robotic arm remote-operation human-machine interaction control.

Description

Technical Field

[0001] The present invention relates to the technical field of teleoperated robots, and in particular to a flexible robotic arm teleoperated human-machine interaction device and method based on mixed reality. Background Art

[0002] Nowadays, robotics technology not only needs to meet the needs of the manufacturing industry, but also goes beyond the factory and expands to a wider range of fields including medicine, aerospace, underwater navigation, etc., to assist or even replace humans in completing some repetitive, heavy or dangerous tasks. Among them, in some narrow and complex space operation scenarios, there is also a growing demand for the application of robotics technology, such as earthquake ruins rescue, cabin drilling operations, nuclear power plant pipeline maintenance, satellite solar panel maintenance, endoscopes and other medical auxiliary equipment, dangerous narrow pipe welding, etc. Such narrow and complex spaces often have the following characteristics: compared with ordinary indoor environments, the space size is limited; the obstacle distribution is more complex, the viewing angle is limited, and the internal information is completely unknown; it is relatively isolated and closed or dangerous; the entrance and exit are limited, and it is difficult for humans to enter directly. Therefore, for the robot arm that enters the narrow space for operation, higher flexibility, intelligence, precision and safety are required. At the same time, due to the unknown and complex characteristics of the internal environment, the robot's intelligence and adaptability are limited, and it is difficult to make decisions completely independently. Sometimes human operators are required to assist the robot arm in interactive navigation control. Since the environmental spatial information is completely unknown, the robotic arm first needs to achieve self-positioning after entering and obtain environmental information through sensors mounted on the robotic arm; for the sake of user-friendliness, the collected raw data undergoes unified data processing and is provided to the operator in a relatively intuitive manner to assist in decision-making and control, and further guide the robotic arm to the next movement and operation without the need for complex professional training for the operator.

[0003] A super-redundant flexible manipulator refers to a manipulator with no less than 16 degrees of freedom. Compared with traditional industrial manipulators with limited degrees of freedom, it has better bending characteristics and flexibility. Therefore, super-redundant flexible manipulators are widely used in nuclear, underwater operations, medical surgery, aerospace and other fields. Among them, the rope-driven super-redundant flexible manipulator is uniformly equipped with a motor on the base, and the joints are driven to rotate by adjusting the length of the rope. Therefore, the manipulator body is lighter and can be used in complex and narrow spaces to achieve functions such as flexible obstacle avoidance. But at the same time, the ultra-high degrees of freedom and complex configuration make the manipulator's spatial perception ability weak, and it is difficult for the manipulator to autonomously perceive and judge the environment and accurately position and plan the trajectory of each joint of the manipulator. Therefore, in the control system of many super-redundant flexible manipulators, it is necessary to use remote human-computer interaction to intervene manually.

[0004] Traditional robotic arm teleoperation usually adopts bilateral teleoperation to establish a master-slave control structure. The master end needs to receive feedback information from the slave end, which includes visual feedback, force feedback, etc. in addition to basic information such as position and speed. Traditional visual feedback is usually based on different types of cameras and 2D displays to provide operators with planar visual information. This technology has a history of more than half a century and has been applied in the fields of hazardous material handling, remote surgery, underwater robots, space robots, mobile robots, etc. However, investigations and studies have shown that this type of traditional teleoperation interaction method has many disadvantages, such as limited viewing angle, large error, image frequency degradation, and inconvenient multi-camera attention switching; in addition, traditional teleoperation usually uses computers or joysticks and other devices to issue commands, and the interaction method is not direct.

[0005] Therefore, the academic community has proposed the concept of immersive interaction, which is to allow operators to interact and make decisions in a simulated environment built based on remote real-time conditions. Research experiments have shown that 3D interactive interfaces can provide operators with a stronger sense of immersion than 2D interfaces, effectively helping operators understand the environment and make decisions. Among them, it is generally believed that visual servoing based on virtual reality (VR), augmented reality (AR), mixed reality (MR) combined with head-mounted displays (HMD) has good prospects. Immersive interaction methods can effectively alleviate the problem of spatial perception performance degradation caused by the high complexity of super-redundant flexible robotic arms, while providing operators with an intuitive 3D operating environment.

[0006] In 2020, Andrés Martín-Barrio's team from the Spanish Center for Automated Robotics (CAR, UPM-CSIC) proposed a human-machine interaction system design for a super-redundant robotic arm for inspection operations based on mixed reality. The super-redundant robotic arm enters a space that is inaccessible or dangerous to human operators to replace humans in the operation. The mixed reality device provides the operator with a 3D environment virtual model and real-time robotic arm position and posture. The operator can change the posture of the end of the robotic arm through interactive means such as gestures to achieve master-slave bilateral position-position control. After the operator adjusts the target posture in the virtual space, he needs to press a button to confirm that the control command is sent to the remote robotic arm. However, this solution has the following disadvantages: (1) In this solution, the virtual environment model is pre-built using the Kinect depth camera and Rtabmap mapping method, and then applied to the robotic arm inspection operation. However, in real-world application scenarios, the super-redundant flexible arm operation environment can basically not be observed and mapped in advance, and real-time and adaptability are more required. (2) To prevent collisions caused by improper operation, this solution uses the method of pressing a button to confirm the control command to control the movement of the robot arm. This method has poor real-time performance, is difficult to deal with emergencies, and has redundant operations. (3) This solution can only operate the end position of the robot arm, which has great limitations and requires high accuracy of the operator.

[0007] Da Sun et al. proposed a new teleoperation system based on mixed reality. Interactive agents were designed and fuzzy logic algorithms were applied to reduce the possibility of accidents caused by improper operation of operators. The system set up an RGB-D camera at a fixed position in the experimental environment to transmit and reconstruct point cloud data in real time, and the operator could observe and operate through a mixed reality interface. However, the workspace of the system is simple, the RGB-D camera is mounted at a fixed position in the workspace, the viewing angle is fixed, the reconstructed point cloud model is real-time feedback, and the information of the occluded part is limited; there is a lack of the first-person perspective of the robotic arm, and it cannot cope with emergencies. Therefore, this solution is only suitable for environments with simpler machinery. Summary of the invention

[0008] In order to make up for the shortcomings of the above-mentioned background technology, the present invention proposes a flexible robotic arm remote-operated human-computer interaction device and method based on mixed reality to solve the problems of low real-time, accuracy, safety and user-friendliness of remote-operated human-computer interaction control.

[0009] The technical problem of the present invention is solved by the following technical solutions:

[0010] The present invention discloses a flexible robotic arm remote-operated human-computer interaction device based on mixed reality, the remote-operated human-computer interaction device comprises: a mixed reality device and a control module, the mixed reality device is connected to the control module, the control module is in communication connection with the flexible robotic arm, the control module performs SLAM real-time positioning and mapping, and presents the mapping result and the posture of the flexible robotic arm in the operation scene to the operator through the mixed reality device, the mixed reality device collects the operator's dynamics, identifies the control mode selected by the operator and the human control instructions issued, the control module receives and parses the instructions, and performs motion control and path-finding and obstacle avoidance operations on the flexible robotic arm, and converts the human control instructions into flexible robotic arm control instructions, thereby performing motion control on the flexible robotic arm.

[0011] In some embodiments, the following technical solutions are also included:

[0012] The mixed reality device is a head-mounted device, which is used to synchronously display an interactive virtual environment and a flexible arm model, perform head tracking, gesture recognition and voice recognition on the operator, and identify the control mode selected by the operator and the manual control instructions issued.

[0013] The control module includes an interaction and model control platform and a data processing and hardware control platform, and the interaction and model control platform and the data processing and hardware control platform are communicatively connected; the interaction and model control platform includes an interactive virtual model, a motion control and pathfinding and obstacle avoidance planning algorithm module and an interactive interface, which are used to build an interactive virtual environment and a flexible arm motion model, perform real-time obstacle avoidance and whole-arm posture planning for the flexible robotic arm, and transmit the mapping results and the posture data of the flexible robotic arm in the scene to the mixed reality device; the data processing and hardware control platform includes a data acquisition module, a SLAM algorithm module and a hardware control system, which are used to collect environmental data, SLAM real-time positioning and mapping, and data communication of the flexible robotic arm.

[0014] The data acquisition module includes an RGB-D depth camera, which is arranged at the end of the flexible robotic arm; the control module performs SLAM real-time positioning and mapping based on the data collected by the RGB-D depth camera.

[0015] The data acquisition module also includes a code disk, which is arranged at each joint of the flexible robotic arm and is used to read the joint angle of the flexible robotic arm.

[0016] The interactive interface includes an interactive panel and a target terminal interactive entity, which is used to provide a human-computer interactive interface.

[0017] The flexible mechanical arm adopts a super redundant design.

[0018] The present invention also discloses a flexible robotic arm remote-operated human-computer interaction system based on mixed reality, comprising a flexible robotic arm remote-operated human-computer interaction device based on mixed reality as described in any of the above items and a flexible robotic arm to be remotely operated.

[0019] In some embodiments, the flexible robotic arm is driven by a rope, each joint of the flexible robotic arm has two degrees of freedom, adjacent joints are perpendicular to each other, and the joints are connected to form an overall super-redundant flexible robotic arm.

[0020] The present invention also discloses a flexible robotic arm remote operation human-computer interaction method based on mixed reality, comprising the following steps: S1, collecting environmental data; S2, performing SLAM real-time positioning and mapping when the flexible robotic arm moves in the space; S3, presenting the mapping results and the posture of the flexible robotic arm in the operation scene to the operator through a mixed reality device; S4, receiving the control mode selected by the operator in the mixed reality device according to the real-time reconstructed local map and the state of the flexible robotic arm and the human control instructions issued; S5, parsing the control mode and the human control instructions, performing motion control and path finding and obstacle avoidance operations on the flexible robotic arm, converting the human control instructions into flexible robotic arm control instructions, and controlling the movement of the flexible robotic arm through a hardware control system, thereby realizing automatic obstacle avoidance and whole arm posture planning during the movement of the flexible robotic arm.

[0021] In some embodiments, the method further includes: repeating steps S1 to S5 until the remote operation task is completed.

[0022] In some embodiments, in step S5, performing pathfinding and obstacle avoidance calculation on the flexible robotic arm includes the following steps:

[0023] S5.1. Construct the point of action of the artificial potential field on the flexible robotic arm;

[0024] S5.2, constructing artificial potential field;

[0025] S5.3. Adopt the whole-arm collaborative planning strategy for maximum work of the flexible robotic arm, that is, move in the direction where the sum of the work done by the forces at each point of action in the potential field within a unit planning period is the largest, so as to realize the whole-arm posture planning with multiple degrees of freedom.

[0026] The beneficial effects of the present invention compared with the prior art include:

[0027] The flexible robotic arm remote-operated human-computer interaction device based on mixed reality provided by the present invention performs SLAM real-time positioning and mapping of the space where the flexible robotic arm is located through a control module, and intuitively presents the mapping results and the posture of the flexible robotic arm in the operation scenario to the operator through a mixed reality device. The operator's dynamics are collected through the mixed reality device, and the control mode selected by the operator and the human control instructions issued are identified. The control module converts the human control instructions into flexible robotic arm control instructions, guides the movement of the end or each joint of the flexible robotic arm according to the real-time reconstructed local map and the state of the flexible robotic arm, flexibly adjusts the posture of the flexible robotic arm, guides the movement and operation of the flexible robotic arm, and performs motion control and path-finding and obstacle avoidance calculations during the movement of the flexible robotic arm, realizes automatic obstacle avoidance and whole-arm posture planning, thereby improving the real-time, accuracy, safety and user-friendliness of the flexible robotic arm remote-operated human-computer interaction control. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a schematic diagram of a flexible robotic arm teleoperation human-machine interaction device based on mixed reality in an embodiment of the present invention;

[0029] Figure 2 is a schematic diagram of a flexible mechanical arm in an embodiment of the present invention;

[0030] Figure 3 It is an operation flow chart of a flexible robotic arm teleoperation human-computer interaction device based on mixed reality in an embodiment of the present invention.

[0031] Figure 4 It is a system information flow diagram of the data acquisition module and the SLAM algorithm module in the embodiment of the present invention.

[0032] Figure 5 It is a system information flow diagram of the SLAM algorithm module and the interactive virtual model in the embodiment of the present invention.

[0033] Figure 6 It is a schematic diagram of the flexible robotic arm in an embodiment of the present invention performing an environment scan through an RGB-D camera (in an actual environment).

[0034] Figure 7 It is a schematic diagram of real-time reconstruction of an environment point cloud model when the RGB-D camera in an embodiment of the present invention scans the environment (in a virtual environment).

[0035] Figure 8 It is a schematic diagram of the expected posture of the interactive entity planning terminal of the mobile target terminal during operator interaction in an embodiment of the present invention (in a virtual environment).

[0036] Fig. 9 It is a schematic diagram of the flexible robotic arm of an embodiment of the present invention moving to reach the target posture according to the calculation results of the path finding and obstacle avoidance planning (in a virtual environment).

[0037] Fig.10 It is a schematic diagram of the obstacle avoidance result of the flexible robotic arm in the actual environment in an embodiment of the present invention (in the actual environment).

[0038] Fig.11 Schematic diagram of the improved artificial potential field in an embodiment of the present invention.

[0039] Fig.12 Schematic diagram of an interactive panel in an embodiment of the present invention.

[0040] Fig.13 It is a schematic diagram of the target terminal interactive entity in an embodiment of the present invention.

[0041] Fig.14 Schematic diagram of a target end anti-shake control handle in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention is further described below with reference to the accompanying drawings and in combination with preferred embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0043] It should be noted that the directional terms such as left, right, up, down, top, and bottom in this embodiment are merely relative concepts or are based on the normal use status of the product, and should not be considered as restrictive.

[0044] The embodiment of the present invention provides a flexible robotic arm remote-operation human-computer interaction device based on mixed reality. The remote-operation human-computer interaction device includes: a mixed reality device and a control module. The mixed reality device is connected to the control module. The control module is communicatively connected with the flexible robotic arm. The control module performs SLAM (Simultaneous Localization and Mapping) to locate each joint of the flexible robotic arm while mapping the environment. The mapping result and the posture of the flexible robotic arm in the operation scene are presented to the operator through the mixed reality device. The mixed reality device collects the operator's dynamics, identifies the control mode selected by the operator and the human control instructions issued. The control module receives and parses the instructions, performs motion control and path-finding and obstacle avoidance operations on the flexible robotic arm, and converts the human control instructions into flexible robotic arm control instructions, thereby performing motion control on the flexible robotic arm.

[0045] Specifically, the mixed reality device (mixed reality interface) is a head-mounted device, which is used to synchronously display the interactive virtual environment and the flexible arm model, perform head tracking, gesture recognition and voice recognition of the operator, and identify the control mode selected by the operator and the manual control instructions issued.

[0046] Specifically, the control module includes an interaction and model control platform and a data processing and hardware control platform, and the interaction and model control platform and the data processing and hardware control platform are communicatively connected; the interaction and model control platform includes an interactive virtual model, a motion control and pathfinding and obstacle avoidance planning algorithm module and an interactive interface, which are used to build an interactive virtual environment and a flexible arm motion model, perform real-time obstacle avoidance and whole-arm posture planning for the flexible robotic arm, and transmit the mapping results and the posture data of the flexible robotic arm in the scene to the mixed reality device; the data processing and hardware control platform includes a data acquisition module, a SLAM algorithm module and a hardware control system, which are used to collect environmental data, SLAM real-time positioning and mapping, and data communication of the flexible robotic arm.

[0047] The data acquisition module includes an RGB-D depth camera, which is set at the end of the flexible robotic arm. The control module performs SLAM real-time positioning and mapping based on the data collected by the RGB-D depth camera.

[0048] Furthermore, the data acquisition module also includes a code disk, which is arranged at each joint of the flexible robotic arm and is used to read the joint angles of the flexible robotic arm.

[0049] Among them, the interactive interface includes three components: an interactive panel, an interactive entity at the target end, and an anti-shake control handle at the target end, which is used to provide a human-computer interaction interface.

[0050] Among them, after the SLAM algorithm module calculates the results in real time, the interactive and model control platform subscribes to the camera trajectory and point cloud data of the data processing and hardware control platform, including the series of poses obtained by the SLAM algorithm module for camera positioning at each sampling point; dense point cloud data of the local map is obtained for SLAM mapping in the current stage, including RGB data and depth data. After data type conversion and coordinate system conversion, the above two types of data are processed into Unity coordinates. The data is used for the subsequent calibration of the flexible robotic arm joint positioning and autonomous obstacle avoidance links; the data is drawn into a color point cloud to form a virtual environmental obstacle model.

[0051] In some embodiments, the flexible robotic arm is based on a rope drive and is a super redundant design.

[0052] In another embodiment, the present invention provides a flexible robotic arm remote-operated human-computer interaction system based on mixed reality, including the flexible robotic arm remote-operated human-computer interaction device based on mixed reality as described above and a flexible robotic arm to be remotely operated.

[0053] Specifically, the flexible robotic arm is driven by a rope, each joint of the flexible robotic arm has two degrees of freedom, adjacent joints are perpendicular to each other, and the joints are connected to form an overall super-redundant flexible robotic arm.

[0054] In another embodiment, the present invention provides a flexible robotic arm teleoperation human-computer interaction method based on mixed reality, comprising the following steps:

[0055] S1. Collect environmental data.

[0056] S2, the flexible robotic arm performs SLAM real-time positioning and mapping when moving in the space. Among them, the SLAM algorithm module is based on the RTabmap method.

[0057] S3. Present the mapping results and the position and posture of the flexible robotic arm in the operation scene to the operator through a mixed reality device.

[0058] S4. Receive the control mode and the manual control instructions selected by the operator in the mixed reality device according to the real-time reconstructed local map and the state of the flexible robotic arm.

[0059] S5. Analyze the control mode and human control instructions, perform motion control and path-finding and obstacle avoidance operations on the flexible robotic arm, convert the human control instructions into flexible robotic arm control instructions, and control the movement of the flexible robotic arm through the hardware control system, thereby realizing automatic obstacle avoidance and whole-arm posture planning during the movement of the flexible robotic arm.

[0060] Among them, the path finding and obstacle avoidance calculation for the flexible robotic arm includes the following steps:

[0061] S5.1. Construct the point of action of the artificial potential field on the flexible robotic arm;

[0062] S5.2, constructing artificial potential field;

[0063] S5.3. Adopt the whole-arm collaborative planning strategy for maximum work of the flexible robotic arm, that is, move in the direction where the sum of the work done by the forces at each point of action in the potential field within a unit planning period is the largest, so as to realize the whole-arm posture planning with multiple degrees of freedom.

[0064] In some embodiments, the method further includes: repeating steps S1 to S5 until the remote operation task is completed.

[0065] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0066] like Figure 1 As shown, the flexible robotic arm teleoperation human-machine interaction device based on mixed reality in this embodiment includes a control module (computer PC end) and a mixed reality device, and the object of interactive control of the teleoperation human-machine interaction device is a rope-driven super-redundant flexible robotic arm.

[0067] The flexible robotic arm 1 is located in a remote and complex environment for operation, as shown in the schematic diagram. Figure 2As shown, the flexible robotic arm 1 has 16 joints and 32 degrees of freedom. 12 driving motors drive 12 ropes, and every 3 ropes drive an arm segment. Each arm segment contains 4 joints and arm rods. Each joint has two orthogonal axes, forming a universal joint, with pitch-yaw dual degrees of freedom; the overall model is composed of a series of universal joints connected in sequence. Therefore, each arm segment has 2 equivalent degrees of freedom, and the overall model has 8 equivalent degrees of freedom. This structure enables the model to have flexible three-dimensional spatial motion capabilities. An RGB-D depth camera 2 of model Intel Realsense D435 is installed at the end of the flexible robotic arm 1 for collecting environmental data. The schematic diagram of the flexible robotic arm 1 scanning the environmental obstacles 3 through the RGB-D camera 2 is shown in the figure. Figure 6 As shown in the figure, encoders are installed at each joint of the flexible robotic arm to read the joint angles.

[0068] The mixed reality device (mixed reality interface) may be a head-mounted device (HoloLens), which is used to synchronously display the interactive virtual environment and the flexible arm model, perform head tracking, gesture recognition, voice recognition, etc. of the operator, and recognize the control mode selected by the operator and the human control instructions issued. HoloLens is a mixed reality head-mounted display developed by Microsoft Corporation (Microsoft), and this embodiment uses Microsoft HoloLens II. HoloLens can scan and obtain environmental information in real time, capture operator displacement, perform hand and eye tracking, and provide voice command services, and the operation is sensitive and convenient.

[0069] The computer PC includes an interactive and model control platform (Unity 3D) and a data processing and hardware control platform (ROS robot operating system). The data processing and hardware control platform includes a data acquisition module, a SLAM algorithm module and a hardware control system. The data acquisition module includes an RGB-D depth camera and a code disk. The computer PC performs SLAM real-time positioning and mapping based on the data collected by the RGB-D depth camera. In Unity 3D, it includes two parts: an interactive virtual model and a motion control and pathfinding obstacle avoidance planning algorithm module. Using a three-dimensional modeling engine, a flexible robotic arm motion model is built, the environmental point cloud map is reconstructed, and an interface for human-computer interaction is provided. Using C# scripts, motion control and pathfinding obstacle avoidance planning algorithm modules are written to achieve real-time obstacle avoidance and whole-arm posture planning of the flexible robotic arm. The schematic diagram of the real-time reconstruction of the environmental point cloud model in the virtual environment when the RGB-D depth camera scans the environment is shown in the figure below. Figure 7 As shown, it specifically includes: a flexible robotic arm virtual model 4, an RGB-D depth camera virtual model 5 and an environmental point cloud model 6; a schematic diagram of the desired pose of the mobile target end interactive entity planning end when the operator interacts is shown in Figure 8As shown, it includes a virtual model 7 for tracking operator gestures; a schematic diagram of the flexible robotic arm moving to the target position according to the calculation results of the path finding and obstacle avoidance planning is shown in Fig. 9 The schematic diagram of the obstacle avoidance result of the flexible robotic arm in the actual environment is shown in Fig.10 shown.

[0070] In addition, Unity 3D has an interactive interface with HoloLens, which is convenient for data transmission and imaging. The ROS robot operating system realizes communication between multiple nodes of the system, connects nodes such as RGB-D depth camera, SLAM algorithm module, flexible arm hardware control system in series, and then communicates with the Unity 3D model through WebSocket to maintain the efficient and normal operation of the remote-controlled human-computer interaction device. Among them, the SLAM algorithm module is based on the RTabmap method. The construction of the system information flow (i.e., the information transmission path) in the embodiment of the present invention is as follows:

[0071] (1) The data acquisition module and the SLAM algorithm module exchange information. This information exchange is established within the ROS robot operating system. In the ROS robot operating system, the RGB-D depth camera acquisition node and the SLAM algorithm node are run successively. When the SLAM node is running, the interactive interface with the RGB-D depth camera data is automatically opened to build the information flow between the two nodes. The system information flow diagram of the data acquisition module and the SLAM algorithm module is shown in the figure. Figure 4 shown.

[0072] (2) The SLAM algorithm module exchanges information with the interactive virtual model. This information interaction is established between the ROS robot operating system end and the Unity 3D end. The embodiment of the present invention uses WebSocket (a protocol for full-duplex communication on a single TCP connection) for communication. The Server end of WebSocket is constructed in the SLAM algorithm module, and the Client end is constructed in Unity 3D. The server end provides the client with two kinds of message data in the SLAM node, / rtabmap / mapPath camera path and / rtabmap / cloud_map point cloud data, to complete the one-way information transmission. The system information flow diagram of the SLAM algorithm module and the interactive virtual model is shown in the figure. Figure 5 shown.

[0073] (3) The interactive virtual model interacts with the motion control and path-finding and obstacle avoidance planning algorithm modules. This information interaction is built inside the Unity 3D end and is simply implemented through scripts.

[0074] (4) HoloLens interacts with the interactive virtual model. This information interaction is established between the PC-side Unity 3D and the HoloLens device. Unity 3D provides MRTK (Mixed Reality Toolkit) for interacting with Microsoft HoloLens. By connecting HoloLens and the PC to the same local area network, information can be transmitted through the TCP / IP protocol.

[0075] The interactive interface of this embodiment includes three components: an interactive panel, an interactive entity at the target end, and an anti-shake control handle at the target end.

[0076] (1) Interactive Panel

[0077] The overall interactive panel is as follows Fig.12 As shown, it contains the main panel and the mode sub-panel.

[0078] The main panel contains mode options and a Reset button. The mode options provide the operator with function selection, and the Reset button provides the operator with the arm reset function. After recognizing that a specific mode is selected, the corresponding mode sub-panel pops up, containing the control premise and variables in the mode, etc., providing the operator with a larger operating space.

[0079] In the Mode sub-panel, there are Anti-ShakeHandle and BaseLock options, which represent the anti-shake control handle mode and end lock mode respectively.

[0080] (2) Target terminal interactive entity

[0081] like Fig.13 As shown in the figure, the collision volume and interactive properties are set in Unity 3D. After the operator wears HoloLens glasses, he can change its posture by grabbing.

[0082] When the target point is far away from the flexible manipulator, the weight of the gravitational field on the posture planning is reduced; it is easily affected by factors such as the movement speed of the flexible manipulator itself and the step length of the path-finding and obstacle-avoidance planning, which can easily introduce greater control errors. Therefore, in Unity 3D, the upper limit of the movement speed of the interactive entity at the end of the target and the upper limit of the distance from the end of the manipulator are set to limit the operator to cooperate with the flexible manipulator to guide in small increments multiple times.

[0083] (3) Target end anti-shake control handle

[0084] Since this embodiment does not provide any tactile feedback, the operator only interacts with the visually visible virtual model. Hand shaking will have a greater impact than when tactile feedback is present, causing unnecessary offsets and errors in the posture planning of the flexible robotic arm, especially in narrow situations where precise control of the end position is required. Switching to the target end anti-shake control handle can greatly reduce the impact of hand shaking.

[0085] like Fig.14 As shown in the figure, the target end anti-shake control handle is composed of three mutually perpendicular handles extending from the x, y, and z axes of the target coordinate system with unit length. When entering the anti-shake mode, the interactive entity at the target end loses the interactive attribute, and the three handles are given interactive attributes. When the operator drags one of the handles, the target end anti-shake control handle will move in a straight line along the axis as a whole, eliminating the deviation in other directions caused by the operator's hand shaking. In addition, under this anti-shake control, the end movement speed is limited by the upper limit of the movement speed ν and the upper limit of the distance s from the end of the robotic arm, just like the interactive entity at the target end, and the operator is also required to guide the flexible robotic arm in small increments and multiple times.

[0086] The operation process of the flexible robotic arm remote-operated human-computer interaction device based on mixed reality in this embodiment is as follows: Figure 3 As shown. In each operation cycle, the flexible robotic arm collects environmental data through the RGB-D depth camera, and sends the joint angle data read by the encoder to the SLAM algorithm module in real time; the SLAM algorithm module runs the SLAM algorithm for positioning and mapping, and synchronously displays the environment and the flexible robotic arm model in HoloLens; the operator generates manual control instructions through HoloLens according to the current mapping and robotic arm posture, and the computer PC receives the instructions and performs motion control and path-finding and obstacle avoidance calculations on the robotic arm through the motion control and path-finding and obstacle avoidance planning algorithm module. The hardware control system converts the manual control instructions into robotic arm control instructions to control the movement of the robotic arm; the movement of the robotic arm brings about the update of the environmental data by the sensor, so that the system forms a closed loop.

[0087] The operation process of the flexible robotic arm remote-operated human-machine interaction device of this embodiment includes:

[0088] (1) The task begins;

[0089] (2) Turn on HoloLens and your computer.

[0090] (3) Start the RGB-D depth camera node, RTabmap node, and robotic arm control node in ROS, and start the Unity 3D operating environment in the computer;

[0091] (4) Starting the flexible robotic arm and completing the posture calibration between the flexible robotic arm and the virtual robotic arm model;

[0092] (5) The flexible robotic arm collects environmental data, feeds it back to the computer PC, performs SLAM mapping, and transmits it to the HoloLens for image presentation;

[0093] (6) Collecting the control mode and gesture and voice data selected by the operator in the mixed reality interface, generating control instructions, and sending them to the flexible robotic arm after passing through the motion control and path-finding and obstacle avoidance planning algorithm module to control the movement of the flexible robotic arm;

[0094] (7) Repeat the above steps (5) to (6);

[0095] (8) Until the teleoperation mission is completed.

[0096] In step (5), after the SLAM algorithm module calculates the results in real time, Unity 3D subscribes to the camera trajectory and point cloud data in ROS through WebSocket, including the series of poses obtained by the SLAM algorithm module for camera positioning at each sampling point; dense point cloud data of the local map is obtained for SLAM mapping in the current stage, including RGB data and depth data. After data type conversion and coordinate system conversion, the above two types of data are processed into Unity coordinates. The data is used for the subsequent calibration of the flexible manipulator joint positioning and autonomous obstacle avoidance; the data is drawn into a colored point cloud in UnityEditor through Unity 3D's Mesh tool to form a virtual environmental obstacle model.

[0097] In this embodiment, based on the improved artificial potential field obstacle avoidance algorithm based on the virtual guiding pipeline, due to the short single planning cycle, according to the actual application scenario, the guiding potential field of the artificial guiding pipeline is removed, the distribution of the guiding potential field is improved, the amount of calculation is reduced, and the real-time performance and flexibility of the system are greatly improved. In step (6), this embodiment considers that the flexible manipulator is in two types of artificial potential fields: the repulsive potential field away from obstacles and the repulsive potential field away from obstacles. rep and the gravitational potential field pointing to the target position att The schematic diagram is as follows Fig.11 The method for performing path finding and obstacle avoidance calculation on the flexible robotic arm is as follows:

[0098] (1) Constructing the point of action of the artificial potential field on the flexible robotic arm

[0099] Repulsive potential field rep Acting on the entire flexible robotic arm, it prevents any part of the flexible robotic arm from colliding with obstacles. However, due to the short single planning cycle, setting too many sampling points will lead to redundant calculations. Therefore, a repulsive potential field is set. rep The action points are the joint points P of the flexible manipulator. i (i∈(1,2,...,n)).

[0100] Gravitational potential field att Acting on the end of the flexible robotic arm, it guides the flexible robotic arm to reach the correct target posture. First, in order to make the end reach the correct position, the target point P Goal The gravitational field generated acts on the end joint point P End In addition, in order to make the end reach the correct posture, two virtual target points are constructed at the distance L between the x-axis and the y-axis of the target coordinate system. Similarly, two virtual action points are constructed at the distance L between the x-axis and the y-axis of the end coordinate system. but The gravitational field generated acts on The gravitational field generated acts on

[0101] The point set consisting of the points where the two artificial potential fields act on the flexible robotic arm is denoted as Q.

[0102] (2) Constructing an artificial potential field

[0103] Repulsive potential field att and gravitational potential field rep The forces generated on the corresponding points of action are expressed as F att 、F rep Expressed as follows, the expression of the improved artificial potential field on the point of action force is as follows:

[0104]

[0105]

[0106] Among them, k att ,k rep ,d1,d2,ρ>0 are artificially set fixed parameters, P Goal is the target point position vector, P is the action point position vector, and P0 is the point position vector closest to the joint point in the reconstructed point cloud.

[0107] (3) Flexible manipulator maximum power whole arm collaborative planning strategy

[0108] In order to realize the whole-arm posture planning with multiple degrees of freedom, the embodiment of the present invention adopts the maximum-work whole-arm collaborative planning strategy, that is, moving in the direction where the sum of the force work done by each action point in the potential field within a unit planning period is the largest.

[0109] For the whole arm posture planning problem of the flexible manipulator in the embodiment of the present invention, the state variables include the joint angles θ i and Base displacement (d x ,d y ,d z), the base rotation Euler angle (α, β, γ), expressed as m = (2*n+6) dimensional state variables n represents the number of joints of the flexible robotic arm, and m represents the dimension of the state variable x.

[0110] The specific principles are as follows:

[0111] For a single action point P i ∈Q, the force in the unit planning period is analyzed as follows:

[0112]

[0113] Then in this planning cycle, the action point P i The work done is:

[0114] w i =F i *dP i ,

[0115] Where dPi represents the differential of Pi;

[0116] Then the total work of each action point in the unit planning cycle is:

[0117]

[0118] Where T is the matrix transpose symbol;

[0119] Total work on state variables Taking partial derivatives we get:

[0120]

[0121] in, That is, the point of action P i The Jacobian matrix at can be expressed as J i The total work of the action point in the unit planning cycle is:

[0122]

[0123] Where Δx represents the change of the state variable x within a unit period.

[0124] Since planning is strictly limited to a relatively short planning cycle, the embodiment of the present invention needs to strictly limit the step size to prevent problems such as collision or uncontrollability caused by large errors.

[0125] In summary, the problem can be summarized as a convex optimization problem, expressed as follows:

[0126]

[0127] Among them, a represents the upper limit of the displacement distance of any action point within the unit planning period, b i represents the upper limit of the single-step length, which is positively correlated with the shortest distance to the obstacle point cloud and the distance to the target point; c i Represents the state variable x i The upper limit is only related to the robot model. In addition, the existence of Other Constraints can enable it to achieve complex functions, such as locking the posture of the end of the flexible robot arm, locking the posture of the base, etc.

[0128] In some embodiments, the functions provided by the present invention include: (1) supporting the operator to control the flexible robotic arm through a mixed reality interface, including adjusting the position and posture of the end of the flexible robotic arm and each joint; (2) supporting full-process autonomous obstacle avoidance of a complete guidance trajectory manually planned in a known obstacle environment, and the flexible robotic arm moves along the planned trajectory and achieves autonomous obstacle avoidance; (3) supporting full-process autonomous obstacle avoidance during real-time single-point operation, that is, when the operator controls the single-point displacement of the robotic arm, the other parts of the robotic arm can also achieve autonomous obstacle avoidance to ensure safety; (4) supporting real-time updating of the obstacle distribution around the flexible robotic arm and the position and posture of each joint of the flexible robotic arm by collecting environmental data and performing SLAM mapping, and presenting them to the operator in real time through the mixed reality interface. The operator observes the real-time scene in the mixed reality interface and issues manual control instructions to achieve remote operation; (5) designing a suitable interactive agent to support more refined human-computer interactive control in specific scenarios.

[0129] The embodiment of the present invention adopts mixed reality (MR) technology to realize human-machine interactive navigation control of a super-redundant flexible manipulator in a narrow and complex environment. Environmental data is obtained by a depth camera mounted at the end of the flexible manipulator, and 3D real-time positioning and mapping (Simultaneous Location and Mapping, SLAM) is performed on the space where the flexible manipulator is located, and the mapping results and the position and posture of the manipulator in the operation scene are intuitively presented to the remote operator through a mixed reality interface (MR Interface); the operator wears an HMD, is in a third-person perspective, selects different control modes, guides the movement of the end of the manipulator or each joint according to the real-time reconstructed local map and the state of the manipulator, flexibly adjusts the position and posture of the manipulator, guides the movement and operation of the manipulator, and realizes automatic obstacle avoidance and whole arm position and posture planning during the movement of the manipulator. In this process, the operator perceives the flexible manipulator and its environment through immersive vision and gives feedback and control, so that the human-machine interactive control forms a closed loop, and the operator can interactively navigate the flexible manipulator in a complex space. The embodiment of the present invention can improve the real-time, accuracy, safety and user-friendliness of the remote human-machine interactive control.

[0130] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A flexible robotic arm remote control human-computer interaction method based on mixed reality, using a flexible robotic arm remote control human-computer interaction device based on mixed reality, characterized in that: The remote-operated human-machine interaction device comprises: a mixed reality device and a control module, wherein the mixed reality device is connected to the control module, and the control module is in communication connection with the flexible robotic arm, wherein the control module performs SLAM real-time positioning and mapping, and presents the mapping result and the position and posture of the flexible robotic arm in the operation scene to the operator through the mixed reality device, wherein the mixed reality device collects the operator's dynamics, identifies the control mode selected by the operator and the human control instructions issued, wherein the control module receives and parses the instructions, and performs motion control and path-finding and obstacle avoidance operations on the flexible robotic arm, and converts the human control instructions into flexible robotic arm control instructions, thereby performing motion control on the flexible robotic arm; The method comprises the following steps: S1. Collect environmental data; S2, the flexible robotic arm performs SLAM real-time positioning and mapping when moving in the space; S3, presenting the mapping results and the position and posture of the flexible robotic arm in the operation scene to the operator through a mixed reality device; S4, receiving the control mode selected and the human control command issued by the operator in the mixed reality device according to the real-time reconstructed local map and the state of the flexible robotic arm; S5. Analyze the control mode and human control instructions, perform motion control and path-finding and obstacle avoidance calculations on the flexible robotic arm, convert the human control instructions into flexible robotic arm control instructions, and control the movement of the flexible robotic arm through the hardware control system, thereby realizing automatic obstacle avoidance and whole-arm posture planning during the movement of the flexible robotic arm; Repeat steps S1 to S5 until the teleoperation task is completed; In step S5, performing pathfinding and obstacle avoidance calculation on the flexible robotic arm includes the following steps: S5.

1. Construct the point of action of the artificial potential field on the flexible robotic arm; S5.2, constructing artificial potential field; S5.

3. Adopt the whole-arm collaborative planning strategy of the flexible manipulator with maximum work, that is, move in the direction where the sum of the work done by the forces at each action point in the potential field within the unit planning cycle is the largest, so as to realize the whole-arm posture planning with multiple degrees of freedom; Adopt the maximum work whole arm coordinated planning strategy, that is, move in the direction where the sum of the force work done by each action point in the potential field within the unit planning period is the largest; For the whole arm posture planning problem of the flexible manipulator, the state variables include the joint angles, base displacement, and base rotation Euler angles, expressed as m = (2*n+6) dimensional state variables n represents the number of joints of the flexible manipulator, and m represents the dimension of the state variable x; The flexible robotic arm is in two types of artificial potential fields: a repulsive potential field away from obstacles and a rep and the gravitational potential field pointing to the target position att , where the repulsive potential field att and gravitational potential field rep The forces generated on the corresponding points of action are expressed as F att 、F rep The expression of the improved artificial potential field on the point of action force is as follows: Among them, k att ,k rep ,d1,d2,ρ>0 are artificially set fixed parameters, P Goal is the target point position vector, P is the action point position vector, and P0 is the point position vector closest to the joint point in the reconstructed point cloud; For a single action point P i ∈Q, the force F in the unit planning period is obtained by analysis i for: F atti represents the gravitational potential force at the i-th point of action, F repi represents the repulsive potential field force at the i-th point of application; During the planning period, the action point P i The work done is: w i =F i *dP i , Where dPi represents the differential of Pi; The total work of each action point in the unit planning cycle is: Where T is the matrix transpose symbol; Total work on m-dimensional state variables Taking partial derivatives we get: in, That is, the point of action P i The Jacobian matrix at , denoted by J i ; Total work of the action point within the unit planning cycle: Among them, Δx represents the change of state variable x in a unit period; The problem is summarized as a convex optimization problem, which can be expressed as follows: Among them, a represents the upper limit of the displacement distance of any action point within the unit planning period, b i represents the upper limit of the single-step step length, which is positively correlated with the shortest distance to the obstacle point cloud and the distance to the target point; c i Represents the state variable x i The upper limit is only related to the robot model; OtherConstraints are other constraints that enable it to achieve complex functions.

2. The mixed reality-based flexible robotic arm teleoperation human-computer interaction method according to claim 1, characterized in that: The mixed reality device is a head-mounted device, which is used to synchronously display an interactive virtual environment and a flexible arm model, perform head tracking, gesture recognition and voice recognition on an operator, and identify a control mode selected by the operator and a manual control command issued.

3. The mixed reality-based flexible robotic arm teleoperation human-computer interaction method according to claim 1, characterized in that: The control module includes an interaction and model control platform and a data processing and hardware control platform, and the interaction and model control platform and the data processing and hardware control platform are communicatively connected; the interaction and model control platform includes an interactive virtual model, a motion control and pathfinding and obstacle avoidance planning algorithm module and an interactive interface, which are used to build an interactive virtual environment and a flexible arm motion model, perform real-time obstacle avoidance and whole-arm posture planning for the flexible robotic arm, and transmit the mapping results and the posture data of the flexible robotic arm in the scene to the mixed reality device; the data processing and hardware control platform includes a data acquisition module, a SLAM algorithm module and a hardware control system, which are used to collect environmental data, SLAM real-time positioning and mapping, and data communication of the flexible robotic arm.

4. The mixed reality-based flexible robotic arm teleoperation human-computer interaction method according to claim 3, characterized in that: The data acquisition module includes an RGB-D depth camera, which is arranged at the end of the flexible robotic arm; the control module performs SLAM real-time positioning and mapping based on the data collected by the RGB-D depth camera.

5. The mixed reality-based flexible robotic arm teleoperation human-computer interaction method according to claim 4, characterized in that: The data acquisition module also includes a code disk, which is arranged at each joint of the flexible robotic arm and is used to read the joint angle of the flexible robotic arm.

6. The mixed reality-based flexible robotic arm teleoperation human-computer interaction method according to claim 3, characterized in that: The interactive interface includes an interactive panel and a target terminal interactive entity, which is used to provide a human-computer interactive interface.

7. The mixed reality-based flexible robotic arm teleoperation human-computer interaction method according to any one of claims 1 to 6, characterized in that: The flexible mechanical arm adopts a super redundant design.

8. The mixed reality-based flexible robotic arm teleoperation human-computer interaction method according to claim 7, characterized in that: The flexible robotic arm is driven by ropes, each joint of the flexible robotic arm has two degrees of freedom, adjacent joints are perpendicular to each other, and the joints are connected to form an overall super-redundant flexible robotic arm.

Images