Mechanical arm teleoperation system and method with spatial position alignment characteristic

Abstract

The invention discloses a mechanical arm teleoperation system and method with a spatial position alignment characteristic, which adopts a master-slave mechanical arm system which is completely isomorphic and a visual perception system which is completely spatially aligned, can provide optimal immediacy feeling experience for operation, can truly perceive the environment of a slave end and the condition of an operated object, and can improve the operation efficiency. An operator has an immersive feeling, the spatial position of an operated object completely coincides with the real position of the tail end of the human hand, operation intuition is better met, the operation efficiency is improved, the problem of image space shielding caused by three-dimensional reconstruction is solved by adopting a real six-degree-of-freedom camera holder, and the operation efficiency is improved. And the influence of system delay on an operator is reduced through model selection and a control strategy.

Description

Technical Field

[0001] The present invention relates to the field of radioactive material processing, and in particular to a robotic arm remote control system and method with spatial position alignment characteristics. Background Art

[0002] In industrial applications, due to the presence of substances in the environment that are harmful to the human body, robotic arms are often used instead of humans to perform operations. For unknown and complex usage scenarios, remote operation can be used to efficiently control the robot.

[0003] During remote operation, VR and other methods are usually used to enhance the sense of presence. The images displayed in VR come from sensors such as cameras and 3D cameras installed on the remote robot. Using cameras, 3D cameras, laser radars and other devices can achieve three-dimensional reconstruction of the remote scene and map it to a virtual 3D scene, allowing the operator to complete the operation task more immersively.

[0004] In the existing solutions, the three-dimensional space reconstruction achieved by sensors such as 3D cameras and lidar has obvious bottlenecks. Due to the limitation of the installation position of the sensors, there is a serious spatial occlusion problem, which leads to a large number of empty areas in the 3D point cloud. When changing the viewing angle for observation, it is impossible to provide effective information to complete the task.

[0005] In summary, a robotic arm teleoperation system and method with spatial position alignment characteristics is needed to address the deficiencies in the prior art. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a robotic arm remote control system and method with spatial position alignment characteristics, aiming to solve the above problems.

[0007] To achieve the above-mentioned object, the present invention provides the following technical solutions: a robotic arm teleoperation system with spatial position alignment characteristics, comprising a visual perception module, a master-slave force feedback robot, a control module, a communication module, an alignment module and a calibration optimization module;

[0008] Visual perception module, including binocular camera and VR helmet. The binocular camera is used to capture remote stereo images, and the VR helmet is used to output its own spatial posture and display remote stereo images.

[0009] The master-slave force feedback robot includes a master end manipulator and a slave end manipulator. The master end manipulator is provided with a space anchor point marker for aligning the master end VR space with the space coordinates of the manipulator. The slave end manipulator is used to perform the actual operation task of the master end manipulator and feed back the environment interaction force on the manipulator to the operator.

[0010] The control module is used to receive the transmission signals from other modules, analyze and process them, and control the operation of other modules;

[0011] Communication module, used for signal communication among other modules;

[0012] An alignment module is used to move the camera to a spatial posture and align the camera view with the operator's view;

[0013] The calibration optimization module is used to calibrate and optimize the systems of the binocular camera and VR helmet.

[0014] Optionally, the control module also includes a visual matching algorithm and an integrated data algorithm, the visual matching algorithm is used to obtain the three-dimensional spatial posture of the marker, and the integrated data algorithm is used to integrate the VR helmet position and IMU data.

[0015] Optionally, the visual matching algorithm is implemented in the following manner:

[0016] Step A1: The camera of the VR helmet captures the spatial anchor point marker on the end handle of the master end robotic arm to obtain a perspective image;

[0017] Step A2: feature extraction, extracting feature information of spatial anchor point markers from the view image;

[0018] Step A3: Using the spatial anchor point marker and its projection position in the image, combined with the camera intrinsic parameter matrix, calculate the three-dimensional spatial pose of the spatial anchor point marker;

[0019] Step A4: By reading the angle position information of the robot arm joints and solving the robot arm kinematic equation, the end position of the robot arm is obtained, and the origin of the VR space coordinate system is adjusted to align with the origin of the robot arm coordinate system;

[0020] Step A5: Real-time tracking and correction. The master and slave end robotic arms use the joint position and force sensors of the robotic arms at both ends to make the master end robotic arm and the slave end robotic arm synchronized.

[0021] Optionally, the calculation method for calculating the three-dimensional space posture in step A3 includes a perspective transformation algorithm and / or a PnP algorithm;

[0022] The perspective transformation algorithm maps a point in a plane from one perspective to the corresponding position in another perspective, and calculates the true position and posture by identifying the positions of its four corner points;

[0023] The PnP algorithm calculates the true position and posture through a three-dimensional point group and its projection on a two-dimensional image, as well as the rotation and translation parameters of the camera.

[0024] Optionally, the perspective transformation algorithm is implemented in the following manner:

[0025] Step B1: Obtain the coordinates of the corner points and detect the coordinates of the spatial anchor corner points on the image;

[0026] Step B2: Establish a corresponding two-dimensional coordinate system based on the actual size of the spatial anchor point in the actual environment;

[0027] Step B3: Solve the transformation matrix and calculate the perspective transformation matrix based on the corner points in the source image and their positions on the target plane;

[0028] Step B4: Apply the transformation matrix to other points in the image to derive the spatial pose of the entire spatial anchor marker.

[0029] Optionally, the PnP algorithm is implemented in the following manner:

[0030] Step C1: Collect known three-dimensional coordinate points and their corresponding two-dimensional projection points on the image;

[0031] Step C2: Use EPnP or UPnP to input the corresponding point pairs of three-dimensional and two-dimensional, output the rotation matrix and translation vector parameters of the camera, and record the posture of the camera relative to the actual environment coordinate system;

[0032] Step C3: Check the reprojection errors of the 2D and 3D point pairs to ensure that the rotation matrix and translation vector parameters of the output camera are accurate.

[0033] A method for remote operation of a manipulator with spatial position alignment characteristics, using a remote operation system of a manipulator with spatial position alignment characteristics, comprises the following steps:

[0034] Step S1: Initialize the system, check the operating system, and ensure that the hardware components and software are correctly installed and connected;

[0035] Step S2: Use the camera on the VR helmet to shoot the spatial anchor point marker on the master end robotic arm, calculate the spatial pose of the spatial anchor point marker relative to the VR helmet, and align the spatial coordinates of the VR helmet space with the spatial coordinates of the master end robotic arm;

[0036] Step S3: The operator presses the button of the master-slave synchronization function through the system human-machine interface. After motion planning, the master-end robot automatically moves to the same posture as the slave-end robot to achieve synchronization. The initial position of the slave-end camera gimbal is synchronized with the current position of the VR helmet.

[0037] Step S4: measuring the interaction force between the slave end robot arm and the environment and the operation force of the master end robot arm, and the control module processes the interaction force and operation force information;

[0038] Step S5: The VR helmet performs position tracking and prediction, and the control module reads the position information of the VR helmet and the data of the built-in IMU sensor in real time, predicts and controls the position of the slave camera gimbal;

[0039] Step S6: The control module updates the position of the camera pan / tilt of the slave end in real time so that the viewing angle completely matches the image.

[0040] Beneficial effects of the present invention:

[0041] 1. In the present invention, a master-slave teleoperation system with force and visual feedback systems is provided, and the space mapping of the master and slave ends is completely aligned. Through a six-degree-of-freedom camera gimbal that can track the operator's head movement, and through a control system, low-latency, smooth and stable master end head position tracking is achieved, providing the operator with the most realistic, three-dimensional, low-latency on-site image perception;

[0042] 2. In the present invention, a completely isomorphic master-slave robot system and a completely spatially aligned visual perception system are used, which can provide the operator with the best telepresence experience and can truly perceive the slave end environment and the situation of the operated object. The operator has an immersive feeling, and the spatial position of the operated object completely coincides with the real position of the end of the human hand, which is more in line with the operation intuition and improves the operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The figure is a schematic diagram of a system structure of the present invention.

[0044] Figure 2 The present invention is a schematic flow chart of a method. DETAILED DESCRIPTION

[0045] In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] like Figure 1 As shown, a robotic arm teleoperation system with spatial position alignment characteristics includes a visual perception module, a master-slave force feedback robot, a control module, a communication module, an alignment module and a calibration optimization module;

[0047] Visual perception module, including binocular camera and VR helmet. The binocular camera is used to capture remote stereo images, and the VR helmet is used to output its own spatial posture and display remote stereo images.

[0048] The active force feedback robot includes a master end robot arm and a slave end robot arm. The master end robot arm is provided with a space anchor point marker for aligning the master end VR space with the space coordinates of the robot arm. The slave end robot arm is used to perform the actual operation task of the master end robot arm and feed back the environment interaction force on the robot arm to the operator.

[0049] The control module is used to receive the transmission signals from other modules, analyze and process them, and control the operation of other modules;

[0050] Communication module, used for signal communication among other modules;

[0051] An alignment module is used to move the camera to a spatial posture and align the camera view with the operator's view;

[0052] The calibration optimization module is used to calibrate and optimize the systems of the binocular camera and VR helmet.

[0053] The control module also includes a visual matching algorithm and an integrated data algorithm. The visual matching algorithm is used to obtain the three-dimensional spatial posture of the marker, and the integrated data algorithm is used to integrate the VR helmet position and IMU data.

[0054] Visual matching algorithm, through the following methods:

[0055] Step A1: The camera of the VR helmet captures the spatial anchor point marker on the end handle of the master end robotic arm to obtain a perspective image;

[0056] Step A2: feature extraction, extracting feature information of spatial anchor point markers from the view image;

[0057] Step A3: Using the spatial anchor point marker and its projection position in the image, combined with the camera intrinsic parameter matrix, calculate the three-dimensional spatial pose of the spatial anchor point marker;

[0058] Step A4: By reading the angle position information of the robot arm joints and solving the robot arm kinematic equation, the end position of the robot arm is obtained, and the origin of the VR space coordinate system is adjusted to align with the origin of the robot arm coordinate system;

[0059] Step A5: Real-time tracking and correction. The master and slave end robotic arms use the joint position and force sensors of the robotic arms at both ends to make the master end robotic arm and the slave end robotic arm synchronized.

[0060] The calculation method for calculating the three-dimensional spatial posture in step A3 includes a perspective transformation algorithm and / or a PnP algorithm;

[0061] The perspective transformation algorithm maps a point in a plane from one perspective to the corresponding position in another perspective, and calculates the true position and posture by identifying the positions of its four corner points;

[0062] The PnP algorithm calculates the true position and posture through a three-dimensional point group and its projection on a two-dimensional image, as well as the rotation and translation parameters of the camera.

[0063] The perspective transformation algorithm is implemented in the following way:

[0064] Step B1: Obtain the coordinates of the corner points and detect the coordinates of the spatial anchor corner points on the image;

[0065] Step B2: Establish a corresponding two-dimensional coordinate system based on the actual size of the spatial anchor point in the actual environment;

[0066] Step B3: Solve the transformation matrix and calculate the perspective transformation matrix based on the corner points in the source image and their positions on the target plane;

[0067] Step B4: Apply the transformation matrix to other points in the image to derive the spatial pose of the entire spatial anchor marker.

[0068] The PnP algorithm is implemented in the following ways:

[0069] Step C1: Collect known three-dimensional coordinate points and their corresponding two-dimensional projection points on the image;

[0070] Step C2: Use EPnP or UPnP to input the corresponding point pairs of three-dimensional and two-dimensional, output the rotation matrix and translation vector parameters of the camera, and record the posture of the camera relative to the actual environment coordinate system;

[0071] Step C3: Check the reprojection errors of the 2D and 3D point pairs to ensure that the rotation matrix and translation vector parameters of the output camera are accurate.

[0072] Visual perception system design:

[0073] The binocular camera is installed on a small 6-axis serial robot arm, which has the ability of three-axis translation and three-axis rotation in the end space, and can move the carried camera to the specified spatial position.

[0074] The operator wears a VR head-mounted display device, which has spatial positioning capabilities and can output its own spatial posture.

[0075] When the system initialization is completed, the coordinates of the master and slave ends need to be aligned.

[0076] A spatial anchor point marker (such as a special color block or a QR code) is set on the end handle of the main hand robotic arm. The marker is photographed by the camera on the VR headset. The three-dimensional spatial position of the marker is obtained through a visual matching algorithm to achieve spatial coordinate alignment between the main VR space and the main robotic arm.

[0077] The master and slave robotic arms adopt the same configuration design in terms of structure design and have the same kinematic forward and inverse solutions, which structurally ensures the spatial alignment of the master and slave motions. In different project application requirements, sometimes the arm span of the slave will far exceed the range of human hand movement (too large / too small). At this time, the spatial dimensions of the master and slave need to be scaled in proportion to the proportion of the hand to ensure the spatial alignment characteristics.

[0078] The master-slave robot can measure the interaction force between the remote robot and the environment and the master-end operating force through force sensors installed on the joints and end tools. The master-slave operation control with force feedback is realized through the controller. The operator can truly feel the force situation on the slave end, and the operator's output can also be truly transmitted to the slave end environment, realizing remote operation control with a high sense of presence.

[0079] When the operator enters the operating station and clicks to confirm the start of the remote operation process through the software, the control system obtains the current operator's VR headset positioning data, calculates the relative posture with the handle, and sends it to the slave end. The camera gimbal (small robotic arm) of the slave end starts to move to the specified posture to align the camera's perspective with the operator's perspective. At this time, the image seen by the operator's two eyes is the image actually captured by the camera, and it is in a state of complete spatial alignment.

[0080] When the operation starts, when the operator moves his head, the control system obtains the VR headset positioning data in real time, calculates the position of the camera gimbal on the slave end in real time and sends it down, achieving complete matching of the observation angle and the image.

[0081] The control system reads the positioning position (60hz) of the VR headset and the built-in IMU sensor data (1000hz) in real time to obtain acceleration and angular velocity information. The moving speed of the headset is calculated by calculating the position difference between the two headset positionings; the position of the headset can be predicted at a high frequency through the IMU data, and then the two are used through Kalman filtering to obtain high-frequency, high-real-time headset position, speed, and acceleration information. The position value is used as the position loop target value, and the speed and acceleration are sent to the camera gimbal as control command feedforward information for motion trajectory planning. The motion trajectory of the camera gimbal is continuous and smooth, and the future trajectory can be predicted to compensate for communication delays and real-time fluctuations.

[0082] The camera uses an SDI interface model. This type of camera data uses an uncompressed format, which can save the encoding and decoding time of the camera video image and minimize image delay. The robot arm uses the ethercat high-speed communication bus and performs position master-slave control at a cycle of 1000Hz, achieving millisecond-level control delay. Ultimately, the overall delay of the entire system is less than 50ms. The operator can hardly perceive the system delay, achieving a maximum sense of presence.

[0083] The distance (baseline) between the binocular cameras is designed to be adjustable. In order to match the pupil distance of different operators, three levels of large, medium and small are designed. It can also be adjusted according to the actual pupil distance of the operator to match the binocular parallax of the operator and reduce distortion.

[0084] The camera image and VR headset are calibrated according to their respective optical systems to improve the image distortion of the optical system and avoid image distortion.

[0085] like Figure 2 As shown, a method for remote operation of a manipulator arm with a spatial position alignment characteristic adopts a remote operation system of a manipulator arm with a spatial position alignment characteristic, comprising the following steps:

[0086] Step S1: Initialize the system, check the operating system, and ensure that the hardware components and software are correctly installed and connected;

[0087] Step S2: The operator operates the master end robot arm, observes the slave end robot arm and makes corrections to synchronize the posture of the master end robot arm with that of the slave end robot arm;

[0088] Step S3: photographing the spatial anchor point marker on the master-end robotic arm through the camera on the VR helmet, calculating the spatial pose of the spatial anchor point marker relative to the VR helmet, and aligning the spatial coordinates of the VR helmet space with the spatial coordinates of the master-end robotic arm;

[0089] Step S4: measuring the interaction force between the slave end robot arm and the environment and the operation force of the master end robot arm, and the control module processes the interaction force and operation force information;

[0090] Step S5: The VR helmet performs position tracking and prediction, and the control module reads the position information of the VR helmet and the data of the built-in IMU sensor in real time, predicts and controls the position of the slave camera gimbal;

[0091] Step S6: The control module updates the position of the camera pan / tilt of the slave end in real time so that the viewing angle completely matches the image.

[0092] The present invention adopts a completely isomorphic master-slave robotic arm system and a completely spatially aligned visual perception system, which can provide the operator with the best telepresence experience and can truly perceive the environment of the slave end and the situation of the operated object.

[0093] The operator has an immersive feeling, and the spatial position of the operated object completely coincides with the real position of the end of the human hand, which is more in line with the operation intuition and improves the operation efficiency.

[0094] A real six-degree-of-freedom camera gimbal is used to solve the image space occlusion problem caused by three-dimensional reconstruction, and the impact of system delay on the operator is reduced through selection and control strategies.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent substitution or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A robotic arm teleoperation system with spatial position alignment characteristics, characterized in that: It includes a visual perception module, a master-slave force feedback robot, a control module, a communication module, an alignment module and a calibration optimization module; Visual perception module, including binocular camera and VR helmet. The binocular camera is used to capture remote stereo images, and the VR helmet is used to output its own spatial posture and display remote stereo images. The master-slave force feedback robot includes a master end manipulator and a slave end manipulator. The master end manipulator is provided with a space anchor point marker for aligning the master end VR space with the space coordinates of the manipulator. The slave end manipulator is used to perform the actual operation task of the master end manipulator and feed back the environment interaction force on the manipulator to the operator. The control module is used to receive the transmission signals from other modules, analyze and process them, and control the operation of other modules; Communication module, used for signal communication among other modules; An alignment module is used to move the camera to a spatial posture and align the camera view with the operator's view; The calibration optimization module is used to calibrate and optimize the systems of the binocular camera and VR helmet.

2. The robotic arm teleoperation system with spatial position alignment characteristics according to claim 1, characterized in that: The control module also includes a visual matching algorithm and an integrated data algorithm. The visual matching algorithm is used to obtain the three-dimensional spatial posture of the marker, and the integrated data algorithm is used to integrate the VR helmet position and IMU data.

3. The robotic arm teleoperation system with spatial position alignment characteristics according to claim 2, characterized in that: The visual matching algorithm is implemented in the following way: Step A1: The camera of the VR helmet captures the spatial anchor point marker on the end handle of the master end robotic arm to obtain a perspective image; Step A2: feature extraction, extracting feature information of spatial anchor point markers from the view image; Step A3: Using the spatial anchor point marker and its projection position in the image, combined with the camera intrinsic parameter matrix, calculate the three-dimensional spatial pose of the spatial anchor point marker; Step A4: By reading the angle position information of the robot arm joints and solving the robot arm kinematic equation, the end position of the robot arm is obtained, and the origin of the VR space coordinate system is adjusted to align with the origin of the robot arm coordinate system; Step A5: Real-time tracking and correction. The master and slave end robotic arms use the joint position and force sensors of the robotic arms at both ends to make the master end robotic arm and the slave end robotic arm synchronized.

4. The robotic arm teleoperation system with spatial position alignment characteristics according to claim 3, characterized in that: The calculation method for calculating the three-dimensional space posture in step A3 includes a perspective transformation algorithm and / or a PnP algorithm; The perspective transformation algorithm maps a point in a plane from one perspective to the corresponding position in another perspective, and calculates the true position and posture by identifying the positions of its four corner points; The PnP algorithm calculates the true position and posture through a three-dimensional point group and its projection on a two-dimensional image, as well as the rotation and translation parameters of the camera.

5. The robotic arm teleoperation system with spatial position alignment characteristics according to claim 4, characterized in that: The perspective transformation algorithm is implemented in the following way: Step B1: Obtain the coordinates of the corner points and detect the coordinates of the spatial anchor corner points on the image; Step B2: Establish a corresponding two-dimensional coordinate system based on the actual size of the spatial anchor point in the actual environment; Step B3: Solve the transformation matrix and calculate the perspective transformation matrix based on the corner points in the source image and their positions on the target plane; Step B4: Apply the transformation matrix to other points in the image to derive the spatial pose of the entire spatial anchor marker.

6. The robotic arm teleoperation system with spatial position alignment characteristics according to claim 4, characterized in that: The PnP algorithm is implemented in the following way: Step C1: Collect known three-dimensional coordinate points and their corresponding two-dimensional projection points on the image; Step C2: Use EPnP or UPnP to input the corresponding point pairs of three-dimensional and two-dimensional, output the rotation matrix and translation vector parameters of the camera, and record the posture of the camera relative to the actual environment coordinate system; Step C3: Check the reprojection errors of the 2D and 3D point pairs to ensure that the rotation matrix and translation vector parameters of the output camera are accurate.

7. A method for remote operation of a robot arm with spatial position alignment characteristics, using the robot arm remote operation system with spatial position alignment characteristics as claimed in any one of claims 1 to 6, characterized in that: The following steps are involved: Step S1: Initialize the system, check the operating system, and ensure that the hardware components and software are correctly installed and connected; Step S2: Use the camera on the VR helmet to shoot the spatial anchor point marker on the master end robotic arm, calculate the spatial pose of the spatial anchor point marker relative to the VR helmet, and align the spatial coordinates of the VR helmet space with the spatial coordinates of the master end robotic arm; Step S3: The operator presses the button of the master-slave synchronization function through the system human-machine interface. After motion planning, the master-end robot automatically moves to the same posture as the slave-end robot to achieve synchronization. The initial position of the slave-end camera gimbal is synchronized with the current position of the VR helmet. Step S4: measuring the interaction force between the slave end robot arm and the environment and the operation force of the master end robot arm, and the control module processes the interaction force and operation force information; Step S5: The VR helmet performs position tracking and prediction, and the control module reads the position information of the VR helmet and the data of the built-in IMU sensor in real time, predicts and controls the position of the slave camera gimbal; Step S6: The control module updates the position of the camera pan / tilt of the slave end in real time so that the viewing angle completely matches the image.

Images