Teleoperation mechanical arm system based on non-electric handheld clamping jaw

Abstract

The invention belongs to the technical field of teleoperation robots, and discloses a teleoperation mechanical arm system based on a non-electric handheld clamping jaw, the teleoperation mechanical arm system comprises the non-electric handheld clamping jaw and a remote mechanical arm which are in communication connection, the non-electric handheld clamping jaw is located on the operator side, and the remote mechanical arm is located in the working environment. The mechanical arm and the end effector connected with the mechanical arm are remotely and accurately controlled through the SLAM algorithm of the non-electric handheld clamping jaw and the finger width positioning algorithm, so that remote operation in the complex or dangerous environment where human beings cannot directly reach is assisted; and meanwhile, the device can be carried on any non-power handheld clamping jaw, so that the problem that long-time operation cannot be performed in a power shortage environment can be solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote-controlled robots, and specifically relates to a remote-controlled mechanical arm system based on a non-powered handheld gripper, which realizes remote and precise control of the mechanical arm and its connected end effector through the SLAM (Simultaneous Localization and Mapping) algorithm of the non-powered handheld gripper and the finger width positioning algorithm, so as to assist in remote operations in complex or dangerous environments that humans cannot directly reach; at the same time, the present invention can be mounted on any non-powered handheld gripper, thus solving the problem of being unable to operate for a long time in a power-deficient environment. Background Art

[0002] With the continuous development of robotics technology, teleoperated robotic arm operation systems have shown great potential in the fields of home services, disaster relief, geological exploration, space exploration, etc. However, existing teleoperation systems mostly rely on complex electrical connections and energy supply, which limits their flexibility and application scenarios.

[0003] In the existing technology, the patent application with publication number CN118238147A proposes a bionic manipulator human-machine interaction system and method based on teleoperation. The method uses head-mounted VR (Virtual Reality) glasses, wearable armbands, force feedback gloves and remote manipulators to communicate and connect, and controls the movement of the remote manipulators through wearable armbands and force feedback gloves. The head-mounted VR glasses provide real-time feedback on the real working environment, achieving consistency between the virtual environment and the real working environment from scene to interaction, and can be used to replace workers in extremely dangerous environments. This method can improve the operator's remote work efficiency and immersion in the working environment, but has obvious disadvantages: 1. It is necessary to wear head-mounted VR glasses, wearable armbands, and force feedback gloves, which increases the weight and discomfort during operation; 2. When wearing head-mounted VR glasses, the operator cannot observe the situation around him, which poses a safety hazard to the operation; 3. Head-mounted VR glasses, wearable armbands, and force feedback gloves all require additional power supply, and cannot work for a long time in an offline state, and the cost is relatively high.

[0004] The patent application with publication number CN118322199A proposes a five-fingered dexterous arm-hand robot teleoperation method based on data gloves, which remotely controls the movement of the five-fingered dexterous arm-hand robot by wearing data gloves. This method can improve the operator's remote work efficiency and operation accuracy, but because high-precision data gloves require additional power supply, they cannot work for a long time in an offline state and are costly.

[0005] The patent application with publication number CN118143961A proposes a B-ultrasound remote operation human-machine shared control system and method based on the attention mechanism. The method uses VR glasses to obtain the operator's attention point position, presents the environment of the attention point position, and controls the remote robot arm to operate under the attention point through the Touch X joystick. This method is mainly used in the field of B-ultrasound and is not replicable for general operations. In addition, the method relies on Touch X for remote operation, requires additional power supply, cannot work for a long time in an offline state, and has a high cost.

[0006] In summary, the existing technologies mostly use AR cameras, data gloves and other devices to remotely control the movement of robotic arms, and their main disadvantages are:

[0007] 1. The equipment cost is high;

[0008] 2. An external power supply is required for the device. It cannot work for a long time in offline state.

[0009] 3. The equipment needs to be worn close to the body, which may cause discomfort to the user; and the equipment needs to work with electricity, which may cause safety hazards such as leakage;

[0010] 4. Most solutions based on VR glasses can easily block the user's real vision, making it impossible to observe the situation around them, so there are certain safety risks. Summary of the invention

[0011] The purpose of the present invention is to provide a remote-controlled robotic arm system based on a non-electric handheld gripper, which does not require an external power supply and can be quickly customized through 3D printing technology. It combines SLAM algorithm with remote communication and other technologies, greatly improving the economy, portability and applicability of the system.

[0012] To achieve the above object, the technical solution adopted by the present invention is:

[0013] A remote-operated robotic arm system based on a non-electric handheld gripper, the remote-operated robotic arm system based on a non-electric handheld gripper comprising a non-electric handheld gripper and a remote robotic arm that establish a communication connection, the non-electric handheld gripper is located at an operator side, the remote robotic arm is located in a working environment, and the non-electric handheld gripper performs the following operations:

[0014] Acquire and display a remote image captured by a remote robotic arm, wherein the remote image is used by an operator to obtain a remote operation progress and operate the non-powered handheld gripper based on the remote image;

[0015] Calculate the position increment of the unpowered hand-held gripper between the latest two adjacent moments;

[0016] The current state of the remote manipulator is obtained, and the moving target point pose of the remote manipulator is obtained according to the pose increment of the non-powered handheld gripper and the current state of the remote manipulator;

[0017] Calculate the clamping width increment of the non-powered hand-held clamp between the latest two adjacent moments;

[0018] The current state of the remote manipulator end gripper is obtained, and the desired target width of the remote manipulator end gripper is obtained according to the gripping width increment of the non-powered hand-held gripper and the current state of the remote manipulator end gripper;

[0019] Sending the moving target point position and the desired target width of the end gripper of the remote robotic arm to the remote robotic arm for the remote robotic arm to perform an action;

[0020] Repeatedly obtain the remote images taken by the remote robotic arm and control the remote robotic arm's movements until the remote operation is completed.

[0021] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution, but are merely further supplements or preferences. Under the premise that there are no technical or logical contradictions, each optional method can be combined with the above-mentioned overall solution separately, and multiple optional methods can also be combined.

[0022] Preferably, the step of calculating the position increment of the non-powered hand-held gripper between two latest adjacent moments includes:

[0023] Estimated time C The position of the hand-held gripper without power at all times T C = {R C ,t C}, where R C It's time C The rotation matrix at time t C It's time C The translation matrix of the moment;

[0024] Get time C-1 The estimated position of the unpowered handheld gripper at the moment is T C-1 = {R C-1 ,t C-1}, where R C-1 It's time C-1 The rotation matrix at time t C-1 It's time C-1 The translation matrix of the moment;

[0025] Calculate the time from the time of the handheld gripper without power C-1 Time to time C The pose increment at time is where dR C-1～C From time C-1 Time to time C The rotation matrix increment at time , and dt C-1～C From time C-1 Time to time C The translation matrix increment at time dt C-1～C =t C -dR C-1～C t C-1 .

[0026] Preferably, the step of obtaining the moving target point pose of the remote manipulator according to the pose increment of the non-powered handheld gripper and the current state of the remote manipulator comprises:

[0027] Convert the six-dimensional pose vector in the current state of the remote manipulator into a 4×4 transformation matrix T robot = {R robot ,t robot}, where R robot is the current rotation matrix of the remote manipulator, t robot is the current translation matrix of the remote manipulator;

[0028] Then the moving target point pose G of the remote manipulator is calculated T =T robot dT C-1～C = {G R ,G t}, where G R is the rotation matrix of the mobile target point of the remote manipulator, and G R =R robot dR C-1～C , G t is the translation matrix of the mobile target point of the remote manipulator, and G t =dR C-1～C t robot +dt C-1～C .

[0029] Preferably, the calculation of the clamping width increment of the non-powered hand-held clamping jaws between two latest adjacent moments includes:

[0030] Get time C The rotation matrix of the first finger on the handheld gripper without power is T l_aruco and the translation matrix is ​​t l_aruco , get time C The rotation matrix of the second finger on the handheld gripper without power is R r_aruco and the translation matrix is ​​t r_aruco ;

[0031] Then calculate time C The clamping width of the handheld gripper without power is Range C =t r_aruco [x]-t l_aruco [x], where t r_aruco [x] is the translation matrix t l_aruco The horizontal x-axis movement distance, t l_aruci [x] is the translation matrix t r_aruco The horizontal x-axis movement distance;

[0032] Get time C-1 The clamping width of the handheld gripper without power is Range C-1 , then the power-free hand-held gripper from time C-1 Time to time C The clamping width increment at the moment is Range C-1～C =Range C -Range C-1 .

[0033] Preferably, the method of obtaining the desired target width of the remote manipulator end gripper according to the gripping width increment of the non-powered handheld gripper and the current state of the remote manipulator end gripper comprises:

[0034] Range expect =Range cur +Range C-1～C

[0035] Among them, Range expect is the desired target width of the remote robot end gripper, Range cur Range is the current width of the remote robot end gripper. C-1～C The latest clamping width increment for non-powered handheld grippers.

[0036] Preferably, the step of sending the moving target point position and the desired target width of the end gripper of the remote manipulator to the remote manipulator comprises:

[0037] If Range expect ≤Range min or Range expect ≥Range max , then discard the latest desired target width Range of the remote robot end gripper expect , only the mobile target point pose is sent to the remote manipulator; otherwise, the mobile target point pose and the desired target width of the remote manipulator end gripper are sent to the remote manipulator, where Range min is the minimum gripping width of the remote robot end gripper, Rangemax The maximum gripping width of the remote robot end gripper.

[0038] Preferably, the non-electric handheld gripper comprises a computing board, a sensor, a display, a handheld base, a button, a finger and a linkage device;

[0039] The computing board is used to provide computing resources;

[0040] The sensor is used to obtain the position and gripping width of the non-powered handheld gripper;

[0041] The display is used to display the remote image captured by the remote robotic arm;

[0042] The handheld base is used as the handheld part of the non-electric handheld clamp;

[0043] The button is used to provide power;

[0044] The linkage device connects the button and the finger, and is used to convert the power provided by the button into controlling the closing / opening of the finger of the non-powered handheld clamp.

[0045] Preferably, the linkage device comprises a first connecting rod, a second connecting rod, a first gear, a second gear, a first pulley, a second pulley and a slide rail, and the finger comprises a first finger and a second finger;

[0046] The two sides of the button are respectively fixed to one end of the first connecting rod and one end of the second connecting rod, the other end of the first connecting rod is connected to the periphery of the first gear, and the other end of the second connecting rod is connected to the periphery of the second gear. The centers of the first gear and the second gear are rotatably mounted on the handheld base through axles, the teeth of the first gear are meshed with the teeth of the first pulley, and the teeth of the second gear are meshed with the teeth of the second pulley. Both the first pulley and the second pulley are buckled with the slide rail, and the first pulley and the second pulley are arranged opposite to each other on the slide rail, the first pulley is fixed to the first finger, and the second pulley is fixed to the second finger.

[0047] The invention provides a remote-controlled mechanical arm system based on a non-electric handheld gripper, which has the following beneficial effects compared with the prior art:

[0048] 1. It can work on any non-powered handheld gripper, and does not require an additional power supply for the handheld gripper. It can also work for a long time in an offline state;

[0049] 2. The design is simple and portable, and can be 3D printed by itself, with extremely low equipment cost;

[0050] 3. Compared with other solutions of wearing close-fitting interactive devices, the present invention relies on a non-powered handheld gripper to remotely operate a remote robotic arm, which does not cause discomfort caused by wearing close-fitting foreign objects, and has a light load;

[0051] 4. The present invention does not require live operation, and there is no leakage. Compared with the solution based on VR glasses, there is no problem of line of sight blocking, so there is no safety hazard in use;

[0052] 5. This method has no requirements for scene restrictions, is highly versatile and has a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is an execution flow chart of a remote-controlled mechanical arm system based on a non-powered handheld gripper according to Embodiment 1 of the present invention;

[0054] Figure 2 This is a schematic diagram of the structure of the non-electric handheld clamp used in Example 2 of the present invention;

[0055] Figure 3 This is a schematic structural diagram of the linkage device of the non-electric handheld clamping jaws used in Example 2 of the present invention;

[0056] Figure 4 This is an execution flow chart of a remote-controlled robotic arm system based on a non-powered handheld gripper according to Example 2 of the present invention.

[0057] In the diagram: 1. LED display; 2. camera; 3. computing board; 4. finger; 401. second finger; 402. first finger; 5. handheld base; 6. QR code area; 7. linkage device; 8. button; 9. second gear; 10. second connecting rod; 11. second pulley; 12. first gear; 13. first connecting rod; 14. first pulley; 15. slide rail. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] It should be noted that when a component is referred to as being “connected” to another component, it may be directly connected to the other component or there may be a central component; when a component is referred to as being “fixed” to another component, it may be directly fixed to the other component or there may be a central component.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0061] Example 1

[0062] In order to overcome the defects of the prior art, the present embodiment provides a remote-controlled robotic arm system based on a non-electrical handheld gripper, which does not need to rely on other powered interactive devices (such as VR glasses, wearable armbands, force feedback gloves, etc.), and only remotely controls the operation of the robotic arm on the gripper without electrical design, thereby improving remote work efficiency.

[0063] The remote-controlled robotic arm system based on the non-powered handheld gripper provided in this embodiment includes a non-powered handheld gripper and a remote robotic arm that establish a communication connection, wherein the non-powered handheld gripper is located at the operator side, and the remote robotic arm is located in the working environment.

[0064] The non-electric handheld clamp is understood as a handheld clamp without electrical design, which at least includes a computing board, a sensor, a display, a handheld base, buttons, fingers and a linkage device.

[0065] The computing board is used to provide computing resources, including establishing a remote connection between the non-powered handheld gripper and the remote robotic arm, estimating the position and posture of the non-powered handheld gripper, estimating the finger width of the non-powered handheld gripper, remotely controlling the remote robotic arm, and moving the gripper at the end of the remote robotic arm. The hardware that can be used includes but is not limited to: ARM Linux development board, mobile phone platform development board, PC, server, FPGA or single-chip microcomputer, etc.

[0066] Sensors are used to obtain the position and gripping width of the non-powered handheld gripper, including but not limited to monocular cameras, binocular cameras, RGBD cameras, IMUs, lidars, or UWB and other sensing devices.

[0067] A display, used to display the remote image captured by the remote robotic arm, including but not limited to LED, OLED, CRT, LCD, PDP, etc.

[0068] The handheld base is used as the handheld part of the non-powered handheld gripper.

[0069] Button for power supply.

[0070] The linkage device connects the button and the finger, and is used to convert the power provided by the button into controlling the closing / opening of the finger of the non-electric handheld gripper. On the basis of satisfying that the non-electric handheld gripper can perform posture and clamping width estimation, no strict restrictions are imposed on the structure of the non-electric handheld gripper.

[0071] The end of the remote robotic arm is connected to a gripper and a camera, and the non-powered handheld gripper establishes a communication connection with the remote robotic arm, which means that the non-powered handheld gripper and the remote robotic arm communicate with each other through remote communication. The remote communication method includes but is not limited to: hardware: WIFI, Bluetooth, wired network, serial port, etc.; software protocol: Socket, Serial, SPI, I2C, UDP / TCP, etc.

[0072] like Figure 1 As shown, the non-powered handheld gripper in the teleoperated manipulator system based on the non-powered handheld gripper of this embodiment performs the following operations:

[0073] Step S1, obtaining and displaying a remote image captured by a remote robotic arm, wherein the remote image is used by an operator to obtain a remote operation progress and to operate a non-powered handheld gripper based on the remote image.

[0074] Step S2, calculating the position increment of the non-powered hand-held gripper between the two latest adjacent moments.

[0075] Specifically, the estimated time C The position of the hand-held gripper without power at all times T C = {R C ,t C}, where R C It's time C The rotation matrix at time t C It's time C The translation matrix at time.

[0076] Get time C-1 The estimated position of the unpowered handheld gripper at the moment is T C-1 = {R C-1 ,t C-1}, where R C-1 It's time C-1 The rotation matrix at time t C-1 It's time C-1 The translation matrix at time.

[0077] Calculate the time from the time of the handheld gripper without power C-1 Time to time C The pose increment at time is where dR C-1～C From time C-1 Time to time C The rotation matrix increment at time , and dt C-1～C From time C-1 Time to time C The translation matrix increment at time dt C-1～C =tC -dR C-1～C t C-1 .

[0078] Among them, through monocular cameras, binocular cameras, RGBD cameras, IMU, lidar or UWB and other sensors, visual SLAM algorithms, laser SLAM algorithms, AOA algorithms, AOD algorithms and other algorithms are adopted to estimate the posture of the unpowered handheld gripper in real time, and control the movement of the remote robotic arm in real time based on the posture of the unpowered handheld gripper.

[0079] Step S3, obtaining the current state of the remote manipulator, and obtaining the moving target point pose of the remote manipulator according to the pose increment of the non-powered handheld gripper and the current state of the remote manipulator.

[0080] The moving target point pose of the remote manipulator refers to estimating the expected target pose of the remote manipulator based on the pose increment of the unpowered handheld gripper and the current pose of the remote manipulator. The methods used include but are not limited to: rotation matrix transformation, quaternion transformation, Euler angle transformation, etc.

[0081] Specifically, in this embodiment, the six-dimensional pose vector in the current state of the remote manipulator is converted into a 4×4 transformation matrix T robot = {R robot ,t robot}, where R robot is the current rotation matrix of the remote manipulator, t robot The current translation matrix of the remote robot.

[0082] Then the moving target point pose G of the remote manipulator is calculated T =T robot dT C-1～C = {G R ,G t}, where G R is the rotation matrix of the mobile target point of the remote manipulator, and G R =R robot dR C-1～C , G t is the translation matrix of the mobile target point of the remote manipulator, and G t =dR C-1～C t robot +dt C-1～C .

[0083] Step S4, calculating the clamping width increment of the non-powered hand-held clamping jaws between the two latest adjacent moments.

[0084] Specifically, get time C The rotation matrix of the first finger on the handheld gripper without power is R l_aruco and the translation matrix is ​​t l_aruco, get time C The rotation matrix of the second finger on the handheld gripper without power is R r_aruco and the translation matrix is ​​t r_aruco .

[0085] Then calculate time C The clamping width of the handheld gripper without power is Range C =t r_aruco [x]-t l_aruco [x], where t r_aruco [x] is the translation matrix t l_aruco The horizontal x-axis movement distance, t l_aruco [x] is the translation matrix t r_aruco The horizontal x-axis movement distance.

[0086] Get time C-1 The clamping width of the handheld gripper without power is Range C-1 , then the power-free hand-held gripper from time C-1 Time to time C The clamping width increment at the moment is Range C-1～C =Range C -Range C-1 .

[0087] Among them, through monocular cameras, binocular cameras, RGBD cameras, IMU, lidar or UWB and other sensors, visual SLAM algorithms, laser SLAM algorithms, AOA algorithms, AOD algorithms, QR code recognition and detection and other methods, the clamping width of the non-powered handheld gripper is estimated in real time, and the movement of the end gripper of the remote robotic arm is controlled in real time based on the clamping width of the non-powered handheld gripper.

[0088] Step S5, obtaining the current state of the remote manipulator end gripper, and obtaining the desired target width of the remote manipulator end gripper according to the gripping width increment of the non-powered handheld gripper and the current state of the remote manipulator end gripper.

[0089] The desired target width estimation of the remote manipulator end gripper refers to the estimation based on the gripping width increment of the non-powered handheld gripper and the current width of the remote manipulator end gripper, and the methods used include but are not limited to: rotation matrix transformation, quaternion transformation, Euler angle transformation, etc. This embodiment is specifically:

[0090] Range expect =Range cur +Range C-1～C

[0091] Among them, Range expectis the desired target width of the remote robot end gripper, Range cur Range is the current width of the remote robot end gripper. C-1～C The latest clamping width increment for non-powered handheld grippers.

[0092] Step S6: Send the moving target point posture and the desired target width of the end gripper of the remote robotic arm to the remote robotic arm so that the remote robotic arm can perform an action.

[0093] Specifically, if Range expect ≤Range min or Range expect ≥Range max , then discard the latest desired target width Range of the remote robot end gripper expect , only the mobile target point pose is sent to the remote manipulator; otherwise, the mobile target point pose and the desired target width of the remote manipulator end gripper are sent to the remote manipulator, where Range min is the minimum gripping width of the remote robot end gripper, Range max The maximum gripping width of the remote robot end gripper.

[0094] It should be noted that, in the present embodiment, steps S2-S3 and steps S4-S5 can be executed successively or simultaneously. Therefore, the moving target point posture and the desired target width of the remote manipulator end gripper obtained by the execution of the two can be sent separately in real time or simultaneously. In addition, in the present embodiment, the non-powered handheld gripper can only establish a communication connection with the remote manipulator, and the remote manipulator forwards the desired target width of the remote manipulator end gripper to the remote manipulator end gripper; or the non-powered handheld gripper can establish a communication connection with the remote manipulator and the remote manipulator end gripper respectively, and the non-powered handheld gripper directly sends the desired target width of the remote manipulator end gripper to the gripper of the remote manipulator. If the non-powered handheld gripper establishes communication with the remote manipulator end gripper, its remote communication method includes but is not limited to: hardware: WIFI, Bluetooth, wired network, serial port, etc.; software protocol: Socket, Serial, SPI, I2C, UDP / TCP, etc.

[0095] The remote manipulator is controlled to move based on the position of the moving target point, and the methods used include but are not limited to: forward solution of manipulator motion, inverse solution of manipulator motion joints, etc. The remote manipulator end gripper is controlled to move based on the desired target width, and the methods used include but are not limited to: forward solution of gripper motion, inverse solution of gripper motion, etc.

[0096] Step S7, repeatedly acquiring the remote images taken by the remote robotic arm and controlling the remote robotic arm to move, and repeating steps S1-S6 until the remote operation is completed.

[0097] Example 2

[0098] This embodiment provides a specific example. The teleoperated robot arm system based on the non-powered handheld gripper of this embodiment includes a remote robot arm and a non-powered handheld gripper. The end of the remote robot arm is connected to a gripper and a camera; Figure 2-3 As shown, the non-powered handheld gripper includes a computing board 3, a camera 2, an LED display 1, a handheld base 5, a button 8, a finger 4 and a linkage device 7.

[0099] The linkage device 7 includes a first connecting rod 13, a second connecting rod 10, a first gear 12, a second gear 9, a first pulley 14, a second pulley 11 and a slide rail 15, and the finger 4 includes a first finger 402 and a second finger 401. The button 8 is located at the bottom of the handheld base 5, and a slide groove is provided at the bottom of the handheld base 5. The button 8 is located in the slide groove, and an elastic component (such as a spring) is connected between the button 8 and the inner wall of the slide groove, so that the button 8 can be pushed to the slide groove end by the elastic component without external force.

[0100] The two sides of the top of the button 8 are respectively fixed to one end of the first connecting rod 13 and one end of the second connecting rod 10, the other end of the first connecting rod 13 is connected to the periphery of the first gear 12, and the other end of the second connecting rod 10 is connected to the periphery of the second gear 9. The centers of the first gear 12 and the second gear 9 are rotatably mounted on the handheld base 5 through the wheel axle, and teeth are provided on the first gear 12 at the side opposite to the connection with the first connecting rod 13, and teeth are provided on the second gear 9 at the side opposite to the connection with the second connecting rod 10. The first pulley 14 and the second pulley The wheels 11 are all engaged with the slide rail 15, and the first pulley 14 is provided with teeth on the side close to the first gear 12, and the second pulley 11 is provided with teeth on the side close to the second gear 9, the teeth of the first gear 12 mesh with the teeth of the first pulley 14, and the teeth of the second gear 9 mesh with the teeth of the second pulley 11, and the first pulley 14 and the second pulley 11 are arranged relatively on the slide rail 15, the first pulley 14 is fixed to the first finger 402 at the end away from the first gear 12, and the second pulley 11 is fixed to the second finger 401 at the end away from the second gear 9. The first finger 402 and the second finger 401 are controlled to translate by pressing / releasing the button 8. In this embodiment, pressing the button 8 controls the first finger 402 and the second finger 401 to approach, and releasing the button 8 controls the first finger 402 and the second finger 401 to move away.

[0101] The present invention aims to remotely operate a remote manipulator through a non-powered handheld gripper, which should include communication between the non-powered handheld gripper and the remote manipulator, posture estimation of the non-powered handheld gripper, clamping width estimation of the non-powered handheld gripper, and control and feedback of the non-powered handheld gripper on the remote manipulator. Figure 4 As shown, the steps are as follows:

[0102] Step 1: Establish a SOCKET communication connection with the remote robotic arm through the remote communication RPC module on the non-powered handheld gripper, and use the UDP protocol for transmission to ensure the real-time nature of the communication process.

[0103] Step 2: If the communication connection in step 1 fails, a connection failure warning is sent; if the communication connection in step 1 is successful, the remote robotic arm starts the network streaming service through FFMEG, and sends the image data collected by the remote robotic arm camera as H.264 raw stream to the RTMP server.

[0104] Step 3: The computing board at the non-powered handheld gripper end connects to the RTMP service of the remote robotic arm through OpenCVCaptureVideo. After RTP unpacking and H.264 decoding, the data in OpenCV Mat format is obtained. By calling the imshow function, the remote image is displayed on the LED display.

[0105] Step 4: The operator adjusts and moves the non-powered handheld gripper based on the remote screen displayed on the LED, and the computing board obtains the time C Image data collected by the camera on the handheld gripper without power at all times Img C 、time C-1 to time C All IMU data Imu at the moment C-1～C , using the ORB_SLAM3 algorithm to estimate time C Hand-held gripper position T without power at all times C = {R C ,t C}, where R C It's time C The rotation matrix at time t C It's time C The translation matrix at time. It is known that the ORB_SLAM3 algorithm has C-1 The estimated position of the unpowered handheld gripper at the moment is T C-1 = {R C-1 ,t C-1}, where R C-1 It's time C-1 The rotation matrix at time t C-1 It's time C-1 The translation matrix at time . Therefore, the ORB_SLAM3 algorithm estimates the translation matrix of the unpowered handheld gripper with respect to time C-1 to time C The pose increment at time is From time C-1 Time to timeC The rotation matrix increment at time is From time C-1 Time to time C The translation matrix increment at time is dt C-1～C =t C -dR C-1～ C t C-1 .

[0106] Step 5: The non-powered handheld gripper sends a request to obtain the status to the remote robotic arm through the remote communication RPC module. The remote robotic arm calls the get_all_state function to obtain the current status of the robotic arm S = {Joints [1:N] ,Position [1:6]}, Joints [1:N] is the joint angle with N degrees of freedom, Position [1:6] is a 6-dimensional pose vector. The remote manipulator transmits the status data S back to the non-powered handheld gripper end through the remote communication RPC module.

[0107] Step 6: The non-powered handheld gripper transforms the state data S obtained in step 5 and converts the 6-dimensional pose vector into a 4×4 transformation matrix T through the scipyfrom_rotvec function. robot = {R robot ,t robot}, where R robot is the current rotation matrix of the remote manipulator, t robot is the current translation matrix of the remote manipulator. Combined with the non-powered handheld gripper position increment dT obtained in step 4 C-1～C , calculate the moving target point pose G of the remote manipulator T =T robot dT C-1～C , the mobile target point rotation matrix G of the remote manipulator R =R robot dR C-1～C , the translation matrix G of the mobile target point of the remote manipulator t =dR C-1～C t robot +dt C-1～C .

[0108] Step 7: The operator determines whether to close / open the fingers of the non-powered handheld gripper by pressing / releasing the button based on the remote screen displayed by the LED, thereby controlling the remote robotic arm end-connected gripper to close / open the same distance. To achieve this function, the camera at the non-powered handheld gripper end captures an image Img finger , image Img fingerThe visual angle includes the first finger, the second finger, the QR code on the first finger, and the QR code on the second finger of the non-powered handheld gripper. finger Enter the opencvdetectMarkes function, the detectMarkes function will detect the image Img finger The two QR codes on the first and second fingers and the four corner vertices corresponding to the QR codes left (including the 4 vertices of the QR code on the first finger) and corner right (including the 4 vertices of the QR code on the second finger), and change the corner left and corner right , and the length of the QR code L aruco = 0.15 Input opencvestimatePoseSingleMarkers function respectively to get the rotation matrix and translation matrix R of the QR code on the first finger and the QR code on the second finger respectively. l_aruco ,t l_aruco ,R r_aruco ,t r_aruco Calculate the gripping width Range of the first and second fingers of the non-powered handheld gripper C =t r_aruco [x]-t l_aruco [x], where t r_aruco [x] is the translation matrix t l_aruco The horizontal x-axis movement distance, t l_aruco [x] is the translation matrix t r_aruco The horizontal x-axis movement distance. The clamping width Range at the previous moment is known C-1 , so the non-powered handheld gripper is about time C-1 to time C The clamping width increment at the moment is Range C-1～C =Range C -Range C-1 .

[0109] Step 8: The non-powered handheld gripper sends a request to obtain the state of the remote robotic arm end connection gripper (also called the remote robotic arm end gripper or gripper) through the remote communication RPC module. The remote robotic arm end connection gripper calls the get_all_state function to obtain the current state of the remote robotic arm end gripper S = {Range cur ,Range max ,Range min}, range cur is the current width of the gripper, range max is the maximum width that the gripper can reach, range minis the minimum width that the gripper can reach. The gripper transmits the status data S back to the non-powered handheld gripper end through the remote communication RPC module of the remote manipulator.

[0110] Step 9: The non-powered handheld gripper end receives the state data S obtained in step 8 and the gripping width increment Range of the first and second fingers of the non-powered handheld gripper obtained in step 7. C-1～C Calculate the desired target width Range of the remote robot end-connected gripper expect =Range cur +Range C-1～C , when Range expect ≤Range min or Range expect ≥Range max , indicating the calculated expected target width Range expect The desired target width cannot be set to Range because the movement range of the end-arm gripper is exceeded. expect Sent to the remote robot end connection gripper, set flag = False, otherwise flag = True, where flag indicates whether to set the expected target width Range expect Sent to the remote end-of-arm gripper.

[0111] Step 10: The non-powered handheld gripper sends the moving target point position G to the remote robotic arm through the remote communication RPC module T , the remote manipulator calls the motion solution move_p function to move to the vicinity of the moving target point pose; if flag = True, the non-powered handheld gripper sends the desired target width Range to the gripper through the remote manipulator expect , the gripper moves the same distance by calling the motion forward solution move_p function.

[0112] Step 11: Repeat steps 4-10, and the operator remotely controls the movement of the remote robot arm and the gripper through the non-powered handheld gripper until the remote operation is completed.

[0113] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0114] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A teleoperated robotic arm system based on a non-powered handheld gripper, characterized in that: The remote-operated manipulator system based on the non-powered handheld gripper comprises a non-powered handheld gripper and a remote manipulator that establish a communication connection, wherein the non-powered handheld gripper is located at the operator side, and the remote manipulator is located in the working environment, and the non-powered handheld gripper performs the following operations: Acquire and display a remote image captured by a remote robotic arm, wherein the remote image is used by an operator to obtain a remote operation progress and operate the non-powered handheld gripper based on the remote image; Calculate the position increment of the unpowered hand-held gripper between the latest two adjacent moments; The current state of the remote manipulator is obtained, and the moving target point pose of the remote manipulator is obtained according to the pose increment of the non-powered handheld gripper and the current state of the remote manipulator; Calculate the clamping width increment of the non-powered hand-held clamp between the latest two adjacent moments; The current state of the remote manipulator end gripper is obtained, and the desired target width of the remote manipulator end gripper is obtained according to the gripping width increment of the non-powered hand-held gripper and the current state of the remote manipulator end gripper; Sending the moving target point position and the desired target width of the end gripper of the remote robotic arm to the remote robotic arm for the remote robotic arm to perform an action; Repeatedly obtain the remote images taken by the remote robotic arm and control the remote robotic arm's movements until the remote operation is completed.

2. The teleoperated manipulator system based on non-powered handheld gripper according to claim 1, characterized in that: The calculation of the position increment of the non-powered hand-held gripper between two latest adjacent moments includes: Estimated time C The position of the hand-held gripper without power at all times T C = {R C ,t C }, where R C It's time C The rotation matrix at time t C It's time C The translation matrix at time; Get time C-1 The estimated position of the unpowered handheld gripper at the moment is T C-1 = {R C-1 ,t C-1 }, where R C-1 It's time C-1 The rotation matrix at time t C-1 It's time C-1 The translation matrix at time; Calculate the time from the time of the handheld gripper without power C-1 Time to time V The pose increment at time is where dR C-1～C From time C-1 Time to time C The rotation matrix increment at time , and dt C-1～C From time C-1 Time to time C The translation matrix increment at time, and dt C-1～C =t C -dR C-1～C t C-1 .

3. The teleoperated manipulator system based on non-powered handheld gripper according to claim 2, characterized in that: The step of obtaining the moving target point pose of the remote manipulator according to the pose increment of the non-powered handheld gripper and the current state of the remote manipulator comprises: Convert the six-dimensional pose vector in the current state of the remote manipulator into a 4×4 transformation matrix T robot = {R robot ,t robot }, where R robot is the current rotation matrix of the remote manipulator, t robot is the current translation matrix of the remote manipulator; Then the moving target point pose G of the remote manipulator is calculated T =T robot dT C-1～C = {G R ,G t }, where G R is the rotation matrix of the mobile target point of the remote manipulator, and G R =R robot dR C-1～C , G t is the translation matrix of the mobile target point of the remote manipulator, and G t =dR C-1～C t robot +dt C-1～C .

4. The teleoperated manipulator system based on non-powered handheld gripper according to claim 1, characterized in that: The calculation of the clamping width increment of the non-powered hand-held clamping jaws between the latest two adjacent moments includes: Get time C The rotation matrix of the first finger on the handheld gripper without power is R l_aruco and the translation matrix is ​​t l_aruco , get time C The rotation matrix of the second finger on the handheld gripper without power is R r_aruco and the translation matrix is ​​t r_aruco ; Then calculate time C The clamping width of the handheld gripper without power is Range C =t r_aruco [x]-t l_aruco [x], where t r_aruco [x] is the translation matrix t l_aruco The horizontal x-axis movement distance, t l_aruco [x] is the translation matrix t r_aruco The horizontal x-axis movement distance; Get time C-1 The clamping width of the handheld gripper without power is Range C-1 , then the power-free hand-held gripper from time C-1 Time to time C The clamping width increment at the moment is Range C-1～C =Range C -Range C-1 .

5. The teleoperated robot arm system based on non-powered handheld gripper according to claim 1, characterized in that: The method of obtaining the desired target width of the remote manipulator end gripper according to the gripping width increment of the non-powered handheld gripper and the current state of the remote manipulator end gripper comprises: Range expect =Range cur +Range C-1～C Among them, Range expect is the desired target width of the remote robot end gripper, Range cur Range is the current width of the remote robot end gripper. C-1～C The latest clamping width increment for non-powered handheld grippers.

6. The teleoperated manipulator system based on non-powered handheld gripper according to claim 1, characterized in that: The step of sending the moving target point position and the desired target width of the end gripper of the remote robotic arm to the remote robotic arm comprises: If Range expect ≤Range min or Range expect ≥Range max , then discard the latest desired target width Range of the remote robot end gripper expect , only the mobile target point pose is sent to the remote manipulator; otherwise, the mobile target point pose and the desired target width of the remote manipulator end gripper are sent to the remote manipulator, where Range min is the minimum gripping width of the remote robot end gripper, Range max The maximum gripping width of the remote robot end gripper.

7. The teleoperated manipulator system based on non-powered handheld gripper according to claim 1, characterized in that: The non-powered handheld gripper includes a computing board, a sensor, a display, a handheld base, a button, a finger, and a linkage device; The computing board is used to provide computing resources; The sensor is used to obtain the position and gripping width of the non-powered handheld gripper; The display is used to display the remote image captured by the remote robotic arm; The handheld base is used as the handheld part of the non-electric handheld clamp; The button is used to provide power; The linkage device connects the button and the finger, and is used to convert the power provided by the button into controlling the closing / opening of the finger of the non-powered handheld clamp.

8. The teleoperated robot arm system based on non-powered handheld gripper according to claim 7, characterized in that: The linkage device includes a first connecting rod, a second connecting rod, a first gear, a second gear, a first pulley, a second pulley and a slide rail, and the finger includes a first finger and a second finger; The two sides of the button are respectively fixed to one end of the first connecting rod and one end of the second connecting rod, the other end of the first connecting rod is connected to the periphery of the first gear, and the other end of the second connecting rod is connected to the periphery of the second gear. The centers of the first gear and the second gear are rotatably mounted on the handheld base through axles, the teeth of the first gear are meshed with the teeth of the first pulley, and the teeth of the second gear are meshed with the teeth of the second pulley. Both the first pulley and the second pulley are buckled with the slide rail, and the first pulley and the second pulley are arranged opposite to each other on the slide rail, the first pulley is fixed to the first finger, and the second pulley is fixed to the second finger.

Images