A remote-controlled robotic arm system based on a non-powered handheld gripper

Abstract

This invention belongs to the field of teleoperated robotics and discloses a teleoperated robotic arm system based on a non-powered handheld gripper. The system comprises a non-powered handheld gripper and a remote robotic arm in communication with each other, with the non-powered handheld gripper located on the operator's side and the remote robotic arm located in the working environment. This invention utilizes the non-powered handheld gripper's SLAM algorithm and finger-width positioning algorithm to achieve precise remote control of the robotic arm and its connected end effector, facilitating remote operations in complex or dangerous environments inaccessible to humans. Furthermore, the system can be installed on any non-powered handheld gripper, thus resolving the issue of prolonged operation in power-deficient environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of teleoperated robots, and specifically relates to a teleoperated robotic arm system based on a non-powered handheld gripper. The system uses the non-powered handheld gripper's SLAM (Simultaneous Localization and Mapping) algorithm and a finger-width positioning algorithm to achieve precise remote control of the robotic arm and its connected end effector, thereby assisting in remote operations in complex or dangerous environments that are inaccessible to humans. Furthermore, the system can be mounted on any non-powered handheld gripper, thus resolving the problem of being unable to operate for long periods of time in power-deficient environments. Background Art

[0002] With the continuous development of robotics technology, teleoperated robotic arm systems have shown great potential in areas such as home services, disaster relief, geological exploration, and space exploration. However, existing teleoperation systems often rely on complex electrical connections and energy supplies, limiting their flexibility and application scenarios.

[0003] Among existing technologies, patent application CN118238147A proposes a teleoperation-based bionic manipulator human-machine interaction system and method. This method utilizes a headset (Virtual Reality) headset, a wearable armband, and force feedback gloves to communicate with a remote manipulator. The wearable armband and force feedback gloves control the movement of the remote manipulator, and the headset provides real-time feedback on the real working environment, achieving consistency between the virtual and real working environments from scene to interaction. This method can be used to replace workers in extremely dangerous environments. This method can improve the operator's remote work efficiency and immersion in the work environment, but it has significant disadvantages: 1. The headset, wearable armband, and force feedback gloves are required, which increases the load and discomfort during operation; 2. When wearing the headset, the operator cannot observe their surroundings, posing a safety hazard; 3. The headset, wearable armband, and force feedback gloves all require additional power supplies, cannot operate for long periods of time offline, and are relatively expensive.

[0004] Patent application CN118322199A proposes a data-glove-based teleoperation method for a five-fingered dexterous arm-hand robot. This method allows operators to remotely control the robot's movements by wearing data gloves. This method improves remote operator efficiency and precision, but the high-precision data gloves require an additional power source, making them limited to long-term offline operation and expensive.

[0005] Patent application publication number CN118143961A proposes a shared control system and method for B-ultrasound teleoperation based on an attention mechanism. This method uses VR glasses to detect the operator's attention point, presents the environment at that point, and controls a remote robotic arm using a Touch X joystick to operate at that point. This method, primarily used in the B-ultrasound field, is not reproducible for general operations. Furthermore, it relies on a Touch X for teleoperation, requiring an additional power supply. It cannot operate for extended periods offline and is costly.

[0006] In summary, existing technologies mostly use AR cameras, data gloves and other devices to remotely control the movement of robotic arms. Their main disadvantages are:

[0007] 1. High equipment cost;

[0008] 2. An external power supply is required for the device, and it cannot work for a long time in an offline state;

[0009] 3. The equipment needs to be worn close to the user, which can easily cause discomfort to the user; and the equipment needs to be operated with electricity, which poses a safety hazard of leakage;

[0010] 4. Most solutions based on VR glasses easily block the user's real vision, making it impossible to observe their surroundings, thus posing certain safety risks. Summary of the Invention

[0011] The purpose of the present invention is to provide a remote-controlled robotic arm system based on a non-electric handheld gripper, which does not require an external power supply and can be quickly customized through 3D printing technology. By combining SLAM algorithms with remote communication technologies, the economy, portability and applicability of the system are greatly improved.

[0012] To achieve the above object, the technical solution adopted by the present invention is:

[0013] A remote-controlled robotic arm system based on a non-powered handheld gripper includes a non-powered handheld gripper and a remote robotic arm that establish a communication connection. The non-powered handheld gripper is located on the operator's side, and the remote robotic arm is located in the working environment. The non-powered handheld gripper performs the following operations:

[0014] Acquiring and displaying a remote image captured by a remote robotic arm, wherein the remote image is used by an operator to obtain remote operation progress and operate the non-powered handheld gripper based on the remote image;

[0015] Calculate the position increment of the unpowered handheld gripper between the two latest adjacent moments;

[0016] Obtain the current state of the remote manipulator, and obtain the moving target point pose of the remote manipulator based on the pose increment of the unpowered handheld gripper and the current state of the remote manipulator;

[0017] Calculate the increment of the gripping width of the unpowered handheld gripper between the two latest adjacent moments;

[0018] Obtaining the current state of the remote manipulator end gripper, and obtaining the desired target width of the remote manipulator end gripper according to the gripping width increment of the non-powered handheld gripper and the current state of the remote manipulator end gripper;

[0019] Sending the moving target point pose and the desired target width of the remote manipulator end gripper to the remote manipulator for the remote manipulator to perform an action;

[0020] Repeatedly obtain the remote images captured by the remote robotic arm and control the remote robotic arm's movements until the remote operation is completed.

[0021] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution. They are merely further supplements or optimizations. Under the premise that there are no technical or logical contradictions, each optional method can be combined separately for the above-mentioned overall solution, or multiple optional methods can be combined.

[0022] Preferably, the step of calculating the position increment of the unpowered handheld gripper between two latest adjacent moments includes:

[0023] Estimated time C The position of the handheld gripper without power at all times T C ={R C ,t C}, where R C It's time C The rotation matrix at time t C It's time C The translation matrix of the moment;

[0024] Get time C-1 The estimated pose of the unpowered handheld gripper at the moment is T C-1 ={R C-1 ,t C-1}, where R C-1 It's time C-1 The rotation matrix at time t C-1 It's time C-1 The translation matrix of the moment;

[0025] Calculate the time from the time of the unpowered handheld gripper C-1 Time to time C The pose increment at time is where dR C-1～C From time C-1 Time to time C The rotation matrix increment at time , and dt C-1～C From time C-1 Time to time C The translation matrix increment at time dt C-1～C =t C -dR C-1～C t C-1 .

[0026] Preferably, the step of obtaining the moving target point pose of the remote manipulator according to the pose increment of the non-powered handheld gripper and the current state of the remote manipulator comprises:

[0027] Convert the six-dimensional pose vector in the current state of the remote manipulator into a 4×4 transformation matrix T robot ={R robot ,t robot}, where R robot is the current rotation matrix of the remote manipulator, t robot is the current translation matrix of the remote manipulator;

[0028] Then the moving target point pose G of the remote manipulator is calculated T =T robot dT C-1～C ={G R ,G t}, where G R is the rotation matrix of the target point of the remote manipulator, and G R =R robot dR C-1～C , G t is the translation matrix of the target point of the remote manipulator, and G t =dR C-1～C t robot +dt C-1～C .

[0029] Preferably, the calculation of the gripping width increment of the non-powered handheld gripper between two latest adjacent moments includes:

[0030] Get time C The rotation matrix of the first finger on the handheld gripper without power is T l_aruco and the translation matrix is ​​t l_aruco , get time C The rotation matrix of the second finger on the handheld gripper without power is R r_aruco and the translation matrix is ​​t r_aruco ;

[0031] Then calculate time C The clamping width of the handheld gripper without power is Range C =t r_aruco [x]-t l_aruco [x], where t r_aruco [x] is the translation matrix t l_aruco The horizontal x-axis movement distance, t l_aruci [x] is the translation matrix t r_aruco The horizontal x-axis movement distance;

[0032] Get time C-1 The clamping width range of the handheld gripper without power at all times C-1 , then the handheld gripper without power is C-1 Time to time C The clamping width increment at the moment is Range C-1～C =Range C -Range C-1 .

[0033] Preferably, the step of obtaining a desired target width of the remote manipulator end gripper according to the gripping width increment of the non-powered handheld gripper and the current state of the remote manipulator end gripper comprises:

[0034] Range expect =Range cur +Range C-1～C

[0035] Among them, Range expect is the desired target width of the remote robot end gripper, Range cur Range is the current width of the remote robot end gripper. C-1～C The latest clamping width increment for non-powered handheld grippers.

[0036] Preferably, the step of sending the moving target point pose and the desired target width of the remote manipulator end gripper to the remote manipulator comprises:

[0037] If Range expect ≤Range min or Range expect ≥Range max , then discard the latest desired target width Range of the remote robot end gripper expect , only the moving target point pose is sent to the remote manipulator; otherwise, the moving target point pose and the desired target width of the remote manipulator end gripper are sent to the remote manipulator, where Range min Range is the minimum gripping width of the remote robot arm end gripper.max The maximum gripping width of the remote robot arm's end gripper.

[0038] Preferably, the non-electric handheld gripper comprises a computing board, a sensor, a display, a handheld base, buttons, fingers and a linkage device;

[0039] The computing board is used to provide computing resources;

[0040] The sensor is used to obtain the position and gripping width of the non-powered handheld gripper;

[0041] The display is used to display the remote image captured by the remote robotic arm;

[0042] The handheld base is used as the handheld part of the non-electric handheld gripper;

[0043] The button is used to provide power;

[0044] The linkage device connects the button and the finger, and is used to convert the power provided by the button into controlling the closing / opening of the finger of the non-electric handheld gripper.

[0045] Preferably, the linkage device includes a first connecting rod, a second connecting rod, a first gear, a second gear, a first pulley, a second pulley and a slide rail, and the finger includes a first finger and a second finger;

[0046] The two sides of the button are respectively fixed to one end of the first connecting rod and one end of the second connecting rod, the other end of the first connecting rod is connected to the periphery of the first gear, and the other end of the second connecting rod is connected to the periphery of the second gear. The centers of the first gear and the second gear are rotatably mounted on the handheld base through axles, the teeth of the first gear are engaged with the teeth of the first pulley, and the teeth of the second gear are engaged with the teeth of the second pulley. The first pulley and the second pulley are both engaged with the slide rail, and the first pulley and the second pulley are arranged opposite to each other on the slide rail, the first pulley is fixed to the first finger, and the second pulley is fixed to the second finger.

[0047] The present invention provides a remote-controlled robotic arm system based on a non-electric handheld gripper, which has the following advantages compared to the prior art:

[0048] 1. It can work on any non-powered handheld gripper, without the need for an additional power supply, and can work for a long time even in an offline state;

[0049] 2. The design is simple and portable, and can be 3D printed by yourself, with extremely low equipment cost;

[0050] 3. Compared with other solutions that require wearing personal interaction devices, this invention relies on a non-powered handheld gripper to remotely operate a remote robotic arm. This eliminates the discomfort of wearing a personal foreign object and is lightweight.

[0051] 4. The present invention does not require live operation and does not cause leakage. Compared with the solution based on VR glasses, there is no problem of line of sight obstruction, so there is no safety hazard during use.

[0052] 5. This method has no requirements for scene restrictions, is highly versatile and has a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flowchart of an execution of a remote-controlled robotic arm system based on a non-powered handheld gripper according to Example 1 of the present invention;

[0054] Figure 2 This is a schematic structural diagram of the non-electric handheld gripper used in Example 2 of the present invention;

[0055] Figure 3 This is a schematic structural diagram of the linkage device of the non-electric handheld gripper used in Example 2 of the present invention;

[0056] Figure 4 This is an execution flow chart of a remote-controlled robotic arm system based on a non-electric handheld gripper according to embodiment 2 of the present invention.

[0057] In the diagram: 1. LED display; 2. Camera; 3. Computing board; 4. Finger; 401. Second finger; 402. First finger; 5. Handheld base; 6. QR code area; 7. Linkage device; 8. Button; 9. Second gear; 10. Second connecting rod; 11. Second pulley; 12. First gear; 13. First connecting rod; 14. First pulley; 15. Slide rail. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0059] It should be noted that when a component is referred to as being “connected” to another component, it may be directly connected to the other component or there may be a component in the middle; when a component is referred to as being “fixed” to another component, it may be directly fixed to the other component or there may be a component in the middle.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0061] Example 1

[0062] In order to overcome the shortcomings of the existing technology, this embodiment provides a remote-controlled robotic arm system based on a non-electrical handheld gripper, which does not need to rely on other powered interactive devices (such as VR glasses, wearable armbands, force feedback gloves, etc.). The robotic arm operation is remotely controlled only on the gripper without electrical design, thereby improving remote work efficiency.

[0063] The teleoperated robotic arm system based on the non-electric handheld gripper provided in this embodiment includes a non-electric handheld gripper and a remote robotic arm that establish a communication connection. The non-electric handheld gripper is located on the operator side, and the remote robotic arm is located in the working environment.

[0064] The non-electric handheld gripper is understood to be a handheld gripper without electrical design, which at least includes a computing board, a sensor, a display, a handheld base, buttons, fingers and a linkage device.

[0065] The computing board is used to provide computing resources, including establishing a remote connection between the unpowered handheld gripper and the remote robotic arm, estimating the position and finger width of the unpowered handheld gripper, remotely controlling the remote robotic arm, and moving the gripper at the end of the remote robotic arm. The hardware that can be used includes but is not limited to: ARM Linux development board, mobile phone platform development board, PC, server, FPGA or single-chip microcomputer, etc.

[0066] Sensors used to obtain the position and gripping width of the non-powered handheld gripper, including but not limited to monocular cameras, binocular cameras, RGBD cameras, IMUs, lidars, or UWB sensing devices.

[0067] A display, used to display the remote image captured by the remote robotic arm, including but not limited to LED, OLED, CRT, LCD, PDP, etc.

[0068] The handheld base is used as the handheld part of the non-electric handheld gripper.

[0069] Button for power supply.

[0070] A linkage connects the button and the finger, converting the power provided by the button into controlling the closing / opening of the fingers of the non-powered handheld gripper. As long as the non-powered handheld gripper can perform pose and grip width estimation, no strict structural restrictions are imposed on the gripper.

[0071] The end of the remote robotic arm is connected to the gripper and the camera, and the non-powered handheld gripper establishes a communication connection with the remote robotic arm, which means that the non-powered handheld gripper and the remote robotic arm communicate through remote communication. The remote communication method includes but is not limited to: hardware: WIFI, Bluetooth, wired network, serial port, etc.; software protocol: Socket, Serial, SPI, I2C, UDP / TCP, etc.

[0072] like Figure 1 As shown, the non-powered handheld gripper in the teleoperated manipulator system based on the non-powered handheld gripper of this embodiment performs the following operations:

[0073] Step S1: Acquire and display a remote image captured by a remote robotic arm. The remote image is used by an operator to obtain the progress of a remote operation and operate a non-powered handheld gripper based on the remote image.

[0074] Step S2: Calculate the position increment of the unpowered handheld gripper between the two latest adjacent moments.

[0075] Specifically, the estimated time C The position of the handheld gripper without power at all times T C ={R C ,t C}, where R C It's time C The rotation matrix at time t C It's time C The translation matrix at time.

[0076] Get time C-1 The estimated pose of the unpowered handheld gripper at the moment is T C-1 ={R C-1 ,t C-1}, where R C-1 It's time C-1 The rotation matrix at time t C-1 It's time C-1 The translation matrix at time.

[0077] Calculate the time from the time of the unpowered handheld gripper C-1 Time to time C The pose increment at time is where dR C-1～C From time C-1 Time to time C The rotation matrix increment at time , and dt C-1～C From time C-1 Time to time C The translation matrix increment at the moment, and dt C-1～C =tC -dR C-1～C t C-1 .

[0078] Among them, through monocular cameras, binocular cameras, RGBD cameras, IMU, lidar or UWB and other sensors, visual SLAM algorithms, laser SLAM algorithms, AOA algorithms, AOD algorithms and other algorithms are used to estimate the posture of the non-powered handheld gripper in real time, and control the movement of the remote robotic arm in real time based on the posture of the non-powered handheld gripper.

[0079] Step S3: Acquire the current state of the remote manipulator, and obtain the moving target point pose of the remote manipulator according to the pose increment of the non-powered handheld gripper and the current state of the remote manipulator.

[0080] The target point pose of the remote manipulator refers to estimating the desired target pose of the remote manipulator based on the pose increment of the unpowered handheld gripper and the current pose of the remote manipulator. The methods used include but are not limited to: rotation matrix transformation, quaternion transformation, Euler angle transformation, etc.

[0081] Specifically, this embodiment converts the six-dimensional pose vector of the current state of the remote manipulator into a 4×4 transformation matrix T robot ={R robot ,t robot}, where R robot is the current rotation matrix of the remote manipulator, t robot The current translation matrix of the remote manipulator.

[0082] Then the moving target point pose G of the remote manipulator is calculated T =T robot dT C-1～C ={G R ,G t}, where G R is the rotation matrix of the target point of the remote manipulator, and G R =R robot dR C-1～C , G t is the translation matrix of the target point of the remote manipulator, and G t =dR C-1～C t robot +dt C-1～C .

[0083] Step S4: Calculate the increment of the gripping width of the unpowered handheld gripper between the two latest adjacent moments.

[0084] Specifically, get time C The rotation matrix of the first finger on the handheld gripper without power is R l_aruco and the translation matrix is ​​t l_aruco, get time C The rotation matrix of the second finger on the handheld gripper without power is R r_aruco and the translation matrix is ​​t r_aruco .

[0085] Then calculate time C The clamping width of the handheld gripper without power is Range C =t r_aruco [x]-t l_aruco [x], where t r_aruco [x] is the translation matrix t l_aruco The horizontal x-axis movement distance, t l_aruco [x] is the translation matrix t r_aruco The horizontal x-axis movement distance.

[0086] Get time C-1 The clamping width range of the handheld gripper without power at all times C-1 , then the handheld gripper without power is C-1 Time to time C The clamping width increment at the moment is Range C-1～C =Range C -Range C-1 .

[0087] Among them, through monocular cameras, binocular cameras, RGBD cameras, IMU, lidar or UWB and other sensors, visual SLAM algorithms, laser SLAM algorithms, AOA algorithms, AOD algorithms, QR code recognition and detection and other methods, the clamping width of the non-powered handheld gripper is estimated in real time, and the movement of the remote robotic arm end gripper is controlled in real time based on the clamping width of the non-powered handheld gripper.

[0088] Step S5: Acquire the current state of the remote manipulator end gripper, and obtain the desired target width of the remote manipulator end gripper according to the gripping width increment of the non-powered handheld gripper and the current state of the remote manipulator end gripper.

[0089] The desired target width of the remote manipulator end gripper is estimated based on the gripping width increment of the unpowered handheld gripper and the current width of the remote manipulator end gripper. The methods used include but are not limited to: rotation matrix transformation, quaternion transformation, Euler angle transformation, etc. In this embodiment, the following are specific examples:

[0090] Range expect =Range cur +Range C-1～C

[0091] Among them, Range expectis the desired target width of the remote robot end gripper, Range cur Range is the current width of the remote robot end gripper. C-1～C The latest clamping width increment for non-powered handheld grippers.

[0092] Step S6: Send the moving target point posture and the desired target width of the remote manipulator end gripper to the remote manipulator for the remote manipulator to perform an action.

[0093] Specifically, if Range expect ≤Range min or Range expect ≥Range max , then discard the latest desired target width Range of the remote robot end gripper expect , only the moving target point pose is sent to the remote manipulator; otherwise, the moving target point pose and the desired target width of the remote manipulator end gripper are sent to the remote manipulator, where Range min Range is the minimum gripping width of the remote robot arm end gripper. max The maximum gripping width of the remote robot arm's end gripper.

[0094] It should be noted that steps S2-S3 and steps S4-S5 in this embodiment can be executed sequentially or simultaneously. Therefore, the moving target point pose and the desired target width of the remote manipulator end gripper obtained by executing both can be sent separately in real time or simultaneously. In addition, in this embodiment, the non-powered handheld gripper can establish a communication connection only with the remote manipulator, and the remote manipulator forwards the desired target width of the remote manipulator end gripper to the remote manipulator end gripper; or the non-powered handheld gripper can establish communication connections with the remote manipulator and the remote manipulator end gripper respectively, and directly send the desired target width of the remote manipulator end gripper to the gripper of the remote manipulator. If the non-powered handheld gripper establishes communication with the remote manipulator end gripper, its remote communication method includes but is not limited to: hardware: WIFI, Bluetooth, wired network, serial port, etc.; software protocol: Socket, Serial, SPI, I2C, UDP / TCP, etc.

[0095] The remote manipulator is controlled based on the position of the moving target point, using methods including but not limited to: forward solution of manipulator motion and inverse solution of manipulator joint motion. The remote manipulator's end gripper is controlled based on the desired target width, using methods including but not limited to: forward solution of gripper motion and inverse solution of gripper motion.

[0096] Step S7: repeatedly obtain the remote image captured by the remote robotic arm and control the remote robotic arm's movements, and repeat steps S1-S6 until the remote operation is completed.

[0097] Example 2

[0098] This embodiment provides a specific example. The teleoperated robotic arm system based on the non-powered handheld gripper of this embodiment includes a remote robotic arm and a non-powered handheld gripper. The end of the remote robotic arm is connected to a gripper and a camera; Figure 2-3 As shown, the non-powered handheld gripper includes a computing board 3, a camera 2, an LED display 1, a handheld base 5, a button 8, a finger 4 and a linkage device 7.

[0099] The linkage device 7 includes a first connecting rod 13, a second connecting rod 10, a first gear 12, a second gear 9, a first pulley 14, a second pulley 11, and a slide rail 15. The finger 4 includes a first finger 402 and a second finger 401. The button 8 is located at the bottom of the handheld base 5. The bottom of the handheld base 5 is provided with a slide groove. The button 8 is located in the slide groove. An elastic component (such as a spring) is connected between the button 8 and the inner wall of the slide groove so that the elastic component can push the button 8 to the end of the slide groove in the absence of external force.

[0100] The two sides of the top of the button 8 are respectively fixed to one end of the first connecting rod 13 and one end of the second connecting rod 10. The other end of the first connecting rod 13 is connected to the periphery of the first gear 12, and the other end of the second connecting rod 10 is connected to the periphery of the second gear 9. The centers of the first gear 12 and the second gear 9 are rotatably mounted on the handheld base 5 through the wheel axle. The first gear 12 is provided with teeth on the side opposite to the connection with the first connecting rod 13, and the second gear 9 is provided with teeth on the side opposite to the connection with the second connecting rod 10. The first pulley 14 and the second pulley The wheels 11 are all engaged with the slide rail 15, and the first pulley 14 is provided with teeth on the side close to the first gear 12, and the second pulley 11 is provided with teeth on the side close to the second gear 9. The teeth of the first gear 12 mesh with the teeth of the first pulley 14, and the teeth of the second gear 9 mesh with the teeth of the second pulley 11. The first pulley 14 and the second pulley 11 are arranged relative to each other on the slide rail 15. The first pulley 14 is fixed to the first finger 402 at the end away from the first gear 12, and the second pulley 11 is fixed to the second finger 401 at the end away from the second gear 9. The first finger 402 and the second finger 401 are controlled to translate by pressing / releasing the button 8. In this embodiment, pressing the button 8 moves the first finger 402 and the second finger 401 closer together, and releasing the button 8 moves the first finger 402 and the second finger 401 away from each other.

[0101] The present invention aims to remotely control a remote manipulator by means of a non-powered handheld gripper, which includes communication between the non-powered handheld gripper and the remote manipulator, posture estimation of the non-powered handheld gripper, estimation of the gripping width of the non-powered handheld gripper, and control and feedback of the non-powered handheld gripper to the remote manipulator. Figure 4 As shown, the steps are as follows:

[0102] Step 1: Establish a SOCKET communication connection between the remote communication RPC module on the non-powered handheld gripper and the remote robotic arm, and use the UDP protocol for transmission to ensure the real-time communication process.

[0103] Step 2: If the communication connection in step 1 fails, a connection failure warning is sent; if the communication connection in step 1 is successful, the remote robotic arm starts the network streaming service through FFMEG, and sends the image data collected by the remote robotic arm camera as an H.264 raw stream to the RTMP server.

[0104] Step 3: The computing board on the unpowered handheld gripper connects to the RTMP service of the remote robotic arm through OpenCVCaptureVideo. After RTP depacketization and H.264 decoding, it obtains the data in OpenCV Mat format. By calling the imshow function, the remote image is displayed on the LED display.

[0105] Step 4: The operator adjusts and moves the non-powered handheld gripper based on the remote screen displayed on the LED, and the calculation board obtains the time C Image data Img collected by the camera on the handheld gripper without power at all times C 、time C-1 to time C All IMU data Imu at this moment C-1～C , using ORB_SLAM3 algorithm to estimate time C Handheld gripper position T without power at all times C ={R C ,t C}, where R C It's time C The rotation matrix at time t C It's time C The translation matrix at time. It is known that the ORB_SLAM3 algorithm is C-1 The estimated position of the unpowered handheld gripper at the moment is T C-1 ={R C-1 ,t C-1}, where R C-1 It's time C-1 The rotation matrix at time t C-1 It's time C-1 The translation matrix at time . Therefore, the ORB_SLAM3 algorithm estimates the unpowered handheld gripper about time C-1 to time C The pose increment at time is From time C-1 Time to timeC The rotation matrix increment at time is From time C-1 Time to time C The translation matrix increment at time dt C-1～C =t C -dR C-1～C t C-1 .

[0106] Step 5: The unpowered handheld gripper sends a status request to the remote robotic arm through the remote communication RPC module. The remote robotic arm calls the get_all_state function to obtain the current state of the robotic arm S = {Joints [1:N] ,Position [1:6]}, Joints [1:N] is the joint angle with N degrees of freedom, Position [1:6] is a 6-dimensional pose vector. The remote manipulator transmits the status data S back to the non-powered handheld gripper through the remote communication RPC module.

[0107] Step 6: The non-powered handheld gripper transforms the state data S obtained in step 5 and converts the 6-dimensional pose vector into a 4×4 transformation matrix T using the scipyfrom_rotvec function. robot ={R robot ,t robot}, where R robot is the current rotation matrix of the remote manipulator, t robot is the current translation matrix of the remote manipulator. Combined with the unpowered handheld gripper position increment dT obtained in step 4 C-1～C , calculate the moving target point pose G of the remote manipulator T =T robot dT C-1～C , the mobile target point rotation matrix G of the remote manipulator R =R robot dR C-1～C , the translation matrix G of the target point of the remote manipulator t =dR C-1～C t robot +dt C-1～C .

[0108] Step 7: The operator judges whether to close / open the fingers of the non-powered handheld gripper by pressing / releasing the button based on the remote image displayed by the LED, thereby controlling the remote robotic arm end-connected gripper to close / open the same distance. To achieve this function, the camera at the non-powered handheld gripper end captures an image Img finger , image Img fingerThe visual angle includes the first finger, the second finger, the QR code on the first finger and the QR code on the second finger of the non-powered handheld gripper. finger Enter the opencvdetectMarkes function, the detectMarkes function will detect the image Img finger The two QR codes on the first and second fingers and the four corner vertices corresponding to the QR codes left (including the 4 vertices of the QR code on the first finger) and corner right (including the 4 vertices of the QR code on the second finger), and the corner left and corner right , and the length of the QR code L aruco =0.15 Input opencvestimatePoseSingleMarkers function respectively to obtain the rotation matrix and translation matrix R of the QR code on the first finger and the QR code on the second finger respectively. l_aruco ,t l_aruco ,R r_aruco ,t r_aruco Calculate the gripping width Range of the first and second fingers of the non-powered handheld gripper C =t r_aruco [x]-t l_aruco [x], where t r_aruco [x] is the translation matrix t l_aruco The horizontal x-axis movement distance, t l_aruco [x] is the translation matrix t r_aruco The horizontal x-axis movement distance. The clamping width Range at the previous moment is known. C-1 , so the non-powered handheld gripper about time C-1 to time C The clamping width increment at the moment is Range C-1～C =Range C -Range C-1 .

[0109] Step 8: The non-powered handheld gripper sends a status request to the remote manipulator end connection gripper (also called remote manipulator end gripper or gripper) through the remote communication RPC module. The remote manipulator end connection gripper calls the get_all_state function to obtain the current state of the remote manipulator end gripper S = {Range cur ,Range max ,Range min}, range cur is the current width of the gripper, range max is the maximum width that the gripper can reach, range minThe gripper transmits the status data S to the non-powered handheld gripper through the remote communication RPC module of the remote manipulator.

[0110] Step 9: The non-powered handheld gripper end uses the status data S obtained in step 8 and the gripping width increment Range of the first and second fingers of the non-powered handheld gripper obtained in step 7. C-1～C Calculate the desired target width Range of the remote robotic arm end-connected gripper expect =Range cur +Range C-1～C , when Range expect ≤Range min or Range expect ≥Range max , indicating the calculated expected target width Range expect The desired target width cannot be set to Range because it exceeds the moving range of the gripper connected to the end of the robot arm. expect Sent to the remote manipulator end connection gripper, set flag = False at this time, otherwise flag = True, where flag indicates whether the desired target width Range expect Sent to the remote end-of-arm gripper.

[0111] Step 10: The unpowered handheld gripper sends the moving target point pose G to the remote robotic arm through the remote communication RPC module. T , the remote manipulator calls the motion forward solution move_p function to move to the vicinity of the moving target point pose; if flag = True, the unpowered handheld gripper sends the desired target width Range to the gripper through the remote manipulator expect , the gripper moves an equal distance by calling the motion forward solution move_p function.

[0112] Step 11: Repeat steps 4-10, and the operator uses the non-powered handheld gripper to remotely control the movement of the remote robotic arm and gripper until the remote operation is completed.

[0113] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0114] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A teleoperated robotic arm system based on a non-powered handheld gripper, characterized in that: The remote-controlled robotic arm system based on the non-powered handheld gripper includes a non-powered handheld gripper and a remote robotic arm that establish a communication connection. The non-powered handheld gripper is located on the operator's side, and the remote robotic arm is located in the working environment. The non-powered handheld gripper performs the following operations: Acquiring and displaying a remote image captured by a remote robotic arm, wherein the remote image is used by an operator to obtain remote operation progress and operate the non-powered handheld gripper based on the remote image; Calculate the position increment of the unpowered handheld gripper between the two latest adjacent moments; Obtain the current state of the remote manipulator, and obtain the moving target point pose of the remote manipulator based on the pose increment of the unpowered handheld gripper and the current state of the remote manipulator; Calculate the increment of the gripping width of the unpowered handheld gripper between the two latest adjacent moments; Obtaining the current state of the remote manipulator end gripper, and obtaining the desired target width of the remote manipulator end gripper according to the gripping width increment of the non-powered handheld gripper and the current state of the remote manipulator end gripper; Sending the moving target point pose and the desired target width of the remote manipulator end gripper to the remote manipulator for the remote manipulator to perform an action; Repeatedly obtain the remote images captured by the remote robotic arm and control the remote robotic arm's movements until the remote operation is completed.

2. The teleoperated robotic arm system based on a non-electric handheld gripper according to claim 1, characterized in that: The calculation of the position increment of the unpowered handheld gripper between two latest adjacent moments includes: Estimated time C The position of the handheld gripper without power at all times T C ={R C ,t C }, where R C It's time C The rotation matrix at time t C It's time C The translation matrix of the moment; Get time C-1 The estimated pose of the unpowered handheld gripper at the moment is T C-1 ={R C-1 ,t C-1 }, where R C-1 It's time C-1 The rotation matrix at time t C-1 It's time C-1 The translation matrix of the moment; Calculate the time from the time of the unpowered handheld gripper C-1 Time to time V The pose increment at time is where dR C-1～C From time C-1 Time to time C The rotation matrix increment at time , and dt C-1～C From time C-1 Time to time C The translation matrix increment at the moment, and dt C-1～C =t C -dR C-1～C t C-1 .

3. The teleoperated robotic arm system based on a non-electric handheld gripper according to claim 2, characterized in that: The step of obtaining the moving target point pose of the remote manipulator according to the pose increment of the non-powered handheld gripper and the current state of the remote manipulator includes: Convert the six-dimensional pose vector in the current state of the remote manipulator into a 4×4 transformation matrix T robot ={R robot ,t robot }, where R robot is the current rotation matrix of the remote manipulator, t robot is the current translation matrix of the remote manipulator; Then the moving target point pose G of the remote manipulator is calculated T =T robot dT C-1～C ={G R ,G t }, where G R is the rotation matrix of the target point of the remote manipulator, and G R =R robot dR C-1～C , G t is the translation matrix of the target point of the remote manipulator, and G t =dR C-1～C t robot +dt C-1～C .

4. The teleoperated robotic arm system based on a non-electric handheld gripper according to claim 1, characterized in that: The calculation of the increment of the gripping width of the unpowered handheld gripper between two latest adjacent moments includes: Get time C The rotation matrix of the first finger on the handheld gripper without power is R l_aruco and the translation matrix is ​​t l_aruco , get time C The rotation matrix of the second finger on the handheld gripper without power is R r_aruco and the translation matrix is ​​t r_aruco ; Then calculate time C The clamping width of the handheld gripper without power is Range C =t r_aruco [x]-t l_aruco [x], where t r_aruco [x] is the translation matrix t l_aruco The horizontal x-axis movement distance, t l_aruco [x] is the translation matrix t r_aruco The horizontal x-axis movement distance; Get time C-1 The clamping width range of the handheld gripper without power at all times C-1 , then the handheld gripper without power is C-1 Time to time C The clamping width increment at the moment is Range C-1～C =Range C -Range C-1 .

5. The teleoperated robotic arm system based on a non-electric handheld gripper according to claim 1, characterized in that: The method of obtaining a desired target width of the remote manipulator end gripper according to the gripping width increment of the non-powered handheld gripper and the current state of the remote manipulator end gripper comprises: Range expect =Range cur +Range C-1～C Among them, Range expect is the desired target width of the remote robot end gripper, Range cur Range is the current width of the remote robot end gripper. C-1～C The latest clamping width increment for non-powered handheld grippers.

6. The teleoperated robotic arm system based on a non-electric handheld gripper according to claim 1, characterized in that: The step of sending the moving target point pose and the desired target width of the remote manipulator end gripper to the remote manipulator comprises: If Range expect ≤Range min or Range expect ≥Range max , then discard the latest desired target width Range of the remote robot end gripper expect , only the moving target point pose is sent to the remote manipulator; otherwise, the moving target point pose and the desired target width of the remote manipulator end gripper are sent to the remote manipulator, where Range min Range is the minimum gripping width of the remote robot arm end gripper. max The maximum gripping width of the remote robot arm's end gripper.

7. The teleoperated robotic arm system based on a non-electric handheld gripper according to claim 1, characterized in that: The non-powered handheld gripper includes a computing board, a sensor, a display, a handheld base, buttons, fingers, and a linkage; The computing board is used to provide computing resources; The sensor is used to obtain the position and gripping width of the non-powered handheld gripper; The display is used to display the remote image captured by the remote robotic arm; The handheld base is used as the handheld part of the non-electric handheld gripper; The button is used to provide power; The linkage device connects the button and the finger, and is used to convert the power provided by the button into controlling the closing / opening of the finger of the non-electric handheld gripper.

8. The teleoperated robotic arm system based on a non-electric handheld gripper according to claim 7, characterized in that: The linkage device includes a first connecting rod, a second connecting rod, a first gear, a second gear, a first pulley, a second pulley and a slide rail, and the finger includes a first finger and a second finger; The two sides of the button are respectively fixed to one end of the first connecting rod and one end of the second connecting rod, the other end of the first connecting rod is connected to the periphery of the first gear, and the other end of the second connecting rod is connected to the periphery of the second gear. The centers of the first gear and the second gear are rotatably mounted on the handheld base through axles, the teeth of the first gear are engaged with the teeth of the first pulley, and the teeth of the second gear are engaged with the teeth of the second pulley. The first pulley and the second pulley are both engaged with the slide rail, and the first pulley and the second pulley are arranged opposite to each other on the slide rail, the first pulley is fixed to the first finger, and the second pulley is fixed to the second finger.

Images