Automatic mechanical hand based on transformer model and mediapipe library and control method thereof

Abstract

The application discloses an automatic mechanical hand based on a Transformer model and a Mediapipe library and a control method thereof. The mechanical hand comprises a mechanical hand wrist, one end of the mechanical hand wrist is connected with a rotating seat, a mechanical hand palm is arranged on the rotating seat, five humanoid hand mechanical fingers are arranged on the mechanical hand palm, a camera is arranged on the rotating seat, and a power supply, a computer and a microprocessor are arranged on the mechanical hand palm. The target object is scanned through the camera, the pictures in the picture library in the computer are matched through the Transformer model, the corresponding optimal action combination is called through the function calling corresponding to the matched pictures, and the action combination is sent to the microprocessor. The microprocessor controls the finger module to grasp the object according to the received action combination, and the grasping process is monitored in real time through the Mediapipe, so that the target object is more stably grasped. The application has reasonable design, fast and accurate finger execution speed and low manufacturing cost.

Description

Technical Field

[0001] This invention relates to the field of robotic arm technology, and in particular to an automated robotic arm based on the Transformer model and the Mediapipe library, and its control method. Background Technology

[0002] For people with disabilities who have lost their hands due to various accidents, current prosthetic hands on the market are mostly decorative and lack the ability to automatically recognize and grasp objects, failing to meet their basic functional needs in daily life. Meanwhile, the cutting-edge robotic hands currently under development, capable of performing various actions according to human will, primarily involve surgically implanting chips in the cerebral cortex or installing specialized electromyography (EMG) sensors at the amputated limb to detect electrical impulses generated by human consciousness, programming these signals into computer language to control the robotic hand's movements. However, this technology is too advanced and too expensive, making large-scale commercialization difficult in the short term, especially for people with disabilities from low- and middle-income backgrounds. Therefore, designing a low-cost robotic hand that can automatically recognize and grasp objects to meet the basic functional needs of people with disabilities is of paramount importance. Summary of the Invention

[0003] This invention addresses the problem of high cost of existing prosthetic hands that can automatically identify and grasp objects to meet the basic functional needs of disabled people. It proposes an automatic robotic hand based on the Transformer model and the Mediapipe library, along with its control method.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] This invention proposes an automated robotic hand based on the Transformer model and the Mediapipe library, including a robotic wrist. One end of the robotic wrist is connected to a rotating base, and a robotic palm is provided on the rotating base. The robotic palm has five humanoid robotic fingers, the surface of which is covered with simulated skin material. A thumb mounting base is provided on the robotic palm, and a thumb crank slider is connected to the thumb mounting base. The thumb crank slider is connected to a thumb cylinder, and the telescopic end of the thumb cylinder is connected to a thumb module. An index finger mounting base is provided on the robotic palm, and an index finger module is connected to the index finger mounting base.

[0006] The rotating base is equipped with a camera, and the robotic hand is equipped with a power supply, a computer, and a microprocessor. The camera, computer, and microprocessor are all connected to the power supply line, and the camera and microprocessor are both connected to the computer line.

[0007] The computer is equipped with a Transformer model and a Mediapipe library; the computer stores an image library for storing photos of everyday items; the computer also stores an action call function library and an action call correction function library for different everyday items; the action call function is used to call the optimal action combination for different everyday items; the action call correction function is used to call the corrected action combination obtained through the Mediapipe library;

[0008] The microprocessor integrates a voice recognition module;

[0009] The system scans the target object with a camera to obtain an image. The computer then uses a Transformer model to match the target image with images in its image library. Based on the matched image, the system calls a function to determine the optimal action combination and sends this combination to the microprocessor. The microprocessor controls the finger module to grasp the object based on the action combination transmitted from the computer. The grasping process is monitored in real time using the Mediapipe library to ensure a more stable grasp of the target object. The system is well-designed, with fast and precise finger execution, and low manufacturing costs, making it affordable for low-income disabled individuals and more accessible to the general public.

[0010] Furthermore, the robotic hand is also equipped with a pressure receiver, which is connected to a power supply and computer circuitry. The thumb module and index finger module are both equipped with pressure sensors at their ends, and the pressure sensor circuitry is connected to the pressure receiver.

[0011] Furthermore, the thumb module includes a fixed block rotatably connected to the telescopic end of the thumb cylinder, a first thumb joint, and a second thumb joint. A pressure sensor is provided on the first thumb joint. A thumb bending cylinder is provided on the outside of the fixed block, and the telescopic end of the thumb bending cylinder is connected to the thumb rotating block.

[0012] Furthermore, the thumb rotating block has an anchor-shaped structure, and is rotatably connected to the fixed block, the thumb bending cylinder, and the second thumb joint; the second thumb joint is rotatably connected to the first thumb joint.

[0013] Furthermore, a rotating block is provided on the fixed block, which can rotate therewith; the rotating block is rotatably connected to a hinge provided on the thumb rotating block. The bending of the thumb is controlled by two cylinders, resulting in a simple structure and low production cost.

[0014] Furthermore, the index finger module includes a motor mounted on an index finger mounting base. A screw is connected to the output end of the motor, and a slider is mounted on the screw. First drive rods are rotatably connected to both sides of the slider. A connector is located at one end of the index finger mounting base, and fixing plates are connected to both sides of the connector. A first index finger joint is rotatably connected to the fixing plates. The index finger module controls the grasping action via a motor, resulting in low production costs.

[0015] Furthermore, a triangular connecting plate is rotatably connected to the first drive rod and the first index finger joint on the same side of the connector. The apex of the triangular connecting plate is connected to the first index finger joint. One of the bases of the triangular connecting plate is connected to the first drive rod, and the other base is rotatably connected to the second drive rod.

[0016] Furthermore, one end of the first index finger joint is provided with two connecting parts. A third driving rod is rotatably connected between the two first index finger joints at the lower connecting parts, and a second index finger joint is rotatably connected to the outer side of the upper connecting parts. One end of the third driving rod is provided with two connecting parts. The lower connecting part of the third driving rod is rotatably connected to the first index finger joint, and the upper connecting part is rotatably connected to the second driving rod.

[0017] Furthermore, a third index finger joint is rotatably connected to the inner side of each of the two second index finger joints. Two connecting parts are provided at one end of each of the two third index finger joints. The outer side of the connecting part at the bottom of the two third index finger joints is connected to the second index finger joint, and the connecting part at the top is rotatably connected to one end of the third drive rod.

[0018] Furthermore, a fingertip protrusion is provided below the third index finger joint, and a connecting block is provided on the fingertip protrusion. The connecting block is located between the two third index finger joints and is connected to the third index finger joint. A pressure sensor is provided on the fingertip protrusion.

[0019] Furthermore, the robotic hand, located away from the wrist end, is sequentially equipped with index finger, middle finger, ring finger, and little finger mounting seats according to the position of the human fingers. Each of these mounting seats has an index finger module, a middle finger module, a ring finger module, and a little finger module, respectively. The middle finger, ring finger, and little finger modules have the same structure as the index finger module. Each of these modules can be individually controlled for grasping actions, resulting in greater flexibility.

[0020] Furthermore, the computer also stores pressure data on each mechanical finger corresponding to the optimal combination of movements for different daily necessities.

[0021] Another aspect of the present invention proposes a method for controlling the automated robotic arm based on the Transformer model and the Mediapipe library, comprising:

[0022] The camera automatically scans the target object to form a target image. The computer uses the Transformer model to match the target image with images in the computer's image library to obtain a matching image. The corresponding action call function is then used to call the best action combination based on the matching image, and the action combination is sent to the microprocessor.

[0023] After the robotic arm extends towards the target object and reaches the appropriate grasping position, the starting voice command is spoken. After receiving the command, the voice recognition module on the microprocessor begins to allow the microprocessor to download the action combination issued by the computer. The robotic arm then completes the grasping action according to the action combination.

[0024] During the process of the robotic arm completing the grasping action, the computer monitors the movement of the robotic arm in real time through Mediapipe. Based on the coordinate data of the real-time hand detection points generated by Mediapipe, the corresponding action call correction function calls the corresponding correction action combination and sends the correction action combination to the microprocessor. The robotic arm adjusts the grasping action based on the correction action combination to grasp the target object more stably.

[0025] Once the user has grasped the target object and completed their task, they can give a voice command to end the action. The voice recognition module on the microprocessor will then receive the command and cause the robotic arm to release the target object, returning it to its initial state, thus ending the entire process.

[0026] Furthermore, it also includes: detecting real-time pressure data on each mechanical finger module through pressure sensors on each mechanical finger module, and adjusting the rotation amplitude of each motor based on the pressure data to increase the pressure value to the corresponding recorded data on the computer.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] This invention uses a camera to scan a target object, obtaining an image. A computer then uses a Transformer model to match the target image with images in its image library. Based on the matched image, a corresponding action function is invoked to call the optimal action combination, which is then sent to a microprocessor. The microprocessor controls the finger modules to grasp the object based on the action combination transmitted from the computer, and the grasping process is monitored in real time via Mediapipe for a more stable grasp. This invention also incorporates pressure sensors on each finger module to detect real-time pressure data and adjust the rotation amplitude of each motor based on this data. This reduces the impact of latex aging and other factors on the pressure data of each finger module, resulting in a more appropriate grip. This invention is rationally designed, offers fast and precise finger execution, and has low manufacturing costs, making it affordable even for low-income disabled individuals, thus making it more accessible to the general public. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of an automated robotic arm based on the Transformer model and the Mediapipe library according to an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of the index finger module in an embodiment of the present invention.

[0031] The attached diagram is labeled as follows: 1 is the mechanical wrist, 2 is the rotating base, 3 is the mechanical hand, 4 is the thumb mounting base, 5 is the index finger module, 51 is the motor, 52 is the screw, 53 is the slider, 54 is the connector, 55 is the fixing plate, 56 is the first index finger joint, 57 is the triangular connecting plate, 58 is the first drive rod, 59 is the second drive rod, 60 is the third drive rod, 61 is the second index finger joint, 62 is the third index finger joint, 63 is the fingertip protrusion, 6 is the thumb crank slider, 7 is the thumb cylinder, 8 is the thumb module, 81 is the first thumb joint, 82 is the second thumb joint, 83 is the thumb rotating block, 85 is the thumb bending cylinder, 9 is the index finger mounting base, 10 is the camera, 11 is the power supply, 12 is the computer, 13 is the microprocessor, 14 is the pressure receiver, 15 is the fixing block, 16 is the hinge, 17 is the rotating shaft, and 18 is the pressure sensor. Detailed Implementation

[0032] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0033] like Figure 1As shown, an automated robotic hand based on the Transformer model and the Mediapipe library includes a robotic wrist 1, one end of which is connected to a rotating base 2. A robotic palm 3 is mounted on the rotating base 2. The robotic palm 3 has five humanoid robotic fingers. A thumb mounting base 4 is mounted on the robotic palm 3. A thumb crank slider 6 is connected to the thumb mounting base 4. A thumb cylinder 7 is connected to the thumb crank slider 6. A thumb module 8 is connected to the telescopic end of the thumb cylinder 7. An index finger mounting base 9 is mounted on the robotic palm 3. An index finger module 5 is connected to the index finger mounting base 9.

[0034] The rotating base 2 is equipped with a camera 10, and the robotic hand 3 is equipped with a power supply 11, a computer 12 and a microprocessor 13. The camera 10, computer 12 and microprocessor 13 are all connected to the power supply 11, and the camera 10 and microprocessor 13 are all connected to the computer 12.

[0035] The computer 12 is equipped with a Transformer model and a Mediapipe library; the computer 12 stores an image library for storing photos of everyday items; the computer 12 also stores an action call function library and an action call correction function library for different everyday items; the action call function is used to call the optimal action combination for different everyday items; the action call correction function is used to call the corrected action combination obtained through the Mediapipe library;

[0036] The microprocessor 13 integrates a voice recognition module.

[0037] The computer 12 scans the target object using camera 10 and matches the scanned object with images in an image library. It then calls the optimal action combination based on the action function corresponding to the matched image and sends this action combination to the microprocessor 13. The microprocessor 13 controls the finger module to grasp the object based on the action combination transmitted by the computer 12, and monitors the grasping process in real time via Mediapipe to ensure a more stable grasp of the target object. In one possible implementation, the robotic arm is wrapped in latex.

[0038] Furthermore, the mechanical hand 3 is also provided with a pressure receiver 14, which is connected to the power supply 11 and the computer 12. The thumb module 8 and the index finger module 5 are both provided with pressure sensors 18, which are connected to the pressure receiver 14.

[0039] The thumb module 8 includes a fixed block 15 rotatably connected to the telescopic end of the thumb cylinder 7, a first thumb joint 81, and a second thumb joint 82. A pressure sensor 18 is provided on the first thumb joint 81. A thumb bending cylinder 85 is located on the outer side of the fixed block 15, and the telescopic end of the thumb bending cylinder 85 is connected to a thumb rotating block 83. The thumb rotating block 83 has an anchor-shaped structure and is rotatably connected to the fixed block 15, the thumb bending cylinder 85, and the second thumb joint 82; the second thumb joint 82 is rotatably connected to the first thumb joint 81. A rotating block is provided on the fixed block 15, which can rotate therewith; the rotating block is rotatably connected to a hinge 16 provided on the thumb rotating block 83. The thumb module 8 is driven and controlled by the coordinated action of the thumb cylinder 7 and the thumb bending cylinder 85, enabling the thumb module 8 to perform a grasping action on an object.

[0040] like Figure 2 The index finger module 5 includes a motor 51 mounted on an index finger mounting base 9. A screw 52 is connected to the output end of the motor 51. A slider 53 is mounted on the screw 52. First drive rods 58 are rotatably connected to both sides of the slider 53. A connector 54 is provided at one end of the index finger mounting base 9. Fixing plates 55 are connected to both sides of the connector 54. A first index finger joint 56 is rotatably connected to the fixing plates 55. A triangular connecting plate 57 is rotatably connected to the first drive rod 58 and the first index finger joint 56 on the same side as the connector 54. The apex of the triangular connecting plate 57 is connected to the first index finger joint 56. One base of the triangular connecting plate 57 is connected to the first drive rod 58, and the other base is rotatably connected to a second drive rod 59. The first index finger joint 56 has two connecting parts at one end. A third drive rod 60 is rotatably connected between the two first index finger joints 56 at the lower connecting part, and a second index finger joint 61 is rotatably connected to the outer side of the upper connecting part. The third drive rod 60 has two connecting parts at one end. The lower connecting part of the third drive rod 60 is rotatably connected to the first index finger joint 56, and the upper connecting part is rotatably connected to the second drive rod 59. A third index finger joint 62 is rotatably connected to the inner side of each of the two second index finger joints 61. Each of the two third index finger joints 62 has two connecting parts at one end. The outer side of the lower connecting part of the two third index finger joints 62 is connected to the second index finger joint 61, and the upper connecting part is rotatably connected to one end of the third drive rod 60.

[0041] The index finger module 5 is driven by the screw to rotate via the motor 51, causing the slider 53 to move on the screw. As the slider moves toward the motor 51, the first drive rod 58 pulls the triangular connecting plate 57, causing the triangular connecting plate 57 to rotate about the connection part with the first index finger joint 56. The triangular connecting plate 57 pushes the third drive rod 60 via the second drive rod 59, causing the connection part of the first index finger joint 56 to rotate inward in a gripping motion. The second and third drive rods 60 rotate about the connection part with the first index finger joint 56, causing the connection part of the second index finger joint 61 to rotate. The second index finger joint 61 pulls the third index finger joint 62, causing the third index finger joint 62 to rotate about the connection part with the third drive rod 60. As the slider 52 moves, the bending amplitude increases, allowing the index finger module 5 to complete the gripping action.

[0042] A fingertip protrusion 63 is provided below the third index finger joint 62. A connecting block is provided on the fingertip protrusion 63. The connecting block is located between the two third index finger joints 62 and connected to the third index finger joints 62. A pressure sensor 18 is provided on the fingertip protrusion 63. The pressure sensor 18 detects the pressure generated by the fingertip protrusion 63 contacting an object. When the pressure reaches a certain value, it will stop the motor 51.

[0043] The mechanical hand 3, located away from the mechanical wrist 1, is provided with an index finger mounting base 9, a middle finger mounting base, a ring finger mounting base, and a little finger mounting base in sequence according to the position of human fingers. The index finger mounting base 9, the middle finger mounting base, the ring finger mounting base, and the little finger mounting base are respectively provided with an index finger module 5, a middle finger module, a ring finger module, and a little finger module. The structure of the middle finger module, the ring finger module, and the little finger module is the same as the structure of the index finger module 5.

[0044] Furthermore, the computer 12 also stores pressure data on each mechanical finger corresponding to the optimal combination of movements for different daily necessities.

[0045] It is worth noting that the robotic hand in this embodiment is suitable for the following groups of people: disabled persons who only have hand defects, whose consciousness and cognitive abilities and the functions of various parts of their bodies are no different from those of ordinary people. After being equipped with this robotic hand, they can quickly and accurately bring the robotic hand close to the target object and place the robotic hand in a suitable spatial position to grasp the object through self-awareness.

[0046] Based on the above embodiments, the present invention also proposes a method for controlling the above-mentioned automated robotic arm based on the Transformer model and the Mediapipe library, comprising:

[0047] The camera 10 automatically scans the target object to form a target image. The computer 12 uses the Transformer model to match the target image with the images in the image library of the computer 12 to obtain a matching image. The corresponding action call function is called to call the best action combination based on the action corresponding to the matching image, and the action combination is sent to the microprocessor 13.

[0048] After the robotic arm extends towards the target object and reaches the appropriate grasping position, it speaks the starting voice command. After receiving the command, the voice recognition module on the microprocessor 13 begins to allow the microprocessor 13 to download the action combination issued by the computer 12. The robotic arm completes the grasping action according to the action combination.

[0049] During the process of the robotic arm completing the grasping action, the computer 12 monitors the movement of the robotic arm in real time through Mediapipe. Based on the coordinate data of the real-time hand detection points generated by Mediapipe, the corresponding action call correction function is called to call the corresponding correction action combination, and the correction action combination is sent to the microprocessor 13. The robotic arm adjusts the grasping action based on the correction action combination to grasp the target object more stably.

[0050] Once the user has grasped the target object and completed their task, they can give a voice command to end the action. The voice recognition module on the microprocessor 13 will receive the command and cause the robotic arm to release the target object and return to its initial state, thus ending the entire process.

[0051] As one possible implementation method, the following preparatory work needs to be completed before performing the above process:

[0052] A large number of commonly used everyday items need to be prepared in advance, and pictures of them (such as various tableware, various foods, various tools, etc.) should be taken to form an image library (assuming the images are named picture1, picture2, ..., picture**). This library serves as the basis for the Transformer model to compare the target objects captured by camera 10. Based on the Transformer model, the target objects captured by camera 10 are compared. Parameters are set to tolerate a certain range of errors. For example, when the target object basically matches the image picture1 in the image library, the program is judged as true and the loop ends; when the target object does not match the image in the image library, the program is judged as false, and the comparison continues with the next image picture2 in the image library until the result is true and the loop ends.

[0053] For the different shapes of objects in the image library, the robotic arm has an optimal action for grasping them. This involves each motor 51 rotating in different sequences, at different time intervals, and with different amplitudes, forming a combination of actions. These parameters are recorded, and various robotic arm action call programs are written to form a custom action call function library. Let's assume the various action call programs in this library are named handaction1(), handaction2(), ..., handaction**(), etc. The significance of this naming is that the action combination represented by handaction1() is most suitable for grasping the object represented by picture1, the action combination represented by handaction2() is most suitable for grasping the object represented by picture2, and so on.

[0054] Write a program for the speech recognition module. When the speech recognition module receives a start command (such as the sound "start"), the microprocessor 13 will receive external command input. When the speech recognition module receives an end command (such as the sound "end"), the microprocessor 13 will issue the initialization command stored in the register to return the robot to its initial state (the state of five fingers outstretched).

[0055] The Mediapipe library is used to monitor the actual operation of a robotic arm. The gesture recognition function in this library can divide a human hand-like object (robotic arm) into 21 points. When the hand makes any movement, the changes in the coordinates of these 21 points in the video image are recorded at any given moment, thus monitoring the hand's movements. For example, when the program executes `handaction1()`, the robotic arm performs a movement but does not fully grasp the target object. In this case, the gesture recognition function in the Mediapipe library records the data changes of the 21 coordinate points of the robotic arm in the video image. After extensive experimentation to form a database, various situations can be recorded where the robotic arm does not fully grasp the object after performing a movement. For example, we name the data combination for each situation as `condition1`, `condition2`, ..., `condition**`, etc., and adjust the parameters of each motor 51 of the robotic arm to ensure full grasp of the target object each time this situation occurs. The adjusted parameters of each motor 51 are then written into new custom functions and added to the action call correction function library, for example, named `re_handaction1`, `re_handaction2`, ..., `re_handaction**`, etc.

[0056] As one possible implementation, the user uses the camera 10 to scan the object to be captured to form a target image. The computer 12 uses the Transformer model to match the target image with the images in the computer's image library to obtain a matching image picture1. The corresponding action call function handaction1() is used to call the corresponding best action combination and send the action combination to the microprocessor 13.

[0057] After the robotic arm extends towards the target object and reaches the appropriate grasping position, it utters the initial voice command. The voice recognition module on the microprocessor 13 receives the command and begins downloading the action (command) combination corresponding to `handaction1()` from the computer 12. The robotic arm's finger modules grasp the object by rotating in time, sequence, and amplitude according to the action combination (pre-set parameters). The thumb module 8 is driven and controlled by the thumb cylinder 7 and the thumb bending cylinder 85 to perform the grasping action. The index finger module 5 drives the screw to rotate via the motor 51, causing the slider 53 to move on the screw. As the slider moves towards the motor 51, the first drive rod 58 pulls... The triangular connecting plate 57 is moved so that it rotates about the connection part with the first index finger joint 56. The triangular connecting plate 57 pushes the third driving rod 60 through the second driving rod 59, so that the connection part of the first index finger joint 56 rotates inward to form a gripping tendency. The second and third driving rods 60 rotate about the connection part with the first index finger joint 56, so that the connection part of the second index finger joint 61 rotates. The second index finger joint 61 pulls the third index finger joint 62, so that the third index finger joint 62 rotates about the connection part with the third driving rod 60. As the slider 52 moves, the bending amplitude increases, so that the index finger module 5 completes the gripping action. The movement principle of the middle finger module, ring finger module and little finger module is the same as that of the index finger module 5, until each finger module completes all the actions in the action combination.

[0058] During the robotic arm's grasping action, computer 12 monitors the robotic arm's movement in real time via Mediapipe. Because the shape and volume of the target object (the program already sets a certain error range during image comparison; otherwise, the system might not be able to effectively identify the target object, causing the program to remain stuck in the recognition loop) may actually differ from picture1, the robotic arm's initial grasping action might not be sufficient to firmly grip the target object. At this point, the coordinate data of the 21 hand detection points generated by Mediapipe might match condition1 described in section 1.3 above. Therefore, the program in the minicomputer calls re_handaction1 from the action program function library again and inputs it to the MCU, causing the MCU to re-execute the new action re_handaction1. In this way, the robotic arm adjusts its movement to more firmly grip the target object.

[0059] Based on the coordinate data of the real-time hand detection points generated by Mediapipe detection, the corresponding action call correction function is called to call the corresponding correction action combination, and the correction action combination is sent to the microprocessor 13. The robotic arm adjusts the grasping action based on the correction action combination to grasp the target object more stably.

[0060] Once the user has grasped the target object and completed their task, they can give a voice command to end the action. The voice recognition module on the microprocessor 13 will receive the command and cause the robotic arm to release the target object and return to its initial state (with fingers outstretched), thus ending the entire action.

[0061] Furthermore, in actual use, due to the aging of the latex material covering the surface of the robotic arm, it may be possible to call handaction1 when recognizing the same target object. However, at this time, the pressure of each finger of the robotic arm on the target object will be less than the pressure data recorded in the computer 12. Therefore, it is necessary to detect the real-time pressure data on each robotic finger module through the pressure sensor 18 on each robotic finger module, and adjust the rotation amplitude of each motor 51 based on the pressure data to increase the pressure value to the corresponding recorded data on the computer 12.

[0062] In summary, this invention uses camera 10 to scan the target object and obtain a target image. Computer 12 uses a Transformer model to match the target image with images in its image library. The corresponding action function is called based on the matched image to retrieve the optimal action combination, which is then sent to microprocessor 13. Microprocessor 13 controls the finger modules to grasp the object based on the action combination transmitted from computer 12, and the grasping process is monitored in real time via Mediapipe for a more stable grasp. This invention also uses pressure sensors 18 on each finger module to detect real-time pressure data and adjust the rotation amplitude of each motor 51 based on the pressure data. This reduces the impact of latex aging and other factors on the pressure data of each finger module, resulting in a more appropriate gripping force. This invention is reasonably designed, with fast and precise finger execution, and low manufacturing cost, making it affordable even for low-income disabled individuals, thus making it more accessible to the general public.

[0063] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An automated robotic arm based on the Transformer model and the Mediapipe library, characterized in that, The device includes a mechanical wrist (1), one end of which is connected to a rotating base (2). The rotating base (2) is provided with a mechanical hand (3), and the mechanical hand (3) is provided with five humanoid mechanical fingers. The mechanical hand (3) is provided with a thumb mounting base (4), and a thumb crank slider (6) is connected to the thumb mounting base (4). The thumb crank slider (6) is connected to a thumb cylinder (7), and the telescopic end of the thumb cylinder (7) is connected to a thumb module (8). The mechanical hand (3) is provided with an index finger mounting base (9), and an index finger module (5) is connected to the index finger mounting base (9). The index finger module (5) includes a motor (51) mounted on an index finger mounting base (9). The output end of the motor (51) is connected to a screw (52). A slider (53) is provided on the screw (52). A first drive rod (58) is rotatably connected to both sides of the slider (53). A connector (54) is provided at one end of the index finger mounting base (9). A fixing plate (55) is connected to both sides of the connector (54). A first index finger joint (56) is rotatably connected to the fixing plate (55). A triangular connecting plate (57) is rotatably connected to the first drive rod (58) and the first index finger joint (56) on the same side as the connector (54). The apex of the triangular connecting plate (57) is connected to the first index finger joint (56). One foot of the triangular connecting plate (57) is connected to the first drive rod (58), and the other foot is rotatably connected to a second drive rod (59). The rotating base (2) is equipped with a camera (10), and the mechanical hand (3) is equipped with a power supply (11), a computer (12), and a microprocessor (13). The camera (10), computer (12), and microprocessor (13) are all connected to the power supply (11) line, and the camera (10) and microprocessor (13) are all connected to the computer (12) line. The computer (12) is equipped with a Transformer model and a Mediapipe library; the computer (12) stores an image library for storing photos of everyday items; the computer (12) also stores an action call function library and an action call correction function library for different everyday items; the action call function is used to call the best action combination for different everyday items; the action call correction function is used to call the corrected action combination obtained through the Mediapipe library; The microprocessor (13) integrates a speech recognition module; The computer (12) is configured to: match the target image captured by the camera (10) with the image library through the Transformer model, and call the corresponding action call function through the matching result to generate a first control instruction to control the robot to perform the best action combination; at the same time, monitor the grasping process of the robot in real time through the Mediapipe library, and call the action call correction function library based on the monitoring data to generate a second control instruction to adjust the action of the robot in real time.

2. The automated robotic arm based on the Transformer model and Mediapipe library according to claim 1, characterized in that, The mechanical hand (3) is also provided with a pressure receiver (14), which is connected to the power supply (11) and computer (12). The thumb module (8) and index finger module (5) are both provided with pressure sensors (18), which are connected to the pressure receiver (14).

3. The automated robotic arm based on the Transformer model and Mediapipe library according to claim 1, characterized in that, The thumb module (8) includes a fixed block (15) rotatably connected to the telescopic end of the thumb cylinder (7), a first thumb joint (81) and a second thumb joint (82). A pressure sensor (18) is provided on the first thumb joint (81). A thumb bending cylinder (85) is provided on the outside of the fixed block (15). The telescopic end of the thumb bending cylinder (85) is connected to the thumb rotating block (83). The structure of the thumb rotating block (83) is anchor-shaped. The thumb rotating block (83) is rotatably connected to the fixed block (15), rotatably connected to the thumb bending cylinder (85), and rotatably connected to the second thumb joint (82); the second thumb joint (82) is rotatably connected to the first thumb joint (81). The fixed block (15) is provided with a rotating block that can rotate therewith, and the rotating block is rotatably connected to the hinge (16) provided on the thumb rotating block (83).

4. The automated robotic arm based on the Transformer model and Mediapipe library according to claim 1, characterized in that, The first index finger joint (56) has two connecting parts at one end. The two first index finger joints (56) are rotatably connected between the lower connecting parts and the second index finger joint (61) is rotatably connected to the outer side of the upper connecting parts. The third driving rod (60) has two connecting parts at one end. The lower connecting part of the third driving rod (60) is rotatably connected to the first index finger joint (56) and the upper connecting part is rotatably connected to the second driving rod (59).

5. The automated robotic arm based on the Transformer model and Mediapipe library according to claim 4, characterized in that, The inner sides of the two second index finger joints (61) are rotatably connected to the third index finger joints (62). The two third index finger joints (62) have two connecting parts at one end. The two third index finger joints (62) are connected to the second index finger joints (61) on the outside of the lower connecting part, and rotatably connected to one end of the third drive rod (60) between the upper connecting parts.

6. The automated robotic arm based on the Transformer model and Mediapipe library according to claim 2, characterized in that, The mechanical hand (3) is provided with an index finger mounting seat (9), a middle finger mounting seat, a ring finger mounting seat and a little finger mounting seat in sequence according to the position of human fingers at the end away from the mechanical wrist (1). The index finger mounting seat (9), the middle finger mounting seat, the ring finger mounting seat and the little finger mounting seat are respectively provided with an index finger module (5), a middle finger module, a ring finger module and a little finger module. The structure of the middle finger module, the ring finger module and the little finger module is the same as the structure of the index finger module (5).

7. The automated robotic arm based on the Transformer model and Mediapipe library according to claim 6, characterized in that, The computer (12) also stores pressure data on each mechanical finger corresponding to the optimal action combination of different daily necessities.

8. A method for controlling an automated robotic arm based on the Transformer model and Mediapipe library as described in any one of claims 1-7, characterized in that, include: The camera (10) automatically scans the target object to form a target image. The computer (12) uses the Transformer model to match the target image with the image in the image library of the computer (12) to obtain a matching image. The corresponding action call function is called to call the best action combination and the action combination is sent to the microprocessor (13). After the robotic arm extends towards the target object and reaches the appropriate grasping position, it speaks the starting voice command. After receiving the command, the voice recognition module on the microprocessor (13) begins to allow the microprocessor (13) to download the action combination issued by the computer (12). The robotic arm completes the grasping action according to the action combination. During the process of the robotic arm completing the grasping action, the computer (12) monitors the movement of the robotic arm in real time through Mediapipe. Based on the coordinate data of the real-time hand detection points generated by Mediapipe, the corresponding action call correction function is called to call the corresponding correction action combination, and the correction action combination is sent to the microprocessor (13). The robotic arm adjusts the grasping action based on the correction action combination to grasp the target object more stably. When the user grasps the target object and completes their purpose, they say the voice command to end. The voice recognition module on the microprocessor (13) receives the command and causes the robotic arm to release the target object and return to the initial state, thus ending the entire action.

9. The method according to claim 8, characterized in that, Also includes: The pressure sensor (18) on each mechanical finger module detects the real-time pressure data on each mechanical finger module, and adjusts the rotation amplitude of each motor (51) based on the pressure data to increase the pressure value to the corresponding recorded data on the computer (12).

Images