Fusion sensor system for under-actuated five-finger simulated hand

Abstract

The invention relates to the field of automation control, in particular to a fusion sensor system for an under-actuated five-finger simulated hand. Comprising an under-actuated five-finger imitated hand, the under-actuated five-finger imitated hand comprises a wrist and a palm rotationally connected to the wrist, the palm is provided with a plurality of fingers capable of completing grabbing actions, a sensor array plate is installed at the palm center of the palm, and a visual perception plate is arranged on the lower side of the sensor array plate; a lens of the visual perception plate collects visual information through a center hole of the sensor array plate; the sensor array plate is a flexible circuit board, the sensor array plate is composed of a 5 * 5 proximity sensor array, and the proximity sensor is used for measuring the proximity distance of an object based on the reflection principle; and the actual requirements of the under-actuated five-finger simulated hand on environment control and self-adaptive grabbing can be met.

Description

Technical Field

[0001] The present invention relates to the field of automatic control, and more specifically to a fusion sensor system for an underactuated five-finger humanoid hand. Background Art

[0002] The underactuated five-finger humanoid hand is a robotic hand design that mimics the structure and function of the human hand. Its core feature is the "underactuated" mechanism; underactuation means that the number of actuators is less than the degrees of freedom. Through ingenious design, the passive adaptability of the mechanical structure is used to achieve complex grasping and operating tasks. However, in the actual use of the underactuated five-finger humanoid hand, problems such as limited vision caused by occlusion faced by the global camera during the use of traditional humanoid robotic hands, and the inability to perform real-time closed-loop control during the grasping stage in traditional visual closed-loop control strategies, and various problems caused by the loss of information due to occlusion of the camera position. Summary of the Invention

[0003] The purpose of the present invention is to provide a fusion sensor system for an underactuated five-finger humanoid hand, which can meet the actual needs of the underactuated five-finger humanoid hand in environmental control and adaptive grasping.

[0004] The purpose of the present invention is achieved through the following technical solutions:

[0005] A fusion sensor system for an underactuated five-finger humanoid hand includes an underactuated five-finger humanoid hand. The underactuated five-finger humanoid hand includes a wrist and a palm rotatably connected to the wrist. A plurality of fingers capable of performing grasping actions are provided on the palm. A sensor array board is installed at the center of the palm. A visual perception board is provided on the lower side of the sensor array board. The lens of the visual perception board collects visual information through the central hole of the sensor array board.

[0006] The sensor array board is a flexible circuit board and is composed of a 5*5 proximity sensor array. The proximity sensor measures the approaching distance of an object based on the reflection principle.

[0007] The plurality of fingers are respectively the little finger, the ring finger, the middle finger, the index finger and the thumb. A plurality of direct current brushless motors for driving the little finger, the ring finger, the middle finger, the index finger and the thumb to move are provided on the palm.

[0008] A main control board is installed on the palm. The direct current brushless motors and the sensor array board are driven and controlled and sensed through the main control board. The sensor array board and the main control board are connected by an FPC flexible line and communicate in an I2C manner.

[0009] The visual perception board transmits the collected visual information to the main control board through wireless Bluetooth after processing.

[0010] The main control board integrates a microcontroller unit (MCU) for processing the raw data of the sensor array board, communicating with the host computer, and controlling the movement of multiple fingers. The main control board receives the output result of the visual feature extraction after processing the visual information transmitted via Bluetooth.

[0011] When the underactuated five-finger anthropomorphic hand is far from the object, end-side reasoning is performed based on the visual perception board. The original image data collected by the input lens, which is a three-channel RGB image containing the target object and the surrounding environment information, is processed by the offline visual perception board on the end side to extract the original image features and perform classification and aggregation, obtaining the semantic perception classification information of the target object and the surrounding environment and the relative position relationship in the pixel coordinate system compared to the palm. The corresponding processing results are transmitted to the main control board via Bluetooth, optimized, and then transmitted to the host computer via the Universal Asynchronous Receiver / Transmitter (UART).

[0012] When the palm reaches the minimum detection distance threshold of the sensor array board, based on the features obtained from the first-stage visual information processing of the visual perception board, the depth information of each point of the array is obtained by interpolation using a fitting function based on the raw data of the sensor array board. The morphological processing and fitting are performed on the obtained dot matrix depth information to obtain the surface contour features collected by the dot matrix. The visual features of the visual perception board are transmitted via Bluetooth, and the sensor array board is transmitted via the Flexible Printed Circuit (FPC) cable in I2C communication mode. The two are fused at the feature level on the main control board, and the final output result reflects the relative pose relationship with details between the underactuated five-finger anthropomorphic hand and the target object, which is transmitted to the host computer via the Universal Asynchronous Receiver / Transmitter (UART).

[0013] When the underactuated five-finger anthropomorphic hand is close to the object, the visual information density of the visual perception board is significantly reduced due to the short distance. At this time, the various processed features obtained from the visual information processing are no longer significant, and the raw data of the sensor array board is more sensitive to the distance change at close range. When the visual information cannot obtain sufficient processed features, the real-time visual information no longer plays a role. At this time, the surface contour features obtained from the dot matrix depth information are mainly compared with the features perceived in the first stage. The two are matched and approximated on the main control board, and the final output result reflects the relative pose relationship with details between the underactuated five-finger anthropomorphic hand and the target object, which is transmitted to the host computer via the Universal Asynchronous Receiver / Transmitter (UART).

[0014] The palm is a hollow structure. The sensor array board is fixedly connected to the outer shell of the palm through the positioning holes, and the main control board is fixed to the outer shell of the back of the palm through the positioning holes.

[0015] The beneficial effects of the present invention are:

[0016] According to the actual requirements of an underactuated five-fingered humanoid hand in environmental control and adaptive grasping, based on the principle of mechatronics design, an underactuated five-fingered humanoid hand with a highly sensitive integrated vision-proximity sensor array system is designed.

[0017] The sensing system involved in the present invention includes infrared proximity sensors, a vision perception board, and a main control board arranged at equal intervals of 5×5; the vision information of this system can effectively improve the discrimination and perception ability of the humanoid mechatronic prosthetic hand for target objects during grasping in a complex environment through end-side information processing. The proximity information makes up for the deficiency of low information density of vision information at close range, improves the perception ability of the underactuated five-fingered humanoid hand for the grasping space, and improves the operating performance of the underactuated five-fingered humanoid hand.

[0018] The sensor information processing circuit involved in the present invention is composed of a sensor array board, a vision perception board, and a main control board. Among them, the sensor array board is a flexible circuit board, which is installed on the palm and contains a 5×5 proximity sensor array. The middle is hollowed out to place the lens of the vision perception board. The 25 sensors are controlled by a multiplexing chip for data transmission in the flexible board, and high-coverage depth data acquisition is realized within the limited resource space and installation space of the embedded chip in the hand. The collected data is connected to the information processing integrated circuit board of the underactuated five-fingered humanoid hand through an FPC flexible line in an I2C communication mode. The vision information acquisition board is placed under the flexible circuit board, and image data acquisition and processing are realized in the board, and it is connected to the control board of the underactuated five-fingered humanoid hand through BLE Bluetooth, effectively alleviating the information processing pressure of the main control board and improving the efficiency of the overall control system.

[0019] The sensor control system involved in the present invention can obtain the relative pose relationship between the target object and the underactuated five-fingered humanoid hand through vision information processing at a long distance, effectively solving the problems of limited vision such as occlusion faced by the global camera during the use of the humanoid manipulator, improving the intelligent level of the control of the underactuated five-fingered humanoid hand, simplifying the complex coordinate transformation involved in the traditional vision scheme, enriching the information level of the traditional in-hand sensors, and improving the control performance of the underactuated five-fingered humanoid hand. At close range, the depth information of the palm area is obtained through the proximity sensor array to achieve closed-loop control for the grasping target. Compared with the traditional vision closed-loop control strategy, which faces various problems such as the inability to perform real-time closed-loop control during the grasping stage and the loss of information due to occlusion by the camera position, the grasping performance of the underactuated five-fingered humanoid hand is effectively improved, the application scenario of the underactuated five-fingered humanoid hand is broadened, and the reliability of the mechatronic system is improved. Description of the Drawings

[0020] The present invention will be further described in detail below with reference to the drawings and specific implementation methods.

[0021] Figure 1It is a schematic structural diagram of the fusion sensor system for the underactuated five-finger humanoid hand of the present invention;

[0022] Figure 2 It is a side view of the fusion sensor system for the underactuated five-finger humanoid hand of the present invention;

[0023] Figure 3 It is a schematic structural diagram of the sensor array board of the present invention;

[0024] Figure 4 It is a framework diagram of the vision-dominated stage algorithm of the present invention;

[0025] Figure 5 It is a framework diagram of the array fusion stage algorithm of the present invention;

[0026] Figure 6 It is a framework diagram of the array stage algorithm of the present invention.

[0027] In the figure: little finger 1; ring finger 2; middle finger 3; index finger 4; DC brushless motor 5; main control board 6; sensor array board 7; vision perception board 8; thumb 9; palm 10; wrist 11. Detailed implementation manners

[0028] The present invention will be further described in detail below with reference to the accompanying drawings.

[0029] As Figures 1 to 6 shown, in order to achieve the technical effect of "meeting the actual requirements of the underactuated five-finger humanoid hand in environmental control and adaptive grasping", the structure and function of a fusion sensor system for the underactuated five-finger humanoid hand will be described in detail below;

[0030] As Figure 1 and Figure 2 shown; a fusion sensor system for the underactuated five-finger humanoid hand includes an underactuated five-finger humanoid hand, and the underactuated five-finger humanoid hand includes a wrist 11 and a palm 10 rotatably connected to the wrist 11. A plurality of fingers capable of performing grasping actions are provided on the palm 10. A sensor array board 7 is installed at the center of the palm 10. A vision perception board 8 is arranged below the sensor array board 7. The lens of the vision perception board 8 collects visual information through the central hole of the sensor array board 7; the outer shell of the palm 10 is designed as a hollow structure according to the configuration of the sensor array board 7. The hollow structure can reduce the weight and reduce the signal interference of the signal acquisition circuit. The sensor array board 7 is fixedly connected to the outer shell of the palm 10 through positioning holes, and the main control board 6 is fixed to the outer shell on the back of the palm 10 through positioning holes;

[0031] Among them, the palm 10 is made of a composite material, preferably an organic polymer material, which reduces the weight of the overall system while meeting the stiffness;

[0032] The sensor array board 7 is a flexible circuit board, which consists of a 5*5 proximity sensor array. The proximity sensor measures the approaching distance of an object based on the reflection principle.

[0033] The multiple fingers are respectively the little finger 1, the ring finger 2, the middle finger 3, the index finger 4 and the thumb 9. A plurality of DC brushless motors 5 for driving the little finger 1, the ring finger 2, the middle finger 3, the index finger 4 and the thumb 9 to move are arranged on the palm 10. The finger body is fixed to the outer shell of the palm 10 and the outer shell of the back of the palm 10 through bolts.

[0034] A main control board 6 is installed on the palm 10. The DC brushless motor 5 and the sensor array board 7 are driven and controlled and sensed and collected through the main control board 6. The sensor array board 7 and the main control board 6 are connected by an FPC flexible line and communicate in an I2C manner.

[0035] The visual perception board 8 transmits the collected visual information to the main control board 6 through wireless Bluetooth after processing.

[0036] The main control board 6 is integrated with a micro control unit MCU to process the original data of the sensor array board 7, communicate with the host computer and control the movement of multiple fingers. The main control board 6 receives the output result of the visual feature extraction after processing the visual information transmitted by Bluetooth. A lot of sensor data such as current and Hall position sensors are set on the main control board 6. Based on the corresponding control strategy, the motor is driven to realize the intelligent grasping of the humanoid five-finger hand. The corresponding control instructions are transmitted to the working carrier such as a robotic arm through various methods such as CAN / UART, so as to provide necessary control information for the closed-loop motion control of the working carrier facing the grasping task.

[0037] Furthermore, compared with the prior art, in the prior art, usually only a single sensor is used to obtain information. For example, only the local visual perception board 8 is used, and the complete relative depth information of the target and the humanoid hand cannot be obtained. And after the distance is relatively close, since only a partial target object can be seen in the field of view, the field of view of the visual perception board 8 is limited, so it is difficult to accurately judge the position, shape and state of the target object, and it is difficult to generate corresponding prompt information after the distance is relatively close. Then the sensor array board 7 is added. The sensor array board 7 can obtain the discrete depth sequence near the grasping range of the palm. By analyzing the sequence data, the object depth situation of the array area can be obtained. Then, based on the complete local visual target result, the judgment of the presence or absence of the target object and the relative pose relationship between the target object and the humanoid hand when the distance is relatively close can be obtained. Furthermore, the guided grasping of the closed loop of the whole grasping process is realized. Through the mutual cooperation of the visual perception board 8 and the sensor array board 7, the information collected by the visual perception board 8 and the sensor array board 7 is converged and transmitted on the main control board 6 to realize the effective grasping of the target object.

[0038] Taking the operation of the humanoid five-finger hand as an example, the specific implementation process is as follows:

[0039] As Figure 4 shown, when the underactuated five-finger humanoid hand is far from the object, end-side reasoning is performed according to the visual perception board 8. The original image data collected by the input lens is a three-channel RGB image containing the target object and the surrounding environment information. It is processed by the offline visual perception board 8 on the end side, the original image features are extracted and classified and aggregated to obtain the semantic perception classification information of the target object and the surrounding environment and the relative position relationship in the pixel coordinate system compared with the palm 10. At this stage, the values of the proximity sensors are not significant, so the control is mainly based on visual information. The corresponding processing results are transmitted to the main control board 6 via Bluetooth, optimized, and then transmitted to the host computer via the Universal Asynchronous Receiver / Transmitter UART; the user is guided by voice to use the underactuated five-finger humanoid hand to gradually approach the target;

[0040] As Figure 5 shown, after the motion control reaches a certain distance, the palm 10 reaches the minimum detection distance threshold of the sensor array board 7. At this time, based on the features obtained from the first-stage visual information processing of the visual perception board 8, the depth information of each point of the array is obtained by interpolation of the original data of the sensor array board 7 through a fitting function, and the surface contour features collected by the dot matrix are obtained by morphological processing and fitting of the obtained dot matrix depth information. The visual features of the visual perception board 8 are transmitted via Bluetooth, and the sensor array board 7 is transmitted via the FPC flexible line in an I2C communication mode. The two are fused at the feature level on the main control board 6, and the final output result reflects the relative pose relationship with details between the underactuated five-finger humanoid hand and the target object, and is transmitted to the host computer via the Universal Asynchronous Receiver / Transmitter UART; the user is guided by voice to use the underactuated five-finger humanoid hand to gradually approach the target;

[0041] As Figure 6As shown in the figure, as the distance approaches, that is, when the underactuated five-finger humanoid hand is close to the object, the visual information density of the visual perception board 8 decreases significantly due to the short distance. At this time, various processed features obtained by visual information processing are no longer prominent, while the raw data of the sensor array board 7 is more sensitive to the distance change at close range. When the visual information fails to obtain sufficient processed features, the real-time visual information no longer plays a role. At this time, the surface contour features obtained by the dot matrix depth information are mainly compared with the features perceived in the first stage, and the two are matched and approximated on the main control board 6. Finally, the output result reflects the relative pose relationship with details between the underactuated five-finger humanoid hand and the target object, and is transmitted to the host computer through the Universal Asynchronous Receiver / Transmitter (UART); the user is guided by voice to use the humanoid five-finger hand until the proximity sensor array detects that the closest distance between the palm 10 and the target object reaches the grasping threshold and the relative pose between the humanoid five-finger hand and the target object reaches the graspable range, and the MCU issues a control command to close the little finger 1, ring finger 2, middle finger 3, index finger 4, and thumb 9, thereby realizing the grasping of the underactuated five-finger humanoid hand;

[0042] Further, as Figures 4 to 6 shown in the figure, the system integrates the sensor array board 7 and the visual perception board 8 on the palm 10. Visual perception is performed through the visual perception board 8, and distance perception is performed through the sensor array board 7. The two are combined with each other. Based on the complete local visual target result of the visual perception board 8, the sensor array board 7 obtains the judgment of the presence or absence of the target object at a short distance and the relative pose relationship between the target object and the humanoid hand; in the scenario of human-machine interaction, corresponding guiding information needs to be obtained in real time for human-computer interaction. The sensor array board 7 and the visual perception board 8 can guide human-computer interaction in real time, and the sensor array board 7 and the visual perception board 8 are integrated in the palm 10, with a high level of integration; for most of the existing vision-based automated grasps, the humanoid hand is combined with a robotic arm, without considering human-computer interaction. After coordinate transformation based on vision, the full-process path planning from approaching to closing the grasp can be obtained. When in use, only the robotic arm and other devices need to be controlled based on the path planning result to achieve closed-loop control. In this application, effective human-computer interaction can be completed through voice control, and through the information feedback transmitted by the sensor array board 7 and the visual perception board 8, human-computer interaction can be guided in real time.

Claims

1. A fusion sensor system for an underactuated five-finger humanoid hand, comprising an underactuated five-finger humanoid hand, characterized in that: The underactuated five-finger humanoid hand comprises a wrist (11) and a palm (10) rotatably connected to the wrist (11); a plurality of fingers capable of performing a grasping action are arranged on the palm (10); a sensor array board (7) is installed at the center of the palm (10); a visual perception board (8) is arranged on the lower side of the sensor array board (7); a lens of the visual perception board (8) collects visual information through a central hole of the sensor array board (7); the sensor array board (7) obtains a judgment on the presence or absence of a target object when the distance is relatively close and a relative position relationship between the target object and the humanoid hand based on a complete local visual target result of the visual perception board (8); and human-computer interaction is guided in real time through information feedback transmitted by the sensor array board (7) and the visual perception board (8).

2. The fusion sensor system for underactuated five-finger humanoid hand simulation according to claim 1, characterized in that: The sensor array board (7) is a flexible circuit board. The sensor array board (7) is composed of a 5*5 proximity sensor array. The proximity sensor measures the proximity distance of an object based on the reflection principle.

3. The fusion sensor system for underactuated five-finger humanoid hand simulation according to claim 2, characterized in that: The plurality of fingers are respectively a little finger (1), a ring finger (2), a middle finger (3), an index finger (4) and a thumb (9); and a plurality of DC brushless motors (5) for driving the little finger (1), the ring finger (2), the middle finger (3), the index finger (4) and the thumb (9) to move are arranged on the palm (10).

4. The fusion sensor system for underactuated five-finger humanoid hand simulation according to claim 3, characterized in that: A main control board (6) is installed on the palm (10), and the DC brushless motor (5) and the sensor array board (7) are driven and controlled and sensed and collected through the main control board (6). The sensor array board (7) and the main control board (6) are connected via an FPC flexible line and communicate in an I2C manner.

5. The fusion sensor system for underactuated five-finger humanoid hand simulation according to claim 4, characterized in that: The visual perception board (8) processes the collected visual information and transmits it to the main control board (6) via wireless Bluetooth.

6. The fusion sensor system for underactuated five-finger humanoid hand simulation according to claim 5, characterized in that: The main control board (6) is integrated with a micro-control unit MCU for processing raw data of the sensor array board (7), communicating with a host computer, and controlling the movement of multiple fingers. The main control board (6) receives visual information transmitted via Bluetooth and processes the visual feature extraction output result.

7. The fusion sensor system for underactuated five-finger humanoid hand according to claim 6, characterized in that: When the underactuated five-finger simulated human hand is far away from the object, the terminal side reasoning is performed according to the visual perception board (8), and the original image data collected by the input lens includes a three-channel RGB image containing information of the target object and the surrounding environment. The original image features are extracted and classified and aggregated on the terminal side by the offline visual perception board (8), and the semantic perception classification information of the target object and the surrounding environment and the relative position relationship in the pixel coordinate system compared with the palm (10) are obtained. The corresponding processing results are transmitted to the main control board (6) via Bluetooth for optimization, and then transmitted to the host computer via the universal asynchronous receiver and transmitter UART.

8. The fusion sensor system for underactuated five-finger humanoid hand simulation according to claim 7, characterized in that: The palm (10) reaches the minimum detection distance threshold of the sensor array board (7). At this time, based on the features obtained by the first stage of visual information processing of the visual perception board (8), the original data of the sensor array board (7) is interpolated through a fitting function to obtain the depth information of each point in the array, and the obtained dot matrix depth information is subjected to morphological processing and fitting to obtain the surface contour features of the dot matrix collection. The visual features of the visual perception board (8) are transmitted via Bluetooth, and the sensor array board (7) is transmitted via an FPC flexible line in an I2C communication mode. The two are fused at the feature level on the main control board (6). The final output result reflects the relative posture relationship including details between the under-actuated five-finger simulated human hand and the target object, and is transmitted to the host computer via a universal asynchronous receiver-transmitter UART.

9. The fusion sensor system for underactuated five-finger humanoid hand simulation according to claim 8, characterized in that: When the underactuated five-fingered simulated human hand is close to an object, the visual information density of the visual perception board (8) is greatly reduced due to the close distance. At this time, the various processed features obtained by visual information processing are no longer significant, and the original data based on the sensor array board (7) is more sensitive to the distance change at close range. When the visual information cannot obtain enough processed features, the real-time visual information no longer plays a role. At this time, the surface contour features obtained by the dot matrix depth information are mainly compared with the features perceived in the first stage. The two are matched and approximated on the main control board (6). The final output result reflects the relative posture relationship between the underactuated five-fingered simulated human hand and the target object, including details, and is transmitted to the host computer via the universal asynchronous receiver-transmitter UART.

10. The fusion sensor system for underactuated five-finger humanoid hand simulation according to claim 1, characterized in that: The palm (10) is a hollow structure, the sensor array board (7) is fixedly connected to the shell of the palm (10) through the positioning holes, and the main control board (6) is fixed to the shell on the back of the palm (10) through the positioning holes.

Images