Dexterous hand based on multi-modal fusion control

Abstract

The invention provides a dexterous hand based on multi-modal fusion control, and relates to the technical field of automatic control, the dexterous hand comprises a palm module, a palm center vision sensing module, a finger module, a fingertip force sensing module and a multi-modal control module; the palm center visual sensing module is used for acquiring visual information of the dexterous hand in a working area; the finger module comprises at least two fingers; the fingertip force sensing module is used for detecting information of contact force applied to the fingertip position; and the multi-mode control module is used for controlling the dexterous hand to grab a target object based on the visual information and the contact force information. According to the dexterous hand based on multi-mode fusion control, data of two modes of visual information and contact force information are fused, so that the dexterous hand can adapt to different grabbing environments, intelligent closed-loop control of visual guidance and force sense regulation and control is realized, and the environmental adaptability and flexibility of the dexterous hand are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automation control, and in particular to a dexterous hand based on multi-modal fusion control. BACKGROUND

[0002] The dexterous hand of a robot has the characteristics of multiple degrees of freedom in structure, so it has adaptability to clamped objects and can operate the clamped objects to adjust their poses. Meanwhile, the dexterous hand also has the ability of perception, and these advantages make the dexterous hand as a general end effector continuously develop and be applied.

[0003] At present, the dexterous hand of a robot mainly realizes the perception of the environment and fine operation through the combination of various end sensors and the use of different visual information acquisition methods. However, the existing dexterous hand of a robot has the problems of poor environmental adaptability and difficult flexible control due to single perception mode and low structural integration. SUMMARY

[0004] The present application provides a dexterous hand based on multi-modal fusion control to solve the technical problem of poor environmental adaptability and difficult flexible control of the dexterous hand of a robot in the prior art.

[0005] The present application provides a dexterous hand based on multi-modal fusion control, comprising a palm module, a palm center visual sensing module, a finger module, a fingertip force sensing module and a multi-modal control module. The palm center visual sensing module is installed at the center position of the palm module, and is used to acquire visual information of the dexterous hand in a working area. The finger module comprises at least two fingers. The fingertip force sensing module is installed at the fingertip position of each finger, and is used to detect contact force information received at the fingertip position. The multi-modal control module is in communication connection with the palm center visual sensing module and the fingertip force sensing module, and is used to control the dexterous hand to grasp a target object based on the visual information and the contact force information.

[0006] According to the dexterous hand based on multi-modal fusion control provided by the present application, the palm center visual sensing module is a high-definition RGB-D camera, which is used to collect RGB images and depth images.

[0007] According to the dexterous hand based on multi-modal fusion control provided by the present application, the lens of the high-definition RGB-D camera is directed in the same direction as the grasping direction of the palm module.

[0008] According to the dexterous hand based on multi-modal fusion control provided by the application, the multi-modal control module is specifically used for: Based on the visual information, an initial grasping strategy is generated. Based on the initial grasping strategy, the dexterous hand is controlled to perform first grasping on the target object, and the contact force information is obtained. Based on the visual information and the contact force information, the initial grasping strategy is adjusted to generate a target grasping strategy. Based on the target grasping strategy, the dexterous hand is controlled to perform second grasping on the target object.

[0009] According to the dexterous hand based on multi-modal fusion control provided by the application, the finger comprises a rigid force-bearing shell, a flexible contact layer and an intermediate hollow layer. The rigid force-bearing shell is used to support the shell of the finger. The flexible contact layer is used to deform when the fingertip position is stressed. The intermediate hollow layer is a cavity of a predetermined size, and the intermediate hollow layer is used to provide a deformation space for the flexible contact layer.

[0010] According to the dexterous hand based on multi-modal fusion control provided by the application, the material of the rigid force-bearing shell is any one of acrylonitrile-butadiene-styrene copolymer, polycarbonate, polystyrene or carbon fiber reinforced composite material.

[0011] According to the dexterous hand based on multi-modal fusion control provided by the application, the material of the flexible contact layer is any one of silica gel, natural rubber or polyurethane elastomer.

[0012] According to the dexterous hand based on multi-modal fusion control provided by the application, the fingertip force sensation sensing module comprises a permanent magnet and a Hall sensor. The permanent magnet is installed on the inner side of the flexible contact layer, and the permanent magnet is used to generate a magnetic field. The Hall sensor is installed on the inner side of the rigid force-bearing shell and has an initial distance of a predetermined length from the permanent magnet, and the Hall sensor is used to generate a voltage signal according to the magnetic field; the voltage signal is used to represent the contact force information.

[0013] According to the dexterous hand based on multi-modal fusion control provided by the application, the permanent magnet is any one of neodymium iron boron, samarium cobalt or ferrite.

[0014] The application also provides an intelligent robot comprising the dexterous hand based on multi-modal fusion control according to any one of the above.

[0015] The dexterous hand based on multi-modal fusion control provided by the application realizes high integration of multi-modal sensors and controllers, maintains compact and lightweight body structure, and improves control flexibility, by integrating a palm module, a palm center visual sensing module, a finger module, a fingertip force sensing module, and a multi-modal control module; the palm center visual sensing module is installed at the center position of the palm module, the palm center visual sensing module is used to acquire visual information of the dexterous hand in a working area, thereby eliminating the visual blind area of the external camera, and providing global visual guidance and prior environmental perception for subsequent grabbing operation; the finger module comprises at least two fingers; the fingertip force sensing module is installed at the fingertip position of each finger, the fingertip force sensing module is used to detect contact force information received by the fingertip position, thereby providing physical basis for realizing various anthropomorphic grabbing postures such as enveloping and pinching for the dexterous hand through collaborative design of multiple fingers, providing micro tactile feedback necessary for fine operation for the dexterous hand through real-time and direct detection of contact force at each fingertip, and solving the problem of force perception lag or inaccuracy in the traditional scheme; the multi-modal control module is in communication connection with the palm center visual sensing module and the fingertip force sensing module, the multi-modal control module is used to control the dexterous hand to grab a target object based on visual information and contact force information, thereby being able to adapt to different grabbing environments through fusion of data of the two modalities of visual information and contact force information, realizing intelligent closed-loop control of visual guidance and force sensation regulation, and improving environmental adaptability of the dexterous hand in grabbing the target object. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0017] Figure 1 is a structural schematic diagram of the dexterous hand based on multi-modal fusion control provided by the application.

[0018] Figure 2 is a structural schematic diagram of the fingertip position of the finger of the dexterous hand provided by the application.

[0019] Figure 3 is a flowchart of the dexterous hand grabbing a target object provided by the application.

[0020] Figure 4 is a visual algorithm processing flowchart provided by the application. DETAILED DESCRIPTION

[0021] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0022] The existing robot dexterous hand mainly has the following defects: (1) Single perception mode, poor environmental adaptability Most existing dexterous hands rely on visual perception only, which can perform positioning planning before grasping, but becomes "blind grasping" once it enters the contact stage, cannot perceive the force, and is prone to insufficient grasping force (object slipping) or excessive grasping force (object damage).

[0023] Or rely on simple binary contact sensors (such as micro switches), which can only determine "whether there is contact", but cannot obtain the key continuous variable "contact force", which severely limits its ability to perform fine force control operations.

[0024] (2) Inherent defects in force sensing scheme The scheme using traditional strain type force sensors usually has the problems of large volume, difficulty in integration in the narrow space of the fingertips, and is easily affected by temperature drift and electromagnetic interference. Long-term use has the problems of mechanical fatigue and calibration failure. In addition, some existing force sensing schemes install sensors at the distal joints of the dexterous hand, rather than directly measuring the fingertip contact force. This method cannot detect the sliding of the object at the fingertip, and cannot perceive the accurate force information of the contact point, has the problems of feedback lag and inaccuracy.

[0025] (3) Low structural integration, bulky and impractical The existing dexterous hand piles up multiple functions by externalizing sensors and cables, resulting in a bloated and bulky system that destroys the motion performance of the body and is difficult to integrate into an actual robot system.

[0026] At the same time, the vision system is usually externally mounted on the wrist or arm, and there is a deviation between the visual angle and the body of the hand claw, which cannot provide the first visual angle of the palm, and is prone to visual blind area.

[0027] (4) Difficulty in balancing dexterity and reliability Part of the dexterous hand improves dexterity by increasing the degree of freedom, but often leads to complex structure, difficult control, high cost, and complex mechanical structure not only reduces reliability, but also increases maintenance cost. At the same time, there is a lack of multi-modal sensing ability matching the high degree of freedom in the whole dexterous hand, which leads to "hand is clever, but no brain", and cannot play its theoretical operation flexibility.

[0028] (5) High cost, difficult to be practical The existing high ability sensor and dexterous hand technology is usually extremely expensive, which limits its application range to laboratory research, and cannot be popularized in the field of industry, service and other fields requiring large scale.

[0029] Therefore, the present application provides a dexterous hand based on multi-modal fusion control, which constructs an intelligent closed loop system of "visual leading and force sensing fine regulation" by deeply integrating macro vision and micro force sensing. Specifically, first, the palm visual sensing module provides macro and prior environmental information to complete object positioning and rough grasping planning, overcoming the blindness of pure tactile operation; then, in the grasping execution stage, based on the fingertip force sensing module, the soft rubber deformation of the fingertip position is converted into accurate force signals in real time; finally, through various visual processing algorithms and bionic control algorithms, the grasping force of each fingertip is dynamically calculated and adjusted, which not only effectively prevents the object from slipping or being damaged, but also adapts to unknown and unstructured environment, so as to realize soft grasping with high success rate, high safety and high humanization, and provides a solid technical foundation for robot to perform complex tasks.

[0030] The dexterous hand based on multi-modal fusion control of the present application will be described below. Figures 1 to 4 The dexterous hand based on multi-modal fusion control of the present application will be described below.

[0031] Figure 1 The structure diagram of the dexterous hand based on multi-modal fusion control provided by the present application is shown in Figure 1 As shown in the figure, the dexterous hand at least includes: a palm module, a palm visual sensing module, a finger module, a fingertip force sensing module and a multi-modal control module; The palm visual sensing module is installed at the center position of the palm module, and the palm visual sensing module is used to acquire visual information of the dexterous hand in the working area; The finger module includes at least two fingers; The fingertip force sensing module is installed at the fingertip position of each finger, and the fingertip force sensing module is used to detect the contact force information received by the fingertip position; The multi-modal control module is in communication connection with the palm visual sensing module and the fingertip force sensing module respectively, and the multi-modal control module is used to control the dexterous hand to grasp the target object based on the visual information and the contact force information.

[0032] Specifically, the application provides a dexterous hand based on multi-modal fusion control, which at least comprises a palm module, a palm center visual sensing module, a finger module, a fingertip force sensing module, and a multi-modal control module. The finger module comprises at least two fingers. The dexterous hand can be designed with multiple fingers according to actual needs, for example, two, five (as shown in Figure 1 The dexterous hand can be designed with multiple fingers according to actual needs, for example, two, five (as shown in

[0033] Further, the finger comprises a rigid force-bearing shell, a flexible contact layer, and an intermediate hollow layer. The rigid force-bearing shell is used to support the shell of the finger. The flexible contact layer is used to deform when the fingertip position is stressed. The intermediate hollow layer is a cavity of a predetermined size, which is used to provide deformation space for the flexible contact layer.

[0034] Further, the material of the rigid force-bearing shell is any one of acrylonitrile-butadiene-styrene copolymer, polycarbonate, polystyrene, or carbon fiber reinforced composite material.

[0035] Further, the material of the flexible contact layer is any one of silicone, natural rubber, or polyurethane elastomer.

[0036] Specifically, Figure 2 is a schematic diagram of the fingertip position structure of the finger of the dexterous hand provided by the application, as Figure 2 As shown in the figure, the fingertip position structure of the finger at least comprises a Hall element, a magnet, a hollow layer, a fingertip soft glue, and a fingertip upper part. Among them, the Hall element is a Hall sensor, the magnet is a permanent magnet, the hollow layer is an intermediate hollow layer, the fingertip soft glue is a flexible contact layer, and the fingertip upper part is a rigid force-bearing shell.

[0037] The rigid force-bearing shell is made of any one of acrylonitrile-butadiene-styrene copolymer, polycarbonate, polystyrene, or carbon fiber reinforced composite material, which is used as a main structure support and an installation base of a sensor device, and is used to simulate human bones and support the shell of the finger.

[0038] The flexible contact layer is made of any one of silicone, natural rubber, or polyurethane elastomer, which is used to simulate the soft skin of human fingertips and elastically deform when in compliant contact with the target object.

[0039] The intermediate hollow layer is a cavity of a predetermined size, which is used to provide deformation space for the flexible contact layer.

[0040] The fingertip force sensing module is a non-contact magnetic sensor installed at the fingertip of each finger. The fingertip force sensing module is used to detect the contact force information at the fingertip.

[0041] Furthermore, the fingertip force sensing module includes a permanent magnet and a Hall sensor; The permanent magnet is installed on the inner side of the flexible contact layer, and the permanent magnet is used to generate a magnetic field. The Hall sensor is installed on the inner side of the rigid load-bearing shell and the initial distance between it and the permanent magnet is a preset length. The Hall sensor is used to generate a voltage signal based on the magnetic field. The voltage signal is used to characterize the contact force information.

[0042] Furthermore, the permanent magnet is any one of neodymium iron boron, samarium cobalt, or ferrite.

[0043] Specifically, such as Figure 2 As shown, a permanent magnet (i.e., a magnet) is installed on the inner side of the flexible contact layer to generate a magnetic field. A Hall sensor is installed on the inner side of the rigid load-bearing shell, with an initial distance of a preset length between it and the permanent magnet. The Hall sensor is used to generate a voltage signal based on the magnetic field; the voltage signal is used to characterize the contact force information.

[0044] When a fingertip grasps or touches an object, an external force is applied to the underlying soft rubber layer (i.e., the flexible contact layer), causing it to compress inward (towards the fingertip). This deformation causes the embedded permanent magnet to move synchronously, thereby changing the relative distance between the permanent magnet and the Hall sensor fixed above.

[0045] Hall effect sensors can accurately detect changes in magnetic field strength caused by variations in the distance between the sensor and a permanent magnet, and output corresponding voltage signals. By pre-calibrating the sensor signal and the applied force, a mapping relationship between voltage signal, relative distance, and contact force can be established, thereby calculating the magnitude of the contact force on the fingertip in real time and accurately, and obtaining contact force information.

[0046] This invention improves the device integration of the dexterous hand by integrating a permanent magnet and a Hall sensor at the fingertip, ensuring the compactness and lightweight of the dexterous hand body. The deformation of the flexible contact layer drives the permanent magnet to generate displacement, thereby accurately sensing the contact force through the Hall effect, improving the force sensitivity and accuracy. The hollow layer in the middle ensures the linearity and sensitivity of the deformation, avoiding the hard contact of traditional rigid fingers, solving the problems of easy wear and difficulty in miniaturization of traditional sensors, and improving the durability and reliability of the dexterous hand.

[0047] The palm vision sensing module is installed at the center of the palm module and is used to acquire visual information of the dexterous hand in the work area.

[0048] Further, the palm visual sensing module is a high-definition RGB-D camera, which is used to collect RGB images and depth images.

[0049] Further, the lens of the high-definition RGB-D camera is directed in line with the grasping direction of the palm module.

[0050] Specifically, the palm visual sensing module can be a high-definition (binocular or RGB-D) camera installed at the center of the palm module, and the lens or camera of the camera is directed in line with the grasping direction of the palm module, so as to collect global RGB images and depth images of the working area.

[0051] The embodiment of the application embeds the camera of the high-definition RGB-D camera at the center of the palm module, which deeply fuses vision and force sense in space, avoids the bulk and redundancy caused by additional sensors, ensures the compactness and lightness of the dexterous hand body, avoids the calibration and blind area problem of external cameras through image collection of the first visual angle, and improves the accuracy of visual information; the high-definition RGB-D camera collects rich and real-time visual information, provides global visual guidance for the dexterous hand, enhances the environmental perception ability, enables the dexterous hand to "see" and understand the environment and target object before operation, and improves the environmental adaptability.

[0052] The multi-modal control module is in communication connection with the palm visual sensing module and the fingertip force sensing module, and is used to control the dexterous hand to grasp the target object based on the visual information and the contact force information.

[0053] Further, the multi-modal control module is specifically used for: generating an initial grasping strategy based on the visual information; controlling the dexterous hand to perform a first grasping on the target object based on the initial grasping strategy, and obtaining the contact force information; adjusting the initial grasping strategy based on the visual information and the contact force information, and generating a target grasping strategy; controlling the dexterous hand to perform a second grasping on the target object based on the target grasping strategy.

[0054] Specifically, Figure 3 is a flowchart of the dexterous hand grasping a target object provided by the application, as Figure 3 shown, the main grasping steps include: Step 1, generating an initial grasping strategy based on visual information.

[0055] (1) Start the system (i.e. power on the dexterous hand), before grasping the target object, collect visual information (including RGB image and depth image) through the palm visual sensing module; (2) Real-time processing of visual information through a preset visual algorithm, specifically including: a) Target detection: determine the target object to be grasped; b) Pose estimation: calculate the 6D position and pose of the target object relative to the dexterous hand, including the horizontal direction coordinate X, the vertical direction coordinate Y, the depth direction coordinate Z, and the orientation (or rotation matrix R) of the target object; c) Rough grasping point planning: based on the geometric shape and stability constraints of the target object, preliminarily calculate the optimal grasping point; Figure 4 is the visual algorithm processing flowchart provided by the application, as Figure 4 shown, the visual algorithm processing flowchart includes the following first stage, second stage and third stage: First stage: RGB-D camera synchronously captures RGB image and depth image, wherein the RGB image is used for target detection; the depth image is used for generating scene point cloud.

[0056] Second stage: through YOLO-v11 model or DEtection Transformer (DETR) based on Transformer end-to-end target detector, the RGB image is detected, and the output is 2D information, including the category, confidence, 2D bounding box and pixel-level mask (M i ) of the target object; using the depth image and the camera intrinsic matrix (K) of the camera, the 3D coordinates (X, Y, Z) of each pixel point are calculated through the back projection formula, and the 3D point cloud of the whole scene is generated, the back projection formula is as follows: X=(u-c x )*Z / f x Y=(v-c y )*Z / f y In the formula, X represents the horizontal direction coordinate of any target point, Y represents the vertical direction coordinate of any target point, and Z represents the depth direction coordinate of any target point (directly read from the pixel value of the depth image at the position (u, v)); u represents the column index (horizontal direction, starting from the left side of the image) of any target point pixel on the RGB image; f x represents the focal length of the camera in the x-axis direction, f x =F / dx, wherein F is the physical focal length and dx is the physical size of each pixel in the x direction; v represents the row index (vertical direction, starting from the top of the image) of any target point pixel on the RGB image; f y represents the focal length of the camera in the y-axis direction, f y=F / dy, where dy is the physical size of each pixel in the y direction; (c x , c y ) represents the principal point coordinates, i.e., the coordinates of the intersection of the camera optical axis and the image plane (the principal point is located near the center of the image), where c x represents the pixel coordinates of the principal point on the u axis (horizontal direction), and c y represents the pixel coordinates of the principal point on the v axis (vertical direction).

[0057] Through the Point Voting Network for 3D Object Detection and Pose Estimation (PVN3D) or the Full Flow Bidirectional Fusion Network for 6D Pose Estimation (FFB6D), the key points of the target object are predicted based on the category and region of interest (ROI) of the target object and the 3D geometric information of the entire scene, and the accurate 6D pose (including rotation matrix R and translation vector t) is calculated through the Perspective-n-Point (PnP) algorithm and the like.

[0058] The 3D point cloud model of the target object is generated by cropping the point set belonging to the target object from the 3D point cloud of the entire scene using the pixel mask M i as a filter.

[0059] The third stage: the output results in the second stage are packaged into structured perception information, including object category, 3D bounding box (which can be calculated from point cloud or pose), 6D pose (R, t), and 3D point cloud model of the target object. According to the above perception information, the geometric shape and stability constraint of the target object are obtained, so as to preliminarily calculate the optimal grasping point for the downstream robot grasping task (or execute a specific downstream grasping task).

[0060] (3) According to the grasping point, the path of the dexterous hand approaching the target object, the pose of the palm module, and the pre-grasp shape of each finger (i.e., the initial open pose of the finger before closing) are calculated, and an initial grasping strategy is generated.

[0061] Step 2, based on the initial grasping strategy, the dexterous hand is controlled to perform the first grasping on the target object, and the contact force information is obtained.

[0062] (1) According to the initial grasping strategy, the dexterous hand is controlled to move to the grasping point according to the planned path; (2) Control all fingers to start performing the closing action and attempt to grasp the object: when the flexible contact layer of the fingertip contacts the object (when the signal output by the Hall sensor is greater than the preset threshold, it can be judged that the fingertip contacts the object at this time), deformation occurs; deformation causes the magnet embedded in the soft rubber of the fingertip to approach the Hall sensor fixed in the rigid force-bearing shell; the Hall sensor detects the change in magnetic field strength (which changes with distance), and outputs a changing voltage signal; through a pre-set signal processing circuit or software algorithm, combined with the pre-calibrated voltage signal-relative distance-contact force mapping relationship curve, the voltage signal is converted into the contact force value of the fingertip position in real time and accurately.

[0063] Step 3, adjust the initial grasping strategy based on visual information and the contact force information, and generate a target grasping strategy.

[0064] (1) Force sensing information reading: in the force control grasping mode, the real-time contact force information of the fingertip position of all fingers is continuously read; (2) Force control algorithm decision: fuse the prior information provided by the visual information (such as the approximate hardness and roughness of the object) with the real-time contact force information, and based on the pre-set force control target (such as "grasp the egg with 2N force" and the like), calculate the force required by the finger through a Proportional-Integral-Derivative (PID) controller or impedance controller, and generate a target grasping strategy.

[0065] Step 4, control the dexterous hand to perform a second grasping of the target object based on the target grasping strategy.

[0066] According to the corresponding control instructions output by the target grasping strategy, the torque or position of the internal motor of each finger is dynamically and finely adjusted, so as to achieve the following control objectives: a) Adaptive grasping: prevent the target object from slipping (too small force) or being crushed (too large force); b) Compliant operation: even if the position of the target object has a slight deviation, it can be "felt" and stably grasped through force feedback; c) Balanced grip: let multiple fingers work together to keep the target object stable.

[0067] Based on the above grasping steps, the system continuously judges whether the grasping success conditions are met (for example: the force of each fingertip reaches the target value and remains stable, the object is stably lifted, etc.).

[0068] Once the conditions are met, the grasping action is completed, the dexterous hand enters a stable holding state, and is ready to perform subsequent tasks (such as placing, assembling, etc.).

[0069] If the grabbing is not successful (for example, accidental slipping), the whole closed-loop system triggers a retry mechanism, including re-performing visual positioning or adjusting the grabbing strategy, etc.

[0070] The embodiment of the application realizes the closed-loop complementation at the perception level by fusing the visual information and the contact force information through the multi-modal control module, and breaks through the perception limitation of a single mode. Specifically, by combining the macroscopic and prior environmental information of the palm visual with the microscopic and accurate contact information of the fingertip force sense, a perception closed loop of "visual pre-judgment and force sense fine regulation" is constructed; the palm visual sensing module provides global pose guidance before grabbing, and the fingertip force sensing module provides real-time force feedback when contacting, which complement each other and enhance the adaptability of the dexterous hand to unstructured and unknown environments, making the grabbing process more intelligent, compliant and reliable, and effectively dealing with fragile, soft or complex-shaped objects, greatly improving the success rate and safety of grabbing; through the organic combination of multi-modal sensing fusion and bionic control algorithm, the problems of blind operation, poor adaptability and unstable grabbing of the dexterous hand in complex real applications are effectively solved, providing key technical support for realizing fine and humanized operation in the fields of service robots, intelligent prostheses, etc.

[0071] The dexterous hand based on multi-modal fusion control provided by the application realizes the high integration of multi-modal sensors and controllers by integrating the palm module, the palm visual sensing module, the finger module, the fingertip force sensing module and the multi-modal control module, maintains a compact and lightweight body structure, and improves the flexibility of control; the palm visual sensing module is installed at the center position of the palm module, and the palm visual sensing module is used to acquire visual information of the dexterous hand in the working area, thereby eliminating the visual blind area of the external camera and providing global visual guidance and prior environmental perception for subsequent grabbing operation; the finger module includes at least two fingers; the fingertip force sensing module is installed at the fingertip position of each finger, and the fingertip force sensing module is used to detect the contact force information received by the fingertip position, thereby providing the physical basis for realizing various anthropomorphic grabbing postures such as enveloping and pinching through the collaborative design of multiple fingers, and providing the microscopic tactile feedback necessary for fine operation for the dexterous hand by detecting the contact force at each fingertip in real time and directly, solving the problem of force perception lag or inaccuracy in traditional schemes; the multi-modal control module is in communication connection with the palm visual sensing module and the fingertip force sensing module, and the multi-modal control module is used to control the dexterous hand to grab the target object based on the visual information and the contact force information, so as to adapt to different grabbing environments by fusing the data of the two modes of visual information and contact force information, realize intelligent closed-loop control of visual guidance and force regulation, and improve the environmental adaptability of the dexterous hand to grab the target object.

[0072] The embodiment of the present application also provides a kind of intelligent robot, including the dexterous hand based on multi-modal fusion control as described in any one of the above embodiments.

[0073] The present application provides an intelligent robot, comprising one or more dexterous hands based on multi-modal fusion control as described above, by integrating palm module, palm visual sensing module, finger module, fingertip force sensing module and multi-modal control module, the high integration of multi-modal sensor and controller is realized, the compact and lightweight body structure is maintained, and the flexibility of control is improved;The palm visual sensing module is installed at the center position of the palm module, and the palm visual sensing module is used to obtain the visual information of the dexterous hand in the working area, so as to eliminate the visual blind area of the external camera, and provide global visual guidance and prior environmental perception for subsequent grasping operation;The finger module includes at least two fingers;The fingertip force sensing module is installed at the fingertip position of each finger, and the fingertip force sensing module is used to detect the contact force information received by the fingertip position, so as to provide the physical basis for realizing various anthropomorphic grasping postures such as envelope and pinch through the cooperative design of multiple fingers, and provide the micro tactile feedback necessary for fine operation for the dexterous hand by detecting the contact force at each fingertip in real time, solve the problem of force perception lag or inaccuracy in traditional scheme;The multi-modal control module is in communication connection with the palm visual sensing module and the fingertip force sensing module respectively, and the multi-modal control module is used to control the dexterous hand to grasp the target object based on the visual information and the contact force information, so as to adapt to different grasping environments by fusing the data of the two modalities of visual information and contact force information, realize the intelligent closed-loop control of visual guidance and force sensation regulation, and improve the environmental adaptability of the dexterous hand to grasp the target object.

[0074] It should be noted that in this document, the terms "comprise", "comprising", or any other variant thereof are intended to cover non-exclusive inclusions, so that processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed, or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to the order of performing the functions shown or discussed, but can also include performing the functions in a substantially simultaneous manner or in reverse order, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted or combined. In addition, the features described with reference to certain examples can be combined in other examples.

[0075] The "determining B based on A" in the embodiments of the present application means that A is considered as a factor when determining B. It is not limited to "determining B based on A only", but also includes "determining B based on A and C", "determining B based on A, C and E", "determining C based on A, and determining B based on C further", and the like. In addition, it can also include taking A as a condition for determining B, for example, "when A meets the first condition, determining B using the first method"; for example, "when A meets the second condition, determining B"; for example, "when A meets the third condition, determining B based on the first parameter"; and the like. Of course, it can also be that A is taken as a condition for determining B, for example, "when A meets the first condition, determining C using the first method, and determining B further based on C"; and the like.

[0076] In addition, it should be noted that the terms "target", "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second" are usually a class, and are not limited to the number of objects, for example, the first object can be one or more.

[0077] In the present application, the term "multiple" means two or more, and other quantifiers are similar.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A dexterous hand based on multi-modal fusion control, characterized in that, The application relates to a dexterous hand based on multi-modal fusion control. The dexterous hand comprises a palm module, a palm center visual sensing module, a finger module, a fingertip force sensing module and a multi-modal control module. The palm center visual sensing module is installed at the center of the palm module, and is used for acquiring visual information of the dexterous hand in a working area. The finger module comprises at least two fingers. The fingertip force sensing module is installed at the fingertip position of each finger, and is used for detecting contact force information received by the fingertip position. The multi-modal control module is in communication connection with the palm center visual sensing module and the fingertip force sensing module, and is used for controlling the dexterous hand to grab a target object based on the visual information and the contact force information.

2. The multi-modal fusion control based dexterous hand of claim 1, wherein, The palm center visual sensing module is a high-definition RGB-D camera, which is used for collecting RGB images and depth images.

3. The multi-modal fusion control based dexterous hand of claim 2, wherein, The lens of the high-definition RGB-D camera is directed in the same direction as the grabbing direction of the palm module.

4. The multi-modal fusion control based dexterous hand of claim 3, wherein, The multi-modal control module is specifically used for: generating an initial grabbing strategy based on the visual information; controlling the dexterous hand to perform first grabbing on the target object based on the initial grabbing strategy, and obtaining the contact force information; adjusting the initial grabbing strategy based on the visual information and the contact force information, and generating a target grabbing strategy; controlling the dexterous hand to perform second grabbing on the target object based on the target grabbing strategy.

5. The multi-modal fusion control based dexterous hand of claim 1, wherein, The finger comprises a rigid force-bearing shell, a flexible contact layer and an intermediate hollow layer. The rigid force-bearing shell is used for supporting the shell of the finger. The flexible contact layer is used for deforming when the fingertip position is subjected to force. The intermediate hollow layer is a cavity with a preset size, and is used for providing a deformation space for the flexible contact layer.

6. The multi-modal fusion control based dexterous hand of claim 5, wherein, The material of the rigid force-bearing shell is any one of acrylonitrile-butadiene-styrene copolymer, polycarbonate, polystyrene or carbon fiber reinforced composite material.

7. The multi-modal fusion control based dexterous hand of claim 5, wherein, The material of the flexible contact layer is any one of silica gel, natural rubber or polyurethane elastomer.

8. The multi-modal fusion control based dexterous hand of claim 5, wherein, The fingertip force sensing module comprises a permanent magnet and a Hall sensor. The permanent magnet is installed on the inner side of the flexible contact layer, and is used for generating a magnetic field. The Hall sensor is installed on the inner side of the rigid force-bearing shell and has an initial distance of a preset length from the permanent magnet, and is used for generating a voltage signal according to the magnetic field; the voltage signal is used for representing the contact force information.

9. The multi-modal fusion control based dexterous hand of claim 8, wherein, The permanent magnet is any one of neodymium iron boron, samarium cobalt or ferrite.

10. An intelligent robot, characterized in that, The application further discloses a dexterous hand based on multi-modal fusion control.

Images