Multi-fingered hand cooperative grasping method and device based on visual-tactile fusion

Abstract

The application provides a multi-fingered dexterous hand cooperative grasping method and device based on visual-haptic fusion. The original fusion image collected by a monocular imaging module in real time is acquired, a binary mask matrix is generated according to a pre-burned physical mask model, an element-wise tensor separation operation is performed on the original fusion image by using the matrix, and a haptic feature image and an environmental visual feature image are decoupled and generated. The haptic feature image is subjected to an optical flow tracking process, a feature point displacement field of a multi-layer concentric circle coded elastic skin surface is extracted, and a three-dimensional contact force vector is calculated; the environmental visual feature image is processed, a standard panoramic environmental image is generated, and the contour and centroid position of a target object are extracted. It is judged whether the elastic skin is deformed beyond a preset threshold, if not, a visual servo mode is activated for pose alignment, and if yes, a haptic force control mode is switched to apply a constant envelope clamping force. The application realizes low-delay cooperative control from visual navigation in the approach stage to constant force envelope after contact.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a multi-finger dexterous hand collaborative grasping method and device based on visual-tactile fusion. Background Technology

[0002] As the end effector of embodied intelligent robots, the grasping ability of a multi-fingered dexterous hand directly depends on its level of perception of the target object. Existing grasping systems typically configure visual perception and force perception separately, with the vision module responsible for target localization and the force module responsible for contact detection. The two are hardware-independent and run serially in data processing. This separate architecture presents two problems in practical applications.

[0003] First, external vision has an inherent blind spot at the grasping end. When a multi-fingered dexterous hand approaches a target object at close range, the mechanical fingers and their drive structure can obstruct the view of the external camera, causing the system to lose visual guidance in the final stroke before contact occurs. This can easily lead to positional deviation between the fingers and the target, resulting in grasping failure. Second, force perception relies on discrete force sensors mounted on joints or fingertips, which can only provide force information at a limited number of measurement points. This makes it difficult to reflect the continuous distribution of contact force on the envelope surface, and the sensors are relatively large, increasing the finger's inertia.

[0004] To address these issues, a new approach has emerged in recent years: integrating visual and tactile sensing into a single module. This approach uses a monocular camera to capture deformation images of the elastomer's surface for tactile measurement, while simultaneously utilizing the same optical path to acquire environmental images for visual perception, potentially enabling multimodal sensing within a compact sensor volume. However, in a monocular coaxial architecture, the tactile illumination source and ambient visual light share the same optical path. To prevent stray light from the internal tactile illumination source from interfering with environmental visual imaging, a physical isolation structure is typically required inside the sensor to block stray light. This isolation structure occupies a significant amount of internal space, resulting in a substantial increase in the sensor's outer diameter. When such a sensor is installed in the palm of a multi-fingered dexterous hand, its large outer diameter severely restricts the inward envelopment space of the surrounding mechanical fingers. When the fingertips grasp small objects, they are prone to physical collisions with the sensor's outer shell, hindering tight, centripetal grasping.

[0005] Therefore, how to enable a multi-finger dexterous hand to have the ability to integrate proximity and tactile perception within a compact palm space without sacrificing the centripetal envelope range of motion of the multi-finger is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0006] Based on this, it is necessary to address the technical problem of how to enable a multi-finger dexterous hand to have the ability to integrate proximity and tactile perception within a compact palm space, without sacrificing the centripetal envelope range of movement of the multi-finger due to the space occupied by the internal isolation structure. A multi-finger dexterous hand collaborative grasping method and multi-finger dexterous hand device based on visual-tactile fusion is proposed.

[0007] In a first aspect, this application provides a multi-finger dexterous hand collaborative grasping method based on visual-tactile fusion, applied to a main control computing unit, wherein the main control computing unit is communicatively connected to a monocular imaging module and a motor drive board of the multi-finger dexterous hand; the method includes: Acquire the original fused image acquired in real time by the monocular imaging module; Based on the pre-programmed physical mask model, a binary mask matrix corresponding to the pixel positions of the original fused image is generated in memory. Using the binary mask matrix, an element-wise tensor separation operation is performed on the original fused image to decouple and generate tactile feature images and environmental visual feature images; The following two feature extraction methods are performed in parallel: optical flow tracing is performed on the tactile feature image to extract the displacement field of feature points on the multi-layer concentric circle coded elastic skin surface covered by the monocular imaging module's field of view caused by contact force, and the three-dimensional contact force vector of the multi-layer concentric circle coded elastic skin surface is calculated based on the displacement field; the environmental visual feature image is processed to generate a standard panoramic environmental image, and the image contour and centroid position information of the target object are extracted. Determine whether the surface of the multi-layer concentric circle coded elastic skin has deformed beyond a preset pixel displacement threshold; if not, activate the visual servo mode, generate a pose alignment control command based on the deviation between the centroid position information of the target object and the preset desired position, and send it to the motor drive board to drive the multi-finger dexterous hand to approach the target object; if yes, switch to the tactile force control mode, generate a force tracking control command based on the deviation between the three-dimensional contact force vector and the preset target force vector, and send it to the motor drive board to drive the multi-finger dexterous hand to apply a constant envelope clamping force to the target object.

[0008] Secondly, this application provides a multi-finger dexterous hand device, comprising: The sensing base housing has a hollow cavity inside, and no physical isolation structure for isolating stray light is provided in the hollow cavity. A monocular imaging module is fixedly installed at the bottom of the sensor base housing, with the field of view of the monocular imaging module facing upwards; An array of light sources is arranged around the outer periphery of the monocular imaging module, and the output optical axis of the array of light sources is set at an inclined angle relative to the optical center normal of the monocular imaging module. A transparent support plate is fixedly placed over the top opening of the sensing base housing; A local diffuser sheet is attached only to the center area of ​​the bottom surface of the transparent support plate, and is used to uniformly diffuse the emitted light from the array light source to the center area above the transparent support plate. A multi-layered concentric circle coded elastic skin covers the upper surface of the transparent support plate. The surface of the multi-layered concentric circle coded elastic skin is divided into multiple staggered transparent annular areas and non-transparent annular areas. Multiple mechanical fingers, the base of each mechanical finger is respectively hinged to different positions on the outer wall of the sensing base shell, and the movement trajectory of the end of each mechanical finger points to the central area of ​​the multi-layer concentric circle coded elastic skin. The motor drive board is fixedly installed inside or outside the housing of the sensing base and is electrically connected to the drive motor of each of the mechanical fingers; The main control computing unit is communicatively connected to the monocular imaging module and the motor drive board, and is used to execute the above-mentioned multi-finger dexterous hand collaborative grasping method based on visual-tactile fusion.

[0009] The beneficial effects of this application are: This application utilizes a pre-programmed physical mask model to directly generate a binary mask matrix, and employs element-wise tensor separation operations to rapidly decouple the original fused image into independent tactile feature images and environmental visual feature images. This approach transforms the multimodal signal separation problem into a numerical comparison and matrix multiplication problem based on physical boundary determination, enabling the separation of visual and tactile signals to be completed within microseconds after the image is loaded into memory. This achieves efficient decoupling of the perceptual layer and provides a real-time data foundation for the subsequent parallel extraction of the two feature paths.

[0010] This application performs inverse polar coordinate transformation on the environmental visual feature image to generate a standard panoramic environmental image and extracts the target contour and centroid position. Simultaneously, it performs optical flow tracing on the tactile feature image to calculate the three-dimensional contact force vector. The two feature extraction paths run in parallel without interference. This parallel architecture enables the system to continuously acquire target pose information during the approach process to guide precise centripetal alignment, while keeping the tactile perception channel in a standby state at all times. This avoids the time-division switching overhead between the two perception modes and can reduce the latency accumulation before the control mode switching.

[0011] This application automatically identifies the moment of contact by determining whether the elastic skin surface undergoes deformation exceeding a preset pixel displacement threshold. Upon detecting contact, it immediately switches from visual servo mode to tactile force control mode, directly generating force tracking control commands using the deviation between the three-dimensional contact force vector and the target force vector. This switching does not rely on external trigger signals or upper-level planner intervention; it is entirely based on the underlying sensing signals themselves, ensuring that control handover is completed at the same moment contact occurs. This allows multi-finger dexterity hands to maintain stable envelope gripping without waiting for replanning in visual blind spots.

[0012] In summary, this application does not rely on external global cameras and articulated force sensors. It can complete the entire process from visual navigation in the approach phase to constant force envelope after contact using only an in-palm monocular imaging module. This can solve the problem of coordinated control of multi-finger dexterous hands with both proximity perception and tactile feedback capabilities in a compact palm configuration. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] in: Figure 1A A three-dimensional view of the multi-finger dexterous hand device provided in the embodiments of this application; Figure 1B A side cross-sectional view of a dexterous hand in an open state, provided for an embodiment of this application; Figure 1C A side cross-sectional view of the central multimodal sensing base after the outer mechanical fingers have been peeled off, as provided in an embodiment of this application; Figure 1D An exploded three-dimensional view of the central multimodal sensing base pulled apart along the Z-axis after the outer mechanical fingers are peeled off, as provided in an embodiment of this application. Figure 2 This is a flowchart illustrating a multi-finger dexterous hand collaborative grasping method based on visual-tactile fusion in one embodiment. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0016] like Figures 1A to 1D As shown, this application provides a multi-finger dexterous hand device integrating a visual-tactile sensor. The device mainly includes a sensing base housing 1, a monocular imaging module 2, an array light source 6, a transparent support plate 5, a local diffuser 7, a narrowband filter 8, a multi-layer concentric circle coded elastic skin 4, multiple mechanical fingers 9, a motor drive board 3, and a main control computing unit. The following is a detailed description of each component and their assembly and connection relationships.

[0017] The sensing base housing 1 serves as the core rigid frame and structural base of the entire multi-finger dexterous hand device, and its interior forms a hollow cavity. This hollow cavity is enclosed by the bottom inner wall and the inner surface of the side walls of the housing, forming a completely transparent chamber structure in physical space. Unlike traditional coaxial multimodal sensor solutions that require an opaque metal isolation cone inside to block stray light, the sensing base housing 1 of this application does not have any physical isolation structure for blocking stray light within its hollow cavity. This design maximizes the utilization of the internal space of the housing, with the entire hollow cavity completely open from bottom to top, without any obstructions blocking the light path. Multiple mounting ears are evenly distributed along the circumference of the outer wall of the sensing base housing 1. These mounting ears can be integrally molded with the housing body or fixedly connected to the outer wall of the housing by welding or threaded fasteners. The number of mounting ears corresponds to the number of mechanical fingers. In the preferred embodiment of this application, there are three mounting ears, which are distributed in a circumferential array at equal intervals of 0 degrees, 120 degrees, and 240 degrees along the outer wall of the housing.

[0018] The monocular imaging module 2 is fixedly mounted on the bottom of the sensor base housing 1. Specifically, a field-of-view centering hole is provided on the bottom end face of the sensor base housing 1, and the axis of this hole coincides with the geometric center axis of the housing. The monocular imaging module 2 is recessed and concealed within this field-of-view centering hole, with its field of view facing upwards, i.e., towards the top opening of the sensor base housing 1. The top of the monocular imaging module 2 does not extend beyond the bottom outer surface of the sensor base housing 1. This recessed mounting method ensures that the imaging module does not occupy additional space outside the housing in the axial direction, which is beneficial for the miniaturization of the entire device. The monocular imaging module 2 consists of an imaging lens and a complementary metal-oxide-semiconductor (CMOS) image sensor chip. The optical axis of the imaging lens coincides with the geometric center axis of the housing, and the image sensor chip is used to convert optical image signals into digital image signals and output them externally through a data interface.

[0019] An array-type light source 6 is arranged around the periphery of the monocular imaging module 2. Specifically, the array-type light source 6 is fixed on a printed circuit board, which is arranged around the monocular imaging module 2 and fixed to the bottom inner side of the sensing base housing 1. The array-type light source 6 contains multiple light-emitting diode chips, which are evenly distributed along the circumference to form a ring-shaped illumination source surrounding the monocular imaging module 2. The emission optical axis of the array-type light source 6 is set at an angle relative to the optical center normal of the monocular imaging module 2. The optical center normal is a straight line passing through the optical center of the lens of the monocular imaging module 2 and perpendicular to the photosensitive surface of the image sensor chip, which coincides with the optical axis of the imaging lens. The angle between the emission optical axis of the array-type light source 6 and this normal is mechanically limited to the range of 20 degrees to 35 degrees, so that the light emitted by the light source illuminates the top area of ​​the sensing base housing 1 in an oblique direction, rather than vertically upward. This oblique illumination method helps to avoid direct light from the light source entering the imaging lens and causing glare. The emission wavelength of the array light source 6 is matched with the passband wavelength of the narrowband filter 8. For example, if the light source uses 850nm infrared light, the narrowband filter is a bandpass filter with the corresponding wavelength.

[0020] A transparent support plate 5 is fixedly attached to the top opening of the sensing base housing 1. The transparent support plate 5 is a thin, circular plate made of a high-transmittance material, with a diameter matching the inner diameter of the top opening of the sensing base housing 1. The transparent support plate 5 can be fixed to the top of the housing by adhesive bonding, mechanical snap-fitting, or threaded clamping, thus sealing the hollow cavity of the housing from the top. The upper surface of the transparent support plate 5 forms a support surface for the multi-layered concentric circle coded elastic skin 4, while its lower surface faces the interior of the hollow cavity.

[0021] The local diffuser 7 is attached only to the central area of ​​the bottom surface of the transparent support plate 5. The local diffuser 7 is a microstructured thin film with light diffusion function, its size roughly equivalent to the size of the central tactile sensing area of ​​the multi-layered concentric circle coded elastic skin 4. The local diffuser 7 is fixedly attached to the very center of the bottom surface of the transparent support plate 5 by optical adhesive or mechanical pressing, and this area is located directly above the field of view of the monocular imaging module 2. The function of the local diffuser 7 is to uniformly diffuse the emitted light from the array light source 6 to the central area above the transparent support plate 5. Specifically, when the array light source 6 illuminates the bottom surface of the transparent support plate 5 at an angle, the light is scattered and homogenized by the local diffuser 7 as it passes through the transparent support plate 5, thus forming a uniformly distributed illumination field in the central area above the transparent support plate 5. Meanwhile, the outer annular area of ​​the bottom surface of the transparent support plate 5 where the local diffuser 7 is not attached remains highly transparent, allowing ambient light to pass through these areas unobstructed into the hollow cavity and reach the monocular imaging module 2.

[0022] A multi-layered concentric circle coded elastic skin 4 covers the upper surface of the transparent support plate 5. The multi-layered concentric circle coded elastic skin 4 is made of a high-transmittance elastomer material, such as high-transmittance liquid silicone. The edges of this elastic skin are fixedly connected to the top edge of the sensor base housing 1 via a mechanical snap-fit ​​structure or a potting seal structure, and the lower surface of the multi-layered concentric circle coded elastic skin 4 is tightly fitted to the upper surface of the transparent support plate 5. The surface of the multi-layered concentric circle coded elastic skin 4 is divided into multiple staggered transparent annular regions and non-transparent annular regions. Specifically, from the inside out, they are the first annular region, the second annular region, the third annular region, the fourth annular region, and the fifth annular region. The first annular region is the central primary tactile sensing area, a non-transparent annular region. The second annular region, surrounding the first annular region, is a transparent annular region, forming a near-field visual window. The third annular region, surrounding the second annular region, is also a non-transparent annular region, forming an auxiliary tactile sensing area. The fourth annular region, surrounding the third annular region, is a transparent annular region, forming a mid-to-far-field visual window. The fifth annular region, surrounding the fourth annular region, is also a non-transparent annular region, forming an edge tactile sensing area. The surfaces of the first, third, and fifth annular regions are all coated with a high-contrast speckle feature coating. These speckle features are used by the monocular imaging module 2 to acquire contact deformation images for optical flow tracing. The radial widths of the second and fourth annular regions increase from the inside out. This increasing pattern matches the isoangular projection distortion model of the lens of the monocular imaging module 2, ensuring that the images formed by the second and fourth annular regions on the photosensitive chip of the monocular imaging module 2 have equal pixel widths.

[0023] The base ends of multiple mechanical fingers 9 are respectively hinged to different circumferential positions on the outer wall of the sensing base housing 1. In a preferred embodiment of this application, there are three mechanical fingers, and the base ends of the three mechanical fingers are distributed in a circumferential array at equal intervals along the outer wall of the sensing base housing 1. Each mechanical finger is a multi-joint mechanical finger, that is, each mechanical finger contains multiple links connected in series by hinge shafts, and a drive motor is provided between adjacent links. Each drive motor is independently driven and controlled. The base end of each mechanical finger is hinged to the mounting ear on the outer wall of the sensing base housing 1 by the hinge shaft, so that the entire mechanical finger can move with multiple degrees of freedom relative to the sensing base housing 1 in three-dimensional space. The end motion trajectory of each mechanical finger points to the central region of the multi-layer concentric circle coded elastic skin 4, forming a physically centripetal grasping configuration. When each mechanical finger closes towards the center under the drive of its respective drive motor, its end ends collectively envelop the target object located above the central region of the multi-layer concentric circle coded elastic skin 4 from different directions.

[0024] In one example, the limit grab geometric boundary constraint equation is: .

[0025] in, This indicates the smallest target object radius that the system can grasp without a rigid collision (physical interference) between the mechanical finger and the palm sensor, expressed in millimeters. This indicates the actual physical outer radius of the CCTV haptic sensor, in millimeters. This indicates the equivalent actuator length of a single mechanical finger, expressed in millimeters. This indicates the maximum allowable inward closing deflection angle of the mechanical finger base joint, expressed in radians or degrees.

[0026] In one example .

[0027] in, This indicates the system's ultimate tangential disturbance rejection capability (maximum static friction force). It represents the equivalent coefficient of friction between the central elastic skin surface and the target object. It represents the equivalent coefficient of friction between the i-th mechanical fingertip and the target object. This refers to the normal reaction force that passively reacts to the surface of the central elastic skin after the target object is squeezed by the peripheral mechanical fingers. This represents the normal contact force actively applied to the target object by the i-th mechanical finger during the closing process. It is obtained by kinematic conversion from the output torque of the joint motor corresponding to the mechanical finger. The subscript i takes the values ​​1, 2, and 3, which correspond to the three mechanical fingers of the circumferential array, respectively.

[0028] The motor drive board 3 is fixedly mounted inside or outside the sensor base housing 1. The motor drive board 3 is a printed circuit board integrating a motor drive circuit, a current sampling circuit, and a communication interface circuit. When the motor drive board 3 is located inside the sensor base housing 1, it can be fixedly installed in the empty space on the inner bottom of the housing, next to the monocular imaging module 2 and the array light source 6. When the motor drive board 3 is located outside the sensor base housing 1, it can be fixedly installed on the outer wall of the sensor base housing 1 or on the end flange of the robotic arm. The motor drive board 3 is electrically connected to the drive motors of each robotic arm via cables, and is used to provide corresponding drive current to each drive motor according to the control commands issued by the main control computing unit.

[0029] The main control computing unit, not shown in the figure, can be integrated inside the sensing base housing 1 or independently located outside the sensing base housing 1. The main control computing unit is an embedded control board or industrial computing device containing a processor, memory, communication interface, and input / output interfaces. The main control computing unit communicates with the monocular imaging module 2 via a high-speed data cable using a universal serial bus interface or a mobile industrial processor interface to receive the raw fused image data acquired by the monocular imaging module 2. The main control computing unit also communicates with the motor drive board 3 via a controller area network bus to send motion control commands to the motor drive board 3 and receive joint status feedback data reported by the motor drive board 3. The main control computing unit is used to execute the aforementioned multi-finger dexterous hand collaborative grasping method based on visual-tactile fusion, including steps such as acquiring the raw fused image, generating a binary mask matrix, tensor separation and decoupling, feature extraction, pattern judgment, and control command generation.

[0030] In a further embodiment of this application, the device further includes a narrowband filter 8. The narrowband filter 8 is fixedly mounted on the lens surface of the monocular imaging module 2, located between the monocular imaging module 2 and the transparent support plate 5. The narrowband filter 8 allows only light within a specific wavelength range to pass through, thereby filtering out stray wavelength components in ambient light and improving the image signal-to-noise ratio.

[0031] The following is combined with Figures 1A to 1D The overall assembly relationship and working principle of the multi-finger dexterous hand device of this application are further explained.

[0032] like Figure 1A The figure shows an overall three-dimensional view of the multi-finger dexterous hand device provided in this embodiment. As can be seen, three independently driven mechanical fingers are tightly hinged to the outer edge of the central tactile sensing base housing 1 in a circumferential array with equal intervals of 0 degrees, 120 degrees, and 240 degrees. Each mechanical finger is independently powered by a corresponding finger joint drive motor. The end-effector trajectories of the three mechanical fingers all point towards the centrally located multi-layered concentric circle coded elastic skin 4, forming a physically centripetal grasping envelope.

[0033] In one possible embodiment of this application, the main control computing unit includes three parallel core software modules: an image mask decoupling and representation module, a multimodal action decision evaluation module, and a ROS2 (Robot Operating System 2) collaborative control node. The peripheral connections of the main control computing unit are as follows: The data output terminal of the monocular imaging module 2 is connected to the image mask decoupling and characterization module via a high-speed data line of USB 3.0 (Universal Serial Bus 3.0) or MIPI-CSI (Mobile Industry Processor Interface - Camera Serial Interface); the kinematic control signal output terminal of the ROS2 collaborative control node is connected to the hand motor drive board 3, which is connected to multiple joint motors of the peripheral multi-finger mechanical claw via a CAN (Controller Area Network) bus; the GPIO (General Purpose Input / Output) interface of the main control computing unit is connected to the light source PWM (Pulse Width Modulation) drive control board, which controls the array light source 6 at high frequency by changing the drive current to achieve hardware-level synchronization with the camera shutter exposure.

[0034] like Figure 1B The figure shows a side cross-sectional view of the dexterous hand provided in this embodiment of the application in its open state. The figure clearly shows the minimalist and compact layout of each hardware component within the limited physical space of the palm. The sensing base housing 1 serves as the core rigid skeleton and palm of the entire multi-finger dexterous hand. Its outer wall is machined with mounting ears for hinged connection of the three peripheral mechanical fingers, and its interior forms a completely transparent hollow cavity. The monocular imaging module 2 is recessed and concealed within the field-of-view centering hole at the bottom of the housing, surrounded by an array of light sources 6 that illuminate upwards at a 25-degree angle. A multi-layered concentric circle coded elastic skin 4 covers the top of the housing, serving as a flexible palm support surface during multi-finger grasping.

[0035] like Figure 1CThe figure shows a side cross-sectional view of the multimodal sensing base in the center after the outer mechanical fingers have been peeled off, as provided in this embodiment of the application. The figure clearly shows the minimalist and compact layout of the various optical and hardware components within a limited physical space. The interior of the sensing base housing 1 forms a completely transparent hollow cavity, eliminating the need for the opaque metal isolation cone that is required in traditional technologies. The monocular imaging module 2 is recessed and concealed within the field-of-view centering hole at the bottom of the housing, surrounded by an array of light sources 6 that illuminate upwards at a 25-degree angle. A multi-layered concentric circle coded elastic skin 4 covers the top of the housing, serving as a flexible palm support surface for multi-finger grasping.

[0036] like Figure 1D The figure shows an exploded three-dimensional view of the multimodal sensing base in the center after the outer mechanical fingers of the present application have been peeled off along the Z-axis. The stacked assembly constraints of the internal core optical components can be clearly seen from the figure. At the bottom layer of the assembly is the central monocular imaging module 2, whose lens surface is fitted with a narrow-band filter 8, and an array of light sources 6 fixed on the PCB board is arranged around it. The output optical axis of the array of light sources 6 is mechanically limited to a directional oblique angle of 20 to 35 degrees off the center. At the center of the bottom surface of the transparent support plate 5 at the top of the cavity, a micro local diffuser 7 is attached, covering only the central tactile sensing area D1. The physical logic of the optical path self-decoupling is that the oblique light passes through the micro local diffuser 7 to uniformly illuminate the central tactile area, while the external ambient light can bypass the diffuser and penetrate the transparent support plate 5. The multi-layer concentric circle coded elastic skin 4 at the top is tightly sealed to the top of the sensing base shell 1 by mechanical snap-fit ​​or edge potting, serving as the physical support surface and deformation generator of the entire multimodal sensor.

[0037] In summary, the multi-finger dexterous hand device integrating visual and tactile sensors of this application achieves stray light elimination and optical path self-decoupling at a purely physical level by designing the interior of the sensing base shell 1 as a completely transparent hollow cavity and eliminating physical partition structures. Combined with the array light source 6 tilted at the bottom and the local diffuser 7 in the top central area, this design allows for extreme compression of the outer diameter of the sensing base shell 1, providing ample physical space for the circumferentially equidistant hinge of multiple mechanical fingers on the outer wall of the shell, thus eliminating the problem of physical interference between the mechanical fingers and the sensor shell when they retract. Simultaneously, the staggered arrangement of transparent and non-transparent annular regions of the multi-layered concentric circle coded elastic skin 4, in conjunction with the monocular imaging module 2, enables spatial multiplexing of the visual and tactile channels on a single photosensitive chip, allowing the device to simultaneously possess environmental visual perception and tactile force perception capabilities within a very small volume. The synergistic effect of the aforementioned structural features enables the device to achieve multi-pointed centripetal grasping, visual-tactile fusion perception, and high-frequency force feedback control with a size the size of a real human fingertip, providing a highly integrated solution for end effectors of embodied intelligent robots.

[0038] like Figure 2 As shown in the diagram, in one embodiment, a flowchart of a multi-finger dexterous hand collaborative grasping method based on visual-tactile fusion is provided, which specifically includes the following steps: Step S1: Acquire the original fused image acquired in real time by the monocular imaging module.

[0039] Before the dexterous hand device begins its grasping task, the main control computing unit first completes system initialization operations, including configuring the registers of the monocular imaging module, setting the exposure time, gain parameters, and output image resolution and frame rate of the image sensor. Simultaneously, the main control computing unit sends an enable signal to the light source pulse width modulation drive control board via a general-purpose input / output interface, driving the array-type directional oblique light source to operate at a preset strobe frequency. The strobe timing of this light source and the shutter exposure timing of the monocular imaging module are synchronized at the hardware level by the main control computing unit to ensure stable and consistent illumination conditions for each frame of image acquisition.

[0040] After initialization, the main control computing unit continuously receives the raw fused image data uploaded by the monocular imaging module via the data cable according to a preset sampling period. This raw fused image is a two-dimensional pixel matrix containing both visual and tactile information. Its pixel values ​​reflect the superposition effect of multiple light signals transmitted from the top of the sensing base housing. Specifically, ambient light entering the transparent annular area from the external environment and reflected light modulated by the elastic skin deformation after illuminating the non-transparent annular area by the internal array light source are simultaneously imaged on the photosensitive chip of the monocular imaging module, forming a raw fused image in which visual and tactile channel information are spatially intertwined and mixed.

[0041] Step S2: Based on the pre-programmed physical mask model, generate a binary mask matrix in memory corresponding to the pixel positions of the original fused image.

[0042] The non-volatile memory of the main control computing unit pre-programs physical mask model data, which includes the boundary polar radius thresholds of each annular region of the multi-layer concentric circle encoded elastic skin. Since the five annular regions of the multi-layer concentric circle encoded elastic skin have fixed radial boundaries in physical space, and these boundaries have been reverse-calibrated during the design phase according to the isometric projection mapping law of the lens used in the monocular imaging module, the boundaries of each annular region are represented as a series of concentric rings with fixed polar radius values ​​in the two-dimensional pixel matrix of the original fused image.

[0043] After each frame of raw fused image data is read into memory, the main control computing unit first uses the optical center of the image as the origin of the image's physical coordinate system, and then calculates the polar radius value of each pixel in the image matrix from this origin. The Euclidean distance formula is used to calculate the polar radius, which is the square root of the sum of the squares of the horizontal and vertical coordinate offsets of the pixel. After obtaining the polar radius value of each pixel, the main control computing unit compares this value with the pre-programmed polar radius thresholds of multiple annular regions to determine which annular region the pixel falls into.

[0044] If a pixel falls within the polar radius interval corresponding to the first, third, or fifth annular region (i.e., within a non-transparent annular region), the main control calculation unit sets the mask value of that pixel to a first logical value. If a pixel falls within the polar radius interval corresponding to the second or fourth annular region (i.e., within a transparent annular region), the main control calculation unit sets the mask value of that pixel to a second logical value. After traversing all pixels of the original fused image, the main control calculation unit arranges the mask values ​​of each pixel according to their pixel positions to form a binary mask matrix with the same size as the original fused image. In this binary mask matrix, the pixel positions with values ​​of the first logical value constitute the reserved region of the tactile channel, and the pixel positions with values ​​of the second logical value constitute the reserved region of the visual channel.

[0045] Step S3: Using a binary mask matrix, perform element-wise tensor separation operation on the original fused image to decouple and generate tactile feature images and environmental visual feature images.

[0046] After obtaining the binary mask matrix, the main control computing unit allocates two image buffers in memory, each the same size as the original fused image, to store the tactile feature image and the environmental visual feature image, respectively. Subsequently, the main control computing unit performs a Hadamard product operation (element-by-element multiplication) on the original fused image and the binary mask matrix. Specifically, the main control computing unit retains the pixel values ​​at the locations of pixels with a mask value of the first logical value in the original fused image, while setting the pixel values ​​at the locations of pixels with a mask value of the second logical value to zero, thereby obtaining the tactile feature image. This tactile feature image retains only the image information within the non-transparent annular regions of the multi-layered concentric circle encoded elastic skin, namely the imaging data of the speckle features on the surface of the first, third, and fifth annular regions.

[0047] Simultaneously, the main control computing unit performs a Hadamard product operation on the logical complement matrix of the original fused image and the binary mask matrix. The logical complement matrix is ​​the matrix obtained by interchanging the first and second logical values ​​in the binary mask matrix. The main control computing unit retains the pixel values ​​at the locations of pixels with mask values ​​of the second logical value in the original fused image, while setting the pixel values ​​at the locations of pixels with mask values ​​of the first logical value to zero, thereby obtaining an environmental visual feature image. This environmental visual feature image only retains the image information within the transparent annular regions of the multi-layered concentric circle coded elastic skin, namely the imaging data of the external environmental field of view corresponding to the second and fourth annular regions.

[0048] The above decoupling operation can be completed in a very short time after the image data is read into memory, because the Hadamard product operation is a simple element-wise arithmetic operation that can be executed with extremely high efficiency at the underlying hardware level of the main control computing unit.

[0049] In one example, the forward generation formula for the original fused image of the multimodal images is: .

[0050] in, This represents the original multimodal mixed image (fused image tensor) captured by a monocular camera. The physical binary mask matrix representing spatial reuse (in this application, it is a 5-layer nonlinear concentric circle mask M(r)). This represents the extracted tactile feature image tensor containing real physical deformations. It represents the visual feature tensor after being affected by internal light source modulation and tactile deformation interference. This represents the Hadamard product operator, which represents element-wise multiplication.

[0051] In one example, the nonlinear coupling relationship of the visual features of the contact area modulated by the illumination within the tactile zone is as follows: .

[0052] in, This represents the visual feature tensor that was actually captured and subjected to strong light modulation interference within the tactile zone. A physical mask representing cross-modal illumination contamination (characterizing the proportion of internal tactile light leaking into the external transparent visual area). It represents the visual tensor of a pure environment under ideal conditions that is completely unaffected by internal light sources. This represents the illumination distribution coefficient tensor of the external natural environment normalized to the [0,1] interval. This represents the internal light source modulation coefficient tensor normalized to the [0,1] interval.

[0053] Step S4: Perform the following two feature extractions in parallel: Perform optical flow tracing processing on the tactile feature image to extract the displacement field of feature points caused by contact force on the multi-layer concentric circle encoded elastic skin surface covering the top of the monocular imaging module's field of view, and calculate the three-dimensional contact force vector of the multi-layer concentric circle encoded elastic skin surface based on the displacement field; process the environmental visual feature image to generate a standard panoramic environmental image, and extract the image contour and centroid position information of the target object.

[0054] Regarding optical flow tracing processing of tactile feature images, the main control computing unit uses the local image window containing the speckle feature points in the tactile feature image as the basic processing unit, and constructs optical flow constraint equations based on the gray-level invariance assumption. This gray-level invariance assumption holds that the gray value of the same speckle feature point remains unchanged between different frames within a short time. Based on this assumption, the main control computing unit establishes the positional change of the speckle feature point in two adjacent frames of the tactile feature image as a linear equation concerning the image spatial gradient and temporal gradient. For multiple pixels within a single local image window, the main control computing unit simultaneously forms multiple optical flow constraint equations, constituting an overdetermined system of equations. Subsequently, the main control computing unit uses the least squares method to solve this overdetermined system of equations to obtain the two-dimensional pixel displacement vector of the speckle feature point corresponding to the local image window on the image plane.

[0055] The main control computing unit performs the above-described solution process on each speckle feature point in the tactile feature image to obtain a two-dimensional pixel displacement field describing the surface deformation of the entire tactile sensing area. Based on this, the main control computing unit calls the pixel-physical scale mapping coefficients pre-programmed into non-volatile memory to convert the pixel displacement vectors of each speckle feature point into physical displacement vectors of the elastic skin surface. These pixel-physical scale mapping coefficients are obtained by linearly fitting calibration patterns with known physical spacing arranged on a multi-layer concentric circle encoded elastic skin surface and collecting the pixel coordinates of these calibration patterns in the tactile feature image. Finally, the main control computing unit inputs the physical displacement vectors of each speckle feature point into a preset contact mechanics model. This model includes a nonlinear hyperelastic material contact equation describing the relationship between the normal compressive deformation and normal contact force of the elastic skin, and a shear strain model describing the relationship between the tangential displacement and transverse shear force of the elastic skin, thereby calculating the three-dimensional contact force vector of the multi-layer concentric circle encoded elastic skin surface.

[0056] Regarding the processing of environmental visual feature images, the main control computing unit first extracts the effective pixel band corresponding to at least one transparent annular region in the environmental visual feature image, that is, extracts the pixel data corresponding to the second and fourth annular regions in the image. Since these effective pixel bands are distributed in an annular shape, they exhibit perspective distortion and cannot be directly used for target object recognition and localization. Therefore, the main control computing unit uses the inner and outer diameter boundaries of the effective pixel bands as constraints to perform an inverse transformation mapping from polar coordinates to rectangular coordinates on each pixel within the effective pixel band. Specifically, the main control computing unit unfolds the annularly distributed effective pixel band along two dimensions: angle and radius. The angle dimension is mapped to the horizontal coordinates of the unfolded image, and the radius dimension is mapped to the vertical coordinates of the unfolded image, thus unfolding the annular effective pixel band into a rectangular standard panoramic environmental image. This standard panoramic environmental image eliminates the barrel distortion inherent in ultra-wide-angle lenses, and the target objects within it exhibit normal geometric shapes that conform to human visual perception.

[0057] After obtaining a standard panoramic environment image, the main control computing unit calls the edge detection operator on the image to extract the geometric contour of the target object, and calculates the pixel coordinates of the centroid of the target object in the standard panoramic environment image based on the pixel distribution of the contour.

[0058] Step S5: Determine whether the surface of the multi-layer concentric circle coded elastic skin has deformed beyond the preset pixel displacement threshold; if not, activate the visual servo mode, generate a pose alignment control command based on the deviation between the centroid position information of the target object and the preset desired position, and send it to the motor drive board to drive the multi-finger dexterous hand to approach the target object; if yes, switch to the haptic force control mode, generate a force tracking control command based on the deviation between the three-dimensional contact force vector and the preset target force vector, and send it to the motor drive board to drive the multi-finger dexterous hand to apply a constant envelope clamping force to the target object.

[0059] The main control computing unit reads the two-dimensional pixel displacement vectors of each speckle feature point on the image plane from the feature point displacement field extracted in step S4, and calculates the magnitude of each two-dimensional pixel displacement vector. The magnitude is calculated by squaring and taking the square root. The main control computing unit compares the magnitude of the displacement vector of each speckle feature point with a preset pixel displacement threshold. If the magnitude of the displacement vector of at least one speckle feature point exceeds the preset pixel displacement threshold, the main control computing unit determines that the surface of the multi-layer concentric circle coded elastic skin has undergone deformation exceeding the preset threshold, that is, the mechanical finger has made contact with the target object and caused measurable deformation of the elastic skin; if the magnitude of the displacement vector of all speckle feature points does not exceed the preset pixel displacement threshold, the main control computing unit determines that the surface of the multi-layer concentric circle coded elastic skin has not undergone deformation exceeding the preset threshold, that is, the mechanical finger has not made effective contact with the target object.

[0060] When the main control computing unit determines that the deformation has not exceeded the preset threshold, the system is in the approach phase, at which point the visual servo mode is activated. In visual servo mode, the main control computing unit generates pose alignment control commands based on the deviation between the centroid position information of the target object extracted in step S4 and the preset desired position, combined with the image Jacobian matrix. Specifically, the main control computing unit first calculates the difference between the current pixel coordinates of the target object's centroid and the preset desired pixel coordinates to obtain the image feature error vector. Subsequently, the main control computing unit calculates the pseudo-inverse matrix of the image Jacobian matrix based on the current joint angle configuration of the mechanical finger. This pseudo-inverse matrix establishes a mathematical mapping relationship between the pixel deviation on the two-dimensional image plane and the movement speed of each joint of the mechanical finger in three-dimensional space. The main control computing unit multiplies the image feature error vector by the pseudo-inverse matrix of the image Jacobian matrix and multiplies it by a preset proportional gain coefficient to obtain the angular velocity control commands for each joint. These control commands are sent to the motor drive board via the controller local area network bus, driving each joint of the multi-fingered dexterous hand to approach the target object at the calculated speed until the centroid of the target object gradually approaches the center position of the image.

[0061] In one example, the image-based visual servoing control law is: .

[0062] in, This represents the target generalized motion velocity vector of each joint of the robotic finger or the robotic arm equipped with it. For rotary joints, the corresponding component is in radians per second (rad / s); for translational joints, the corresponding component is in meters per second (m / s). λ is a positive definite scalar proportional gain coefficient used to adjust the system's response speed to visual errors. The dimension of this coefficient is Hertz (Hz), and its specific value needs to be determined through closed-loop stability analysis based on the sampling period and the numerical range of the image Jacobian matrix, and is usually pre-tuned during engineering debugging. This is represented as the pseudo-inverse of the image Jacobian matrix. This matrix establishes a mapping between 2D image pixel deviations and 3D joint angular velocities. The dimensions of its elements vary depending on the joint type: for rotary joints, the corresponding column elements are in radians per pixel (rad / pixel); for translational joints, the corresponding column elements are in meters per pixel (m / pixel). The value of this matrix depends on the current joint configuration q of the robotic arm or finger and the current feature coordinates s of the target in the image. It needs to be calculated online in each control cycle based on the kinematic model and camera projection model. The pseudo-inverse matrix is ​​obtained using the standard Moore-Penrose singular value decomposition method. Represented as the current time from the standard two-dimensional environment image I panoThe centroid coordinates of the target object pixels extracted from (u,v). The pixel coordinates are used to center-align the desired target object in the image. Both are dimensionless pixel coordinate vectors, and their values ​​depend on the image resolution and the projected position of the target on the imaging plane. The difference between the two is... This constitutes the visual pixel contour error vector at the current moment.

[0063] In one example, upon entering the haptic force control mode, the generation of master control commands follows a force control law. In this application, the force control law is specifically implemented using an explicit force control law. Its core idea is to map the deviation between the haptic-perceived three-dimensional contact force vector and the preset target force vector into a target torque output in joint space via a proportional-derivative (PD) force error adjuster. The specific formula for the force control law is: .

[0064] Where, τ cmd This indicates the output torque control command vector of each joint of a multi-point gripper. This represents the transpose of the kinematic Jacobian matrix of the multi-fingered manipulator in the current pose, and is responsible for mapping the three-dimensional contact forces in Cartesian space to each motor-driven joint. This represents the preset force error proportional gain matrix. This represents the preset force error differential gain matrix. This represents the set force vector for the safe grasping target. This represents the actual three-dimensional contact force vector. This represents the rate of change of the contact force over time, providing dynamic damping to prevent rigid impacts and oscillations during the grasping process. g(q) represents the torque vector compensating for gravity and friction.

[0065] When the main control unit determines that deformation has exceeded a preset threshold, the system enters the contact phase, switching to tactile force control mode. In tactile force control mode, the main control unit no longer relies on environmental visual features for pose servoing, but instead performs force closed-loop control entirely based on the three-dimensional contact force vector calculated in step S4. The main control unit compares the current three-dimensional contact force vector with the preset target force vector, calculating the difference as the force control error. Subsequently, based on the preset force control stiffness matrix and force control damping matrix, combined with the force control error and the time change rate of the contact force, the main control unit calculates the torque control command that each joint motor needs to output. This torque control command is sent to the motor drive board via the controller local area network bus, driving the multi-finger dexterous hand to apply a constant envelope clamping force to the target object. As the actual contact force gradually approaches the preset target force, the force control error approaches zero, and the torque control command converges to a constant value. At this point, the motor drive board automatically enters a constant torque output state, ensuring that the multi-finger dexterous hand continuously envelops the target object with a preset safe clamping force.

[0066] In one possible embodiment, during the execution of step S2, the main control computing unit reads the polar radius thresholds of multiple annular boundaries of the multi-layer concentric circle coded elastic skin from the pre-programmed physical mask model. This physical mask model is pre-stored in the non-volatile memory of the main control computing unit, and its data content is calibrated and programmed after the dexterous hand device is assembled. Since the five annular regions of the multi-layer concentric circle coded elastic skin have fixed physical geometric dimensions, and the optical parameters of the monocular imaging module remain constant after manufacturing, the polar radius thresholds corresponding to the boundaries of each annular region on the image plane are fixed values ​​after calibration and do not need to be recalculated in each grasping task.

[0067] It should be noted that a multi-layered concentric circle coded elastic skin covers the top of the monocular imaging module's field of view. Its surface is composed of alternating radially arranged, concentrically arranged opaque and transparent annular regions. Specifically, from the inside out, the annular regions are: the first annular region (central primary tactile sensing area), the second annular region (near-field environment visual window), the third annular region (auxiliary tactile sensing area), the fourth annular region (mid-to-far-field environment visual window), and the fifth annular region (edge ​​tactile sensing area). The first, third, and fifth annular regions are opaque, coated with a high-contrast speckle feature layer to allow the imaging module to capture displacement changes of feature points when deformed by a target object. The second and fourth annular regions are transparent, allowing ambient light to pass through and reach the bottom of the imaging module. There are six pre-programmed annular boundary polarity thresholds, representing the polarity of the boundary line between two adjacent annular regions on the image plane. These six polarity thresholds divide the image plane into five consecutive polarity intervals from the inside out, each interval corresponding to a specific annular region.

[0068] After each frame of raw fused image data is read into the memory of the main control computing unit, the unit establishes an image physical coordinate system with the optical center of the raw fused image as the origin. This optical center refers to the intersection of the lens optical axis of the monocular imaging module and the image sensor plane, which is usually located near the geometric center of the image pixel matrix in a real image. After establishing the image physical coordinate system with the optical center as the origin, the main control computing unit calculates the polar radius value from each pixel in the raw fused image to this optical center origin. The polar radius value is calculated using the Euclidean distance formula, that is, the polar radius of a pixel is equal to the square root of the sum of the squares of the x-coordinate offset and the squares of the y-coordinate offset of that pixel.

[0069] After calculating the extreme radius value of each pixel, the main control unit compares this value with multiple annular boundary extreme radius thresholds read from the physical mask model to determine which annular region the pixel belongs to. Since the six annular boundary extreme radius thresholds divide the entire extreme radius value range into five consecutive intervals from smallest to largest, and these five intervals correspond to the first, second, third, fourth, and fifth annular regions respectively in ascending order, the annular region to which the pixel belongs can be uniquely determined by comparing the pixel's extreme radius value with these six thresholds.

[0070] If the extreme radius value of a pixel falls within the extreme radius interval corresponding to a non-transparent annular region, i.e., within the interval corresponding to the first, third, or fifth annular region, the main control calculation unit sets the mask value of that pixel to the first logical value. If the extreme radius value of a pixel falls within the extreme radius interval corresponding to a transparent annular region, i.e., within the interval corresponding to the second or fourth annular region, the main control calculation unit sets the mask value of that pixel to the second logical value. The first and second logical values ​​are opposite binary values, for example, the first logical value is 1 and the second logical value is 0, or the first logical value is 0 and the second logical value is 1. The specific values ​​are determined by preset encoding rules, as long as the tactile channel preservation region and the visual channel preservation region can be accurately distinguished in the subsequent Hadamard product operation.

[0071] The main control computing unit traverses all pixels of the original fused image in the manner described above. After performing polar radius calculation, threshold comparison, and mask value assignment for each pixel, it constructs a binary mask matrix identical in both rows and columns to the original fused image, based on the mask values ​​of all pixels and their original arrangement positions in the original fused image. In this binary mask matrix, the set of pixel positions with the first logical value corresponds exactly to the projection area of ​​all non-transparent annular regions in the multi-layer concentric circle coded elastic skin onto the image plane, while the set of pixel positions with the second logical value corresponds exactly to the projection area of ​​all transparent annular regions onto the image plane.

[0072] In one possible embodiment, after the binary mask matrix is ​​generated in step S2, it is stored in the memory of the main control computing unit, awaiting invocation in step S3. Each element of the binary mask matrix corresponds one-to-one with a pixel in the original fused image. Elements with a first logical value indicate that the corresponding pixel belongs to the effective area of ​​the tactile channel, that is, the content imaged by the pixel comes from the non-transparent annular area in the multi-layer concentric circle coded elastic skin; elements with a second logical value indicate that the corresponding pixel belongs to the effective area of ​​the visual channel, that is, the content imaged by the pixel comes from the transparent annular area in the multi-layer concentric circle coded elastic skin.

[0073] When step S3 begins, the main control computing unit first allocates two independent image buffers in memory. The first buffer is used to store tactile feature images, and the second buffer is used to store environmental visual feature images. The number of rows and columns in both buffers are exactly the same as the original fused image to ensure that the spatial resolution of each feature image remains consistent in subsequent image processing steps.

[0074] After allocating the two image buffers, the main control computing unit performs element-wise separation operations on each pixel position of the original fused image. Specifically, the main control computing unit reads the mask value at the current pixel position in the binary mask matrix and determines whether the mask value is equal to the first logical value or the second logical value.

[0075] If the binary mask value at the current pixel location is equal to the first logical value, it indicates that the pixel location belongs to the reserved location of the tactile channel. The main control computing unit copies the original pixel value at that pixel location in the original fused image to the same pixel location in the tactile feature image buffer. At the same time, the main control computing unit sets the pixel value at the same pixel location in the environmental visual feature image buffer to zero.

[0076] If the binary mask value at the current pixel location is equal to the second logical value, it indicates that the pixel location belongs to the reserved location of the visual channel. The main control computing unit copies the original pixel value at that pixel location in the original fused image to the same pixel location in the environmental visual feature image buffer. At the same time, the main control computing unit sets the pixel value at the same pixel location in the tactile feature image buffer to zero.

[0077] The mathematical essence of the aforementioned decoupling operation is a double Hadamard product operation: element-wise multiplication between the binary mask matrix and the original fused image, and element-wise multiplication between the logical complement of the binary mask matrix and the original fused image. The logical complement is obtained by swapping the first and second logical values ​​in the binary mask matrix. In actual hardware implementations, the main control unit typically employs a single-instruction multiple-data parallel processing approach, simultaneously performing the aforementioned judgment and assignment operations on multiple pixel positions, thereby significantly improving the efficiency of the decoupling operation.

[0078] The main control computing unit traverses all pixel positions of the original fused image in the manner described above. After traversing all pixel positions, the image data stored in the tactile feature image buffer is the decoupled tactile feature image, and the image data stored in the environmental visual feature image buffer is the decoupled environmental visual feature image.

[0079] The tactile feature image obtained after the above element-by-element separation operation retains only the image information of the non-transparent annular regions in the multi-layer concentric circle coded elastic skin. In other words, all the effective pixels of the tactile feature image come from the projection areas of the first, third, and fifth annular regions onto the photosensitive chip of the imaging module. In these effective regions, the imaging of the high-contrast speckle feature coating sprayed on the surface of the non-transparent annular regions can be clearly observed. When a target object presses against the surface of the multi-layer concentric circle coded elastic skin, these speckle features will produce displacements in the image plane corresponding to the magnitude and direction of the contact force. This displacement information will be used by the subsequent optical flow tracing algorithm. In the tactile feature image, the pixel positions that originally corresponded to the second and fourth annular regions are all set to zero values. Therefore, these regions appear as completely black invalid regions in the tactile feature image and will not cause any interference to the subsequent tactile feature extraction.

[0080] Meanwhile, the environmental visual feature image obtained after the above element-by-element separation operation retains only the image information within the transparent annular regions of the multi-layered concentric circle coded elastic skin. In other words, all the effective pixels of the environmental visual feature image originate from the projection areas of the second and fourth annular regions onto the image sensor chip of the imaging module. In these effective regions, light from the external environment can pass through the transparent support plate and the transparent annular regions to reach the imaging module at the bottom, thus allowing for a clear observation of the optical image of the target object and its surrounding environment. In the environmental visual feature image, the pixel positions originally corresponding to the first, third, and fifth annular regions are all set to zero values. Therefore, these regions appear as completely black invalid areas in the environmental visual feature image and will not interfere with subsequent environmental visual feature extraction.

[0081] In one possible embodiment, after the tensor separation operation is completed in step S3, the main control computing unit stores two independent data streams in its memory: a tactile feature image and an environmental visual feature image. The effective pixels of the environmental visual feature image are distributed only at the image positions corresponding to the transparent annular regions of the multi-layered concentric circle encoded elastic skin, including the second annular region (projection area of ​​the near-field environmental visual window) and the fourth annular region (projection area of ​​the mid-to-far-field environmental visual window). Since these two annular regions are physically circular, their projection onto the image sensor chip is also circular. Therefore, when directly observing the environmental visual feature image, the effective information appears as two concentrically arranged donut shapes, while the invalid areas, corresponding to the non-transparent annular regions, are all zero values ​​and appear black. This annular distribution introduces severe perspective distortion, making it unsuitable for direct object recognition and localization.

[0082] In one example, the environmental visual image feature extraction model in a complex background is as follows: .

[0083] in, This represents the original environmental visual image stream matrix acquired against a complex background. This represents a binary indicator mask used to distinguish foreground objects from the background in an environmental image. This represents the pure environment background image tensor. A pure visual image tensor representing the approaching target object (foreground).

[0084] The main control computing unit first extracts the effective pixel bands corresponding to at least one transparent annular region from the environmental visual feature image. In a preferred embodiment, the main control computing unit simultaneously extracts the effective pixel bands corresponding to two transparent annular regions, the second and fourth annular regions, to obtain environmental information with a wider field of view. When extracting the effective pixel bands, the main control computing unit determines the inner and outer diameter boundaries of each transparent annular region on the image plane based on the annular boundary extreme diameter thresholds in the pre-programmed physical mask model. Specifically, the effective pixel band of the second annular region is located between its inner and outer diameter extreme diameter thresholds, and the effective pixel band of the fourth annular region is located between its inner and outer diameter extreme diameter thresholds. The main control computing unit retains only the pixel data within these effective pixel bands, discarding the pixel data of other regions, thereby obtaining the annular effective pixel bands to be unfolded.

[0085] After extracting the effective pixel band, the main control computing unit uses the inner and outer diameter boundaries of the effective pixel band as geometric constraints to perform an inverse polar coordinate to Cartesian coordinate transformation mapping on each pixel within the effective pixel band, unfolding the annularly distributed effective pixel band into a rectangular standard panoramic environment image. This inverse transformation mapping essentially converts the pixel positions on the image plane from polar coordinates with the optical center as the origin to Cartesian coordinates with the upper left corner of the standard panoramic environment image as the origin. The main control computing unit first sets the width and height parameters of the unfolded standard panoramic environment image. The width setting determines the number of sampling points in the angular direction of the unfolded image, and is usually set to the average perimeter of the effective pixel band to ensure that the horizontal resolution of the unfolded image is comparable to the original annular image. The height setting is equal to the difference between the outer and inner diameters of the effective pixel band, i.e., the radial pixel width of the effective pixel band. This width has been pre-compensated by isometric projection mapping during the design phase to have a fixed pixel value.

[0086] After parameter settings are completed, the main control computing unit establishes a mapping relationship from the coordinates of any pixel in the standard panoramic environment image to the corresponding pixel coordinates in the original environment visual feature image. For a pixel in the standard panoramic environment image located in the u-th column horizontally and the v-th row vertically, the main control computing unit first calculates the polar coordinate angle value corresponding to the pixel. This angle value is equal to the horizontal column number divided by the total width of the panoramic image and then multiplied by the circumference angle. The main control computing unit also calculates the polar coordinate radius value corresponding to the pixel. This radius value is equal to the inner diameter of the effective pixel band plus the vertical row number divided by the total height of the panoramic image and then multiplied by the difference between the outer diameter and the inner diameter of the effective pixel band. After obtaining the angle and radius values, the main control computing unit finds the pixel position in the original environment visual feature image with the optical center as the origin, along the angular direction, at a distance of the radius value from the origin. The horizontal coordinate of this pixel position is equal to the horizontal coordinate of the optical center plus the cosine function of the radius value multiplied by the angle value, and the vertical coordinate is equal to the vertical coordinate of the optical center plus the sine function of the radius value multiplied by the angle value. The original pixel value at that pixel location is then filled into the pixel position in the v-th row and u-th column of the standard panoramic environment image.

[0087] According to the above mapping relationship, the main control computing unit traverses every pixel position of the standard panoramic environment image, picking up the corresponding pixel value from the original environmental visual feature image point by point and filling it in, thus completing the expansion of the entire annular effective pixel band. The rectangular standard panoramic environment image generated after the above inverse transformation mapping from polar coordinates to rectangular coordinates eliminates the perspective distortion and barrel distortion caused by the original annular distribution. The target objects in the image present normal geometric shapes that conform to human visual habits, the straight line features of the object edges are restored, and the relative positional relationship between adjacent objects is also restored to the actual spatial layout in physical space.

[0088] After obtaining a standard panoramic environment image, the main control computing unit invokes an edge detection operator on this image to extract the geometric contours of the target object. The edge detection operator identifies pixel locations in the standard panoramic environment image where grayscale values ​​change drastically. These pixel locations typically correspond to the outer boundary of the target object, the edge of a surface texture, or the boundary between the object and the background. In a preferred embodiment of this application, the main control computing unit uses the Cannibal edge detection operator to perform edge detection. This operator effectively extracts the complete closed contours of the target in the image while suppressing false edges caused by noise through steps such as Gaussian filtering to smooth the image, calculating gradient magnitude and direction, non-maximum suppression, and double thresholding.

[0089] After extracting the geometric contour of the target object, the main control computing unit calculates the centroid of the target object in the standard panoramic environment image based on the coordinate distribution of all pixels within the area enclosed by the contour. The centroid coordinates are calculated using the image moment method. The main control computing unit first calculates the sum of the horizontal and vertical coordinates of all pixels within the contour area. Then, it divides the sum of the horizontal coordinates by the total number of pixels in the contour area to obtain the horizontal centroid coordinates, and divides the sum of the vertical coordinates by the total number of pixels in the contour area to obtain the vertical centroid coordinates. These centroid pixel coordinates represent the center position of the target object in the standard panoramic environment image, serving as the benchmark for calculating image feature errors in the subsequent visual servo control law.

[0090] In one possible embodiment, after the tensor separation operation is completed in step S3, the main control computing unit stores two independent data streams in its memory: a tactile feature image and an environmental visual feature image. The effective pixels of the tactile feature image are distributed only at the image positions corresponding to the non-transparent annular regions of the multi-layered concentric circle encoded elastic skin, including the first annular region (projection area of ​​the central main tactile sensing area), the third annular region (projection area of ​​the auxiliary tactile sensing area), and the fifth annular region (projection area of ​​the edge tactile sensing area). These non-transparent annular regions are coated with a high-contrast speckle feature coating, which appears in the tactile feature image as a large number of randomly distributed bright and dark speckle particles. When a target object presses against the surface of the elastic skin, these speckle particles will move accordingly with the deformation of the elastic skin. By tracking the positional changes of these speckle particles in multiple consecutive frames, the deformation field of the elastic skin surface can be deduced, and the distribution of the contact force can be calculated.

[0091] In one example, the formula for extracting the contact-induced real tactile deformation difference tensor is: .

[0092] in, The tensor representing the difference in real tactile deformation caused by contact is used to characterize the relative displacement and changes in light and shadow before and after the force is applied. This represents the tensor of the raw tactile image captured in real time from the contact target object. This represents the pre-calibrated initial background tactile image tensor in the unloaded state when not in contact with the target object.

[0093] When the main control computing unit performs optical flow tracing on the tactile feature images, it uses the local image window containing the speckle feature points as the basic processing unit. Each speckle feature point is surrounded by a local image window, the size of which is typically set to an odd-numbered pixel block, such as 7 pixels by 7 pixels or 9 pixels by 9 pixels, so that the speckle feature point is exactly located at the center of the window. The main control computing unit identifies all speckle feature points in each frame of the tactile feature image and tracks the new positions of these feature points in the next frame. To achieve tracking, the main control computing unit constructs an optical flow constraint equation based on the gray-level invariance assumption. This gray-level invariance assumption means that the gray value of the same speckle feature point does not change within the acquisition interval between two adjacent frames; that is, the gray value of the feature point in the previous frame is equal to its gray value in the next frame. The assumption of grayscale invariance has good applicability in tactile perception scenarios because the tactile feature images are acquired under stable internal light source illumination conditions controlled by the main control computing unit, and there is no influence of external ambient light changes on the grayscale values ​​of speckle features.

[0094] Based on the assumption of grayscale invariance, the main control computing unit establishes a Taylor expansion of the grayscale value for each pixel in the local image window. After ignoring higher-order terms (second order and above), a linear equation containing three unknowns is obtained. These unknowns are the optical flow velocity in the horizontal direction, the optical flow velocity in the vertical direction, and the change in grayscale value over time. The change in grayscale value over time can be directly calculated from the grayscale difference between two consecutive frames; therefore, the equation actually contains two unknowns: the horizontal optical flow velocity and the vertical optical flow velocity. Since a single equation cannot uniquely determine two unknowns, the main control computing unit sets up a system of overdetermined equations for the optical flow constraints corresponding to all pixels within a local image window. The number of equations in this overdetermined system equals the total number of pixels within the local image window. For example, for a 7-pixel by 7-pixel local image window, the overdetermined system contains 49 equations, while there are only two unknowns; therefore, this system of equations is highly overdetermined.

[0095] After obtaining the overdetermined system of equations formed by multiple local image windows, the main control computing unit solves the overdetermined system of equations using the least squares method. The basic idea of ​​the least squares method is to find a set of estimated optical flow velocities that minimizes the sum of squares of the errors between the calculated values ​​on the left side and the actual values ​​on the right side of all equations. The main control computing unit writes the overdetermined system of equations in matrix form and then solves for the least squares solution of the overdetermined system of equations through matrix operations. Specifically, the main control computing unit calculates the sum of the squares of the horizontal gray-level gradients, the sum of the squares of the vertical gray-level gradients, and the sum of the products of the horizontal and vertical gray-level gradients of all pixels within the local image window, forming a 2x2 coefficient matrix; at the same time, it calculates the negative values ​​of the sum of the products of the horizontal and temporal gray-level gradients and the negative values ​​of the sum of the products of the vertical and temporal gray-level gradients, forming a 2x1 constant vector. The main control computing unit then solves the 2x2 linear equation system to obtain the two-dimensional pixel displacement vector of the speckle feature point at the center of the local image window on the image plane. The two components of this vector represent the pixel displacement of the feature point in the horizontal and vertical directions, respectively.

[0096] The main control computing unit performs the aforementioned least-squares solution process on each speckle feature point in the tactile feature image, obtaining a two-dimensional pixel displacement vector for each speckle feature point on the image plane. The two-dimensional pixel displacement vectors of all speckle feature points are arranged and combined according to their spatial positions, thus forming a two-dimensional pixel displacement field describing the deformation of the entire tactile sensing area. This displacement field reflects the deformation distribution of the multi-layered concentric circle coded elastic skin surface under the contact pressure of a target object; areas with larger displacement correspond to locations with larger contact forces, while areas with smaller displacement correspond to locations with smaller contact forces or no force.

[0097] After obtaining the two-dimensional pixel displacement vectors of each speckle feature point, the main control computing unit needs to convert these pixel-level image planar displacements into physical displacements on the elastic skin surface in millimeters. To achieve this conversion, the main control computing unit calls the pixel-physical scale mapping coefficient pre-programmed into non-volatile memory. This pixel-physical scale mapping coefficient is a pre-calibrated scaling factor, whose physical meaning is how many millimeters of physical displacement on the elastic skin surface correspond to one pixel on the image plane. Since the multi-layer concentric circle coded elastic skin is a planar structure closely attached to the top of the imaging module's field of view, and the geometric dimensions of each annular region are precisely determined during manufacturing, this pixel-physical scale mapping coefficient is a fixed value after calibration and does not change with different grasping tasks. The main control computing unit multiplies the horizontal and vertical components of the two-dimensional pixel displacement vectors of each speckle feature point by the pixel-physical scale mapping coefficient to obtain the horizontal and vertical physical displacement vectors of each speckle feature point on the multi-layer concentric circle coded elastic skin surface.

[0098] After obtaining the physical displacement vectors of each speckle feature point, the main control computing unit inputs these physical displacement vectors into a preset contact mechanics model to calculate the three-dimensional contact force vector of the multi-layer concentric circle encoded elastic skin surface. This contact mechanics model is a mathematical model pre-established and stored in the main control computing unit's memory, used to describe the quantitative relationship between the physical displacement of the elastic skin and the applied contact force. The model consists of two main components. The first part describes the relationship between the normal contact force and the normal compressive deformation depth of the elastic skin, i.e., the nonlinear hyperelastic material contact equation between the depth of skin compression and the applied normal pressure when the target object presses the skin in a direction perpendicular to the elastic skin surface. This equation includes a linear stiffness coefficient and a nonlinear hardening coefficient, corresponding to the stiffness characteristics of the elastic skin under small and large deformation conditions, respectively. The second part describes the relationship between the tangential contact force and the tangential displacement of the elastic skin, i.e., the shear strain model between the tangential displacement of the speckle feature points on the skin surface and the applied shear force when the target object tends to slide in a direction parallel to the elastic skin surface. This model includes parameters such as the equivalent shear modulus, elastic skin thickness, and effective contact area. The main control computing unit substitutes the normal component of the physical displacement vector of each speckle feature point into the normal contact force equation to obtain the value of the normal contact force; it also substitutes the tangential component of the physical displacement vector into the shear strain model to obtain the value of the tangential shear force. The normal contact force and the tangential shear force are combined into a three-dimensional contact force vector, which fully describes the magnitude and direction of the spatial force exerted by the target object on the surface of the elastic skin.

[0099] In one example, the main control computing unit performs optical flow tracing processing step S4 on all speckle feature points in the tactile feature image one by one to obtain a two-dimensional pixel displacement field describing the surface deformation of the entire tactile sensing area.

[0100] Based on this, the main control computing unit calls the pixel-physical scale mapping coefficients pre-programmed into non-volatile memory to convert the pixel displacement vectors of each speckle feature point into the physical displacement vectors of the elastic skin surface. These pixel-physical scale mapping coefficients are obtained by linear fitting after arranging calibration patterns with known physical spacing on a multi-layered concentric circle coded elastic skin surface and acquiring the pixel coordinates of the calibration patterns in the tactile feature image. Specifically, during calibration, multiple concentric circular calibration rings are printed on the elastic skin surface at equal physical radial spacing. The pixel radius of each calibration ring in the image is measured, and the known physical radius of each calibration ring is divided by the corresponding pixel radius to obtain the local mapping coefficient at that radial position. Radial interpolation is performed on the local mapping coefficients at each calibration ring position to obtain a mapping coefficient table continuously distributed along the polar radius direction. During conversion, the main control computing unit reads the corresponding mapping coefficient from the mapping coefficient table based on the polar radius coordinates of each speckle feature point to the optical center of the image, and multiplies this coefficient by the two components of the pixel displacement vector to obtain the two components of the physical displacement vector.

[0101] Finally, the main control computing unit inputs the physical displacement vectors of each speckle feature point into a preset contact mechanics model. This contact mechanics model contains two sub-models: one is a nonlinear hyperelastic material contact equation describing the relationship between the normal compressive deformation and normal contact force of the elastic skin; the other is a shear strain model describing the relationship between the tangential displacement and transverse shear force of the elastic skin. The main control computing unit completes the calculation of the three-dimensional contact force vector according to the following steps.

[0102] The first step is to calculate the normal deformation depth. The main control computing unit tracks the pixel area change of the local window containing each speckle feature point in the tactile feature image. Let the pixel area of ​​the local window of a certain speckle feature point in the idle state be denoted as... The local window pixel area under the current stress state is Calculate the area magnification. Based on the assumption that elastic skin materials are volumetrically incompressible, the normal compression deformation depth... Calculate using the following formula: h skin Δz represents the physical thickness of the elastic skin in an unstressed state, and is a preset constant, with the unit being millimeters (mm). When α>1, Δz takes a positive value, indicating that the elastic skin is compressed by the target object and produces a normal indentation.

[0103] The second step is to calculate the normal contact force. The main control calculation unit substitutes the Δz obtained in the first step into the nonlinear hyperelastic material contact equation: Where k1 is the linear stiffness coefficient, with units of Newtons per millimeter (N / mm); k2 is the nonlinear hardening coefficient, with units of Newtons per cubic millimeter (N / mm³), both of which are preset constants, obtained by performing uniaxial compression tests on elastic skin samples and fitting force-displacement curves. N,skin This is the normal contact force, and its unit is Newton (N).

[0104] When multiple speckle feature points exist within the tactile sensing area, the main control computing unit can calculate the normal deformation depth at each feature point and substitute it into the above equation to obtain the normal contact force at each point. Then, the arithmetic mean of the measured normal contact forces at all feature points within the contact area is taken as the reported F at that moment. N,skin Alternatively, the measured value of a single feature point at the center of the contact area can be pre-set as the reporting value according to the actual control accuracy requirements. Both of the above selection methods are pre-programmed into non-volatile memory.

[0105] The third step is to calculate the transverse shear force and its components. The main control computing unit calculates the physical displacement vectors of each speckle feature point obtained from the calculation. Calculate transverse shear force based on shear strain model : .

[0106] in, The equivalent shear modulus of elastic skin material is expressed in megapascals (MPa) or newtons per square millimeter (N / mm²), and is obtained by performing standard shear tests on elastic skin samples. Let Δx be the magnitude of the physical displacement vector, representing the distance a feature point deviates from its initial position along a horizontal direction on the elastic skin surface. Δy represents the distance the same feature point deviates from its initial position along another direction perpendicular to the aforementioned horizontal direction on the elastic skin surface, in millimeters (mm). The effective contact area between the target object and the elastic skin at the current moment is expressed in square millimeters (mm²). It is obtained by the main control computing unit through the image mask segmentation algorithm, which counts the number of pixels in the contact area in the tactile feature image and multiplies it by the physical area corresponding to each pixel.

[0107] Subsequently, the main control computing unit will determine the direction angle of the physical displacement vector. It can be decomposed into two orthogonal tangential components, each component being in Newtons (N): , .

[0108] like ,but .

[0109] When multiple speckle feature points exist within the tactile sensing area, the main control computing unit can calculate the transverse shear force and its component at each feature point, and then take the arithmetic mean of the transverse shear force measurements at all feature points within the contact area as the reported F at that moment. shear Alternatively, the median of the measured values ​​of several feature points at the center of the contact area can be pre-set as the reporting value according to the actual control accuracy requirements. Both of the above selection methods are pre-programmed into non-volatile memory.

[0110] The fourth step is to synthesize the three-dimensional contact force vector. The main control computing unit will then use the normal contact force obtained in the second step. The tangential component obtained in the third step , Combined into a three-dimensional contact force vector : Each component in this vector is calculated at the same time and together constitutes the contact force feedback value reported to the tactile force control closed loop within the current control cycle.

[0111] In one possible embodiment, when the main control computing unit determines that the multi-layer concentric circle coded elastic skin surface has deformed beyond a preset pixel displacement threshold and switches to haptic force control mode, the main control computing unit continuously executes force closed-loop control, driving the multi-fingered dexterous hand to envelop the target object with a constant envelope clamping force. During the continuous operation of the haptic force control mode, the main control computing unit simultaneously initiates an independent parallel monitoring process, which is specifically used to detect whether the target object shows signs of slippage on the elastic skin surface, and actively intervenes in the force control commands when necessary to prevent slippage accidents.

[0112] In this parallel monitoring process, the main control computing unit extracts two force components from the three-dimensional contact force vector calculated in step S4: tangential shear force and normal contact force. The three-dimensional contact force vector is a vector in three-dimensional space containing three mutually orthogonal components. The normal contact force refers to the pressure component applied by the target object along a direction perpendicular to the elastic skin surface; this component determines the degree of compression of the elastic skin and the magnitude of the normal pressure between the contact surfaces. The tangential shear force refers to the frictional force component applied by the target object along a direction parallel to the elastic skin surface; this component reflects the magnitude of the external force or inertial force acting on the target object in the tangential direction. The main control computing unit extracts the values ​​of the normal and tangential components from the three-dimensional contact force vector through vector decomposition operations.

[0113] During the processing cycle of each frame of tactile image, after obtaining the normal contact force at the current moment, the main control computing unit immediately calculates the upper limit of the maximum static friction force under the current contact state based on the normal contact force and a preset friction coefficient. This calculation is based on the classical Coulomb's law of friction, that is, the maximum static friction force equals the normal contact force multiplied by the static friction coefficient between the contact surfaces. This static friction coefficient is a preset physical parameter, the value of which depends on the surface material properties of the multi-layer concentric circle coded elastic skin and the frictional properties of the target object's surface material. In practical applications, this friction coefficient can be calibrated through pre-tested friction experiments and pre-programmed as a system parameter into the non-volatile memory of the main control computing unit.

[0114] After calculating the maximum static friction limit, the main control unit subtracts this limit from a preset safety margin value to obtain the anti-slip critical boundary value. This preset safety margin value is a pre-defined constant parameter. Its function is to set an early trigger warning line based on the maximum static friction limit, allowing anti-slip compensation control to intervene before the lateral shear force approaches the maximum static friction limit, thus theoretically preventing slippage. The setting of the safety margin value needs to balance the timeliness of anti-slip measures with the stability of control. If the safety margin is too large, it may lead to frequent false triggers of anti-slip compensation, affecting the smoothness of the grip; if the safety margin is too small, it may not be able to effectively prevent slippage.

[0115] After obtaining the critical anti-slip boundary value, the main control calculation unit compares the current tangential shear force with this critical boundary value. If the tangential shear force is less than or equal to the critical anti-slip boundary value, it indicates that the current tangential load is within a safe range, and the target object has not yet shown any risk of slippage. The main control calculation unit does not perform any additional intervention operations, and the tactile force control mode continues to maintain the current envelope clamping force output. If the tangential shear force exceeds the critical anti-slip boundary value, it indicates that the tangential load of the target object on the elastic skin surface has approached or even exceeded the safe boundary of the maximum static friction force, and slippage is about to occur. At this time, the main control calculation unit immediately triggers anti-slip compensation control.

[0116] The specific operation of anti-slip compensation control is as follows: without changing the basic framework of the tactile force control mode, the main control computing unit sends an additional emergency torque increment command to the motor drive board, driving the mechanical fingers of the multi-fingered dexterous hand to further close, thereby increasing the enveloping clamping force on the target object. This increase in enveloping clamping force directly leads to an increase in normal contact force, which in turn increases the upper limit of maximum static friction, thus providing a higher load-bearing capacity for tangential shear force. After triggering anti-slip compensation control, the main control computing unit continues to monitor the tangential shear force and normal contact force in real time. As the normal contact force continues to increase, the upper limit of maximum static friction rises continuously, and the anti-slip critical boundary value also increases accordingly. When the anti-slip critical boundary value is raised to exceed the current value of tangential shear force, the tangential shear force falls back below the anti-slip critical boundary value. At this point, the anti-slip compensation control condition is no longer met, the main control computing unit stops further increasing the enveloping clamping force, and the tactile force control mode returns to the normal constant force output state.

[0117] It should be noted that the aforementioned anti-slip compensation control is implemented as a fast response mechanism at the low-level hardware interrupt. In tactile force control mode, the main control computing unit monitors the tangential shear force and normal contact force at a frequency consistent with the tactile image acquisition frequency, typically above 120 Hz. Once the tangential shear force is determined to exceed the critical anti-slip boundary value, the main control computing unit issues the anti-slip compensation command within the same control cycle, without waiting for upper-level motion planning or decision-making algorithms. This direct low-level response mechanism significantly reduces the delay between detecting slip precursors and executing anti-slip actions, enabling the system to actively adjust the envelope clamping force before any visible slippage occurs on the target object.

[0118] In one possible embodiment, when the main control computing unit switches to haptic force control mode, while continuously executing force closed-loop control, it also acquires the current angular displacement vector and current output torque vector of each joint from the motor drive board in real time via the controller area network bus. In each control cycle, the motor drive board collects the angular displacement readings of the motor encoders of each joint and the actual output torque calculated from the motor phase current, and packages this data before reporting it to the main control computing unit. The acquisition frequency of this feedback data is independent of the processing frequency of the haptic feature images, typically at the kilohertz level, far exceeding the acquisition frequency of the haptic images, thus providing sufficient time resolution for position correction.

[0119] After obtaining the current angular displacement vectors of each joint, the main control computing unit calculates the current position of the fingertip of the multi-fingered dexterous hand using forward kinematics. Forward kinematics is a fundamental computational method in robotics; its input is the angular displacement of each joint, and its output is the position and orientation of the end effector, i.e., the mechanical fingertip, in three-dimensional space. The main control computing unit pre-stores the kinematic model parameters of the multi-fingered dexterous hand, including the length of each link, the direction and position of the rotation axis of each joint, and the angular displacement range of each joint. Based on these parameters and the current angular displacement vectors of each joint, the main control computing unit sequentially applies the homogeneous transformation matrix of each joint, progressively transferring coordinate transformations from the base of the mechanical finger towards the fingertip, ultimately calculating the three-dimensional position coordinates of the fingertip in the base coordinate system.

[0120] After calculating the current position of the fingertip, the main control computing unit compares this current position with the desired fingertip envelope pose and calculates the deviation between the two. The desired fingertip envelope pose refers to the target position that each robotic fingertip should reach in the current grasping task. This pose is preset by the grasping planning module based on the size and shape of the target object. The main control computing unit calculates the difference between the desired pose and the current position to obtain a position deviation vector. Subsequently, based on this position deviation vector and a preset position control gain, the main control computing unit generates a joint position correction command. The function of this joint position correction command is to drive each joint of the robot to produce a corresponding angular displacement, thereby reducing the deviation between the current position of the fingertip and the desired pose.

[0121] While generating joint position correction commands, the main control calculation unit also maintains a continuous output of force tracking control commands. These force tracking control commands are calculated based on the deviation between the three-dimensional contact force vector and the preset target force vector, combined with a force control law. At this point, the main control calculation unit faces a coordination problem between position control and force control; that is, it needs to combine the joint position correction commands and force tracking control commands into a unified, coordinated control command and send it to the motor drive board. To solve this coordination problem, the main control calculation unit uses either superposition or selective switching to fuse the two commands.

[0122] In the superposition mode, the main control computing unit directly adds the joint position correction command and the force tracking control command in the joint torque space to obtain the synthesized cooperative control command. The superposition mode is suitable for grasping target objects with elastic or compliant characteristics. In this mode, position correction and force control can work simultaneously. Position correction is responsible for guiding the fingertip toward the desired envelope configuration, while force control is responsible for adjusting the magnitude of the clamping force after contact occurs. The two complement each other and work in parallel.

[0123] When selecting the switching mode, the main control calculation unit selects between the joint position correction command and the force tracking control command as the effective output at the current moment based on the current contact state. This switching mode is suitable for grasping rigid objects or when the contact state changes drastically, in which case it is necessary to clearly distinguish the priority of position control and force control. In haptic force control mode, the force tracking control command is selected as the main control command; only when the fingertip position deviates from the desired envelope pose by more than a preset tolerance range is it temporarily switched to the joint position correction command, and the force tracking control command is restored after correction is completed.

[0124] Regardless of whether an overlay method or a selective switching method is used, the main control computing unit ultimately sends the fused and generated collaborative control commands to the motor drive board via the controller local area network bus. After receiving the command, the motor drive board converts it into drive current signals for each joint motor, driving each joint of the multi-fingered dexterous hand to output corresponding torque or angular displacement according to the command.

[0125] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processing unit built into the embodied robot, causes the processing unit to perform the methods described above.

[0126] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0127] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this patent application should be determined by the appended claims.

Claims

1. A multi-finger dexterous hand collaborative grasping method based on visual-tactile fusion, characterized in that, The method is applied to a main control computing unit, which is communicatively connected to a monocular imaging module and a motor drive board of a multi-finger dexterous hand; the method includes: Acquire the original fused image acquired in real time by the monocular imaging module; Based on the pre-programmed physical mask model, a binary mask matrix corresponding to the pixel positions of the original fused image is generated in memory. Using the binary mask matrix, an element-wise tensor separation operation is performed on the original fused image to decouple and generate tactile feature images and environmental visual feature images; The following two feature extraction methods are performed in parallel: Optical flow tracing is performed on the tactile feature image to extract the displacement field of feature points on the multi-layer concentric circle coded elastic skin surface covered by the monocular imaging module's field of view, induced by contact force. The three-dimensional contact force vector of the multi-layer concentric circle coded elastic skin surface is calculated based on the displacement field. Specifically, using the local image window containing the speckle feature points as the basic processing unit, an optical flow constraint equation based on the gray-level invariance assumption is constructed. The tactile feature image includes multiple speckle feature points. The overdetermined equation system formed by multiple local image windows is solved using the least squares method to obtain the two-dimensional pixel displacement vector of each speckle feature point on the image plane. Based on the two-dimensional pixel displacement vector and the pre-calibrated pixel-physical scale mapping coefficient, the physical displacement vector of each speckle feature point on the multi-layer concentric circle coded elastic skin surface is calculated. The physical displacement vector is then input into a preset contact mechanics model to calculate the three-dimensional contact force vector. The environmental visual feature image is processed to generate a standard panoramic environmental image, and the image contour and centroid position information of the target object are extracted. Determine whether the surface of the multi-layer concentric circle coded elastic skin has deformed beyond a preset pixel displacement threshold; if not, activate the visual servo mode, generate a pose alignment control command based on the deviation between the centroid position information of the target object and the preset desired position, and send it to the motor drive board to drive the multi-finger dexterous hand to approach the target object; if yes, switch to the tactile force control mode, generate a force tracking control command based on the deviation between the three-dimensional contact force vector and the preset target force vector, and send it to the motor drive board to drive the multi-finger dexterous hand to apply a constant envelope clamping force to the target object.

2. The multi-finger dexterous hand collaborative grasping method based on visual-tactile fusion according to claim 1, characterized in that, The step of generating a binary mask matrix in memory corresponding to the pixel positions of the original fused image based on the pre-programmed physical mask model includes: Multiple annular boundary polarity thresholds of multi-layer concentric circle encoded elastic skin are read from the pre-burned physical mask model; wherein, the multi-layer concentric circle encoded elastic skin covers the top of the monocular imaging module's field of view and is composed of alternating concentric arrangements of non-transparent annular regions and transparent annular regions, and the surface of the non-transparent annular regions has speckle characteristics. Using the optical center of the original fused image as the origin, an image physical coordinate system is established, and the polar radius value of each pixel is calculated one by one. The extreme radius value of each pixel is compared with the extreme radius threshold of the multiple annular boundaries to determine the annular region into which each pixel falls. If a pixel falls within the polar radius range corresponding to the non-transparent annular region, then the mask value of that pixel is set to the first logical value; if a pixel falls within the polar radius range corresponding to the transparent annular region, then the mask value of that pixel is set to the second logical value. The binary mask matrix is ​​formed by traversing all pixels of the original fused image and arranging the mask values ​​of each pixel according to their pixel positions.

3. The multi-finger dexterous hand collaborative grasping method based on visual-tactile fusion according to claim 2, characterized in that, The step of performing element-wise tensor separation operations on the original fused image using the binary mask matrix to decouple and generate tactile feature images and environmental visual feature images includes: The pixel positions in the binary mask matrix that have the first logical value are used as the reserved positions of the tactile channel, and the pixel values ​​at the corresponding pixel positions in the original fused image that have the second logical value are set to zero to obtain the tactile feature image; The pixel positions in the binary mask matrix that have the second logical value are used as the reserved positions of the visual channel, and the pixel values ​​at the corresponding pixel positions in the original fused image that have the first logical value are set to zero to obtain the environmental visual feature image. The tactile feature image retains only the image information of the non-transparent annular region in the multi-layered concentric circle coded elastic skin, while the environmental visual feature image retains only the image information of the transparent annular region in the multi-layered concentric circle coded elastic skin.

4. The multi-finger dexterous hand collaborative grasping method based on visual-tactile fusion according to claim 1, characterized in that, The step of processing the environmental visual feature image to generate a standard panoramic environmental image includes: Extract the effective pixel band corresponding to at least one transparent annular region in the environmental visual feature image; Using the inner and outer diameter boundaries of the effective pixel band as constraints, an inverse transformation mapping from polar coordinates to rectangular coordinates is performed on each pixel point within the effective pixel band, expanding the annularly distributed effective pixel band into a rectangular standard panoramic environment image. An edge detection operator is invoked on the standard panoramic environment image to extract the geometric contour of the target object, and the pixel coordinates of the centroid of the target object in the standard panoramic environment image are calculated.

5. The multi-finger dexterous hand collaborative grasping method based on visual-tactile fusion according to claim 1, characterized in that, In the tactile force control mode, the method further includes: The tangential shear force and normal contact force on the surface of the multi-layer concentric circle coded elastic skin are continuously monitored. The tangential shear force and normal contact force are obtained by the decomposition of the three-dimensional contact force vector. Based on the current normal contact force and the preset friction coefficient, the upper limit of the maximum static friction force is calculated, and the difference between the upper limit of the maximum static friction force and the preset safety margin value is used to obtain the anti-slip critical boundary value. The tangential shear force is compared with the anti-slip critical boundary value; if the tangential shear force exceeds the anti-slip critical boundary value, anti-slip compensation control is triggered, increasing the envelope clamping force until the tangential shear force falls back below the anti-slip critical boundary value.

6. A multi-finger dexterous hand device integrating a visual-tactile sensor, characterized in that, include: The sensing base housing has a hollow cavity inside, and no physical isolation structure for isolating stray light is provided in the hollow cavity. A monocular imaging module is fixedly installed at the bottom of the sensor base housing, with the field of view of the monocular imaging module facing upwards; An array of light sources is arranged around the outer periphery of the monocular imaging module, and the output optical axis of the array of light sources is set at an inclined angle relative to the optical center normal of the monocular imaging module. A transparent support plate is fixedly placed over the top opening of the sensing base housing; A local diffuser sheet is attached only to the center area of ​​the bottom surface of the transparent support plate, and is used to uniformly diffuse the emitted light from the array light source to the center area above the transparent support plate. A multi-layered concentric circle coded elastic skin covers the upper surface of the transparent support plate. The surface of the multi-layered concentric circle coded elastic skin is divided into multiple staggered transparent annular areas and non-transparent annular areas. Multiple mechanical fingers, the base of each mechanical finger is respectively hinged to different positions on the outer wall of the sensing base shell, and the movement trajectory of the end of each mechanical finger points to the central area of ​​the multi-layer concentric circle coded elastic skin. The motor drive board is fixedly installed inside or outside the housing of the sensing base and is electrically connected to the drive motor of each of the mechanical fingers; The main control computing unit is communicatively connected to the monocular imaging module and the motor drive board, and is used to execute the multi-finger dexterous hand collaborative grasping method based on visual-tactile fusion as described in any one of claims 1 to 5.

7. The multi-finger dexterous hand device according to claim 6, characterized in that, The number of the plurality of mechanical fingers is three, and the base ends of the three mechanical fingers are distributed in a circumferential array at equal intervals along the outer wall of the sensing base housing.

8. The multi-finger dexterous hand device according to claim 6, characterized in that, The surface of the multi-layered concentric circle encoded elastic skin, from the inside out, includes: The first ring-shaped area is a non-transparent ring-shaped area, which constitutes the central main tactile sensory area; The second annular area is a transparent annular area, which is set around the outer periphery of the first annular area to form a visual window of the near-field environment. The third annular area is a non-transparent annular area, which is arranged around the outer periphery of the second annular area to form an auxiliary tactile sensing area; The fourth annular area is a transparent annular area that surrounds the outer perimeter of the third annular area, forming a visual window for the mid-to-far view environment. The fifth annular region is a non-transparent annular region that surrounds the outer periphery of the fourth annular region, forming an edge tactile sensing area.

9. The multi-finger dexterous hand device according to claim 6, characterized in that, The multi-finger dexterous hand device also includes: A narrow-band filter is fixedly mounted on the lens surface of the monocular imaging module and located between the monocular imaging module and the transparent support plate.

Images