A dexterous hand multi-modal perception and control method, system, device and medium

Abstract

The application discloses a dexterous hand multi-modal perception and control method, system, device and medium, relates to the field of dexterous hand multi-modal perception and control, and comprises the following steps: acquiring original tactile data and original visual data corresponding to each finger; performing data compression and semantic fusion on the original tactile data and the original visual data to obtain a fusion feature vector corresponding to each finger; inputting the fusion feature vector corresponding to each finger into a finger-level KAN subnet corresponding to the finger to obtain a motor torque suggestion value; inputting an intermediate layer feature of the finger-level KAN subnet into a global collaborative KAN network to obtain a torque correction coefficient set; the torque correction coefficient set comprises torque correction coefficients of all joints of the fingers; and the motor torque value of each joint of each finger and the torque correction coefficient are used for multi-finger collaborative control of the dexterous hand, so that high-precision and high-adaptability control of the dexterous hand is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of dexterous hand multi-modal perception and control, in particular to a dexterous hand multi-modal perception and control method, system, device and medium. BACKGROUND

[0002] With the rapid development of humanoid robot technology, dexterous hands, as the interaction interface between algorithms and the physical world, are becoming a key technology to break through application bottlenecks. Dexterous hand research began in the 1970s, from the early Okada dexterous hand, Stanford / JPL dexterous hand to Utah / MIT dexterous hand, laying a theoretical foundation for subsequent humanoid multi-fingered dexterous hand research. At the end of the 20th century, with the development of embedded hardware, dexterous hand research shifted to high system integration and rich perception capabilities, such as DLR-I and DLR-II dexterous hands integrating 25 sensors, including tactile sensors, joint torque sensors, etc., significantly improving the flexibility and perception capabilities of dexterous hands. In addition, in the construction of modern fishing ports, the integration of dexterous hand technology and fishing port digital platform and communication key technology can provide new solutions for remote control and intelligent autonomous operation in marine operating environments. However, complex systems result in high manufacturing costs and reduce system reliability and maintainability, so in recent years, an important direction of multi-fingered dexterous hand design is to simplify the system and improve robustness to meet high control precision requirements. SUMMARY

[0003] The purpose of the present application is to provide a dexterous hand multi-modal perception and control method, system, device and medium, which can realize high-precision and high-adaptability control of dexterous hands.

[0004] To achieve the above purpose, the present application provides the following solutions.

[0005] In a first aspect, the present application provides a dexterous hand multi-modal perception and control method, comprising the following steps.

[0006] Obtaining original tactile data and original visual data corresponding to each finger; the original tactile data is measured by a flexible sensor array on the fingertips and the palm of the dexterous hand; the original visual data is collected by an image collection device on the dexterous hand.

[0007] Data compression and semantic fusion are performed on the original tactile data and the original visual data to obtain a fusion feature vector corresponding to each finger.

[0008] For each finger of the dexterous hand, the fusion feature vector corresponding to the finger is input into a finger-level KAN subnetwork corresponding to the finger to obtain a motor torque suggestion value; the motor torque suggestion value includes motor torque values of each joint of the finger.

[0009] input the intermediate layer features into the global collaborative KAN network to obtain a torque correction coefficient set; the intermediate layer features are outputs of an intermediate layer of a finger-level KAN subnetwork; the torque correction coefficient set includes torque correction coefficients of all joints of the fingers; and the motor torque value of each joint of each finger and the torque correction coefficient are used for multi-finger collaborative control of the dexterous hand.

[0010] Optionally, the finger-level KAN subnetwork includes an input layer, a first hidden layer, a second hidden layer and an output layer; the input layer is configured to receive a fusion feature vector; the first hidden layer includes 120 first nodes, each of which is configured to combine the fusion feature vector by using a weighted multivariate B-spline basis function to obtain a first feature corresponding to each of the first nodes; the second hidden layer includes 60 second nodes, each of which is configured to combine the first features corresponding to all the first nodes by using a weighted multivariate B-spline basis function to obtain a second feature corresponding to each of the second nodes; the second features corresponding to all the second nodes constitute the intermediate layer features; and the output layer includes four third nodes, each of which corresponds to a joint of a finger, and each of the third nodes is configured to process the second feature corresponding to the second node by using a linear activation function to obtain a motor torque value of each joint of the finger.

[0011] Optionally, the global collaborative KAN network includes a feature aggregation layer, a nonlinear mapping layer and a correction coefficient generation layer; the feature aggregation layer is configured to: splice the 60-dimensional second features output by the second hidden layers of the five finger-level KAN subnetworks to obtain spliced features; and generate a global feature representation by dynamically weighting the spliced features through an attention mechanism; the nonlinear mapping layer is configured to extract features from the global feature representation by using a basis function combination mechanism to obtain basis function combination features; and the correction coefficient generation layer is configured to predict the torque correction coefficient set according to the basis function combination features.

[0012] Optionally, the original haptic data and the original visual data are subjected to data compression and semantic fusion to obtain a fusion feature vector corresponding to each finger, specifically including: pre-processing the original haptic data and the original visual data to obtain pre-processed haptic data and pre-processed visual data; compressing the pre-processed haptic data by using a pyramid spatial compression network to obtain compressed haptic data; compressing the pre-processed visual data by using an encoder based on a time convolution network to obtain compressed visual data; and performing semantic fusion on the compressed haptic data and the compressed visual data by using a bidirectional cross-attention mechanism to obtain a fusion feature vector corresponding to each finger.

[0013] Optionally, the pyramid spatial compression network comprises a pooling module, a strided convolution network and a linear layer connected in sequence; the pooling module is configured to reduce the original matrix of the preprocessed tactile data to one quarter size by using a non-overlapping mean pooling operation to obtain first reduced dimension tactile data; the strided convolution network comprises a plurality of convolution modules connected in sequence and is configured to perform feature dimension compression on the first reduced dimension tactile data to obtain second reduced dimension tactile data; each convolution module comprises a plurality of convolution layers connected in sequence; and the linear layer is configured to process the second reduced dimension tactile data to obtain compressed tactile data.

[0014] Optionally, the time convolution network-based encoder comprises a causal convolution module and a linear layer; the causal convolution module comprises a plurality of causal convolution layers connected in sequence.

[0015] Optionally, before the fusion feature vector corresponding to the finger is input into the finger-level KAN subnetwork corresponding to the finger to obtain the motor torque recommendation value, the dexterous hand multi-modal perception and control method further comprises: obtaining a data set; the data set comprises a training set corresponding to each finger; the training set corresponding to each finger comprises a plurality of sample fusion feature vectors corresponding to each finger and a sample motor torque value corresponding to each sample fusion feature vector; for each finger, the initial KAN network is trained using the training set corresponding to the finger to obtain the finger-level KAN subnetwork corresponding to the finger; the intermediate layer features in the finger-level KAN subnetwork training process are taken as input, and the torque correction coefficient of each joint of the finger is taken as a label to train the initial KAN network to obtain a global collaborative KAN network.

[0016] In a second aspect, the present application provides a dexterous hand multi-modal perception and control system, comprising a data acquisition device and a server; the data acquisition device comprises a flexible sensor array and an image acquisition device.

[0017] The flexible sensor array is arranged on the fingertips and the fingerpads of the dexterous hand and is configured to acquire original tactile data corresponding to a finger.

[0018] The image acquisition device is arranged on the dexterous hand and is configured to acquire original visual data.

[0019] The server is configured to execute the dexterous hand multi-modal perception and control method described above.

[0020] In a third aspect, the present application provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the dexterous hand multi-modal perception and control method described above.

[0021] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the dexterous hand multi-modal perception and control method.

[0022] According to the specific embodiments provided in the present application, the present application discloses the following technical effects: the present application provides a dexterous hand multi-modal perception and control method, system, device and medium. The original tactile data and the original visual data are subjected to data compression and semantic fusion. Not only the key information of each modality is retained, but also the complementary relationship between the modalities is captured. The multi-modal fusion mechanism can overcome the limitations of single sensing mode. The motor torque prediction is performed based on the fusion feature vector obtained through semantic fusion. Stable object recognition and state estimation ability can be maintained in a complex and variable environment. Even in the case of insufficient light or partial occlusion of the object, the system can still operate reliably. The finger-level KAN subnetwork and the global collaborative KAN network are adopted. While maintaining high precision, better interpretability and faster neural scaling law are provided. High-precision and high-adaptability control of the dexterous hand can be achieved. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application. Those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0024] Figure 1 An application environment diagram of a dexterous hand multi-modal perception and control method according to an embodiment of the present application.

[0025] Figure 2 A flowchart of a dexterous hand multi-modal perception and control method according to an embodiment of the present application.

[0026] Figure 3 A functional module diagram of a dexterous hand multi-modal perception and control system according to an embodiment of the present application.

[0027] Figure 4 A structural diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort are within the scope of protection of the present application.

[0029] Traditional industrial robots, lacking force perception, are like "blind men feeling an elephant" when facing irregularly shaped workpieces in unstructured environments. However, dexterous hands equipped with multi-degree-of-freedom joints and tactile sensors enable robots to achieve "hand-eye-mind" coordination for the first time—through multiple pressure sensor arrays at the fingertips, they can detect minute force differences of 0.1 Newtons in real time, achieving millimeter-level operational precision in conjunction with a vision system. This biomimetic interaction capability is giving rise to disruptive applications, from replacing precision components in satellite on-orbit servicing to minimally invasive instruments in operating rooms that can identify tissue stiffness.

[0030] Tactile perception is a key technology for the fine manipulation of dexterous hands. Related academic papers reveal that humans naturally possess the ability to optimally fuse visual and tactile information, but how to construct such capabilities for engineering systems through computational models remains a significant challenge. Currently, tactile sensor research has evolved from early stages of force and pressure measurement to the current array tactile sensors, such as the commercially available BioTac sensor, and multimodal tactile sensors. These sensors provide robotic hands with a wealth of information, including posture, contact information with objects, and physical information about the environment. However, existing tactile sensing systems still have limitations in terms of information processing efficiency, flexibility, self-healing capabilities, and power consumption.

[0031] Multimodal fusion is a crucial approach to improving the performance of dexterous hands. When a person grasps an object, its weight, size, temperature, texture, and other characteristics are simultaneously perceived. Human dexterity is based on this perception. The knowledge learned from visual and tactile sensors can be correlated and complementary, enabling robots to fully master high-precision tasks. However, multimodal data typically possesses different physical properties and informational characteristics, and effectively fusing this information to achieve more accurate object recognition and manipulation remains a challenge.

[0032] In dexterous hand control, traditional methods, primarily based on simple rule systems or linear control models, struggle to handle complex and changing environments and tasks. Recently, machine learning, particularly deep learning methods, has shown great potential in dexterous hand control. However, these methods typically require large amounts of training data and suffer from problems such as model complexity, high computational cost, and poor interpretability. Inspired by the Kolmogorov-Arnold representation theorem, KAN networks (Kolmogorov-Arnold Networks) offer a novel neural network architecture. Unlike traditional multilayer perceptrons (MLPs), they fundamentally eliminate the dependence on linear weight matrices by replacing fixed activation functions with learnable functions. This architecture maintains high accuracy while providing better interpretability and faster neural scaling, offering new possibilities for solving complex pattern recognition and nonlinear mapping problems in dexterous hand control.

[0033] Currently, dexterous hand technology is experiencing a transition from "mechanical replication" to "intelligent surpassing", but still faces many challenges in integrating multi-modal perception data and achieving precise and flexible control. In particular, how to efficiently fuse tactile and visual information under limited computing resources and establish an accurate perception-control mapping relationship has become a key bottleneck restricting the practical application of dexterous hands. To solve these problems, the present application proposes a multi-modal fusion and control scheme based on KAN network, aiming to realize high-precision and high-adaptability control of dexterous hands.

[0034] The above objects, features and advantages of the present application will be more apparent from the following detailed description in conjunction with the accompanying drawings and specific embodiments.

[0035] The dexterous hand multi-modal perception and control method provided by the embodiments of the present application can be applied in an application environment as shown in the figure. Figure 1 The terminal 102 communicates with the server 104 through the network. The data storage system can store the data required to be processed by the server 104. The data storage system can be separately arranged, or integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the original tactile data and the original visual data to the server 104. After receiving the original tactile data and the original visual data, the server 104 performs data compression and semantic fusion on the original tactile data and the original visual data to obtain a fusion feature vector corresponding to each finger. The fusion feature vector corresponding to each finger is input into the finger-level KAN subnetwork corresponding to the finger to obtain a motor torque suggestion value. The intermediate layer features of the finger-level KAN subnetwork are input into the global collaborative KAN network to predict a set of torque correction coefficients. The server 104 can feed back the set of torque correction coefficients obtained for the original tactile data and the original visual data to the terminal 102. In addition, in some embodiments, the dexterous hand multi-modal perception and control method can also be realized by the server 104 or the terminal 102 alone, such as directly performing multi-modal perception processing on the original tactile data and the original visual data by the terminal 102, or obtaining the original tactile data and the original visual data from the data storage system by the server 104 and performing multi-modal perception processing on the original tactile data and the original visual data.

[0036] The terminal 102 can be, but is not limited to, various desktop computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be realized by an independent server or a server cluster composed of multiple servers, and can also be a cloud server.

[0037] In one exemplary embodiment, as shown in Figure 2 A dexterous hand multi-modal perception and control method is provided, which is executed by a computer device, specifically can be executed by a terminal or a server alone, or can be executed by a terminal and a server together. In the embodiments of the present application, the method is applied to the server 104 in the Figure 1 The method includes the following steps 201 to 204.

[0038] Step 201, obtaining original tactile data and original visual data corresponding to each finger; the original tactile data is measured by a flexible sensor array on the fingertips and the palm of the dexterous hand; the original visual data is collected by an image collection device on the dexterous hand.

[0039] Step 202, data compression and semantic fusion are performed on the original tactile data and the original visual data to obtain a fusion feature vector corresponding to each finger.

[0040] Step 203, for each finger of the dexterous hand, input the fusion feature vector corresponding to the finger into the finger-level KAN subnetwork corresponding to the finger to obtain a motor torque suggestion value; the motor torque suggestion value includes a motor torque value of each joint of the finger.

[0041] Step 204, input the intermediate layer features into the global collaborative KAN network to predict a set of torque correction coefficients; the intermediate layer features are the outputs of the intermediate layer of the finger-level KAN subnetwork; the set of torque correction coefficients includes torque correction coefficients of each joint of all fingers; each joint motor torque value and the torque correction coefficient of each finger are used for multi-finger collaborative control of the dexterous hand.

[0042] By implementing the above steps 201 to 204, through the fusion of high-dimensional tactile information and visual data, combined with the innovative KAN network architecture, it is expected to achieve a new level in the field of dexterous operation. From the perspective of perception, the multi-modal fusion mechanism can overcome the limitations of single sensing method, enabling the system to maintain stable object recognition and state estimation ability in complex and variable environments, even in the case of insufficient light or partial occlusion of objects. In terms of control precision, due to the introduction of tactile information and the fusion of physical constraints, the system can apply appropriate grasping force to objects with different hardness, shape and surface characteristics, avoiding the common problem of grasping too tightly or too loosely in traditional methods.

[0043] The effective operation of the dexterous hand system depends on the accurate perception of complex environment, which requires intelligent fusion of multi-source heterogeneous data. In the embodiment, a high-dimensional perception system is adopted to integrate information streams from the tactile sensor array and the vision system. The original tactile data is acquired by a high-density flexible sensor array embedded in the fingertips and fingerpads of the dexterous hand, and each finger is configured with a binocular vision unit which can generate a 500x500 pressure distribution matrix and a deformation distribution matrix after precise calibration, and these data contain micro-characteristic information of the object contact surface. The original vision data is from a Realsense D435i depth camera fixed on the dexterous hand, which provides RGBD images with a resolution of 1280x720 and a frame rate of 30Hz, and is used to extract macro features of the target object, such as color, texture and surface defect features.

[0044] The data preprocessing and feature fusion mainly include the measurement alignment link, the data preprocessing link, the data compression link and the semantic fusion link, and the specific process is as follows.

[0045] 1) Measurement alignment link: the measurement alignment mechanism is the basic link of multi-modal fusion, mainly solving the inconsistency problem of tactile and vision sensing systems in time and space dimensions. The application monitors the sampling time stamp of each sensor in real time, establishes a sliding sampling window, and dynamically adjusts the window width to adapt to the sampling frequency difference of different sensors. For example, the sampling rate of the tactile sensor (i.e. flexible sensor) is usually higher than that of the vision system, and the dynamic window mechanism can intelligently aggregate multiple tactile sampling points in the tactile data stream corresponding to a single frame of vision data in time, to generate time-consistent data packets.

[0046] 2) Data preprocessing link: the original perception data preprocessing link adopts differentiated processing strategies for different types of sensing data, effectively improving the data quality.

[0047] For tactile data, a multi-level filtering framework is implemented, first applying median filtering to eliminate burst noise points, and then using a wavelet transform-based time series filtering method to eliminate high-frequency jitter and retain effective frequency band information in the tactile signal. To deal with the dead zones or saturated points that may occur in the sensor array, an adaptive outlier detection algorithm is designed, which identifies abnormal points by analyzing the statistical characteristics of the local region, and performs interpolation repair based on the surrounding valid pixel values to obtain preprocessed tactile data.

[0048] For vision data, the preprocessing focuses on improving the reliability of depth information, using bilateral filtering to preserve edge information while smoothing depth noise. For the measurement blind area of the depth camera on the surface of transparent, reflective or black objects, a depth completion algorithm based on the surface continuity assumption is implemented to effectively fill the missing area and obtain preprocessed vision data.

[0049] Both the adaptive outlier detection algorithm and the depth completion algorithm based on the surface continuity assumption are to detect and complete data according to the measurement value should not be mutated.

[0050] 3) Data compression link: the application proposes a multi-modal data compression and fusion method for dexterous hand control system. The method aims to solve the problem of heterogeneous fusion of high-dimensional tactile data and visual information. Through the construction of a multi-scale, cross-modal feature processing system, efficient perception of complex operating environment is realized.

[0051] The first challenge is the high dimensionality of the preprocessed tactile data - the flexible sensor of each fingertip produces a 500 resolution pressure and deformation dual-channel matrix. Direct processing of these data will result in excessive consumption of computing resources and difficulty in meeting real-time control requirements. Therefore, the application designs a pyramid spatial compression network architecture, which realizes efficient information extraction through step-by-step dimension reduction. In the primary processing stage, the system uses non-overlapping mean pooling operation to reduce the original matrix to one-fourth size, obtaining the first dimension-reduced tactile data, which retains the macro pressure distribution pattern; then, through a step convolution network optimized by parameters, the feature dimension is further compressed, focusing on capturing the gradient change characteristics of the contact edge, obtaining the second dimension-reduced tactile data, and finally outputting a 128-dimensional vector , which is the compressed tactile data.

[0052] For the dynamic characteristics of preprocessed visual data, the application uses a time convolution network-based encoder to capture the temporal evolution law in the operation process. The time convolution network-based encoder expands the receptive field range step by step through a multi-layer causal convolution structure, which can establish local correlation between adjacent frames and model long-term dependence, and finally outputs a 128-dimensional vector , which is the compressed visual data. The preprocessed visual data includes several frames of images.

[0053] 4) Semantic fusion link: to realize deep semantic fusion, the application designs a bidirectional cross-attention mechanism. The bidirectional cross-attention mechanism first maps the compressed tactile data (tactile features) and the compressed visual data (visual features) to a 64-dimensional shared semantic space respectively through two groups of learnable projection matrices and : , , where is the mapped tactile feature, is the mapped visual feature, and​​ are projection matrices corresponding to the compressed haptic data and the compressed visual data respectively. In this shared space, the system computes the correlation matrix between the haptic features and the visual features achieves cross-modal attention allocation, where is a scaling factor to stabilize the gradient, is a normalized exponential function. The application expects that the network is more likely to rely on haptic information in the area where the pressure signal is explicit, while more likely to rely on visual cues in the area where the contact is insufficient. The fused features are passed through to obtain the final output of the fused feature vector not only preserves the key information of each modality, but also captures the complementary relationship between the modalities, such as the consistency constraint between the local hardness of the object perceived by the haptic and the overall material identified by the vision, denotes the concatenation operation.

[0054] The step 202 includes the following steps 301-304.

[0055] Step 301: Preprocess the original haptic data and the original visual data respectively to obtain preprocessed haptic data and preprocessed visual data.

[0056] Step 302: Use a pyramid spatial compression network to compress the preprocessed haptic data to obtain compressed haptic data.

[0057] The pyramid spatial compression network includes a pooling module, a stride convolution network and a linear layer connected in sequence; the pooling module is used to reduce the original matrix of the preprocessed haptic data to one-fourth size by using a non-overlapping mean pooling operation to obtain first reduced dimension haptic data; the stride convolution network includes a plurality of convolution modules connected in sequence, and is used to compress the feature dimension of the first reduced dimension haptic data to obtain second reduced dimension haptic data; each convolution module includes a plurality of convolution layers connected in sequence; and the linear layer is used to process the second reduced dimension haptic data to obtain the compressed haptic data.

[0058] Step 303: Use an encoder based on a time convolution network to compress the preprocessed visual data to obtain compressed visual data. The encoder based on the time convolution network includes a causal convolution module and a linear layer; the causal convolution module includes a plurality of causal convolution layers connected in sequence.

[0059] Step 304: Use a bidirectional cross-attention mechanism to fuse the semantic information of the compressed haptic data and the compressed visual data to obtain a fused feature vector corresponding to each finger.

[0060] The KAN-based dexterous hand multi-modal controller design: based on multi-modal data preprocessing and feature fusion, the application designs a dexterous hand controller architecture based on Kolmogorov-Arnold network (KAN). The KAN network is derived from the Kolmogorov-Arnold representation theorem, has strong function fitting ability and interpretability, and is particularly suitable for establishing a complex mapping relationship between high-dimensional tactile-vision features and fine torque control.

[0061] To cope with the complexity of multi-degree-of-freedom cooperative control of the dexterous hand, the application proposes a hierarchical KAN network architecture based on divide-and-conquer cooperation, including a finger-level KAN subnetwork and a global cooperative KAN network.

[0062] Finger-level KAN subnetwork: each finger is equipped with an independent KAN subnetwork, which is responsible for mapping the multi-modal joint features of the corresponding finger to 4-dimensional motor torque recommendation values. The finger-level KAN subnetwork includes an input layer, a first hidden layer, a second hidden layer, and an output layer.

[0063] The input layer is used to receive a fusion feature vector. The fusion feature vector is a 256-dimensional fusion feature vector, which includes tactile dynamic features and visual context information integrated through an attention mechanism.

[0064] The first hidden layer includes 120 first nodes, each of which uses a weighted multivariate B-spline basis function to combine the fusion feature vector to obtain a first feature corresponding to each first node. The first hidden layer uses 120 first nodes, each of which uses an adaptive B-spline basis function as an activation unit. These basis functions can accurately express the nonlinear relationship in the pressure-torque mapping and can better predict the motor torque value.

[0065] The second hidden layer includes 60 second nodes, each of which uses a weighted multivariate B-spline basis function to combine the first features corresponding to all the first nodes to obtain a second feature corresponding to each second node. The second features corresponding to all the second nodes constitute the intermediate layer features. The second hidden layer includes 60 second nodes, each of which uses a weighted multivariate B-spline basis function to combine, in the form of the following formula.

[0066] (1).

[0067] wherein, is the output of the second node, is the B-spline basis function, is the total number of B-spline basis functions used to construct the activation function of each second node, is the learnable weights of B-spline basis functions. This design enables the network to adaptively allocate more expressive power to the key nonlinear intervals of the contact force-deformation relationship.

[0068] An output layer, comprising four third nodes, each of the third nodes corresponding to a joint of the finger, each third node adopting a linear activation function to process the second features corresponding to the second nodes to obtain a motor torque value of each joint of the finger. The four nodes correspond to the motor torque of the four joints of the finger.

[0069] A global coordination KAN network: the global coordination network is responsible for optimizing the multi-finger coordination effect, and the global coordination KAN network comprises a feature aggregation layer, a nonlinear mapping layer and a correction coefficient generation layer.

[0070] The feature aggregation layer is configured to: splice the 60-dimensional second features output by the second hidden layers of the five finger-level KAN subnetworks to obtain spliced features; and generate a global feature representation by dynamically weighting the spliced features through an attention mechanism.

[0071] The nonlinear mapping layer is configured to perform feature extraction on the global feature representation by using a basis function combination mechanism to obtain basis function combination features. The nonlinear mapping layer uses a basis function combination mechanism unique to KAN to express a mixture of sine basis functions of different frequencies and polynomial basis functions of different orders, in the form shown in the following formula.

[0072] (2).

[0073] wherein, is the output of the nonlinear mapping layer, i.e., the basis function combination features; is an input variable, i.e., the feature vector transmitted from the feature aggregation layer, i.e., the global feature representation; is a weight coefficient of the sine term, which is a parameter learned by the network through training and controls the contribution degree of each sine basis function; is a frequency parameter of the sine function, which determines the periodicity of the sine wave. Different values enable the network to capture periodic patterns of different frequencies; is the total number of sine basis functions, which determines the network's ability to express periodic patterns; is a weight coefficient of the polynomial term, which is also a parameter learned through training and controls the contribution size of each order polynomial; is an order polynomial used to capture non-periodic trends and power-law relationships in the data; is the highest order of the polynomial basis function, which is usually set according to the nonlinearity complexity to be expressed. This design enables the network to simultaneously capture periodic patterns in the multi-finger coordination process.

[0074] A correction coefficient generation layer is configured to predict a set of torque correction coefficients based on the base function combined features. Each joint of each finger corresponds to a motor, and there are a total of 20 joints for all fingers. The correction coefficient generation layer outputs a 20-dimensional collaborative correction vector to finely adjust the motor torque suggested value of each finger-level KAN subnetwork, thereby achieving global optimization. Therefore, the output torque of each motor is the sum of the output of the finger-level KAN subnetwork and the corresponding torque correction coefficient.

[0075] Before step 203, the dexterous hand multi-modal perception and control method further includes steps 401-403.

[0076] Step 401: Obtain a data set, wherein the data set includes a training set corresponding to each finger, and each training set corresponding to each finger includes a plurality of sample fusion feature vectors corresponding to each finger and a sample motor torque value corresponding to each sample fusion feature vector.

[0077] Step 402: For each finger, train the initial KAN network using the training set corresponding to the finger to obtain a finger-level KAN subnetwork corresponding to the finger.

[0078] Step 403: Train the initial KAN network using the intermediate layer features in the finger-level KAN subnetwork training process as input and the torque correction coefficients of the joints of the finger as labels to obtain a global collaborative KAN network.

[0079] The present application proposes a scientific and efficient training strategy. The training method fully considers the multi-modal data characteristics and the complexity of dexterous hand control, so as to realize precise and robust grasping operation in the actual application environment.

[0080] The training process first needs comprehensive data collection. The present application adopts a combination of human expert demonstration and machine autonomous exploration to construct a data set containing multi-object operation. These objects cover different materials, shapes and hardness characteristics, thereby ensuring that the network can learn a wide range of tactile-visual-control mapping relationships. In the data collection process, the tactile sensor array data, the RGBD images captured by the vision system and the corresponding motor torque values are recorded synchronously. To enhance the robustness of the model, the present application implements enhancement processing on the original data, including adding synthetic noise, performing perspective transformation and simulating partial sensor information occlusion, thereby effectively expanding the diversity of the training samples.

[0081] Considering the hierarchical characteristics of the dexterous hand control system, the present application designs a phased training process. First, the pre-training phase of the independent subnetwork is performed, and the finger-level KAN subnetwork is trained for each finger. In this phase, the loss function is mainly composed of mean square error, that is, the loss function used in the training of the finger-level KAN subnetwork is mean square error loss (MSE). MSE LossThe learning rate adopts a decay strategy, with an initial value of 0.001, and is reduced by 10% after every fixed training round to balance the training speed and accuracy.

[0082] After the training of each finger-level KAN subnetwork converges, the training phase of the global collaborative KAN network is entered. At this time, the network parameters of the finger-level KAN subnetwork are temporarily frozen, and the optimization of the global collaborative KAN network is focused on. The global collaborative KAN network receives the intermediate layer features (i.e., the output of the second hidden layer) of the five finger-level KAN subnetworks and outputs the torque correction coefficient, realizing multi-finger collaborative control. At the same time, the task success rate loss is combined to incorporate the actual completion of the grasping task into the optimization goal, so that the network learns a more practical control strategy. The task success rate can be defined as the ratio of the number of successful times to the total number of times in the last 1000 training.

[0083] Finally, the end-to-end fine-tuning phase is performed, and the frozen state of all parameters is released to allow unified adjustment of the entire network system. A small learning rate is used for fine-tuning to prevent the destruction of the learned effective feature expression. The comprehensive loss function integrates the loss terms of each stage, and the different learning goals are balanced by adjusting the weight coefficients.

[0084] During the training process, a multi-dimensional evaluation index system is designed. The torque prediction error is used to quantify the control accuracy, including the mean absolute error and the root mean square error; the task success rate directly reflects the control effect in the actual task; and the stability index evaluates the smoothness of the grasping process. Through this evaluation mechanism, the model performance can be comprehensively grasped to guide the training direction. The stability index is defined as the sum of the squares of the acceleration.

[0085] What is particularly noteworthy is the adaptability exhibited when facing unknown objects. Through the excellent function fitting ability of the KAN network, it can be generalized to object types outside the training data set, exhibiting a certain degree of "tactile intelligence", which is crucial for the inevitable handling of new objects in practical application environments. In addition, the multi-finger collaborative control mechanism can realize complex operation actions, not limited to simple grasping and placing, but also possibly supporting rotation, adjustment, and other fine operations, laying the foundation for achieving human-level dexterous operation.

[0086] From a practical point of view, the present application reduces the task planning time while improving the task success rate, especially when dealing with irregularly shaped or fragile objects. This technological advancement will open up new possibilities for the application of robots in manufacturing, medical assistance, disaster rescue, and other fields, and will promote the transformation of dexterous operation capabilities from laboratory environments to practical application scenarios.

[0087] The application also provides an application scenario of the dexterous hand multi-modal perception and control method. Specifically, the dexterous hand multi-modal perception and control method provided in the embodiment can be applied in a dexterous hand grasping scenario. The dexterous hand grasping scenario includes a data acquisition link, a multi-modal perception link and a torque distribution link. The original tactile data and the original visual data enter the multi-modal perception link from the data acquisition link, and the corresponding finger joint motor torque value and the torque correction coefficient are obtained through human-machine cooperation, and then enter the downstream torque distribution link. The dexterous hand multi-modal perception and control method provided in the embodiment belongs to the multi-modal perception link. Specifically, in the multi-modal perception link process of the original tactile data and the original visual data, the original tactile data and the original visual data can be compressed and semantically fused to obtain a fusion feature vector corresponding to each finger. The fusion feature vector corresponding to each finger is input into a finger-level KAN subnetwork corresponding to the finger to obtain a motor torque suggestion value. The intermediate layer features of the finger-level KAN subnetwork are input into a global collaborative KAN network to predict a torque correction coefficient set.

[0088] Based on the same inventive concept, the embodiment of the application also provides a dexterous hand multi-modal perception and control system for implementing the dexterous hand multi-modal perception and control method described above. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme described in the above method, so the specific limitations in one or more dexterous hand multi-modal perception and control system embodiments provided below can refer to the limitations of the dexterous hand multi-modal perception and control method described above, and will not be repeated here.

[0089] In one exemplary embodiment, as shown in Figure 3 A dexterous hand multi-modal perception and control system is provided, which includes a data acquisition device and a server; the data acquisition device includes a flexible sensor array and an image acquisition device.

[0090] The flexible sensor array is arranged on the fingertips and the finger pads of the dexterous hand, and is used to acquire original tactile data corresponding to a finger.

[0091] The image acquisition device is arranged on the dexterous hand, and is used to acquire original visual data.

[0092] The server is used to execute the dexterous hand multi-modal perception and control method described above.

[0093] In an exemplary embodiment, a computer device is provided, which can be a server or a terminal, and the internal structure diagram thereof can be as shown in Figure 4As shown in the figure. The computer device includes a processor, a memory, an Input / Output (I / O) interface, and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store dexterous hand multi-modal perception data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a dexterous hand multi-modal perception and control method.

[0094] Those skilled in the art can understand that, Figure 4 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0095] In an exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above method embodiments.

[0096] In an exemplary embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program is executed by the processor to implement the steps in the above method embodiments.

[0097] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0098] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0099] The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0100] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0101] The principles and implementation manners of the present application are described herein by using specific examples, and the above examples are only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will have changes. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A dexterous hand multi-modal sensing and control method, characterized by, The dexterous hand multi-modal perception and control method comprises: Obtaining original tactile data and original visual data corresponding to each finger; the original tactile data is measured by a flexible sensor array on the fingertips and finger pads of the dexterous hand; the original visual data is collected by an image collection device on the dexterous hand; Performing data compression and semantic fusion on the original tactile data and the original visual data to obtain a fusion feature vector corresponding to each finger; For each finger of the dexterous hand, inputting the fusion feature vector corresponding to the finger into a finger-level KAN subnetwork corresponding to the finger to obtain a motor torque suggestion value; the motor torque suggestion value comprises motor torque values of joints of the finger; the finger-level KAN subnetwork comprises an input layer, a first hidden layer, a second hidden layer and an output layer; The input layer is used for receiving the fusion feature vector; The first hidden layer comprises 120 first nodes, each of which uses a weighted multivariate B-spline basis function to combine the fusion feature vector to obtain a first feature corresponding to each first node; The second hidden layer comprises 60 second nodes, each of which uses a weighted multivariate B-spline basis function to combine the first features corresponding to all the first nodes to obtain a second feature corresponding to each second node; the second features corresponding to all the second nodes constitute an intermediate layer feature; The output layer comprises four third nodes, each of which corresponds to a joint of the finger, and each third node uses a linear activation function to process the second feature corresponding to the second node to obtain the motor torque value of the joint of the finger; Inputting the intermediate layer feature into a global collaborative KAN network to predict a torque correction coefficient set; the intermediate layer feature is the output of the intermediate layer of the finger-level KAN subnetwork; the torque correction coefficient set comprises torque correction coefficients of the joints of all the fingers; each joint motor torque value of each finger and the torque correction coefficient are used for multi-finger collaborative control of the dexterous hand.

2. The dexterous hand multi-modal perception and control method of claim 1, wherein, The global collaborative KAN network comprises a feature aggregation layer, a nonlinear mapping layer and a correction coefficient generation layer; The feature aggregation layer is used for splicing the 60-dimensional second features output by the second hidden layers of the five finger-level KAN subnetworks to obtain spliced features; The spliced features are dynamically weighted through an attention mechanism to generate a global feature representation; The nonlinear mapping layer is used for feature extraction of the global feature representation through a basis function combination mechanism to obtain basis function combination features; The correction coefficient generation layer is used for predicting the torque correction coefficient set according to the basis function combination features.

3. The dexterous hand multi-modal perception and control method of claim 1, wherein, Performing data compression and semantic fusion on the original tactile data and the original visual data to obtain a fusion feature vector corresponding to each finger, specifically comprising: Preprocessing the original tactile data and the original visual data respectively to obtain preprocessed tactile data and preprocessed visual data; Using a pyramid spatial compression network to compress the preprocessed tactile data to obtain compressed tactile data; The preprocessed visual data is compressed by using an encoder based on a time convolution network to obtain compressed visual data; The compressed haptic data and the compressed visual data are fused by using a bidirectional cross-attention mechanism to obtain a fusion feature vector corresponding to each finger.

4. The dexterous hand multi-modal perception and control method of claim 3, wherein, The pyramid spatial compression network comprises a pooling module, a stride convolution network and a linear layer connected in sequence; The pooling module is configured to reduce the original matrix of the preprocessed haptic data to one fourth size by using a non-overlapping mean pooling operation to obtain first reduced-dimension haptic data. The stride convolution network comprises a plurality of convolution modules connected in sequence and configured to compress the feature dimension of the first reduced-dimension haptic data to obtain second reduced-dimension haptic data; each convolution module comprises a plurality of convolution layers connected in sequence. The linear layer is configured to process the second reduced-dimension haptic data to obtain compressed haptic data.

5. The dexterous hand multi-modal perception and control method of claim 3, wherein, The encoder based on the time convolution network comprises a causal convolution module and a linear layer; the causal convolution module comprises a plurality of causal convolution layers connected in sequence.

6. The dexterous hand multi-modal perception and control method of claim 1, wherein, Before the fusion feature vector corresponding to each finger is input into the finger-level KAN subnetwork corresponding to the finger to obtain a motor torque recommendation value, the dexterous hand multi-modal perception and control method further comprises: obtaining a data set; the data set comprises a training set corresponding to each finger; the training set corresponding to each finger comprises a plurality of sample fusion feature vectors corresponding to each finger and a sample motor torque value corresponding to each sample fusion feature vector; for each finger, training an initial KAN network by using the training set corresponding to the finger to obtain a finger-level KAN subnetwork corresponding to the finger; training the initial KAN network by taking the intermediate layer features in the training process of the finger-level KAN subnetwork as input and taking the torque correction coefficient of each joint of the finger as a label to obtain a global collaborative KAN network.

7. A dexterous hand multi-modal sensing and control system, characterized by, The dexterous hand multi-modal perception and control system comprises a data acquisition device and a server; the data acquisition device comprises a flexible sensor array and an image acquisition device; The flexible sensor array is arranged on the fingertips and the fingerpads of the dexterous hand and is configured to acquire original haptic data corresponding to a finger; The image acquisition device is arranged on the dexterous hand and is configured to acquire original visual data; The server is configured to execute the dexterous hand multi-modal perception and control method according to any one of claims 1-6.

8. A computer device comprising: A memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the dexterous hand multi-modal perception and control method according to any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the dexterous hand multi-modal perception and control method according to any one of claims 1-6.

Images