Dexterous hand multi-mode sensing and control method, system and equipment and medium

Abstract

The invention discloses a dexterous hand multi-mode perception and control method, system and device and a medium, and relates to the field of dexterous hand multi-mode perception and control, and the method comprises the steps: obtaining original tactile data and original visual data corresponding to each finger; performing data compression and semantic fusion on the original tactile data and the original visual data to obtain a fusion feature vector corresponding to each finger; inputting the fusion feature vector corresponding to each finger into a finger-level KAN subnet corresponding to the finger to obtain a motor torque suggestion value; inputting the intermediate layer features of the finger-level KAN subnet into a global collaborative KAN network, and predicting to obtain a torque correction coefficient set; the torque correction coefficient set comprises torque correction coefficients of all joints of all fingers; and the motor torque value and the torque correction coefficient of each joint of each finger are used for performing multi-finger cooperative control on the dexterous hand. According to the invention, high-precision and high-adaptability control of the dexterous hand is realized.

Description

Technical Field

[0001] This application relates to the field of dexterous hand multimodal sensing and control, and in particular to a dexterous hand multimodal sensing and control method, system, device and medium. Background Technology

[0002] With the rapid development of humanoid robot technology, dexterous hands, as the interface connecting algorithms and the physical world, are becoming a key technology for overcoming application bottlenecks. Dexterous hand research began in the 1970s, from the early Okada dexterous hand and Stanford / JPL dexterous hand to the Utah / MIT dexterous hand, laying the theoretical foundation for subsequent research on humanoid multi-fingered dexterous hands. At the end of the 20th century, with the development of embedded hardware, dexterous hand research shifted towards high system integration and rich sensing capabilities. For example, the DLR-I and DLR-II dexterous hands integrate 25 sensors, including tactile sensors and joint torque sensors, significantly improving the dexterous hand's flexibility and sensing capabilities. Furthermore, in modern fishing port construction, integrating dexterous hand technology with the construction of intelligent platforms for fishing vessels and ports, and combining key communication technologies, can provide new solutions for remote control and intelligent autonomous operation in marine environments. However, complex systems lead to high manufacturing costs and reduce system reliability and maintainability. Therefore, in recent years, an important direction in the design of multi-finger dexterous hands has been to simplify the system and improve robustness in order to meet the requirements of high control precision. Summary of the Invention

[0003] The purpose of this application is to provide a method, system, device and medium for multimodal sensing and control of dexterous hands, which can realize high-precision and highly adaptive manipulation of dexterous hands.

[0004] To achieve the above objectives, this application provides the following solution.

[0005] In a first aspect, this application provides a method for multimodal sensing and control of a dexterous hand, comprising the following steps.

[0006] The raw tactile data and raw visual data corresponding to each finger are acquired; the raw tactile data is measured by a flexible sensor array on the fingertips and fingertips of the dexterous hand; the raw visual data is acquired by an image acquisition device on the dexterous hand.

[0007] The original tactile data and the original visual data are compressed and semantically fused to obtain a fused feature vector for each finger.

[0008] For each finger of a dexterous hand, the fused feature vector corresponding to the finger is input into the finger-level KAN subnet corresponding to the finger to obtain the motor torque suggestion value; the motor torque suggestion value includes the motor torque value of each joint of the finger.

[0009] The intermediate layer features are input into the global collaborative KAN network to predict the torque correction coefficient set; the intermediate layer features are the outputs of the intermediate layer of the finger-level KAN subnetwork; the torque correction coefficient set includes the torque correction coefficients of each joint of all fingers; the motor torque value and torque correction coefficient of each joint of each finger are used for multi-finger collaborative control of dexterous hand.

[0010] Optionally, the finger-level KAN subnetwork includes an input layer, a first hidden layer, a second hidden layer, and an output layer; the input layer is used to receive the fused feature vector; the first hidden layer includes 120 first nodes, each of which uses a weighted multivariate B-spline basis function to combine the fused feature vector to obtain a first feature corresponding to each first node; the second hidden layer includes 60 second nodes, each of which uses a weighted multivariate B-spline basis function to combine all the first features corresponding to the first nodes to obtain a second feature corresponding to each second node; the second features corresponding to all the second nodes constitute the intermediate layer features; the output layer includes 4 third nodes, each of which corresponds to a joint of the finger, and each third node uses a linear activation function to process the second feature corresponding to the second node to obtain the motor torque values ​​of each joint of the finger.

[0011] Optionally, the global collaborative KAN network includes a feature aggregation layer, a nonlinear mapping layer, and a correction coefficient generation layer; the feature aggregation layer is used to: concatenate the 60-dimensional second features output from the second hidden layers of the five finger-level KAN subnetworks to obtain concatenated features; and dynamically weight the concatenated features through an attention mechanism to generate a global feature representation; the nonlinear mapping layer is used to extract features from the global feature representation using a basis function combination mechanism to obtain basis function combination features; and the correction coefficient generation layer is used to predict a set of torque correction coefficients based on the basis function combination features.

[0012] Optionally, the original tactile data and the original visual data are subjected to data compression and semantic fusion to obtain a fused feature vector corresponding to each finger. Specifically, this includes: preprocessing the original tactile data and the original visual data respectively to obtain preprocessed tactile data and preprocessed visual data; using a pyramid-shaped spatial compression network to compress the preprocessed tactile data to obtain compressed tactile data; using an encoder based on a temporal convolutional network to compress the preprocessed visual data to obtain compressed visual data; and using a bidirectional cross-attention mechanism to perform semantic fusion on the compressed tactile data and compressed visual data to obtain a fused feature vector corresponding to each finger.

[0013] Optionally, the pyramid-shaped spatial compression network includes a pooling module, a strided convolutional network, and a linear layer connected in sequence; the pooling module is used to reduce the original matrix of the preprocessed tactile data to a quarter size using non-overlapping mean pooling to obtain first dimensionality-reduced tactile data; the strided convolutional network includes several convolutional modules connected in sequence, used to compress the feature dimensions of the first dimensionality-reduced tactile data to obtain second dimensionality-reduced tactile data; each convolutional module includes several convolutional layers connected in sequence; the linear layer is used to process the second dimensionality-reduced tactile data to obtain compressed tactile data.

[0014] Optionally, the encoder based on the temporal convolutional network includes a causal convolutional module and a linear layer; the causal convolutional module includes a plurality of sequentially connected causal convolutional layers.

[0015] Optionally, before inputting the fused feature vector corresponding to the finger into the finger-level KAN subnetwork corresponding to the finger to obtain the motor torque proposal value, the dexterous hand multimodal perception and control method further includes: acquiring a dataset; the dataset includes a training set corresponding to each finger; the training set corresponding to each finger includes several sample fused feature vectors corresponding to each finger and sample motor torque values ​​corresponding to each sample fused feature vector; for each finger, the initial KAN network is trained using the training set corresponding to the finger to obtain the finger-level KAN subnetwork corresponding to the finger; the intermediate layer features in the training process of the finger-level KAN subnetwork are used as input, and the torque correction coefficients of each joint of the finger are used as labels to train the initial KAN network to obtain a global collaborative KAN network.

[0016] Secondly, this application provides a dexterous hand multimodal perception and control system, including a data acquisition device and a server; the data acquisition device includes a flexible sensor array and an image acquisition device.

[0017] The flexible sensor array is set on the fingertips and pads of the fingers to collect raw tactile data corresponding to a finger.

[0018] The image acquisition device is mounted on a dexterous hand and is used to acquire raw visual data.

[0019] The server is used to execute the aforementioned dexterous hand multimodal sensing and control method.

[0020] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described dexterous hand multimodal sensing and control method.

[0021] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described dexterous hand multimodal sensing and control method.

[0022] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a dexterous hand multimodal perception and control method, system, device and medium, which performs data compression and semantic fusion on the original tactile data and original visual data, not only retaining the key information of each modality, but also capturing the complementary relationship between modalities. The multimodal fusion mechanism can overcome the limitations of a single sensing method. Based on the fusion feature vector obtained by semantic fusion, motor torque prediction is performed, which can maintain stable object recognition and state estimation capabilities in complex and variable environments, and can still operate reliably even in cases of insufficient lighting or partial occlusion of objects. By adopting a finger-level KAN subnet and a global collaborative KAN network, while maintaining high accuracy, better interpretability and faster neural scaling rules are provided, which can realize high-precision and highly adaptive manipulation of the dexterous hand. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is an application environment diagram of a dexterous hand multimodal sensing and control method according to an embodiment of this application.

[0025] Figure 2 This is a flowchart illustrating a dexterous hand multimodal sensing and control method provided in one embodiment of this application.

[0026] Figure 3 This is a schematic diagram of the functional modules of a dexterous hand multimodal sensing and control system provided in an embodiment of this application.

[0027] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] Traditional industrial robots, lacking force perception, are like "blind men feeling an elephant" when facing irregularly shaped workpieces in unstructured environments. However, dexterous hands equipped with multi-degree-of-freedom joints and tactile sensors enable robots to achieve "hand-eye-mind" coordination for the first time—through multiple pressure sensor arrays at the fingertips, they can detect minute force differences of 0.1 Newtons in real time, achieving millimeter-level operational precision in conjunction with a vision system. This biomimetic interaction capability is giving rise to disruptive applications, from replacing precision components in satellite on-orbit servicing to minimally invasive instruments in operating rooms that can identify tissue stiffness.

[0030] Tactile perception is a key technology for the fine manipulation of dexterous hands. Related academic papers reveal that humans naturally possess the ability to optimally fuse visual and tactile information, but how to construct such capabilities for engineering systems through computational models remains a significant challenge. Currently, tactile sensor research has evolved from early stages of force and pressure measurement to the current array tactile sensors, such as the commercially available BioTac sensor, and multimodal tactile sensors. These sensors provide robotic hands with a wealth of information, including posture, contact information with objects, and physical information about the environment. However, existing tactile sensing systems still have limitations in terms of information processing efficiency, flexibility, self-healing capabilities, and power consumption.

[0031] Multimodal fusion is a crucial approach to improving the performance of dexterous hands. When a person grasps an object, its weight, size, temperature, texture, and other characteristics are simultaneously perceived. Human dexterity is based on this perception. The knowledge learned from visual and tactile sensors can be correlated and complementary, enabling robots to fully master high-precision tasks. However, multimodal data typically possesses different physical properties and informational characteristics, and effectively fusing this information to achieve more accurate object recognition and manipulation remains a challenge.

[0032] In dexterous hand control, traditional methods, primarily based on simple rule systems or linear control models, struggle to handle complex and changing environments and tasks. Recently, machine learning, particularly deep learning methods, has shown great potential in dexterous hand control. However, these methods typically require large amounts of training data and suffer from problems such as model complexity, high computational cost, and poor interpretability. Inspired by the Kolmogorov-Arnold representation theorem, KAN networks (Kolmogorov-Arnold Networks) offer a novel neural network architecture. Unlike traditional multilayer perceptrons (MLPs), they fundamentally eliminate the dependence on linear weight matrices by replacing fixed activation functions with learnable functions. This architecture maintains high accuracy while providing better interpretability and faster neural scaling, offering new possibilities for solving complex pattern recognition and nonlinear mapping problems in dexterous hand control.

[0033] Currently, dexterous hand technology is undergoing a transition from "mechanical replication" to "intelligent transcendence," but it still faces many challenges in integrating multimodal sensory data and achieving precise and flexible control. In particular, how to efficiently integrate tactile and visual information and establish an accurate perception-control mapping relationship under limited computing resources has become a key bottleneck restricting the practical application of dexterous hands. To address these issues, this application proposes a multimodal fusion and control scheme based on KAN networks, aiming to achieve high-precision and highly adaptive manipulation of dexterous hands.

[0034] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] The dexterous hand multimodal sensing and control method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send raw tactile data and raw visual data to server 104. After receiving the raw tactile data and raw visual data, server 104 performs data compression and semantic fusion on the raw tactile data and raw visual data to obtain a fused feature vector corresponding to each finger. The fused feature vector corresponding to each finger is input into the finger-level KAN subnetwork corresponding to that finger to obtain the motor torque proposal value. The intermediate layer features of the finger-level KAN subnetwork are input into the global collaborative KAN network to predict the torque correction coefficient set. Server 104 can feed back the obtained torque correction coefficient set for the raw tactile data and raw visual data to terminal 102. In addition, in some embodiments, the dexterous hand multimodal perception and control method can also be implemented by the server 104 or the terminal 102 separately. For example, the terminal 102 can directly perform multimodal perception processing on the original tactile data and the original visual data, or the server 104 can obtain the original tactile data and the original visual data from the data storage system and perform multimodal perception processing on the original tactile data and the original visual data.

[0036] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0037] In one exemplary embodiment, such as Figure 2 As shown, a dexterous hand multimodal sensing and control method is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 201 to 204.

[0038] Step 201: Obtain the raw tactile data and raw visual data corresponding to each finger; the raw tactile data is measured by a flexible sensor array on the fingertips and pads of the dexterous fingers; the raw visual data is acquired by an image acquisition device on the dexterous hand.

[0039] Step 202: Perform data compression and semantic fusion on the original tactile data and the original visual data to obtain the fused feature vector corresponding to each finger.

[0040] Step 203: For each finger of the dexterous hand, input the fused feature vector corresponding to the finger into the finger-level KAN subnet corresponding to the finger to obtain the motor torque suggestion value; the motor torque suggestion value includes the motor torque value of each joint of the finger.

[0041] Step 204: Input the intermediate layer features into the global collaborative KAN network to predict the torque correction coefficient set; the intermediate layer features are the outputs of the intermediate layer of the finger-level KAN subnetwork; the torque correction coefficient set includes the torque correction coefficients of each joint of all fingers; the motor torque value and torque correction coefficient of each joint of each finger are used for multi-finger collaborative control of the dexterous hand.

[0042] By implementing steps 201 to 204 above, and fusing high-dimensional tactile information with visual data, combined with an innovative KAN network architecture, it is expected to reach a new level in the field of dexterity manipulation. From a perceptual perspective, the multimodal fusion mechanism can overcome the limitations of a single sensing method, enabling the system to maintain stable object recognition and state estimation capabilities in complex and changing environments, and to operate reliably even in low light conditions or when objects are partially occluded. In terms of manipulation accuracy, due to the introduction of tactile information and the fusion of physical constraints, the system can apply appropriate grasping force to objects with different hardness, shapes, and surface properties, avoiding the problems of excessively tight or loose grasping common in traditional methods.

[0043] The effective operation of a dexterous hand system relies on accurate perception of complex environments, which requires the intelligent fusion of multi-source heterogeneous data. In this embodiment, a high-dimensional perception system is employed, integrating information flows from tactile sensor arrays and a vision system. Raw tactile data is acquired through a high-density flexible sensor array embedded in the fingertips and pads of the dexterous hand. After precise calibration, the binocular vision unit configured in each finger can generate a 500×500 pressure distribution matrix and deformation distribution matrix. These data contain microscopic characteristic information of the contact surface with the object. Raw visual data comes from a Realsense D435i depth camera fixed to the dexterous hand, providing RGBD images with a resolution of 1280×720 and a frame rate of 30Hz, used to extract macroscopic features of the target object, such as color, texture, and surface defect features.

[0044] Data preprocessing and feature fusion mainly include measurement alignment, data preprocessing, data compression, and semantic fusion. The specific process is described below.

[0045] 1) Measurement Alignment: The measurement alignment mechanism is a fundamental step in multimodal fusion, primarily addressing the spatiotemporal inconsistencies between tactile and visual sensing systems. This application monitors the sampling timestamps of each sensor in real time, establishes a sliding sampling window, and dynamically adjusts the window width to accommodate the differences in sampling frequencies among different sensors. For example, the sampling rate of tactile sensors (i.e., flexible sensors) is typically higher than that of visual systems. The dynamic window mechanism can intelligently aggregate multiple tactile sampling points in the tactile data stream that correspond temporally to a single frame of visual data, generating time-consistent data packets.

[0046] 2) Data preprocessing stage: The raw sensing data preprocessing stage adopts differentiated processing strategies for different types of sensing data to effectively improve data quality.

[0047] For tactile data, a multi-level filtering framework was implemented. First, a medium-level filter was applied to eliminate sudden noise points. Then, a wavelet transform-based temporal filtering method was used to eliminate high-frequency jitter while retaining the effective frequency band information in the tactile signal. To handle potential dead zones or saturation points in the sensor array, this embodiment designed an adaptive outlier detection algorithm. This algorithm identifies outliers by analyzing the statistical characteristics of local areas and performs interpolation repair based on the surrounding effective pixel values ​​to obtain preprocessed tactile data.

[0048] For visual data, preprocessing focuses on improving the reliability of depth information, employing bilateral filtering to preserve edge information while smoothing depth noise. To address the measurement blind spots of depth cameras on transparent, reflective, or black surfaces, a depth completion algorithm based on the surface continuity assumption is implemented to effectively fill in missing regions, resulting in preprocessed visual data.

[0049] Both adaptive outlier detection algorithms and depth completion algorithms based on the assumption of surface continuity refer to the detection and completion of data based on the premise that the measured values ​​should not change abruptly.

[0050] 3) Data Compression Stage: This application proposes a multimodal data compression and fusion method for dexterous hand control systems. This method aims to solve the challenge of heterogeneous fusion of high-dimensional tactile data and visual information by constructing a multi-scale, cross-modal feature processing system to achieve efficient perception of complex operating environments.

[0051] The first challenge is the high dimensionality of the preprocessed tactile data—each fingertip's flexible sensor generates 500... The 500-resolution dual-channel matrix of pressure and deformation would consume excessive computational resources and fail to meet real-time control requirements if processed directly. Therefore, this application designs a pyramid-shaped spatial compression network architecture, which achieves efficient information extraction through step-by-step dimensionality reduction. In the initial processing stage, the system uses non-overlapping mean pooling to reduce the original matrix to one-quarter of its size, obtaining the first dimensionality-reduced tactile data while preserving the macroscopic pressure distribution. Subsequently, a parameter-optimized strided convolutional network further compresses the feature dimensions, focusing on capturing the gradient change features at the contact edges, resulting in the second dimensionality-reduced tactile data. Finally, a 128-dimensional vector is output through a linear layer. This 128-dimensional vector To compress tactile data.

[0052] To address the dynamic characteristics of preprocessed visual data, this application employs a temporal convolutional network-based encoder to capture the temporal evolution patterns during the operation process. This temporal convolutional network-based encoder expands the receptive field progressively through a multi-layer causal convolutional structure, enabling both the establishment of local correlations between adjacent frames and the modeling of long-term temporal dependencies. Finally, it outputs a 128-dimensional vector through linear layers. This 128-dimensional vector To compress the visual data. The preprocessed visual data consists of several frames of images.

[0053] 4) Semantic Fusion Stage: To achieve deep semantic fusion, this application designs a bidirectional cross-attention mechanism. This bidirectional cross-attention mechanism first uses two sets of learnable projection matrices... and This will compress tactile data (tactile features). And compressed visual data (visual features) Mapped to a shared 64-dimensional semantic space respectively: , , of which For the mapped tactile features, For the mapped visual features, and These are the projection matrices corresponding to the compressed tactile data and compressed visual data, respectively. Within this shared space, the system calculates the correlation matrix between tactile features and visual features. To achieve cross-modal attention allocation, where This is a scaling factor used to stabilize the gradient. This is a normalized exponential function. This application aims to make the network rely more on tactile information in areas with clear pressure signals, and more on visual cues in areas with insufficient contact. The fused features are obtained through... The calculated final output fused feature vector It not only preserves key information of each modality, but also captures complementary relationships between modalities, such as the consistency constraint between the local hardness of an object perceived by touch and the overall material of the object recognized by vision. This indicates a splicing operation.

[0054] Then step 202 above includes steps 301 to 304.

[0055] Step 301: Preprocess the original tactile data and the original visual data respectively to obtain preprocessed tactile data and preprocessed visual data.

[0056] Step 302: Use a pyramid-shaped spatial compression network to compress the preprocessed tactile data to obtain compressed tactile data.

[0057] The pyramid-shaped spatial compression network comprises a pooling module, a strided convolutional network, and a linear layer connected in sequence. The pooling module is used to reduce the original matrix of the preprocessed tactile data to one-quarter of its size using non-overlapping mean pooling to obtain first-dimensionality reduced tactile data. The strided convolutional network comprises several convolutional modules connected in sequence to compress the feature dimensions of the first-dimensionality reduced tactile data to obtain second-dimensionality reduced tactile data. Each convolutional module comprises several convolutional layers connected in sequence. The linear layer is used to process the second-dimensionality reduced tactile data to obtain compressed tactile data.

[0058] Step 303: Using a temporal convolutional network-based encoder, the preprocessed visual data is compressed to obtain compressed visual data. The temporal convolutional network-based encoder includes a causal convolutional module and linear layers; the causal convolutional module includes several sequentially connected causal convolutional layers.

[0059] Step 304: Using a bidirectional cross-attention mechanism, semantic fusion is performed on compressed tactile data and compressed visual data to obtain the fused feature vector corresponding to each finger.

[0060] Design of a Dexterous Hand Multimodal Controller Based on KAN: Based on multimodal data preprocessing and feature fusion, this application designs a dexterous hand controller architecture based on the Kolmogorov-Arnold network (KAN). The KAN network originates from the Kolmogorov-Arnold representation theorem and has strong function fitting ability and interpretability, making it particularly suitable for establishing complex mapping relationships between high-dimensional tactile-visual features and fine torque control.

[0061] To address the complexity of multi-degree-of-freedom collaborative control of dexterous hands, this application proposes a divide-and-conquer collaborative hierarchical KAN network architecture, comprising two layers: a finger-level KAN subnetwork and a global collaborative KAN network.

[0062] Finger-level KAN subnetwork: Each finger is equipped with an independent KAN subnetwork responsible for mapping the multimodal joint features of the corresponding finger to 4D motor torque proposal values. The finger-level KAN subnetwork consists of an input layer, a first hidden layer, a second hidden layer, and an output layer.

[0063] The input layer is used to receive the fused feature vector. This fused feature vector is a 256-dimensional fused feature vector, which contains tactile dynamic features and visual context information integrated through an attention mechanism.

[0064] The first hidden layer comprises 120 first nodes. Each first node uses a weighted multivariate B-spline basis function to combine the fused feature vectors, resulting in a first feature corresponding to each first node. The first hidden layer uses 120 first nodes, each employing an adaptive B-spline basis function as its activation unit. These basis functions can accurately express the nonlinear relationship in the pressure-torque mapping, enabling better prediction of motor torque values.

[0065] The second hidden layer comprises 60 second nodes. Each second node uses a weighted multivariate B-spline basis function to combine the first features corresponding to all the first nodes, resulting in a second feature corresponding to each second node. The second features corresponding to all the second nodes constitute the intermediate layer features. The second hidden layer contains 60 second nodes, and each second node is combined using a weighted multivariate B-spline basis function, in the form shown in the following equation.

[0066] (1).

[0067] in, This is the output of the second node. For the first A B-spline basis function, It is the total number of B-spline basis functions used to construct the activation function of each second node. For the first Learnable weights for each B-spline basis function. This design enables the network to adaptively allocate more expressive power across key nonlinear regions of the contact force-deformation relationship.

[0068] The output layer includes four third nodes, each corresponding to a joint of the finger. Each third node uses a linear activation function to process the second feature corresponding to the second node, obtaining the motor torque values ​​of each finger joint. The four nodes correspond to the motor torques of the four finger joints.

[0069] Global Cooperative KAN Network: The global cooperative network is responsible for optimizing the multi-finger cooperative effect. The global cooperative KAN network includes a feature aggregation layer, a nonlinear mapping layer, and a correction coefficient generation layer.

[0070] The feature aggregation layer is used to: concatenate the 60-dimensional second features output from the second hidden layer of the five finger-level KAN subnetworks to obtain concatenated features; and dynamically weight the concatenated features through an attention mechanism to generate a global feature representation.

[0071] The nonlinear mapping layer is used to extract features from the global feature representation using a basis function combination mechanism, resulting in basis function combination features. It employs a basis function combination mechanism unique to KAN, using a mixture of sinusoidal basis functions of different frequencies and polynomial basis functions of different orders, in the form shown below.

[0072] (2).

[0073] in, This is the output of the nonlinear mapping layer, i.e., the feature of the basis function combination; The input variable is the feature vector passed from the feature aggregation layer, which is the global feature representation. The weight coefficients of the sine terms are parameters learned by the network during training, which control the contribution of each sine basis function. The frequency parameter of the sine function determines the periodicity of the sine wave; different... This value enables the network to capture periodic patterns at different frequencies; The total number of sinusoidal basis functions determines the network's ability to express periodic patterns; These are the weighting coefficients of the polynomial terms, which are also parameters learned through training, controlling the contribution of polynomials of different orders. for Power-law polynomials are used to capture non-periodic trends and power-law relationships in data; The highest order of the polynomial basis functions is usually set to the required nonlinear complexity so that the network can capture periodic patterns in multi-finite cooperative processes.

[0074] A correction coefficient generation layer is used to predict the torque correction coefficient set based on the combined features of the basis functions. Each finger joint corresponds to a motor, and there are 20 joints in total. This correction coefficient generation layer outputs a 20-dimensional collaborative correction vector, which finely adjusts the motor torque proposal values ​​of each finger-level KAN subnet, achieving global optimization. Therefore, the output torque of each motor is the sum of the output of the finger-level KAN subnet and the torque correction coefficients.

[0075] Before step 203, the dexterous hand multimodal sensing and control method further includes the following steps 401 to 403.

[0076] Step 401: Obtain the dataset; the dataset includes the training set corresponding to each finger; the training set corresponding to each finger includes several sample fusion feature vectors corresponding to each finger and the sample motor torque value corresponding to each sample fusion feature vector.

[0077] Step 402: For each finger, train the initial KAN network using the training set corresponding to the finger to obtain the finger-level KAN subnetwork corresponding to the finger.

[0078] Step 403: Using the intermediate layer features from the finger-level KAN subnetwork training process as input and the torque correction coefficients of each finger joint as labels, train the initial KAN network to obtain the global collaborative KAN network.

[0079] This application proposes a scientific and efficient training strategy that fully considers the characteristics of multimodal data and the complexity of dexterous hand control, aiming to achieve accurate and robust grasping operations in practical application environments.

[0080] The training process begins with comprehensive data acquisition. This application employs a combination of human expert demonstration and machine autonomous exploration to construct a dataset containing diverse object operations. These objects encompass different materials, shapes, and hardness characteristics, ensuring the network can learn a wide range of tactile-visual-control mappings. During data acquisition, tactile sensor array data, RGBD images captured by the vision system, and corresponding motor torque values ​​are recorded simultaneously. To enhance the model's robustness, this application performs augmentation processing on the raw data, including adding synthetic noise, performing viewpoint transformations, and simulating partial sensor information occlusion, effectively expanding the diversity of training samples.

[0081] Considering the hierarchical characteristics of the dexterous hand control system, this application designs a phased training process. First, a pre-training phase is conducted for independent subnets, training a finger-level KAN subnet separately for each finger. In this phase, the loss function mainly consists of the mean squared error; that is, the loss function used during the training of the finger-level KAN subnet is the mean squared error loss (MSE). MSELossThe learning rate employs a decay strategy, with an initial value of 0.001, decreasing by 10% after each fixed number of training rounds to balance training speed and accuracy.

[0082] After the training of each finger-level KAN subnetwork converges, the training phase of the global collaborative KAN network begins. At this stage, the network parameters of the finger-level KAN subnetworks are temporarily frozen, focusing on optimizing the global collaborative KAN network. This global collaborative KAN network receives intermediate layer features (i.e., the output of the second hidden layer) from the five finger-level KAN subnetworks and outputs torque correction coefficients to achieve multi-finger collaborative control. Simultaneously, a task success rate loss is incorporated, integrating the actual completion of the grasping task into the optimization objective, enabling the network to learn more practical control strategies. The task success rate can be defined as the proportion of successful attempts out of the total number of attempts in the most recent 1000 training iterations.

[0083] Finally, an end-to-end fine-tuning phase is performed, releasing all parameters from their frozen state and allowing for unified adjustments across the entire network. A relatively small learning rate is used for fine-tuning to prevent disruption of the learned, effective feature representations. The comprehensive loss function integrates the loss terms from each of the preceding stages, balancing different learning objectives by adjusting the weight coefficients of each term.

[0084] During training, this application designed a multi-dimensional evaluation index system. Torque prediction error is used to quantify control accuracy, including mean absolute error and root mean square error; task success rate directly reflects the control effect in actual tasks; and stability index evaluates the smoothness during the grasping process. Through this systematic evaluation mechanism, the model performance can be comprehensively grasped, guiding the training direction. The stability index is defined as the sum of squares of accelerations.

[0085] Of particular note is its adaptability when faced with unknown objects. Leveraging the excellent function fitting capabilities of the KAN network, it can generalize to object types beyond the training dataset, exhibiting a degree of "tactile intelligence," which is crucial for handling unfamiliar objects inevitably encountered in real-world applications. Furthermore, the multi-finger collaborative control mechanism enables complex maneuvers, going beyond simple grasping and placement to potentially support finer operations such as rotation and adjustment, laying the foundation for achieving human-level dexterity.

[0086] From a practical perspective, this application reduces task planning time while increasing task success rate, demonstrating a significant advantage, particularly when handling irregularly shaped or fragile items. This technological advancement will open up new possibilities for the application of robots in manufacturing, medical assistance, disaster relief, and other fields, driving the transformation of dexterity from laboratory environments to real-world applications.

[0087] This application also provides an application scenario in which the above-mentioned dexterous hand multimodal perception and control method is applied. Specifically, the dexterous hand multimodal perception and control method provided in this embodiment can be applied in a dexterous hand grasping scenario. The dexterous hand grasping scenario includes a data acquisition stage, a multimodal perception link, and a torque distribution stage. Raw tactile data and raw visual data enter the multimodal perception link from the data acquisition stage, and obtain the corresponding finger joint motor torque values ​​and torque correction coefficients through human-machine collaboration, and then enter the downstream torque distribution stage. The dexterous hand multimodal perception and control method provided in this embodiment belongs to the multimodal perception link. Specifically, in the multimodal perception link process for raw tactile data and raw visual data, data compression and semantic fusion can be performed on the raw tactile data and raw visual data to obtain the fused feature vector corresponding to each finger. The fused feature vector corresponding to each finger is input into the finger-level KAN subnetwork corresponding to the finger to obtain the motor torque suggestion value. The intermediate layer features of the finger-level KAN subnetwork are input into the global collaborative KAN network to predict the torque correction coefficient set.

[0088] Based on the same inventive concept, this application also provides a dexterous hand multimodal sensing and control system for implementing the dexterous hand multimodal sensing and control method described above. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the dexterous hand multimodal sensing and control system provided below can be found in the limitations of the dexterous hand multimodal sensing and control method described above, and will not be repeated here.

[0089] In one exemplary embodiment, such as Figure 3 As shown, a dexterous hand multimodal sensing and control system is provided, including a data acquisition device and a server; the data acquisition device includes a flexible sensor array and an image acquisition device.

[0090] The flexible sensor array is set on the fingertips and pads of the fingers to collect raw tactile data corresponding to a finger.

[0091] The image acquisition device is mounted on a dexterous hand and is used to acquire raw visual data.

[0092] The server is used to execute the aforementioned dexterous hand multimodal sensing and control method.

[0093] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 4As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores dexterous hand multimodal sensing data. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a dexterous hand multimodal sensing and control method.

[0094] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0095] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0096] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0097] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0098] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0099] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0100] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for multimodal sensing and control of a dexterous hand, characterized in that, The dexterous hand multimodal sensing and control method includes: The system acquires raw tactile data and raw visual data for each finger; the raw tactile data is measured by a flexible sensor array on the fingertips and pads of the dexterous fingers; the raw visual data is acquired by an image acquisition device on the dexterous hand. The original tactile data and the original visual data are compressed and semantically fused to obtain a fused feature vector for each finger. For each finger of a dexterous hand, the fused feature vector corresponding to the finger is input into the finger-level KAN subnet corresponding to the finger to obtain the motor torque suggestion value; the motor torque suggestion value includes the motor torque value of each joint of the finger. The intermediate layer features are input into the global collaborative KAN network to predict the torque correction coefficient set; the intermediate layer features are the outputs of the intermediate layer of the finger-level KAN subnetwork; the torque correction coefficient set includes the torque correction coefficients of each joint of all fingers; the motor torque value and torque correction coefficient of each joint of each finger are used for multi-finger collaborative control of dexterous hand.

2. The dexterous hand multimodal sensing and control method according to claim 1, characterized in that, The finger-level KAN subnet includes an input layer, a first hidden layer, a second hidden layer, and an output layer; The input layer is used to receive the fused feature vector; The first hidden layer includes 120 first nodes. Each first node uses a weighted multivariate B-spline basis function to combine the fused feature vectors to obtain the first feature corresponding to each first node. The second hidden layer includes 60 second nodes. Each second node uses a weighted multivariate B-spline basis function to combine the first features corresponding to all the first nodes to obtain the second features corresponding to each second node. The second features corresponding to all the second nodes constitute the intermediate layer features. The output layer includes four third nodes, each corresponding to a joint of the finger. Each third node uses a linear activation function to process the second feature corresponding to the second node to obtain the motor torque value of each joint of the finger.

3. The dexterous hand multimodal sensing and control method according to claim 2, characterized in that, The global collaborative KAN network includes a feature aggregation layer, a nonlinear mapping layer, and a correction coefficient generation layer; The feature aggregation layer is used to: concatenate the 60-dimensional second features output from the second hidden layer of the five finger-level KAN subnets to obtain concatenated features; A global feature representation is generated by dynamically weighting the concatenated features through an attention mechanism. The nonlinear mapping layer is used to extract features from the global feature representation using a basis function combination mechanism to obtain basis function combination features. The correction coefficient generation layer is used to predict the torque correction coefficient set based on the basis function combination features.

4. The dexterous hand multimodal sensing and control method according to claim 1, characterized in that, The original tactile data and the original visual data are compressed and semantically fused to obtain a fused feature vector for each finger, specifically including: The original tactile data and the original visual data are preprocessed to obtain preprocessed tactile data and preprocessed visual data, respectively. Using a pyramid-shaped spatial compression network, the preprocessed tactile data is compressed to obtain compressed tactile data; A temporal convolutional network-based encoder is used to compress the preprocessed visual data to obtain compressed visual data. By utilizing a bidirectional cross-attention mechanism, semantic fusion is performed on compressed tactile data and compressed visual data to obtain a fused feature vector for each finger.

5. The dexterous hand multimodal sensing and control method according to claim 4, characterized in that, The pyramid-shaped spatial compression network includes a pooling module, a strided convolutional network, and a linear layer connected in sequence. The pooling module is used to reduce the original matrix of the preprocessed tactile data to a quarter size by non-overlapping mean pooling to obtain the first dimensionality-reduced tactile data. The strided convolutional network includes several sequentially connected convolutional modules for compressing the feature dimensions of the first dimensionality-reduced tactile data to obtain the second dimensionality-reduced tactile data; each convolutional module includes several sequentially connected convolutional layers. The linear layer is used to process the second dimensionality-reduced tactile data to obtain compressed tactile data.

6. The dexterous hand multimodal sensing and control method according to claim 4, characterized in that, The encoder based on the temporal convolutional network includes a causal convolutional module and a linear layer; the causal convolutional module includes several causal convolutional layers connected in sequence.

7. The dexterous hand multimodal sensing and control method according to claim 1, characterized in that, Before inputting the fused feature vector corresponding to the finger into the finger-level KAN subnet corresponding to the finger to obtain the motor torque proposal value, the dexterous hand multimodal perception and control method further includes: Obtain the dataset; the dataset includes a training set corresponding to each finger; the training set corresponding to each finger includes several sample fusion feature vectors corresponding to each finger and sample motor torque values ​​corresponding to each sample fusion feature vector. For each finger, the initial KAN network is trained using the training set corresponding to that finger to obtain the finger-level KAN subnetwork; Using the intermediate layer features from the finger-level KAN subnetwork training process as input and the torque correction coefficients of each finger joint as labels, the initial KAN network is trained to obtain a global collaborative KAN network.

8. A dexterous hand multimodal sensing and control system, characterized in that, The dexterous hand multimodal perception and control system includes a data acquisition device and a server; the data acquisition device includes a flexible sensor array and an image acquisition device; The flexible sensor array is set on the fingertips and finger pads of the dexterous fingers to collect the original tactile data corresponding to a finger. The image acquisition device is mounted on a dexterous hand and is used to acquire raw visual data; The server is used to execute the dexterous hand multimodal sensing and control method according to any one of claims 1-7.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the dexterous hand multimodal sensing and control method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the dexterous hand multimodal sensing and control method as described in any one of claims 1-7.

Images