A tactile point cloud completion method based on dexterous hand topology and multi-level feature coding

Abstract

The application discloses a kind of based on dexterous hand topological structure and multi-level feature coding haptic point cloud completion method, belong to robot control technical field.For solve the problem of improving the operation ability and perception accuracy of robot in non-vision scene.The application includes that mechanical arm carries dexterous hand moves to target position, dexterous hand gradually closes until electronic skin sensor records contact signal, and the haptic point cloud data of target is collected;Construct the haptic point cloud completion model based on dexterous hand topological structure and multi-level feature coding;Design the layered loss function of haptic point cloud completion model based on dexterous hand topological structure and multi-level feature coding, which consists of two parts of coarse level point cloud loss and dense point cloud loss, and the core measurement adopts chamfer distance CD as geometric similarity index to measure the spatial deviation between predicted point cloud and real point cloud.The application realizes high-quality completion to complex object, so as to improve the operation ability and perception accuracy of robot in non-vision scene.

Description

Technical Field

[0001] This invention belongs to the field of robot control technology, specifically relating to a tactile point cloud completion method based on dexterous hand topology and multi-level feature encoding. Background Technology

[0002] Point cloud completion, a crucial task in 3D reconstruction and environmental understanding, aims to reconstruct complete and detailed 3D forms from partially missing or sparse point clouds. This technology has significant application value in fields such as intelligent manufacturing, robotics, autonomous driving, and virtual reality. For example, in robotic grasping or assembly scenarios, sensors often only acquire information about occluded or partially contacted surfaces, resulting in incomplete object shapes and affecting the accuracy of subsequent recognition, localization, and manipulation. Therefore, achieving high-fidelity 3D point cloud completion under limited observation conditions has become a key focus of academic and industrial research in recent years.

[0003] Existing point cloud completion methods can be broadly classified into two categories: one is non-contact completion methods based on vision or LiDAR; the other is contact completion methods based on tactile perception. The former typically uses cameras, structured light, or LiDAR sensors to acquire sparse or dense point cloud data and infers missing regions through deep neural networks or shape prior models; the latter uses embedded tactile sensors or electronic skin to acquire multimodal tactile signals such as contact force, pressure distribution, and temperature changes, and reconstructs the local or global geometry of an object from limited contact segments.

[0004] The first type of technique typically uses monocular or multi-view vision systems, depth cameras, or LiDAR to collect surface point cloud data of the environment or objects, and then uses deep learning networks to infer the complete geometric structure of the objects. This type of method has good adaptability and data coverage in macroscopic scenes, but it also has some inherent problems: because visual point clouds are greatly affected by lighting, viewing angle, and occlusion, they often cannot accurately model objects with complex shapes, transparent or reflective materials; at the same time, in the detailed structure of objects or occluded areas, visual point clouds are often extremely sparse or missing, resulting in insufficient accuracy of the completion results. In addition, visual completion models usually only rely on spatial geometric distribution for reconstruction, making it difficult to effectively capture the physical characteristics of objects at the contact level, such as hardness, friction coefficient, and texture.

[0005] In contrast, tactile point cloud completion methods have gained increasing attention in recent years. Tactile sensors can directly contact an object's surface, collecting multi-dimensional information such as pressure distribution, shear force, and temperature changes at local contact points, thus providing surface properties and microstructure information that are imperceptible to the naked eye. Especially in robotic dexterity hands or electronic skin systems, multi-point array tactile units can acquire high-resolution local surface information during dynamic contact, offering new possibilities for reconstructing three-dimensional geometry from a contact perspective. Tactile point cloud completion methods typically construct a tactile point cloud by spatially mapping the collected tactile points, and then utilize deep networks for shape reasoning and representation completion. Compared to vision-based solutions, tactile completion offers higher contact accuracy and surface detail fidelity, making it particularly suitable for occluded, invisible, or material-specific objects.

[0006] However, existing tactile point cloud completion techniques still have significant shortcomings. Most current research focuses on extracting overall features from tactile signals, using convolutional networks or transformer models to model the distribution patterns of tactile point clouds at a global level. While these approaches improve the global consistency of completion to some extent, they neglect the structural characteristics of the tactile data itself. Specifically, tactile point clouds not only contain geometric-physical information such as position and pressure, but also exhibit strong spatial local correlations and topological constraints between fingers. During contact, dexterous hands exhibit natural spatial coupling and mechanical transmission between different fingers and joints; this topological structure directly affects the distribution patterns and feature associations of tactile information. However, traditional methods often encode each tactile point or local block as an independent sample, lacking modeling of the physical structure of the dexterous hand, resulting in models failing to accurately capture the hierarchy of tactile signals and the patterns of inter-finger interactions.

[0007] In summary, existing technologies for tactile point cloud completion in robotic dexterous hand applications largely rely on global tactile feature encoding, neglecting the influence of the dexterous hand's physical topology on the distribution of tactile information. This leads to insufficient detail restoration and local structural distortion in point cloud completion. Therefore, there is an urgent need for a point cloud completion technology that incorporates a dexterous hand topology perception mechanism based on multi-level tactile feature encoding. This would improve local detail recovery capabilities, enhance the utilization rate of multi-finger collaborative features, and increase the model's adaptability to complex contact environments and different object shapes, thereby overcoming the limitations of existing technologies. Summary of the Invention

[0008] The problem this invention aims to solve is to improve the robot's operational capabilities and perception accuracy in non-visual scenarios. It proposes a tactile point cloud completion method based on dexterous hand topology and multi-level feature encoding.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A tactile point cloud completion method based on dexterous hand topology and multi-level feature encoding includes the following steps:

[0011] S1. The robotic arm carries the dexterous hand to the target position. The dexterous hand gradually closes until the electronic skin sensor records the contact signal, and collects the tactile point cloud data of the target, including the pressure data of the contact point and the three-dimensional coordinate data of the contact point, to build training and test sets.

[0012] S2. Construct a tactile point cloud completion model based on dexterous hand topology and multi-level feature encoding, including a pressure feature encoding module, a local tactile attention encoding module, a topology-aware map feature extraction module, a temporal aggregation module, and a point cloud reconstruction decoding module connected in sequence. The local tactile attention encoding module is also connected to the point cloud reconstruction decoding module. The pressure data of the touch point obtained in step S1 is input to the pressure feature encoding module, and the three-dimensional coordinate data of the touch point is input to the local tactile attention encoding module.

[0013] S3. Design a hierarchical loss function for a tactile point cloud completion model based on dexterous hand topology and multi-level feature encoding. The loss function consists of two parts: coarse point cloud loss and dense point cloud loss. The core metric uses chamfer distance (CD) as a geometric similarity index to measure the spatial deviation between the predicted point cloud and the real point cloud.

[0014] S4. Train the tactile point cloud completion model based on dexterous hand topology and multi-level feature encoding constructed in step S2 to obtain the trained tactile point cloud completion model based on dexterous hand topology and multi-level feature encoding for tactile point cloud completion based on dexterous hand topology and multi-level feature encoding.

[0015] Furthermore, the specific implementation method of step S1 includes the following steps:

[0016] S1.1. Each finger of the dexterous hand is configured to have three phalanges, and each phalange has an electronic skin array. The data acquisition process is a fixed sampling time, which collects the pressure data of the contact points and the pose information of the robotic arm and the dexterous hand.

[0017] Let the pressure data of the contact points of the i-th electronic skin in the t-th frame be... There are a total of There are 1 contact point, among which Let H be the set of real numbers, H be the height of the electronic skin matrix, and W be the width of the electronic skin matrix.

[0018] Based on the physical layout of the electronic skin, the local three-dimensional coordinates of the contact points of the i-th electronic skin in the t-th frame are calculated. for:

[0019]

[0020] in, Let x, y, and z represent the x-axis coordinates, y-axis coordinates, and z-axis coordinates of the contact point of the t-th frame and the i-th piece of electronic skin, respectively. The total number of electronic skins;

[0021] Let the joint angle vectors of the seven joints of the robotic arm at time t be... The corresponding joint length vector is ,in, and These represent the angle and length of the k-th joint, respectively.

[0022] The pose of the robotic arm's end effector in the global coordinate system is:

[0023]

[0024] in, Represents the positive kinematics function of the robotic arm;

[0025] Let the fixed installation pose of the i-th electronic skin in the dexterous hand coordinate system be... Then, its pose matrix in the global coordinate system at time t The expression is:

[0026] ;

[0027] S1.2. Transform the local point cloud to the global coordinate system:

[0028]

[0029] in, Represents the global coordinates of the t-th frame and the i-th piece of electronic skin;

[0030] S1.3. In a fixed sampling frame Inside, the tactile arrays and three-dimensional coordinates of all electronic skin touch points are combined to form a tactile point cloud. :

[0031]

[0032] in, This represents the total number of sampled frames.

[0033] Furthermore, the ratio of the training set to the test set constructed in step S1 is 8:2.

[0034] Furthermore, the specific implementation method of step S2 includes the following steps:

[0035] S2.1. Construct a pressure feature encoding module, including a first convolutional layer, a second convolutional layer, a batch normalization layer, and an activation layer. Input the pressure data of the touch points obtained in step S1 into the pressure feature encoding module, and output the extracted local tactile texture and pressure gradient features, expressed as:

[0036]

[0037] in, This represents the extracted local tactile texture and pressure gradient features of the i-th electronic skin in frame t. For pressure feature encoding function, is a rectified linear unit function, a commonly used activation function in artificial neural networks. It is a convolutional layer. For the t-th frame, the pressure reading at the contact point of the i-th electronic skin;

[0038] S2.2. Construct a local tactile attention encoding module, including a first linear transformation layer, a second point attention layer, a third linear transformation layer, and a fourth max pooling layer. Input the 3D coordinate data of the touch points obtained in step S1 and the extracted local tactile texture and pressure gradient features obtained in step S2.1 into the local tactile attention encoding module to obtain the locally tactile attention-encoded features, expressed as:

[0039]

[0040] in, This represents the local tactile attention encoding features of the i-th electronic skin in frame t. For the attention layer, These are the model parameters of the attention layer;

[0041] S2.3. Construct a topology-aware graph feature extraction module, including a first graph convolutional layer, a second graph convolutional layer, a third max pooling layer, and a fourth linear transformation layer. Input the local tactile attention encoding features obtained in step S2.2 into the topology-aware graph feature extraction module. Through graph neural networks, feature propagation and updates based on topological priors are performed between nodes to obtain the features of the t-th frame and the i-th piece of electronic skin. ,in For graph neural networks, It has a multi-layered topological structure of knuckles, fingers, and palm. These are the model parameters for the graph neural network;

[0042] Then, all the electronic skin features are fed into a linear transform layer to obtain the global features of frame t, expressed as:

[0043]

[0044] in, For the model parameters of the linear layer;

[0045] S2.4. Construct a time aggregation module, using a Long Short-Term Memory (LSTM) network to aggregate the time-dimensional features of the global features obtained in step S2.3, resulting in time-aggregated features. ,in, For global features from time frame 1 to t, These are the model parameters for the LSTM;

[0046] S2.5. Construct a point cloud reconstruction decoding module, including a first linear transformation layer, a second cross-attention layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer. Input the temporal aggregation features obtained in step S2.4 and the features encoded by local tactile attention obtained in step S2.2 into the point cloud reconstruction decoding module. First, obtain a coarse point cloud through a multilayer perceptron (MLP). ,in These are the model parameters for the MLP layer;

[0047] Then, through the cross-attention module, each point of the coarse point cloud is used as a query, and the local tactile attention encoding features of each electronic skin are used as keys and values. The attention score between the query point and these features is calculated, and the features are weighted and summed to obtain the weighted tactile context representation, i.e., TC, corresponding to each point of the coarse point cloud.

[0048] Then feature concatenation is performed to obtain ,in, The features are stitched together; the grid is a 2D mesh. For feature concatenation function;

[0049] Finally, dense point clouds were obtained. .

[0050] Furthermore, the specific implementation method of step S3 includes the following steps:

[0051] S3.1. Set coarse point cloud loss ,in This is a coarse point cloud output by the network. For realistic point clouds, This is the chamfer distance;

[0052] S3.2. Set the dense point cloud loss ,in This represents the high-resolution point cloud output by the network.

[0053] S3.3. Constructing a hierarchical loss function for a tactile point cloud completion model based on dexterous hand topology and multi-level feature encoding. ,in, This is the adjustment coefficient for the loss function.

[0054] The beneficial effects of this invention are:

[0055] This invention discloses a tactile point cloud completion method based on dexterous hand topology and multi-level feature encoding. By fully utilizing the physical topology and local tactile features of the dexterous hand during the tactile point cloud completion process, the accuracy and detail fidelity of point cloud completion are improved. This method overcomes the problems of existing technologies that only extract features from the whole and ignore tactile characteristics and dexterous hand topological information, achieving high-quality completion of complex objects, thereby improving the robot's operational capabilities and perception accuracy in non-visual scenarios.

[0056] This invention discloses a tactile point cloud completion method based on dexterous hand topology and multi-level feature encoding. Based on the topological constraints of the dexterous hand and a multi-level feature encoding strategy, it achieves high-fidelity reconstruction of tactile point clouds at both the spatial geometric and semantic levels. Through hierarchical feature fusion and local structure modeling, it effectively improves the accuracy of the completed point cloud in terms of shape continuity, contact area details, and overall structural consistency, providing a reliable means for high-quality reconstruction of tactile data.

[0057] The tactile point cloud completion method based on dexterous hand topology and multi-level feature encoding described in this invention has significant application value in tasks such as contact perception and object manipulation, and is particularly suitable for object recognition and flexible grasping in complex contact scenarios. When a dexterous hand partially contacts an unknown object, this method can generate a complete object point cloud based on limited tactile input, thereby assisting the robot in reasoning about the object's shape and contact characteristics, and achieving adaptive manipulation of non-rigid or complex-shaped targets. Attached Figure Description

[0058] Figure 1 The flowchart shows a tactile point cloud completion method based on dexterous hand topology and multi-level feature encoding as described in this invention.

[0059] Figure 2 The photos show the hardware environment of this invention, where (a) is the RM75-B robotic arm and (b) is a five-finger dexterous hand and electronic skin.

[0060] Figure 3 This is a single-frame tactile point cloud diagram of the present invention;

[0061] Figure 4 This is a diagram of the tactile point cloud completion model based on dexterous hand topology and multi-level feature encoding of the present invention.

[0062] Figure 5This is the topological structure of the topology-aware graph feature extraction module of the present invention;

[0063] Figure 6 Block diagram of point cloud reconstruction decoding module;

[0064] Figure 7 Let (a) be the loss curve, where (b) is the loss curve of the training set and (a) is the loss curve of the test set.

[0065] Figure 8 Comparison images of the input tactile point cloud, coarse point cloud, dense point cloud, and real point cloud. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention; that is, the described specific embodiments are merely a part of the embodiments of the invention, and not all of them. The components of the specific embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations, and the invention may also have other embodiments.

[0067] Therefore, the following detailed description of specific embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected specific embodiments of the invention. All other specific embodiments obtained by those skilled in the art based on these specific embodiments without inventive effort are within the scope of protection of this invention.

[0068] To further understand the invention's content, features, and effects, the following specific embodiments are provided, along with accompanying drawings. Figure 1 -Appendix Figure 8 Detailed explanation is as follows:

[0069] Example 1:

[0070] A tactile point cloud completion method based on dexterous hand topology and multi-level feature encoding includes the following steps:

[0071] S1. The robotic arm carries the dexterous hand to the target position. The dexterous hand gradually closes until the electronic skin sensor records the contact signal, and collects the tactile point cloud data of the target, including the pressure data of the contact point and the three-dimensional coordinate data of the contact point, to build training and test sets.

[0072] Furthermore, the specific implementation method of step S1 includes the following steps:

[0073] S1.1. Each finger of the dexterous hand is configured to have three phalanges, and each phalange has an electronic skin array. The data acquisition process is a fixed sampling time, which collects the pressure data of the contact points and the pose information of the robotic arm and the dexterous hand.

[0074] Let the pressure data of the contact points of the i-th electronic skin in the t-th frame be... There are a total of There are 1 contact point, among which Let H be the set of real numbers, H be the height of the electronic skin matrix, and W be the width of the electronic skin matrix.

[0075] Based on the physical layout of the electronic skin, the local three-dimensional coordinates of the contact points of the i-th electronic skin in the t-th frame are calculated. for:

[0076]

[0077] in, Let x, y, and z represent the x-axis coordinates, y-axis coordinates, and z-axis coordinates of the contact point of the t-th frame and the i-th piece of electronic skin, respectively. The total number of electronic skins;

[0078] Let the joint angle vectors of the seven joints of the robotic arm at time t be... The corresponding joint length vector is ,in, and These represent the angle and length of the k-th joint, respectively.

[0079] The pose of the robotic arm's end effector in the global coordinate system is:

[0080]

[0081] in, Represents the positive kinematics function of the robotic arm;

[0082] Let the fixed installation pose of the i-th electronic skin in the dexterous hand coordinate system be... Then, its pose matrix in the global coordinate system at time t The expression is:

[0083] ;

[0084] S1.2. Transform the local point cloud to the global coordinate system:

[0085]

[0086] in, Represents the global coordinates of the t-th frame and the i-th piece of electronic skin;

[0087] S1.3. In a fixed sampling frame Inside, the tactile arrays and three-dimensional coordinates of all electronic skin touch points are combined to form a tactile point cloud. :

[0088]

[0089] in, This represents the total number of sampled frames.

[0090] Furthermore, the ratio of the training set to the test set constructed in step S1 is 8:2.

[0091] Furthermore, during data acquisition, the robotic arm, carrying a dexterous hand, moves to the target bottle location. The dexterous hand then gradually closes until the electronic skin sensor records a contact signal. The entire acquisition process is sampled in fixed 20 frames to ensure the consistency and repeatability of the tactile data. The acquisition of the tactile point cloud relies on pressure readings collected by the electronic skin of the dexterous hand, as well as the pose information of the robotic arm and the dexterous hand. Forward kinematics maps the local positions of the sensors to the global coordinate system, thereby generating a high-precision tactile point cloud. The tactile point cloud includes pressure readings and three-dimensional coordinates of fifteen electronic skin contact points.

[0092] Furthermore, the hardware used in this embodiment consists of the RM75-B seven-DOF standard robotic arm independently developed and manufactured by Ruiman Intelligent, and a five-finger dexterous hand and electronic skin independently developed by Tujian Company. Each finger of the dexterous hand has three joints, and each joint has an array of electronic skin, totaling fifteen pieces.

[0093] Furthermore, the point cloud This represents the complete tactile perception result, which can be used for subsequent point cloud completion. A single-frame tactile point cloud is shown below. Figure 3 As shown.

[0094] S2. Construct a tactile point cloud completion model based on dexterous hand topology and multi-level feature encoding, including a pressure feature encoding module, a local tactile attention encoding module, a topology-aware map feature extraction module, a temporal aggregation module, and a point cloud reconstruction decoding module connected in sequence. The local tactile attention encoding module is also connected to the point cloud reconstruction decoding module. The pressure data of the touch point obtained in step S1 is input to the pressure feature encoding module, and the three-dimensional coordinate data of the touch point is input to the local tactile attention encoding module.

[0095] Furthermore, the specific implementation method of step S2 includes the following steps:

[0096] S2.1. Construct a pressure feature encoding module, including a first convolutional layer, a second convolutional layer, a batch normalization layer, and an activation layer. Input the pressure data of the touch points obtained in step S1 into the pressure feature encoding module, and output the extracted local tactile texture and pressure gradient features, expressed as:

[0097]

[0098] in, This represents the extracted local tactile texture and pressure gradient features of the i-th electronic skin in frame t. For pressure feature encoding function, is a rectified linear unit function, a commonly used activation function in artificial neural networks. It is a convolutional layer. For the t-th frame, the pressure reading at the contact point of the i-th electronic skin;

[0099] The pressure feature encoding module can effectively capture information such as contact intensity distribution and force direction changes. The detailed structure of the pressure feature encoding module is as follows:

[0100] First convolutional layer:

[0101] Convolutional layer: Input channel Output channel The convolutional kernel size is 3×3, stride=1, and padding p=1. It extracts features from the 2D image. The following convolutional layers function similarly and will not be described further.

[0102] Batch normalization layer: Normalizes the output of convolution.

[0103] Activation layer: For any element in the convolutional layer Applying ReLU, which is defined as

[0104] Second convolutional layer

[0105] Convolutional layer: Input channel Output channel The kernel size is 3×3, the stride is 1, and the padding is p=1.

[0106] Batch normalization layer: Normalizes the output of convolution.

[0107] Activation layer: For any element in the convolutional layer Applying ReLU, which is defined as .

[0108] S2.2. Construct a local tactile attention encoding module, including a first linear transformation layer, a second point attention layer, a third linear transformation layer, and a fourth max pooling layer. Input the 3D coordinate data of the touch points obtained in step S1 and the extracted local tactile texture and pressure gradient features obtained in step S2.1 into the local tactile attention encoding module to obtain the locally tactile attention-encoded features, expressed as:

[0109]

[0110] in, This represents the local tactile attention encoding features of the i-th electronic skin in frame t. For the attention layer, These are the model parameters of the attention layer;

[0111] Furthermore, to enhance feature aggregation between local geometric regions, this module models the relationships between points within each electronic skin array, thereby capturing the geometric and contact pattern features between tactile points in different parts of the dexterous hand. The detailed structure of the local tactile attention encoding module is as follows:

[0112] First linear transformation layer: Input feature dimension Output feature dimension Its main function is to perform linear mapping of the tactile features of the input points, achieving a preliminary feature space transformation. Through a fully connected layer (linear layer), this layer performs linear combination and weighting of the original features while keeping the feature dimensions unchanged, so that the subsequent attention layer can perform feature interaction in a normalized feature space.

[0113] Second-layer attention layer: Input feature dimension Output feature dimension In this layer, for each point, based on its spatial location and feature distribution, the model searches for its 16 nearest neighbors in its local neighborhood and calculates the attention score between that point and its neighbors. Through a weighted summation operation based on attention weights, the model can adaptively aggregate information from local regions, thereby strengthening important features related to geometric structure and reducing redundant or noisy information. This process enables dynamic modeling of local topological relationships, allowing the output features of each point to reflect its local geometric context.

[0114] Third linear transformation layer: Input feature dimension Output feature dimension Its function is to map the local features that have been aggregated through attention to a high-dimensional latent feature space, thereby improving the model's ability to represent complex tactile patterns.

[0115] The fourth max pooling layer (kernel=1, stride=1) mainly aggregates local features by maximizing the features of all points within each point cloud patch in the feature dimension, thereby extracting the most representative feature response.

[0116] S2.3. Construct a topology-aware graph feature extraction module, including a first graph convolutional layer, a second graph convolutional layer, a third max pooling layer, and a fourth linear transformation layer. Input the local tactile attention encoding features obtained in step S2.2 into the topology-aware graph feature extraction module. Through graph neural networks, feature propagation and updates based on topological priors are performed between nodes to obtain the features of the t-th frame and the i-th piece of electronic skin. ,in For graph neural networks, It has a multi-layered topological structure of knuckles, fingers, and palm. These are the model parameters for the graph neural network;

[0117] Then, all the electronic skin features are fed into a linear transform layer to obtain the global features of frame t, expressed as:

[0118]

[0119] in, For the model parameters of the linear layer;

[0120] Furthermore, this module incorporates the physical topology of a dexterous hand, organizing the sensing units into a multi-layered graph structure of knuckles, fingers, and palm. .node The features obtained after local tactile attention encoding for each piece of electronic skin ,side This represents the connections between electronic skin components. The topology is as follows: Figure 5 As shown. The detailed structure of the topology-aware map feature extraction module is as follows:

[0121] First layer: Graph convolutional layer; Input feature dimension Output feature dimension This layer constructs a graph structure based on the connection relationships of tactile points of a dexterous hand. Through graph convolution operations, node features are aggregated and updated within their neighborhoods, thereby realizing the feature propagation and fusion of tactile topological information.

[0122] Second layer: Graph convolutional layer; Input feature dimension Output feature dimension This layer further performs multi-hop aggregation on the graph features output from the previous layer to extract global topological information across regions.

[0123] The third max-pooling layer (kernel=1, stride=1) aggregates the feature dimension to 1. This layer performs max-pooling on node features across the entire graph, aggregating the feature responses of all nodes and extracting the most representative topological feature components.

[0124] Fourth linear transformation layer: Input feature dimension Output feature dimension This layer is used to perform high-dimensional mapping and compressed encoding of global topological features, so as to align multimodal features with other modules (such as the temporal aggregation module and the point cloud decoding module).

[0125] S2.4. Construct a time aggregation module, using a Long Short-Term Memory (LSTM) network to aggregate the time-dimensional features of the global features obtained in step S2.3, resulting in time-aggregated features. ,in, For global features from time frame 1 to t, These are the model parameters for the LSTM;

[0126] Furthermore, the input feature dimension of the LSTM layer The hidden layer has a dimension of 1024. By iteratively calculating the entire time series, the module can effectively capture the dynamic changes of tactile signals over time, enhancing the understanding of the contact process. The output is the hidden state of the last time step, serving as a global temporal representation of the sequence for use by the point cloud decoding module.

[0127] S2.5. Construct a point cloud reconstruction decoding module, including a first linear transformation layer, a second cross-attention layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer. Input the temporal aggregation features obtained in step S2.4 and the features encoded by local tactile attention obtained in step S2.2 into the point cloud reconstruction decoding module. First, obtain a coarse point cloud through a multilayer perceptron (MLP). ,in These are the model parameters for the MLP layer;

[0128] Then, through the cross-attention module, each point of the coarse point cloud is used as a query, and the local tactile attention encoding features of each electronic skin are used as keys and values. The attention score between the query point and these features is calculated, and the features are weighted and summed to obtain the weighted tactile context representation, i.e., TC, corresponding to each point of the coarse point cloud.

[0129] Then feature concatenation is performed to obtain ,in, The features are stitched together; the grid is a 2D mesh. For feature concatenation function;

[0130] Finally, dense point clouds were obtained. .

[0131] Furthermore, the detailed structure of the point cloud reconstruction decoding module is as follows:

[0132] First linear transformation layer: Input feature dimension Output feature dimension This linear mapping transforms the global representation into the three-dimensional coordinates of a coarse point cloud, which contains 1024 points, each with (x, y, z) three-dimensional information.

[0133] Second cross-attention layer: Input feature dimension Output feature dimension This layer achieves information interaction and weighted fusion by calculating the attention score between the coarse point cloud and the local tactile patch features, providing rich local contextual information for subsequent convolutional layers.

[0134] Third convolutional layer:

[0135] Convolutional layer: Input channel Output channel The kernel size is 1, the stride is 1, and the padding is p=1.

[0136] Batch normalization layer: Normalizes the output of convolution.

[0137] Activation layer: For any element in the convolutional layer Applying ReLU, which is defined as

[0138] Fourth convolutional layer:

[0139] Convolutional layer: Input channel Output channel The kernel size is 1, the stride is 1, and the padding is p=1.

[0140] Batch normalization layer: Normalizes the output of convolution.

[0141] Activation layer: For any element in the convolutional layer Applying ReLU, which is defined as

[0142] Fifth convolutional layer:

[0143] Convolutional layer: Input channel Output channel The convolution kernel size is 1, stride=1, and padding p=1. The output is the final 3D coordinate information (x, y, z) of the dense point cloud, completing the haptic point cloud.

[0144] S3. Design a hierarchical loss function for a tactile point cloud completion model based on dexterous hand topology and multi-level feature encoding. The loss function consists of two parts: coarse point cloud loss and dense point cloud loss. The core metric uses chamfer distance (CD) as a geometric similarity index to measure the spatial deviation between the predicted point cloud and the real point cloud.

[0145] Furthermore, the specific implementation method of step S3 includes the following steps:

[0146] S3.1. Set coarse point cloud loss ,in This is a coarse point cloud output by the network. For realistic point clouds, This is the chamfer distance;

[0147] S3.2. Set the dense point cloud loss ,in This represents the high-resolution point cloud output by the network.

[0148] S3.3. Constructing a hierarchical loss function for a tactile point cloud completion model based on dexterous hand topology and multi-level feature encoding. ,in, This is the adjustment coefficient for the loss function.

[0149] Furthermore, to simultaneously ensure the overall shape consistency of the point cloud and the accuracy of local structural details, a hierarchical loss function structure was designed. The overall loss consists of two parts: a coarse-level point cloud loss and a dense-level point cloud loss. The core metric uses Chamfer Distance (CD) as a geometric similarity index to measure the spatial deviation between the predicted and ground truth point clouds. Let the predicted point cloud be... The real point cloud is Then CD is defined as:

[0150]

[0151] The first term measures the closest distance from each point in the predicted point cloud to the real point cloud, reflecting the coverage of the prediction results; the second term measures the minimum distance from the real point cloud to the predicted point cloud, reflecting the completeness of the prediction results.

[0152] The output of the model of this invention includes two stages: coarse point cloud, used to recover the global geometry of the object; and dense point cloud, used to further complete the detailed structure and surface continuity.

[0153] S4. Train the tactile point cloud completion model based on dexterous hand topology and multi-level feature encoding constructed in step S2 to obtain the trained tactile point cloud completion model based on dexterous hand topology and multi-level feature encoding for tactile point cloud completion based on dexterous hand topology and multi-level feature encoding.

[0154] This embodiment conducts experiments on a swipe-up grip dataset containing 6 bottles, with each bottle being swipe-up gripped 500 times, for a total of 3000 samples. Figure 7 The figure shows the loss curves for the training and test sets during training. After 100 training iterations, the losses for both the training and test sets tend to plateau. The loss for the training set is 10.796, and the loss for the test set is 12.174. To better illustrate the loss, the loss in the figure is calculated by multiplying the loss by 1000. Figure 8 It displays the input tactile point cloud, the completed coarse point cloud, dense point cloud, and real point cloud.

[0155] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0156] Although this application has been described above with reference to specific embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of this application. In particular, as long as there is no structural conflict, the features in the specific embodiments disclosed in this application can be combined with each other in any way. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, this application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A tactile point cloud completion method based on dexterous hand topology and multi-level feature encoding, characterized in that, Includes the following steps: S1. The robotic arm carries the dexterous hand to the target position. The dexterous hand gradually closes until the electronic skin sensor records the contact signal, and collects the tactile point cloud data of the target, including the pressure data of the contact point and the three-dimensional coordinate data of the contact point, to build training and test sets. S2. Construct a tactile point cloud completion model based on dexterous hand topology and multi-level feature encoding, including a pressure feature encoding module, a local tactile attention encoding module, a topology-aware map feature extraction module, a temporal aggregation module, and a point cloud reconstruction decoding module connected in sequence. The local tactile attention encoding module is also connected to the point cloud reconstruction decoding module. The pressure data of the touch point obtained in step S1 is input into the pressure feature encoding module, and the extracted local tactile texture and pressure gradient features are output. The three-dimensional coordinate data of the touch point and the local tactile texture and pressure gradient features obtained in step S1 are input into the local tactile attention encoding module to obtain the features after local tactile attention encoding. The local tactile attention encoding features are input into the topology-aware graph feature extraction module. The graph neural network performs feature propagation and updating based on topology priors between nodes to obtain the features of the t-th frame and the i-th piece of electronic skin. Then, the features of all pieces of electronic skin are sent to the linear transformation layer to obtain the global features of the t-th frame. A Long Short-Term Memory (LSTM) network is used to aggregate the global features in the time dimension, resulting in time-aggregated features. The features encoded by temporal aggregation and local tactile attention are input into the point cloud reconstruction decoding module; S3. Design a hierarchical loss function for a tactile point cloud completion model based on dexterous hand topology and multi-level feature encoding. The loss function consists of two parts: coarse point cloud loss and dense point cloud loss. The core metric uses chamfer distance (CD) as a geometric similarity index to measure the spatial deviation between the predicted point cloud and the real point cloud. S4. Train the tactile point cloud completion model based on dexterous hand topology and multi-level feature encoding constructed in step S2 to obtain the trained tactile point cloud completion model based on dexterous hand topology and multi-level feature encoding for tactile point cloud completion based on dexterous hand topology and multi-level feature encoding.

2. The tactile point cloud completion method based on dexterous hand topology and multi-level feature encoding according to claim 1, characterized in that, The specific implementation method of step S1 includes the following steps: S1.

1. Each finger of the dexterous hand is configured to have three phalanges, and each phalange has an electronic skin array. The data acquisition process is a fixed sampling time, which collects the pressure data of the contact points and the pose information of the robotic arm and the dexterous hand. Let the pressure data of the contact points of the i-th electronic skin in the t-th frame be... There are a total of There are 1 contact point, among which Let H be the set of real numbers, H be the height of the electronic skin matrix, and W be the width of the electronic skin matrix. Based on the physical layout of the electronic skin, the local three-dimensional coordinates of the contact points of the i-th electronic skin in the t-th frame are calculated. for: in, Let x, y, and z represent the x-axis coordinates, y-axis coordinates, and z-axis coordinates of the contact point of the t-th frame and the i-th piece of electronic skin, respectively. The total number of electronic skins; Let the joint angle vectors of the seven joints of the robotic arm at time t be... The corresponding joint length vector is ,in, and These represent the angle and length of the k-th joint, respectively. The pose of the robotic arm's end effector in the global coordinate system is: in, Represents the positive kinematics function of the robotic arm; Let the fixed installation pose of the i-th electronic skin in the dexterous hand coordinate system be... Then, its pose matrix in the global coordinate system at time t The expression is: ； S1.

2. Transform the local point cloud to the global coordinate system: in, Represents the global coordinates of the t-th frame and the i-th piece of electronic skin; S1.

3. In a fixed sampling frame Inside, the tactile arrays and three-dimensional coordinates of all electronic skin touch points are combined to form a tactile point cloud. : in, This represents the total number of sampled frames.

3. The tactile point cloud completion method based on dexterous hand topology and multi-level feature encoding according to claim 2, characterized in that, The ratio of training set to test set constructed in step S1 is 8:

2.

4. The tactile point cloud completion method based on dexterous hand topology and multi-level feature encoding according to claim 3, characterized in that, The specific implementation method of step S2 includes the following steps: S2.

1. Construct a pressure feature encoding module, including a first convolutional layer, a second convolutional layer, a batch normalization layer, and an activation layer. Input the pressure data of the touch points obtained in step S1 into the pressure feature encoding module, and output the extracted local tactile texture and pressure gradient features, expressed as: in, This represents the extracted local tactile texture and pressure gradient features of the i-th electronic skin in frame t. For pressure feature encoding function, is a rectified linear unit function, a commonly used activation function in artificial neural networks. It is a convolutional layer. For the t-th frame, the pressure reading at the contact point of the i-th electronic skin; S2.

2. Construct a local tactile attention encoding module, including a first linear transformation layer, a second point attention layer, a third linear transformation layer, and a fourth max pooling layer. Input the 3D coordinate data of the touch points obtained in step S1 and the local tactile texture and pressure gradient features extracted in step S2.1 into the local tactile attention encoding module to obtain the locally tactile attention-encoded features, expressed as: in, This represents the local tactile attention encoding features of the i-th electronic skin in frame t. For the attention layer, These are the model parameters of the attention layer; S2.

3. Construct a topology-aware graph feature extraction module, including a first graph convolutional layer, a second graph convolutional layer, a third max pooling layer, and a fourth linear transformation layer. Input the local tactile attention encoding features obtained in step S2.2 into the topology-aware graph feature extraction module. Through graph neural networks, feature propagation and updates based on topological priors are performed between nodes to obtain the features of the t-th frame and the i-th piece of electronic skin. ,in For graph neural networks, It has a multi-layered topological structure of knuckles, fingers, and palm. These are the model parameters for the graph neural network; Then, all the electronic skin features are fed into a linear transform layer to obtain the global features of frame t, expressed as: in, For the model parameters of the linear layer; S2.

4. Construct a time aggregation module, using a Long Short-Term Memory (LSTM) network to aggregate the time-dimensional features of the global features obtained in step S2.3, resulting in time-aggregated features. ,in, For global features from time frame 1 to t, These are the model parameters for the LSTM; S2.

5. Construct a point cloud reconstruction decoding module, including a first linear transformation layer, a second cross-attention layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer. Input the temporal aggregation features obtained in step S2.4 and the features encoded by local tactile attention obtained in step S2.2 into the point cloud reconstruction decoding module. First, obtain a coarse point cloud through a multilayer perceptron (MLP). ,in These are the model parameters for the MLP layer; Then, through the cross-attention module, each point of the coarse point cloud is used as a query, and the local tactile attention encoding features of each electronic skin are used as keys and values. The attention score between the query point and these features is calculated, and the features are weighted and summed to obtain the weighted tactile context representation, i.e., TC, corresponding to each point of the coarse point cloud. Then feature concatenation is performed to obtain ,in, The features are stitched together; the grid is a 2D mesh. For feature concatenation function; Finally, dense point clouds were obtained. .

5. The tactile point cloud completion method based on dexterous hand topology and multi-level feature encoding according to claim 4, characterized in that, The specific implementation method of step S3 includes the following steps: S3.

1. Set coarse point cloud loss ,in This is a coarse point cloud output by the network. For realistic point clouds, This is the chamfer distance; S3.

2. Set the dense point cloud loss ,in This represents the high-resolution point cloud output by the network. S3.

3. Constructing a hierarchical loss function for a tactile point cloud completion model based on dexterous hand topology and multi-level feature encoding. ,in, This is the adjustment coefficient for the loss function.

Images