Robot multi-mode touch sensing device and method based on electrical capacitance tomography

Abstract

The invention provides a robot multi-mode touch sensing device based on electrical capacitance tomography, and the device comprises a flexible electrical capacitance tomography sensor which is disposed at an execution tail end of a robot and is used for sensing capacitance measurement values in a non-contact stage and a contact stage respectively; the multi-mode signal processing unit is used for acquiring the capacitance measurement value in real time, processing the capacitance measurement value to generate multi-mode touch information and feeding back the multi-mode touch information to the robot control system; wherein object category identification is carried out based on the capacitance measurement value in the non-contact stage, if the identified object is a contactable object, ECT image reconstruction is carried out by using the capacitance measurement value in the contact stage to obtain a contact image, and the contact image is converted into contact force distribution through a gray scale-pressure mapping relation. And carrying out object contour extraction by using the geometric features of the contact image. The adaptability and safety of the robot in a complex environment are improved, and the method is particularly suitable for the field of robots needing fine operation and environment interaction.

Description

Technical Field

[0001] This invention relates to the field of robot tactile sensing technology, specifically to a multimodal tactile sensing device and its sensing method based on capacitance tomography, which is applicable to scenarios requiring high-precision tactile feedback, such as robot grasping, human-computer interaction, and intelligent prosthetics. Background Technology

[0002] Robotics plays an indispensable role in industrial automation. However, most current robotic systems still rely primarily on visual sensors for environmental perception and object recognition. This approach often proves ineffective in poor lighting conditions, when targets are occluded, or when blind spots exist. Furthermore, while traditional direct-contact tactile sensors can provide richer physical interaction information, such as pressure, material properties, and deformation, they require actual contact with objects to acquire data. This can potentially damage the object or the sensor itself and limits the robot's early object perception capabilities.

[0003] In tasks such as sorting, picking, and precision assembly, robots often need to identify detailed properties of objects, such as material, hardness, and geometry. Especially in low-light or other complex environments, vision systems alone are insufficient to meet the demands for high-precision identification. While tactile sensing can provide this crucial information, this method of detection can cause irreversible damage to fragile, sensitive, or tiny objects, a problem particularly prominent in fields such as electronics manufacturing, food sorting, and logistics handling.

[0004] Therefore, the application of non-contact sensors in the field of robotics has gradually attracted attention in recent years, aiming to achieve preliminary perception of the material, position, or shape of objects without physical contact through methods such as capacitance, inductance, ultrasound, or electromagnetic fields. However, existing non-contact sensing technologies still have many limitations: on the one hand, most non-contact sensors can only provide distance or presence information, lacking the ability to finely distinguish the material, dielectric properties, or surface structure of objects; on the other hand, although some sensors can sense material properties, their outputs are mostly single numerical values ​​or array signals, making it difficult to achieve spatial distribution imaging and providing visualization information of object contours or contact areas. In addition, non-contact sensors generally suffer from small sensing range, susceptibility to environmental electromagnetic interference, and weak response to non-metallic materials, resulting in insufficient stability and reliability in complex industrial scenarios. More importantly, most current non-contact sensing systems lack effective integration with subsequent contact operations, failing to achieve a continuous and unified tactile feedback loop from "proximity perception" to "contact interaction," thus limiting the robot's ability to make autonomous decisions and perform fine operations in dynamic environments. Therefore, there is an urgent need for a multimodal tactile system that can both predict materials and shapes in the non-contact stage and seamlessly transition to high-resolution force distribution perception in the contact stage, in order to break through the bottleneck of traditional perception modes. Summary of the Invention

[0005] The present invention aims to at least partially solve one of the technical problems existing in the prior art.

[0006] To address this, this invention proposes a multimodal tactile sensing device and its sensing method for robots based on capacitance tomography (CT) technology. The aim is to provide a highly sensitive, flexible tactile sensing device capable of integrating multiple functions such as object recognition, contact force sensing, and shape detection. This device is suitable for various application scenarios, particularly in the field of robotics requiring fine manipulation and environmental interaction.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] The first aspect of this invention provides a multimodal tactile sensing device for robots based on capacitance tomography, comprising:

[0009] A flexible capacitance tomography sensor is installed at the end effector of a robot to sense capacitance measurements during both non-contact and contact phases.

[0010] A multimodal signal processing unit is used to acquire the capacitance measurement value in real time, process it to generate multimodal tactile information and feed it back to the robot control system. Among them, based on the capacitance measurement value in the non-contact stage, object category recognition is performed, which is divided into touchable objects and non-touchable objects. If the recognized object is a touchable object, the capacitance measurement value in the contact stage is used to reconstruct the ECT image to obtain a contact image. The contact image is converted into a contact force distribution through the gray-scale-pressure mapping relationship, and the geometric features of the contact image are used to extract the object contour.

[0011] In some embodiments, the flexible capacitive tomography sensor employs a double-layer flexible electrode array manufactured using flexible printed circuit board technology, with the top layer being the electrode array and the bottom layer being a flexible substrate.

[0012] In some embodiments, the capacitance measurements sensed by the flexible capacitance tomography sensor during the non-contact and contact phases are used to form the mutual capacitance between each pair of the M×N electrodes in the flexible capacitance tomography sensor, thus forming a total of One mutual capacitance measurement channel.

[0013] In some embodiments, the object category identification based on capacitance measurements during the non-contact phase includes:

[0014] The capacitance measurement value is characterized as the capacitance change rate:

[0015] Before each detection, an initial capacitance matrix C0 is recorded in the absence of a test object. After detecting an object approaching the flexible capacitance tomography sensor in a non-contact state, the mutual capacitance value C of each electrode pair (i,j) is scanned and acquired sequentially. ij This forms the current capacitance matrix C, where i,j = 1, 2, 3, ..., M × N, and i ≠ j; for each electrode pair, the capacitance change rate δ is calculated. ij :

[0016]

[0017] Among them, C 0,ij ΔC represents the initial capacitance between the i-th and j-th electrodes in the absence of a analyte. ij To detect the change in capacitance between the i-th electrode and the j-th electrode before and after;

[0018] Will The capacitance change rates are arranged in a set order to form... 3D eigenvector X:

[0019] X = [δ] 12 ,…,δ 1(M×N) ,…,δ ij ,…,δ(M×N-1)(M×N) ];

[0020] The feature vector X is input into a pre-trained classification model to determine whether a nearby object is a reachable object.

[0021] In some embodiments, the training steps of the classification model include:

[0022] For different items, corresponding feature vectors are constructed according to the method of obtaining the feature vector X, and their category labels are labeled. Each item is sampled multiple times to cover the situation of the item in different postures, proximity distances and environmental interference. The feature vectors of all items and their category labels form the training dataset D.

[0023] A support vector machine is used as a classifier, and the classifier is trained based on the training dataset D to obtain a classification model. The classification model outputs the predicted category of the object and the classification confidence.

[0024] In some embodiments, the classification model supports an incremental learning mechanism. When the confidence level output by the classification model is lower than a preset threshold, the current input sample is determined to be of the "unknown / uncertain" category. The feature vector and capacitance measurement value of the input sample are cached in a local database, which is deployed in the memory of the multimodal signal processing unit. The memory has a manual annotation interface to add true category labels to the cached samples. If the labels are not added in time, the multimodal signal processing unit automatically marks the current input sample as "unknown". After accumulating a set number of new samples, the classification model is locally updated using an incremental learning algorithm of support vector machine.

[0025] In some embodiments, the step of reconstructing the ECT image using capacitance measurements during the contact phase to obtain a contact image includes:

[0026] A three-dimensional simulation geometric model of M×N electrodes in the flexible capacitance tomography sensor is established. A small dielectric perturbation is applied to each electrode pair, and the resulting change in mutual capacitance is calculated. The influence coefficient of each imaging unit in the imaging region on the mutual capacitance measurement value is calculated, and a sensitivity matrix is ​​constructed accordingly. in, This indicates the number of pixel units that divide the imaging region;

[0027] Construct the capacitance change vector: ΔC = [C 12 -C 0,12 ,...,C ij -C 0,ij ,...,C (M×N-1)(M×N) -C 0,(M×N-1)(M×N) Initialize the image as either an all-zero image or a uniformly distributed image;

[0028] Introducing the Tikhonov regularization term, we have:

[0029] G k+1 =G k +η(S T (ΔC-SG k )+λG k )

[0030] Among them, S T G is the transpose of the sensitivity matrix. k Let η be the reconstructed image obtained in the k-th iteration, η be the iteration step size, and λ be the Tikhonov regularization parameter.

[0031] After iteration, the reconstructed image G is output. k Each pixel value represents the change in dielectric constant of that pixel region, and the output reconstructed image G k Normalized to a grayscale image, which serves as the contact image.

[0032] In some embodiments, converting the contact image into a contact force distribution through a grayscale-pressure mapping relationship includes:

[0033] Pixels with gray values ​​greater than a set threshold in the contact image are marked as high-response regions, generating a binarized potential contact region image; connected component analysis is performed on the potential contact region image to identify all interconnected pixel regions, and the connected region with the largest area is selected as the ROI region. The gray values ​​of all pixels in the ROI region are extracted, and the median is calculated as the feature value of the ROI region.

[0034] The contact force distribution is obtained based on the feature values ​​of the ROI region and the pre-constructed gray-scale-pressure mapping relationship;

[0035] The grayscale-pressure mapping relationship is a "grayscale-pressure" mapping table, and the construction steps include:

[0036] Under known force conditions, multiple sets of capacitance measurements are obtained using the flexible capacitance tomography sensor. A series of calibrated contact images are obtained through ECT image reconstruction. For each set of experiments, the median gray value of the corresponding ROI region and the actual applied pressure value are recorded to obtain a "gray-scale-pressure" mapping table.

[0037] In some embodiments, when an object is identified as a touchable object, the multimodal signal processing unit feeds back the obtained contact force distribution and object contour information to the robot control system in real time; when an object is identified as a non-touchable object, the robot control system adjusts the working path of the robot's end effector according to the object category identification result obtained by the multimodal signal processing unit.

[0038] A second aspect of the present invention provides a robot tactile perception method, comprising:

[0039] The robot's multimodal tactile sensing device according to any embodiment of the first aspect of the present invention is assembled at the robot's end effector;

[0040] During the non-contact phase, the changes in capacitance measurements obtained through the robot's multimodal tactile sensing device identify approaching objects to obtain object categories.

[0041] If the object is a touchable object, then during the stable contact phase, the robot's multimodal tactile sensing device is used to simultaneously acquire the contact force distribution and the object's outline.

[0042] The robot control system adjusts the motion strategy of the robot's end effector based on the multimodal tactile information from the robot's multimodal tactile sensing device.

[0043] The present invention provides a multimodal tactile sensing device and its sensing method based on capacitance tomography, which has the following significant advantages:

[0044] This invention enables continuous tactile perception throughout the entire process from non-contact to contact.

[0045] This invention employs a miniature flexible electrode array design, overcoming the difficulty of integrating traditional capacitance tomography systems into robot end effectors.

[0046] This invention introduces the Landweber iterative algorithm and regularization strategy to solve the problem of high computational cost in conventional image reconstruction algorithms for capacitance tomography, thereby meeting the needs of real-time robot control.

[0047] This invention constructs a multimodal signal processing architecture based on capacitance tomography.

[0048] The visualized tactile information of this invention improves the interpretability of the system. In contrast, traditional tactile sensors mostly output dot matrix pressure values ​​and lack spatial structure information.

[0049] This invention enhances the adaptability and safety of robots in complex environments. When handling fragile, sensitive, or fragile items, this invention supports non-contact preliminary identification, avoiding damage caused by mechanical contact and improving the safety and efficiency of operations. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the structure of the robot multimodal tactile sensing device provided in an embodiment of the present invention;

[0051] Figure 2The confusion matrix represents the 15-class classification accuracy of the SVM algorithm used in the robot multimodal tactile sensing device provided in this embodiment of the invention.

[0052] In the figure: 1. Sensor mounting end, 11. Electrode, 12. Flexible substrate, 2. Multimodal signal processing unit, 3. Robot end effector, 31. Mounting area. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application. The following solutions are merely illustrative of the inventive concept, and specific solutions are not limited thereto. Furthermore, for ease of description, the accompanying drawings show only the parts relevant to the present invention, not the entire process.

[0054] Conversely, this application covers any alternatives, modifications, equivalent methods, and schemes made within the spirit and scope of this application as defined by the claims. Furthermore, to provide the public with a better understanding of this application, certain specific details are described in detail below. However, this application can be fully understood by those skilled in the art even without these detailed descriptions.

[0055] See Figure 1 The present invention provides a multimodal tactile sensing device based on capacitance tomography, comprising:

[0056] A flexible capacitance tomography sensor 1 is installed at the end effector 3 of a robot to sense capacitance measurement values ​​during the non-contact and contact phases, respectively.

[0057] The multimodal signal processing unit 2 is connected to the flexible capacitance tomography sensor 1 via a connecting line. It acquires the capacitance measurement values ​​collected by the sensor in real time, processes them to generate multimodal tactile information, and feeds it back to the robot control system. Specifically, it performs object category recognition based on the capacitance measurement values ​​collected in the non-contact phase. If the identified object is a touchable object, it uses the capacitance measurement values ​​collected in the contact phase to reconstruct the ECT image, obtains a contact image, converts the contact image into a contact force distribution through a grayscale-pressure mapping relationship, and extracts the object contour using the geometric features of the contact image. The contact force distribution and object contour information obtained in this process are fed back to the robot control system in real time to guide the robot to achieve safe and accurate physical interaction. If the identified object is a non-touchable object, it immediately sends the object category recognition result to the robot control system. The robot control system adjusts the working path of the robot's end effector based on the recognition result to ensure the safety and intelligence of the operation.

[0058] In some embodiments, the flexible capacitive tomography sensor 1 employs a double-layer flexible electrode array manufactured using flexible printed circuit board technology. The top layer is an M×N rectangular copper electrode array, with the size of a single electrode varying between 10mm and 60mm, depending on application requirements. The net spacing between adjacent electrodes can also be adjusted between 3mm and 10mm to meet different resolution requirements. The thickness of the copper electrodes can vary between 12µm and 25µm to accommodate different conductivity and flexibility requirements. An anti-oxidation nickel-gold layer with a total thickness between 10µm and 15µm can be deposited on the surface of the copper electrodes to further enhance conductivity and long-term stability. The bottom layer is a flexible substrate 12, the planar dimensions of which should completely cover the electrode array of the top layer. The flexible substrate 12 is selected from a polyimide film with a thickness ranging from 110µm to 300µm, possessing excellent flexibility and bending resistance. Wiring is performed on the flexible substrate 12, with the wiring width selectable between 0.1mm and 0.3mm and the wiring spacing ≥0.5mm to ensure good electrical isolation between the electrodes. Electrode 11 is interconnected with the wiring via laser drilling. The diameter of the via is generally controlled between 200um and 250um to ensure the reliability and durability of the connection.

[0059] The capacitance measurements sensed by the flexible capacitance tomography sensor 1 during the non-contact and contact phases represent the mutual capacitance between M×N electrodes, forming a total of One mutual capacitance measurement channel.

[0060] In one specific embodiment of this application, the top layer of the flexible capacitance tomography sensor 1 adopts a 4×4 square copper electrode array. Each electrode 11 measures 40mm×40mm, with a net spacing of 5mm between adjacent electrodes, comprising a total of 16 independent electrodes. The copper electrodes are 18µm thick and coated with a 12.5µm thick anti-oxidation nickel-gold layer. The flexible substrate 12 is a 110µm thick polyimide film with a wiring width of 0.2mm and a wiring spacing ≥0.5mm. The capacitance measurements sensed by the flexible capacitance tomography sensor 1 in both the non-contact and contact phases are the total capacitance between each pair of the 16 electrodes. Mutual capacitance.

[0061] Understandably, based on the flexible characteristics of the flexible capacitive tomography sensor 1, it can be attached to the surface of the robot end effector 3 or the bionic finger, and is suitable for various curved surface installation environments, avoiding interference with the motion of the robot end effector 3. Near the robot end effector 3, the installation area of ​​the flexible capacitive tomography sensor 1 is formed by the outer peripheral surface of the robot end effector 3 itself, which meets the tactile perception needs of industrial robots in tasks such as grasping, sorting, and assembly.

[0062] In some embodiments, the multimodal signal processing unit 2 includes an acquisition module, a preprocessing module, an object recognition module, an image reconstruction module, a contact force sensing module, and a shape detection module.

[0063] The acquisition module can use a high-performance capacitance measurement chip TS3F605AQ5. It obtains the capacitance measurement value sensed by the flexible capacitance tomography sensor 1 by scanning the mutual capacitance values ​​between each electrode pair in the sensor pair one by one. The mutual capacitance measurement channels form a complete capacitance matrix.

[0064] Furthermore, to improve measurement accuracy, the acquisition module employs a rate-of-change measurement method to record the rate of capacitance change before and after sensing by the flexible capacitance tomography sensor 1, serving as the basis for subsequent analysis. Specifically: before each detection, an initial capacitance matrix C0 is recorded in the absence of a test object; after detecting an approaching object in a non-contact state, the module enters a non-contact sensing mode, sequentially scanning and acquiring the mutual capacitance value C of each electrode pair (i,j). ij This forms the current capacitance matrix C, where i,j = 1, 2, 3, ..., M × N, and i ≠ j. The current capacitance matrix C should include all excitation-detection combinations. For a given electrode pair (i,j), the i-th electrode serves as the excitation electrode, and the j-th electrode serves as the detection electrode, forming a mutual capacitance channel. The capacitance change rate δ is calculated for each electrode pair. ij :

[0065]

[0066] Among them, C 0,ij ΔC represents the initial capacitance between the i-th and j-th electrodes in the absence of a analyte. ij To detect the change in capacitance between the i-th electrode and the j-th electrode before and after.

[0067] The preprocessing module is used to process the data acquired by the acquisition module. The capacitance change rates are arranged in a set order to form a... 1-dimensional eigenvector X;

[0068] X = [δ] 12 ,δ 13 ,…,δ 1(M×N) ,δ 23 ,…,δ 2,M×N ,…,δ ij ,…,δ (M×N-1)(M×N) ]

[0069] The aforementioned feature vector X will be used as the input to the object recognition module.

[0070] The object recognition module has a pre-trained classification model. The feature vector obtained from the preprocessing module is input into the classification model to identify the object category, thereby determining whether the approaching object is a touchable object.

[0071] Furthermore, the classification model is trained according to the following steps:

[0072] Step 11: For various items of different materials and shapes, including but not limited to: metal blocks, plastic bottles, ceramic cups, glass, fabric, foam, fruit, and other common items, construct corresponding feature vectors according to the above-mentioned feature vector acquisition method, and label them with category labels. For each category of item, perform P data collections to obtain P feature vectors (P≥500), covering different postures (e.g., upright, tilted, sideways), different approach distances, and various environmental disturbances (e.g., temperature and humidity changes). The feature vectors and category labels of all items form the training dataset. q is the index of a sample in the training dataset D, and Q is the number of samples in the training dataset D.

[0073] Step 12: Use Support Vector Machine (SVM) as the classifier. Train the SVM classifier using toolkits such as Scikit-learn or LIBSVM and the training dataset D obtained in Step 11. Choose RBF (Gaussian kernel) as the kernel function. Optimize the penalty coefficient C and kernel function parameter γ through grid search to obtain the best classification performance. After training, save the SVM classifier parameters to obtain the classification model. This classification model outputs the predicted category of the object and the classification confidence.

[0074] Furthermore, the classification model integrates an incremental learning mechanism to handle unknown objects or samples with high uncertainty in practical applications. When the confidence level output by the classification model is lower than a preset threshold (set to 85% in this embodiment), the system determines the current input sample as "unknown / uncertain"; the feature vector and capacitance measurement value of the sample are cached in a local database, which is deployed in the memory of the multimodal signal processing unit; this memory supports a manual annotation interface, allowing operators to add true category labels to cached samples through a human-machine interface; if not labeled in time, the multimodal signal processing unit marks it as "unknown" for subsequent online learning; after accumulating a certain number of new samples, the incremental learning algorithm of support vector machines is used to locally update the existing classification model. This process only adjusts the support vector set and related parameters, avoiding retraining the original training dataset D; the updated classification model is immediately deployed to the object recognition module to achieve dynamic optimization of the model.

[0075] The image reconstruction module employs the Landweber iterative algorithm combined with the Tikhonov regularization strategy to reconstruct a two-dimensional image of the mutual capacitance acquired during the contact phase. Specifically, by setting an initial image distribution and continuously correcting errors, it gradually approximates the true capacitance distribution, generating a grayscale image reflecting the dielectric properties of the contact area. The specific steps include:

[0076] Step 21, Sensitivity Matrix Construction:

[0077] Establish a three-dimensional geometric model of an M×N flexible electrode array, where M and N are the number of rows and columns of the electrode array, respectively (e.g., 4×4). Import this geometric model into finite element simulation software (such as COMSOL or ANSYS) and set the material properties (including the air dielectric constant ε0, substrate material ε0, etc.). r ), boundary conditions (including grounding and excitation voltage), and the solution domain; apply a small dielectric perturbation to each electrode pair and calculate the resulting change in mutual capacitance; calculate the influence coefficient of each imaging unit in the imaging region on the mutual capacitance measurement, and construct the sensitivity matrix accordingly. in, Indicates the total number of measurable mutual capacitance channels; The number of pixel units representing the imaging region (2164 in this embodiment); elements in the sensitivity matrix S Indicates the first The change in dielectric constant of the pixel unit affects the _th Sensitivity of mutual capacitance measurements.

[0078] Step 22, Regularized Landweber Iterative Calculation:

[0079] Construct the capacitance change vector: ΔC = [C 12 -C 0,12 ,...,C ij -C 0,ij ,...,C (M×N-1)(M×N) -C 0,(M×N-1)(M×N) The image is initialized as either an all-zero image or a uniformly distributed image. The Landweber iteration formula is as follows:

[0080] G k+1 =G k +ηS T (ΔC-SG k )

[0081] Introducing the Tikhonov regularization term improves the result to:

[0082] G k+1 =G k +η(S T (ΔC-SGk )+λG k )

[0083] Among them, S T G is the transpose of the sensitivity matrix. k The reconstructed image is obtained in the k-th iteration. The maximum number of iterations is usually set to 100-200. η is the iteration step size, and λ is the Tikhonov regularization parameter used to control the smoothness of the image.

[0084] After iteration, the reconstructed image G is output. k Each pixel value represents the change in dielectric constant of that pixel region; the output reconstructed image G k Normalized to grayscale, it serves as a contact image, facilitating subsequent analysis and visualization.

[0085] The image reconstruction module provided in this embodiment has a response time of less than 50ms, which can meet the requirements for real-time feedback.

[0086] The contact force detection module is used to extract the region of interest (ROI) from the contact image under stable contact conditions, calculate the median grayscale value, and convert it into an actual force value through a pre-calibrated "grayscale-pressure" mapping relationship, outputting the contact force distribution. This enables contact force detection between the robot's end effector and the object. Specific steps include:

[0087] Step 31: Perform threshold segmentation on the contact image to extract the contact area:

[0088] In the contact image generated by the image reconstruction module, pixels with gray values ​​greater than a set threshold T are marked as high-response regions, generating a binarized potential contact region image. Connectivity analysis is performed on this potential contact region image to identify all interconnected pixel regions, and the largest connected region is selected as the Region of Interest (ROI). Within the determined ROI, the gray values ​​of all pixels are extracted, and the median of these gray values ​​is calculated as the feature value of the ROI. Using the median effectively reduces the impact of outliers on the results.

[0089] Step 32: Establish grayscale-pressure mapping relationship:

[0090] Under known force conditions, multiple sets of capacitance data were collected using a contact force detection module, and images were reconstructed using the aforementioned image reconstruction module to obtain a series of calibrated contact images. For each set of experiments, the median grayscale value of the corresponding ROI region and the actual applied pressure value were recorded. Based on this, a "grayscale-pressure" mapping table was constructed, which essentially represents the conversion relationship between the median grayscale value and the actual pressure.

[0091] Step 33, Contact Force Lookup Table:

[0092] When contact force needs to be detected, the median grayscale value of the current ROI region is first calculated according to step 31. Then, the force value corresponding to the closest median grayscale value is found from the "grayscale-pressure" mapping table established in step 32. Finally, the found actual force value is output for feedback adjustment of the robot control system.

[0093] The shape detection module performs Gaussian filtering to denoise the contact image, then uses an adaptive thresholding method to obtain a binary image. It extracts closed contours from the binary image using edge tracking. Subsequently, it calculates the minimum bounding rectangle, outputs the aspect ratio and orientation angle information, and identifies standard geometric shapes (such as circles, squares, and triangles) using the Hu invariant moment method, assisting the robot in determining the object's pose and shape category. Specifically, it includes the following steps:

[0094] Step 41, Contact Image Preprocessing:

[0095] Gaussian filtering is used to denoise the contact image output by the image reconstruction module. The cv2.GaussianBlur() function in the OpenCV library is used to filter the image. An adaptive thresholding method is used to convert the filtered grayscale image into a binary image to highlight the object outline.

[0096] Step 42, Contour Extraction:

[0097] Use the cv2.findContours() function from the OpenCV library to locate and extract object contours from a binary image. Determine the minimum bounding rectangle of each contour to obtain the basic geometric properties of the object.

[0098] Step 43, Shape Sorting and Feedback:

[0099] Standard geometric shapes (circles, squares, triangles, etc.) are classified based on Hu invariant moment features. The object shape is determined based on the extracted geometric features and Hu invariant moment results, and this shape information is fed back to the robot control system. The robot control system then adjusts the robot's motion state based on the multimodal perception results.

[0100] To verify the effectiveness of the robot multimodal tactile sensing device provided in the embodiments of the present invention, an experimental platform based on the UR5 collaborative robot was built to test objects of various materials and shapes, including plastic bottles, metal blocks, ceramic cups, and glass.

[0101] Test results show that:

[0102] In a non-contact state, the system can complete preliminary material identification within 30mm of the object, with an average identification accuracy of 93%.

[0103] In contact, the image reconstruction is highly accurate and can accurately identify pressure concentration points within the contact area;

[0104] The shape recognition module successfully identified basic geometric shapes such as circles, squares, and triangles.

[0105] The overall system response time is less than 100ms, which meets the real-time control requirements of the robot.

[0106] It is understood that the embodiments of the present invention provide a multimodal tactile sensing solution that can achieve both non-contact detection and contact perception. This allows for the early detection of an object's presence and attributes without physical contact, and the comprehensive acquisition of force distribution and morphological characteristics during contact, thereby improving the robot's safety and intelligence. Furthermore, based on ECT technology, the present invention enables continuous perception throughout the entire process from object approach to stable contact. ECT technology has the following significant advantages:

[0107] (1) Supports full-stage detection from non-contact to contact: ECT technology can identify objects and their dielectric properties in a non-contact state, and reconstruct the capacitance distribution map in a contact state, so as to achieve early identification and fine control.

[0108] (2) Strong visualization and perception capabilities: It generates a two-dimensional capacitance distribution map, which intuitively displays the change of dielectric constant, providing high-quality data for pressure calculation and material identification.

[0109] (3) Strong adaptability and unaffected by light: It does not depend on the light source and can adapt to dark, reflective or dusty environments, enhancing the stability and reliability of the system.

[0110] A second aspect of the present invention provides a robot tactile perception method, comprising:

[0111] The aforementioned multimodal tactile sensing device is assembled at the end effector of the robot;

[0112] In the non-contact phase, as the robot's end effector approaches the object, the object category is identified by the change in capacitance measurement obtained from the robot's multimodal tactile sensing device.

[0113] During the stable contact phase, contact force distribution images and object contour features are acquired simultaneously.

[0114] Based on the object type identified by the multimodal tactile sensing device, the obtained contact force, and the object's contour features, the robot's end effector motion strategy is dynamically adjusted. For example, if the detected contact force exceeds a preset safety threshold, the contact force is automatically reduced; if the object's shape is complex or contains fragile parts, path planning is optimized to avoid damage. This adjustment mechanism combines real-time perception and adaptive control. It not only improves the accuracy and safety of operations but also enhances the robot's autonomous operation capabilities in unknown environments.

[0115] It should be noted that the foregoing explanation of an embodiment of a robot multimodal tactile sensing device also applies to a robot tactile sensing method in this embodiment, and will not be repeated here.

[0116] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0117] Although examples of the invention have been shown and described above, it is understood that the above examples are exemplary and should not be construed as limiting the invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above examples within the scope of the invention.

Claims

1. A robot multimodal tactile sensing device based on capacitance tomography, characterized in that, include: A flexible capacitance tomography sensor is installed at the end effector of a robot to sense capacitance measurements during both non-contact and contact phases. A multimodal signal processing unit is used to acquire the capacitance measurement value in real time, process it to generate multimodal tactile information and feed it back to the robot control system. Among them, based on the capacitance measurement value in the non-contact stage, object category recognition is performed, which is divided into touchable objects and non-touchable objects. If the recognized object is a touchable object, the capacitance measurement value in the contact stage is used to reconstruct the ECT image to obtain a contact image. The contact image is converted into a contact force distribution through the gray-scale-pressure mapping relationship, and the geometric features of the contact image are used to extract the object contour.

2. The robot multimodal tactile sensing device according to claim 1, characterized in that, The flexible capacitive tomography sensor employs a dual-layer flexible electrode array manufactured using flexible printed circuit board technology, with the top layer being the electrode array and the bottom layer being a flexible substrate.

3. The robot multimodal tactile sensing device according to claim 1, characterized in that, The capacitance measurements sensed by the flexible capacitance tomography sensor during the non-contact and contact phases constitute the mutual capacitance between each pair of the M×N electrodes in the flexible capacitance tomography sensor, forming a total of One mutual capacitance measurement channel.

4. The robot multimodal tactile sensing device according to claim 3, characterized in that, The object category identification based on capacitance measurements during the non-contact phase includes: The capacitance measurement value is characterized as the capacitance change rate: Before each detection, an initial capacitance matrix C0 is recorded in the absence of a test object. After detecting an object approaching the flexible capacitance tomography sensor in a non-contact state, the mutual capacitance value C of each electrode pair (i,j) is scanned and acquired sequentially. ij This forms the current capacitance matrix C, where i,j = 1, 2, 3, ..., M × N, and i ≠ j; for each electrode pair, the capacitance change rate δ is calculated. ij : Among them, C 0,ij ΔC represents the initial capacitance between the i-th and j-th electrodes in the absence of a analyte. ij To detect the change in capacitance between the i-th electrode and the j-th electrode before and after; Will The capacitance change rates are arranged in a set order to form... 3D eigenvector X: X=[δ 12 ,…,d 1(M×N) ,…,d ij ,…,d (M×N-1)(M×N) ]; The feature vector X is input into a pre-trained classification model to determine whether a nearby object is a reachable object.

5. The robot multimodal tactile sensing device according to claim 4, characterized in that, The training steps for the classification model include: For different items, corresponding feature vectors are constructed according to the method of obtaining the feature vector X, and their category labels are labeled. Each item is sampled multiple times to cover the situation of the item in different postures, proximity distances and environmental interference. The feature vectors of all items and their category labels form the training dataset D. A support vector machine is used as a classifier, and the classifier is trained based on the training dataset D to obtain a classification model. The classification model outputs the predicted category of the object and the classification confidence.

6. The robot multimodal tactile sensing device according to claim 4, characterized in that, The classification model supports an incremental learning mechanism. When the confidence level output by the classification model is lower than a preset threshold, the current input sample is determined to be in the "unknown / uncertain" category. The feature vector and capacitance measurement value of the input sample are cached in a local database, which is deployed in the memory of the multimodal signal processing unit. The memory has a manual annotation interface to add true category labels to the cached samples. If the labels are not added in time, the multimodal signal processing unit automatically marks the current input sample as "unknown". After accumulating a set number of new samples, the classification model is locally updated using an incremental learning algorithm based on a support vector machine.

7. The robot multimodal tactile sensing device according to claim 4, characterized in that, The method of reconstructing the ECT image using capacitance measurements during the contact phase to obtain the contact image includes: A three-dimensional simulation geometric model of M×N electrodes in the flexible capacitance tomography sensor is established. A small dielectric perturbation is applied to each electrode pair, and the resulting change in mutual capacitance is calculated. The influence coefficient of each imaging unit in the imaging region on the mutual capacitance measurement value is calculated, and a sensitivity matrix is ​​constructed accordingly. in, This indicates the number of pixel units that divide the imaging region; Construct the capacitance change vector: ΔC = [C 12 -C 0,12 ,...,C ij -C 0,ij ,...,C (M×N-1)(M×N) -C 0,(M×N-1)(M×N) Initialize the image as either an all-zero image or a uniformly distributed image; Introducing the Tikhonov regularization term, we have: G k+1 =G k +η(S T (ΔC-SG k )+λG k ) Among them, S T G is the transpose of the sensitivity matrix. k Let η be the reconstructed image obtained in the k-th iteration, η be the iteration step size, and λ be the Tikhonov regularization parameter. After iteration, the reconstructed image G is output. k Each pixel value represents the change in dielectric constant of that pixel region, and the output reconstructed image G k Normalized to a grayscale image, which serves as the contact image.

8. The robot multimodal tactile sensing device according to claim 1, characterized in that, The step of converting the contact image into a contact force distribution through a grayscale-pressure mapping relationship includes: Pixels with gray values ​​greater than a set threshold in the contact image are marked as high-response regions, generating a binarized potential contact region image; connected component analysis is performed on the potential contact region image to identify all interconnected pixel regions, and the connected region with the largest area is selected as the ROI region. The gray values ​​of all pixels in the ROI region are extracted, and the median is calculated as the feature value of the ROI region. The contact force distribution is obtained based on the feature values ​​of the ROI region and the pre-constructed gray-scale-pressure mapping relationship; The grayscale-pressure mapping relationship is a "grayscale-pressure" mapping table, and the construction steps include: Under known force conditions, multiple sets of capacitance measurements are obtained using the flexible capacitance tomography sensor. A series of calibrated contact images are obtained through ECT image reconstruction. For each set of experiments, the median gray value of the corresponding ROI region and the actual applied pressure value are recorded to obtain a "gray-scale-pressure" mapping table.

9. The robot multimodal tactile sensing device according to claim 1, characterized in that, When an object is identified as a touchable object, the multimodal signal processing unit feeds back the obtained contact force distribution and object contour information to the robot control system in real time. When an object is identified as untouchable, the robot control system adjusts the working path of the robot's end effector based on the object category recognition result obtained by the multimodal signal processing unit.

10. A robot tactile perception method, characterized in that, include: The robot's end effector is equipped with a robot multimodal tactile sensing device according to any one of claims 1 to 9; During the non-contact phase, the changes in capacitance measurements obtained through the robot's multimodal tactile sensing device identify approaching objects to obtain object categories. If the object is a touchable object, then during the stable contact phase, the robot's multimodal tactile sensing device is used to simultaneously acquire the contact force distribution and the object's outline. The robot control system adjusts the motion strategy of the robot's end effector based on the multimodal tactile information from the robot's multimodal tactile sensing device.

Images