Capacitive tactile tomography method and system based on simulation-real migration learning

Abstract

This invention proposes a capacitive tactile tomography method and system based on simulation-real transfer learning. The method includes: constructing a three-dimensional finite element simulation model of the capacitive tactile sensor to be used for imaging, thereby generating a simulation dataset by simulating capacitance changes in an empty field and under the condition of placing a detection object, and pre-training the imaging network model; using the capacitive tactile sensor to acquire capacitance changes in an empty field and under the condition of placing a detection object in a real scene to generate a real dataset; and using the real dataset to fine-tune the parameters of the pre-trained imaging network model to obtain the final imaging network model, thereby realizing capacitive tactile tomography. This invention can effectively utilize low-cost simulation data, correct physical deviations through transfer learning, and achieve high-precision, robust real-time tactile imaging.

Description

Technical Field

[0001] This invention belongs to the field of tactile sensing and computational imaging technology, and in particular relates to a capacitive tactile tomography method and system based on simulation-real transfer learning. Background Technology

[0002] Sensors based on the principle of capacitive tactile tomography (ECT) have attracted attention due to their simple structure and flexible extensibility. Traditional algorithms rely on precise sensitivity matrices and are affected by the "soft field" effect, resulting in low imaging resolution. Although deep learning methods have improved imaging quality in recent years, acquiring tens of thousands of sets of labeled real-world tactile data in practical systems is time-consuming and labor-intensive. Simulation software can generate massive amounts of data at low cost, but simulation models cannot fully simulate real hardware noise, edge effects, and material nonlinearities, resulting in a "simulation-reality gap." Therefore, achieving efficient and robust capacitive tactile tomography is a pressing problem that needs to be solved. Summary of the Invention

[0003] To overcome the shortcomings of existing capacitive tactile imaging technologies, such as the difficulty in acquiring realistic labeled data, high training costs, and poor generalization ability of pure simulation models in practical applications, this invention proposes a capacitive tactile tomography method and system based on simulation-real transfer learning. This invention can effectively utilize low-cost simulation data and correct physical biases through transfer learning, achieving high-precision and robust real-time tactile imaging.

[0004] A first aspect of this invention proposes a capacitive tactile tomography method based on simulation-real transfer learning, comprising:

[0005] A three-dimensional finite element simulation model is constructed for the capacitive tactile sensor to be used for imaging. Using the simulation model, a simulation dataset is generated by simulating the capacitance changes in the empty field state and the state in which the detection object is placed. The samples in the simulation dataset contain normalized feature vectors of the simulated capacitance changes and corresponding simulated tactile image labels.

[0006] A CAVE conditional variational autoencoder model is constructed as an imaging network model; based on the transfer learning strategy, the imaging network model is pre-trained using the simulation dataset to obtain the pre-trained imaging network model.

[0007] Using the capacitive tactile sensor, a real dataset is generated by acquiring the capacitance changes in an empty field and when a detection object is placed in a real scene. The input of the samples in the real dataset is a normalized feature vector of the real capacitance change, and the label is the conductivity distribution matrix corresponding to the location of the detection object.

[0008] Based on the transfer learning strategy, the parameters of the pre-trained imaging network model are fine-tuned using the real dataset to obtain the final imaging network model.

[0009] Capacitive tactile tomography is achieved using the aforementioned imaging network model.

[0010] In one specific embodiment of the present invention, it further includes:

[0011] During simulation, the simulated capacitance vector under the condition of being placed in the virtual detection object is obtained through finite element analysis. Simulation reference vector under no-load condition The capacitance difference between the two is calculated and vector-level normalization is performed, which is then used as input to the simulation sample.

[0012] The physical distribution of virtual detection objects on the sensing plane of the simulation model is mapped to imaging grid labels. The area covered by the virtual detection objects is marked with a normalized response value, and the background area is marked with 0. Finally, a binary tactile image is generated as the label of the corresponding simulation sample.

[0013] In one specific embodiment of the present invention, it further includes:

[0014] The imaging network model includes an encoding subnetwork and a decoding subnetwork; wherein, the encoding subnetwork is composed of multiple fully connected layers, which is used to compress the original input features and map them to a high-dimensional latent space through nonlinear transformation, so as to extract the deep spatial features after the capacitance field is disturbed.

[0015] The decoding subnetwork consists of multiple transposed convolutional layers, which are used to reconstruct and upsample the features of the latent space, and finally restore them to a two-dimensional tactile distribution image.

[0016] In one specific embodiment of the present invention, it further includes:

[0017] When fine-tuning the parameters of the pre-trained imaging network model using the real dataset, after loading the weight parameters of the pre-trained model, the weights of the encoding sub-network are kept frozen, and the weight parameters of the decoding sub-network are fine-tuned by backpropagation using the real dataset.

[0018] A second aspect of the present invention provides a capacitive tactile tomography device based on simulation-real transfer learning, comprising:

[0019] The simulation dataset construction module is used to construct a three-dimensional finite element simulation model of the capacitive tactile sensor to be used for imaging. Using the simulation model, the capacitance changes generated by simulating the empty field state and the state of placing the detection object are used to generate a simulation dataset. The samples in the simulation dataset contain normalized feature vectors of the simulated capacitance changes and corresponding simulated tactile image labels.

[0020] The model pre-training module is used to construct a CAVE conditional variational autoencoder model as an imaging network model; based on the transfer learning strategy, the imaging network model is pre-trained using the simulation dataset to obtain the pre-trained imaging network model.

[0021] The real dataset construction module is used to generate a real dataset by acquiring the capacitance changes generated in the empty field state and the state with the detection object placed in a real scene using the capacitive tactile sensor; the input of the samples in the real dataset is the normalized feature vector of the real capacitance change, and the label is the conductivity distribution matrix corresponding to the position of the detection object;

[0022] The model fine-tuning module is used to fine-tune the parameters of the pre-trained imaging network model based on the transfer learning strategy and using the real dataset to obtain the final imaging network model.

[0023] An imaging module is used to realize capacitive tactile tomography using the imaging network model.

[0024] In one specific embodiment of the present invention, it further includes:

[0025] During simulation, the simulated capacitance vector under the condition of being placed in the virtual detection object is obtained through finite element analysis. Simulation reference vector under no-load condition The capacitance difference between the two is calculated and vector-level normalization is performed, which is then used as input to the simulation sample.

[0026] The physical distribution of virtual detection objects on the sensing plane of the simulation model is mapped to imaging grid labels. The area covered by the virtual detection objects is marked with a normalized response value, and the background area is marked with 0. Finally, a binary tactile image is generated as the label of the corresponding simulation sample.

[0027] In one specific embodiment of the present invention, it further includes:

[0028] The imaging network model includes an encoding subnetwork and a decoding subnetwork; wherein, the encoding subnetwork is composed of multiple fully connected layers, which is used to compress the original input features and map them to a high-dimensional latent space through nonlinear transformation, so as to extract the deep spatial features after the capacitance field is disturbed.

[0029] The decoding subnetwork consists of multiple transposed convolutional layers, which are used to reconstruct and upsample the features of the latent space, and finally restore them to a two-dimensional tactile distribution image.

[0030] In one specific embodiment of the present invention, it further includes:

[0031] When fine-tuning the parameters of the pre-trained imaging network model using the real dataset, after loading the weight parameters of the pre-trained model, the weights of the encoding sub-network are kept frozen, and the weight parameters of the decoding sub-network are fine-tuned by backpropagation using the real dataset.

[0032] A third aspect of the present invention provides an electronic device comprising:

[0033] At least one processor; and a memory communicatively connected to said at least one processor;

[0034] The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to perform the above-described capacitive tactile tomography method based on simulation-real transfer learning.

[0035] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions for causing the computer to execute the above-described capacitive tactile tomography method based on simulation-real transfer learning.

[0036] The features and beneficial effects of this invention are as follows:

[0037] By combining simulation data augmentation with transfer learning techniques, this invention significantly reduces the need for real labeled samples and lowers training costs. The CAVE autoencoder structure effectively solves the "soft field" nonlinear mapping problem in capacitive imaging, improving imaging quality. Through targeted transfer fine-tuning, this invention eliminates the physical feature differences between simulation and real hardware, significantly improving the system's positioning accuracy and shape recognition robustness in complex real-world environments. Attached Figure Description

[0038] Figure 1 This is an overall flowchart of a capacitive tactile tomography method based on simulation-real transfer learning according to an embodiment of the present invention.

[0039] Figure 2 This is a schematic diagram of a simulation model of a capacitive tactile sensor in a specific embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the distribution of randomly generated detection objects and their corresponding image labels in a simulation environment, according to a specific embodiment of the present invention. Detailed Implementation

[0041] This invention proposes a capacitive tactile tomography method and system based on simulation-real transfer learning. The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0042] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] A first aspect of this invention proposes a capacitive tactile tomography method based on simulation-real transfer learning, comprising:

[0044] A three-dimensional finite element simulation model is constructed for the capacitive tactile sensor to be used for imaging. Using the simulation model, a simulation dataset is generated by simulating the capacitance changes in the empty field state and the state in which the detection object is placed. The samples in the simulation dataset contain normalized feature vectors of the simulated capacitance changes and corresponding simulated tactile image labels.

[0045] A CAVE conditional variational autoencoder model is constructed as an imaging network model; based on the transfer learning strategy, the imaging network model is pre-trained using the simulation dataset to obtain the pre-trained imaging network model.

[0046] Using the capacitive tactile sensor, a real dataset is generated by acquiring the capacitance changes in an empty field and when a detection object is placed in a real scene. The input of the samples in the real dataset is a normalized feature vector of the real capacitance change, and the label is the conductivity distribution matrix corresponding to the location of the detection object.

[0047] Based on the transfer learning strategy, the parameters of the pre-trained imaging network model are fine-tuned using the real dataset to obtain the final imaging network model.

[0048] Capacitive tactile tomography is achieved using the aforementioned imaging network model.

[0049] In a specific embodiment of the present invention, the overall process of the capacitive tactile tomography method based on simulation-real transfer learning is as follows: Figure 1 As shown, it includes:

[0050] 1) Construct a three-dimensional finite element simulation model of the capacitive tactile sensor to be used for imaging. Use this simulation model to generate a simulation dataset by simulating capacitance changes under empty and object-placed conditions. The samples in the simulation dataset contain normalized feature vectors of simulated capacitance changes and corresponding simulated tactile image labels. Specific steps are as follows:

[0051] 1-1) Construct a three-dimensional finite element simulation model of the capacitive tactile sensor to be used for imaging.

[0052] In one specific embodiment of the present invention, a three-dimensional electrostatic field model of the capacitive tactile sensor is established in the finite element simulation software COMSOL Multiphysics. Figure 2 This is a schematic diagram of the simulation model of the capacitive tactile sensor in this embodiment. Figure 2 As shown, this sensor uses The planar array consists of 16 square electrodes evenly distributed on an FR4 PCB substrate. Each small square in the diagram represents an electrode with a side length of 2 mm, used to detect local electric field or capacitance changes, enabling touch positioning and pressure distribution sensing. The center-to-center spacing of the electrodes in both the horizontal and vertical directions is 18 mm, and the overall array area spans 78 mm horizontally and vertically, exhibiting a symmetrical distribution. The PCB substrate size is 100 mm × 100 mm, and the electrode array is 11 mm from the edge of the substrate. The small dots in the diagram represent signal lead-out pads for each electrode, typically located on one side of the electrode or at the edge of the substrate, facilitating connection to external circuitry. To protect the electrodes and simulate a realistic contact environment, a flexible dielectric overlay layer of a predetermined thickness is placed above the electrodes.

[0053] 1-2) Use the simulation model generated in step 1-1) to generate a simulation dataset.

[0054] In one specific embodiment of the present invention, the COMSOL LiveLink for MATLAB interface is used to automate the generation of simulation data. In the simulation environment, virtual detection objects are randomly placed above the capacitive tactile sensor using a script. Figure 3 This is a schematic diagram illustrating the distribution of randomly generated detection objects and their corresponding image labels in a simulation environment, according to a specific embodiment of the present invention. In this embodiment, the randomly generated virtual detection objects during the simulation process include different quantities (one or more), shapes (such as cylinders, cubes), and spatial coordinates. And geometric dimensions to simulate diverse tactile contact scenarios, and simultaneously generate corresponding image labels. For example... Figure 3 As shown in the upper middle section of the attached diagram, the cylindrical detection object is located above the electrode plate. Its position, size, material properties, etc., will affect the distribution of the sensing signal of the electrode array below.

[0055] Specifically, in one embodiment of the present invention, each time a virtual detection object is placed, the mesh is re-divided and the Maxwell's equations for the electrostatic field are solved during the finite element analysis. Through a cyclic excitation scheme, the 120-dimensional independent mutual capacitance feature vectors formed by pairwise pairing of the 16 electrodes are extracted sequentially. This process derives the simulated capacitance vectors under the condition of the detection object being placed. Simulation reference vector under no-load condition The capacitance difference between the two is calculated and vector-level normalization is performed, which is then used as input to the simulation sample.

[0056] Simultaneously, the physical distribution of the virtual detected objects on the sensing plane is mapped to imaging grid labels. In this embodiment, the following is employed: A pixel grid is used to label the area covered by the virtual detection object with a normalized response value, while the background area is labeled with 0, ultimately generating a binarized tactile image as the label for the corresponding simulated sample. For example... Figure 3 As shown in the lower middle of the attached diagram, the large black area represents the background area, while the white circular area in the middle represents the projection area of ​​the cylindrical object being detected, which intuitively shows the relative position of the object being detected above the electrode plate.

[0057] In this embodiment, 10,000 sets of simulation datasets containing 120-dimensional simulated capacitance difference vectors and corresponding binarized tactile image labels are finally generated as source domain data for transfer learning.

[0058] 2) Construct a CAVE convolutional autoencoder model as an imaging network model, and pre-train the imaging network model using the simulation dataset obtained in step 1) to obtain a pre-trained imaging network model.

[0059] In this embodiment, an imaging network model based on the CAVE (Conditional Variational Autoencoder) architecture is constructed. The input layer of this model has 120 nodes, directly corresponding to the extracted cross-capacitance differential features. The imaging network model includes an encoding sub-network and a decoding sub-network. The encoding sub-network consists of multiple fully connected layers (three fully connected layers are used in a specific embodiment of this invention), responsible for compressing and mapping the original input features to a high-dimensional latent space through nonlinear transformations to extract the deep spatial features after the capacitance field is perturbed.

[0060] Furthermore, the imaging network model also includes a decoding sub-network composed of multiple transposed convolutional layers (in a specific embodiment of this invention, three transposed convolutional layers are used). The role of the decoding sub-network is to reconstruct and upsample the abstract features of the latent space, ultimately restoring them into a high-resolution two-dimensional tactile distribution image. Supervised training is performed using a simulation dataset, and mean squared error (MSE) is used as the loss function, enabling the model to initially grasp the physical mapping relationship between changes in capacitance field and the distribution of objects in space.

[0061] In this embodiment, a transfer learning strategy is used to train the model. Specific training parameters are set as follows: the Adam optimizer is used, the initial learning rate is 1e-4 with dynamic decay support, the batch size is 32, and the maximum number of training epochs is 200. The loss function is the mean squared error loss, selected based on the task type. Training termination conditions include reaching the maximum number of training epochs, the validation loss not significantly decreasing for 20 consecutive epochs (early termination), signs of overfitting appearing (validation loss continuously increasing), or the model reaching a preset performance threshold on the validation set (e.g., positioning error below 2 mm). This transfer learning strategy employs a phased training approach, first fixing the backbone and training the top layer, then fine-tuning the overall model, thereby achieving efficient and stable training with limited sensor data.

[0062] 3) A real dataset is generated by using a capacitive tactile sensor to obtain the capacitance changes generated in the empty field state and the state with the detection object placed in a real scene; the input of the samples in the real dataset is the normalized feature vector of the real capacitance change, and the label is the two-dimensional coordinates of the detection object.

[0063] In this embodiment, in a real-world scenario, a capacitive tactile sensor is used to collect reference capacitance data under an empty field condition. And capacitance data after placing the actual test object. The difference between the two is calculated and vector-level normalization is performed, which is then used as the input to the real sample.

[0064] In one specific embodiment of the present invention, the capacitive tactile sensor is a 16-electrode sensor. The planar array sensor's geometric parameters are completely consistent with the simulation model. The sensor's electrode array incorporates a high-precision capacitance detection module (CDC) and a high-speed multiplexing switch, enabling full-cycle scanning of 120 independent mutual capacitance signals. The acquired digital signals are uploaded to the host computer computing unit via a universal communication bus interface.

[0065] Considering the unavoidable presence of parasitic capacitance, trace coupling, and environmental background noise in actual hardware circuits, directly using pre-trained models trained solely on simulation data often fails to achieve ideal imaging results. Therefore, this embodiment uses a standard pressure head to collect approximately 200-500 sets of real labeled data in a real experimental environment.

[0066] In a specific embodiment of the present invention, the data acquisition process for the real dataset is described as follows: First, the empty-field reference capacitance vector is measured when there is no obstruction or contact on the surface of the capacitive tactile sensor. In this embodiment, a standard cylindrical metal weight is used as the actual object to be tested, and it is placed directly on the surface of the sensor hardware planar circuit board (FR4 PCB electrode array) for the experiment. The regular geometry, precisely known diameter, height, and mass of the weight provide a stable and repeatable detection benchmark for the system. The real-time capacitance vector under the current state is measured. Finally, calculate the difference vector between the two. By calculating this difference vector and performing overall mapping normalization, the influence of the system's inherent parasitic capacitance can be effectively offset. The core content of the label is the precise two-dimensional coordinates (x, y) of the weight on the circuit board plane (in millimeters), and the conductivity distribution matrix is ​​ultimately used as the label for each sample in the real dataset.

[0067] 4) Based on the transfer learning strategy, the imaging network model pre-trained in step 2) is fine-tuned using the real dataset obtained in step 3) to obtain the final imaging network model.

[0068] In this embodiment, a transfer learning strategy is employed to optimize the pre-trained imaging network model. After loading the weight parameters of the pre-trained model, the weights of the encoding sub-network are kept frozen (i.e., not participating in gradient updates) to preserve its ability to extract general electric field spatial features learned from simulations. Backpropagation fine-tuning training is performed on the weight parameters of the decoding sub-network using only real datasets. The backbone network used for feature extraction is first frozen, and only the head is trained. Then, the end layers near the output of the backbone network are unfrozen to participate in fine-tuning. Simultaneously, weight decay, gradient pruning, and an early stopping mechanism based on validation loss are introduced to ensure stable convergence of the model on limited real weight detection data, ultimately achieving millimeter-level positioning accuracy and over 99% presence recognition accuracy.

[0069] Through this local fine-tuning transfer learning approach, the imaging network model can quickly compensate for and adapt to the electrical signal characteristics, parasitic capacitance bias, and noise distribution of the real hardware system while preserving the physical mapping rules. The final generated imaging network model not only possesses the breadth of simulation data but also the accuracy of real-world environments.

[0070] 5) Use the imaging network model obtained in step 4) to realize capacitive tactile tomography.

[0071] In this embodiment, a capacitive tactile sensor is used to collect capacitance data of the object to be imaged. The difference between this capacitance data and the collected reference capacitance data in the empty field is calculated and vector-level normalization is performed to generate a normalized capacitance difference feature vector. This capacitance difference feature vector is input into the final imaging network model to output a tactile tomographic image.

[0072] When imaging using the method described in this embodiment, the geometric contours and center positions of single-point and multi-point contact targets can be clearly and accurately reproduced, with positioning errors maintained at the millimeter level, demonstrating extremely strong robustness.

[0073] To implement the above embodiments, a second aspect of the present invention proposes a capacitive tactile tomography device based on simulation-real transfer learning, comprising:

[0074] The simulation dataset construction module is used to construct a three-dimensional finite element simulation model of the capacitive tactile sensor to be used for imaging. Using the simulation model, the capacitance changes generated by simulating the empty field state and the state of placing the detection object are used to generate a simulation dataset. The samples in the simulation dataset contain normalized feature vectors of the simulated capacitance changes and corresponding simulated tactile image labels.

[0075] The model pre-training module is used to construct a CAVE conditional variational autoencoder model as an imaging network model; based on the transfer learning strategy, the imaging network model is pre-trained using the simulation dataset to obtain the pre-trained imaging network model.

[0076] The real dataset construction module is used to generate a real dataset by acquiring the capacitance changes generated in the empty field state and the state with the detection object placed in a real scene using the capacitive tactile sensor; the input of the samples in the real dataset is the normalized feature vector of the real capacitance change, and the label is the conductivity distribution matrix corresponding to the position of the detection object;

[0077] The model fine-tuning module is used to fine-tune the parameters of the pre-trained imaging network model based on the transfer learning strategy and using the real dataset to obtain the final imaging network model.

[0078] An imaging module is used to realize capacitive tactile tomography using the imaging network model.

[0079] In one specific embodiment of the present invention, it further includes:

[0080] During simulation, the simulated capacitance vector under the condition of being placed in the virtual detection object is obtained through finite element analysis. Simulation reference vector under no-load condition The capacitance difference between the two is calculated and vector-level normalization is performed, which is then used as input to the simulation sample.

[0081] The physical distribution of virtual detection objects on the sensing plane of the simulation model is mapped to imaging grid labels. The area covered by the virtual detection objects is marked with a normalized response value, and the background area is marked with 0. Finally, a binary tactile image is generated as the label of the corresponding simulation sample.

[0082] In one specific embodiment of the present invention, it further includes:

[0083] The imaging network model includes an encoding subnetwork and a decoding subnetwork; wherein, the encoding subnetwork is composed of multiple fully connected layers, which is used to compress the original input features and map them to a high-dimensional latent space through nonlinear transformation, so as to extract the deep spatial features after the capacitance field is disturbed.

[0084] The decoding subnetwork consists of multiple transposed convolutional layers, which are used to reconstruct and upsample the features of the latent space, and finally restore them to a two-dimensional tactile distribution image.

[0085] In one specific embodiment of the present invention, it further includes:

[0086] When fine-tuning the parameters of the pre-trained imaging network model using the real dataset, after loading the weight parameters of the pre-trained model, the weights of the encoding sub-network are kept frozen, and the weight parameters of the decoding sub-network are fine-tuned by backpropagation using the real dataset.

[0087] This enables the effective use of low-cost simulation data and the correction of physical biases through transfer learning, achieving high-precision and robust real-time tactile imaging.

[0088] To implement the above embodiments, a third aspect of the present invention provides an electronic device, comprising:

[0089] At least one processor; and a memory communicatively connected to said at least one processor;

[0090] The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to perform the above-described capacitive tactile tomography method based on simulation-real transfer learning.

[0091] To implement the above embodiments, a fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions for causing the computer to execute the above-described capacitive tactile tomography method based on simulation-real transfer learning.

[0092] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0093] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform a capacitive tactile tomography method based on simulation-real transfer learning according to the above embodiments.

[0094] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0095] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0096] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0097] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0098] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0099] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0100] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0101] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0102] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A capacitive tactile tomography method based on simulation-real transfer learning, characterized in that, include: A three-dimensional finite element simulation model is constructed for the capacitive tactile sensor to be used for imaging. Using the simulation model, a simulation dataset is generated by simulating the capacitance changes in the empty field state and the state in which the detection object is placed. The samples in the simulation dataset contain normalized feature vectors of the simulated capacitance changes and corresponding simulated tactile image labels. A CAVE conditional variational autoencoder model is constructed as an imaging network model; based on the transfer learning strategy, the imaging network model is pre-trained using the simulation dataset to obtain the pre-trained imaging network model. Using the capacitive tactile sensor, a real dataset is generated by acquiring the capacitance changes in an empty field and when a detection object is placed in a real scene. The input of the samples in the real dataset is a normalized feature vector of the real capacitance change, and the label is the conductivity distribution matrix corresponding to the location of the detection object. Based on the transfer learning strategy, the parameters of the pre-trained imaging network model are fine-tuned using the real dataset to obtain the final imaging network model. Capacitive tactile tomography is achieved using the aforementioned imaging network model.

2. The method according to claim 1, characterized in that, Also includes: During simulation, the simulated capacitance vector under the condition of being placed in the virtual detection object is obtained through finite element analysis. Simulation reference vector under no-load condition The capacitance difference between the two is calculated and vector-level normalization is performed, which is then used as input to the simulation sample. The physical distribution of virtual detection objects on the sensing plane of the simulation model is mapped to imaging grid labels. The area covered by the virtual detection objects is marked with a normalized response value, and the background area is marked with 0. Finally, a binary tactile image is generated as the label of the corresponding simulation sample.

3. The method according to claim 2, characterized in that, Also includes: The imaging network model includes an encoding subnetwork and a decoding subnetwork; wherein, the encoding subnetwork is composed of multiple fully connected layers, which is used to compress the original input features and map them to a high-dimensional latent space through nonlinear transformation, so as to extract the deep spatial features after the capacitance field is disturbed. The decoding subnetwork consists of multiple transposed convolutional layers, which are used to reconstruct and upsample the features of the latent space, and finally restore them to a two-dimensional tactile distribution image.

4. The method according to claim 3, characterized in that, Also includes: When fine-tuning the parameters of the pre-trained imaging network model using the real dataset, after loading the weight parameters of the pre-trained model, the weights of the encoding sub-network are kept frozen, and the weight parameters of the decoding sub-network are fine-tuned by backpropagation using the real dataset.

5. A capacitive tactile tomography device based on simulation-real transfer learning, characterized in that, include: The simulation dataset construction module is used to construct a three-dimensional finite element simulation model of the capacitive tactile sensor to be used for imaging. Using the simulation model, the capacitance changes generated by simulating the empty field state and the state of placing the detection object are used to generate a simulation dataset. The samples in the simulation dataset contain normalized feature vectors of the simulated capacitance changes and corresponding simulated tactile image labels. The model pre-training module is used to construct the CAVE conditional variational autoencoder model as an imaging network model. Based on the transfer learning strategy, the imaging network model is pre-trained using the simulation dataset to obtain the pre-trained imaging network model. The real dataset construction module is used to generate a real dataset by acquiring the capacitance changes generated in the empty field state and the state with the detection object placed in a real scene using the capacitive tactile sensor; the input of the samples in the real dataset is the normalized feature vector of the real capacitance change, and the label is the conductivity distribution matrix corresponding to the position of the detection object; The model fine-tuning module is used to fine-tune the parameters of the pre-trained imaging network model based on the transfer learning strategy and using the real dataset to obtain the final imaging network model. An imaging module is used to realize capacitive tactile tomography using the imaging network model.

6. The apparatus according to claim 5, characterized in that, Also includes: During simulation, the simulated capacitance vector under the condition of being placed in the virtual detection object is obtained through finite element analysis. Simulation reference vector under no-load condition The capacitance difference between the two is calculated and vector-level normalization is performed, which is then used as input to the simulation sample. The physical distribution of virtual detection objects on the sensing plane of the simulation model is mapped to imaging grid labels. The area covered by the virtual detection objects is marked with a normalized response value, and the background area is marked with 0. Finally, a binary tactile image is generated as the label of the corresponding simulation sample.

7. The apparatus according to claim 6, characterized in that, Also includes: The imaging network model includes an encoding subnetwork and a decoding subnetwork; wherein, the encoding subnetwork is composed of multiple fully connected layers, which is used to compress the original input features and map them to a high-dimensional latent space through nonlinear transformation, so as to extract the deep spatial features after the capacitance field is disturbed. The decoding subnetwork consists of multiple transposed convolutional layers, which are used to reconstruct and upsample the features of the latent space, and finally restore them to a two-dimensional tactile distribution image.

8. The apparatus according to claim 7, characterized in that, Also includes: When fine-tuning the parameters of the pre-trained imaging network model using the real dataset, after loading the weight parameters of the pre-trained model, the weights of the encoding sub-network are kept frozen, and the weight parameters of the decoding sub-network are fine-tuned by backpropagation using the real dataset.

9. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method according to any one of claims 1-4.

Images