Sensing apparatus and apparatus

By distributing electrodes on the surface of deformable objects to capture capacitive signals and using machine learning models to process data, the problem of insufficient monitoring accuracy of shape and deformation information in the prior art is solved, and high-precision 3D shape reconstruction and deformation detection are realized, suitable for Soft robot perception application.

CN118922683APending Publication Date: 2024-11-08THE UNIV COURT OF THE UNIV OF EDINBURGH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380029039.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-23
Filing Date
2023-03-22
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The prior art is difficult to effectively monitor and perceive the shape and deformation information of deformable objects, especially in complex multimodal deformation scenarios, traditional protothetic perception systems have insufficient accuracy and flexibility.

Method used

Using multiple electrodes distributed on the surface of deformable objects, shape and deformation information are captured through capacitive signal output, and capacitive signal data is processed in combination with machine learning models to achieve high-resolution 3D shape reconstruction and deformation detection.

Benefits of technology

It realizes high-precision 3D shape reconstruction and deformation detection in complex deformation scenarios, improves the soft robot's perception of motion state, and is suitable for fields such as biomedical and human-computer interaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118922683A_ABST
    Figure CN118922683A_ABST
Patent Text Reader

Abstract

A sensing device comprising: a plurality of electrodes configured to be distributed around one or more surfaces of a deformable object, where the plurality of electrodes are operable to generate a capacitive signal output from a plurality of selected pairs of the plurality of electrodes, where the plurality of electrodes are distributed around the one or more surfaces of the deformable object, where the plurality of electrodes are operable to generate a capacitive signal output from a plurality of selected pairs of the plurality of electrodes; such that the plurality of selected pairs comprise at least one proximal pair and at least one non-proximal pair, wherein the generated capacitive signal output of the pair of electrodes depends on at least one of: a distance between the electrodes; the shape and / or orientation of the electrodes and at least one characteristic of the material between the pairs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a sensing device and apparatus, for example, a sensing device and apparatus for use with a deformable object. Background Art

[0002] Humans can effortlessly perceive their posture, movement, and position via a network of mechano-sensory networks throughout their bodies. This ability, known as proprioception, enables humans to control their bodies efficiently and accurately and is also an important requirement for smart robots to perform dexterous manipulation. The proprioceptive system of rigid robots is well known and used in multiple applications. Summary of the invention

[0003] According to a first aspect, there is provided a sensing device comprising: a plurality of electrodes configured to be distributed around one or more surfaces of a deformable object, wherein the plurality of electrodes are operable to generate a signal output from a plurality of selected pairings of the plurality of electrodes. The plurality of electrodes may be distributed around the one or more surfaces of the deformable object such that the plurality of selected pairings include at least one proximal pairing and at least one non-proximal pairing. The generated signal output of the electrode pairings may depend on at least one of: a distance between the electrodes; a shape and / or orientation of the electrodes and at least one property of a material between the pairings. The generated signal output may include a capacitive signal output.

[0004] The sensing device may be used with a deformable object. The sensing device may be integrated into the deformable object. The plurality of electrodes may be integrated into the deformable object and / or disposed on a surface or exterior of the deformable object. The plurality of electrodes may form part of a conformable layer or skin. The conformable layer or skin may be conformed to at least a portion of a surface of the deformable object. The at least one characteristic may include a dielectric constant and / or conductivity of the material.

[0005] The generated capacitive signal output may be processable to determine information associated with the deformable object, wherein the information includes at least one of: shape information; deformation information; force information; and / or velocity field information.

[0006] At least one of the plurality of electrodes may be deformable, stretchable and / or compressible.

[0007] The plurality of electrodes may include two or more proximal electrodes and two or more distal electrodes. The selected pairings may include at least one pairing between a proximal electrode and a distal electrode, and at least one pairing between two proximal electrodes and / or at least one pairing between two distal electrodes.

[0008] The plurality of selected pairs may include at least one pair between two electrodes disposed far from each other and at least one pair between two electrodes disposed close to each other.

[0009] At least one pairing between distal electrodes and at least one pairing between proximal electrodes may provide global shape and / or deformation information as well as local shape and / or deformation information.

[0010] The plurality of electrodes may be distributed in one or more layers, and at least one pairing may include a pairing between electrodes in the same or adjacent layers.

[0011] At least two of the plurality of selected pairs may include a common electrode.

[0012] The capacitance signal output may include a capacitance value for each selected pair of the plurality of pairs. The capacitance value may depend on at least one of the following: a spatial relationship between the pairs; a distance between the pairs; a surface area of ​​each electrode of the pairs; a relative orientation of the pairs; at least one material property of the material between the pairs, such as a dielectric constant.

[0013] Multiple electrodes may be distributed along one or more surfaces to form a three-dimensional distribution, wherein the three-dimensional configuration is continuously deformable from a planar configuration.

[0014] The plurality of electrodes may be distributed along and / or across one or more surfaces of the deformable object. The plurality of electrodes may be distributed laterally around one or more surfaces of the deformable object. The plurality of electrodes may define a sensing volume corresponding to the shape of at least a portion of the object.

[0015] The capacitive signal output may be processable to obtain deformation information associated with at least one of: bending, twisting, elongation, expansion, compression of the object.

[0016] The plurality of electrodes may be disposed laterally along the first surface of the deformable object and at least the second surface of the deformable object.

[0017] The electrodes may define a sensing volume having a shape corresponding to at least a portion of the shape of the deformable object.A plurality of electrodes may be distributed around an exterior of the deformable object.

[0018] The electrodes may be distributed to cover at least a portion of the exterior of the deformable object. The electrodes may cover at least 50%, optionally 75%, optionally 90% of the exterior. The electrodes may span substantially all of the exterior of the deformable object.

[0019] The plurality of selected pairings may include at least one pairing of a first electrode of the plurality of electrodes with a non-adjacent electrode.

[0020] The plurality of electrodes may be operable to generate a capacitive signal output from a plurality of selected pairings according to a predetermined sequence.

[0021] The plurality of selected pairings may include a subset, such as a degenerate subset, of all pairings of the plurality of electrodes.

[0022] The plurality of electrodes may include at least two electrodes arranged in a first plane and at least two electrodes arranged in a second plane, wherein the first plane is not substantially parallel to the second plane.

[0023] The plurality of electrodes may be disposed in or on one or more deformable substrates that are integrated into the surface by deformation.

[0024] The one or more deformable substrates may be stretchable.The one or more deformable substrates may be stretchable in at least a lateral direction.

[0025] The one or more deformable substrates may be stretchable to increase or decrease a distance between two or more electrodes of the plurality of electrodes.

[0026] The plurality of electrodes may form one or more sensor modules, wherein each sensor module may be continuously deformable from a planar configuration and / or integrated into a surface. In a planar configuration, the plurality of electrodes may be arranged in one or more layers and / or rows and / or columns. In a planar configuration, the plurality of electrodes may be arranged in a grid-like distribution.

[0027] The plurality of electrodes may comprise a stretchable conductive material. The sensing device may comprise a deformable connection between the plurality of electrodes. The deformable connection may comprise a stretchable conductive material.

[0028] The stretchable conductive material may include at least one of the following: a carbon black elastomeric conductive hydrogel and / or a liquid metal.

[0029] The plurality of electrodes may include an elongated conductive portion, wherein the elongated conductive portion is configured to further elongate in response to a force. The elongated conductive portions of the plurality of electrodes may be provided in a parallel arrangement.

[0030] A plurality of electrodes may be disposed on or integrated into one or more deformable substrates for application to a deformable object.

[0031] One or more electrodes may be integrated into the surface of the deformable object.

[0032] The deformable object may include at least one of: a robotic arm; other robotic manipulator; a part of the human body and / or a wearable object.

[0033] The plurality of electrodes may be distributed according to a predetermined layout. The predetermined layout may be determined using a machine learning derived process.

[0034] The sensing device may further include a processing resource configured to process the capacitive signal output or capacitive data obtained from the capacitive signal output to obtain said information associated with the deformable object.

[0035] The processing circuit system can be configured to apply at least one predetermined model to obtain the information. The at least one predetermined model can include a model trained using a machine learning derived process.

[0036] Obtaining shape information may include at least one of: obtaining reconstructed shape data; obtaining a graphical representation of at least a portion of the deformable object; determining one or more dimensions or other physical parameters of the deformable object; performing a shape reconstruction process. The reconstructed shape data may include full geometry high resolution 3D shape data.

[0037] Obtaining deformation information may include determining at least one of: a magnitude of a deformation applied to the deformable object; a type of deformation applied to the deformable object.

[0038] Obtaining force information may include determining at least one of a magnitude and / or a force applied to at least a portion of the deformable object. Obtaining force information may include determining forces on the deformable object from other objects. Obtaining force information may include determining touch information that depends at least on a change in a dielectric constant.

[0039] Obtaining velocity field information may include determining at least one of: a velocity field map of the deformable object.

[0040] The sensing device may be configured to perform at least one of: a shape reconstruction process and / or a deformation sensing process and / or a force sensing process and / or a deformation classification process and / or a velocity field map generation process.

[0041] The sensing device may include an electrode driving module configured to selectively drive one or more of the plurality of electrodes to generate a capacitive signal output from the selected plurality of pairs. The sensing device may include a capacitive signal readout value module configured to read the generated capacitive signal output and generate capacitive signal data.

[0042] One or more of the selected pairings may include two or more electrodes, optionally including three or more electrodes. A plurality of selected pairings may include pairings between a first group of one or more electrodes and a second group of one or more electrodes. The first group of electrodes may be operable to form a first combined electrode, and the second group of one or more electrodes may be operable to form a second combined electrode. The first and second combined electrodes may be operable to produce a capacitive signal output.

[0043] According to a second aspect, there is provided a sensing apparatus comprising the sensing device of the first aspect. The sensing apparatus may further comprise a display for displaying a visual representation of the shape of the deformable object.

[0044] The sensing device may also include at least one of the following: a processing resource configured to process the capacitance signal output or capacitance data obtained from the capacitance signal output to obtain the information associated with the deformable object; an electrode driving module configured to selectively drive one or more of the plurality of electrodes to generate a capacitance signal output from the selected plurality of pairs. The sensing device may include a capacitance signal readout value module configured to read the generated capacitance signal output and generate capacitance signal data.

[0045] The sensing device may include signal routing circuitry operable to connect each of the plurality of electrodes to at least one of a drive circuit and a signal readout circuitry. The signal routing circuitry may be controllable using one or more control signals.

[0046] According to a third aspect, which may be provided independently, there is provided a method comprising: obtaining signal output data representing signal outputs from a plurality of selected pairs of electrodes distributed around one or more surfaces of a deformable object, wherein the generated signal outputs of the electrode pairs depend on at least one of: a distance between the pairs; a shape and / or orientation of the electrodes and at least one material property of a material between the pairs; and processing the signal output data to determine information associated with the deformable object. The signal output may include a capacitive signal output, and the signal output data may include the capacitive signal output data.

[0047] The processing of the capacitance signal output data may include using at least one predetermined model. The at least one model may be predetermined using a process derived from machine learning. The at least one predetermined model may relate the capacitance signals of the plurality of electrodes to the information.

[0048] The processing of the capacitance signal output may be performed as part of at least one of the following: a shape reconstruction process, a deformation detection process. The processing of the capacitance signal output may be performed as part of a touch detection process. The processing of the capacitance signal output may be performed as part of a simultaneous touch detection and deformation detection process.

[0049] At least one model may be configured to output touch information and deformation information. At least one model may include at least two models, the at least two models including: a first model for outputting touch information and a second model for outputting deformation information. The touch information may include touch position information. The touch information may include contact position information.

[0050] According to a fourth aspect that may be provided independently, there is provided a method for training at least one model, comprising: obtaining training data, the training data comprising: signal output data representing a plurality of selected pairs of signal outputs from a plurality of electrodes distributed around one or more surfaces of a deformable object; and further data representing information of the plurality of electrodes; performing a model training process using the obtained training data to obtain at least one trained model to obtain further information of interest of the plurality of electrodes using the further obtained signal outputs. The signal output may comprise a capacitive signal output, and the signal output data may represent the capacitive signal output. Further signal outputs may comprise further capacitive signal outputs.

[0051] The obtained capacitive signal output data may be obtained for one or more spatial configurations of the plurality of electrodes and / or in response to one or more deformations applied to the deformable object. The obtained information may include shape and / or deformation information data corresponding to the one or more spatial configurations and / or one or more applied deformations.

[0052] Obtaining shape and / or deformation information data may include performing a sensing process on the deformable object while the deformable object is in each of the one or more spatial configurations and / or in response to the one or more deformations.

[0053] Obtaining shape and / or deformation information data may include obtaining image and / or depth sensor data of the deformable object while in multiple spatial configurations and / or in response to one or more deformations and processing the image and / or depth sensor data.

[0054] The image and / or depth sensor data may include 3D shape representation data, such as point cloud data, and / or data representing further derived parameters.

[0055] The obtained capacitance signal output data may be obtained in response to performing a series of contact actions at multiple locations on one or more surfaces. The obtained information may include contact location information.

[0056] According to a fifth aspect, which may be provided independently, there is provided an apparatus comprising a processing resource configured to: obtain signal output data representing signal outputs from a plurality of selected pairs of electrodes distributed around one or more surfaces of a deformable object, wherein the generated signal outputs of the electrode pairs depend on at least one of: a distance between the pairs; a shape and / or orientation of the electrodes and at least one material property of a material between the pairs; and process the signal output data to determine information associated with the deformable object. The signal output may comprise a capacitive signal output, and the signal output data may comprise the capacitive signal output data.

[0057] According to a sixth aspect that may be provided independently, there is provided an apparatus comprising a processing resource configured to: obtain training data, the training data comprising: signal output data representing a plurality of selected pairs of signal outputs from a plurality of electrodes distributed around one or more surfaces of a deformable object; and further data representing information of the plurality of electrodes; perform model training processing using the obtained training data to obtain at least one trained model to obtain further information of interest of the plurality of electrodes using the further obtained signal outputs. The signal output may comprise a capacitive signal output, and the signal output data may represent the capacitive signal output. Further signal outputs may comprise further capacitive signal outputs.

[0058] According to a seventh aspect, a non-transitory computer-readable medium is provided, comprising instructions operable by a processor to perform the method of the third aspect or the fourth aspect.

[0059] Features in one aspect may be provided as features in any other aspect as appropriate. For example, features of a device or apparatus may be provided as features of a method, and vice versa. Any feature in one aspect may be provided in combination with any suitable feature in any other aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Various aspects of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0061] Figure 1 is a schematic diagram of a sensor device according to an embodiment;

[0062] FIG2 shows the sensing device in unassembled and assembled configurations;

[0063] FIG3 illustrates the sensed capacitance between proximal and non-proximal pairs of electrodes;

[0064] FIG. 4( a ) is a top view of the sensor module, FIG. 4( b ) is a first cross-sectional view of the sensor module, and FIG. 4( c ) is an exploded cross-sectional view of the sensor module;

[0065] Figure 5 depicts the type of deformation applied to the sensing device;

[0066] Figure 6 is a flow chart generally illustrating a method of obtaining deformation information using a trained model;

[0067] Figure 7 is a flow chart generally illustrating a method of training a model for obtaining deformation information;

[0068] Figure 8 is a schematic diagram of a model architecture according to an embodiment;

[0069] 9(a) and 9(b) illustrate the output of a trained model according to an embodiment;

[0070] Fig.10 shows a plurality of electrode arrangements according to further embodiments;

[0071] Fig.11 depicts results from a neural network trained to classify a type of deformation applied to an object using a signal output, according to an embodiment;

[0072] Fig.12 depicts results from a neural network trained to determine the magnitude and direction of a force applied to an object using a signal output, according to an embodiment;

[0073] Fig.13 depicts a sensing device in unassembled and assembled configurations according to further embodiments;

[0074] Fig.14 depicts a sensing device according to a further embodiment;

[0075] Fig.15 (a) is a top view of a sensor module according to a further embodiment, Fig.15 (b) is a second top view of the sensor module, and Fig.15 (c) is an exploded cross-sectional view of the sensor module;

[0076] Fig.16 is the view of the manipulator;

[0077] Fig.17 Describes the use Fig.15 The results obtained by the sensor module;

[0078] Fig.18 Describes the use Fig.15 Further results obtained from the sensor module;

[0079] Fig.19 Describes the use Fig.15 The touch area of ​​the sensor module;

[0080] Fig. 20 is a schematic diagram of a model architecture according to an embodiment;

[0081] Fig.21 depicts a confusion matrix obtained during training of a neural network model; and

[0082] Fig. 22 Describes the use Fig.15 Further experimental results obtained with the sensor module. DETAILED DESCRIPTION

[0083] The following embodiments relate to a sensing device for use with a deformable object, which, as a non-limiting example, is a robotic arm or a robotic manipulator. The sensing device is configured to obtain a capacitive signal output that can be processed to obtain information about or associated with the deformable object. In the following, the embodiments describe obtaining, for example, shape and / or deformation information. However, it will be understood that other information (e.g., force information, velocity field information) can be derived from the capacitive signal output.

[0084] Figure 1 1 is a schematic diagram of a sensing device 10 according to an embodiment. The sensing device 10 has a sensing device 12, which includes a plurality of electrodes 14, which are referred to as electrodes 14 for simplicity. The sensing device 10 also has a driving module 16, a readout module 18, a processing resource 20, a memory resource 22, and a display 24. The electrodes 14 are configured to move along a deformable object ( Figure 1 The readout module may also be referred to as a readout electronic device and / or a readout circuit system. Similarly, the drive module may also be referred to as a drive electronic device and / or a drive circuit system.

[0085] A plurality of electrodes 14 are distributed around an outer surface of a deformable object, for example, a portion of a soft robot, such as a robotic arm or a robotic manipulator. The electrodes may be distributed along one or more surfaces of the deformable object, for example, laterally along one or more surfaces and / or across one or more surfaces. The plurality of electrodes may be considered to define a sensing volume that covers at least a portion of the shape of the object, and in some embodiments, substantially all of the object. While examples of a robotic arm and a robotic manipulator are described below, it will be understood that the apparatus may be used in a number of different applications and at a number of different scales.

[0086] The plurality of electrodes 14 have a distribution that allows each electrode to form proximal and non-proximal (also referred to as remote) pairings with other electrodes in the plurality of electrodes. The electrodes in such a distribution are therefore operable to obtain capacitance readings from proximal pairings and remote pairings. The proximal pairings and remote pairings are further described in detail with reference to FIG3.

[0087] Generally speaking, any pair of multiple electrodes can generate a capacitive output. For the capacitance formed between different electrode pairs, the capacitance is sensitive to the characteristics of the area between the electrodes.

[0088] The capacitance of the electrode pair can be approximately expressed mathematically by the following mathematical relationship:

[0089]

[0090] The capacitance value (C) between a pair of electrodes therefore depends on the properties of the material between the two electrodes (in this case the dielectric constant ε), the distance between the two electrodes (d), and the surface area S of the electrodes. The surface area of ​​the visible electrodes will change depending on the relative orientation of the electrodes. S in the above formula is a measure of the overlapping area of ​​the two electrodes. In addition, it will be understood that the above formula is an approximation.

[0091] In response to deformation or changes in, for example, environmental conditions, the shape of the electrodes may change (corresponding to the area S), the distance may also change, and the dielectric constant (or other relevant material properties, such as conductivity) may also change. Thus, the measured capacitance value will change in response to any such changes.

[0092] The sensing strategy described below includes collecting multiple capacitance values ​​from multiple selected pairs of electrodes. The specific number and selection of the collected capacitance values ​​can be referred to as a sensing strategy. The capacitance signal output can be processed to obtain information (e.g., deformation information or shape information) of the deformed object to be obtained. In some embodiments, the information obtained is deformation information collected across the 3D domain of the object to enable shape reconstruction processing of the object to be performed. In other embodiments, the information obtained is force or velocity field information. Since the capacitance signal output includes information related to the above-mentioned measurements (e.g., shape, distance, material properties), the capacitance signal output can be processed in principle to obtain quantities that can be derived from these measurements.

[0093] The sensing device 10 has an electronic connection between the sensor device 12 and the drive / readout value module, which is configured to transmit the drive signal from the drive module 16 to the electrode 14 and transmit the capacitance signal from the electrode 14 to the readout value module 18. According to an embodiment, each electrode has a corresponding electronic connection to allow addressing of each electrode. A portion of this electronic connection is provided as a conductive link in the sensing device itself, as described with reference to Figures 2 and 4.

[0094] The processing resource is configured to combine the capacitive signal outputs of the proximal pair and the non-proximal pair from the plurality of electrodes and process the output signals to obtain shape and / or deformation information. The processing resource can perform shape reconstruction or deformation detection processing using the capacitive signal output. In other embodiments, the processing resource can derive one or more further characteristics based on at least one of the shape, distance, and material properties.

[0095] In such a shape reconstruction or deformation detection process, the shape information obtained may be, for example, a graphical representation of a reconstructed 3D shape of at least a portion of a deformable object. The shape information may be the dimensions of the deformable object. The deformation information obtained may be, for example, the magnitude of a type of force or a type of deformation applied to the object.

[0096] The processing resource 20 may be any suitable processing circuit system. In some embodiments, the processing resource 20 is, for example, FPGA or ASIC hardware. The processing resource may be an edge-based processing resource, for example, NVIDIA Jetson Nano.

[0097] In use, the drive module 16 drives the plurality of electrodes according to the sensing strategy to obtain a capacitance signal from the selected electrode pairing. The sensing strategy may target a subset of electrodes. In some embodiments, a degenerate subset is targeted, where, for example, capacitance readings from repeated or degenerate pairings are not read. In response to the drive signal, the plurality of electrodes 14 then generate capacitance signal outputs between the selected pairings (including proximal pairings and distal pairings). The capacitance signal outputs are then processed by the processing resource 20 to obtain, for example, shape information or deformation information of a deformable object. As described below, the processing of the capacitance signal output may include transforming the capacitance signal output data into desired information (e.g., shape or deformation information) using a trained model.

[0098] FIG2 depicts a sensing device 112 according to an embodiment. It will be understood that the sensing device 112 corresponds to the sensing device 12 and can be used as a reference. Figure 1 The operation of a portion of the sensing device is described.

[0099] FIG. 2( a) depicts the sensing device 112 in a disassembled configuration. In this embodiment, the disassembled configuration is an unfolded configuration such that the electrodes 114 are in a planar grid. In this embodiment, the grid corresponds to an 8×4 array of electrodes. As depicted in FIG. 2( a), each electrode 114 has a corresponding conductive link 122. In this embodiment, the sensing device 112 is composed of 8 sensor modules, each of which is a four-electrode sensor module. For illustrative purposes, the sensor modules 124 are indicated in FIG. 2( a). A further description of a single sensor module 124 is provided with reference to FIG. 4.

[0100] As depicted in Fig. 2(a), each electrode is numbered. It will be appreciated that each electrode has an address (or array number). Since each electrode has a corresponding conductive link, the drive module can address one or more specific electrodes at a given time or according to a sequence or pattern, and the readout module can also derive a readout value from a specific electrode.

[0101] In this embodiment, the drive sequence includes proximal electrode pairs and non-proximal electrode pairs to ensure the diversity of readout values ​​and expand the sensing field. In some embodiments, if the two electrodes are too far apart, the capacitance value will be small. In such a case, the drive sequence can only use readings from electrode pairs in the same layer and across adjacent layers.

[0102] FIG. 2( b ) depicts the sensing device 112 in an assembled configuration, wherein a substrate of the sensing device is folded and integrated into a surface of a deformable object. In the present embodiment, the deformable object is a robotic arm. In the assembled configuration, a plurality of electrodes are laterally distributed along four surfaces of the deformable object. The sensing device 112 in the integrated configuration has first, second, third, and fourth sets of electrodes laterally distributed along first, second, third, and fourth surfaces, respectively. When assembled and integrated into the deformable object, the electrodes may be considered to form a sensing volume having a shape corresponding to the shape of the deformable object.

[0103] In this embodiment, the 32-electrode sensing device 122 is deployed on a mock-up robot arm and consists of eight 4-electrode modules. The capacitance value is read by the circuit (readout value module) at a speed of approximately 30fps (however, other sampling rates can also be used). Figure 6 and Figure 7 As described, a neural network based approach can be used to recover the 3D deformation from the capacitive signal readout.

[0104] Although FIG. 2 depicts a foldable sensing device, it will be understood that this is provided only as a non-limiting example, and that the plurality of electrodes may be distributed on one or more surfaces of an object having different shapes. As non-limiting examples, the shapes may include a glove, a snake, or other irregular shapes.

[0105] The embodiment of Figure 2 illustrates that in some embodiments, the sensing device can be modular and consist of one or more deformable sensor modules that can be fixed to the surface of an object. The sensor module can be regarded as forming a part or patch of the stretchable skin of the object. However, it will also be understood that in further embodiments, the electrodes themselves are integrated into a part of the object or deployed on the outer layer of the object. In further embodiments, some electrodes can be integrated into the object itself, while other electrodes can be provided as modules or separate layers applied and fixed to the object.

[0106] FIG. 2 can be viewed as depicting 8 layers of electrodes across different modules (e.g., electrodes 1, 9, 17, and 25 are viewed as being in the same layer). Layers herein may also be referred to as rows. In some embodiments, the device is configured to obtain capacitance values ​​from electrodes in the same layer (e.g., between electrodes 12 and 20) or from electrodes in adjacent layers (e.g., electrodes 4 and 5).

[0107] Figure 2(c) depicts the layers of the manipulator in more detail. The left hand side depicts a 16-electrode capacitive sensor. Figure 2(c) depicts four layers: Layer 1, Layer 2, Layer 3, and Layer 4. Each layer has four electrodes, which together provide up to 6 capacitance readouts. The right hand side provides a grid representation of the manipulator. In this embodiment, the top of the manipulator (above layer 4) is fixed. The rest of the manipulator is free to move in response to applied (deformation-causing) forces.

[0108] As described above, multiple electrodes are operable to generate a capacitive signal between selected electrode pairs. FIG. 3 illustrates a deformable object 302, in this example a soft robotic arm. In the illustration of FIG. 3, four electrodes are depicted: a first electrode 312a, a second electrode 312b, a third electrode 312c, and a fourth electrode 312d. Object 302 extends outward from a surface and has a proximal portion and a distal portion. The first electrode and the second electrode (312a, 312b) are disposed at the surface of the proximal portion and may be referred to as proximal electrodes. The third electrode and the fourth electrode (312c, 312d) are disposed at the surface of the distal portion and may be referred to as distal electrodes.

[0109] In FIG. 3( a), two pairings are indicated: a first pairing 304a (between the first electrode 312a and the second electrode 312b) and a second pairing 304b (between the third electrode 312c and the fourth electrode 312d). The first pairing 304a can be regarded as a pairing between two proximal electrodes, and the second pairing 304b can be regarded as a pairing between two distal electrodes. In FIG. 3( b), two further pairings between electrodes are indicated: a third pairing 304c (between the first electrode 312a and the fourth electrode 312d) and a fourth pairing 304d (between the second electrode 312b and the third electrode 312c). Both the third pairing 304c and the fourth pairing 304d can be regarded as a pairing between a proximal electrode and a distal electrode.

[0110] Whether an electrode pair is considered proximal or distal can depend on, for example, the distance between the two electrodes. In some embodiments, if there are other electrodes between the two electrodes, then the capacitance between them may not be measured. However, if the electrodes are located in adjacent layers, for example, they are located in the same or adjacent layers, for example, as electrodes 1-2 in Figure 2, then a capacitance value can be returned. In some embodiments, the proximal pairing can be considered as two adjacent electrodes. The distal pairing can be considered as two non-adjacent electrodes, for example, there is an electrode between them. In some embodiments, only the capacitance formed by the adjacent electrodes is measured.

[0111] The above embodiments provide examples of subsets of all possible pairings that can be used. It will be appreciated that in some embodiments, a capacitance output from a subset of all possible pairings is used. Such an output may include, for example, a degenerate subset pairing that includes only degenerate or non-repeating pairings.

[0112] As shown in Fig. 3 (a) and Fig. 3 (b), at least two pairs have a common electrode. For example, the first pair 304a and the third pair 304c have a common electrode (first electrode 312a). Similarly, for example, the first pair 304a and the fourth pair 304d have a common electrode (second electrode 312b).

[0113] It will be understood that the terms "proximal" and "distal" as used herein refer to the placement of the proximal and distal portions of the robotic arm extending outward from the surface. However, for this object and for objects of other shapes, the electrode pairings selected refer to their proximity or closeness to each other. Thus, the first pairing 304a can be referred to as a proximal pairing, and similarly, the second pairing 304b can be referred to as a proximal pairing. The third pairing 304c can be referred to as a non-proximal pairing, and similarly, the fourth pairing 304d can be referred to as a non-proximal pairing. Non-proximal pairings can also be referred to as remote pairings.

[0114] The proximal pairing of an electrode may include any electrode that falls within a predetermined area (e.g., area or volume) around the electrode. The predetermined area may define a neighborhood so that any electrode within the neighborhood is referred to as a neighboring electrode, and any electrode outside the neighborhood is referred to as a non-neighboring electrode. In this description, the proximal pairing may correspond to a group of neighboring electrodes. The proximal pairing of an electrode may include, but is not limited to, the nearest neighbor of an electrode.

[0115] As described above, the magnitude of the capacitance measured between a pair of electrodes depends at least on the distance between the two electrodes. Therefore, in operation, a stronger capacitance signal is sensed for the first pairing 304a and the second pairing 304b due to their proximity to each other than the weaker capacitance signal sensed for the third pairing 304c and the fourth pairing 304d. In general, a near-side pairing will measure a larger capacitance value than a remote pairing.

[0116] For the purpose of shape reconstruction and deformation sensing, it has been found that if only strong signals between adjacent electrodes are used, a reliable shape may not be reconstructed. Similarly, if only weak signals between remote pairs are used, a reliable shape cannot be reconstructed. As depicted in Figure 3(c), by combining the capacitive signal outputs from proximal and remote electrode pairs, the model can reconstruct accurate geometry and shape. Figure 3(c) illustrates a plurality of points 306, which together provide a reconstructed point cloud for the robotic soft arm 302. Therefore, it has been found that by receiving capacitive signal outputs from proximal and non-proximal electrode pairs and processing the outputs, local and global shape and / or deformation information can be obtained. The point cloud used here is an example representation of the shape information of an object. It will be understood that the point cloud is independent of the number of electrodes.

[0117] The distribution of electrodes can define a sensing volume that covers a portion of the volume of the object. Similarly, the distribution of electrodes can define a sensing area that spans a portion of the external surface area of ​​the object. For example, the portion can be a portion of the exterior of the deformable object. As a non-limiting example, the portion can be at least 50%, optionally 75%, optionally 90% of the exterior of the deformable object. In some embodiments, the electrodes can span substantially all of the exterior of the deformable object. The distribution of electrodes can be regularly spaced or unevenly spaced (so that the density of the electrodes varies on the surface). The density of the distributed electrodes will depend on the object being measured.

[0118] 4(a) is a top view of a four-electrode sensor module 124 used to form a sensing device 112. The four-electrode sensor module has a first electrode 114a, a second electrode 114b, a third electrode 114c, and a fourth electrode 114d. The sensor module is manufactured to be deformable under the action of a force, for example, stretchable, twistable, and / or compressible.

[0119] Each electrode has a corresponding conductive link for linking the electrode to other electronic devices, such as a readout module and a drive module. The conductive link for the electrode can be referred to as an electrode link or simply a link. The link provides a conductive path between the electrode at the first part of the sensor module and the connector at the other parts of the sensor module. FIG. 4 (a) depicts a first electrode link 122a, a second electrode link 122b, a third electrode link 122c, and a fourth electrode link 122d.

[0120] The electrode link is configured to transmit the drive signal from the drive module to the electrodes (to activate the selected electrodes). The electrode link is also configured to transmit the capacitive readout signal from the electrodes (to the readout module).

[0121] In this embodiment, the electrode links of the sensor module are embedded in the hierarchical structure of the sensor module. Each electrode link has a first and a second connector (also referred to as a terminal) and a connecting portion between the first and the second connector. For clarity, FIG. 4( a) depicts a first connector 123, a second connector 126, and a connecting portion 125 for a first link 122a.

[0122] The links are arranged in the module according to the link pattern so that the first connectors of the four links are aligned at the first end of the module and the second connector of each electrode link terminates at the corresponding electrode. The link pattern is such that the connected parts of each link do not overlap or cross. It will be understood that when forming part of the sensing device, other connections (e.g., wiring or cables or wireless capabilities) connect the electrode links to the readout value and the drive module, in particular the connector.

[0123] FIG4( b) depicts a cross-sectional view of the sensor module 124 at the electrode region. As depicted in FIG4( b), the sensor module has a layered structure. At the electrode region, the sensor module has a sealing layer 142, an isolation layer 144, an electrode layer 146, and a protective substrate layer 148. Each layer is disposed on a base substrate 140. In this embodiment, the base substrate is made of silicone.

[0124] Channels for electrode links are provided in the isolation layer 144. In this embodiment, the channels are microchannels and are formed in the isolation layer 144 using a laser engraving process. The channels are engraved on the isolation layer by a laser machine.

[0125] FIG4( c) depicts an alternative view of the layered structure of a portion of a sensor module. FIG4( c) depicts an electrode link 122 embedded in an isolation layer 144. FIG4( c) also depicts an electrode 114 formed in the electrode layer. FIG4( c) also shows a protective substrate layer and a sealing layer. FIG4( c) also depicts holes 150 formed in the isolation layer 144. The holes 150 are vertical interconnect holes. Each hole provides an opening for connecting an electrode layer to an electrode link.

[0126] The first and second connectors and electrodes may be formed of conductive and deformable materials (e.g., stretchable conductive materials). In this embodiment, the electrodes are carbon black (CB) dispersed elastomers. It has also been found that such materials may not be well suited for use as connectors and connecting portions of electrode links due to their high resistance and nonlinear, irreversible conductivity response under deformation. In this embodiment, eutectic gallium 75.5% indium 24.5% (EGaIn) is used due to its conductive properties (3.4×10 7 S m-1) and a stable response to deformation for electrode links (wires and connectors).

[0127] In more detail, the 4-electrode sensing module has dimensions of (20×20×120 mm) and consists of 4 different functional layers, namely, a protective substrate layer 148 (thickness of approximately 0.39 mm), an electrode layer 146 (thickness of approximately 0.08 mm), an isolation layer 144 (thickness of approximately 0.24 mm) and a sealing layer (thickness of 0.3 mm). Engraving is performed by a laser machine, microchannels for wires (width 0.5 mm) are engraved and connections (3×2 mm) are formed on the isolation layer, and then the sealing layer is bonded to the outer surface of the isolation layer. EGaIn ink is injected into the channel with a small syringe. The connection between the CB electrode and the EGain wire is ensured by vertical interconnect holes. Although the thicknesses are provided above, it will be understood that these thicknesses can vary.

[0128] The selection of the above materials and design parameters provides a relative capacitance response range of 16% to 19% for 40% strain, depending on the electrode pair activated. The response curves show excellent linearity and consistency over multiple cycles (over 500 cycles). The EGaIn wires are shown to provide superior performance to the CB wire in terms of sensitivity (more significant response under the same deformation), linearity (no distortion in the response curve), and cyclic stability (no shift after 500 stretching cycles).

[0129] Although FIG. 2 and FIG. 4 illustrate a sensing device made of a sensing module according to an embodiment, it will be understood that the design parameters of the sensing device and / or the module may vary. The design depicted in FIG. 2 balances reconfiguration performance with manufacturing complexity. The sensing module of FIG. 4 may be manufactured using known elastomer processing techniques to provide patterning accuracy, repeatability, and scalability. Using these manufacturing techniques, the sensing modules may be manufactured in parallel.

[0130] Figure 5 A representative set of deformations that can be sensed using the sensing device are depicted. These deformations represent a non-limiting set of deformation states of the sensing device 112. Figure 5 FIG. 10( a ) shows the sensing device 112 in an undeformed state. The undeformed configuration may be referred to as the natural state of the sensing device 112 . Figure 5 (b) shows the sensing device in a bent state under the influence of bending deformation. Figure 5 (c) shows that the sensing device is undergoing bending and twisting deformation and is in a bent and twisted state. Figure 5 (d) depicts the sensing device undergoing an elongated deformation (stretching) and is depicted as being in an elongated or stretched state. Figure 5 (e) depicts the sensing device undergoing elongation and distortion and being in an elongated and distorted state. It will be understood that these figures are non-limiting examples of deformed states, and that under the influence of force, the sensing device can be placed in a plurality of different deformed states. For example, Figure 5(f) and Figure 5 (g) depicts further bending and / or bending and twisting states for sensing the device. Figure 5 As shown, it is apparent that the sensing device can experience and sense multi-modal deformations. The sensed deformations may include one or more of the following: bending, twisting, elongation, expansion, compression, stretching, shearing. It should be understood that the type of deformation of the sensing device is inferred.

[0131] Figure 6 6 is a flow chart generally describing a method of obtaining deformation information using a capacitive signal output from a sensing device. The method 600 uses a trained model. The trained model may also be referred to as a capacitance to deformation transformer (C2DT). Figure 7 Describing the Training of Models It will be appreciated that in some embodiments, more than one trained model may be used.

[0132] At step 602, capacitance signal output data is obtained. As described above, according to an embodiment, capacitance signal output data is obtained using a sensing device. At step 604, the trained model is applied to the capacitance signal output data. In this embodiment, the capacitance signal output data is provided as input to the trained model. At step 606, deformation information is obtained as an output of the trained model. Figure 8 Provides further details on the specific neural network implementation of the model.

[0133] Figure 7 is a flow chart generally describing a method 700 for training a model used in, for example, method 600 .

[0134] At step 702, capacitance signal output training data is obtained. The capacitance signal output training represents the capacitance signal output from the sensing device, as described in the embodiments. The capacitance signal output data obtained is obtained for one or more spatial configurations of multiple electrodes and / or in response to one or more deformations applied to the deformable object. At step 704, deformation information training data is obtained. The deformation information training data corresponds to the obtained capacitance signal output data, and the two data sets together provide a training data set. Although steps 702 and 704 are described as two steps of method 700, they can also be viewed as a single step of obtaining training data.

[0135] Although a variety of different methods can be used to obtain training data, in this embodiment, the training data is obtained by applying a test deformation to the sensing device and measuring the corresponding capacitance signal output data of the deformation, as shown in FIG. Figure 1Described. When the sensing device is in a test deformation, one or more depth sensing cameras located around the sensing device are used to obtain depth data of the sensing device. In this embodiment, the depth data includes or can be processed to obtain 3D point cloud data. The depth data is processed to obtain deformation information. In this embodiment, multiple depth cameras (providing 3D point cloud data) are used to capture the deformation. The depth data is processed and cleaned for use as ground truth in training.

[0136] The above process is then repeated for a plurality of test deformations to form a training data set including capacitance signal output training data and corresponding deformation information training data.

[0137] At step 706, a model training process is performed to train the model using the training data according to one or more model training algorithms. Although a variety of different training algorithms can be used to obtain a sufficiently trained model for use in, for example, method 500, as a non-limiting example, the training process includes providing the capacitance signal output training data as input to the model and performing a comparison of the output of the model with the deformation information training data. The model is trained by iteratively providing the capacitance signal output training data to the model and comparing the output with the deformation information training data, refining the values ​​of the model weights.

[0138] At step 708, the trained model is stored. The trained model is stored for use, for example, during method 700. Although training may be performed on other processing resources of the sensing device, the trained model, or at least the trained weights, may be stored on the memory resource 22 for use by the sensing apparatus.

[0139] Although Figure 6 and Figure 7 This paper briefly describes the methods of training models and using them to obtain deformation information, but Figure 8 The model architecture of the neural network model according to the embodiment is described in more detail.

[0140] The model takes as a first input the capacitance readout data 704 and as a second input the source point cloud 706. The input source point cloud provided as input corresponds to the source point cloud for the undeformed sensing device. Thus, the input source point cloud includes spatial information corresponding to, for example, the shape and volume of the sensing device. The model is trained to output a reconstructed target point cloud based on the input. In summary, the model displacement (without any deformation) of each point in the source point cloud comes from the capacitance readout data.

[0141] The transformer can be viewed as having three modules (also called layers: encoding module 710, decoding module 712, and loss counting module 716).

[0142] The encoder module receives the capacitance signal readout data 704 as input. Features from the capacitance data are extracted based on the self-attention mechanism. The features are then fed into the decoder part to "deform" the source point cloud. In the encoding part, the neural network encodes the input capacitance readout and the geometric structure information of the electrode pair into a high-dimensional space and feeds them to the transformer encoder to refine the proprioceptive information.

[0143] The decoder module receives as input the source point cloud data 702 and the output of the encoder. In use, the trained decoder module outputs reconstructed target point cloud data 708. In the decoding part, the network tries to assign the correct displacement to each point in the source point cloud based on the output sequence of the encoding part.

[0144] The loss count module is used during the training of the model. As part of the training process, the ground truth of the target point cloud (also referred to as training data) is provided to the loss count module. The loss count module 714 is used to train the model. The loss count module 714 includes a loss function that is minimized during the training process. The loss function consists of a squared distance term for the visual marker and a chamfer distance term for the remaining points.

[0145] It will be understood that Figure 8 The neural network is a structure optimized for this specific problem, however, other suitable network structures and models can also be used.

[0146] In addition, it will be understood that for different applications (e.g., determining different types of other information from the capacitive signal), different trained models will be used. Therefore, for a particular application, the processing resource 20 can be configured to retrieve appropriate model information (e.g., model weights and / or other model data) from the memory resource 22. For example, for shape reconstruction purposes, a trained shape reconstruction model is obtained and used. As another example, to determine the type of deformation, a trained model that converts the capacitive signal output into a label corresponding to the type of deformation is obtained.

[0147] Figure 9(a) depicts example output graphical representations of the shape reconstruction process. These representations were generated from the sensed capacitance values ​​using different trained models. The region of interest is the middle portion in the source point cloud. The reconstruction results of the trained models represent good, high-quality reconstructions and capture the range of complex deformations tested (as measured by multiple performance metrics: average distance (AD), maximum distance (MD), chamfer distance, Hausdorff distance). As part of the analysis, the impact of the visual labeling term in the loss function is analyzed.

[0148] Figure 9(a) depicts simulated data and Figure 9(b) depicts examples of real-world data. Figure 9(b) depicts different frames of image data and the corresponding ground truth and trained model (C2DT) outputs for three different types of deformations.

[0149] The sensing devices and methods described above have advantages over known methods. For example, high-definition shape reconstruction and / or accurate dimensional determination can be performed.

[0150] In other embodiments, a dense array of electrodes may be used to meet the requirements of real-world applications. Increased electrode density may result in increased burdens for wiring, data collection, and computation. In some scenarios, such as large-scale tactile sensing, a sparse array of electrodes may be more preferred. In other embodiments, a trained machine learning algorithm is used to determine the electrode layout. Fig.10 Five different electrode layouts deployed on the surface of a soft rectangular body are depicted. The first three layouts are human-designed (HD). The last two layouts are randomly generated (RD) and designed by a machine learning algorithm using sensor layout optimization (SLO).

[0151] In the above embodiments, shape reconstruction and deformation sensing are described, where the output of the trained neural network is, for example, a 3D point cloud or other spatial representation of an object. Fig.11 and Fig.12 Results related to other applications of sensing devices are depicted, where a neural network is trained to receive capacitive signal outputs from multiple electrodes as input and provide different types of outputs.

[0152] Fig.11 Results related to the deformation classifier are depicted. Fig.11 Image data for two types of deformations are depicted: a first image 1102a for bending forces applied to an object, and a second image 1102b for bending and torsional forces applied to the object. Capacitive signal readouts are depicted for two types of deformations: a first capacitance signal readout 1104a for bending forces, and a second capacitance signal readout 1104b for bending and torsional forces. The x-axis represents the index of the electrode layer (e.g., see the layer structure of Figure 2(c)). In this embodiment, there are four layers and each layer has 4 electrodes to obtain 6 capacitance readouts. The y-axis represents the measurement of the calibrated capacitance. Regarding parameter estimation, in this embodiment, it is not necessary to deploy a dense array of electrodes.

[0153] Fig.11The network output is also depicted. In this embodiment, the network output is the probability score that the applied force includes a distortion. For the first deformation (no distortion), the probability is 0.005, and for the second deformation (including distortion), the probability is 0.993. In some embodiments, a threshold on the output probability can be used to convert the probability to a binary value, or the network can be trained to output a binary value.

[0154] Fig.12 Results related to the force estimation network are depicted. Fig.12 A first image 1202a for a first curvature applied to the object and a second image 1202b for a second curvature applied to the object are depicted. Fig.12 The corresponding output from the trained neural network is also depicted. The trained neural network used for this application is trained to receive as input the capacitive signal output from multiple electrodes and output the magnitude of the force applied to the object. As can be observed from the results 1204a and 1204b, the estimated force is closely related to the true value measured force.

[0155] Below, the experimental methods using the above-mentioned embodiments are described. The above-mentioned embodiments relate to the technology of giving highly compliant systems 3D proprioception through a new type of intrinsic stretchable electronic skin and advanced machine learning algorithms. The electronic skin with a uniquely designed planar stretchable electrode array and sensing scheme is able to capture boundary deformations on soft bodies. By making full use of electronic skin signals and a custom-designed deep neural architecture based on a self-attention mechanism, it has been demonstrated that the proprioceptive system can uniquely reconstruct complete 3D geometry under complex multimodal deformations in dense point clouds, with an accuracy (millimeter-level error) comparable to that of external commercial RGB-D cameras. This represents a step change from existing proprioceptive systems that only provide sparse geometric reasoning in constrained deformation modes. Proprioception technology can enable soft robots to have the ability to accurately perceive their motion state like natural organisms, paving the way for their application in important real-world scenarios from biomedicine to human-computer interaction.

[0156] A class of intrinsically stretchable capacitive electronic skins (SCAS) embedded with planar electrode arrays is described to capture information, such as 3D proprioceptive information. SCAS is combined with a custom-designed deep network to reconstruct dense point clouds under complex multimodal deformations, which can be considered one of the most challenging proprioceptive problems. The capacitance formed by the non-redundant combination of planar stretchable electrodes distributed on the 3D surface can characterize the boundary deformation and spatial electrical properties in the 3D domain of interest. Compared with parallel electrode arrays, the electrode array can have a simpler structure and can be easier to manufacture, miniaturize and modularize. A model robotic arm actuated by an external force (a square prism silicone structure similar to a typical soft robotic manipulator) was chosen as a test bed. The motivation for this particular choice is the computational simplicity in simulation and the need to test the widest range of possible deformations, which would be unattainable in an intrinsically actuated system. However, the proposed method is in principle independent of the shape of the soft body being studied, as it does not require any prior geometric knowledge, which makes it generalizable to soft robotic platforms with various geometric configurations. The approach is first investigated via coupled electrostatic and solid mechanics simulations and then transferred to a physical platform via appropriate electrode layout and network architecture modifications based on the conclusions drawn from the simulation results.

[0157] In the coupled simulation, the SCAS consisting of 64 planar electrodes was deployed on a model robot arm to characterize various deformations (elongation, twisting, bending and their combinations) caused by external forces via 392 measurable independent capacitance readouts at each measurement frame. A simulated dataset consisting of 39,334 frames of different deformations (in point cloud format) and corresponding capacitance readouts was generated. This dataset was used to evaluate the performance of the SCAS and 3D deformation reconstruction methods. The results from the simulation phase were used to guide the design of sensor and network architectures in physical systems.

[0158] Ablation studies were implemented to better understand the role of each loss term and position encoding. It was observed that C2DT was unable to learn correct point-to-point correspondences without including visual markers in the training process. When C2DT performs reconstruction without markers, points in the region of interest in the source point cloud may not be mapped to the correct corresponding regions. By minimizing the chamfer distance term, the reconstruction can have similarity to the true value while the point-to-point error remains large. It was also found that by retaining only the squared distance term of the visual markers during training, local distortions appeared in the reconstructed set of frames. This indicates that the chamfer distance term can be beneficial to the geometric quality of the reconstruction. Finally, poor convergence was observed when attempting to train the network after removing the position encoding portion. It was observed that electrode pairs with high geometric correlation tend to cluster together after position encoding.

[0159] The high density of markers and electrodes of SCAS adopted in simulation environments poses practical challenges when applying them to the fabrication and experiments of physical systems. Therefore, the impact of the number of markers and the layout of electrodes on the performance of C2DT was investigated, with the ultimate goal of guiding the design and deployment of functional real-world SCAS systems. The results of this analysis show that the improvement in accuracy brought by increasing the number of markers tends to level off. This provides evidence that a small set of visual markers can be sufficient for C2DT to establish correct point-to-point correspondences. Similarly, reconstruction performance can improve with the density of electrodes, but the improvement is very limited after the number of electrodes exceeds a certain value (e.g., above 32). Therefore, there seems to be a positive trade-off between reconstruction accuracy and electrode / marker unit, confirming that it is safe to sacrifice a minor performance degradation to greatly simplify the fabrication and deployment of SCAS.

[0160] Eight 4-electrode SCAS modules are deployed on the surface of a model robotic arm with dimensions of 20×20×240mm, which is one-fifth of the size studied in the simulation (the additional 40mm height is the interface area). The 32-electrode SCAS consisting of 8 SCAS modules is connected to an Electrical Capacitance Tomography (ECT) system to extract individual capacitance values. Two RGB-D cameras are used and placed directly opposite to each other to capture the real-time, true-value 3D deformation of the measured object in a colored point cloud format from two complementary views and fuse them in one coordinate system. The sides of the model robotic arm are dyed white because its original transparency would negatively affect the quality of the data collected by the RGB-D camera. 16 yellow visual markers are placed to encourage the network to learn the correct point-to-point correspondence during training. The entire experimental platform can synchronously record capacitance and point cloud data at a speed of approximately 30fps.

[0161] The reliability of the SCAS allowed recording of capacitance readout frames (each frame comprising 76 separate readouts) while the model robot arm was deformed over a long period of time by manual manipulation of the bottom holder. Approximately 1220 seconds of deformation data were collected during the 10 hour experiment.

[0162] A series of random complex deformations were implemented during the experiment, including bending, stretching, twisting and their combinations. In most frames, the point cloud collected by the camera may not represent the complete 3D deformation because of missing points caused by inevitable visual occlusion. For frames with slight missing point problems, these points can be filled by shape reconstruction, and frames facing severe occlusion are directly filtered, after which a total of 31,380 frames of data are obtained.

[0163] Other challenges in experimental deformable reconstruction may come from the quality of the point cloud (limited by camera accuracy, occlusions, lighting conditions), noise in the SCAS signal, and imperfect synchronization between different devices. The C2DT framework compensates for these additional sources of inaccuracy in the experimental setting by increasing the number of input frames (Ni adjacent frames of SCAS readouts) and by introducing a regularization term in its loss function that limits the amount of change in the distance between neighboring points before and after deformation. Several C2DTs were trained with different numbers of input frames using a filtered real-world dataset. It was found that the reconstruction performance improves with the number of input frames and achieves its minimum error with 3 adjacent input frames. This improvement may indicate that increasing the number of input frames can reduce the negative impact of noise in the SCAS signal and asynchrony between different devices. The temporal correlation between adjacent frames can be considered to be beneficial for deformable reconstruction. The results achieve a quality level comparable to the ground-truth point cloud collected by an external RGB-D camera.

[0164] According to the results of ablation studies, visual markers play a similar role in both real and virtual environments, encouraging the network to learn correct point-to-point correspondences. It is also observed that adding a neighborhood regularization term slightly improves the reconstruction quality. The position encoding part is crucial for extracting useful proprioceptive information from physical SCAS signals. Similar to its contribution in simulation, the position encoding can assign discriminative high-dimensional representations to different electrode pairs based on their geometric structures.

[0165] The described proprioceptive system can capture a variety of complex deformed 3D geometries in real time (30fps) with a quality (millimeter error) comparable to that of commercial RGB-D cameras. This can demonstrate significant advantages over many previous attempts that mainly involve primary and simple proprioceptive scenarios. The system can also be independent of the geometry of the object being measured and can therefore be extended to many other types of soft robots through simple learning processing with the help of RGB-D cameras. The performance demonstrated by this technology offers great promise for solving some of the most complex challenges in the control of soft robots, thereby promoting their adoption in fields such as biomedicine and human-robot interaction. In addition, dynamic coupled field simulation methods that simultaneously combine sensors and soft robot deformations can become a powerful tool to promote automatic sensor design and optimization, for example in fields such as digital twins of soft robots. The improvements of the SCAS system presented in this article can also allow integration with other sensor units. Such integrated methods can allow multimodal sensing of both proprioception and external stimuli, further aligning the performance of artificial systems with that of biological bodies.

[0166] The simulation and experimental methods are further described below based on the embodiments, but are not limited thereto.

[0167] As part of the experimental study, coupled simulations were implemented in COMSOL Multiphysics to generate rich capacitance and deformation data to demonstrate the effectiveness of the proposed method. The subject of the study is a square prism model robot made of silicone (length: 271mm, width: 100mm, height: 1000mm). An electrode array with 64 electrodes (8×8) is placed on the surface of the model robot to form a 64-electrode SCAS. Each electrode is a 105×30mm flat surface with no thickness. The horizontal and vertical distances between two adjacent electrodes on the same side are both 20mm. The distance between each edge and the nearest electrode is 10mm. The relevant material properties are set as follows: Young's modulus E=4.15MPa, Poisson's ratio v=275 0.022, density p=1.28×103kg m-3, relative dielectric constant Sr=3.276.

[0168] In the simulation, 956 different episodes were implemented. Each episode simulated a time-continuous deformation process and was discretized into approximately 40 frames. In each frame, the deformation and the corresponding capacitance readout from the SCAS were recorded.

[0169] Four different types of loads were applied to generate various complex deformations: 1. Combination of elongation and twisting L(z,r), where a torsional force and a tensile force along the z-axis were simultaneously applied to the tip of the model robot arm. 2. Pure bending L(x,y), where a tensile force in the xy plane was applied to the tip of the arm. 3. Two-phase torsion and bending L e (x, y), where a torsional force is applied to the tip of the arm in the first r frames (r ranges from 6 to 16), and then a tensile force in the xy plane is applied to the tip while maintaining the torsional state. 4. A combination of torsion and bending L(x, y, r), where a torsional force and a tensile force in the xy plane are simultaneously applied to the tip of the arm. Each deformation is represented by a 3D point cloud with 1716 points. Because it is impractical to determine the exact point-to-point correspondence of all points in all deformations under real-world conditions, a scene that can be realistically implemented in a physical experiment is used. 64 points were selected as visual markers whose correspondences were available during network training, and the correspondences of the remaining points were only used for testing for evaluation.

[0170] In theory, any two electrodes can form a capacitor. Fig.13 A sensing device with 64 electrodes is depicted in an assembled configuration (right hand side) and a disassembled configuration (left hand side). Fig.13As shown in Figure 2, 2016 independent capacitance readouts can be generated in each measurement frame. However, many of them output very weak signals due to the long distance (e.g., the capacitance between electrode 1 and electrode 64). In a physical platform, such signals will be difficult to detect, so they can be completely ignored. Therefore, only the capacitance of electrode pairs in the same layer and the capacitance of some electrode pairs between two adjacent layers are recorded.

[0171] In particular, in this example, the 28 electrodes in the first layer are paired to form independent capacitors that can be measured. A subset of the pairings includes the following independent (non-repeating) pairings: electrode 1 is paired with each electrode in the layer (9, 17, 25, 33, 41, 49, and 57) to give 7 pairings; electrode 9 forms an additional pairing with all electrodes in the layer except electrode 1 (17, 25, 33, 41, 49, 57). It will be understood that in this scheme, electrode 25 (with electrodes 33, 41, 49, and 57) forms 4 additional pairings; electrode 33 (with 41, 49, 57) forms 3 additional pairings; electrode 41 (with electrodes 49 and 57) forms 2 additional pairings, and electrode 49 (with electrode 57) forms 1 additional pairing.

[0172] In particular, in this example, 24 electrode pairs between the first and second layers form measurable independent capacitors. A subset of the pairs includes the following independent (non-repeating) pairs: electrode 1 forms a pair with each electrode in the layer (9, 17, 25, 33, 41, 49, and 57) to give 7 pairs; electrode 9 forms an additional pair with all electrodes in the layer except electrode 1 (17, 25, 33, 41, 49, 57). It will be understood that in this scheme, electrode 25 (with electrodes 33, 41, 49, and 57) forms 4 additional pairs; electrode 33 (with 41, 49, 57) forms 3 additional pairs; electrode 41 (with electrodes 49 and 57) forms 2 additional pairs, and electrode 49 (with electrode 57) forms 1 additional pair.

[0173] The 24 electrode pairs between the first layer and the second layer form independent capacitors that can be measured. In general, SCAS can generate 392 independent capacitance readouts per measurement frame. For the first layer, each electrode forms 3 additional pairs with adjacent electrodes in the second layer. For example, electrode 1 in a layer forms a pair with adjacent electrodes 2, 10, and 58, and electrode 9 forms an additional pair with adjacent electrodes 2, 10, and 18. It will be understood that each electrode forms 3 additional pairs.

[0174] In total, SCAS can generate 392 independent capacitance readings in each measurement frame. Each reading is calibrated as follows: c = (c′-c emp ) / c emp, where c is the calibrated capacitance reading, c' is the raw reading, and c emp A total of 39334 frames (956 rounds) of deformation and capacitance readout values ​​were generated by the coupled simulation, of which 2319 frames (53 rounds) had deformation type 1, 12552 frames (300 rounds) had deformation type 2, 12269 frames (303 rounds) had deformation type 3, and 12194 frames (300 rounds) had deformation type 4.

[0175] Recovering deformations from SCAS sensing data is actually a sequence-to-sequence problem, mapping a sequence of capacitance readout values ​​to a corresponding sequence of point coordinates (point cloud). As described above, a capacitance-to-deformation transformer (C2DT) with a self-attention mechanism can be used to achieve dense 3D deformation reconstruction. In the following additional comments on the architecture, the details of the implementation of C2DT and the evaluation metrics are described according to the embodiment.

[0176] In general, C2DT is a method that can measure feature tensors The source point cloud P s Deform to target point cloud Here, is a point cloud without any deformation, N p YesP s The number of points in , and the value in this case is 1719; is a reconstruction of a point cloud with a specific deformation; is the corresponding calibration capacitance readout value; N m is the number of readouts in c, which in this case is 392; and and are the coordinates of the electrode that generated c.

[0177] The C2DT architecture consists of two parts, namely, encoding and decoding. The input of the encoding part is c, and are regarded as position signals that can help distinguish different elements in c. They are passed through a multi-layer perceptron (MLP) f q To obtain a geometric representation of each electrode. An element-wise max function is chosen to integrate the two electrode representations into the final geometric representation of the electrode pair, because the capacitance is independent of the order of the electrodes (i.e., the capacitance readout between electrode 1 and electrode 2 is the same as the readout between electrode 2 and electrode 1).

[0178] MLP c Map c to a high-dimensional representation, and the sum of the capacitance representation and the geometric representation is of length N m The input sequence of the transformer encoder E.

[0179] For the decoding part, firstly, P S Feed into the MLP f s , and then the converter decoder D is used to decode f s The output of E realizes multi-head attention. Using MLP f d The output sequence of D(·) is mapped to the displacement of each point and is obtained by comparing it with P S Add them together to obtain the reconstruction P.

[0180] It is expected that P is as close as possible to the true value of the target point cloud P. This goal is achieved by minimizing the following loss function:

[0181]

[0182] in represents the remaining points; is the coordinate of the jth remaining point; is the coordinate of the i-th visual marker; N v and N r are the number of visual markers and remaining points, respectively; is the distribution of P; λ1 and λ2 are the weights of the squared distance term of the visual marker and the chamfered distance term of the remaining points, respectively.

[0183] The structure of the C2DT sub-network is as follows:

[0184] ·f s :Linear(3,h em )->ReLU->LayerNorm(hem)->Linear(h em , d model )

[0185] ->ReLU->LayerNorm(d model )

[0186] ·f q :Linear(3,h em )->ReLU->LayerNorm(h em )->Linear(h em , d model )

[0187] ·f c :Linear(1,h em )->ReLU->LayerNorm(h em )->Linear(h em , d model )

[0188] ·f d:Linear(d model , 3)->a*Tanh

[0189] E:

[0190] ·D

[0191] ->

[0192] in

[0193] h em =32,d model =128, a=1.2, d ff =256, h=8, P drop =0.1, n m-layer =1,n e-layer =3,n d-layer =2.

[0194] f q and f c The linear layers in have no learnable bias, while the other layers do. The LayerNorm in E takes as input the sum of the capacitive representation and the geometric representation. The Transformer.EncoderLayer and Transformer.DecoderLayer are exactly the same as the original Transformer. The first self-attention unit of the Transformer.DecoderLayer is removed and the rest is used as a Transformer.MutualLayer since Ps remains constant. Indicates n e-layer A stack of Transformed.EncoderLayer.

[0195] The simulated dataset is split into three mutually exclusive parts, namely, training set, validation set and test set. The training set includes 22517340 frames (548 rounds), of which 1334 frames (31 rounds) have the first type of deformation; 7204 frames (172 rounds) have the second type of deformation; 6980 frames (173 rounds) have the third type of deformation and 6999 frames (172 rounds) have the fourth type of deformation. The validation set includes 9721 frames (236 rounds), of which 550 frames (12 rounds) have the first type of deformation; 3093 frames (74 rounds) have the second type of deformation; 3098 frames (76 rounds) have the third type of deformation and 2980 frames (74 rounds) have the fourth type of deformation. The test set includes 7096 frames (172 epochs), of which 435 frames have the first type of deformation; 2255 frames (54 epochs) have the second type of deformation; 2191 frames (54 epochs) have the third type of deformation and 2215 frames 3.5* (54 epochs) have the fourth type of deformation.

[0196] C2DT is implemented in PyTorch. The Adam optimizer (β1 = 0.9, β2 = 0.98, ∈ = 10 -9 ) Update the learnable parameters and minimize the loss function. Use an initial learning rate of 0.001 and decay by 1.2 times every 15 epochs. λ1 and λ2 are calculated as follows:

[0197] λ1=λ / 3(λN v +2N r ), λ2=1 / 3(λN v +2N r ), where λ = max(1,300-2*(epoch-1)).

[0198] Gradients were clipped with a threshold of 0.5 and C2DT was trained for 300 epochs using the training set with a batch size of 24. Each epoch took about 9 minutes on 3 Nvidia Quadro P5000s. The network with the minimum validation loss was saved as the final model.

[0199] The performance of C2DT is evaluated by four error metrics, namely, average distance (AD), maximum distance (MD), chamfer distance (CD), and Hausdorff distance (HD).

[0200]

[0201] To understand the impact of each loss term and position encoding on the performance, an ablation study was implemented. The squared distance term and the chamfer distance term were removed separately, and the same training process was performed to obtain the results of C2DT without markers and C2DT without chamfer distance. Attempts to train the network without the position encoding part failed to converge. The position representation of the trained C2DT can be visualized via t-SNE29, which can help discover geometric correlations between different electrode pairs. The performance of C2DT with different network hyperparameters, different numbers of visual markers, and different electrode layouts was investigated using the same method to guide real-world sensor and network design.

[0202] According to the embodiment, a 32-electrode SCAS consisting of 8 modular 4-electrode SCASs was manufactured, each SCAS having 4 different functional layers, namely, a protective substrate, an electrode layer, an isolation layer, and a sealing layer. Each module was manufactured layer by layer as follows:

[0203] i. Mix part A (1.0) and part B (1.0) of Smooth-on Ecoflex 00-30 and pour on a glass plate. Flatten the silicone using a TQC gloss micrometer film applicator and cure in an oven at 100°C for 3 minutes.

[0204] ii. Mix Imerys Enasco 250P conductive carbon black (0.2) with isopropyl alcohol (2.0) and then add uncured silicone mixture (2.0) and stir for 3 minutes. Apply a layer of uncured conductive silicone on the protective substrate and cure in an oven at 100°C for 3 minutes.

[0205] iii. The CB electrodes were patterned using a 40W Aeon MIRA 5 laser engraving and cutting machine. The parameters were set as follows: 28% 386 power, 300 mm s-1 speed, and 0.05 mm spacing. The planar size of each electrode was 21 mm × 6 mm, which is one fifth of the size studied in the simulation.

[0206] iv. A silicone film for the isolation layer on top of the electrode layer was made using the same method as (i).

[0207] v. Use 20.5% power, 300 mm s -1 Two rounds of engraving were performed at 19.5% power and 25 mms intervals to generate 380 microchannel liquid metal lines and connections to the readout electronics. Four rounds of engraving were performed with the same parameters to generate vertical interconnect holes. The planar dimensions of the readout connections and vertical interconnect holes were 3 mm x 2 mm, and the line width was 0.5 mm. -1 Speed ​​cut a rectangular area of ​​the modular SCAS and remove the rest.

[0208] vi. A new piece of silicone film was made as in step i and a very thin layer of uncured silicone mixture was evenly coated on its surface as an adhesive. The cut SCAS formed in step v was then attached to the top of the film and any trapped bubbles were manually squeezed out. After curing at room temperature for about 4 hours, a good quality bond was obtained.

[0209] vii. Eutectic Gallium 75.5% Indium 24.5% (EGaIn, Sigma Aldrich) ink was injected from the readout connection via a syringe with a 0.33 mm needle and the air in the microchannel was expelled through the vertical interconnect holes. The injection point was then sealed using an uncured silicone mixture.

[0210] viii. Obtain the final modular 4-electrode SCAS.

[0211] The planar dimensions of the SCAS module are 120 mm x 20 mm, of which 100 mm x 20 mm is the area of ​​the electrodes, which is one fifth of the equivalent area in the simulation, and 20 mm x 20 mm is the interface area with the readout electronics. The layer thicknesses are 0.39 mm, 0.08 mm, 0.24 mm, and 0.3 mm, respectively. Five SCAS modules were connected in parallel in this example because the manufacturing is easy to scale up.

[0212] A square cylinder model robot arm with a casting size of 20mm×20mm×240mm was cast, which is one-fifth of the size in the simulation. The additional 40mm height is the interface area for driving deformation and bonding with the fixed ceiling. Eight 4-electrode SCAS modules were bonded to its surface to form a 32-electrode SCAS proprioceptive system. The silicone layer was coated with white Smooth-on Silc Pig silicone pigment for better reflection. 16 yellow dots were attached as visual markers, which can assist network training with correspondence information, and the readout value electronic device interface was covered with black acrylic tape to reduce its interference with point cloud collection.

[0213] To characterize the response of the SCAS module and demonstrate the superior performance of EGaIn wires over CB wires, a 4-electrode SCAS with CB wires and a 4-electrode SCAS with EGaIn wires were placed on the front and back of a square-prism silicone structure (20mm×20mm×140mm) and cyclically stretched using a Nema23 stepper motor with an SFU1605 ball screw. Each cycle took 20 seconds and the strain of the SCAS was as high as 40%. The entire test took about 3 hours (over 500 cycles). The relative capacitance readouts of each SCAS were compared, and the results showed that the SCAS with EGaIn wires can have better sensitivity (greater response under the same deformation), linearity (no distortion in the response curve), and cyclic stability (no displacement after 500 cycles of stretching).

[0214] In the experiments, the experimental platform consists of a simulated soft arm equipped with a 32-electrode SCAS, readout electronics, two Microsoft Azure Kinect RGB-D cameras, and a laptop computer to control the readout electronics and record data from the cameras and SCAS. The readout electronics are based on a 32-electrode ECT system that supports arbitrary switching schemes. The capacitance measurement resolution is 3fF, and the signal-to-noise ratio is higher than 60dB for all channels.

[0215] Two cameras are placed directly opposite and in line with the model robot to capture the real-time 3D deformation of the model robot from two complementary perspectives. The deformation is saved and represented via a color point cloud format. The camera and the readout electronics record the data synchronously. If only the point cloud and capacitance data are recorded, the frame rate can reach about 30fps, while if the RGB image is recorded at the same time, the frame rate is reduced to about 20fps.

[0216] During the real-world experiments, the hand-held holder glued to the bottom of the model robotic arm was manually manipulated to induce various complex deformations, including stretching, twisting, bending, and their combinations. At the same time, the SCAS and the camera synchronously recorded the data (capacitance readouts, color point clouds, and sometimes RGB images) under deformation. A total of 36,465 frames (about 1,220 seconds) of data were collected, of which only the capacitance readouts and color point clouds were recorded in the first 36,013 frames (about 1,200 seconds), and additional RGB images were saved during the last 452 frames (about 20 seconds). The 32-electrode SCAS can produce 76 capacitance readouts in each frame, which were calibrated using the same method as the simulation. The point clouds from different cameras were fused in one coordinate system by the chessboard calibration method. The raw data is noisy and contains a large number of meaningless background points, so it cannot be used for training purposes in this format. The data was cleaned and preprocessed using Matlab and its computer vision toolbox to selectively retain only the points on the surface of the model robotic arm. The points on the black acrylic tape and the red holder were eliminated via color filtering. To further reduce the negative impact of noise and outliers, areas with local point density below a preset threshold were filtered. Due to inevitable visual occlusion during the experiment, the cleaned point cloud cannot fully represent the 3D deformation in many frames. To avoid this problem, further preprocessing is required before training. First, average grid downsampling is performed with a 4mm box grid filter to achieve computational efficiency. The alpha shape is then reconstructed based on the downsampled point cloud to alleviate the problem of incomplete representation. The triangle mesh of the alpha shape is subdivided three times and the vertices are extracted as a new point cloud with supplementary points. In the C2DT framework, the source point cloud is expected to have the same number of points as the target point cloud. To meet this requirement, average grid downsampling is implemented with a 4mm box grid filter, and then 1300 points are finally selected in each point cloud using farthest point sampling. Yellow visual markers are extracted from the cleaned point cloud before downsampling, and shape reconstruction is performed based on the RGB information of each point. A graph based on the labeled points for one frame is created. The connection between each two points in the graph is determined by their distance. The threshold of the distance to be connected is 6mm. Each connected subgraph with more than 10 points is considered as a visual marker, and the average of the coordinates of all points in the subgraph is used to represent the marker position. Due to camera occlusion, the number of extracted visual markers is not always 16. The visual markers are aligned layer by layer. The 16 visual markers can be divided into 4 layers, and each layer includes 4 markers. The figure is based on the coordinates of the extracted markers connected to a frame with a distance threshold of 26mm. Each subgraph is a layer of markers. The arrangement of the layers is determined by the relative position of the y-axis of the fused coordinate system among the 4 layers. All abnormal frames where the number of extracted markers is greater than 16 and / or the number of layers is not equal to 4 are deleted.Layers with less than 4 markers are padded with (0,0,0) to ensure that all layers have the same number of points, which can improve computational efficiency during training. In addition, frames with severe missing points are problematic due to the low quality of their reconstructed shapes. The number of markers in each layer indicates the severity of the missing points. Frames with at least 2 markers in all layers are retained, while other frames are ignored. After the above filtering process, a total of 31,380 frames of data are available for analysis. A random check was performed on 500 frames in the dataset, and no samples with severe missing point problems were found.

[0217] The basic framework of C2DT in real-world experiments is similar to that in simulation experiments. However, some modifications may be required due to the differences between real and virtual environments. First, the loss function in simulation is no longer applicable to the above experimental conditions because there are no visually marked point-to-point correspondences. Instead, a new loss function similar to:

[0218]

[0219] The first term of the loss function counts the chamfer distance between the reconstruction of the mark and the true value layer by layer, where It is the first k The coordinates of the visual markers in the layer; YesP lk The coordinates of the i-th point in , yes With P lk The square distance between the closest points; N l is the number of layers; N lv is the number of tokens in each layer, and in this case N l and N lv The value of is 4. When calculating the loss, only the labeled points extracted during data preprocessing are considered and the padding points are ignored. All points in are labeled points because they are generated by the network based on the corresponding capacitance readouts and source points excluding the padding points. To eliminate the influence of the padding points during training, the mask is synthesized as follows:

[0220] ·if is a marking point, then is set to 1. If is the filling point, then is set to 0.

[0221] ·if Excluding any fill points, then is set to 1, otherwise is set to 0.

[0222] The second term in the loss function is exactly the same as its simulated counterpart, counting the chamfer distance between the reconstruction and the true value of the remaining points. The third term is a regularizer that encourages the distance between neighboring points to not change significantly before and after the deformation. Here, yes The lth neighbor of s j,i is the distance between the corresponding two points in the source point cloud; δ d and δ u is the coefficient of the threshold. The loss is only calculated when the neighbor distance in the reconstruction falls outside the preset range. This is achieved with a mask like this:

[0223] ·if So is set to 1, otherwise is set to 0.

[0224] ·if So is set to 1, otherwise is set to 0.

[0225] The number of input frames in the physical world is not constant at 1. In contrast, C2DT takes several (Ni) adjacent frames as input. Therefore, the first linear unit in fc is modified to Linear(N i ,h em ). The hyperparameters of C2DT are set to: h em =32,d model =64,d ff =128, h=4, P drop =0.1, n e-layer =2,n m-layer =1,n d-layer =1 and a=1.9472.

[0226] The values ​​of the point cloud data are scaled up five times to make it close to the simulation scale. The network is trained and evaluated using a process almost identical to the one presented above. The real-world dataset is split into three mutually exclusive parts. The first 26771 frames (about 1020 seconds) are used for training (20693 frames) and validation (6018 frames), and the last 4669 frames (about 200 seconds) are used for testing. d is set equal to 0.5 and δ u is set equal to 2. λ1, λ2, and λ3 are calculated as follows:

[0227]

[0228] where λ = max(1300-10*(epoch-1)). A total of 200 epochs are run with a batch size of 39, and the network with the smallest validation loss is retained. Ablation studies are implemented to evaluate the impact of each loss term, and real-world scale reconstruction is quantitatively evaluated using CD and HD metrics that do not require point-to-point correspondence. Finally, the representation of each electrode pair is visualized via t-SNE to illustrate the geometric correlation between different capacitance readouts.

[0229] In the above-described embodiments, obtaining a measurement of a selected electrode pairing involves activating a single electrode of the pairing and measuring the capacitance at the corresponding single electrode of the pairing. In other embodiments, one or more of the selected pairings may include three or more electrodes. As a non-limiting example, such pairing may involve pairing between a first group of electrodes and a second group of electrodes. In operation, the first group of electrodes may be activated simultaneously to form a combined electrode, and the second group of electrodes may also be combined to form a corresponding combined electrode to measure capacitance. In the context of electrical capacitance tomography, Yang et al. describe in more detail an example of such a multi-electrode sensing strategy involving pairing of more than two electrodes in "A novel multi-electrode sensing strategy for electrical capacitance tomography with ultra-low dynamic range".

[0230] Fig.14 Sensing devices according to other associated embodiments are schematically depicted, illustrating combining groups of electrodes to form a pairing between a first group of electrodes and a second group of electrodes. Fig.14 Depicted are excitation circuitry 1402 and measurement circuitry 1404 configured to apply excitation signals to a selected group of electrodes. Fig.14 Also illustrated is the sensing device in a disassembled configuration (1406) and an assembled configuration (1408) according to an embodiment.Excitation circuitry 1402 may be considered to constitute part of drive circuitry, while measurement circuitry 1406 may be considered to constitute part of signal readout circuitry.

[0231] Fig.14 Depicted is an equivalent capacitor, which is a linear combination of a series of capacitors formed by two individual electrodes. Fig.14 The two electrodes (24 and 32) are shown tied together to form a combined measurement electrode. In more detail, both electrodes are connected to the same terminal in the measurement circuit so that the capacitance of the combined electrode (24 and 32) can be measured by the measurement circuit. Fig.14 According to the linear superposition principle, C 8,24-32 =C 8,24 +C8,32 , the capacitance formed by the example pairing of electrode 8 and combined electrodes 24 and 32 is equal to the capacitance between electrodes 8 and 24 plus the capacitance between electrodes 8 and 32.

[0232] In other example embodiments, a 32-electrode multiplexer array is provided to allow suitable connections to be formed between electrodes, signal readout value circuitry, and signal drive circuitry to form a desired electrode group. In this example embodiment, the 32-electrode multiplexer array allows each electrode to be connected to one of the excitation circuitry (for receiving an excitation signal), a measurement terminal (of the measurement circuitry), or a ground terminal. The 32-electrode multiplexer array is controlled by a control signal. The 32-electrode multiplexer array allows the electrode to be connected to one of the excitation circuitry, the measurement circuit, and the ground terminal in response to receiving the control signal.

[0233] In more detail, a group of multiple electrodes can be simultaneously excited by the sensing device by controlling the multiplexer to connect each of the multiple electrodes to the excitation source at the same time. Similarly, another group of multiple electrodes can form a combined measurement electrode by controlling the multiplexer to connect each electrode of the other group to the measurement terminal. The multiplexer is further controlled to connect all other electrodes (all electrodes not in the first group or the second group) to the ground terminal. Although a 32-electrode multiplexer array is described, it will be understood that other suitable signal routing circuit systems can be used.

[0234] In such example embodiments, a specific combination electrode strategy can be designed and used. The electrode strategy can include using a 32-way multiplexer array or other signal routing circuit system to send control signals to the electrodes. The control signal can include a sequence of digital signals, for example, a control word for controlling the 32-electrode multiplexer array.

[0235] The skilled person will appreciate that variations in the accompanying arrangements are possible without departing from the invention.

[0236] For example, in the above embodiments, the electrodes are described as being distributed around the surface of the object, however, it will be understood that in other embodiments, one or more electrodes may be embedded within the object or at a depth from the surface of the object.

[0237] In the above embodiment, the number of electrodes is 32. However, it will be appreciated that this number is not fixed. In practice, the number of electrodes used for a selected object may depend on the complexity of the object shape or the size of the object (e.g., the larger the object, the more electrodes that can be implemented).

[0238] In addition, although Figure 1 The electrodes are depicted as part of the sensing device, but it will be understood that Figure 1Other elements of the sensing device depicted in the figure may form part of the sensing device (e.g., the driving module and / or the readout value module may form part of the sensing device). In some embodiments, the sensing device may include a driving module, electrodes, a readout value module, processing resources, and memory resources.

[0239] In addition, in the above-described embodiments, the sensing of capacitance signals is described. However, it will be understood that in other embodiments, other types of electronic signal outputs may be sensed. For example, the signal output may include voltage and / or potential difference. In particular, the sensing device measures voltage. In some embodiments, the electrode measures a voltage having a magnitude (e.g., amplitude) that is linearly related to the capacitance to be measured. In other embodiments, the sensing device may be extended to measure impedance. In some embodiments, the signal is all in the form of voltage and is in a sine / cosine form with a certain phase and / or amplitude, similar to a modulated signal.

[0240] In the above embodiments, a learning-based approach for full geometry reconstruction is described. Although other approaches can be used, very strong constraints and prior information will be essential, and the model is usually applicable to specific objects and lacks generalization capabilities. Learning-based approaches can be easy to implement and generalize to different scenarios through established training pipelines.

[0241] In addition to or as an alternative to sensing deformation and shape information, the sensing device may also be able to detect, for example, electrical properties and touch. For example, as described above, due to the nature of capacitive electrodes, the capacitance value is sensitive to changes in the dielectric constant in the close proximity of the skin surface. Therefore, an object close to the skin may cause a change in the dielectric constant, thereby affecting the capacitance value. This feature can be used to sense touch or, for example, collision detection.

[0242] Another embodiment is described below. The capacitance between two boundary electrodes can be expressed as:

[0243]

[0244] where C is the capacitance; Q is the stored charge; V is the potential difference between the boundary electrode pair; E(x,y,z) and φ(x,y,z) are the dielectric constant and potential distribution in the sensing domain, and Г represents the electrode surface. Compared to other applications, when applied to soft robotics, both the dielectric constant distribution and the geometry of the electrodes are subject to changes, which increases the complexity of inferring any desired information from the obtained capacitance data. It has been found that the dielectric constant distribution and geometry of the boundary electrodes can trigger different modes in the capacitance measurement, enabling the simultaneous decoding of tactile / touch and deformation information. It will be understood that touch refers to the proximity or contact of an object, which causes a change in the properties of the material between the electrodes, in particular a change in the dielectric constant in the sensing area.

[0245] Fig.15 (a) shows a sensor module (also called an electronic skin module) according to other embodiments. Fig.15 The sensor module has four electrodes: a first electrode 1514a, a second electrode 1514b, a third electrode 1514c and a fourth electrode 1514d. The sensor module is manufactured to be deformable under the action of force, for example, stretchable, twistable and / or compressible. In particular, Fig.15 (a) depicts the module in a first, unstretched and undeformed configuration, while Fig.15 (b) depicts the module in a second, stretched or elongated configuration. Compared to the sensor module of FIG. 4 (which depicts electrodes, connectors, and connector links deployed in the material), Fig.15 In an embodiment of the present invention, the electrode is a liquid metal wire substantially elongated along the first direction.

[0246] exist Fig.15 In the embodiment of (a), the electrodes are arranged in a parallel arrangement and each electrode has a conductive portion elongated in the longitudinal direction of the sensor. Fig.15 (a) Also depicted are connectors (1522a, 1522b, 1522c, 1522d) for the electrodes. The connectors provide an interface between the liquid metal electrodes and the corresponding metal readout value lines. A metal readout value line 1523 for the fourth electrode 1514d and connector 1522d is indicated. In this embodiment, the electrodes are liquid metal wires and are provided together with connectors (also referred to as interfaces). In this embodiment, the dimensions of the sensor module in the first (unstretched) configuration are 400mm in width and 1100mm in length. The width of each electrode is 1mm, and the lengths are 20mm, 45mm, 70mm, and 95mm, respectively. The area of ​​each liquid metal interface is 5mm×5mm.

[0247] The four liquid metal wires provide elongated conductive portions operable as electrodes, which in combination can form a capacitor as described above. It will be appreciated that the sensor module is configured to be combined with one or more other sensor modules to assemble a sensing device, substantially as described with reference to the embodiment of, for example, FIG.

[0248] As from Fig.15 (a) As can be seen, each of the four electrodes of the module has a different length and they are arranged in a parallel arrangement. In this embodiment, each length is successively longer than the previous length. It will be understood that Fig.15Each electrode of (a) has a conductive portion and is configured to perform a capacitive reading substantially along its length. The different lengths of each individual electrode can provide advantages for touch sensing because the difference in length can increase the difference in the sensed capacitive signal, for example, when contacting different areas of a module or sensing device.

[0249] As from Fig.15 (b) As can be seen, the sensor module is configured to be deformed and stretched at least in the longitudinal direction. The electrodes are arranged substantially parallel to the longitudinal direction. Stretching of the sensor module causes each electrode to extend or further lengthen. In a first undeformed configuration, each electrode has a first length, and in a second deformed configuration, the electrode has a second, longer length.

[0250] Fig.15 (c) depicts Fig.15 Hierarchical structure of sensor modules. Fig.16 A sensor module is depicted having two layers: a substrate layer 1546 and a protective layer 1542. The electrodes 1514a, 1514b, 1514c, 1514d and their corresponding connectors and interfaces are disposed in the substrate layer 1546. Fig.15 (c) The electrode layer and the protective substrate layer are depicted together. The electrodes and interfaces are both provided in the electrode layer.

[0251] The electrode layer includes a substrate formed with platinum-catalyzed silicone. In this embodiment, the platinum-catalyzed silicone is Ecoflex00-30 silicone. Microchannels are fabricated in substrate layer 1546 using 3D printed casting. Once the substrate is formed, a protective layer 1542 is formed from a silicone film fabricated by membrane coating and bonded to substrate layer 1546 using uncured silicone as an adhesive. Liquid metal ink is injected into the formed microchannels to form sensing electrodes. In this embodiment, the liquid metal ink is eutectic gallium 75.5%, indium 24.5% (EGaIn) ink and is injected from an interface. Air is exhausted through the end of the wire. Compared to the embodiment described in reference FIG. 4( c ), no carbon black electrode is fabricated, so the number of layers is reduced to two. As shown in FIG. Fig.15 (b) It can be seen that the elongated conductive portion is embedded in the substrate so that the conductive portion can be deformed together with the substrate.

[0252] Fig.16 A 3-chamber pneumatic manipulator (size 500 mm × 1200 mm) is depicted as a test bed for validating the proposed method. Pneumatic soft robots are often used in many applications and can provide different deformations, e.g., inflation and bending. The structure of the manipulator is shown in Fig.16The manipulator has three chambers, each chamber is 400mm×300mm in size, and each chamber has an air inlet to allow air to be injected into each chamber. The width of each inlet is 1.5mm. Two sensor modules (as shown in reference Fig.15 The two sensor modules together form an 8-electrode capacitive sensor, as each module has 4 electrodes, capable of generating 28 capacitive readout values ​​per measurement frame.

[0253] The following is a non-limiting experimental method and results. The three-dimensional vector p = (p1, p2, p3) is used to describe the inflation state of the robot, where p1, p2 and p3 are the volumes of air injected into the first, second and third air chambers, respectively. In the experiment, 20 ml of air was injected into each chamber of the robot at the same time (i.e., p = (20, 20, 20) ml) and the sensor response signal was recorded.

[0254] Fig.17 The results of measurements performed during such an inflation of the manipulator are shown. Fig.17 (a) shows all 28 capacitance readings during the inflation process. The y-axis is the calibrated capacitance C (i.e., the relative change in capacitance), which can be calculated by the following formula:

[0255]

[0256] Among them C t is the capacitance readout in the current state and C0 is the capacitance readout in the reference state (state without inflation). p = (20, 20, 20) ml inflation can stimulate a maximum change in relative capacitance change of about 40%. The capacitance readout of a capacitor formed by electrodes in the same surface generally increases as the pneumatic robot is inflated (example in Fig.17 (b) The capacitance readouts of capacitors formed by electrodes on different surfaces show opposite trends (examples are in Fig.17 (b) is shown on the right hand side. Inflation of the robot body makes the area of ​​the electrodes (positively correlated with capacitance) larger and the distance between the electrodes (negatively correlated with capacitance) longer. For capacitors formed by electrodes on the same surface, the increase in electrode area dominates the change in capacitance. For capacitors formed by electrodes on different surfaces, it is found that the change in the distance between the electrodes can be more significant and can dominate the capacitance change.

[0257] Fig.18Results from two phase experiments are shown, illustrating the difference in capacitance changes in response to touch and deformation. Capacitance changes in response to touch can be dominated by dielectric constant changes. Capacitance changes in response to deformation can correspond to geometric changes. According to the capacitance calculation equation, touch will cause dielectric constant changes, which will lead to further changes in the signal. The same set of electrodes is used to detect both touch and deformation, but different signal patterns will be detected. Touch can refer to the application of a localized contact force on the surface of the sensor.

[0258] In the first stage, the robot was inflated to (0, 20, 20) ml without additional contact. The front surface of the robot was then divided into 9 parts (see Fig.19 ), and touch each part individually to form a touch measurement. Each contact can be called a touch event or a touch action and is performed at the contact location. Fig.19 Indicates a location on the surface of the robot. In use, contacting one of the areas of the surface will result in identification of the contact location on the surface. For example, a local contact in the area labeled 1 will return a label of "1" from the trained model. It will be understood that more than one touch event or action occurring substantially simultaneously may be detected. Although Fig.19 Nine touch areas are depicted, but it will be understood that more electrodes may be used to provide greater distinguishability and more touch areas.

[0259] Fig.18 (a) shows the overall response of the sensor. Capacitive response of (0, 20, 20) ml gas filling and Fig.17 The capacitive response of the (20,20,20) ml inflation shown in (a) is similar. Inflation can trigger a change in every capacitive readout simultaneously (global response), while touch only stimulates a change in a portion of the readouts based on the location of the touch point (local response). Fig.17 Examples of capacitance readouts from four different electrode pairs are shown in (b). Fig.17 (b) illustrates that different electrode pairs have different sensing fields. The capacitance readout reflects only touches within its own sensing field and is insensitive to touches outside the area. This feature results in different patterns of capacitance signals caused by inflation and touch. Small fluctuations in the capacitance readout during touch are also observed. This is caused by deformation of the robot body caused by touch / contact (e.g., bending).

[0260] Further experiments are described below, where data were collected to verify the feasibility of using the proposed flexible sensor module to identify touch during deformation (e.g., during inflation). Ideally, it requires both the location of the touch point and the sensor signal. However, it has been found that determining the exact touch (or contact) location as the robot moves and deforms during the experiments is a significant challenge. Therefore, the surface of the robot was divided into 18 sub-areas (see Fig.19 ). Then, the sub-regions are randomly touched in the experiment and the response signals are recorded. The index of the sub-region reflects the rough location of the touch point.

[0261] In addition, the possibility of extracting deformation information from the sensor signals was investigated, as deformation tracking is a key topic and sensing devices with multiple functions are expected in soft robots. For pneumatic robotic platforms, inflation information is usually known because the amount of air injected into the chamber can be controlled. Therefore, deformations caused by user interactions, such as bending caused by touch, were investigated. In order to obtain deformation labels during the experiment, five reflective visual markers were glued to the sides of the robot. Three OptiTrack Flex 13 cameras were then deployed around the robot to capture the real-time 3D coordinates of the markers. The coordinates provide a brief description of the deformation and are used as deformation labels later. The experimental platform includes a pneumatic robot equipped with an 8-electrode capacitive skin (as described above) and five reflective visual markers, using OptiTrack Flex cameras and readout electronics and can achieve 3fF capacitance measurement resolution and a signal-to-noise ratio of more than 60dB for all measurement channels. The data recording speed of the camera and readout electronics was set to 30fps.

[0262] The experiments were performed at 27 different inflation states, ranging from (0,0,0) to (20,20,20) ml. In each inflation state, data were collected from 7 separate periods. Each period lasted 30 seconds. During the first period, the robot was inflated to a preset state without touch. In subsequent periods, sub-areas of the robot were randomly touched to induce deformation through contact force. A series of contact actions were performed at multiple locations on one or more surfaces of the sensing device, and the position information of the touch points was recorded.

[0263] For touch recognition, 189 (27*7) sets of different inflation states and touch points were obtained, each set of data included 30 seconds of capacitance signal (i.e., 900 frames when the sampling speed is 30fps) and the index of the corresponding touch point. The data was specifically divided into a training set (125×30=3750 seconds, 3750×30=112500 frames), a validation set (32×30=960 seconds, 960×30=28800 frames), and a test set (32×30=960 seconds, 960×30=28800 frames). For deformation tracking, a set of data consists of 30 seconds of capacitance signal and the trajectory of the visual marker recorded by the camera. The data with no touch and occlusion problems were manually filtered. 146 sets of data were obtained, which were specifically divided into a training set (108×30=3240 seconds, 3240×30=97200 frames), a validation set (18×30=540 seconds, 540×30=16200 frames) and a test set (20×30=600 seconds, 600×30=18000 frames).

[0264] For deformation tracking, a set of data consists of 30 seconds of capacitive signals and the trajectory of visual markers recorded by the camera. The data with no touch and occlusion problems were manually filtered. 146 sets of data were obtained, which were specifically divided into training set (108×30=3240 seconds, 3240×30=97200 frames), validation set (18×30=540 seconds, 540×30=16200 frames) and test set (20×30=600 seconds, 600×30=18000 frames).

[0265] Fig. 20 is a schematic diagram of the neural network architecture used to implement touch point classification. The neural network architecture can also be called a multi-layer perceptron (MLP). In this experiment, 19 different classes were used in this study, corresponding to 18 different touch points and one no-touch condition. The input to the MLP is 28 calibrated capacitance readouts in a frame. The MLP outputs a vector of size 19 indicating the class probabilities. A cross-entropy based loss function is used. The MLP has one hidden layer with 128 neurons. The activation function used for the hidden layer is ReLU. Dropout (p=0.1) is used to prevent overfitting. The output of the neural network is the location of the touch or the touch area.

[0266] The training of the MLP was implemented in Pytorch. The Adam optimizer was used to update the learnable parameters to minimize the cross entropy loss between the predicted touch points and the true value touch points. The initial learning rate was set to 0.001 and decayed by a factor of 1.2 every 15 epochs. The training set was used for 100 epochs with a batch size of 256, and the network with the minimum loss was saved on the validation set. The training process took 10 minutes on an Nvidia Quadro P5000 GPU card.

[0267] After the training process, the MLP demonstrated a classification accuracy of 99.88% on the test set. This shows that the proposed soft skin can estimate touch points using a simple deep learning model even when the signal is severely disturbed by the inflation of the robot body. The confusion plot of the classification results is shown in Fig.21 The confusion map shows that only 34 frames are misclassified out of 28,800 test frames. All misclassifications occur between adjacent sub-regions. For example, 34 touches on sub-region 2 are incorrectly classified as sub-region 3. This is because the signals caused by touches on adjacent sub-regions have relatively high similarities, which confuses the network.

[0268] Although the reference Fig.15The sensor module of describes a trained model that outputs touch position information, but it will be understood that sensor modules of other embodiments (e.g., the sensor module of FIG. 4) can be used to train such a model.

[0269] Turning to deformation tracking, estimating the coordinates of visual markers based on capacitive signals can be viewed as a set-to-set problem. We adopt the Capacitance-to-Deformation Transformer (C2DT) described above. C2DT is a transformer-based architecture deployed to reconstruct the point cloud geometry of a soft robot. Figure 8 Describe the structure of C2DT.

[0270] The cameras and readout electronics were synchronized via an automated click script, which resulted in a slight delay between the data recorded by different devices. Therefore, 10 frames of calibrated capacitance signals were fed into C2DT to mitigate this problem. The position signal consists of the position information of the electrode pairs forming the capacitor, which can help C2DT distinguish the capacitance readouts generated by different electrode pairs. The squared error between the estimate and the true value was chosen as the loss function.

[0271] The above description focuses on deformations caused by user interaction (e.g., touch) rather than inflation (which is known in most scenarios because the amount of air injected into the chamber is controllable). However, the signals caused by these deformations are much smaller than those triggered by inflation. Directly estimating the coordinates of the visual markers can be challenging. To address this issue, the first frame in each trajectory is used as a priori knowledge, i.e., the capacitance readout value is used as a reference to calibrate the capacitance input and the coordinates of the markers are used as the source sequence for the transformer decoder.

[0272] The training of the MLP is implemented in Pytorch. We use the Adam optimizer to update the learnable parameters to minimize the squared loss between the predicted and true value coordinates. The initial learning rate is set to 0.001 and decays by a factor of 1.2 every 15 epochs. We run 150 epochs of training with a batch size of 255 using the training set and save the network with the minimum loss on the validation set. The training process takes 2.5 hours on three Nvidia Quadro P5000 GPU cards.

[0273] The average distance (AD) between the estimated and true coordinates of the markers is used to evaluate the performance of the modified C2DT, which is defined as: where N is the number of samples in the test set, M is the number of visual markers, pid is the true coordinate of the ith visual marker of the ith test sample, and pid is the estimated coordinate of the ith visual marker of the ith test sample. After training, C2DT can achieve 2.905±2.207mm AD error. This shows that with the prior knowledge (capacitive signal and coordinates of the marker in the first frame of each trajectory), the proposed skin can be applied to track deformation using C2DT even in environments with severe disturbances (inflation and dielectric constant changes caused by touch). Fig. 22 Several examples of tracking results are shown in . In all cases, it is observed that the estimated visual markers (red) are close to the ground-truth visual markers (blue), indicating high accuracy of deformation tracking.

[0274] The ground truth markers (e.g., marker 2202) represent the location of the physical marker attached to the manipulator, which is captured by the tracking camera. The blue ground truth represents the coordinates of the visual marker collected by the camera. The red estimate (e.g., marker 2204) represents the output of the network. Fig. 22 The overlap between the output of the network and the locations of the physical markers for all points is shown.

[0275] Therefore, the description of the above specific embodiments is for illustrative purposes only and not for limiting purposes. It will be clear to the skilled person that minor modifications can be made without major changes to the described operations.

Claims

1. A sensing device, comprising: a plurality of electrodes arranged to be distributed around one or more surfaces of the deformable object, wherein the plurality of electrodes are operable to generate a capacitive signal output from a plurality of selected pairs of the plurality of electrodes, wherein the plurality of electrodes are distributed about the one or more surfaces of the deformable object such that the plurality of selected pairs include at least one proximal pair and at least one non-proximal pair, Wherein the generated capacitive signal output of the electrode pair depends on at least one of: the distance between the electrodes; the shape and / or orientation of the electrodes and at least one property of the material between the pair.

2. A sensing device as claimed in any one of the preceding claims, wherein the generated capacitive signal output can be processed to determine information associated with the deformable object, wherein the information includes at least one of the following: shape information; deformation information; force information; and / or velocity field information.

3. A sensing device as claimed in any preceding claim, wherein at least one electrode of the plurality of electrodes is deformable, stretchable and / or compressible.

4. A sensing device as described in any of the preceding claims, wherein the plurality of electrodes includes two or more proximal electrodes and two or more distal electrodes, and wherein the selected pairings include at least one pairing between a proximal electrode and a distal electrode and at least one pairing between two proximal electrodes and / or at least one pairing between two distal electrodes.

5. A sensing device as claimed in any preceding claim, wherein the plurality of electrodes are distributed along the one or more surfaces to form a three-dimensional distribution, wherein the three-dimensional configuration is continuously deformable from a planar configuration.

6. A sensing device as claimed in any preceding claim, wherein the capacitive signal output is processable to obtain deformation information associated with at least one of: bending, twisting, elongation, expansion, compression of the object.

7. A sensing device, wherein the plurality of electrodes are disposed laterally along a first surface of a deformable object and at least a second surface of the deformable object.

8. The sensing device of any one of the preceding claims, wherein the plurality of selected pairings comprises at least one pairing of a first electrode of the plurality of electrodes with a non-adjacent electrode.

9. A sensing device as claimed in any preceding claim, wherein the plurality of electrodes are operable to generate capacitive signal outputs from the plurality of selected pairs according to a predetermined sequence.

10. A sensing device as claimed in any preceding claim, wherein the plurality of selected pairings comprises a subset, such as a degenerate subset, of all pairings of the plurality of electrodes.

11. The sensing device of any of the preceding claims, wherein the plurality of electrodes comprises at least two electrodes arranged in a first plane and at least two electrodes arranged in a second plane, wherein the first plane is not substantially parallel to the second plane.

12. A sensing device as claimed in any one of the preceding claims, wherein the plurality of electrodes are disposed in one or more deformable substrates which are conformable to a surface by deformation.

13. A sensing device as described in claim 12, wherein the one or more deformable substrates are stretchable in at least a lateral direction, and optionally, wherein the one or more deformable substrates are stretchable to increase or decrease the distance between two or more electrodes of the plurality of electrodes.

14. The sensing device of any of the preceding claims, wherein the plurality of electrodes forms one or more sensor modules, wherein each sensor module is continuously deformable from a planar configuration and / or integrated to a surface.

15. A sensing device as claimed in any preceding claim, wherein the plurality of electrodes comprises a stretchable conductive material.

16. A sensing device as claimed in any one of the preceding claims, wherein the plurality of electrodes comprise elongated conductive portions, wherein the elongated conductive portions are configured to further elongate in response to a force, optionally wherein the elongated conductive portions of the plurality of electrodes are arranged in a parallel arrangement.

17. A sensing device as claimed in any of the preceding claims, wherein the plurality of electrodes are arranged on or integrated into one or more deformable substrates for application to a deformable object, and / or wherein one or more electrodes are integrated into the surface of the deformable object.

18. A sensing device as claimed in any preceding claim, wherein the deformable object comprises at least one of: a robotic arm; other robotic manipulator; a part of a human body and / or a wearable object.

19. A sensing device, wherein the plurality of electrodes are distributed according to a predetermined layout.

20. A sensing device as described in any one of claims 2 to 19, further comprising a processing resource configured to process the capacitive signal output or capacitive data obtained from the capacitive signal output to obtain said information associated with the deformable object, optionally wherein the processing circuit system is configured to apply at least one predetermined model to obtain the information.

21. A method comprising: obtaining capacitive signal output data representative of capacitive signal outputs from a plurality of selected pairs of a plurality of electrodes distributed about one or more surfaces of the deformable object, wherein the generated capacitive signal outputs of the electrode pairs depend on at least one of: a distance between the pairs; a shape and / or orientation of the electrodes and at least one material property of a material between the pairs; The capacitive signal output data may be processed to determine information associated with the deformable object.

22. The method of claim 21, wherein the method further comprises processing the capacitive signal output to determine the information, wherein the processing comprises applying at least one model to the capacitive signal output, for example, a model determined using machine learning derived processing.

23. The method of claims 21 to 22, wherein processing of the capacitive signal output is performed as part of at least one of: a shape reconstruction process, a deformation detection process.

24. The method of claims 21 to 23, wherein the at least one model is configured to output touch information and deformation information.

25. A method for training at least one model, comprising: Obtain training data, which includes: capacitive signal output data representative of capacitive signal outputs from a plurality of selected pairs of a plurality of electrodes disposed laterally along one or more surfaces of the deformable object; and further data representing information associated with the deformable object; and A model training process is performed using the obtained training data to obtain at least one trained model to obtain further information of the deformable object using the further obtained capacitive signal output.

26. A method as claimed in claim 25, wherein the capacitance signal output data obtained is obtained for one or more spatial configurations of the multiple electrodes and / or in response to one or more deformations applied to the deformable object; and wherein the information obtained includes shape and / or deformation information data corresponding to the one or more spatial configurations and / or the one or more deformations applied.

27. The method of claim 25 or 26, wherein the obtained capacitive signal output data is obtained in response to performing a series of contact actions at a plurality of locations on the one or more surfaces, and wherein the obtained information includes contact position information.

28. A non-transitory computer readable medium comprising instructions operable by a processor to perform the method of claims 21 to 24 or the method of claims 25 to 27.