Sensing devices and equipment

Abstract

A sensing device comprising a plurality of electrodes configured to be distributed near one or more surfaces of a deformable object, the plurality of electrodes being operable to generate a capacitive signal output from a plurality of selected electrode pairs in the plurality of electrodes, the plurality of electrodes being distributed near the one or more surfaces of the deformable object such that the plurality of selected electrode pairs comprises at least one pair of adjacent electrodes and at least one pair of non-adjacent electrodes, and the capacitive signal output generated at the electrode pairs depends on at least one of: the distance between the electrodes of the electrode pair; the shape and / or orientation of the electrodes of the electrode pair; and at least one property of the material between the electrodes of the electrode pair.

Description

[Technical field]

[0001] The present invention relates to sensing devices and apparatus, for example for use with deformable objects. [Background technology]

[0002] Humans are able to effortlessly perceive their posture, movement, and position due to mechanosensory neural networks distributed throughout their bodies. This ability, known as proprioception, allows humans to control their bodies efficiently and precisely, and is also a vital requirement for intelligent robots to undertake dexterous movements. Proprioceptive systems for rigid robots are known and have been applied in a number of applications. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Yang et al., “A novel multi-electrode sensing strategy for electrical capacitance tomography with ultra-low dynamic range” Summary of the Invention [Means for solving the problem]

[0004] According to a first aspect, a sensing device is provided, comprising a plurality of electrodes configured to be distributed near one or more surfaces of a deformable object, the plurality of electrodes operable to generate a signal output from a plurality of selected electrode pairs of the plurality of electrodes. The plurality of electrodes may be distributed near one or more surfaces of the deformable object such that the plurality of selected electrode pairs comprises at least one pair of adjacent electrodes and at least one pair of non-adjacent electrodes. A generated signal output at an electrode of an electrode pair may depend on at least one of: a distance between the electrodes of the electrode pair; a shape and / or an orientation of the electrodes of the electrode pair; and at least one property of a material between the electrodes of the electrode pair. The generated signal output may include a capacitance signal output.

[0005] The sensing device can be used with a deformable object. The sensing device can be integrated into the deformable object. The electrodes can be integrated within the deformable object and / or disposed on a surface or outer surface of the deformable object. The electrodes can form part of a mating layer or skin. The mating layer or skin can be mating with a portion of the surface of the deformable object. The at least one property can include a dielectric constant and / or a conductivity of the material.

[0006] The generated capacitance signal output can be processed to determine information related to the deformable object, which can include at least one of: shape information; deformation information; force information; and / or velocity field information.

[0007] At least one of the plurality of electrodes may be deformable, stretchable, and / or compressible.

[0008] The plurality of electrodes can include two or more proximal electrodes and two or more distal electrodes. The selected electrode pairs can include at least one electrode pair consisting of a proximal electrode and a distal electrode, at least one electrode pair consisting of two proximal electrodes, and / or at least one electrode pair consisting of two distal electrodes.

[0009] The plurality of selected electrode pairs may include at least one pair of two electrodes disposed distally from one another, and at least one pair of two electrodes disposed proximally from one another.

[0010] At least one pair of distal electrodes and at least one pair of proximal electrodes may provide global shape and / or deformation information and local shape and / or deformation information.

[0011] The multiple electrodes can be distributed in at least one layer, and the at least one electrode pair can include a pair of electrodes in the same layer or in adjacent layers.

[0012] At least two of the plurality of selected electrode pairs may include a common electrode.

[0013] The capacitance signal output may include a capacitance value for each selected electrode pair of the plurality of electrode pairs that depends on at least one of: a spatial relationship between the electrodes of the electrode pair, a distance between the electrodes of the electrode pair, a surface area of ​​each electrode of the electrode pair, a relative orientation of the electrodes of the electrode pair, and at least one material property, such as a dielectric constant, of a material between the electrodes of the electrode pair.

[0014] A number of electrodes can be distributed along one or more surfaces, thereby forming a three-dimensional spatial distribution, which can be continuously deformed from a planar arrangement.

[0015] The multiple electrodes can be distributed along and / or across one or more surfaces of the deformable object, and the multiple electrodes can define a sensing volume that corresponds to the shape of at least a portion of the object.

[0016] The capacitance signal output may be processable to obtain deformation information relating to at least one of: bending, twisting, stretching, expansion, compression of the object.

[0017] The plurality of electrodes may be disposed laterally along the first surface of the deformable mass and along the second surface of the deformable mass.

[0018] The electrodes may define a sensing volume having a shape corresponding to at least a portion of a shape of the deformable object. The electrodes may be distributed about an exterior surface of the deformable object.

[0019] The electrodes may be distributed to cover at least a portion of the deformable mass. The electrodes may cover at least 50%, optionally 75%, optionally 90%. The electrodes may extend over substantially the entire outer surface of the deformable mass.

[0020] The plurality of selected electrode pairs can include at least one electrode pair of a first electrode and a non-adjacent electrode of the plurality of electrodes.

[0021] The plurality of electrodes may be operable to generate capacitive signal outputs from a plurality of selected electrode pairs according to a predetermined sequence.

[0022] The plurality of selected electrode pairs may include a subset, eg, a degenerate subset, of all electrode pairs in the plurality of electrodes.

[0023] The plurality of electrodes can include at least two electrodes arranged in a first plane and at least two electrodes arranged in a second plane, the first plane being substantially non-parallel to the second plane.

[0024] The electrodes can be disposed in or on one or more deformable substrates, the one or more deformable substrates being capable of mating with a surface upon deformation.

[0025] The one or more deformable substrates may be stretchable. The one or more deformable substrates may be stretchable in at least the lateral direction.

[0026] The one or more deformable substrates may be stretchable to increase or decrease the distance between two or more of the multiple electrodes.

[0027] The electrodes can form one or more sensor modules, each sensor module being continuously deformable from a planar configuration and / or mating with a surface. In a planar configuration, the electrodes can be arranged in one or more layers and / or in rows and / or columns. In a planar configuration, the electrodes can be arranged in a grid distribution.

[0028] The electrodes may comprise a stretchable conductive material.The sensing device may comprise a deformable connection between the electrodes.The deformable connection may comprise a stretchable conductive material.

[0029] The stretchable conductive material can include at least one of: carbon black, an elastomer, a conductive hydrogel, and / or a liquid metal.

[0030] The plurality of electrodes may comprise elongated conductor portions configured to elongate further in response to a force, and the elongated conductor portions of the plurality of electrodes may be disposed in a parallel arrangement.

[0031] The electrodes may be provided on one or more deformable substrates for attachment to the deformable mass, or may be integrated within such substrates.

[0032] One or more electrodes may be integrated into the surface of the deformable object.

[0033] The deformable object can include at least one of: a robotic arm; another robotic manipulator; a human body part; and / or a wearable object.

[0034] The plurality of electrodes may be distributed according to a predetermined layout, which may be determined using a machine learning generated process.

[0035] The sensing device may further comprise a processing resource configured to process the capacitance signal output, or capacitance data obtained from the capacitance output, to obtain said information related to the deformable object.

[0036] The processing circuitry can be configured to apply at least one predefined model to obtain the information, and the at least one predefined model can include a model trained using a machine learning generated process.

[0037] Obtaining the shape information may include at least one of: obtaining reconstructed shape data; obtaining a graphical representation of at least a portion of the deformable object; measuring one or more dimensions or other physical parameters of the deformable object; performing a shape reconstruction process. The reconstructed shape data may include high-resolution three-dimensional (3D) shape data of the entire geometric shape.

[0038] Obtaining the deformation information may include measuring at least one of: a magnitude of the deformation applied to the deformable object; a type of deformation applied to the deformable object.

[0039] Obtaining the force information can include: measuring at least one of a magnitude and / or a force acting on at least a portion of the deformable object. Obtaining the force information can include measuring a force on the deformable object from another object. Obtaining the force information can include determining touch information as a function of at least the change in dielectric constant.

[0040] Obtaining the velocity field information may include determining a velocity field map for the deformable object.

[0041] The sensing device may be configured to perform at least one of: a shape reconstruction process, and / or a deformation sensing (detection) process, and / or a force sensing process, and / or a deformation classification process, and / or a velocity field map generation process.

[0042] The sensing device may include an electrode drive module configured to selectively drive one or more of a plurality of electrodes to generate a capacitive signal output from a selected plurality of electrode pairs, and a capacitive signal readout module configured to read the generated capacitive signal output to generate capacitive signal data.

[0043] One or more of the selected electrode pairs may comprise two or more electrodes, and optionally three or more electrodes. The plurality of selected electrode pairs may comprise electrode pairs of a first group of one or more electrodes and a second group of one or more electrodes. The first group of electrodes may be operable to form a first combined electrode, and the second group of electrodes may be operable to form a second combined electrode. The first and second combined electrodes may be operable to generate a capacitive signal output.

[0044] According to a second aspect there is provided a sensing apparatus comprising the sensing device of the first aspect. The sensing apparatus may further comprise a display for displaying a visual representation of the shape of the deformable object.

[0045] The sensing device may further comprise at least one of: a processing resource configured to process the capacitance signal output, or capacitance data obtained from the capacitance signal output, to obtain the above information related to the deformable object; an electrode drive module configured to selectively drive one or more of the plurality of electrodes to generate a capacitance signal output from a selected plurality of electrode pairs. The sensing device may comprise a capacitance readout module configured to read out the generated capacitance signal output to generate capacitance signal data.

[0046] The sensing device can include signal routing circuitry operable to connect each of the plurality of electrodes to at least one of the drive circuitry and signal readout circuitry, the signal readout circuitry being controllable using one or more control signals.

[0047] According to a third aspect, which may be provided independently, there is provided a method, the method comprising: obtaining signal output data representative of a signal output from a plurality of selected electrode pairs of a plurality of electrodes distributed about one or more surfaces of a deformable object, the signal output generated at the electrode pairs being dependent on at least one of: a distance between the electrodes of the electrode pairs; a shape and / or orientation of the electrodes of the electrode pairs; and at least one material property of a material between the electrodes of the electrode pairs; and processing the signal output data to determine information related to the deformable object. The signal output may comprise a capacitance signal output, and the signal output data may comprise capacitance signal output data.

[0048] Processing the capacitance signal output data can include using at least one predefined model. The at least one model can be predefined using a machine learning generated process. The at least one predefined model can relate the capacitance signal for the plurality of electrodes to the information above.

[0049] The processing of the capacitive signal output may be performed as part of at least one of: a shape reconstruction process; a deformation detection process. The processing of the capacitive signal output may be performed as part of a touch detection process. The processing of the capacitive signal output may be performed as part of a simultaneous touch detection and deformation detection process.

[0050] The at least one model can be configured to output touch information and deformation information. The at least one model can include at least two models, including: a first model that outputs touch information and a second model that outputs deformation information. The touch information can include touch position information. The touch information can include contact position information.

[0051] According to a fourth aspect which may be provided independently, there is provided a method of training at least one model, the method comprising: acquiring training data, the training data including signal output data representative of signal outputs from a plurality of selected electrode pairs of a plurality of electrodes distributed about one or more planes of a deformable object; and additional data representative of information about the plurality of electrodes; and performing a model learning process using the acquired training data to acquire at least one trained model for acquiring additional information of interest about the plurality of electrodes using the additional acquired signal outputs. The signal outputs may include capacitance signal outputs, and the signal output data may represent the capacitance signal outputs. The additional signal outputs may include additional capacitance signal outputs.

[0052] The obtained capacitive signal output data may be obtained for one or more spatial arrangements of the plurality of electrodes and / or in response to one or more deformations applied to the deformable object, and the obtained information may include shape and / or deformation information data corresponding to said one or more spatial arrangements and / or said one or more applied deformations.

[0053] Obtaining the shape and / or deformation information data may include performing a sensing process on the deformable object while the deformable object is within each spatial location in the one or more spatial locations and / or in response to the one or more deformations.

[0054] Acquiring the shape and / or deformation information data may include acquiring image and / or depth sensor data of the deformable object when the deformable object is in multiple spatial configurations and / or in response to one or more deformations, and processing the image and / or depth sensor data.

[0055] The image and / or depth sensor data may include 3D shape representation data, for example point cloud data, and / or data representing additional derived parameters.

[0056] The obtained capacitive signal output data may be obtained in response to performing a sequence of touch actions at multiple locations on one or more surfaces. The obtained information may include touch location information.

[0057] According to a fifth aspect which may be provided independently, there is provided an apparatus comprising a processing resource configured to: obtain signal output data representative of a signal output from a plurality of selected electrode pairs of a plurality of electrodes distributed near one or more surfaces of the deformable object; and process the signal output data to determine information relating to the deformable object, wherein a signal output generated at the electrode pair depends on: a distance between the electrodes of the electrode pair; a shape and / or orientation of the electrodes; and at least one material property of a material between the electrodes of the electrode pair. The signal output may comprise a capacitance signal output, and the signal output data may comprise the capacitance signal output data.

[0058] According to a sixth aspect, which may be provided independently, there is provided an apparatus comprising a processing resource configured to: acquire training data; and execute a model learning process using the acquired training data to acquire at least one trained model for acquiring additional information of interest about a plurality of electrodes using additionally acquired signal outputs, the training data including: signal output data representative of signal outputs from a plurality of selected electrode pairs of a plurality of electrodes distributed near one or more surfaces of the deformable object; and additional data representative of information about the plurality of electrodes. The signal output may comprise a capacitance signal output, and the signal output data may represent the capacitance signal output. The additional signal output may comprise an additional capacitance signal output.

[0059] According to a seventh aspect, there is provided a non-transitory computer readable medium, the non-transitory computer readable medium comprising instructions operable by a processor to perform a method of the third or fourth aspect.

[0060] Features in one embodiment may be provided as features in any other embodiment, if necessary. For example, features of a device or apparatus may be provided as features of a method, and vice versa. Any one or more features in one embodiment may be combined with any suitable one or more features in any other embodiment.

[0061] Various aspects of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0062] [Figure 1] 1 is a schematic diagram of a sensor device according to an embodiment. [Diagram 2] 1A and 1B show a sensing device in an unassembled and assembled configuration. [Diagram 3] 13A and 13B are diagrams illustrating detection of capacitance between a pair of electrodes in close proximity and between a pair of electrodes not in close proximity. [Figure 4] FIG. 4(a) is a top view of the sensor module, FIG. 4(b) is a first cross-sectional view of the sensor module, and FIG. 4(c) is an exploded cross-sectional view of the sensor module. [Diagram 5] 1A and 1B are diagrams illustrating types of deformations that can be applied to a sensing device. [Figure 6] 1 is a flowchart outlining a method for acquiring deformation information using a trained model. [Figure 7] 1 is a flow chart outlining a method for training a model for obtaining deformation information. [Figure 8] FIG. 1 is a schematic diagram of a model architecture according to one embodiment. [Figure 9] 9(a) and 9(b) are diagrams illustrating the output of a trained model according to an embodiment. [Figure 10] FIG. 13 illustrates a multiple electrode array according to another embodiment. [Figure 11] FIG. 13 illustrates results from a neural network trained to use signal output to classify the type of deformation applied to an object, according to one embodiment. [Figure 12] FIG. 13 illustrates results from a neural network trained to use the signal output to determine the magnitude and direction of a force applied to an object, according to one embodiment. [Figure 13] 13A-13D show a sensing device in an unassembled and assembled configuration according to another embodiment. [Figure 14] FIG. 13 illustrates a sensing device according to another embodiment. [Figure 15] FIG. 15(a) is a top view of a sensor module according to another embodiment, FIG. 15(b) is a second top view of the sensor module, and FIG. 15(c) is an exploded cross-sectional view of the sensor module. [Figure 16] FIG. [Figure 17] FIG. 16 shows results obtained using the sensor module of FIG. 15. [Figure 18] FIG. 16 shows further results obtained using the sensor module of FIG. 15. [Figure 19] FIG. 16 is a diagram showing a touch area using the sensor module of FIG. [Figure 20] FIG. 1 is a schematic diagram of a model architecture according to one embodiment. [Figure 21] FIG. 1 shows a confusion matrix obtained during training of a neural network model. [Figure 22] 16A and 16B are diagrams showing other experimental results obtained using the sensor module of FIG. 15. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0063] Detailed Description The following embodiments relate to a sensing device for use with a deformable object, as a non-limiting example a robotic arm or robotic manipulator. The sensing device is configured to obtain a capacitance signal output, which can be processed to obtain information about or related to the deformable object. In the following, the described embodiments obtain, for example, shape and / or deformation information. However, it will be understood that other information (e.g. force information, velocity field information) can be derived from the capacitance signal output.

[0064] FIG 1 is a schematic diagram of a sensing apparatus 10 according to an embodiment. The sensing apparatus 10 comprises a sensing device 12, which comprises a number of electrodes 14, referred to as electrodes 14 for simplicity. The sensing apparatus 10 also comprises a driving module 16, a readout module 18, processing resources 20, memory resources 22, and a display 24. The electrodes 14 are configured to be distributed along one or more surfaces of a deformable object (not shown in FIG 1). The readout module may also be referred to as readout electronics and / or readout circuitry. Similarly, the driving module may also be referred to as drive electronics and / or drive circuitry.

[0065] A plurality of electrodes 14 are distributed on the exterior of a deformable object, e.g., a portion of a soft robot, such as a robot arm or robot manipulator. The electrodes may be distributed along, e.g., laterally and / or across, one or more surfaces of the deformable object. The plurality of electrodes may be considered to define a sensing volume that covers at least a portion of the shape of the object, and in some embodiments, substantially the entire object. Although an example of a robot arm or robot manipulator is described below, it will be understood that the device can be used at many different scales in many different applications.

[0066] The plurality of electrodes 14 has a distribution such that each electrode forms a close and non-close (also referred to as remote) electrode pair with other electrodes in the plurality of electrodes. The electrodes in such a distribution are thus operable to obtain capacitance readings from both the close and the remote electrode pairs. Close and remote electrode pairs are described in more detail with reference to FIG.

[0067] In general, every electrode pair in the plurality of electrodes can generate a capacitive output. For capacitances formed between electrodes of different electrode pairs, the capacitance is sensitive to the characteristics of the area between the electrodes.

[0068] The capacitance value of an electrode pair can be approximately represented by the following mathematical relationship:

number

[0069] Therefore, the capacitance value (C) between a pair of electrodes depends on the properties of the material between the two electrodes (in this case, the dielectric constant ε), the distance (d) between the two electrodes, and the surface area S of the electrodes. The surface area of ​​an electrode that is visible varies depending on the relative orientation of the electrodes. S in the above equation is a measure of the overlap area of ​​the two electrodes. Furthermore, the above equation is an approximation.

[0070] In response to deformation or, for example, changes in environmental conditions, the shape of the electrodes (corresponding to area S) may change, as may the distance and the dielectric constant (or other relevant material properties, e.g., conductivity), and the measured capacitance value will therefore change in value in response to any such changes.

[0071] The sensing strategy described below involves collecting a number of capacitance values ​​from a number of selected electrode pairs. The specific number and selection of capacitance values ​​to be collected can be referred to as the sensing strategy. The capacitance signal output can be processed to obtain information to be obtained about the deformed object (e.g., deformation information or shape information). In some embodiments, the information obtained is deformation information of the object collected over a 3D area, allowing a shape reconstruction process to be performed on the object. In other embodiments, the information obtained is force field or velocity field information. Since the capacitance output signal contains information related to the measurements mentioned above (e.g., shape, distance, material properties), in principle the capacitance signal output can be processed to obtain quantities that can be derived from these measurements.

[0072] The sensing apparatus 10 comprises electronic connections between the sensor device 12 and a drive / readout module configured to convey drive signals from the electrodes 14 to a readout module 18. According to an embodiment, each electrode has a corresponding electronic connection allowing individual electrodes to be addressed. Some of the electronic connections are provided as conductive links within the sensing device itself, as will be described with reference to Figures 2 and 4.

[0073] The processing resource is configured to combine capacitive signal outputs from both adjacent and non-adjacent electrode pairs of the plurality of electrodes and process these output signals to obtain shape and / or deformation information. The processing resource may perform a shape reconstruction or deformation detection process using the capacitive signal outputs. In other embodiments, the processing resource may derive one or more additional properties that depend on at least one of shape, distance, and material properties.

[0074] In such a shape reconstruction or deformation detection process, the obtained shape information can be, for example, a graphical (schematic) representation of a reconstructed 3D shape of at least a part of the deformable object. The shape information can be the dimensions of the deformable object. The obtained deformation information can be, for example, the magnitude of a certain force applied to the object or a certain deformation.

[0075] The processing resource 20 may be any suitable processing circuit. In some embodiments, the processing resource 20 may be, for example, a field programmable gate array (FPGA) or application-specific integrated circuit (ASIC) hardware. The processing resource may be an edge-based processing resource, for example, an NVIDIA Jetson Nano (registered trademark, development kit name).

[0076] In use, the drive module 16 drives the plurality of electrodes according to a sensing strategy to obtain capacitance signals from selected electrode pairs. The sensing strategy may target a subset of the electrodes. In some embodiments, a degenerate subset is targeted, e.g., the degenerate subset does not read capacitance readings from repetitive or degenerate electrode pairs. In response to the drive signal, the plurality of electrodes 14 generate capacitance signal outputs between the electrodes of the selected electrode pairs, including both close and distant electrode pairs. The capacitance output signals are then processed by the processing resources 20 to obtain, e.g., shape or deformation information about the deformable object. As described below, processing the capacitance output signals may include using a trained model to convert the capacitance signal output data into desired information (e.g., shape or deformation information).

[0077] 2 illustrates a sensing device 112 according to one embodiment. It will be understood that the sensing device 112 corresponds to the sensing device 12 and can operate as part of the sensing arrangement described with reference to FIG.

[0078] FIG. 2(a) illustrates the sensing device 112 in an unassembled configuration. In this embodiment, the unassembled configuration is an unfolded configuration, such that the electrodes 114 are in the form of a planar grid. In this embodiment, the grid corresponds to an 8×4 array of electrodes. As shown in FIG. 2(a), each electrode 114 has a corresponding conductive link 122. In this embodiment, the sensing device 112 is comprised of eight sensor modules, each of which is a four-electrode sensor module. For illustrative purposes, one sensor module 124 is shown in FIG. 2(a). Further description of the individual sensor modules 124 is provided with reference to FIG. 4.

[0079] As shown in Figure 2(a), each electrode is numbered. It is understood that each electrode has an address (or array number). Because each electrode has its own conductive link, the drive module can address one or more specific electrodes at a given time or according to a certain sequence or pattern, and the readout module can also attribute readouts to specific electrodes.

[0080] In this embodiment, the driving sequence includes adjacent and non-adjacent electrode pairs to ensure diversity of readings and expand the sensing field. In some embodiments, if the two electrodes are spaced too far apart, the capacitance value may be small. In such cases, the driving sequence may only use readings from electrode pairs in the same layer or across adjacent layers.

[0081] FIG. 2(b) shows the sensing device 112 in an assembled configuration, where the substrate of the sensing device is folded to be flush with the surface of the deformable object. In this embodiment, the deformable object is a robot arm. In the assembled configuration, a plurality of electrodes are distributed laterally along the four surfaces of the deformable object. In the flush configuration, the sensing device 112 has a second, third and fourth set of electrodes, which are distributed laterally along the first, second, third and fourth surfaces, respectively. When assembled and flush with the deformable object, the electrodes can be considered to form a sensing volume, which has a shape corresponding to the shape of the deformable object.

[0082] In this embodiment, a 32-electrode sensing device 122 is deployed on a mock-up robot arm and consists of eight 4-electrode modules. The capacitance values ​​are read out by an electrical circuit (readout module) at 30 fps (30 frames per second) (although other sample rates can be used). As will be described with reference to Figures 6 and 7, a neural network-based method can be employed to recover the 3D deformation from the capacitance signal readout.

[0083] 2 illustrates a foldable sensing device, it is understood that this is provided merely as a non-limiting example and that multiple electrodes can be distributed along one or more surfaces of objects having different shapes. By way of non-limiting example, the shapes can include a sphere, a serpentine shape, or other irregular shapes.

[0084] The embodiment of FIG. 2 illustrates that in some embodiments the sensing device can be modular and can consist of one or more deformable sensor modules that can be fixed to the surface of the object. The sensor modules can be thought of as forming a stretchable part of the skin or a patch for the object. However, it is understood that in other embodiments the electrodes themselves are integrated into a part of the object or are located on the outermost layer of the object. In other embodiments some electrodes can be integrated within the object itself and other electrodes can be provided as modules or separate layers added and fixed to the object.

[0085] 2 can be thought of as showing eight layers of electrodes across different modules (e.g., electrodes 1, 9, 17 and 25 are thought of as being in the same layer), which can also be referred to as rows. In some embodiments, the device is configured to obtain capacitance values ​​from electrodes in the same layer (e.g., between electrodes 12 and 20) or from electrodes in adjacent layers (e.g., electrodes 4 and 5).

[0086] The layers of the robot arm are shown in more detail in Figure 2(c). A 16-electrode capacitive sensor is shown on the left. Figure 2(c) shows four layers: layer 1, layer 2, layer 3, and layer 4. Each layer has four electrodes that together provide up to six capacitive readouts. A mesh representation of the robot arm is provided on the right. In this embodiment, the top of the robot arm (above layer 4) is fixed. The rest of the robot arm is free and moves in response to applied forces (which cause deformations).

[0087] As mentioned above, the plurality of electrodes are operable to generate a capacitive signal between the electrodes of a selected electrode pair. In Fig. 3, a deformable object 302, in this example a soft robotic arm, is shown. In the illustration of Fig. 3, four electrodes are shown: a first electrode 312a, a second electrode 312b, a third electrode 312c, and a fourth electrode 312d. The object 302 extends outward from a surface and has a proximal portion and a distal portion. The first and second electrodes (312a, 312b) are provided on the surface within the proximal portion and may be referred to as the proximal electrodes. The third and fourth electrodes (312c, 312d) are provided on the surface of the distal portion and may be referred to as the distal electrodes.

[0088] FIG. 3(a) shows two electrode pairs: a first electrode pair 304a (between a first electrode 312a and a second electrode 312b) and a second electrode pair 304b (between a third electrode 312c and a fourth electrode 312d). The first electrode pair 304a can be considered as a pair of two proximal electrodes, and the second electrode pair 304b can be considered as a pair of two distal electrodes. FIG. 3(b) shows two other pairs between electrodes: a third electrode pair 304c (between a first electrode 312a and a fourth electrode 312d) and a fourth electrode pair 304d (between a second electrode 312b and a third electrode 312c). Both the third electrode pair 304c and the fourth electrode pair 304d can be considered as pairs of proximal and distal electrodes.

[0089] Whether an electrode pair is considered proximal or distal can depend, for example, on the distance between the two electrodes. In some embodiments, it may not be possible to measure the capacitance between two electrodes if there is another electrode between them. However, if the electrodes are in adjacent layers, such as electrodes 1-2 in FIG. 2, a capacitance value can be returned if the electrodes are in the same layer or adjacent layers. In some embodiments, a proximal electrode pair can be considered as two adjacent electrodes. A distal electrode pair can be considered as a non-adjacent electrode pair, such as having an electrode between them. In some embodiments, only the capacitance formed by adjacent electrodes is measured.

[0090] The above embodiments provide examples of subsets of all possible electrode pairs that may be used. Some embodiments use capacitance outputs from a subset of all possible electrode pairs. Such outputs may include, for example, a degenerate subset of electrode pairs that includes only degenerate or non-repeating electrode pairs.

[0091] As shown in Figures 3(a) and 3(b), at least two of the electrode pairs have a common electrode. For example, the first electrode pair 304a and the third electrode pair 304c have a common electrode (the first electrode 312a). Similarly, for example, the first electrode pair 304a and the fourth electrode pair 304d have a common electrode (the second electrode 312b).

[0092] As used herein, proximal and distal refer to placement relative to the proximal and distal portions of the robotic arm that extends outward from the surface. However, for this object, and for other shapes of objects, selected electrode pairs are referenced in terms of their proximity or closeness to one another. Thus, the first electrode pair 304a may be referred to as a proximate electrode pair, and similarly, the second electrode pair 304b may be referred to as a proximate electrode pair. The third electrode pair 304c may be referred to as a non-proximate electrode pair, and similarly, the fourth electrode pair 304d may be referred to as a non-proximate electrode pair. Non-proximate electrode pairs may also be referred to as remote electrode pairs.

[0093] A proximal electrode pair for an electrode may include any electrode that falls within a predetermined region (e.g., area or volume). The predetermined region may define a neighborhood zone, whereby any electrode within the neighborhood zone is referred to as a neighboring electrode, and any electrode outside the neighborhood zone is referred to as a non-neighboring electrode. In this description, a proximal electrode pair may correspond to a collection of neighboring electrodes. A proximal electrode pair for an electrode may include, but is not limited to, the nearest neighboring electrodes.

[0094] As described above, the capacitance measured between a pair of electrodes depends at least on the distance between the two electrodes. Thus, during operation, a stronger capacitance signal is detected for the first electrode pair 304a and the second electrode pair 304b due to the electrodes of these electrode pairs being closer to each other than a weaker capacitance signal detected for the third electrode pair 304c and the fourth electrode pair 304d. In general, a larger capacitance value is measured for the close electrode pairs than for the distant electrode pairs.

[0095] It has been found that for the purpose of shape reconstruction and deformation sensing, when only strong signals between adjacent electrodes are used, a reliable shape may not be reconstructed. Similarly, when only weak signals between electrodes of a distant electrode pair are used, a reliable shape is not reconstructed. By combining the capacitance signal output from both the near and far electrode pairs of electrodes, as shown in FIG. 3(c), the model can reconstruct the exact geometry and shape. FIG. 3(c) shows a number of points 306 that collectively provide a reconstructed point cloud for the soft robot arm 302. Thus, it has been found that by receiving the capacitance signal output from the near and far electrode pairs and processing the output, local and global shape and / or deformation information can be obtained. As used herein, a point cloud is an example of a representation of shape information for an object. The point cloud depends on the number of electrodes.

[0096] The distribution of the electrodes can define a sensing volume that covers a certain percentage of the volume of the object. Similarly, the distribution of the electrodes can define a sensing area that extends over a certain percentage of the external surface area of ​​the object. For example, the percentage can be a portion of the outside of the deformable object. By way of non-limiting example, the percentage can be at least 50%, optionally 75%, and optionally 90% of the outside of the deformable object. In some embodiments, the electrodes can extend over substantially the entire outside of the deformable object. The distribution of the electrodes can be regularly spaced (so that the density of the electrodes varies across the surface) or unevenly spaced. The density of the distributed electrodes depends on the object being measured.

[0097] 4(a) is a top view of a four-electrode sensor module 124 used to form the sensing device 112. The four-electrode sensor module has a first electrode 114a, a second electrode 114b, a third electrode 114c, and a fourth electrode 114d. The sensor module is fabricated to be deformable under force, for example stretchable, twistable, and / or compressible.

[0098] Each electrode has a corresponding conductive link for linking the electrode to other electronic devices, such as a readout module and a drive module. The conductive link for an electrode can be referred to as an electrode link or simply a link. The link provides a conductive path between the electrode on a first portion of the sensor module and a connector on another portion of the sensor module. A first electrode link 122a, a second electrode link 122b, a third electrode link 122c, and a fourth electrode link 122d are shown in FIG. 4(a).

[0099] The electrode link is configured to convey drive signals from the drive module to the electrodes (activating selected electrodes). The electrode link is further configured to convey (to the readout module) capacitive readout signals from the electrodes.

[0100] In this embodiment, the electrode links of the sensor module are embedded in the layer structure of the sensor module. Each electrode link has a first and a second connector (also called a terminal) and a connection part between the first and second connectors. For clarity, the first connector 123, the second connector 126, and the connection part 125 are shown for the first link 122a in FIG. 4(a).

[0101] The links are arranged within the module according to a linking pattern such that the first connectors of the four links are aligned with a first end of the module and the second connector of each electrode link terminates at a respective electrode. The linking pattern is such that the connections of each link do not overlap or cross over one another. It will be appreciated that in forming part of the sensing device, other connections (e.g., wires or cabling or wireless capabilities) connect the electrode links to the readout and drive module, and in particular its connectors.

[0102] Fig. 4(b) shows a cross-sectional view of the electrode region of the sensor module 124. As shown in Fig. 4(b), the sensor module has a layered structure. In the electrode region, the sensor module has a sensing layer 142, an insulating layer 144, an electrode layer 146, and a protective substrate layer 148. These layers are provided on a base substrate 140. In this embodiment, the base substrate is made of silicone.

[0103] Channels for the electrode links are provided in the insulating layer 144. In this embodiment, these channels are microchannels and are formed in the insulating layer 144 using a laser etching process, where the channels are etched on the insulating layer by a laser machine.

[0104] FIG. 4(c) shows an alternative view of the layer structure of a portion of the sensor module. FIG. 4(c) shows electrode links 122 embedded within an insulating layer 144. FIG. 4(c) further shows electrodes 114 formed within the electrode layer. FIG. 4(c) also shows a protective substrate layer and an encapsulation layer. FIG. 4(c) also shows holes 150 formed within the insulating layer 144. The holes 150 are vertical interconnect holes. Each hole provides an opening for connecting an electrode layer to an electrode link.

[0105] The first and second connectors and the electrodes can be made of a conductive and deformable material, for example a stretchable conductive material. In this embodiment, the electrodes are carbon black (CB) dispersed elastomers. This material has been found to be less suitable for the connector and electrode link connections due to its high resistance and non-linear and irreversible conductivity response under deformation. In this embodiment, gallium 75.5% indium 24.5% eutectic (EGain) is used due to its conductive properties (3.4×10 7 Sm -1 ) and stable response to deformation, it is adopted for electrode links.

[0106] More specifically, the above four-electrode sensing module has dimensions of (20x20x120mm) and consists of four different functional layers: a protective substrate layer 148 (about 0.39mm thick), an electrode layer 146 (about 0.08mm), an insulating layer 144 (about 0.24mm thick), and a sensing layer (0.3mm thick). Microchannels (0.5mm wide) for wiring are etched, and connectors (3x2mm) on the insulating layer are formed by etching with a laser machine, after which the sensing layer is bonded onto the outward surface of the insulating layer. EGain ink is injected into the channels by a small syringe. The connection between the CB electrodes and the EGain wiring is ensured by vertical interconnect holes.

[0107] The above material and design parameter selection provides a relative capacitance response over a 40% strain range, ranging from 16% to 19%, depending on the active electrode pair. The response curves show excellent linearity and consistency over multiple cycles (over 500 cycles). The EGain wires are shown to provide superior performance over the comparable CB wires in terms of sensitivity (greater response under the same deformation), linearity (no distortion in the response curve), and cycle stability (no shift after 500 cycles of stretching).

[0108] While Figures 2 and 4 show sensing devices fabricated with sensing modules, it is understood that design parameters of the sensing devices and / or modules can be varied according to an embodiment. The design shown in Figure 2 balances reconfigurability performance with manufacturing complexity. The sensing module of Figure 4 provides patterning precision, repeatability, and stability using known elastomer processing techniques. These manufacturing techniques can be used to fabricate sensing modules in parallel.

[0109] FIG. 5 illustrates a set of representative deformations that can be detected using the sensing device. These deformations represent a non-limiting set of deformation states for the sensing device 112. FIG. 5(a) illustrates the sensing device 112 in an undeformed state. The undeformed form can be referred to as the natural state of the sensing device 112. FIG. 5(b) illustrates the sensing device in a bent state under the effect of a bending deformation. FIG. 5(c) illustrates the sensing device in a bent and twisted state after being subjected to bending and twisting deformations. FIG. 5(d) illustrates the sensing device being subjected to an elongation deformation (stretching), illustrating the elongated (stretched) or stretched state. FIG. 5(e) illustrates the sensing device being subjected to an elongation and twisting, illustrating the elongated and twisted state. It will be understood that these illustrations are non-limiting examples of deformation states and that the sensing device can be placed in many different deformation states under the effect of a force. For example, FIGS. 5(f) and (g) illustrate different bending and / or bending and twisting states for the sensing device. Apparently, the sensing device can generate and detect multi-mode deformations, as illustrated by Fig. 5. The detected deformations can include one or more of bending, twisting, elongation, expansion, compression, tension (stretching), and shear. It is understood that the type of deformation of the sensing device can be inferred.

[0110] 6 is a flow chart that generally describes a method for obtaining deformation information using a capacitance signal output from a sensing device. The method 600 uses a trained model. The trained model may also be referred to as a capacitance to deformation transformer (C2DT). The training of the model is described with reference to FIG. 7. It is understood that in some embodiments, more than one trained model may be used.

[0111] In step 602, a capacitive signal output is obtained. As described above, according to an embodiment, the capacitive signal output data is obtained using a sensing device. In step 604, a trained model is applied to the capacitive signal output data. In this embodiment, the capacitive signal output data is provided as an input to the trained model. In step 606, deformation information is obtained as an output from the trained model. Further details regarding a specific neural network implementation of the model are provided with reference to FIG. 8.

[0112] FIG. 7 is a flow chart that generally illustrates a method 700 for training a model for use, for example, in method 600 .

[0113] In step 702, capacitance signal output training data is acquired. As described in the embodiments, the capacitance signal output training data represents a capacitance signal output from the sensing device. The capacitance signal output data is acquired for one or more spatial arrangements of the plurality of electrodes and / or in response to one or more deformations applied to the deformable object. In step 704, deformation information training data is acquired. The deformation information training data corresponds to the acquired capacitance signal output data, and these two data sets together provide a training data set. Although steps 702 and 704 are described as two steps of method 700, these steps may also be considered as a single step of acquiring training data.

[0114] In this embodiment, the training data can be obtained using many different methods, for example by applying a test deformation to the sensing device and measuring the corresponding capacitance signal output data for the deformation, as described with reference to FIG. 1. Depth data of the sensing device is obtained using one or more depth sensing cameras placed above the sensing device while the sensing device is under the test deformation. In this embodiment, the depth data includes 3D point cloud data or can be processed to obtain 3D point cloud data. The depth data is processed to obtain the deformation information. In this embodiment, the deformation is captured using multiple depth cameras (which provide 3D point cloud data). The depth data is processed to clean it for use as ground truth in training.

[0115] The above process is then repeated for multiple test deformations to form a training data set including capacitance signal output training data and corresponding deformation information training data.

[0116] In step 706, a model training process is performed using the training data by one or more model training algorithms to train a model. For example, while many different training algorithms can be used to obtain a fully trained model for use in method 500 as a non-limiting example, the training process includes providing the capacitance signal output training data as an input to the model and performing a comparison between the output of the model and the deformation information training data. By iteratively providing the capacitance signal output training data to the model and comparing the output with the deformation information training data, the values ​​of the model weights are improved, thus training the model.

[0117] At step 708, the trained model is stored. The trained model is stored, for example, for use in method 700. Although training can be performed in conjunction with training of the sensing device on a separate processing resource, the trained model, or at least the trained weights, can be stored on memory resource 22 for use by the sensing apparatus.

[0118] 6 and 7 show generally how the model is trained and used to obtain deformation information, while FIG. 8 shows a more detailed model architecture for a neural network model according to one embodiment.

[0119] The model receives the capacitive readout data 704 as a first input and the source point cloud as a second input. The input source point cloud provided as input corresponds to the source point cloud for the undeformed sensing device. The input source point cloud thus contains spatial information corresponding, for example, to the shape and volume of the sensing device. The model is trained to obtain a reconstructed target point cloud based on the input, which is roughly the displacement of each point in the source point cloud (without any deformation) from the capacitive readout data.

[0120] The converter can be thought of as having three modules (also called layers): an encoder module 710; a decoder module 712; and a loss count module 716.

[0121] The encoder module receives the capacitance signal readout data as input. Features from the capacitance data are extracted based on a self-attention mechanism. These features are then fed to the decoder to "deform" the source point cloud. In the encoder, a neural network encodes the input capacitance readouts and geometric structure information of the electrode pairs into a high-dimensional space and feeds these to the transducer encoder to extract the proprioceptive information.

[0122] The decoder module receives as input the source point cloud data 702 and the output of the encoder. In use, the trained decoder module outputs reconstructed target point cloud data 708. At the decoder end, the network manages to assign the correct displacement to each point in the source point cloud based on the encoder output sequence.

[0123] The loss count module is used during model training. As part of the training process, the ground truth (also referred to as training data) of the target point cloud is provided to the loss count module. The model is trained using the loss count module 714. The loss count module 714 contains a loss function that is minimized during the training process. The loss function consists of a squared distance term for the visual markers and a chamfered distance term for the remaining points.

[0124] The neural network of FIG. 8 is a structure optimized for this particular problem, but it will be appreciated that other suitable network structures and models can be used.

[0125] In addition, it is understood that different trained models are used for different applications (e.g., to determine different types of additional information from the capacitance signal). Thus, for a particular application, the processing resource 20 can be configured to retrieve appropriate model information (e.g., model weights and / or other model data) from the memory resource 22. For example, for shape reconstruction purposes, a trained shape reconstruction model is obtained and used. As another example, to identify a type of deformation, a trained model is obtained that converts the capacitance signal output into a label corresponding to the type of deformation.

[0126] Figure 9(a) shows an example of a graphical representation of the output from the shape reconstruction process. These representations are generated from the detected capacitance values ​​using different trained models. The region of interest is the central part in the source point cloud. The reconstruction results of the trained models represent good high-quality reconstructions, capturing the range of complex deformations tested (measured by a number of performance metrics, namely: average distance (AD), maximal distance (MD), chamfer distance, and Hausdorff distance). The impact of the visual marker term in the loss function is analyzed as part of the analysis.

[0127] Fig. 9(a) shows simulated data, while Fig. 9(b) shows an example of real-world data. Fig. 9(b) shows different frames of image data and the corresponding ground truth and trained model (C2DT) outputs for three different types of deformations.

[0128] The above-described sensing devices and methods offer advantages over known methods, for example, being able to perform high-resolution shape reconstructions and / or precise measurements of dimensions.

[0129] In another embodiment, a high density electrode array can be used to meet the demands of real-world applications. Increasing the density of electrodes can result in increased wiring, data collection, and computational burdens. In some scenarios, such as large-scale tactile detection, a sparse (low density) electrode array may be more preferable. In another embodiment, a trained machine learning algorithm is used to determine the electrode layout. Figure 10 shows five different electrode layouts deployed on the surface of a soft rectangular object. The first three layouts are human designed (HD). The latter two layouts are randomly generated (RD) and designed by a machine learning algorithm using sensor layout optimization (SLO).

[0130] In the above embodiments, shape reconstruction and deformation sensing are described, where the trained neural network was, for example, a 3D point cloud or other spatial representation of an object. Figures 11 and 12 show results for another application of the sensing device, where a neural network is trained to receive capacitive signal output from multiple electrodes as input and provide a different type of output.

[0131] The results for deformation classification are shown in Fig. 11. Fig. 11 shows image data for two types of deformation: a first image 1102a for bending force applied to the object, and a second image 1102b for bending and twisting force applied to the object. Capacitive readouts are shown for both types of deformation: a first capacitive signal readout 1104a for bending force and a second capacitive signal readout 1104b for bending and twisting force. The x-axis shows the electrode layer identification (number) (see for example the layer structure in Fig. 2(c)). In this embodiment, there are four layers, each layer with four electrodes resulting in six capacitive readouts. The y-axis represents the calibrated capacitance measurement. Regarding parameter estimation, in this embodiment there is no need to deploy a dense array of electrodes.

[0132] 11 further illustrates the network output. In this embodiment, the network output is a probability score that the applied force includes a twist. For the first deformation (no twist), the probability is 0.005, and for the second deformation (including twist), the probability is 0.993. In some embodiments, a threshold on the output probability can be used to convert the probability to a binary value, or the network can be trained to output a binary value.

[0133] Results for the force estimation network are shown in Fig. 12. Fig. 12 shows a first image 1202a for a first bend applied to an object, and a second image 1202b for a second bend applied to the object. Fig. 12 also shows the corresponding output from the trained neural network. The trained neural network for this application was trained to receive as input the capacitance signal output from multiple electrodes, and output the force applied to the object. As can be observed from the results 1203a and 1204b, the estimated forces correlate closely with the ground truth measured forces.

[0134] In the following, experimental methods using the above-mentioned embodiments are described. The above-mentioned embodiments relate to a technology that provides a highly complex system that enables 3D proprioception through a new kind of inherently stretchable e-skin and advanced machine learning algorithms. The e-skin can enable boundary deformation capture throughout soft objects through its uniquely designed stretchable planar electrode array and sensing scheme. By utilizing e-skin signals and a custom-designed deep neural architecture based on a self-attention mechanism, it is demonstrated that the proprioception system can uniquely reconstruct complete 3D geometry under complex and diverse deformations in the form of a dense point cloud with accuracy (mm-scale error) comparable to that of an external commercial RGB-D (red, green, blue and depth) camera. This represents a step change for existing proprioception systems that only provide sparse geometric inference under constrained mode deformations. Proprioception technology can equip soft robots with the ability to perceive their own motion states as precisely as natural living beings, thus paving the way for the adoption of these robots in real-world scenarios ranging from biomedicine to human-robot interaction.

[0135] We describe a class of intrinsically stretchable capacitive e-skins (SCAS) that are embedded with planar electrode arrays to capture information, e.g., 3D proprioceptive information. SCAS is combined with a custom-designed deep net to reconstruct dense point clouds under complex and diverse deformations, which can be considered as one of the most challenging proprioceptive problems. The capacitance formed by the non-redundant combination of stretchable planar electrodes distributed on a 3D surface can characterize the boundary deformations and spatial electrical properties within the 3D region of interest. Compared to parallel electrode arrays, this electrode array can have a more concise structure and is easier to manufacture, miniaturize, and modularize. We select a mock-up robot arm (a square cylindrical silicone structure resembling a stylized soft robot manipulator) actuated by external forces as a testbed. This specific choice is motivated by the computational simplicity in the simulations and the need to test the maximum range of possible deformations, which are not feasible for an internally actuated system. However, the proposed method is in principle agnostic about the shape of the soft object under investigation since it does not require any prior geometric knowledge, making it generalizable to soft robotic platforms with a wide variety of geometric morphologies. The method is explored through coupled electrostatics and solid mechanics simulations and then transferred to a physical platform with appropriate electrode layout and network architecture based on the conclusions drawn from the simulation results.

[0136] In the coupled simulation, a SCAS consisting of 64 planar electrodes is deployed on a mock-up robot arm and various deformations (stretching, twisting, bending, and their combinations) caused by external forces are characterized by 392 measurable independent capacitance readouts in each measurement frame. A simulation data set is generated that contains 39,334 frames of different deformations (in point cloud format) and the corresponding capacitance readouts. This data set is used to evaluate the performance of the SCAS and the 3D deformation reconstruction method. The results from the simulation phase serve the purpose of guiding the design of the sensor and network architecture in the physical system.

[0137] The study of ablation is realized to better understand the role of each loss term and the positional coding. It is observed that without including visual markers in the training process, C2DT is unable to learn the correct point-to-point correspondences. When reconstruction is performed by C2DT without markers, points in the region of interest in the source point cloud may not be mapped to the correct corresponding region. By minimizing the chamfer distance term, the reconstruction can have similarity to the ground truth even if the point-to-point error remains large. It is also found that by retaining only the squared distance term of the visual markers during training, local distortions are introduced in the set of frames of the reconstruction. This indicates that the chamfer distance term helps the geometric quality of the reconstruction. Finally, poor convergence is observed when trying to train the network after removing the positional coding part. It is observed that electrode pairs with a high degree of geometric correlation tend to cluster together after positional coding.

[0138] The high density of SCAS markers and electrodes employed in a simulated environment poses practical challenges for fabrication and experimentation when applied to physical systems. Therefore, we investigated the impact of marker and electrode layout on the performance of C2DT with the ultimate aim of guiding the design and deployment of a functional real-world SCAS system. The results of this analysis show that the improvement in accuracy by increasing the number of markers reaches a plateau. This provides evidence that a small set of visual markers is sufficient for C2DT to establish correct point-to-point correspondence. Similarly, the reconstruction performance can be improved with the density of electrodes, but this improvement is very limited after the number of electrodes exceeds a certain value (e.g., more than 32). Thus, there appears to be a positive trade-off between reconstruction accuracy and electrode / marker unit, confirming that it is safe to sacrifice a small degradation in performance for a significant simplification of SCAS fabrication and deployment.

[0139] Eight 4-electrode SCAS modules are deployed on a mock-up robot arm with a size of 20 × 20 × 240 mm, which is one-fifth of the one considered in the simulation (an additional 40 mm height is the interface area). A 32-electrode SCAS consists of eight SCAS modules and is connected to an electrical capacitance tomography (ECT) system to extract the individual capacitance values. RGB-D cameras are used, positioned directly opposite each other, to capture the real-time ground-truth 3D deformation of the measured object as a color point cloud format from two complementary views and fuse them into one coordinate system. The sides of the mock-up robot arm are dyed white, since their original transparency can adversely affect the quality of the data collected by the RGN-D camera. Sixteen yellow visual markers are placed to encourage the network to learn the correct point-to-point correspondences during training. The entire experimental platform is capable of synchronously recording capacitance and point cloud data at approximately 30 fps.

[0140] The reliability of SCAS enabled us to record capacitance readout frames (each frame consisting of 76 independent readouts) as the mock-up robot arm was deformed by hand manipulation of the lower holder over an extended period of time. Approximately 1,220 s of deformation data were collected during the 10-hour experiment.

[0141] A random sequence of complex deformations was realized during the experiment, including bending, stretching, twisting, and loss combinations. In most frames, the point cloud collected by the camera may not represent the complete 3D deformation due to missing points caused by unseen visual occlusions. These points can be filled by shape reconstruction for frames with missing points problem and by directly filtering out frames with severe occlusion problem, after which 31,380 frames of data were acquired.

[0142] Further challenges for experimental deformation reconstruction are posed by the quality of the point cloud (constrained by camera accuracy, occlusion, and light conditions), noise in the SCAS signal, and imperfect synchronization between different devices. To compensate for these additional sources of inaccuracy in the experimental setup, the C2DT framework increases the number of input frames (Ni adjacent frames of SCAS readouts) and by introducing a regularization term in its loss function that can limit the amount of change in the distance between adjacent points before and after deformation. Several C2DTs have been trained with different numbers of input frames using filtered real-world datasets. It is found that the reconstruction performance improves as the number of input frames increases, achieving its minimum error for three adjacent input frames. This improvement can indicate that increasing the number of input frames can reduce the adverse effects of noise in the SCAS signal and synchronization between different devices. The linear correlation between adjacent frames can be considered to be beneficial for deformation reconstruction. The results achieve a level of quality comparable to that of ground truth point clouds collected by an external RGB-D camera.

[0143] Results from ablation studies show that visual markers play a similar role in both real and visual environments, helping the network learn the correct point-to-point correspondences. It is also observed that the addition of neighboring regularization terms slightly improves the quality of the reconstruction. The positional coding part is crucial for extracting useful proprioceptive information from physical SCAS signals. Similar to its contribution in simulations, positional coding can assign distinct high-dimensional representations to different electrode pairs based on the geometric structure of these electrode pairs.

[0144] The described proprioception system can capture real-time (30 fps) geometry of various complex deformations with a quality (mm-scale error) comparable to that of commercial RGB-D cameras. This can demonstrate a significant advantage over many previous attempts that mainly involve first-order simple proprioception scenarios. The system can also be made agnostic about the geometry of the measured object, and thus can be extended to many other kinds of soft robots by a straightforward learning process with the aid of the RGB-D camera. The performance demonstrated by this technique offers great promise in addressing some of the most complex challenges in soft robot control. In addition, the method of dynamic coupled domain simulation, which simultaneously includes the deformation of the sensor and the deformation of the soft robot, can be a powerful tool to facilitate automatic sensor design and optimization, for example, in fields such as digital twins of soft robots. The improvement of the SCAS system presented here can also enable integration with other sensor units. Such an integration method can enable diverse sensing of both proprioception and external stimuli to further align the performance of artificial systems with that of living organisms. Without limitation, further description of the simulation and experimental method is described below by embodiments.

[0145] As part of the experimental investigation, a coupled simulation is realized in COMSOL Multiphysics, a modeling software, to generate abundant capacitance and deformation data to demonstrate the effectiveness of the proposed method. The investigated object is a square cylindrical mock-up robot arm (length 271 mm, width 100 mm, height 1000 mm) made of silicone. An (8 × 8) electrode array with 64 electrodes is placed on the surface of the mock-up robot arm to form a 64-electrode SCAS. Each electrode is a flat surface of 105 × 30 mm and has no thickness. The distance between two adjacent electrodes on the same side is 20 mm in both horizontal and vertical directions. The distance from each edge to the nearest electrode is 10 mm. The relevant material properties are set as follows: Young's modulus E = 4.15 MPa, Poisson's ratio v = 275 / 0.022, and density ρ = 1.28 × 103 kgm -3 , relative dielectric constant Sr = 3.276.

[0146] The simulation realizes 956 different episodes, each of which mimics a time-continuous deformation process and is discretized into about 40 frames, during which the deformation and the corresponding capacitance readout from the SCAS are recorded.

[0147] Four different types of loads were applied to generate a variety of complex deformations: 1. Combined stretch and twist L(z,r), where a twisting force and a pulling force along the z-axis are simultaneously applied to the end of a mock-up robot arm; 2. Pure bending L(x,y), where a pulling force in the xy plane is applied to the end of the arm; 3. Two-stage twist and bend Le(x,y), where a twisting force is applied to the arm during the first r frames (r ranges from 6 to 16), and then a pulling force in the xy plane is applied to the end while maintaining the twisted state; 4. Combined twist and bend L(x,y,r), where a twisting force and a pulling force in the xy plane are simultaneously applied to the end of the arm. Each deformation is represented by a 3D point cloud of 1,716 points. Since it is impractical to determine the exact point-to-point correspondence of all points over all deformations in real-world conditions, we used scenarios that can be realistically realized in physical experiments. 64 points are selected as visual markers, whose correspondences are available during network training, and the remaining point correspondences are only used in testing for evaluation.

[0148] Theoretically, any two electrodes can form a capacitor. In FIG. 13, a sensing device with 64 electrodes is shown in assembled form (on the right) and in unassembled form (on the left). The SCAS with 64 electrodes (shown in FIG. 13) generates 2,016 independent capacitance readings during each measurement frame. However, many of these electrodes generate very weak signals (e.g., the capacitance between electrode 1 and electrode 64) due to the long distance between the electrodes. In a physical platform, such signals are difficult to detect, which leads to cases where these signals are ignored altogether. Therefore, we only record the capacitance of electrode pairs within the same layer and the capacitance of a specific electrode pair between two adjacent layers.

[0149] In particular, in this example, the 28 electrode pairs in the first layer form independent, measurable capacitors. The subset of electrode pairs includes the following independent (non-repeating) electrode pairs: electrode 1 forms an electrode pair with each electrode in the layer (9, 17, 25, 33, 41, 49, and 57) giving seven electrode pairs; electrode 9 forms additional electrode pairs with every electrode in the layer other than electrode 1 (17, 25, 33, 41, 49, 57). In this manner, it is understood that electrode 25 forms four additional electrode pairs (with electrodes 33, 41, 49, and 57); electrode 33 forms three additional electrode pairs (with electrodes 41, 49, 57); electrode 41 forms two additional electrode pairs (with electrodes 49 and 57); and electrode 49 forms one additional electrode pair (with electrode 57).

[0150] In particular, in this example, the 24 electrode pairs between the first and second layers form independent, measurable capacitors. The subset of electrode pairs includes the following independent (non-repeating) electrode pairs: electrode 1 forms an electrode pair with each electrode in the layer (9, 17, 25, 33, 41, 49, and 57) to give seven electrode pairs; electrode 9 forms additional electrode pairs with every electrode in the layer other than electrode 1 (17, 25, 33, 41, 49, 57). In this manner, it is understood that electrode 25 forms four additional electrode pairs (with electrodes 33, 41, 49, and 57); electrode 33 forms three additional electrode pairs (with electrodes 41, 49, 57); electrode 41 forms two additional electrode pairs (with electrodes 49 and 57); and electrode 49 forms one additional electrode pair (with electrode 57).

[0151] The 24 electrode pairs between the first and second layers form independent capacitances that can be measured. In total, the SCAS can generate 392 independent capacitance readings per measurement frame. For the first layer, each electrode forms three additional electrode pairs with adjacent electrodes in the second layer. For example, electrode 1 in layer 1 forms an electrode pair with adjacent electrodes 2, 10, and 58, and electrode 9 forms additional electrode pairs with adjacent electrodes 2, 10, and 18. It is understood that each electrode forms three additional electrode pairs.

[0152] In total, the SCAS can generate 392 independent capacitance readings per measurement frame. Each reading is calibrated as follows: c = (c' - c emp ) / c emp , where c is the calibrated capacitance reading, c' is the original reading, and c empis the readout without deformation. A total of 39,334 frames (956 episodes) of deformation and capacitance readouts were generated by the coupled simulation, of which 2,319 frames (53 episodes) are from deformation type 1, 12,562 frames (300 episodes) are from deformation type 2, 12,269 frames (303 episodes) are from deformation type 3, and 12,194 frames (300 episodes) are from deformation type 4.

[0153] Deformation recovery from SCAS sensing data is actually a sequence-sequence problem, mapping a sequence of capacitance readouts to a corresponding sequence of point coordinates (point cloud). As mentioned above, a capacitance-to-deformation transducer (C2DT) with a self-attention mechanism can be used to achieve dense 3D deformation reconstruction. In the following additional comments on the architecture, the realization and evaluation criteria of C2DT are described by an embodiment.

[0154] In general, C2DT uses a source point cloud P S The measurement characteristic tensor (c,Q e1 ,Q e2 ) to target point cloud (outside 1) Here is a deep model that can be transformed to reconstruct JPEG2025510756000003.jpg129. (outside 2) JPEG2025510756000004.jpg2464 is a point cloud without any deformation, p P S is the number of points in , which in this case is 1,719; (Outside 3) JPEG2025510756000005.jpg1139 is a reconstruction of a point cloud with a specific deformation, (outside 4) JPEG2025510756000006.jpg1939 is the corresponding calibrated capacitance readout; mis the number of read values ​​in c, which in this case has a value of 392; (outside 5) JPEG2025510756000007.jpg1539 and (outside 6) JPEG2025510756000008.jpg1639 is the coordinate of the electrode used to generate c.

[0155] The C2DT architecture mainly consists of two parts: encoding and decoding. The inputs of the encoding part are c, Q e1 , Q e2 Q e1 and Q e2 can be thought of as position signals that can help distinguish between different elements in c. These are the inputs to the multi-layer perceptron (MLP) f q The geometric representation of each electrode is obtained by passing the MLP f c maps c to a high-dimensional representation, and the capacitance representation together with the geometric representation is the input sequence of the transducer encoder E, and N m The length of the

[0156] For the decryption part, S First of all, MLP f S Then, the multi-head attention is fed to f S This is realized over the outputs of the MLP f S We use,to map the output sequence of,D(*),to the displacement of each point, and the reconstruction,P,is,P,. S is obtained by adding

[0157] P is the ground truth target point cloud (outer 7) The expected result is as close as possible to JPEG2025510756000009.jpg1813. This goal is achieved by minimizing the following loss function:

number

[0158] The structure of the C2DT subnetwork is as follows:

number

[0159] f q and f c The linear layer in does not have a learnable bias, whereas the other layers do. The LayerNorm in E takes the capacitance representation and the geometric representation together as input. The Transformer.EncoderLayer and Transformer.DecoderLayer are exactly the same as the original transformers. The first self-attention cell of the Transformer.DecoderLayer is removed, and the remaining part is used as a Transformer.MutualLayer, since P S remains constant. (Outside 11) JPEG2025510756000015.jpg17127 is n e-layer Represents a stack of Transformer.EncoderLayer s.

[0160] The simulation dataset is divided into three exclusive parts: a training set, a validation set, and a test set. The training set contains 22,517 frames (548 episodes), of which 1,334 frames (31 episodes) have a first type of variant; 7,204 frames (172 episodes) have a second type of variant; 6,980 frames (173 episodes) have a third type of variant; and 6,999 frames (172 episodes) have a fourth type of variant. The validation set contains 9,721 frames (236 episodes), of which 550 frames (12 episodes) have a first type of variant; 3,093 frames (74 episodes) have a second type of variant; 3,098 frames (76 episodes) have a third type of variant; and 2,980 frames (74 episodes) have a fourth type of variant. The test set contained 7,096 frames (172 episodes), of which 435 frames had the first type of variant; 2,255 frames (54 episodes) had the second type of variant; 2,191 frames (54 episodes) had the third type of variant; and 2,215 frames 3.5*(54 episodes) had the fourth type of variant.

[0161] C2DT is implemented in Pytorch. It uses the Adam optimizer (β1=0.9, β2=0.98, ε=10 -9 ), and update the learnable parameters to minimize the loss function. We use an initial learning rate of 0.001, which is decreased by a factor of 1.2 every 15 epochs. λ1 and λ2 are: λ1=λ / 3(λN ν +2N γ ), λ2=1 / 3(λN ν +2N γ ), where λ=max(1,300-2*(epoch-1)). max( ) represents the maximum value. Gradient is clipped at a threshold of 0.5 and the training set is used to train C2DT for 300 epochs with a batch size of 24. Each epoch takes about 9 minutes on three NVIDIA Quadro P500s. The network with the smallest validation loss is saved as the final model.

[0162] The performance of C2DT is evaluated by four error metrics: average distance (AD), maximum distance (MD), chamfer distance (CD), and Hausdorff distance (HD).

number

[0163] To understand the impact of each loss term and the position coding on the performance, an ablation study was realized. The squared distance term and the chamfer distance term, respectively, were removed and the same training procedure was performed to obtain the results of C2DT without markers and C2DT without chamfer distance. Attempts to train the network without the position coding part failed to converge. The position representation of the trained C2DT can be visualized by t-distributed stochastic neighbor embedding (t-SNE),29 which can help to discover the geometric correlation between different electrode pairs. The performance of C2DT with different network hyperparameters, different numbers of visual markers, and different electrode layouts was investigated using the same method to guide sensor and network design in the real world.

[0164] A 32-electrode SCAS consisting of 8 modules of 4-electrode SCAS with 4 different functional layers, namely a protection substrate, an electrode layer, an insulating layer, and an encapsulation layer, was fabricated according to the embodiment. Each module was fabricated layer by layer as follows.

[0165] Part A (1.0) and Part B (1.0) of i.Smooth-on Ecoflex® 00-30 were mixed and poured onto a glass plate. The silicone was plated using a TQC Sheen micrometered film applicator and cured in a 100°C oven for 3 minutes.

[0166] ii. Imerys Enasco® 250P conductive carbon black (0.2) was mixed with isopropyl alcohol (2.0), followed by the uncured silicone mixture (2.0) and stirred for 3 minutes. The uncured conductive silicone was coated onto a protective substrate and cured in an oven at 100° C. for 3 minutes.

[0167] iii. Pattern the CB electrodes using an AEON MIRA 5 laser etching and cutting machine at 40 W. The parameters are set as follows: 386 power at 28%, 300 mms -1 The speed was 0.05 mm and the spacing was 0.05 mm. The planar size of each electrode was 21 × 6 mm, which is one-fifth of that considered in the simulation.

[0168] iv. Using the same method as (i), a silicone film for the insulating layer was fabricated on the electrode layer.

[0169] v.2 round etching, 20.5 power, 300mms -1 The etching is performed at a speed of 1000 s and spacing of 0.05 mm to produce 380 microchannel liquid metal wiring and connections to the readout electronics. Four etching rounds are performed with the same parameters to produce the vertical interconnect holes. The planar size of the readout connections and vertical interconnect holes is 3 x 2 mm, and the width of the wiring is 0.5 mm. The rectangular area of ​​the modular SCAS is expanded at 19.5% power and 25 mms. -1 Cut (cut) at a speed of 0.05 to remove the remaining part.

[0170] iv. A new piece of silicone membrane is prepared according to step i and coated evenly on its surface with a very thin layer of uncured silicone mixture as adhesive. The SCAS cut and shaped in step v is then applied onto this membrane and any trapped bubbles are squeezed out manually. After about 4 hours of curing at room temperature, a good quality bond is obtained.

[0171] vii. Eutectic gallium 75.5% indium 24.5% (EGain, Sigma Aldrich) ink is injected through the readout connection by a syringe with a 0.33 mm needle, and the air in the microchannel is evacuated through the vertical interconnect hole. The injection point is then sealed with uncured silicone mixture.

[0172] viii. The final modular 4-electrode SCAS is obtained.

[0173] The planar size of the SCAS module is 120x20mm, of which 100x20mm is the area of ​​the electrodes, which is one-fifth of the area of ​​the comparison in the simulation, and 20x20mm is the interface area with the readout electronics. The layer thicknesses are 0.39mm, 0.08mm, 0.24mm, and 0.3mm, respectively. Since the manufacturing is easily scalable, five SCAS modules can be manufactured in parallel in this example.

[0174] A square cylinder mockup robot arm is molded with a size of 20x20x240mm, which is one-fifth of the size in the simulation. An additional 40mm height is the interface area used to boost deformation and coupling with the fixed ceiling. Eight 4-electrode SCAS modules are fixed on the surface to form a 32-electrode SCAS fixed-spacing system. The silicone layer is coated with white Smooth-on Silc Pig® silicone dye for better reflection. 16 yellow dots are attached as visual markers, which can assist the network learning with correspondence information, and the readout electronics interface is covered with black acrylic tape to reduce its interference in point cloud collection.

[0175] To characterize the response of the SCAS and demonstrate the superior performance of EGain wiring compared to CB wiring, a 4-electrode SCAS with CB wiring and a 4-electrode SCAS with EGain wiring are placed on the front and back of one segment of a square cylindrical silicone structure (20 × 20 × 140 mm), and they are cyclically stretched using a Nema23 stepper motor with an SFU1605 ball screw. Each cycle takes 20 s, and the SCAS is distorted by 40%. The whole test takes about 3 hours (more than 500 cycles). The relative capacitance readouts of each SCAS are compared, and it is shown that the SCAS with EGain wiring can have better sensitivity (larger response under the same deformation), linearity (no distortion in the response curve), and cyclic stability (no shift after 500 cycles of stretching).

[0176] The experiment uses an experimental platform consisting of a mock-up soft arm equipped with a 32-electrode SCAS, readout electronics, two Microsoft Azure Kinect® RGB-D cameras, and a laptop (computer) to control the readout electronics and record the data from the cameras and SCAS. The readout electronics is based on a 32-electrode ECT system that supports an arbitrary switching scheme. Its capacitance measurement resolution is 3fF (femtofarads) and the signal-to-noise ratio of all channels is better than 60dB.

[0177] Two cameras are positioned directly facing and in line with the mock-up robot arm to capture the real-time 3D deformation from two complementary fields of view. The deformation is stored and represented in a color point cloud format. The cameras and readout electronics record the data synchronously. The frame rate can reach about 30 fps when only recording point cloud and capacitance data, and drops to around 20 fps when recording RGB images simultaneously.

[0178] During real-world experiments, a handholder attached to the bottom of a mock-up robot arm is manually manipulated to produce complex deformations including stretching, twisting, bending, and combinations thereof. At the same time, the SCAS and cameras are synchronized to record data (capacitive readouts, color point clouds, and sometimes RGB images) under deformation. A total of 36,465 frames (~1,220 s) of data are collected, of which the first 36,013 frames (~1,200 s) record only capacitive readouts and color point clouds, and the last 452 frames (~20 s) store additional RGB images. The 32-electrode SCAS can generate 76 capacitive readouts in each frame, which are calibrated using the same method as in the simulation. The point clouds from the different cameras are fused in one coordinate system by a chessboard calibration method. The raw data is noisy and contains many meaningless background points, making the raw data unusable for learning purposes in this format. Using MATLAB® and its computer vision toolbox, these data are cleaned and preprocessed to retain only the points on the surface of the mock-up robot arm. Points on the black acrylic tape and the red holder are color filtered out. To further reduce the adverse effects of noise and other outliers, regions with local point density lower than a pre-set threshold are filtered out. Due to unavoidable visual occlusions that occur during the experiment, the cleaned point cloud cannot fully represent the 3D deformation in many frames. To overcome this issue, additional preprocessing is required before training. First, average grid downsampling with a 4 mm box grid filter is realized for computational efficiency. Then, the alpha shape is reconstructed based on the downsampled point cloud to alleviate the problem of incomplete representation. The triangle mesh of the alpha shape is subdivided three times and the vertices are extracted as a new point cloud with auxiliary points.In the C2DT framework, it is expected that the number of points in the source point cloud is the same as the number of points in the target point cloud. To meet this requirement, we implement average grid downsampling with a 4 mm box grid filter, and then use farthest point sampling to finally select 1,300 points in each point cloud. Before downsampling and shape reconstruction based on the RGB information of each point, we extract yellow visual markers from the cleaned point cloud. We create a one-frame graph of marker points. The connection of every two points in the graph is determined by the distance between these points. The connection distance threshold is 6 mm. Each connected subgraph with 11 or more points is considered as a visual marker, and the average value of the coordinates of all points in the subgraph is used to represent the marker position. The number of extracted markers is not necessarily 16 due to camera occlusion. The visual markers are aligned between layers. The 16 visual markers can be divided into four layers, with each layer containing four markers. The graph is based on one-frame coordinates of extracted markers with a connection distance threshold of 26 mm. Each subgraph is a layer of markers. The permutation of the layers is determined by the relative positions in the y-axis of the fused coordinate system among all four layers. We remove outlier frames where the number of extracted markers is greater than 16 and / or the number of layers is not equal to 4. Layers with fewer than four markers are filled with (0,0,0) to ensure that all layers have the same number of points, which can improve computational efficiency during training. Furthermore, we filter out frames with critical missing points issues due to poor quality of the reconstructed alpha-shape. The number of markers in a particular layer indicates the severity of missing points. Frames with at least two points in all layers are kept while other frames are rejected. Upon the filtering process described above, a total of 31,380 frames of data are still available for analysis. We performed a random inspection of 500 frames of the data set and found no samples with severe missing points issues.

[0179] The basic framework of C2DT in real-world experiments is similar to its counterpart in simulation. However, differences between real and virtual environments may require some modifications. First, the loss function in simulation is no longer applicable as in the experimental conditions described above, since there is no point-to-point correspondence of visual markers. Instead, we use a new loss function similar to the following:

number

[0180] The first term of this loss function counts the chamfer distance between the marker reconstruction and the marker ground truth for each layer, where (Outside 12) JPEG2025510756000018.jpg3689 is k are the coordinates of the visual markers in the layer; (Outside 13) JPEG2025510756000019.jpg5089 is P lk are the coordinates of the i-th point in ; (Outside 14) JPEG2025510756000020.jpg4689 is (Outside 15) JPEG2025510756000021.jpg1513 and P lk is the squared distance between the nearest point in l is the number of layers, N lν is the number of markers in each stratum; N l and N lν The value of is 4 in this case. When calculating the loss, we only consider the marker points extracted during data processing and ignore the padding points. (Outside 16) All points in JPEG2025510756000022.jpg1313 are marker points because they are generated by the network based on the corresponding capacitance readings and source points that do not contain padding points. To eliminate the effect of padding points during training, we synthesize a mask as follows:

number

number

[0181] The number of input frames in the physical world is not constant and is 1. In contrast, C2DT uses some (N i It takes as input the next 2 adjacent frames. Thus, the first linear cell in fc is Linear(N i ,h em ) The hyper parameters of C2DT are set as follows: em =32, d model =64, d ff =128, h=4, p drop = 0.1, n e-layer= 2, n m-layer =1, n d-layer =1, and a=1.9472.

[0182] We scale up the point cloud data values ​​five times to be closer to the simulation scale. We train and evaluate the network using almost the same procedure presented above. We split the real-world dataset into three exclusive parts. The first 26,771 frames (~1,020 s) are used for training (20,693 frames) and validation (6,018 frames), and the last 4,660 frames (~200 s) are used for testing. δ d is set equal to 0.5, and δ u is set equal to 2. λ1, λ2 are calculated as follows:

number

[0183] In the above-described embodiment, obtaining measurements for the selected electrode pairs includes activating a single electrode of the electrode pair to measure the capacitance at a corresponding single electrode of the electrode pair. In another embodiment, one or more of the selected electrode pairs can include three or more electrodes. As a non-limiting example, such electrode pairs can include an electrode pair of a first group of electrodes and a second group of electrodes. In operation, the electrodes of the first group are simultaneously activated to form a combination electrode, and the electrodes of the second group are also combined to form a corresponding combination electrode for measuring the capacitance. Such multi-electrode sensing strategies involving electrode pairs of three or more electrodes are described in more detail in the context of electrical capacitance tomography in "A novel multi-electrode sensing strategy for electrical capacitance tomography with ultra-low dynamic range" by Yang et al. (Non-Patent Document 1).

[0184] FIG. 14 illustrates a sensing apparatus according to another related embodiment, showing the combination of groups of electrodes to form electrode pairs with a first group of electrodes and a second group of electrodes. FIG. 14 illustrates an excitation circuit 1402 and a measurement circuit 1404, where the excitation circuit 1402 is configured to apply an excitation signal to a selected electrode group. FIG. 14 also illustrates the sensing device according to the embodiment in an unassembled configuration (1406) and an assembled configuration (1408). The excitation circuit 1402 can be considered to form part of a drive circuit, and the measurement circuit 1406 can be considered to form part of a signal readout circuit.

[0185] FIG. 14 shows an equivalent capacitor, which is a linear combination of the series of capacitors formed by two individual electrodes. FIG. 14 shows two electrodes (24 and 32) combined together to form a combined measurement electrode. More specifically, both electrodes are connected to the same terminal in a measurement circuit, which allows the capacitance of the combined electrodes (24 and 32) to be measured by the measurement circuit. In FIG. 14, the capacitance formed by electrode 8 in this example and electrodes 24 and 32 in the combination is equal to the capacitance between electrodes 8 and 24 plus the capacitance between electrodes 8 and 32, following the principle of linear superposition of the equation: C 8,24-32 =C 8,24 +C 8,32

[0186] In another embodiment, a 32-electrode multiplexer array is provided to allow appropriate connections to be made between the electrodes, the signal readout circuitry, and the signal driver circuitry to form the desired electrode group. In this embodiment, the 32-electrode multiplexer array allows each electrode to be connected to one of an excitation circuit (to receive an excitation signal), a measurement terminal (of a measurement circuit), or a ground terminal. The 32-electrode multiplexer array can be controlled by a control signal. In response to receiving the control signal, the 32-electrode multiplexer array allows the electrodes to be connected to one of the excitation circuit, the measurement circuit, and the ground terminal.

[0187] More specifically, the multiplexer can be controlled to simultaneously excite a group of multiple electrodes by simultaneously connecting each of the multiple electrodes to an excitation source. Similarly, the multiplexer can be controlled to simultaneously connect another group of multiple electrodes to a measurement terminal, forming a combination of measurement electrodes. The multiplexer is further controlled to connect all other electrodes (all electrodes not in the first and second groups) to a ground terminal. Although a 32-electrode multiplexer array is described, it will be appreciated that other suitable signal routing circuitry can be used.

[0188] In such an embodiment, a specific combination electrode strategy can be designed and used that can include sensing control signals to the electrodes using a 32-electrode multiplexer array or other signal routing circuitry. The control signals can include a series of digital signals, such as a control word for controlling the 32-electrode multiplexer array.

[0189] Those skilled in the art will appreciate that variations in the configurations contained herein are possible without departing from the invention.

[0190] For example, while in the embodiments described above the electrodes are described as being distributed near the surface of the object, it will be understood that in other embodiments one or more electrodes can be embedded within the object or at a depth below the surface of the object.

[0191] In the embodiment described above, the number of electrodes is 32. However, it is understood that this number is not fixed. In practice, the number of electrodes for a selected object may depend on the complexity of the object's shape, or the size of the object (e.g., larger objects may be implemented with a larger number of electrodes).

[0192] In addition, although Figure 1 illustrates electrodes as part of the sensing device, it is understood that other elements of the sensing apparatus illustrated in Figure 1 may form part of the sensing device (e.g., a drive module and / or a readout module may form part of the sensing device). In some embodiments, the sensing device may include a drive module, electrodes, a readout module, processing resources, and memory resources.

[0193] Further, in the above embodiments, sensing of capacitive signals is described. However, it is understood that in alternative embodiments, other types of electrical signal outputs can be detected. For example, the signal output can include voltage and / or potential difference. In particular, the sensing device measures voltage. In some embodiments, the electrodes measure voltages having a size (e.g., amplitude) that has a linear relationship with the capacitance being measured. In alternative embodiments, the sensing device can be extended to measure impedance. In some embodiments, the signals are all in the form of voltages and have a sine / cosine form with a particular phase and / or amplitude, similar to modulated signals.

[0194] In the above embodiment, the learning-based method is described for complete geometric reconstruction. Other methods can be used, but very strong constraints and prior information are essential, and the above models are usually applicable to specific objects and lack the ability to generalize. The established learning pipeline makes the learning-based method practical and allows it to generalize to different scenarios.

[0195] In addition to or instead of detecting deformation and shape information, the sensing device can be capable of detecting, for example, electrical properties and touch. For example, as described above, due to the nature of capacitive electrodes, the capacitance value is sensitive to changes in the dielectric constant at a close distance near the surface of the skin. Thus, an object approaching the skin can cause a change in the dielectric constant and thus affect the capacitance value. This property can be used to sense touch detection, or, for example, collision detection.

[0196] Another embodiment is described below. The capacitance between two boundary electrodes can be expressed by the following formula:

number

[0197] FIG. 15(a) illustrates a sensor module (also referred to as an e-skin module) according to another embodiment. The sensor module of FIG. 15 has four electrodes: a first electrode 1514a, a second electrode 1514b, a third electrode 1514c, and a fourth electrode 1514d. The sensor module is fabricated to be deformable, e.g., stretchable, twistable, and / or compressible, under force. In particular, FIG. 15(a) illustrates the module in a first unstretched and undeformed configuration, and FIG. 15(b) illustrates the module in a second stretched or elongated configuration. In contrast to the sensor module of FIG. 4, which shows electrodes, connectors, and connector links disposed in a material, in the embodiment of FIG. 15, the electrodes are liquid metal wires that are significantly elongated along a first direction.

[0198] In the embodiment of FIG. 15(a), electrodes are provided in a parallel arrangement, each electrode having a conductive portion extending along the longitudinal axis of the sensor. FIG. 15(a) also shows connectors (1522a, 1522b, 1522c, 1522d) for the electrodes. The connectors provide an interface between the liquid metal electrodes and the respective readout metal wires. Readout metal wire 1523 is shown for the fourth electrode 1514d and connector 1522d. In this embodiment, the electrodes are liquid metal wires and are provided with connectors (also referred to as interfaces). In this embodiment, the size of the sensor module in the first unstretched form is 400 mm wide and 1100 mm long. The width of each electrode is 1 mm, and the lengths are 20, 45, 70, and 95 mm, respectively. Each of the liquid metal interfaces has an area of ​​5×5 mm.

[0199] The four liquid metal wires provide elongated conductive portions operable as electrodes, the combination of which may form a capacitor, as described above. It will be appreciated that the sensor module may be combined with one or more other sensor modules to assemble a sensing device, for example, substantially as described with reference to the embodiment of FIG.

[0200] As can be seen in Figure 15(a), each of the four electrodes of the module has a different length and these electrodes are arranged in a parallel arrangement, in this embodiment, the length of each electrode is successively longer than the previous electrode. It is understood that each of the electrodes in Fig. 15(a) has a conductive portion and is configured to perform a capacitive readout substantially along its length. The different lengths of each individual electrode can provide advantages for touch sensing, for example, because the difference in length can increase the difference in the detected capacitive signal when different areas of the module or sensing device are touched.

[0201] As can be seen in Fig. 15(b), the sensor module is configured to deform and stretch at least in the longitudinal direction. The electrodes are arranged substantially parallel to the longitudinal direction. Stretching of the sensor module causes the lengthening or further elongation of each electrode. In a first, undeformed configuration, each electrode has a first length, and in a second, deformed configuration, the electrodes have a second, longer length.

[0202] Figure 15(c) shows the layer structure of the sensor module of Figure 15. Figure 16 shows a sensor module having two layers: a substrate layer 1546 and a protective layer 1542. Electrodes 1514a, 1514b, 1514c, 1514d and their corresponding connectors and interfaces are provided in the layer structure 1546. Figure 15(c) shows the electrode layer and protective substrate layer together. Both the electrodes and the interfaces are provided in the electrode layer.

[0203] The electrode layer includes a substrate formed of platinum-catalyzed silicone. In this embodiment, the platinum-catalyzed silicone is Ecoflex® 00-30 silicone. Microchannels are fabricated in the substrate layer 1546 by 3D printing and casting. Once the substrate is formed, a protective layer 1542 is fabricated by film coating from a silicone membrane and bonded to the substrate layer 1546 using uncured silicone as an adhesive. Liquid metal ink is injected into the formed microchannels to form the sensing electrodes. In this embodiment, the liquid metal ink is a eutectic 75.5% gallium, 24.5% indium (EGain) ink and is injected through the interface. Air is exhausted through the ends of the lines. In contrast to the embodiment described with reference to FIG. 4(c), no carbon black electrodes are fabricated, and therefore the number of layers is reduced to two. As can be seen in FIG. 15(b), the elongated conductive parts are embedded in the substrate, which allows the conductive parts to deform together with the substrate.

[0204] Figure 16 shows a three-chamber pneumatic manipulator (size 500x1200mm) used as a testbed to validate the proposed method. Pneumatic soft robots are frequently used in many applications and can impart different deformations, e.g., swelling and bending. The structure of the manipulator is shown in Figure 16. The manipulator has three chambers, each measuring 400x300mm, and each chamber acts as an air inlet to allow air to be injected into each chamber. The width of each inlet is 1.5mm. The two sensor modules described with reference to Figure 15 are bonded to the surface of the external manipulator (one sensor module on the front side and the second sensor module on the back side) using uncured silicone as adhesive. The two sensor modules together form an 8-electrode capacitive sensor, since each module has four electrodes and can generate 28 capacitive readings per measurement frame.

[0205] Non-limiting experimental methods and results are described below. In the following, a three-dimensional vector p=(p1, p2, p3) is used to describe the inflation state of the robot, where p1, p2, and p3 are the volumes of air injected into the first, second, and third chambers, respectively. In this experiment, 20 ml of air is injected into each chamber of the robot simultaneously (i.e., p=(20,20,20)) and the sensor response signals are recorded.

[0206] The results of measurements performed during such inflation of the manipulator are shown in Figure 17. Figure 17(a) shows all 28 capacitance readings during the inflation process. The y-axis is the calibrated capacitance C (i.e. the relative change in capacitance), which can be calculated by the following formula:

number

[0207] FIG. 18 shows the results from a two-stage experiment, illustrating the difference in capacitance change in response to touch and deformation. The capacitance change in response to touch may be dominated by the change in dielectric constant. The capacitance change in response to deformation may correspond to the change in geometry. The touch induces a change in dielectric constant, which leads to a further change in signal according to the capacitance formula. The same set of electrodes detects both touch and deformation, but different signal patterns are detected. Touch can be referred to as the application of a localized contact force on the surface of the sensor.

[0208] In the first stage, the robot is inflated to (0,20,20) ml without any additional touches. Next, the front of the robot is divided into nine sections (see FIG. 19) and each section is touched individually to form touch measurements. Each touch can be referred to as a touch event or touch action, and is performed at the touch location. FIG. 19 shows the location of the surface of the robot. In use, touching one of the areas of the surface results in identification of the touch location on the surface. For example, a touch within the area marked with a 1 returns the label "1" from the trained model. It is understood that more than one touch event or action at approximately the same time can be detected. Although FIG. 19 shows nine touch areas, it is understood that a larger number of electrodes can be used to provide greater discriminability and a larger number of touch areas.

[0209] The overall response of the sensor is shown in Fig. 18(a). The capacitance response of a (0.20.20) ml bulge is similar to that of a (20,20,20) ml bulge shown in Fig. 17(a). A bulge can trigger a variation in all capacitance readouts simultaneously (global response), whereas a touch promotes a change in only a portion of the readouts based on the location of the touch point (local response). Examples of capacitance readouts from four different electrode pairs are shown in Fig. 17(b). Fig. 17(b) shows that different electrode pairs have different sensory fields. The capacitance readouts reflect touches only within their own sensory field and are insensitive to touches outside that area. This feature gives rise to different patterns in the capacitance signals induced by the bulge and the touch. Small variations are also observed in the capacitance readouts during touch. These small variations are induced by deformations (e.g., bending) of the robot body caused by the touch / contact.

[0210] Another experiment is described below, in which data is collected to verify the feasibility of recognizing touch during deformation (e.g., bulging) using the proposed flexible sensor module. Ideally, the location of the touch point and the sensor signal would be determined simultaneously. However, it proves to be a great challenge to identify the exact touch (or contact) location as the robot moves and deforms during the experiment process. Therefore, we divide the surface of the robot into 18 sub-regions (see Figure 19). Then, we touch the sub-regions randomly during the experiment and record the response signals. The sub-region landmarks (numbers) reflect the coarse location of the touch point.

[0211] Furthermore, since deformation tracking is an important topic, we investigate the possibility of extracting deformation information from the sensor signals, and in soft robots, a sensing device with multiple functions is desired. For pneumatic robotic platforms, the bulge information is usually known, since the volume of air injected into the chamber can be controlled. Therefore, we investigate the deformations caused by user interaction, such as touch-induced bending. To obtain deformation labels during the experiment, five reflective visual markers are coupled to the sides of the robot. Then, OptiTrack® Flex 13 cameras are deployed around the robot to capture the real-time 3D coordinates of the markers. These coordinates provide a concise description of the deformation and are used as deformation labels in the concise description of the deformation. The experimental platform includes a pneumatic robot with an 8-electrode capacitive skin as described above, together with five reflective visual markers, and uses OptiTrack® Flex cameras and readout electronics, which can reach a measurement resolution of 3 fF and a signal-to-noise ratio of more than 60 dB for all measurement channels. The data recording rate in the camera and readout electronics is set to 30 fps.

[0212] The experiments were performed under 27 different inflation conditions ranging from (0,0,0) ml to (20,20,20) ml. For each inflation condition, data is collected from seven separate periods. Each period lasts for 30 seconds. During the first period, the robot is allowed to inflate to a pre-set state without touch. During subsequent separate periods, small areas of the robot are randomly touched to induce deformation due to contact forces. A series of contact actions are performed at multiple locations on one or more surfaces of the sensing device to record location information for the touch points.

[0213] For touch recognition, we acquire 189 (27×7) groups of different bulging states. Each group of data contains 30 seconds (i.e., 900 frames since the sensing rate is 30 fps) of capacitance signals and corresponding touch point landmarks. These data are exclusively divided into a training set (125×30=3750 seconds, 3750×30=112500 frames), a validation set (32×30=960 seconds, 960×30=28800 frames), and a test set (32×30=960 seconds, 960×30=28800 frames). For deformation tracking, one group of data consists of 30 seconds of capacitance signals and the trajectory of visual markers recorded by the camera. Data without touch and with occlusion problems are manually filtered out. 146 groups of data are obtained and exclusively divided into a training set (108×30=3240 s, 3240×30=97200 frames), a validation set (18×30=540 s, 540×30=16200 frames), and a test set (20×30=600 s, 600×30=18000 frames).

[0214] For deformation tracking, one group of data consists of 30 seconds of capacitance signals and visual marker trajectories recorded by the camera. Data with no touch and occlusion issues are manually filtered out. 146 groups of data are obtained and these data are exclusively divided into training set (108×30=3240 seconds, 3240×30=97200 frames), validation set (18×30=540 seconds, 540×30=16200 frames), and test set (20×30=600 seconds, 600×30=18000 frames).

[0215] Figure 20 is a schematic diagram of the neural network architecture adopted for touch point classification. This neural network architecture is sometimes called multi-layer perceptron (MLP). In this experiment, 19 different classes corresponding to 18 different touch points and one case of no touch are used in this study. The input of the MLP is the 28 calibrated capacitance readings in one frame. The output of the MLP is a vector with a size of 19, indicating the probability of the classes. A cross-entropy based loss function is used. The MLP has one hidden layer with 128 neurons. The activation function for the hidden layer is ReLU (rectified linear unit / function). A dropout probability (p=0.1) is used to prevent overfitting. The output of the neural network is the location of the touch or touch area.

[0216] Training of the MLP is realized in Pytorch®. The Adam optimizer is used to update the learnable parameters to minimize the cross entropy loss between the predicted and ground truth touch points. The initial learning rate is set to 0.001 and is decreased by a factor of 1.2 every 15 epochs. Using the training set, 100 epochs of training are used with a batch size of 256, and the network with the smallest loss is saved on the validation set. The training process takes 10 minutes on an NVIDIA Quadro® P5000 GPU (graphics processing unit) card.

[0217] Following the training process, the MLP demonstrated a classification accuracy of 99.88% on the test set. The MLP demonstrates that the proposed flexible skin can estimate touch points using a simple deep learning model, even when the signal is severely disturbed by the bulging of the robot body. The confusion matrix of the classification results is shown in Figure 21. The confusion matrix shows that only 34 frames out of 28,800 frames of the test sample are misclassified. All misclassifications occur between adjacent subregions. For example, 34 touches on subregion 2 are misclassified as subregion 3. This is because the signals induced by touches on adjacent subregions have a relatively high similarity, which can confuse the network.

[0218] Although the trained model for outputting touch position information has been described with reference to the sensor module of FIG. 15, it will be appreciated that such a model can be trained using a sensor module of another embodiment, such as the sensor module of FIG. 4.

[0219] Turning to deformation tracking, estimating the coordinates of visual markers based on capacitance signals can be treated as a set-to-set problem. We use the Capacitance-to-Deformation Transducer (C2DT) described above. C2DT is a transducer-based architecture developed to reconstruct point clouds of soft robots. The structure of C2DT is described with reference to Figure 8.

[0220] The cameras and readout electronics are synchronized by Auto Click Script (a software that automates mouse clicks), which introduces a small amount of delay between the data recorded by different devices. Therefore, 10 frames of calibrated capacitance signals are input to the C2DT to mitigate this problem. The position signal consists of the position information of the electrode pairs to form a capacitor, which can help the C2DT to distinguish the capacitance readouts generated by different electrode pairs. The squared error between the estimated value and the ground truth is selected as the loss function.

[0221] The above description focuses on deformations caused by user interaction (e.g., touch) rather than bulges (which are known in most scenarios since the volume of air injected into the chamber is controllable). However, the signals induced by these deformations are much smaller than those triggered by bulges. Directly estimating the coordinates of visual markers can be challenging. To address this problem, we use the first frame in each trajectory as a priori knowledge, i.e., we use the capacitance readout as a reference to calibrate the capacitive input, and use the marker coordinates as the source sequence for the transducer decoder.

[0222] Training of the MLP is realized in Pytorch. We use the Adam optimizer to update the learnable parameters to minimize the squared loss between the predicted and ground truth coordinates. The initial learning rate is 0.001 and is decreased by a factor of 1.2 every 15 epochs. We train the network on the training set for 150 epochs with a batch size of 255 and save the network with the smallest loss on the validation set. The training process takes 2.5 hours on an NVIDIA Quadro P5000 GPU card.

[0223] The performance of the modified C2DT is evaluated using the average distance between the estimated and ground truth coordinates of the markers, where N is the number of samples in the test set, M is the number of visual markers, pid is the ground truth coordinate of the i-th visual marker for the i-th test sample, and pid is the estimated coordinate of the i-th visual marker for the i-th test sample. After training, the C2DT can achieve an AD error of 2.905 ± 2.207 mm. This demonstrates that with a priori knowledge (capacitive signals and marker coordinates in the first frame of each trajectory), the proposed skin can be applied to track deformations using the C2DT even in environments with severe disturbances (bulging and permittivity variations caused by touch). Some examples of tracking results are shown in Figure 22. In all cases, it is observed that the estimated visual markers (red) and the ground truth visual markers (blue) are close, indicating high accuracy of deformation tracking.

[0224] The ground truth markers (e.g., marker 2202) represent the positions of physical markers attached to the manipulator, which are captured by a tracking camera. The blue ground truth represents the coordinates of the visual markers collected by the camera. The red estimates (e.g., marker 2204) represent the output of the network. Figure 22 shows the overlap between the network output and the physical marker positions for all points.

[0225] Thus, the description of specific embodiments of anomalies is provided by way of example only and not for purposes of limitation, as those skilled in the art will appreciate that minor modifications can be made without significantly altering the operation described.

Claims

1. 1. A sensing device comprising a plurality of electrodes configured to be distributed near one or more surfaces of a deformable object, the plurality of electrodes is operable to generate a capacitive signal output from a plurality of selected electrode pairs in the plurality of electrodes, the plurality of electrodes being distributed about the one or more surfaces of the deformable body such that the plurality of selected electrode pairs comprises at least one pair of adjacent electrodes and at least one pair of non-adjacent electrodes; A sensing device in which the capacitive signal output generated in an electrode pair in the plurality of electrodes depends on at least one of the distance between the electrodes of the electrode pair, the shape and / or orientation of the electrodes of the electrode pair, and at least one property of the material between the electrodes of the electrode pair.

2. 2. The sensing device of claim 1, wherein the generated capacitive signal output is processable to determine information related to the deformable object, the information including at least one of shape information, deformation information, force information, and / or velocity field information.

3. The sensing device of claim 1 or 2, wherein at least one of the plurality of electrodes is deformable, stretchable, and / or compressible.

4. 3. The sensing device of claim 1 or 2, wherein the plurality of electrodes includes two or more proximal electrodes and two or more distal electrodes, and the selected electrode pairs include at least one electrode pair consisting of the proximal electrode and the distal electrode, at least one electrode pair consisting of two of the proximal electrodes, and / or at least one electrode pair consisting of two of the distal electrodes.

5. 3. The sensing device of claim 1, wherein the plurality of electrodes are distributed along one or more surfaces, thereby forming a three-dimensional spatial distribution, the arrangement of which is continuously deformable from a planar arrangement.

6. 3. The sensing device of claim 1 or 2, wherein the capacitance signal output is processable to determine information relating to at least one of bending, twisting, stretching, expansion, and compression of the object.

7. The sensing device of claim 1 or 2, wherein the plurality of electrodes are arranged laterally along a first surface of the deformable mass and a second surface of the deformable mass.

8. The sensing device of claim 1 or 2, wherein the plurality of selected electrode pairs includes at least one electrode pair of a first electrode and an electrode not adjacent to the first electrode among the plurality of electrodes.

9. 3. The sensing device of claim 1, wherein the plurality of electrodes is operable to generate the capacitive signal output from the plurality of selected electrode pairs according to a predetermined sequence.

10. The sensing device of claim 1 or 2, wherein the plurality of selected electrode pairs comprises a subset, such as a degenerate subset, of all electrode pairs in the plurality of electrodes.

11. 3. The sensing device of claim 1, wherein the plurality of electrodes includes at least two electrodes arranged in a first plane and at least two electrodes arranged in a second plane, the first plane being substantially non-parallel to the second plane.

12. 3. The sensing device of claim 1, wherein the plurality of electrodes are disposed within one or more deformable substrates, the one or more deformable substrates being capable of being deformed to mate with a surface.

13. 13. The sensing device of claim 12, wherein the one or more deformable substrates are stretchable in at least a lateral direction, and optionally the one or more deformable substrates are stretchable to increase or decrease a distance between two or more electrodes of the plurality of electrodes.

14. 3. The sensing device of claim 1 or 2, wherein the plurality of electrodes form one or more sensor modules, each of which is continuously deformable from a planar configuration and / or mates with a surface.

15. The sensing device of claim 1 or 2, wherein the plurality of electrodes comprises a stretchable conductive material.

16. 3. The sensing device of claim 1, wherein the plurality of electrodes comprises elongated conductor portions configured to elongate further in response to a force, and optionally the elongated conductor portions of the plurality of electrodes are arranged in a parallel arrangement.

17. 3. A sensing device as described in claim 1 or 2, wherein the plurality of electrodes are provided on or integrated into one or more deformable substrates for attachment to the deformable object, and / or one or more of the plurality of electrodes are integrated into the surface of the deformable object.

18. The sensing device of claim 1 or 2, wherein the deformable object comprises at least one of a robotic arm, another robotic manipulator, a human body part, and / or a wearable object.

19. The sensing device according to claim 1 or 2, wherein the plurality of electrodes are distributed according to a predetermined layout.

20. 3. The sensing device of claim 2, further comprising a processing resource configured to process the capacitance signal output, or capacitance data obtained from the capacitance signal output, to obtain the information related to the deformable object, and optionally the processing resource configured to apply at least one predetermined model to obtain the information.

21. acquiring capacitance signal output data representing capacitance signal outputs from a plurality of selected electrode pairs of a plurality of electrodes distributed near one or more surfaces of a deformable object, wherein the capacitance signal output generated at an electrode pair in the plurality of electrodes depends on at least one of a distance between the electrodes of the electrode pair, a shape and / or orientation of the electrodes of the electrode pair, and at least one material property of a material between the electrodes of the electrode pair; A method wherein the capacitance signal output data is processable to determine information relating to the deformable object.

22. 22. The method of claim 21 , further comprising processing the capacitance signal output to determine the information, the processing comprising applying at least one model to the capacitance signal output, for example a model determined using a machine learning generated process.

23. 23. The method of claim 21 or 22, wherein processing the capacitance signal output is performed as part of at least one of a shape reconstruction process, a deformation detection process.

24. The method of claim 22 , wherein the at least one model is configured to output touch information and deformation information.

25. 1. A method of training at least one model, comprising: A step of acquiring training data, the training data comprising: capacitance signal output data representing capacitance signal outputs from a plurality of selected electrode pairs of a plurality of electrodes arranged laterally along one or more surfaces of the deformable object; and including additional data representing information related to said deformable object; performing a model learning process using the acquired training data to obtain at least one trained model for obtaining additional information about the deformable object using additionally acquired capacitance signal outputs; A method comprising:

26. 26. The method of claim 25, wherein the acquired capacitance signal output data is acquired for one or more spatial arrangements of the plurality of electrodes and / or acquired in response to one or more deformations applied to the deformable object, and the acquired information comprises shape and / or deformation information data corresponding to the one or more spatial arrangements and / or the one or more applied deformations.

27. 27. The method of claim 25 or 26, wherein the acquired capacitive signal output data is acquired in response to performing a series of touch actions at a plurality of locations on the one or more surfaces, and the acquired information includes touch location information.

28. A non-transitory computer-readable medium comprising instructions for performing the method of claim 21, 22, 24, or the method of claim 25 or 26, when executed by a processor.

Images