INTELLIGENT HUMAN-MACHINE INTERFACE AND METHOD EXECUTED WITH IT

Description

[0001] The invention presented relates to a novel non-invasive human-machine interface and a method which can be carried out with such an intelligent non-invasive human-machine interface.

[0002] The increasing networking and digitalization of today's world requires intuitive communication between a person (or even an animal) and electronic devices of all kinds. A key term here is the Internet of Things (IoT), which refers to the connection to a large number of devices via the Internet and simultaneously enables users to operate or control these devices.

[0003] Without any claim to completeness, examples of such devices include: vehicles of all kinds, household appliances, building heating systems, fitness trainers, gaming equipment, drones, medical prostheses, robots and much more.

[0004] To operate and / or control such and other devices, intelligent human-machine interfaces are required that enable a person (or animal) to use the devices as simply and intuitively as possible.

[0005] US 2018 / 307314 A1 and WO 2016 / 168117 A2 disclose the detection of an electrical signature that changes depending on muscle activity using surface electrodes and the determination of movement based on this data. US 2018 / 067552 A1 and the articles "An Imperceptible Magnetic Skin" in ADVANCED MATERIALS TECHNOLOGIES, Vol. 4, No. 10, October 10, 2019, XP055688067, and "Magnetosensitive e-skins with directional perception for augmented reality" in Science Advances, January 1, 2018, XP055688173, each disclose the detection of movement based on magnetic data.

[0006] The present invention aims to provide a preferably portable, non-invasive human-machine interface.

[0007] To achieve the stated object, a preferably portable human-machine interface according to claim 1 is proposed, which is designed for non-invasive attachment to a skin area of ​​a human or animal body and which comprises: a flexible, first magnetic layer for contacting a surface of the skin area, which contains or carries a first plurality of magnetic or magnetizable elements whose magnetic fields together form a magnetic signature at any time, wherein the flexibility of the layer is such that the layer can at least substantially follow deformations of the surface of the skin area, and wherein the magnetic signature changes in a measurable manner upon deformation of the surface of the skin area; a detection device for measuring the magnetic signature as a function of a deformation of the flexible, first magnetic layer, which faces the side of the flexible layer facing away from the skin area; a second plurality of potential-measuring surface electrodes arranged in or on the flexible layer to come into direct contact with the surface of the skin area;The totality of the measurement signals generated by the potential measurement surface electrodes at a given time forms an electrical signature that changes depending on the muscle activity, and an integrated or external processing unit configured to assign a pair consisting of a magnetic signature and an associated electrical signature to a configuration and / or movement and / or a force with which the movement is performed. An essential property of the human-machine interface according to the invention is that its components as well as its functions are located or take place entirely outside the human or animal body.

[0008] In the context of this disclosure, the term "magnetic or magnetizable element" refers to any element that is either magnetic itself or can be magnetized, for example, permanent magnets or magnetizable particles, which can also be arranged, for example, in a magnetic tape or a magnetic strip. In particular, a magnetic or magnetizable element also includes a conductor through which current flows, such as an electromagnet. It is important that every magnetic or magnetizable element leads to an outwardly acting magnetization that can be detected by the detection device.

[0009] The flexibility of the first magnetic layer must be such that it can at least largely follow deformations of the surface of the skin area. This can be achieved, for example, by intrinsic flexibility of the first magnetic layer or by a flexible coupling of otherwise rigid individual elements.

[0010] In an advantageous embodiment of the human-machine interface according to the invention, the flexible, first magnetic layer comprises a plurality of magnets arranged in the form of an array. Such a flexible magnetic layer can be realized, for example, by embedding commercially available permanent magnets with a simple north-south pole configuration into a ribbon-like, elastic silicone layer. Alternatively or additionally, current-carrying conductors, e.g., electromagnets in the form of an array, or combined sensor elements (see the relevant explanation below) can be embedded in a ribbon-like, elastic silicone layer. Other arrangements are, of course, conceivable and equally suitable.

[0011] The flexible, first magnetic layer of the human-machine interface according to the invention is further characterized in that a second plurality of potential-measuring surface electrodes are embedded in or applied to the flexible layer in such a way that these potential-measuring surface electrodes come into direct contact with the surface of the skin area during operation of the human-machine interface. The first plurality and the second plurality can be equal in number, but this is not a requirement for the functioning of the human-machine interface according to the invention. The potential-measuring surface electrodes can, in particular, be electromyography (EMG) electrodes, as used in electromyography (EMG). Other flat electrode designs are also conceivable.The electrical potentials derived from the surface of the skin area using the potential measurement surface electrodes are, in particular, electromyographic potentials, which allow a statement to be made about the electrical activity of the muscles involved in a given skin deformation.

[0012] In a preferred embodiment in which the flexible, first magnetic layer comprises a plurality of magnets arranged in the form of an array, the number of magnets corresponds to the number of potential measuring surface electrodes and each potential measuring surface electrode is rigidly coupled to a magnet, for example by attaching the potential measuring surface electrode to the surface of the magnet facing the skin area.

[0013] A combination of a potential-measuring surface electrode and a magnetic or magnetizable element rigidly or at least firmly coupled thereto is referred to in this disclosure as a "combined sensor element." The entirety of such combined sensor elements is therefore capable of not only measuring a magnetic signature composed of the magnetic fields of all magnetic or magnetizable elements, but also providing an electrical signature composed of the totality of the measurement signals generated by the potential-measuring surface electrodes at a specific time. These signals can be measured by means of muscle activity derived from the potential-measuring surface electrodes when the surface of the skin area is deformed.The magnetic signature resulting from measurements of the magnetic properties and the electrical signature obtained from measurements of the electromyographic potentials can together also be referred to as a combined signature that characterizes a displacement / twisting of one or more magnetic or magnetizable elements in conjunction with the associated skin-conformal potential measurement surface electrodes resulting from a muscle-induced skin deformation.

[0014] According to the invention, a detection device designed for this purpose is used to measure the magnetic signature, which is directed toward the side of the flexible, first magnetic layer facing away from the skin area. In other words, the detection device is designed to detect at least those changes in the magnetic signatures of the magnetic layers that, with respect to an initial state, are caused by a deformation of the surface of the skin area, and in particular by a muscle-induced deformation of the surface of the skin area.

[0015] In preferred embodiments, this detection device contains a plurality of magnetometers, which are preferably arranged in the form of an array. The shape of this array does not have to match the array in which the first plurality of magnetic or magnetizable elements is arranged. Nor does the number of magnetometers need to match the number of magnetic or magnetizable elements comprised by the flexible, first magnetic layer.

[0016] If the previously explained combined sensor elements are used in the human-machine interface according to the invention, then each potential-measuring surface electrode and the magnetic or magnetizable element(s) firmly coupled to it are preferably separated from each other by an electromagnetic insulating layer to shield against magnetic interference fields. Such an electromagnetic insulating layer can be a layer of a material with high magnetic permeability, such as iron, µ-metal, carbon fibers, etc.

[0017] The human-machine interface according to the invention can optionally comprise a second magnetic layer with a plurality of magnetic or magnetizable elements. This second magnetic layer can be arranged between the flexible, first magnetic layer and the sensing device or can be located on the side of the sensing device facing away from the flexible, first magnetic layer. The second magnetic layer does not need to be flexible, but can have the same or similar flexibility as the flexible, first magnetic layer to facilitate non-invasive application and improve skin conformity. The second magnetic layer interacts with the first magnetic layer through its magnetic field.

[0018] The magnetic properties of the first and second magnetic layers are preferably such that the polarity / magnetic charges of the two magnetic layers are oppositely directed. According to one embodiment, the flexible, first magnetic layer can contain magnets or magnetic particles or have material properties such that it can be externally magnetized in a desired direction, e.g., by current-carrying conductors or permanent magnets of the second magnetic layer.

[0019] The resulting magnetic fields generated by the first magnetic layer or by the first and second magnetic layers are measured as a magnetic signature by the detection device, preferably by one or more magnetometers arranged in an array. In particular, the second magnetic layer can contain current-carrying conductors suitable for measuring, controlling, and / or compensating the strength of magnetic fields. Furthermore, the second magnetic layer can preferably have a magnetic field strength and polarity that causes the magnetic or magnetizable elements and / or combined sensor elements used in the flexible, first magnetic layer to repel toward the surface of the skin area.In this way, the conformity of the flexible, first magnetic layer to the skin surface is increased, even during rapid deformation of the skin surface, and hysteresis phenomena resulting from the flexibility of the flexible layer are reduced. Furthermore, this method can counteract unwanted, premature saturation of the magnetometers. Such an intentional influence on the first magnetic layer by the optionally present second magnetic layer can be calculated out of the measured magnetic signature, for example, in the form of an offset (which can also vary over time), in order to compensate for any possible distortion of the measured magnetic signature resulting from the intentional influence.

[0020] If the magnetic properties of at least one of the magnetic layers are controlled by electromagnets, the magnetic fields of the magnets, magnetized particles, or combined sensor elements used can be controlled, resulting in further possibilities for compensating and / or controlling the magnetic field strength or shape. Such control of the magnetic properties can achieve significant advantages in terms of the precision and measurement speed of the deformation of the surface of the skin area in question.

[0021] The provision of a second magnetic layer and the resulting advantages as described above can also occur independently of the presence of the second plurality of potential-measuring surface electrodes. In other words, a likewise very advantageous embodiment of the human-machine interface according to the invention has the second magnetic layer, but not the second plurality of potential-measuring surface electrodes. In such an embodiment, the assignment to a configuration and / or movement and / or a force with which the movement is executed is made exclusively on the basis of the detected magnetic signature.

[0022] If desired or required depending on the application, an additional electromagnetic insulating layer made of a material with high magnetic permeability, such as iron, µ-metal, or carbon fiber, can be provided on the side of the first magnetic layer facing away from the skin area. This layer serves, among other things, to shield against magnetic interference fields. In embodiments with a first and a second magnetic layer, such an additional electromagnetic insulating layer is to be arranged on the side of the second magnetic layer facing away from the skin area.

[0023] In the context of the present disclosure, "magnetic properties" is understood to mean that the magnetic properties of the magnetic layers are designed in such a way that a deformation of the surface of the skin area, which in particular is followed by the flexible, first magnetic layer, leads to a measurable change in the magnetic properties of at least one of the layers, preferably the flexible, first magnetic layer.

[0024] Finally, the human-machine interface according to the invention comprises a processing unit configured to assign the measured magnetic and electrical signatures to a configuration and / or movement and / or their intensity by executing an algorithm. In particular, the processing unit can be configured to assign a pair consisting of a magnetic signature and an associated electrical signature to a configuration and / or movement and / or a force with which the movement is executed.

[0025] The processing unit associated with the human-machine interface can be integrated or remote. "Integrated" here means that the processing unit is located directly on the portable human-machine interface. However, this processing unit can also be located remotely from the portable human-machine interface and communicate with the portable human-machine interface, for example, via a wired and / or wireless connection.

[0026] The deformation of the surface of a skin area can have voluntary or involuntary causes. For example, a person may voluntarily reposition their limbs, resulting in corresponding deformations of certain skin areas. On the other hand, involuntary deformations of certain skin areas may occur, for example, due to a person's illness (e.g., due to involuntary muscle contractions or organ movements). The identification and classification of these deformations can enable a physician to make a better diagnosis.

[0027] In the context of the present invention, the term "configuration" refers to minimal coordinates necessary to describe the degrees of freedom of the respective extremity—in the case of a human hand, for example, the joint angles. In other words, a "configuration" is a specific position of a human or animal extremity (examples include: leg extended, leg bent, hand open, hand closed, arm straight, arm bent, hand twisted or not twisted relative to the elbow, etc.). Of course, a configuration can also indicate a state between two end configurations, such as arm partially extended, hand partially closed, etc.

[0028] In the context of the present invention, the term "movement" is understood to mean a temporal change (particularly with regard to extent and speed) of a configuration.

[0029] In the context of the present invention, the term "intensity of movement" refers to the force with which a movement is carried out.

[0030] Generally speaking, voluntary or involuntary movements of muscles, tendons, ligaments, bones, and / or organs in a human or animal body result in deformation of the surface of skin areas. One aim of the present invention is to detect such deformations and reliably assign them to specific states (configurations) and / or movements, as well as, if desired, to the force with which the movement is performed.

[0031] The magnetic layers can take on various designs. However, sufficient conformity, especially of the flexible, first magnetic layer, to the surface of the skin area of ​​interest is important, so that any deformations of the skin surface and the associated deformations of the flexible, first magnetic layer lead to measurable changes in its magnetic properties. Furthermore, skin conformity is a prerequisite for optimal measurement using the potential measurement surface electrodes.

[0032] More precisely, changes in the magnetic field strength and / or changes in the magnetic flux density are preferably measured with respect to both magnetic layers. In other words, the resulting magnetic field is measured, whereby an existing magnetic field does not necessarily have to change itself, but merely needs to appear as a different magnetic field from the perspective of the detection device after a skin deformation.

[0033] In this context, it is advantageous if the magnetic layers contain or consist of a pole configuration with many north-south pole pairs. In this way, a shift or rotation of the magnetic layers allows for externally measurable magnetization by measuring the magnetic field. The pole configuration, along with the strength of the respective magnetization, is crucial, so that changes in the magnetic flux density and / or magnetic field strength can be measured by magnetometers.

[0034] In addition to measuring the magnetic and electrical signature, it may be advantageous to capture additional information that arises during a deformation of the surface of a skin area of ​​interest. For this reason, preferred embodiments of human-machine interfaces according to the invention also include one or more acceleration sensors and / or yaw rate sensors, the output signals of which can also be fed to the processing unit.

[0035] Based on the detected magnetic and electrical signatures of the magnetic layers resulting from a deformation of the surface of a skin area of ​​interest, the processing unit executes a mapping algorithm that maps the measured change (or changes) in the magnetic and electrical signature(s) to a configuration and / or a movement and / or a force with which the movement is performed.

[0036] Based on the assigned configuration and / or movement and / or a force with which the movement is executed, the processing unit or a downstream control unit can, for example, generate a control command that is fed to a device, which then executes an action corresponding to the control command. Thus, the action performed by the device corresponds to the at least one configuration and / or movement assigned by the processing unit or the assignment algorithm and / or a force with which the movement is executed. The assigned configuration and / or movement and / or a force with which the movement is executed can, for example, be a gesture or continuous movement of one or more fingers.

[0037] The assignment of the change in the magnetic properties of the magnetic layers detected by the detection device, for example by means of magnetometer measurements, to, for example, a configuration of a human extremity can be advantageously achieved with the aid of artificial intelligence.

[0038] The processing unit of a human-machine interface according to the invention is therefore preferably configured to assign the measured change in the magnetic signatures of the magnetic layer or layers to a configuration and / or movement and / or a force with which the movement is carried out, with the aid of Bayes classifiers, an artificial neural network (for example, convolutional long short-term memory networks or graph neural networks), a support vector machine, a finite state machine, a hidden Markov model, a relevance vector machine, a dynamic time warping method, a conditional random field method, a decision tree, a random forest method, a k-nearest neighbors algorithm, a discriminant analysis, a linear regression, a logistic regression, a Gaussian process, a perceptron and / or ensemble methods (Bayes optimal classifier, bagging, boosting).with additional inclusion of the data obtained from the potential measurement surface electrodes, and if necessary, with inclusion of the data supplied by angular rate and / or acceleration sensors. In general, it can be said that first, data sets consisting of specific magnetic and electrical signatures and an associated, arbitrary configuration and / or movement are recorded, and then the relationship between these variables is determined in a so-called training. A distinction must be made between a learning or training mode, in which sufficient measurement data must be recorded and assigned in order to subsequently enable, in a prediction mode, an accurate assignment of a single or multiple measurements of the human-machine interface to a configuration and / or movement and / or a force with which the movement is executed. Since the measured changes in the magnetic properties, in addition to tendon,Bone or organ movements are essentially due to the skin deformation resulting from muscle activation and ultimately muscle contraction, these muscle activations or associated measurable electrode potential changes are recorded by the potential measurement surface electrodes according to the invention and used by the processing unit to further improve the recognition of configurations and / or movements and / or the forces with which the movement is carried out.

[0039] Advantageous embodiments of the human-machine interface according to the invention are further provided with a communication unit for exchanging data with at least one external device which, based on the data exchanged with the human-machine interface, can execute an action which corresponds to the at least one associated configuration and / or movement and / or a force with which the movement is executed.

[0040] For example, a human-machine interface according to the invention can be integrated into a smartwatch, so that functions of the smartwatch can be executed with the aid of the human-machine interface according to the invention. For example, it would be conceivable for certain finger and / or hand movements to be assigned to certain functions and / or control commands of a smartwatch, so that the smartwatch can be operated easily and intuitively by executing these movements. Another application could be the control of robots in the sense of telerobotics, wherein the degrees of freedom to be controlled of the robot or its end effector are controlled taking into account the configuration and / or movement identified by the processing unit and / or a force with which the movement is executed.

[0041] The present invention is also directed to a method according to claim 12 for detecting a deformation of the surface of a skin area of ​​a human or animal body and associating the detected deformation with a configuration and / or movement and / or a force with which the movement is performed. The method comprises the steps: Arranging a first plurality of magnetic or magnetizable elements on or near a skin area such that the first array can at least substantially follow a skin deformation; Arranging a second plurality of potential-measuring surface electrodes such that the potential-measuring surface electrodes come into direct contact with the surface of the skin area; Determining at least one measurable change in a magnetic signature of the entirety of the first plurality of magnetic or magnetizable elements resulting from a skin deformation by means of a detection device for measuring the magnetic signature; Recording the measurement signals generated by the entirety of the second plurality of potential-measuring surface electrodes at a specific time to obtain at least one electrical signature that changes depending on the muscle activity;and executing an algorithm for assigning the determined at least one magnetic signature and the obtained at least one electrical signature to a configuration and / or movement and / or a force with which the movement is carried out. ,

[0042] As previously explained, the deformation of the surface of the skin area can be based on involuntary or voluntary muscle, tendon, ligament, bone and / or organ movement, whereby muscle-induced movements are accompanied by the activation of muscles and thus enable the recording of (electrical) surface potentials.As already explained above in connection with the human-machine interface according to the invention, the assignment algorithm is preferably a trained artificial intelligence that has determined or learned the relationship between magnetic signatures and electrical signatures and associated configurations and / or movements and / or a force with which the movement is executed, based on a data set consisting of at least one magnetic signature and at least one electrical signature and the associated configuration and / or movement and / or a force with which the movement is executed. The assignment algorithm can preferably be those algorithms that have already been mentioned above in connection with the human-machine interface.

[0043] The terms "magnetic signature" and "electrical signature" refer to the totality of the measured or measured changes in the magnetic and electrical properties of the magnet and electrode array that result from a deformation of the surface of a skin area of ​​interest or the underlying muscles. Another magnetic / electrical signature can serve as a reference, describing an initial state before a skin deformation occurs.

[0044] In particular, signal conditioning or initialization can be performed based on a predetermined reference configuration before beginning to detect a change in the magnetic / electrical signature. For example, all measurement signals can be zeroed for such a reference configuration, and changes in the magnetic / electrical signature can be measured based on this reference configuration. According to one embodiment, each magnetic signature is composed of measured magnetic field strengths and / or measured magnetic flux densities of a magnet array that uses combined sensor elements. According to the invention, the magnetic signatures are thus augmented by electrical signatures, with the magnetic signature resulting from a deformation of the skin surface and the electrical signature being obtained by electromyographic surface measurements of the muscles lying beneath the deformed skin areas.

[0045] In the present invention, the measured changes in the magnetic and electrical properties of the magnetic layer(s) resulting from skin deformation represent the useful signal. Other changes, which may be caused by interference such as the earth's magnetic field or magnetic objects, may need to be filtered out of the measured signals. If, for example, there are magnetic components in the human-machine interface, these can be permanently compensated using a so-called hard-soft-iron calibration, i.e., such a calibration only needs to be performed once for a given human-machine interface. The earth's magnetic field at a given application location of the human-machine interface can also be permanently compensated.To compensate for magnetic interference fields, additional magnetometers can be referenced and / or electromagnetic materials can be used for shielding, e.g. highly permeable iron or µ-metals.

[0046] Exemplary embodiments of a human-machine interface according to the invention and of the method according to the invention are explained in more detail below with reference to the attached schematic drawings. It shows: Fig. 1: a schematic representation of the structure of a first embodiment of a human-machine interface according to the invention, Fig. 2: a schematic representation of the structure of a second embodiment of a human-machine interface according to the invention, Fig. 3: a block diagram of a method for detecting continuous movements using the example of finger and hand movements, and Fig. 4: a flow chart to illustrate the experimental setup for evaluating the human-machine interface according to the invention using the method according to the invention.

[0047] Fig. 1 shows, in an exploded view, a schematic representation of the layered structure of a human-machine interface, generally designated 10, which was used to carry out an exemplary embodiment of a method according to the invention, which will be described in more detail below. In the exemplary embodiment described here, the human-machine interface 10 has an overall band-like shape and comprises a band-shaped support element 12, by means of which the human-machine interface 10 can be fastened, for example, to the wrist of a user facing a skin area (not shown). For this purpose, the support element 12 is preferably designed to be flexible, but can also be rigid in other embodiments not shown.

[0048] A layer 14 is arranged on the support element 12, which serves to position an array of magnetometers 16. In this exemplary embodiment, each magnetometer 16 contains combined Hall-effect sensors and can thus measure the magnetic flux density in the x, y, and z directions. The magnetometer array is designed here as a printed circuit board 18 with 4x12 sensor chips 20 in which the magnetometers 16 are arranged. In addition to a magnetometer 16, each sensor chip 20 also contains an acceleration sensor and a yaw rate sensor, thus enabling 9-axis measurement if desired.

[0049] In Fig. 1 Above layer 14 is a flexible, first magnetic layer 22, also ribbon-shaped and made of silicone rubber, into which an 8x16 array of a total of 128 so-called combined sensor elements 24 is embedded. The silicone rubber used in the embodiment described here is transparent.

[0050] Each combined sensor element 24 comprises a magnet 26, here configured as a permanent magnet, and a potential-measuring surface electrode 28 located on the side of the magnet 26 facing the skin area and rigidly coupled to it. In the example shown, the potential-measuring surface electrodes 28 are electromyography (EMG) electrodes for measuring electromyographic potentials.

[0051] The flexible layer 22, referred to as the first magnetic layer, with the combined sensor elements 24 embedded therein, rests against the underside of the forearm near the wrist of a user, conforming to the skin, in accordance with the assumed application example. Due to the magnetic measuring principle, the magnetometers 16 and the combined sensor elements 24 do not need to be coupled by a physical medium, as is necessary, for example, with force or pressure sensors. The flexible layer 22, made of silicone rubber, is highly elastic to minimize hysteresis phenomena, yet sufficiently compliant to be able to follow even small skin deformations.

[0052] The magnetometer array with the magnetometers 16 arranged thereon, together with a multiplexer (not shown here) and a microcontroller (likewise not shown), forms a detection device 30 of the human-machine interface 10 according to the invention. As a microcontroller, for example, a microcontroller already present in a smartwatch or another electronic device or a microprocessor can be used.

[0053] The EMG electrodes 28 are connected to a measuring electronics known to experts in this field and therefore not further explained or shown.

[0054] The microcontroller feeds the acquired magnetometer measurements and electromyographic measurement signals to a processing unit (not shown here), which, using an algorithm implemented within it, can assign a measured magnetic and electrical signature to a configuration and / or movement and / or its intensity. This processing unit can, for example, be integrated into a smartwatch and would then be an integral part of the human-machine interface 10. However, it can also be located remotely from the human-machine interface and, for example, communicate wirelessly with the arrangement shown in the figures.Due to the use of the previously discussed combined sensor elements 24, in addition to the magnetic signature generated by the magnets 26, electrical signatures (caused by the muscle activity) can be measured and transmitted to the processing unit (not shown).

[0055] Fig. 2 schematically shows a second embodiment of a human-machine interface 10, in which an additional, second magnetic layer 32 with a total of forty electromagnets 34 is present, arranged in the form of a 5x8 array. Instead of the electromagnets 34 used here, other magnetic or magnetizable elements could also be used (not shown). Fig. 2 In the example shown, the second magnetic layer 32 is located between the support element 12 and the layer 14, but the second magnetic layer could also be arranged between the layer 14 and the flexible, first magnetic layer 22.

[0056] In the illustrated embodiment, the second magnetic layer 32 serves to establish a magnetic field strength and polarity during operation of the human-machine interface 10 such that a repulsion of the magnets 26 within the flexible, first magnetic layer 22 toward the surface of the aforementioned skin area is effected. In this way, the skin conformity of the flexible layer 22 is increased, even during rapid deformations of the skin surface, and hysteresis phenomena caused by the flexibility of the flexible layer 22 are also reduced. Furthermore, premature saturation of the magnetometers 16 can be counteracted in this way.

[0057] To avoid disturbances in the magnetic field strengths of the first magnetic layer 22 and the second magnetic layer 32, Fig. 2 In the embodiment shown, a highly magnetically permeable layer 36 is provided between the support element 12 and the second magnetic layer 32 for shielding purposes. Furthermore, a further highly magnetically permeable layer 38 is provided between each magnet 26 and the associated EMG electrode 28 in order to minimize the influence of interference between the EMG electrode 28 and the magnet 26 of a combined sensor element 24 within the first magnetic layer 22. Inventive method

[0058] In order to enable the recognition of continuous finger and hand movements using the described portable human-machine interface 10, a method based on neural networks was developed, see in particular Figure 3. The method implements a supervised learning algorithm which assigns magnetic, possibly electrical, and possibly other signatures, e.g. capacitive and optical sensor arrays, to measured output, i.e., target variables. The method, i.e., the neural network architecture, typically uses data originating from domain-specific data conditioning. Although the described method within the scope of the present disclosure is primarily aimed at assigning a configuration and / or movement and / or its intensity based on detected magnetic signatures, possiblyin combination with measured electrical and other signatures, it should be expressly pointed out at this point that such a method is suitable for processing any type of signature and combinations thereof. Such a method can therefore enable the assignment to a configuration and / or movement and / or its intensity, for example also based on capacitive signatures detected by one or more capacitive sensors, if desired also in combination with other electrical signatures, e.g. electromyographic signatures, and possibly other signatures. Likewise, such a method can enable the assignment to a configuration and / or movement and / or its intensity, for example also based on optical signatures detected by one or more optical sensors, if desired again in combination with other signatures of an electrical and / or magnetic nature.Such alternative embodiments of the method described herein are also part of the present disclosure.

[0059] Since the data are measurements of temporal sequences with a spatial arrangement, additional statistical or physical calculation variables can be constructed from the measurements and advantageously incorporated into the process as a supplementary filter channel: For example, the vector magnitude can be calculated from the magnetometer measurements in the x, y, and z directions and concatenated as an image-equivalent channel. The same applies to the high-frequency electrical signatures, which can be continuously evaluated using statistical parameters by defining sliding windows (sliding-window approach). In this sense, signals from magnetic and electrical signatures as well as signals from different time steps can also be combined.This not only significantly improves the performance of predicting the actual finger and hand movements, but also allows additional physical or statistical information to be obtained for the evaluation of medical or user-specific questions.

[0060] The described method is not limited to finger and hand movements, but generally allows the assignment of measured, combined signatures with physiologically associated movements or movement intensities as output variables.

[0061] Fig. 3 shows that the inventive method for detecting continuous movements Convolutional Long-Short-Term-Memory ( convLSTM, i.e., LSTM with convolutional kernel) and 3D convolutional (Conv3D) neural networks. Preprocessors perform domain-specific signal conditioning, e.g., artifact removal in the case of electrical signatures from EMG measurements, typically also normalization to the interval, here [0,1], and especially rendering into the input format of image-equivalent matrices. The resulting signals from the preprocessor are each processed using convLSTM- Blocks with 1×1-kernel is projected non-linearly and passed to the blocks in the bracket expression. The method is modular and scalable thanks to the parameters s = 1, 2,..., S (stage) and t = 1, 2,..., T (depth). The parameter ns,t represents the total number of filters n in stage s with depth t and kernel size K. The parameter fs,t represents the filter base number, which, in combination with the filter multiplier ws,t, controls the width of the architecture. The entire architecture can thus be flexibly adapted to a variety of problem sizes with varying complexity by varying the number of input and output variables to be predicted, since width and depth can be scaled. The path with the bold arrows represents a skip connection.

[0062] The number and selection of output variables is application-specific. For example, in the case of predicting individual gestures in a classification problem, the only output variable is a one-hot matrix, with the activation function of the associated conv3D block defined as a softmax activation by default.

[0063] The domain-specific preprocessor delivers artifact-free, typically scaled or normalized to the interval [0,1] and rendered in an image-like, topographical manner, i.e., spatially meaningfully arranged, input data as numerically processable matrices of the dimension (sequence length, number of sensor elements in the x-direction, number of sensor elements in the y-direction, channel depth: the total dimension of a measured value and the dimensions of constructed calculation variables). Measured values ​​with high statistical fluctuation can be replaced or supplemented by sliding statistical calculation variables of a sequence of measured values. This is particularly advantageous in the case of EMG measurements, since their high sampling rate compared to magnetometer measurements allows statistical calculation variables to be collected over the time horizon from the last to the next magnetometer measurement, thus allowing the different sampling rates to be meaningfully synchronized.Statistical calculation parameters for the evaluation of EMG signals include mean absolute value, mean absolute value slope, slope sign changes, waveform lengths, zero crossings, root mean square, waveform length, number of zero-crossings, variance and maximum in the time domain and wavelet packet transforms in the frequency domain.

[0064] For the sake of explanation, let's assume that the magnetometers measure the magnetic field in the x, y, and z directions, and that a measured value has a dimension of three. This allows us to construct physical calculation quantities, e.g., the vector magnitude, the unit vector, etc.

[0065] Electromyography measurements typically have a dimension of one, i.e., the potential change measured by an electrode. Thus, measured values ​​from an array of magnetometers as well as the electromyography measurements, after artifact removal, analog / digital conversion, and filtering, can be rendered into image-equivalent matrices. These matrices, when arranged spatially appropriately—usually according to their position in the sensor array—represent a topographical mapping of the measured values ​​corresponding to their respective domain. In the present embodiment of the method, these domain-specific sequences of image-equivalent matrices are concatenated along the channel depth. The channel depth therefore increases with the number of domain-specific measured values ​​and the number of constructed computational variables. A sequence of a temporal progression of these combined image-equivalent matrices can then be fed into the method.The blocks in the architecture of the procedure are summarized in Table 1. The described method allows for the reliable prediction of both discrete classes in a multi-class classification problem (e.g., gestures) and continuous output variables in a regression problem (e.g., joint angle profiles of the extremities) based on the data measured by the human-machine interface. The solution space described by the classes is often limited to the number of gestures in gesture identification. However, in an unsupervised learning problem such as the k-means cluster algorithm, these classes do not necessarily have to be predefined. On the one hand, it is possible to define many discrete gestures depending on the application scenario (e.g., controlling a smartwatch), i.e., to expand the solution space, and to implement the described method as a multi-class classification.On the other hand, it is possible to specify biomechanically meaningful and smooth movements between the defined, discrete gestures that represent the desired final configurations, so that even more complex applications (e.g., controlling a hand prosthesis) can be realized.

[0066] The following describes how the method enables identification that goes beyond the recognition of discrete classes and can map the solution space of many temporally continuous output variables (regression). This involves not only recording a single magnetic and electrical signature, but also recording and evaluating entire sequences of these signatures over an arbitrarily definable time horizon. Thus, a temporal domain is added to the spatial domain, enabling a very comprehensive and application-oriented evaluation of the measurements provided by the human-machine interface shown. Example application

[0067] As part of the experimental testing of the novel human-machine interface, measurements of joint angles of the fingers of the right hand (measured by a commercially available data glove) synchronized and normalized with the measurements of the human-machine interface were defined as output variables. Subsequently, a regression problem was formulated, i.e., a prediction of joint angles based on measurements of the human-machine interface, and with the help of the Fig. 3 This is achieved by the method according to the invention. The joint angles of the right hand can thus be predicted based on the measurements of the human-machine interface.

[0068] To generate the necessary data, the data glove is placed on the right hand and the inventive human-machine interface is placed on the right forearm in such a way that, during fine, unrestricted finger movements, the measured values ​​from the data glove and the inventive human-machine interface can be recorded synchronously. The inventive human-machine interface measures the changes in the electrical and magnetic signature that occur as a result of the skin deformations caused by the finger movements on the right forearm, or the myoelectric activity of the underlying muscles. The movements are performed as naturally as possible with varying intensities and with some variation in speed over a period of 10 minutes and recorded accordingly.

[0069] It should be noted that defining the output variables using the measured values ​​from the data glove is only an example. The output variables can be defined arbitrarily or even reduced to a classification problem. However, to be meaningful, they must be directly or indirectly explainable by the sensor values ​​measured using the wearable human-machine interface. Experimental setup

[0070] The human-machine interface 10 used in the experiment consisted of an array of 128 (8x16 array) combined sensor elements 24 with a distance between the combined sensor elements of 7 mm. The schematic setup of the experiment and the domain-specific processing steps are shown in Fig. 4 , which schematically depicts the data glove and the synchronized measurement of the joint angles with the corresponding magnetic and electrical signatures measured by the detection device. The synchronized measurements enable the assignment of the hand-finger configuration defined by the totality of the joint angles and their temporal change to the magnetometer / electrode measurements using the method according to the invention.

[0071] The measured electrical signatures were sampled via Ag / AgCl electrodes at a frequency of 1000 Hz. A reference electrode was attached to the user's elbow. First, the power line interference was eliminated using a notch filter at 60 Hz with a Q factor of 50. Subsequently, the electrical signatures were Butterworth bandpass filtered (2nd order, corner frequencies: 20–390 Hz), the absolute value was calculated, and the moving average was determined over a sliding window of 10 ms in accordance with the sampling rate of the magnetometers. The processed electrical signatures were finally normalized to the value range [0,1]. Subsequently, the processed electrical signature measurements were rendered into an image-equivalent matrix in accordance with the arrangement of the combined sensor elements.The magnetic signatures were measured at a sampling rate of 100 Hz and, analogous to the electrical signatures, normalized to the value range [0, 1] and rendered into an image-equivalent matrix.

[0072] As a final step of data conditioning (preprocessing), the acquisition device temporally synchronized the filtered and averaged electrical signatures with the magnetic signatures.

[0073] The description of continuous finger movements is advantageously achieved by defining joint angles, taking into account the appropriate degrees of freedom for describing the movement. These are temporally continuous, dynamic quantities and, as already mentioned above, are determined by formulating and solving a regression problem from the pure magnetometer measurements and the processed measurements of myoelectric activity, i.e., the processed electrical signatures. The data glove enables an indirect measurement of degrees of freedom distributed across the hand and the individual fingers for a detailed description of finger movements (see also Fig. 4 ). Calibration can then convert the measured sensor values ​​into the biomechanical joint angles of the human hand. The synchronized joint angle measurements of the calibrated data glove were used as output variables and also normalized to the value range [0, 1]. The sampling rate of the data glove was 100 Hz.

[0074] Prediction of continuous movements using the method according to the invention and the human-machine interface according to the invention

[0075] For the determination of finger movements using the method according to the invention from Fig. 3 The following parameters were defined (Table 2) and Table 1. The cost function was defined as the mean squared error between the prediction and output variables. The optimization method was selected as ADAM (learning rate = 0.0001; β 1 = 0.9; β 2 = 0.999; decay = 0.0; e = None, amsgrad = False). The residual scaling (R, Table 1) was defined as R=1. As already mentioned, the terminology and default implementations are based on KERAS (https: / / keras.io / ) version v2.2.4 with the Tensorflow backend (https: / / www.tensorflow.org / ) version v1.15.0.

[0076] The thus completely defined instance of the procedure, i.e. the resulting neural network, was trained within 150 epochs with a batch size of 64. Sequences (sequence length s L = 10) of the processed, combined signatures were fed to the procedure as rendered image-equivalent matrices, with the concatenated filter channel having a depth of four (resulting from the x-, y-, z-measurements of the magnetic signature and the processed measurements of the electrical signature, see Figur 4 ). During the experimental testing, the resulting neural network was trained and evaluated with 42,000 (training data set), 12,000 sequences (test data set), and 6,000 (validation data set) randomized sequences.

[0077] The accuracy or precision of the prediction of finger movements naturally depends on the synchronized measurement frequency of the data glove and the wearable human-machine interface, so higher measurement frequencies must be maintained for faster movements. Furthermore, a sufficiently long measurement window must be selected, containing sufficient information (magnetometer and electrical measurements and the variables constructed from them) for the prediction. The resulting synchronized measurement frequency was f M = 100 Hz, which corresponded to the maximum sampling rate of the magnetometer measurements and the data glove. This comparatively high frequency enabled excellent results even for above-average grasping movements.

[0078] To evaluate the performance of the overall system (the inventive human-machine interface in combination with the inventive method), the root mean square error (RMSE) and the coefficient of determination (R2) between the actual joint angle profiles measured by the data glove and the joint angle profiles predicted by the overall system for the 20 measured joint angle degrees of freedom were evaluated for the cross-validation dataset over a time horizon of 3 minutes. The root mean square error averaged 2.6%, and the coefficient of determination averaged 97.5% for all gestures.

[0079] The described method in conjunction with the portable non-invasive human-machine interface is thus a versatile real-time system that not only allows the identification of continuous finger and hand movements using artificial intelligence algorithms, for example for the intuitive manipulation of digital systems, but also enables application-specific analyses, for example for medical processing or diagnosis, by taking into account physical and statistical quantities constructed from the measured or recorded magnetic and electrical signatures (and, if necessary, additional data from acceleration and angular rate sensors).

Claims

1. A human-machine interface (10) formed for non-invasive attachment to a skin area of a human or animal body, with: - a flexible, first magnetic layer (22) for coming into contact with a surface of the skin area, which contains or carries a first plurality of magnetic or magnetisable elements, whose magnetic fields together at any time form a magnetic signature, wherein the flexibility of the layer (22) is such that the layer (22) is able to at least substantially follow deformations of the surface of the skin area, and wherein the magnetic signature changes in a measurable manner in the event of a deformation of the surface of the skin area, - a detection device (30) for measuring the magnetic signature in dependence on a deformation of the flexible layer (22), which faces the side of the flexible, first magnetic layer (22) facing away from the skin area, - a second plurality of potential-measurement surface electrodes (28) arranged in or on the flexible layer (22) so as to come into direct contact with the surface of the skin area, wherein the totality of the measurement signals generated by the potential-measurement surface electrodes (28) at a given point in time forming an electrical signature that varies in dependence on the muscle activity, and - an integrated or external processing unit that is configured to assign in each case a pair consisting of a magnetic signature and an associated electric signature to a configuration and / or movement and / or a force with which the movement is carried out.

2. The human-machine interface according to claim 1, characterised in that the flexible, first magnetic layer (22) has a plurality of magnets (26) that are arranged in a shape of an array.

3. The human-machine interface according to claim 2, characterised in that the number of magnets (26) corresponds to the number of potential-measurement surface electrodes (28), wherein each potential-measurement surface electrode (28) is fixedly coupled to a magnet.

4. The human-machine interface according to one of claims 1 to 3, characterised in that the detection device comprises multiple magnetometers (16) that are preferably arranged in a shape of an array.

5. The human-machine interface according to one of the preceding claims, characterised in that the potential-measurement surface electrodes (28) are electromyography (EMG) electrodes.

6. The human-machine interface according to one of the preceding claims, characterised in that an electromagnetic insulating layer (38) is arranged between the potential-measurement surface electrodes (28) and the magnetic or magnetisable elements or magnets (26).

7. The human-machine interface according to one of the preceding claims, characterised in that a second magnetic layer (32) with a plurality of magnetic or magnetisable elements is present between the flexible, first magnetic layer (22) and the detection device (30) or on the side of the detection device (30) facing away from the flexible, first magnetic layer (22), its magnetic or magnetisable elements during operation of the human-machine interface having a magnetic field strength and polarity such that the flexible, first magnetic layer (22) is biased in a direction of the skin area.

8. The human-machine interface according to one of the preceding claims, characterised in that a deformation of the surface of the skin area is based on an involuntary or voluntary muscle, tendon, ligament, bone and / or organ movement.

9. The human-machine interface according to one of the preceding claims, characterised in that furthermore one or more acceleration sensors and / or angular rate sensors are present on the human-machine interface (10), the output signals of which are supplied to the processing unit.

10. The human-machine interface according to one of the preceding claims, characterised in that the processing unit is configured to assign the measured magnetic and electrical signatures to a configuration and / or movement and / or a force with which the movement is carried out with the assistance of Bayes classifiers, an artificial neural network, a support vector machine, a finite state machine, a hidden Markov model, a relevance vector machine, a dynamic time warping method, a conditional random field method, a decision tree, a random forest method, a k-nearest neighbours algorithm, a discriminant analysis, a linear regression, a logistic regression, a Gaussian process, a perceptron and / or ensemble methods (Bayes optimal classifier, bagging, boosting).

11. The human-machine interface according to one of the preceding claims, further comprising a communication unit for data exchange with at least one external device which, based on the data exchanged with the human-machine interface (10), is able to carry out an action that corresponds to the at least one assigned configuration and / or movement and / or a force with which the movement is carried out.

12. A method for detecting a deformation of the surface of a skin area of a human or animal body and assigning the detected deformation to a configuration and / or movement and / or a force with which the movement is carried out, with the steps of: - arranging a first plurality of magnetic or magnetisable elements on or near a skin area, such that the first array is able to at least substantially follow a skin deformation, - arranging a second plurality of potential-measurement surface electrodes such that the potential-measurement surface electrodes come into direct contact with the surface of the skin area, - determining at least one measurable change in a magnetic signature of the totality of the first plurality of magnetic or magnetisable elements resulting from skin deformation by a detection device for measuring the magnetic signature, - recording the measurement signals that are generated by the totality of the second plurality of potential-measurement surface electrodes at a specific point in time to obtain at least one electrical signature that varies as a function of muscle activity, and - executing an algorithm for assigning the determined at least one magnetic signature and the obtained at least one electrical signature to a configuration and / or movement and / or a force with which the movement is carried out.

13. The method according to claim 12, characterised in that the algorithm is a trained artificial intelligence that has ascertained or learned the relationship between magnetic and electrical signatures and associated configurations and / or movements and / or a force with which the movement is carried out on the basis of a data set consisting of at least one magnetic and electrical signature and the associated configuration and / or movement and / or a force with which the movement is carried out.

14. The method according to claim 12, characterised in that the magnetic and electrical signatures are assigned to a configuration and / or movement and / or a force with which the movement is carried out with the assistance of Bayes classifiers, an artificial neural network, a support vector machine, a finite state machine, a hidden Markov model, a relevance vector machine, a dynamic time warping method, a conditional random field method, a decision tree, a random forest method, a k-nearest neighbours algorithm, a discriminant analysis, a linear regression, a logistic regression, a Gaussian process, a perceptron and / or ensemble methods (Bayes optimal classifier, bagging, boosting).

15. The method according to one of claims 12 to 14, characterised in that the magnetic signatures to be measured are detected by several magnetometers that are preferably arranged in a shape of an array.

16. The method according to claim 15, characterised in that, during the measurement of magnetic signatures, magnetic field strengths and / or magnetic flux densities in the x and / or y and / or z direction are detected.

17. The method according to any one of claims 12 to 16, characterised in that the deformation of the surface of the skin area is based on an involuntary or voluntary muscle, tendon, ligament, bone and / or organ movement.

18. The method according to any one of claims 12 to 17, further characterised by the step of determining accelerations and / or rotation rates in addition to determining magnetic signatures.

19. The method according to any one of claims 12 to 18, further characterised by the steps - allocating a command with an assigned configuration and / or movement and / or a force with which the movement is carried out, and - transmitting the command to a device.