Wearable human-machine interface and possible methods using same

The wearable human-machine interface addresses the challenge of intuitive device control by using a flexible magnetic layer with magnetometers and algorithms to recognize skin deformations, providing accurate gesture recognition and diagnostic capabilities.

JP7596313B2Active Publication Date: 2024-12-09ヒューメインソリューションズウーゲー(ハフトゥングスベシュレンクト)
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2021571353
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2019-05-28
Publication Date
2024-12-09
Estimated Expiration
2039-05-28

AI Technical Summary

Technical Problem

Existing human-machine interfaces are not intuitive and efficient for controlling electronic devices, especially in wearable forms, lacking the ability to accurately detect and interpret voluntary and involuntary deformations of the skin surface for seamless interaction.

Method used

A wearable human-machine interface with a flexible layer having magnetic properties that conforms to the skin, equipped with magnetometers and optional sensors, processes changes in magnetic properties to recognize structures and movements, using algorithms like CNNs to assign these deformations to control commands.

Benefits of technology

Enables accurate and intuitive control of devices by interpreting skin deformations, allowing seamless interaction through gestures and movements, enhancing usability and enabling diagnostic applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007596313000002
    Figure 0007596313000002
  • Figure 0007596313000003
    Figure 0007596313000003
  • Figure 0007596313000001
    Figure 0007596313000001
Patent Text Reader

Abstract

A wearable human-machine interface and possible methods using the same. The present invention provides a wearable human-machine interface (10), comprising: a wearable element (12) configured to be attached to a skin area of ​​a human or animal body; a flexible layer (26) with magnetic properties attached to the wearable element on the side of the human or animal body facing the skin area so as to contact the surface of the skin area, the layer (26) with magnetic properties being such that the layer (26) can at least substantially follow deformations of the surface of the skin area, the magnetic properties of the layer (26) being configured such that deformations of the surface of the skin area result in a measurable change in the magnetic properties; a detection unit for measuring the change in the magnetic property of the flexible layer (26); a processing unit adapted to execute an algorithm for assigning the measured changes in magnetic properties to structure and / or movement and / or movement intensity; It is characterized by comprising: The present invention also relates to a method that can be implemented in this human-machine interface.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a novel human machine interface and to a method operable in such a human machine interface. [Background technology]

[0002] Today, due to the increasing interconnectedness and digitalization of the environments in which humans and animals live, it is becoming increasingly necessary for communication to take place between humans (or animals) and electronic devices of all kinds. A key word that may be mentioned here is the term Internet of Things (IoT), which is commonly understood to mean that a connection with multiple devices is or will be possible via the Internet, and that these devices can be operated or controlled by a user. Examples of such devices that may be mentioned, without claiming completeness, include vehicles of all kinds, domestic appliances, heating systems in buildings, fitness trainers, gaming devices, drones, robots, etc. To operate or control these and other devices, intelligent human-machine interfaces are needed to make the use of such devices as easy and intuitive as possible for humans (or animals).

[0003] The underlying objective of the present invention is to provide such a human-machine interface, which is preferably wearable. Summary of the Invention

[0004] In order to achieve the above object, a preferably wearable human-machine interface is proposed, which comprises a wearable element adapted to be attached to a skin area of ​​the human or animal body. Any element suitable for fastening, preferably removably, the human-machine interface to a desired skin area of ​​the human or animal body can be used as wearable element. The human-machine interface according to the invention further comprises a flexible layer with magnetic properties attached to the element worn on the human or animal body on its side facing the skin area so as to be in contact with the surface of the skin area. The flexibility of the layer with magnetic properties must be such that this layer can at least substantially follow the deformations of the surface of the skin area. In the context of the present invention, "magnetic properties" is understood to mean that the magnetic properties of the layer are adapted such that a deformation of the surface of the skin area, followed by a deformation of the flexible layer with magnetic properties, results in a measurable change in the magnetic properties. The human-machine interface according to the invention further comprises a detection unit for measuring the change in the magnetic properties of the flexible layer. In other words, the detection unit is adapted to measure, at least relative to an initial state, the change in the magnetic properties of the flexible layer caused by the deformation of the surface of the skin area. Finally, the human-machine interface according to the invention comprises a processing unit adapted to execute algorithms for assigning the changes in the measured magnetic properties to structure and / or movement and / or movement intensity.

[0005] The cause of the deformation of the surface of a skin area can be voluntary or involuntary. For example, a person can voluntarily move a limb to a different position, which causes a corresponding deformation of a particular area of ​​the skin. On the other hand, an involuntary deformation of a particular area of ​​the skin can occur due to, for example, a person's disease (e.g., through involuntary muscle contraction or organ movement), the detection and allocation of which can allow a physician to make a better diagnosis.

[0006] In the context of the present invention, the term "configuration" refers to the totality of joint angles necessary to describe the degrees of freedom of the limb in question. In other words, "configuration" refers to a particular position of a human or animal limb (examples mentioned here are extended or bent legs, open or closed hands, extended or bent arms, hands twisted or not twisted at the elbow, etc.). Of course, a configuration can also represent intermediate states between two extreme configurations, for example, a slightly extended arm, a slightly closed hand, etc.

[0007] In the context of the present invention, the term "motion" refers to the change in structure over time (especially with regard to size and speed).

[0008] In the context of the present invention, the expression "intensity of exercise" means in particular the force with which an exercise is performed.

[0009] Generally speaking, voluntary or involuntary muscle, tendon, ligament, bone and / or organ movements of the human or animal body result in deformations of the surface of the skin area. The aim of the invention is to detect such deformations and reliably assign them to specific conditions (structures) and / or movements, and optionally to movement intensities.

[0010] The above-mentioned processing unit belonging to the human machine interface can be integrated or separated. "Integrated" here means that the processing unit is located directly in the wearable human machine interface. However, the processing unit can also be located remotely from the wearable human machine interface and communicate with the human machine interface, for example, via a wired or wireless connection.

[0011] The flexible layer with magnetic properties can take various forms. However, it is important that it conforms sufficiently to the surface of the skin area of ​​interest, so that the resulting deformation of the skin surface and the associated deformation of the flexible layer result in a measurable change in the magnetic properties. More precisely, it is preferred to measure the change in magnetic field strength and / or the change in magnetic flux density. In other words, the magnetic field generated in each case is measured, whereby the existing magnetic field itself does not necessarily have to change, but only the deformation of the skin has to form a different magnetic field as seen by the detection unit. In this regard, it is advantageous if the flexible layer comprises a pole structure with many N-S pole pairs. In this way, the displacement or twisting of the flexible layer with magnetic properties makes it possible to perform a magnetization that is measurable from the outside by measuring the magnetic field. The key element here is the pole structure with the magnetization strength in question, so that the change in magnetic flux density and / or magnetic field strength can be measured with a magnetometer.

[0012] According to a preferred embodiment of the human-machine interface according to the invention, the flexible layer comprises a plurality of magnets, preferably arranged in the form of an array, where, as already mentioned, a measurable magnetic property of the flexible layer is to be understood as a measurable change in magnetic field strength and / or a measurable change in magnetic flux density. The term "magnet" here means a single permanent magnet, but also a conductor through which a current flows, or a magnetizable particle or a group of magnetizable particles (which can be arranged, for example, on a magnetic tape or a magnetic strip). It is only important that each "magnet" produces a magnetization that is effective externally and that can be detected by a detection unit. A flexible layer with magnetic properties and an advantageous pole structure can be realized, for example, by setting commercially available permanent magnets with a simple N-S pole structure in a strip-shaped elastic silicone layer. Of course, many other configurations are conceivable and are equally suitable.

[0013] The flexible layer with magnetic properties does not have to be separate from the worn element. In an advantageous embodiment of the human-machine interface according to the invention, the worn element and the flexible layer with magnetic properties are in fact suitably combined in one component. Such a component combining both elements can for example be in the form of a strap, as is also used in watches.

[0014] The detection unit of the human-machine interface according to the invention preferably comprises a number of magnetometers for measuring the changes in the magnetic properties of the flexible layer, which are advantageously likewise arranged in an array, which does not necessarily correspond to the array in which the magnets are arranged, nor does the number of magnetometers necessarily correspond to the number of magnets contained in the flexible layer.

[0015] In addition to measuring the changes in the magnetic properties of the flexible layer, it may be advantageous to obtain further information occurring during the deformation of the surface of the skin area of ​​interest. For this reason, in preferred embodiments of the human-machine interface according to the invention, one or more acceleration and / or rotational speed sensors are further present, the output signals of which may likewise be fed to a processing unit.

[0016] Based on the changes in the detected magnetic properties of the flexible layer resulting from the deformation of the surface of the skin area of ​​interest, the processing unit assigns the changes in the measured magnetic properties to structures and / or movements and / or intensities of movements. Based on the assigned structures and / or movements and / or intensities of movements, the processing unit or a downstream control unit can then generate, for example, control commands, which are then provided to the device for performing corresponding actions. Thus, the actions performed by the device correspond to at least one structure and / or movement and / or movement intensity assigned by the processing unit or the assignment algorithm. The assigned structures and / or movements and / or movement intensity can be, for example, gestures.

[0017] For example, the assignment of the changes in the magnetic properties of the flexible layer detected by the detection unit, for example by the measurements of a magnetometer, to the structure of the human limb can be advantageously achieved with the aid of artificial intelligence. The processing unit of the human-machine interface according to the invention is therefore preferably adapted to assign the changes in the measured magnetic properties of the flexible layer to structures and / or movements and / or movement intensities with the aid of Bayesian classifiers, artificial neural networks, support vector machines, finite state machines, hidden Markov models, relevance vector machines, dynamic time warping methods, conditional random fields methods, decision trees, random forest methods, k-nearest neighbor algorithms, discriminant analysis, linear regression, logistic regression, Gaussian processes, perceptrons and / or ensemble methods (Bayesian optimal classifiers, bagging, boosting), optionally additionally including data provided by rotational speed and / or acceleration sensors. In general, it can be said that a data set consisting of a particular magnetic property and any structures and / or movements associated therewith is first recorded and then the relationship between these parameters is determined in a so-called training. Therefore, a distinction must be made between learning and training modes and sufficient measurement data must be acquired to subsequently enable a correct assignment of a single magnetometer measurement or multiple magnetometer measurements to structures and / or movements and / or movement intensities in a prediction mode.

[0018] In an advantageous embodiment of the human-machine interface according to the invention, a communication unit is further provided for exchanging data with at least one external device which is able to perform an action corresponding to at least one assigned structure and / or movement and / or movement intensity based on data exchanged with the human-machine interface. For example, the human-machine interface according to the invention can be integrated into a smartwatch such that the functions of the smartwatch can be performed with the aid of the human-machine interface according to the invention. For example, it is conceivable that specific finger or hand movements can be assigned to specific functions or control commands of the smartwatch, and that performing these movements allows easy and intuitive operation of the smartwatch.

[0019] The invention also relates to a method for detecting deformations of a human skin area or an animal body surface and for assigning the detected deformations to a structure and / or a movement and / or a movement intensity, the method comprising the following steps:

[0020] Placing an array of magnets on or near the skin area so as to follow deformations of the skin.

[0021] Determining at least one magnetic characteristic of the array of magnets by measuring a change in a measurable magnetic property of the array of magnets resulting from deformation of the skin.

[0022] Executing an algorithm for assigning the determined at least one magnetic characteristic to a structure and / or a movement and / or a movement intensity.

[0023] As previously mentioned, the deformation of the surface of the skin area can be based on the movement of involuntary or voluntary muscles, tendons, ligaments, bones and / or organs.

[0024] As mentioned above in connection with the human machine interface according to the invention, the allocation algorithm is preferably a trained artificial intelligence which determines or learns the relationship between the magnetic feature and the associated structure and / or movement and / or movement intensity on the basis of a data set consisting of at least one magnetic feature and the associated structure and / or movement and / or movement intensity. As allocation algorithm it is preferable to use the algorithms already mentioned above in connection with the human machine interface.

[0025] The expression "magnetic signature of the magnet array" refers to the totality of changes in the magnetic properties of the magnet array to be measured or measured, which changes are the result of deformations occurring on the surface of the skin area of ​​interest. Another magnetic signature describing the initial state before the skin deformation occurs can be used as a reference. In particular, the signal conditioning or signal initialization can be performed based on a predefined reference structure before starting to detect the changes in the magnetic signature. For example, such a reference structure allows to zero all measured signals and the changes in the magnetic signature can be measured starting from this reference structure. According to an exemplary embodiment, each magnetic signature is composed of a measured magnetic field strength and / or a measured magnetic flux density of the magnet array.

[0026] Thus, in the present invention, the change in the measured magnetic properties of the magnetic layer or magnet array, which is a result of the deformation of the skin, constitutes the desired signal. Other changes, which may be caused by external disturbing variables, such as the Earth's magnetic field or magnetic objects, must be excluded from the measured signal if necessary. For example, if the human-machine interface has magnetic components, these can be permanently compensated for by a so-called hard-soft-iron calibration, which requires a calibration to be performed only once for the specific human-machine interface. The Earth's magnetic field at the place of use of the human-machine interface can also be permanently compensated for by such a hard-soft-iron calibration. In order to cancel other external interference sources, one or more magnetometers of the human-machine interface can be positioned to measure only the interfering magnetic fields. In this way, the interference sources can be canceled out when magnetometers are used to determine the changes in the magnetic properties of the magnetic layer or magnet array, which are a result of the deformation of the skin.

[0027] As mentioned above in connection with the human-machine interface, the magnetic field strength to be measured and / or the magnetic flux density to be measured are advantageously detected by a plurality of magnetometers preferably arranged in the form of an array (but not necessarily corresponding to the array in which the magnets are arranged), whereby the magnetic field strength to be measured and / or the magnetic flux density to be measured are preferably detected in the x-direction and / or the y-direction and / or the z-direction. In preferred exemplary embodiments of the method according to the invention and the human-machine interface according to the invention, the detection of the mentioned parameters takes place in the x-direction, the y-direction and the z-direction.

[0028] In addition to determining the magnetic signature, embodiments of methods according to the present invention may comprise determining the acceleration and / or rotational speed.

[0029] As already explained, the method according to the invention optionally comprises the steps of allocating control commands to the allocated structure and / or movement and / or movement intensity, and of transmitting the control commands to a device, which may form a unit with the human machine interface according to the invention or may be separate from the human machine interface. The transmission of the control commands to the device may be done in a wired or wireless manner, for example involving the use of the Internet.

[0030] Exemplary embodiments of the human-machine interface according to the invention and of the method according to the invention are explained in more detail below with reference to the accompanying schematic drawings, in which: FIG. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 is a perspective view of an exemplary embodiment of a human-machine interface from an oblique top view. [Diagram 2] FIG. 2 is a side view of the human-machine interface of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] Figures 1 and 2 show diagrammatically a human-machine interface, generally numbered 10, used to carry out the method according to the invention, which will be explained in more detail below. In the exemplary embodiment shown, the human-machine interface 10 resembles a smartwatch, here having a rectangular and hollow casing 11 at the top, generally forming a frame surrounding a recess 12 serving to receive a smartwatch module (not shown) or a microcontroller and / or microprocessor (not shown). A strap-like wearing element 13 allows the human-machine interface 10 to be fixed to the wrist of a user. The wearing element 13 can be adjusted in the manner of a watch strap in order to be able to adapt to different wrist diameters.

[0033] In the lower region, the wearable element 13 has three segments 14, 16 and 18, which are articulated and connected to one another and serve in each case to receive a 3x3 array of magnetometers 20. In this exemplary embodiment, each magnetometer 20 comprises three non-collinear Hall effect sensors, which allow the measurement of the magnetic flux density in the x, y and z directions. Each magnetometer of the 3x3 array is here configured as a printed circuit board 21 with 3x3 sensor chips 20a on which the magnetometers 20 are arranged. Each sensor chip 20a also comprises, in addition to the magnetometer 20, an acceleration sensor and a rotation speed sensor, thus allowing a 9-axis measurement. However, here only the magnetometer function of each sensor chip 20a has been used. Sensor chips of the above type and configuration are commercially available. Instead of the commercially available sensor chips used here, it is also possible to use pure magnetometers, which would allow the magnetometer array to be smaller or to obtain more magnetometer measurements with the same size.

[0034] The magnetometer 20 is positioned inside the wearable element 13 and therefore faces the skin surface of the wrist on which the human-machine interface 10 is worn.

[0035] A band-like flexible layer 26 made of silicone rubber is attached to the wearable element 13 by two interlocking connection elements 22, 24, in which three segments 14, 16 and 18 are shown holding the magnetometer 20, and in which an array of 4 x 12 magnets 28 is embedded. The silicone rubber used in the exemplary embodiment described here is transparent. The flexible layer 26 with the embedded magnets 28 rests on the underside of the user's wrist so that it conforms to the skin. The magnetometer 20 and the magnets 28 are not connected by a physical medium, as is necessary, for example, in the case of force or pressure sensors. The flexible layer 26 of silicone rubber is very elastic to minimize hysteresis phenomena, and at the same time is sufficiently flexible to be able to follow small deformations of the skin.

[0036] The magnets 28 used here are disc-shaped NdFeB neodymium permanent magnets with a diameter of 1.5 mm and a height of 0.5 mm. The magnetization was N50 and the direction of magnetization was axial, i.e. parallel to the height. The north pole of each magnet 28 was placed on the skin side, so that the south pole of each magnet 28 faced the magnetometer 20. The distance between the magnets 28 was 5 mm in each case.

[0037] The three articulated segments 14, 16 and 18, each of which here has dimensions of approximately 22 mm (width) by 29 mm (length), on which a magnetometer 20 is arranged together with a multiplexer, not shown here (the use of the multiplexer is optional and serves only to enable communication with a large number of sensors via a small number of data lines), form the detection unit of the human-machine interface 10, a microcontroller, already described but not shown here, which can be used, for example, as a microcontroller already present in a smart watch or other electronic device, or a microprocessor.

[0038] The detected magnetometer measurements are supplied by a microcontroller to a processing unit, not shown here, which, by means of a CNN implemented therein, can carry out an assignment of the measured magnetic characteristics to structures and / or movements and / or movement intensities. This processing unit can for example be integrated into a smartwatch, in which case it is an integral part of the human machine interface 10, but it can also be separate from the human machine interface and, for example, communicate wirelessly with the arrangement shown in the figures.

[0039] In the exemplary embodiment of the method according to the invention now to be described, as mentioned above, the magnetometer measurements can be assigned to a number of freely definable hand / finger configurations, for example in conjunction with gesture recognition by the human-machine interface 10.

[0040] The human-machine interface 10 was fastened to the user's left wrist like a wristwatch and magnetic signatures associated with specific hand / finger structures were measured using this human-machine interface 10. The user's hand, held flat, was used as the reference structure.

[0041] The human machine interface 10 used included a silicone layer with an array of 4x12 magnets embedded therein as the flexible layer with magnetic properties 26. The relative distance between the magnets 28 was 5mm.

[0042] To measure the magnetic characteristics of this magnet array, a magnetometer array having 9 × 3 magnetometers 20 and divided into three subarrays with 33 × 3 magnetometers was used.

[0043] To assign the magnetometer measurements provided by the human-machine interface 10 to the structure of the individual fingers of the left hand, an exemplary embodiment of an artificial neural network, the so-called Convolutional Neural Network (CNN), was used here. Each magnetometer 20 detected the magnetic fields acting on it in the x, y and z directions. Therefore, 81 measurements had to be processed per measurement step, that is, each magnetic feature was composed of 81 distinct values. Based on a large number of such measurements, the CNN in predictive mode is able to predict a certain class from the magnetometer measurements. Such classes here correspond to distinct gestures, whereby each gesture should have a different magnetic feature. Therefore, in the case described here, the assignment problem was formulated as a multi-class classification problem.

[0044] The use of CNN is not a condition for the method according to the invention. Alternatively, it is also possible to use, for example, support vector machines, random tree forests or further methods or algorithms already mentioned above. However, the use of CNN gives very good results and seems to be advantageous since CNNs can be very easily scaled for large amounts of data. Furthermore, the use of CNNs allows a particularly compact representation of the parameters to be learned, which is advantageous in terms of classification accuracy and resource efficiency, especially for the intended use in mobile applications and IoT (Internet of Things) applications. Furthermore, CNNs are robust to translation, rotation, scale and intensity variances, so that on the one hand a large pool of data can be generated for optimizing the free parameters in order to increase the effectiveness of the CNN, and on the other hand the generalization of the learned relationships between skin deformations and related structures can be easily transferred to several users and their individual characteristics.

[0045] CNN is generally one of the algorithms belonging to artificial intelligence, more precisely supervised learning by artificial neural networks. The magnetometer measurements serve as the input of the CNN, and any associated gestures are recorded or defined as the output of the CNN. A data set consisting of the magnetometer measurements (input of the CNN) and any associated gestures (output of the CNN) is recorded or prepared, and the relationship between these parameters is determined by optimizing the free parameters of the CNN by so-called training.

[0046] In the described exemplary embodiment, each training example consists of the qualitative structure of the hand, i.e. distinct classes encoded as so-called one-hot vectors, and the entire set of magnetometer measurements detected in x, y and z directions while preserving the qualitative structure of the hand.

[0047] The magnetometers 20 are arranged as a 3x9 array in the human machine interface 10 used, so that in each case, for one measurement, 27 support points are obtained for measuring the magnetic characteristics in the x, y and z directions. The magnetic characteristics of the flexible layer 26 resulting from the skin deformations, obtained from the individual magnetometer measurements, can therefore be collected as a geometric measurement of the skin deformation with 27 support points as a matrix for processing in the CNN. As already mentioned above, the training examples are completed by their respective labels indicating the relevant class, i.e. the relevant structure.

[0048] Thus, in the described exemplary embodiment, the training data set consists of a matrix X of dimensions (m, nC, nH, nW) where m is the number of training examples, nH is the number of magnetometers in the y direction (height of the magnetometer array), nW is the number of magnetometers in the x direction (width of the magnetometer array), and nC is the number of channels (x, y, z directions), and a matrix Y of dimensions (m, nG) where m is the number of training examples and nG is the number of gestures.

[0049] Each of the 81 measured values ​​of the magnetic signature was in each case standardized by its arithmetic mean and standard deviation (so-called Z-score standardization or Z-transformation) determined from the matrix X. These arithmetic means and standard deviations determined from the measured values ​​in the training mode can also be used later, i.e. when the training is completed, in the prediction mode of the signal adjustment of new measured values ​​of the magnetic signature for standardization.

[0050] In the context of this experiment, m=18,000 training examples (forming the training dataset), 3,375 test examples (forming the test dataset), 1,125 validation examples (forming the validation dataset), and nG=6 gestures were first studied to train the CNN. The CNN was trained using only the training dataset. The evaluation of the test dataset served to evaluate and fine-tune the CNN in order to achieve optimal performance in terms of prediction accuracy and generalizability of the CNN. The validation dataset is used at the end of the final evaluation to ensure that the CNN can indeed classify, i.e. assign, a previously unknown dataset.

[0051] The training of the CNN consists of so-called forward and backward propagation. Forward propagation represents the forward branch of the calculation, i.e. the prediction of the class based on the magnetometer measurements and the instantaneous parameters of the CNN. Then, using a cost function, the free parameters of the CNN are changed in the backward propagation, i.e. the reverse branch, so that the value of the cost function is reduced, in particular minimized. In the exemplary embodiment described here, the cost function was formulated as the so-called inter-category cross-entropy loss. Backward propagation is therefore an optimization method for determining the free parameters of the CNN. The CNN used was trained for 50 epochs, with a batch size of 16, i.e. the entire training dataset was passed to the CNN for 50 optimizations, whereby for reasons of numerical stability, 16 training datasets are always fed to the CNN simultaneously. The Adadelta optimization method was used with the following parameters: That is, the initial,adaptive learning rate lR=1.0, the Adadelta decay coefficient rho=0.95, the fuzz factor epsilon=1e-07, and the initial learning rate decay coefficient decay=0.0.

[0052] The CNN architecture used is shown in Table 1 below.

[0053] [Table 1]

[0054] To this end, the filter kernels of the convolutional layers and the fully connected layers were initialized by Glorot's uniform distribution (Xavier) method. Furthermore, the filter kernels of the convolutional layers were normalized by the L2 norm with weight coefficients of 0.001. Finally, the filter kernels of the convolutional layers included biases that were initialized to zero.

[0055] In conclusion, magnetometer measurements were classified into the correct classes with the following accuracy: training 99.34%, testing 99.53%, validation 99.29%.

[0056] In the described exemplary embodiment, in the predictive mode of the thus trained CNN, a single measurement step of all magnetometers in the human-machine interface is sufficient to identify a static gesture. However, it is possible to measure not only the static state but also the time-dependent changes of the magnetic signature, which would require multiple measurement steps. For the evaluation, so-called recurrent neural networks (LSTM [long short-term memory] or GNN [gate neural network]) can be used for the classification of gestures, i.e. for identifying continuous finger movements.

[0057] The wearable human-machine interface 10 is fixed by a strap like a wristwatch, so that the sensor module of the detection unit records the deformation of the skin of the wrist via the changes in magnetic flux density. More precisely, due to the movements of the hand and fingers, the skin is deformed by the movements of the muscles, ligaments, tendons and bones. The flexible layer 26 in which the magnets 28 are embedded follows this deformation, resulting in a relative displacement and rotation of the magnets 28 with respect to the magnetometers 20. These relative changes in magnetic flux density as a result of the movements of the hand and fingers can thus be reconstructed from the magnetometer measurements. The training phase is usually computationally intensive and is advantageously carried out on a powerful computer or processing unit optimized for artificial neural networks.

[0058] However, the trained CNN can make predictions in a very computationally efficient way. Due to the large number of data sets obtained in the training phase, not only a high accuracy of predictions is achieved, but also a high degree of robustness. The prediction mode, except for the process of signal conditioning, only involves so-called forward propagation, which makes the predictions very computationally efficient compared to training and can be carried out, for example, on smartphones and similar mobile devices. In the specifically described example, gestures can be identified very quickly by magnetometer measurements at 81 locations, depending on the sampling frequency, which are transmitted in one step via the CNN optimized by training. Control signals assigned to the identified gestures can then be transmitted directly to the device to be controlled by the processing unit or a downstream control unit.

Claims

1. a wearing element (13) adapted to be attached to a skin area of ​​a human limb or an animal body; a flexible layer (26) with magnetic properties attached to the wearing element on a side of the human or animal body facing the skin area so as to be in contact with the surface of the skin area, the layer (26) with magnetic properties being such that the layer (26) can at least substantially follow deformations of the surface of the skin area due to involuntary or voluntary movements of muscles, tendons, ligaments, bones and / or organs, the magnetic properties of the layer (26) being configured such that deformations of the surface of the skin area result in a measurable change in the magnetic properties; a detection unit for measuring the change in said magnetic property of said flexible layer (26); - an integrated or separate processing unit adapted to execute an algorithm for assigning the changes in the measured magnetic properties of the flexible layer (26) to a structure and / or a movement and / or a movement intensity with the aid of a Bayesian classifier, an artificial neural network, a support vector machine, a finite state machine, a hidden Markov model, a relevance vector machine, a dynamic time warping method, a conditional random field method, a decision tree, a random forest method, a k-nearest neighbor algorithm, a discriminant analysis, a linear regression, a logistic regression, a Gaussian process, a perceptron and / or an ensemble method (Bayesian optimal classifier, bagging, boosting), characterized in that the structure is a specific position of a limb of a human or an animal and the movement is a change in the structure over time; a communication unit for exchanging data with at least one external device capable of performing an action corresponding to the assigned structure and / or the exercise and / or the exercise intensity based on data exchanged with the human-machine interface (10); A wearable human-machine interface (10) comprising:

2. 2. A human-machine interface as described in claim 1, characterized in that the flexible layer (26) includes a plurality of magnets (28), preferably arranged in the form of an array, and the measurable change in the magnetic property of the flexible layer (26) is a measurable change in magnetic field strength and / or a measurable change in magnetic flux density.

3. A human-machine interface according to claim 1 or 2, characterized in that the wearable element (13) and the flexible layer (26) provided with magnetic properties are combined into one part.

4. A human-machine interface according to any preceding claim, wherein the detection unit includes a plurality of magnetometers.

5. 5. A human machine interface according to any one of claims 1 to 4, further comprising one or more acceleration sensors and / or rotational speed sensors provided in the human machine interface (10), the output signals of the sensors being supplied to the processing unit.

6. The human-machine interface according to any one of claims 1 to 5, characterized in that the human-machine interface is integrated into a smart watch.

7. 1. A method for detecting deformations on the surface of a skin area of ​​a human limb or an animal body and allocating said detected deformations to a structure and / or a movement and / or a movement intensity, the structure being a specific location of a human or animal limb and the movement being a change in the structure over time, - positioning an array of magnets on or near a skin area such that the array follows deformations of the skin due to involuntary or voluntary movements of muscles, tendons, ligaments, bones and / or organs; determining at least one magnetic signature of the magnet array by measuring a change in a measurable magnetic property of the magnet array resulting from deformation of the skin; - executing an algorithm for assigning said at least one determined magnetic characteristic to a structure and / or a movement and / or a movement intensity; Equipped with said algorithm being a trained artificial intelligence that has determined or learned a relationship between a magnetic signature and an associated structure and / or movement and / or movement intensity based on a data set consisting of at least one magnetic signature and said associated structure and / or said movement and / or said movement intensity; The magnetic features are assigned to structures and / or movements and / or movement intensities with the help of Bayesian classifiers, artificial neural networks, support vector machines, finite state machines, hidden Markov models, relevance vector machines, dynamic time warping methods, conditional random fields, decision trees, random forests, k-nearest neighbor algorithms, discriminant analysis, linear regression, logistic regression, Gaussian processes, perceptrons and / or ensemble methods (Bayesian optimal classifiers, bagging, boosting), moreover allocating control commands to the allocated structures and / or movements and / or movement intensities; sending said control command to a device; executing, by the device, an action corresponding to the control command; 13. A method comprising:

8. 8. The method of claim 7, wherein each magnetic characteristic comprises a change in measured magnetic field strength and / or a measured magnetic flux density.

9. 9. The method according to claim 7, further comprising the step of: prior to measuring the change in the measurable magnetic properties of the magnet array resulting from the deformation of the skin, an initialization of the signals is performed based on a predetermined reference structure.

10. 10. Method according to claim 8 or 9, characterized in that the changes in magnetic field strength to be measured and / or the magnetic flux density to be measured are detected by means of a plurality of magnetometers, preferably arranged in the form of an array.

11. 11. The method according to claim 10, characterized in that the magnetic field strength to be measured and / or the magnetic flux density to be measured are detected in the x-direction and / or the y-direction and / or the z-direction.

12. 10. The method according to claim 9, characterized in that for signal initialization, the magnetometer is zeroed for each detection direction based on said predetermined reference structure.

13. 13. A method according to any one of claims 7 to 12, further comprising the step of determining an acceleration and / or a rotational speed in addition to the step of determining a magnetic signature.

Citation Information

Patent Citations

  • Biological information measuring apparatus

    JP2001112725A

  • Operation signal output unit and operation system

    JP2008027252A

  • Band-type device and method for controlling detection of the same

    JP2015114182A

  • Method and apparatus for combining myoelectric sensor signals and inertial sensor signals for gesture-based control

    JP2016507851A

  • Plusimeter sensor and method for controlling the same

    KR1020180028861A