DEVICE AND METHOD FOR DETECTING AND PREDICTING BODY MOVEMENTS

Description

TECHNICAL AREA

[0001] The application relates to a system for detecting and predicting a user's movements, comprising a detection area with at least one sensor, a modeling area, and an actuator, wherein the sensor is capable of acquiring measurement data about the user's body state and transmitting it to the modeling area. The application also relates to methods for detecting and predicting a user's movements. STATE OF THE ART

[0002] There are numerous complex movement sequences of a user's body that require a high degree of body control and necessitate correspondingly intensive training to achieve. A variety of technical devices and methods exist for monitoring and improving the training effect, enabling the acquisition and processing of measurement data and the provision of feedback on the body's current movement status.

[0003] Methods for detecting and predicting movements are known from the following publications: WO-2020 / 102693-A1, US-2017 / 061817-A1, US-2017 / 123487-A1, US-2017 / 238812-A1, US-2016 / 038083-A1, and WU YU ET AL: "A Human-Machine Interface Using Electrical Impedance Tomography for Hand Prosthesis Control", IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS, IEEE, US, Vol. 12, No. 6, December 2018, ISSN: 1932-4545, DOI: 10.1109 / TBCAS.2018.2878395.

[0004] However, the possibilities for analysis and resulting training instructions based on the visualization of current movement states are limited. While visualization of the movement state can be performed without significant time delays, drawing conclusions about the training program requires a separate evaluation. TASK OF INVENTION

[0005] Based on this, the object of the present invention is to provide a system and method which collect measurement data on the body condition of a user and process it in such a way as to enable a latency-free and reliable provision of training instructions or training data. TECHNICAL SOLUTION

[0006] This problem is solved by a system according to claim 1 and a method according to claim 11.

[0007] The described system for detecting and predicting user movements comprises a detection area with at least one sensor, a modeling area, and an actuator. The sensor is capable of acquiring measurement data about the user's body state and transmitting it to the modeling area. The acquisition of the user's body state is performed using EIT (electrical impedance tomography) and / or EMG (electromyography) and / or UWB (ultra-wideband) real-time localization. The modeling area is capable of creating a data processing model based on the measurement data using machine learning to calculate the user's current and / or future body state. The actuator is capable of acting on the user's body and / or on a device used by the user, based on the data processing model.

[0008] The detection zone serves as the user level, where the sensor, either alone or as part of a sensor system, is compactly attached to and integrated with the user and / or an interaction object, or within their measurement environment. Data acquisition, i.e., the collection of measurement data, takes place within the detection zone.

[0009] Even during data acquisition, the measurement data can be reduced or minimally preprocessed (e.g., filtering, compression, fusion). For example, measured values ​​that lie outside a permissible range of relevant measurement data and / or that can be disregarded from the outset can be ignored during the measurement process. The goal here is to reduce the size of the measurement data. This involves compressing the measurement data to reduce its volume.

[0010] The data is transferred after the modeling area.

[0011] In the modeling area, the measurement data is used to create a data processing model through machine learning.

[0012] Machine learning, a subfield of artificial intelligence, is the generation of knowledge from experience by a machine-based, artificial system. This system learns based on existing datasets and examples, and can generalize these after the learning phase. To achieve this, machine learning algorithms build a statistical model based on training data—in this case, the provided measurement data. This means that the system doesn't simply memorize examples, but rather recognizes patterns and regularities in the training data. This allows the system to evaluate even unknown data (learning transfer).

[0013] Machine learning can be performed through analysis in Convolutional Neural Networks (CNNs).

[0014] A Convolutional Neural Network (CNN or ConvNet) is an artificial neural network. It is a concept inspired by biological processes for the machine processing of image or audio data. The result of the machine learning process is a profiling and anticipation model, which is used to calculate motion data sets within the system.

[0015] In principle, movements or corresponding measurement data of the user are thus recorded via the sensor / sensor system and profiled by the modeling area, so that future movements can also be predicted on the basis of the data processing model.

[0016] The user's physical condition can be assessed using CS (compressed sensing).

[0017] To prevent large amounts of (measurement) data from growing too large, data is often stored in compressed form to reduce the storage space required for archiving. One common standard for image data compression is ISO / IEC 10918-1, often referred to as the JPEG standard (Joint Photographic Experts Group). In principle, lossless compression (JPEG Lossless Mode) is possible according to the JPEG standard.

[0018] The measurement data can be reduced during acquisition, that is, immediately upon taking the raw data. Such methods are called CS (Compressed Sensing or Compressed Sampling). CS methods combine a compression step with the data acquisition step by collecting—as far as possible—only relevant data of the object under observation. With CS methods, the compression performed during sampling ensures that very high image quality is maintained despite significant undersampling.

[0019] A special feature of data acquired using CS methods is that the acquired input data can be smaller than the further processed, unpacked output data, which replaces the usually compressed output data.

[0020] The actuator can be any device suitable for acting on the user's body and / or on a device used by the user, based on the data processing model. This can occur through direct influence, e.g., through haptic signals, or indirectly, e.g., through optical or acoustic signals.

[0021] In a preferred embodiment, EIT is used to detect muscle activity and / or the condition of body tissue, and detection by means of EMG is used to detect muscle activity and / or the condition of body tissue, and UWB real-time localization is used to detect joint positions and / or the spatial arrangement of the user's bones.

[0022] Electrical conduction studies (EIT) are an imaging technique based on measurements resulting from changes in impedance and capacitance within the human body. Compared to other tomographic methods, such as CT scans (X-rays) and PET scans (gamma rays), EIT is non-invasive. The sensors rest on the skin without the need for conductive gel, for example. EIT is non-invasive and suitable for continuous operation (e.g., low voltage, no ionizing radiation). EIT is used in medicine, for example, to assess cardiac function, pulmonary hypertension, and regional lung function.

[0023] In tomographic methods, the internal structure and, if applicable, the composition of objects are determined by using external signals and observing their changes over time. EIT uses surface electrodes and high-frequency alternating current signals to measure the internal electrical impedance. By using multiple electrodes, an internal impedance distribution can be determined. Since different materials within an object respond differently to signals, the cross-sectional internal structure of the object can be reconstructed based on the variations in the impedance distribution.

[0024] Several surface electrodes are positioned on the skin around a specific body region. High-frequency, low-amplitude alternating currents flow between each pair of electrodes, while the electrical potential is simultaneously recorded using the other electrodes. Repeated measurements with any variation in the stimulation electrode pair generate a cross-sectional image (tomogram) from which conclusions can be drawn about the tissue composition within the examined body region.

[0025] In this way, several individual images are created from different layers of the object being viewed and combined to form a detailed three-dimensional overall image.

[0026] EMG is an electrophysiological method in which electrical muscle activity is measured based on action currents of the muscles and, if necessary, displayed (graphically).

[0027] Using concentric needle electrodes or electrodes placed on the skin, the potential fluctuations of individual motor units and the skin can be recorded with surface electrodes.

[0028] UWB is an ultra-wideband technology and is a form of short-range radio communication using extremely large frequency ranges with a bandwidth of at least 500 MHz or at least 20% of the arithmetic mean of the lower and upper cutoff frequencies of the frequency band used.

[0029] Ultra-wideband (UWB) technology detects the position of the joints and thus the posture of the skeleton. Corresponding movements can also be detected from the joint position. UWB may be supported by a camera system (stereo camera).

[0030] A combination of the aforementioned methods according to the invention is EIT and UWB.

[0031] In a particular embodiment of the invention, the actuator is suitable for acting on the user's body by means of vibrations.

[0032] In this case, the actuator rests against the user's body to transmit the vibration directly. The vibration is particularly useful for alerting the user to, for example, a misalignment of the body or body parts. The vibration can be applied at varying intensities to indicate the degree of misalignment.

[0033] Preferably, the actuator includes a detector for detecting the position and / or movement of the actuator relative to the user's body and / or relative to the environment.

[0034] In addition to detecting muscle and tissue contractions, the position of the actuator itself can be determined in relation to the body and / or the environment. If multiple actuators are used, their positions can be compared to each other to capture relative positions and movements.

[0035] In addition to the UWB described above, an inertial measurement unit (IMU) can also be used.

[0036] IMU is a spatial combination of several inertial sensors such as e.g.

[0037] Accelerometers and gyroscopes. It represents the sensory measurement unit of an inertial navigation system and can capture the dynamics of movement.

[0038] The movements of the body or body parts are measured here.

[0039] Advantageously, the actuator is suitable for collecting measurement data about the condition of an area of ​​the user's body where the actuator is located.

[0040] The measurement can be performed using the aforementioned EIT, EMT, UWB and IMU methods.

[0041] In a particularly advantageous embodiment, the actuator is incorporated into and / or attached to a carrier, in particular a garment, which the user can wear on their body.

[0042] The actuator can also be designed as a so-called wearable in the form of a band, cuff or belt, which is positioned over the relevant area of ​​the body.

[0043] In an advantageous embodiment, the system includes a prediction area suitable for creating a motion data set based on a large number of measurement data and the data processing model.

[0044] The prediction area is typically a local computing instance with sufficient processing power to perform the essential processing steps for analyzing and predicting motion and creating a corresponding motion dataset. In the prediction area, the measurement data detected in the detection area is received, decompressed, and processed.

[0045] Machine learning can potentially be performed through analyses / implementations in convolutional neural networks (CNNs). In this case, the analyses or motion data sets are calculated in the prediction domain using the data processing model created by convolutional neural networks (CNNs).

[0046] The geographical proximity and dedicated resource provision make it possible to achieve the required low latencies between sensors and actuators.

[0047] In addition, the prediction area may be suitable for storing the data processing model.

[0048] The prediction area has appropriate storage media for this purpose, e.g. local storage such as hard drives or non-local storage, such as access to cloud storage.

[0049] Preferably, the prediction area is suitable for receiving and processing measurement data from the detection area.

[0050] Furthermore, the measurement data can be passed on to the modeling area to update the data processing model.

[0051] Advantageously, the prediction area is suitable for recognizing a user's movement sequence from measurement data received from the detection area.

[0052] In particular, a multitude of measurement data that track spatial changes in the body can be used to predict corresponding movement sequences. In addition to one or more movement sequences, gestures or user movements can also be recognized. For example, a contraction of a specific muscle group might indicate the lifting of an object, or a particular angle of the user's leg might suggest an upward or downward movement, such as jumping. Gesture recognition can support or simplify data processing within the data processing model and improve the predictive accuracy or reliability of movements.

[0053] The processing of the measurement data includes decompression of the measurement data in the prediction area.

[0054] In this process, the measurement data compressed by the detection area is decompressed again; for example, in the case of images (recording of muscle contractions), the image layers are reassembled.

[0055] Preferably, the prediction area is suitable for visualizing a whole or partial body model of the user and / or current and / or future movement sequences of the user from the movement data set.

[0056] Visualization can be performed internally within the system or externally. For example, muscle activity can be visualized on a device (such as a tablet).

[0057] In a particular embodiment, the system includes at least one actuator suitable for acting on the user's body and / or on a device used by the user, based on the motion data set.

[0058] The actuator serves to influence the user or their training device in order to trigger or initiate a specific user response. This can be done purely visually, showing the user how, for example, their body or body parts must be moved to achieve or optimize a desired movement. Alternatively, a user response can be triggered through direct influence, such as warning tones, buzzing, electrical or mechanical impulses, etc., so that they, for example, correct a physical misalignment.

[0059] In addition, the actuator can also measure the current state (actual state) of the body and provide corresponding feedback to the prediction area. In this way, the body's state is continuously monitored, for example, to ensure a specific posture is maintained.

[0060] An actuator's influence on the sports equipment used by the user can, for example, consist of the sports equipment supporting or counteracting a desired or predicted movement of the user, depending on the training goal.

[0061] The invention further comprises a method for detecting and predicting a user's movements towards the object, comprising the following method steps: a) Provision of a system according to the invention. b) Acquisition of measurement data by means of measurements using at least one sensor on the user's body and / or by means of the actuator, optionally also by means of measurements on a sports device in use. c) Forwarding the measurement data to the modeling area. d) Creation of a data processing model based on the measurement data using machine learning. e) Action on the body and / or feedback by means of the actuator.

[0062] Machine learning can be performed through analysis in Convolutional Neural Networks (CNNs).

[0063] As described above, CNN is a concept for the machine processing of image or audio data.

[0064] Another method for detecting and predicting a user's movements includes the following procedural steps: a) Provision of a system according to the invention. b) Acquisition of measurement data by means of measurements using the sensor on the user's body and / or by means of the actuator, optionally also by means of measurements on a sports device in use. c) Forwarding the measurement data to the prediction area. d) Creation of a motion data set using the data processing model. e) Processing and / or storage and / or display of the motion data set in the prediction area. f) Action on the body and / or feedback by means of the actuator.

[0065] Preferably, the procedure includes the further procedure step: g) Adapting the data processing model using the measurement data forwarded to the modeling domain.

[0066] By adapting the data processing model, a continuous update and improvement of the data processing model can take place.

[0067] The acquisition of measurement data in the detection area to record the user's body condition is carried out using EIT (electrical impedance tomography) and / or EMT and / or UWB.

[0068] Using EIT (as already described in detail above), several individual images from different layers of the part of the body under consideration are created by measuring impedance or capacitance changes between measuring points, and a detailed overall image is created from these.

[0069] EMG measures electrical muscle activity based on action currents of the muscles and can be displayed (graphically) if necessary.

[0070] UWB is a form of short-range radio communication using large frequency ranges.

[0071] Preferably, the method includes the further process step: action of the actuator of the provided system on the body of the user and / or on a device used by the user, e.g. a sports device, based on the motion data set.

[0072] The actuator serves to implement the movement model by influencing the user's body accordingly. This can be done purely visually, showing the user how, for example, the body or body parts must be moved to achieve a desired movement. Or it can be done through direct intervention, such as warning tones, buzzing, etc., to alert the user to a physical misalignment. BRIEF DESCRIPTION OF THE FIGURES

[0073] Figure 1 Schematic representation of the individual components of the system according to the invention; Figure 2 Representation of the arrangement of sensors on the upper and lower arm of a user; Figure 3 Schematic representation of the point arrangement of one or more actuators on the body of a user; Figure 4A Schematic representation of the point arrangement on the body of a user according to Figure 3 With regard to a coordinate system for determining the position of one or more actuators, Figure 4B shows a schematic representation of the determination of joint movements based on the point arrangement on the body of a user according to Figure 4A . DESCRIPTION OF A PREFERRED EXECUTION EXAMPLE

[0074] The invention will be explained in more detail using the following exemplary embodiments.

[0075] Out of Figure 1It becomes apparent that the system 1 according to the invention is basically divided into three parts, namely User 3, Mobile-Edge Cloud 4 and Cloud-Backend 5, wherein User 3 corresponds to the detection area, Mobile-Edge Cloud 4 to the prediction area and Cloud-Backend 5 to the modeling area.

[0076] In detection area 3, which is the user level, the sensor 31 / sensor is attached as compactly as possible to the user 2 themselves and / or to an interaction object and / or integrated into the latter. The sensor 31 uses EIT (electronic impedance tomography) to acquire measurement data 6 about the user 2's body, e.g., through pressure mapping 311 or muscle contractions 312.

[0077] During data acquisition, multiple individual images from different "viewpoints" are layered to create a detailed three-dimensional (3D) image. This method is used, for example, to determine the weight distribution of a surfer on their surfboard. A cross-sectional image of muscles (e.g., the thigh muscles) is generated using a cuff, preferably attached to the user's legs and / or arms. This allows muscle contractions to be measured and inferences to be made about movements. The number of images generated is quite substantial, and the resulting data volume is correspondingly large. Therefore, the sensor values ​​are acquired using a "compressed sensing" method, which significantly speeds up data acquisition because the data is already measured in a compressed format.

[0078] In addition to pressure display 311 or measurement of muscle contraction 312, the measurement of certain keypoints 313 of the user's body is also possible, either concurrently or cumulatively. The (joint) position / arrangement of the extremities (e.g., arms, legs, torso, and head) is particularly important in this regard, as it allows conclusions to be drawn about the user's body position.

[0079] Measurement data 6 is recorded through the various measurements 311, 312, 313.

[0080] In detection area 3, only data acquisition and minimal preprocessing (filtering, compression for size reduction, fusion) take place. The measurement data 6 is transmitted to the Mobile-Edge Cloud 4 and / or the Cloud Backend Server 5.

[0081] In the cloud backend server, or modeling area 5, the transmitted measurement data 6 are collected and compiled into training datasets 62 for the creation of the machine learning models. As part of this training, the CNNs (convolutional neural networks) for the Mobile Edge cloud instances are created and updated there upon completion.

[0082] Using Machine Learning 7, a data processing model 8 is created, which is then forwarded to the prediction area / Mobile-Edge Cloud 4.

[0083] The central area, the processing area / Mobile-Edge Cloud 4, is a local computing instance with sufficient processing capacity to perform the essential processing steps for analyzing and predicting the movements of the user's body 2. In processing area 4, the measurement data 6 provided by detection area 3 is received, decompressed, processed, and the analyses (profiling and anticipation) are calculated using the profiling and anticipation model generated by convolutional neural networks (CNNs). During decompression, the image layers are combined and displayed on an end device 911 (e.g., a tablet) to visualize the body's activities.

[0084] In this way, the data processing model 8 provided by modeling area 5 is further developed into a motion data set 9 (profiling / anticipation). This can be visualized 91 and used to influence 92 the body of user 2 or the sports equipment via an actuator 10. It also visualizes predicted movements. The actuator 10 can act on the body of user 2, but also control it and provide corresponding feedback. Therefore, the actuator 10 is also located in detection area 3.

[0085] Due to the spatial proximity and dedicated resource provision, low latencies between sensors and actuators can be achieved.

[0086] Fig. 2 shows the arrangement of two sensors 31, one each located on the forearm and upper arm of a user,

[0087] Fig. 3Figure 10 shows the arrangement of one or more sensors / actuators 10 (in the form of so-called wearables) at points on the body (e.g., on the upper arm, forearm, and upper and lower leg). The wearables are in the form of bands that are positioned at the respective points 21 on the body of a user 2. Muscle and tissue activity is measured using electrical impulse transmission (EIT) and / or electromyography (EMG). The position of the joints can be detected using ultra-wideband (UWB). For example, if one actuator 10 is attached to each upper and lower arm, the position of the joint between the upper and lower arm can be measured, and thus the position and movement of the arm can also be measured. The actuators 10 can also provide the user with feedback, e.g., via vibrations, for example, if there is a misalignment or incorrect movement.

[0088] Fig. 4A The detection of joint position using actuators (wearable) 10 shows the joints at body points 21 according to the Fig. 3are attached. A wearable typically has an integrated IMU and a UWB chip. A three-point anchor system (in relation to the body or the environment) triangulates the absolute position of the wearable in three-dimensional space at adjustable intervals.

[0089] A reference system is established via the three anchor points 22 using the UWB's air interface, and a multilateration calculation is performed using TDoA (Time Difference of Arrival) to determine the X, Y, and Z coordinates of the wearable. This allows for the absolute position of each wearable to be determined at cyclical intervals (e.g., approximately every 5 seconds). Between these measurements, the relative movement of the limbs in space is measured by the IMUs mounted on the actuators 10. This ensures consistently reliable motion capture data for each limb equipped with a wearable. The anchor points can be positioned as needed, for example, directly on the body or at the edge of the playing field, etc.

[0090] Fig. 4B This shows the relative movement of the joints, represented by points 23, which are measured by the IMU. The IMU sensor drift is obtained using the UWB coordinate system.

Claims

1. System (1) for identifying and predicting movements of a user (2), comprising a detection unit (3) with at least one sensor (31), a modeling unit (5), a prediction unit (4), and an actuator (10), wherein the sensor (31) is capable of collecting measurement data (6) through the user's (2) physical condition and to transmit it to the modeling unit (5), wherein the detection of the user's (2) physical condition and / or a body region is performed using EIT (Electrical Impedance Tomography) and UWB (Ultra-wideband) real-time localization, wherein, for the detection of the physical condition and / or the body region by measuring the internal electrical impedance, a cross-sectional image of the tissue composition within the examined body region is generated within the impedance distribution, and multiple individual images from different layers of the object under examination are created and merged into a detailed three-dimensional overall image, and wherein the modeling unit (5) is capable of using machine learning to create a data processing model (7) based on the measurement data (6) for calculating a current and / or future physical state of the user (2), and wherein the actuator (10) is configured for acting on the user's (2) body and / or on a device used by the user (2), wherein the prediction unit (4) is configured to generate a motion data set (8) based on a plurality of measurement data (6; 61) using the data processing model (7), and is configured to visualize a full-body model of the user from the motion data set (8).

2. System (1) for identifying and predicting movements of a user (2) according to claim 1, wherein EIT is used to measure muscle activity and / or the state of body tissue, and the detection by means of EMG serves to record muscle activity and / or the state of the body tissue, and UWB real-time localization serves to record joint positions and / or the spatial arrangement of the user's bones.

3. System (1) for identifying and predicting movements of a user (2) according to claims 1 to 2, wherein the actuator (10) is configured for act on the user's body by means of vibrations.

4. System (1) for identifying and predicting movements of a user (2) according to any one of claims 1 to 3, wherein the actuator (10) comprises a detector for identifying the position and / or the movement of the actuator (10) relative to the user's body and / or relative to the environment.

5. System (1) for identifying and predicting movements of a user (2) according to any one of claims 1 to 4, wherein the actuator (10) is adapted to collect measurement data (61) regarding the condition of a region of the user's (2) body on which the actuator (10) is located.

6. System according to any one of claims 1 to 5, wherein the actuator (10) is incorporated into and / or attached to a carrier that can be worn on the user's body, in particular a garment.

7. System (1) for identifying and predicting movements of a user (2) according to claim 1, wherein the prediction unit (4) is configured for receive and process measurement data (6; 61) from the detection unit (3) and / or from the actuator (10).

8. System (1) for identifying and predicting movements of a user (2) according to claim 7, wherein the prediction unit (4) is configured for detect a sequence of movements of the user (2) based on measurement data (6; 61) received from the detection unit (3) and / or from the actuator (10).

9. System (1) for identifying and predicting movements of a user (2) according to any one of claims 7 to 8, wherein the prediction unit (4) is configured for visualize, from the motion data set (8), a complete or partial body model of the user (2) and / or motion sequences of the user (2).

10. System (1) for identifying and predicting movements of a user (2) according to any one of claims 1 to 10, wherein the actuator (10) is configured for act on the body of the user (2) and / or on a device used by the user (2) based on the motion data set (8).

11. Method for identifying and predicting movements of a user (2), comprising the following method steps: a) Providing a system (1) according to any one of claims 1 to10. b) Acquiring measurement data (6; 61) through measurements using at least one sensor (3) on the user's (2) body and / or using the actuator (10). c) Generating of multiple individual images by measuring changes in impedance from different layers of the part of the body being examined and layering these into a detailed three-dimensional composite image. c) Forwarding the measurement data (6; 61) to the modeling unit (5) or the prediction unit (4). d) Generating of a data processing model (7) based on the measurement data (6; 61) using machine learning. e) Generating of a motion data set (8) using the data processing model (7). f) Processing and / or storing and / or displaying the motion data set (8) in the prediction unit (4), g) Acting on the body and / or providing feedback via the actuator (10).

12. Method for identifying and predicting movements of a user (2) according to claim 11, comprising the further method step: a) Adapting the data processing model (7) using the measurement data (6; 61) forwarded to the modeling unit (5).

13. Method for identifying and predicting movements of a user (2) according to any one of claims 11 to 12, comprising the further method step: b) Acting the actuator (10) of the provided system (1) on the body of the user (2) and / or on a device used by the user (2) based on the motion data set (8).