Device and method for prediction of a user activity status

Abstract

The present invention relates to a method and a device (1) for obtaining a trained prediction model (31) for prediction of at least one activity status (51) of a user, the invention further relates to a method and a device for predicting at least one activity status of a user using a trained prediction model (31) obtained from the device (1).

Description

DEVICE AND METHOD FOR PREDICTION OF A USER ACTIVITY STATUSFIELD OF INVENTION

[0001] The present invention relates to artificial intelligence applied to a user ’ s behavior analysis. In particular, the invention relates to a method and a device for obtaining a trained prediction model for prediction of at least one activity status of a user. The invention further relates to a method and a device for predicting at least one activity status of a user using a trained prediction model obtained from the device or method described above.BACKGROUND OF INVENTION

[0002] Interpretation of human activities through the utilization of sensors embedded within wearable devices has garnered significant attention in recent years. In particular, discerning human activities based on data gathered from various sensors, including Inertial Measurement Units (IMUs), Electromyography (EMG) sensors, and pressure sensors. The fusion of data gathered from these sensors allows for the comprehensive analysis of human activities, facilitating applications such as healthcare monitoring, rehabilitation, and personalized assistance systems.

[0003] However, despite significant advancements in this field, challenges persist. Chief among these is the need to accurately analyze diverse activities in real-time. This necessitates robust algorithms capable of processing complex data streams promptly and effectively. Moreover, ensuring the reliability and precision of activity status recognition, particularly in rehabilitation contexts, presents a major obstacle. Conditions such as stroke, multiple sclerosis, and Parkinson's disease introduce additional complexities, demanding heightened accuracy and adaptability from activity status recognition systems.

[0004] Therefore, the invention endeavors to enhance the precision and adaptability of wearable sensor-based activity recognition. By refining algorithms and optimizing sensor data fusion techniques, the innovation aims to improve the reliability of activity prediction, thereby optimizing outcomes in healthcare and rehabilitation settings. Additionally, the invention seeks to reinforce real-time adaptability, enabling seamlessintegration into dynamic environments and facilitating personalized assistance adapted to individual user needs. Through these advancements, the invention aims to overcome existing limitations and drive the field of wearable sensor technology toward greater efficacy and utility in diverse applications.SUMMARY

[0005] This invention thus relates to a device for obtaining a trained prediction model for prediction of at least one activity status of a user, wherein the device comprises: at least one input configured to receive for each subject of a plurality of subjects, at least one recording previously obtained using at least one sensor, wherein during each recording, said subject is performing at least one activity; at least one processor configured to: o generate a training dataset by, for each recording of each subject of said plurality of subjects:■ extracting at least one feature representative of said activity status;■ labelling said recording using said at least one feature; o obtain said trained prediction model by feeding said training dataset to a trainable neural network, wherein the trained prediction model is configured to:■ receive as an input at least one recording of said user performing said at least one activity obtained using said at least one sensor; and,■ provide as output said prediction of said at least one user activity status; at least one output configured to provide said trained prediction model.

[0006] Advantageously, training the prediction model as described above allows to predict the activity status of the user based on sensor recordings. The device can recognize patterns indicative of different activity statuses. This enables real-time monitoring and assessment of the user activity status, facilitating applications such as personalized health tracking, fitness monitoring, and rehabilitation management. Additionally, the device'scapability to adapt to individual variations and diverse activity scenarios enhances its versatility and reliability in predicting user activity statuses across different contexts and conditions.

[0007] According to other advantageous aspects of the invention, the device comprises one or more of the features described in the following embodiments, taken alone or in any possible combination.

[0008] According to one embodiment, said at least one sensor is a wearable sensor configured to be worn by said plurality of subjects.

[0009] According to one embodiment, said at least one sensor is a wearable sensor.

[0010] Additionally, the sensor may be positioned at a predefined distance of the subject and capture the subject’s movements. For instance, such sensor may be a depth camera, a LiDAR sensor a Passive Infrared sensor, an ultrasonic sensor, a radar sensor, a doppler sensor, an infrared motion sensor, a thermal imaging camera and the like.

[0011] According to one embodiment, said at least one activity feature is extracted using at least one of a statistical analysis, a frequency domain analysis, and a time domain analysis.

[0012] According to one embodiment, said at least one recording comprises a first recording obtained using a first sensor and a second recording obtained using a second sensor, and wherein generating said training dataset further comprises aligning and synchronizing the first recording and the second recording.

[0013] According to one embodiment, said at least one recording is segmented into time windows of fixed size using a sliding window, the extraction of said at least one feature being performed on each time window.

[0014] According to one embodiment, generating said training dataset further comprises extracting at least two features and selecting one feature from the at least two features based on at least one of a correlation analysis, a feature importance ranking, and a domain expertise.

[0015] According to one embodiment, said trainable neural network is a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0016] According to the invention, Convolutional Neural Networks (CNNs) are a class of deep neural networks commonly used for tasks recognition in sensor-based systems, particularly those involving motion or activity detection. CNNs are designed to automatically learn and extract spatial and temporal features from sensor data, such as accelerometer or gyroscope readings, which capture movements and patterns associated with different tasks. CNNs apply convolutional filters across the input data to detect local patterns and structures, enabling hierarchical feature extraction. By learning from labeled data, CNNs can classify input sensor sequences into predefined tasks categories, such as walking, standing, turning, or jumping. CNNs have demonstrated strong performance in tasks recognition applications, providing robust and accurate predictions even in noisy or complex sensor data environments.

[0017] Recurrent Neural Networks (RNNs) are a class of neural networks commonly used for tasks recognition in sequential sensor data. RNNs are well-suited for modeling temporal dependencies and sequential patterns inherent in tasks recognition tasks, such as the transitions between different activities over time or the different phases of a gait. The recurrent architecture of RNNs enables them to capture temporal dynamics and context within the input sequences, facilitating tasks recognition based on the sequential patterns of sensor data. By training on labeled sensor sequences, RNNs can learn to predict the sequence of tasks performed by an individual, such as walking, standing, turning, or jumping, based on the observed sensor readings. Variants of RNNs, such as Long Short-Term Memory (LSTM) networks or Gated Recurrent Units (GRUs), are commonly used in tasks recognition applications to address challenges such as capturing long-term dependencies and handling variable-length sequences effectively. RNNs have demonstrated effectiveness in tasks recognition tasks, providing accurate predictions and capturing complex temporal relationships in sensor data sequences.

[0018] According to one embodiment, said at least one processor is further configured to receive at least one recording of said user performing said at least one activity obtained using said at least one sensor, said at least one processor being further configured to generate a patient (e.g., user) dataset by, for each recording, extracting said at least one feature representative of said activity status and labelling said recording using said at leastone feature; said at least one processor being further configured to retrain said trainable neural network using said patient dataset.

[0019] According to one embodiment, said at least one processor is further configured to receive at least one recording of said user performing said at least one activity obtained using said at least one sensor, said at least one processor being further configured to generate a user dataset by, for each recording, extracting said at least one feature representative of said activity status and labelling said recording using said at least one feature; said at least one processor being further configured to finetune the trained prediction model using said user dataset so as to obtain a finetuned user prediction model.

[0020] This method is also called “transfer learning” and it allows to personalize activity detection, ensuring that the neural network better recognizes and interprets a user’s unique movement patterns. In the context of activity recognition for rehabilitation training, this enhances accuracy in tracking rehabilitation progress and detecting deviations from expected recovery trajectories. Additionally, since the model is finetuned with a limited set of user- specific data, it reduces the need for extensive retraining from scratch, leading to faster adaptation and more efficient use of computational resources. Ultimately, this improves the effectiveness of real-time feedback systems, leading to more precise and responsive rehabilitation interventions.

[0021] According to one embodiment, said at least one processor is configured to optimize at least one parameter of said trainable neural network using stochastic gradient descent.

[0022] According to the invention, stochastic gradient descent (SGD) is an optimization algorithm used in machine learning and deep learning for minimizing the loss function during model training. It is a variant of the traditional Gradient Descent algorithm and is particularly well-suited for large-scale datasets or models with many parameters. In SGD, instead of computing the gradient of the loss function using the entire training dataset (as in traditional Gradient Descent), the gradient is computed using a randomly selected subset of the training data, often referred to as a "mini-batch". This subset typically contains a small, fixed number of samples from the dataset.

[0023] The key advantages of Stochastic Gradient Descent include:

[0024] Efficiency: Computing the gradient using a mini-batch of data is computationally more efficient than using the entire dataset, especially for large datasets.

[0025] Generalization: The stochastic nature of SGD introduces noise into the parameter updates, which can help the model generalize better to unseen data and avoid overfitting.

[0026] Convergence Speed: SGD can converge faster than traditional Gradient Descent, as each parameter update is based on a smaller subset of data, allowing for more frequent updates to the model parameters.

[0027] The present invention further relates to a device for prediction of at least one activity status of a user using a trained prediction model obtained from the device for obtaining a trained prediction model, wherein the device for predicting comprises: at least one input configured to receive at least one recording of said user performing said at least one activity obtained using at least one sensor; at least one processor configured to: o provide as input to said trained prediction model said at least one recording so as to obtain said prediction of said at least one activity status; at least one output configured to provide said prediction of said at least one activity status.

[0028] According to one embodiment, said prediction of said at least one user activity status is provided as an input to a device for electrical neuromuscular stimulation so as to obtain a stimulation profile for said user, said stimulation profile corresponding to said user activity status.

[0029] Indeed, integration of sensor-based activity status recognition with devices for electrical neuromuscular stimulation offers the advantage of guided therapy. By accurately identifying the activity status of a user, clinicians can customize neuromuscular stimulation protocols tailored to address identified abnormalities. This targeted approach ensures that electrical stimulation is applied to the appropriate muscles and at the optimal timing during the gait cycle, maximizing therapeutic effectiveness. Additionally, real-time feedback from sensors can inform adjustments to stimulation parameters, ensuring adaptive and personalized treatment plans that better addresspatients' specific gait impairments. As a result, rehabilitation outcomes are enhanced, leading to more efficient and effective treatment strategies.

[0030] The present invention further relates to a computer implemented method for obtaining a trained prediction model for prediction of at least one activity status of a user, said method comprising: receiving for each subject of a plurality of subjects, at least one recording previously obtained using at least one sensor, wherein during each recording, said subject is performing at least one activity; generating a training dataset by, for each recording of each subject of said plurality of subjects: o extracting at least one feature representative of said activity status; o labelling said recording using said at least one feature; obtaining said trained prediction model by feeding said training dataset to a trainable neural network, wherein the trained prediction model is configured to: o receive as an input at least one recording of said user performing said at least one activity obtained using said at least one sensor; and, o provide as output said activity status of the user; outputting said trained prediction model.

[0031] The present invention further relates to a computer implemented method for prediction of at least one activity status of a user using a trained prediction model obtained from the method for obtaining a trained prediction model, said method for predicting comprising: receiving at least one recording of said user performing said at least one activity obtained using at least one sensor; providing as input to said trained prediction model said at least one recording so as to obtain said prediction of said at least one activity status of said user; outputting said at least one prediction of the activity status of said user.

[0032] In addition, the disclosure relates to a computer program comprising software code adapted to perform a computer implemented method compliant with any of the above execution modes when the program is executed by a processor.

[0033] The present disclosure further pertains to a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method compliant with any of the above execution modes.

[0034] The present disclosure further relates to a non-transitory program storage device (i.e. computer-readable storage medium), readable by a computer, tangibly embodying a program of instructions executable by the computer to perform a computer-implemented method, compliant with the present disclosure.

[0035] Such a non-transitory program storage device can be, without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor device, or any suitable combination of the foregoing. It is to be appreciated that the following, while providing more specific examples, is merely an illustrative and not exhaustive listing as readily appreciated by one of ordinary skill in the art: a portable computer diskette, a hard disk, a ROM, an EPROM (Erasable Programmable ROM) or a Flash memory, a portable CD-ROM (Compact-Disc ROM).DEFINITIONS

[0036] In the present invention, the following terms have the following meanings:

[0037] The terms “adapted” and “configured” are used in the present disclosure as broadly encompassing initial configuration, later adaptation or complementation of the present device, or any combination thereof alike, whether effected through material or software means (including firmware).

[0038] The term “processor” should not be construed to be restricted to hardware capable of executing software, and refers in a general way to a processing device, which can for example include a computer, a microprocessor, an integrated circuit, or a programmable logic device (PLD). The processor may also encompass one or more Graphics Processing Units (GPU), whether exploited for computer graphics and image processing or other functions. Additionally, the instructions and / or data enabling to perform associated and / or resulting functionalities may be stored on any processor- readable medium such as, e.g., an integrated circuit, a hard disk, a CD (Compact Disc),an optical disc such as a DVD (Digital Versatile Disc), a RAM (Random- Access Memory) or a ROM (Read-Only Memory). Instructions may be notably stored in hardware, software, firmware or in any combination thereof.

[0039] “Machine learning (ML)” designates in a traditional way computer algorithms improving automatically through experience, on the ground of training data enabling to adjust parameters of computer models through gap reductions between expected outputs extracted from the training data and evaluated outputs computed by the computer models.

[0040] A “hyper-parameter” presently means a parameter used to carry out an upstream control of a model construction, such as a remembering-forgetting balance in sample selection or a width of a time window, by contrast with a parameter of a model itself, which depends on specific situations. In ML applications, hyper-parameters are used to control the learning process.

[0041] “Datasets” are collections of data used to build an ML mathematical model, so as to make data-driven predictions or decisions. In “supervised learning” (i.e. inferring functions from known input-output examples in the form of labelled training data), three types of ML datasets (also designated as ML sets) are typically dedicated to three respective kinds of tasks: “training”, i.e. fitting the parameters, “validation”, i.e. tuning ML hyperparameters (which are parameters used to control the learning process), and “testing”, i.e. checking independently of a training dataset exploited for building a mathematical model that the latter model provides satisfying results.

[0042] A “neural network (NN)” designates a category of ML comprising nodes (called “neurons”), and connections between neurons modeled by “weights”. For each neuron, an output is given in function of an input or a set of inputs by an “activation function”. Neurons are generally organized into multiple “layers”, so that neurons of one layer connect only to neurons of the immediately preceding and immediately following layers.

[0043] The above ML definitions are compliant with their usual meaning, and can be completed with numerous associated features and properties, and definitions of related numerical objects, well known to a person skilled in the ML field. Additional terms will be defined, specified or commented wherever useful throughout the following description.BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The present disclosure will be better understood, and other specific features and advantages will emerge upon reading the following description of particular and non- restrictive illustrative embodiments, the description making reference to the annexed drawings wherein:

[0045] Figure 1 is a block diagram representing schematically a particular mode of a device for obtaining a trained prediction model for prediction of at least one activity status of a user, compliant with the present disclosure;

[0046] Figure 2 is a flow chart showing successive steps executed with the device for obtaining a trained prediction model of figure 1 ;

[0047] Figure 3 is a block diagram representing schematically a particular mode of a device for predicting at least one activity status of a user using a trained prediction model obtained from the device of figure 1 ;

[0048] Figure 4 shows an apparatus integrating the functions of the device for obtaining a trained prediction model of figure 1 and of the device for predicting at least one activity status of a user of figure 3.

[0049] On the figures, the drawings are not to scale, and identical or similar elements are designated by the same references.ILLUSTRATIVE EMBODIMENTS

[0050] The present description illustrates the principles of the present disclosure. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the disclosure and are included within its scope.

[0051] All examples and conditional language recited herein are intended for educational purposes to aid the reader in understanding the principles of the disclosure and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions.

[0052] Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

[0053] Thus, for example, it will be appreciated by those skilled in the art that the block diagrams presented herein may represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, and the like represent various processes which may be substantially represented in computer readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0054] The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which may be shared.

[0055] It should be understood that the elements shown in the figures may be implemented in various forms of hardware, software or combinations thereof. Preferably, these elements are implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, memory and input / output interfaces.

[0056] The present disclosure will be described in reference to a particular functional embodiment of a device 1 for obtaining a trained prediction model 31 for prediction of at least one activity status 51 of a user (i.e. activity status of a user), as illustrated on Figure 1.

[0057] The device 1 is configured to receive for each subject of a plurality of subjects, at least one recording 21 previously obtained using at least one sensor, wherein during each recording, said subject is performing at least one activity, and to provide as output a trained prediction model 31 for prediction of at least one activity status 51 of a user.

[0058] The device 1 is associated with a device 6 for predicting at least one activity status 51 of a user using the trained prediction model 31 obtained from the device 1. The device 6 is represented on Figure 3, and will be subsequently described.

[0059] Though the presently described devices 1 and 6 are versatile and provided with several functions that can be carried out alternatively or in any cumulative way, other implementations within the scope of the present disclosure include devices having only parts of the present functionalities.

[0060] Each of the devices 1 and 6 is advantageously an apparatus, or a physical part of an apparatus, designed, configured and / or adapted for performing the mentioned functions and produce the mentioned effects or results. In alternative implementations, any of the device 1 and the device 6 is embodied as a set of apparatus or physical parts of apparatus, whether grouped in a same machine or in different, possibly remote, machines. The device 1 and / or the device 6 may e.g. have functions distributed over a cloud infrastructure and be available to users as a cloud-based service, or have remote functions accessible through an API.

[0061] The device 1 and the device 6 for may be integrated in a same apparatus or set of apparatus, and intended to same users. In other implementations, the structure of device 6 may be completely independent of the structure of device 1, and may be provided for other users

[0062] In what follows, the modules are to be understood as functional entities rather than material, physically distinct, components. They can consequently be embodied either as grouped together in a same tangible and concrete component, or distributed into several such components. Also, each of those modules is possibly itself shared between at least two physical components. In addition, the modules are implemented in hardware, software, firmware, or any mixed form thereof as well. They are preferably embodied within at least one processor of the device 1 or of the device 6.

[0063] The device 1 comprises a module 11 for receiving, a trainable neural network 22 and, for each subject of a plurality of subjects, at least one recording 21 previously obtained using at least one sensor, wherein during each recording said subject is performing at least one activity. In other words, each recording 21 previously obtainedusing at least one sensor comprises at least one signal representative of an activity status of the subject. The trainable neural network 22 and the at least one recording 21 may be stored in one or more local or remote database(s) 10. The latter can take the form of storage resources available from any kind of appropriate storage means, which can be notably a RAM or an EEPROM (Electrically-Erasable Programmable Read-Only Memory) such as a Flash memory, possibly within an SSD (Solid-State Disk).

[0064] For instance, the activity status may reflect any physical task that a subject or a user might be engaging in, such as: sitting down, standing up, walking, jogging, running, jumping, turning, climbing stairs, cycling, stretching, or performing specific exercises. In other words, the activity status is associated with a physical activity and / or movement type.

[0065] For example, the sensor may be a pressure sensor (e.g., a force-sensitive resistor, a piezoelectric sensor, a capacitive pressure sensor), an Inertial Measurement Unit (IMU), an Electromyography (EMG) sensor, an optical sensor (e.g., pulse oximeter), a stretch sensor, a flex sensor or an electrocardiography (ECG or EKG) sensor.

[0066] The sensor may also be an Inertial Measurement Unit (IMU) configured to measure a recording of accelerometer signal, presenting a time-series plot depicting the acceleration along the x, y, and z axes as a person walks, with peaks corresponding to each step. Additionally, it can measure a gyroscope signal, displaying angular velocity changes as a person performs head movements, with oscillations indicating rotations. In other words, the recording may be an accelerometer signal or a gyroscope signal, obtained from an accelerometer or a gyroscope wear by the subject.

[0067] Another type of sensor could be an Electromyography (EMG) sensor, which records surface EMG signal (i.e. the recording) during gripping exercises, showcasing a waveform depicting muscle activation with spikes corresponding to muscle contractions. Furthermore, it may capture an Intramuscular EMG signal (i.e. the recording), presenting a graph showing electrical activity within a muscle during running, with patterns indicating muscle recruitment and fatigue.

[0068] Moreover, a pressure sensor, such as a Force-Sensitive Resistor (FSR), may be utilized to record a signal illustrating pressure distribution (i.e. the recording) on the soleof the foot while walking, with peaks corresponding to heel strikes and toe-offs. Similarly, a capacitive pressure sensor can record a signal including changes in pressure (i.e. the recording) exerted on a wearable glove while grasping objects, with peaks indicating grip strength variations.

[0069] One sensor or a combination of sensors may be used to obtain the recordings 21. For instance, at least two types of sensors may be used. To that end, each subject of the plurality of subjects may be asked to perform at least one activity among for instance: sitting down, standing up, walking, jogging, running, jumping, turning, climbing stairs, cycling, stretching or dancing. The plurality of sensors may simultaneously record the at least one activity performed by the subject and multiple recordings of the same sequence of activities may be obtained. Moreover, the at least one sensor may be a wearable sensor.

[0070] The device 1 further comprises optionally a module 12 for preprocessing the received at least one recording 21. The module 12 may notably be adapted to clean and denoise the recordings 21 for sake of efficient and reliable processing. Preprocessing may for instance include data cleaning, normalization and temporal alignment, resampling, filtering and noise reduction.

[0071] It may also transform the recording 21, (e.g. by signal compression or decompression).

[0072] According to various configurations, the module 12 is adapted to execute only part or all of the above functions, in any possible combination, in any manner suited to the following processing stage.

[0073] In advantageous modes, the module 12 is configured for preprocessing the recording 21 so as to have the recordings standardized. Standardizing the recordings 21 from different sensors involves ensuring that the recordings 21 from each sensor is transformed or processed in a consistent manner, such that they can be directly compared or combined for further analysis. This may enhance the efficiency of the downstream processing by the device 1. Such standardization may be particularly useful when exploited recordings originate from different sources (e.g. different sensors), including possibly different wearable sensors types or non-wearable sensors types (e.g., a camera).

[0074] The standardization is advantageously applied to the recording 21 and to recording(s) of a user (e.g. by the device 2) in a similar way. Notably, the device 6 may deal with recordings coming from a given sort of source, while the prediction model obtained from device 1 is trained with different types of sources (e.g. recordings from different sensors). Thanks to standardization, differences between sources may then be neutralized or minimized. This may make the devices 1 and 6 more efficient and reliable.

[0075] Standardization may be performed by converting the sensor recordings 21 into a common format or data structure, normalizing the recordings 21 to a common scale or range to eliminate differences in magnitude or units, aligning the recordings 21 to synchronize them in time and the like.

[0076] The device 1 may further comprise a module 13 for generating a training dataset by, for each recording of each subject of said plurality of subjects:- extracting at least one feature representative of said activity status; and- labelling said recording using said at least one feature.

[0077] Notably, module 13 is configured to extract from one or more of the recordings 21 at least one feature representative of said activity status. For example, one feature may be extracted from the accelerometer signal alone while another feature may be extracted from the accelerometer signal and the EMG signal.

[0078] The at least one feature may be extracted using at least one of a statistical analysis, a frequency domain analysis, and a time domain analysis.

[0079] Statistical analysis involves the examination and interpretation of the recordings 21 through the application of statistical methods and techniques. It aims to describe, summarize, and draw inferences from the recordings 21 by quantifying various aspects of its distribution, central tendency, variability, and relationships between variables.

[0080] Frequency domain analysis involves the examination of the recordings 21 in terms of their frequency components. It typically involves transforming time-domain signals into the frequency domain using techniques such as Fourier analysis. This analysis reveals the presence and characteristics of periodic components, oscillations, and spectral features within the recordings.

[0081] Time domain analysis involves the direct analysis of the recordings 21 in the time domain. It focuses on the behavior and characteristics of signals over time, including amplitude variations, temporal patterns, and dynamic changes. Time domain analysis provides insights into the temporal aspects of the recordings 21 without considering frequency components.

[0082] More precisely, such feature may be at least one of the following: mean acceleration: average acceleration values along each axis (x, y, z) during a gait cycle. peak acceleration: maximum acceleration values observed along each axis during a gait cycle. root mean square (RMS) acceleration: a measure of the overall magnitude of acceleration, calculated as the square root of the average of the squares of acceleration values. jerk: rate of change of acceleration over time, indicating abrupt changes in movement. stride length: distance covered during a single gait cycle, typically calculated from the integration of acceleration recordings 21. stride time: time duration of a single gait cycle, representing the time taken to complete a step. step time: time duration of a single step, providing insights into the timing of foot placement and lift-off. step symmetry: comparison of temporal parameters (e.g., step time) between left and right steps, indicating asymmetry in gait. cadence: number of steps per unit time, often expressed as steps per minute, step width: lateral distance between the feet during walking, indicating stability and balance. foot clearance: minimum height of the foot during the swing phase, reflecting the risk of tripping or foot drop. range of motion: maximum and minimum angles of joint movements during the gait cycle, indicating flexibility and mobility.gait variability: variability in gait parameters such as step length, step time, and stride time, providing insights into gait stability and consistency. frequency components: analysis of frequency domain characteristics, such as dominant frequencies or spectral features, revealing oscillatory patterns or rhythmicity in movement. temporal patterns: identification of specific patterns or sequences of movements within the gait cycle, such as heel strike, toe-off, and mid-stance phases.

[0083] When multiple features are identified, a feature selection may be performed to identify the most informative and discriminative features for classification.

[0084] In order to extract the features (e.g. as a function of time), the recordings 21 may be segmented into time windows (i.e., segments) using a sliding window. This allows to capture the temporal context within the recordings 21. Different types of features, such as statistical, frequency-domain, or time-domain features, can be computed within each window, allowing for a diverse set of features to be extracted and utilized for analysis. By aggregating data points within a window, sliding windows can help mitigate the effects of noise and variability in the recordings 21. Sliding windows can adapt to changes in the dynamics of the signal over time. By adjusting the window size or sliding step, the feature extraction process can be tuned to capture different temporal scales or resolutions in the recordings 21, allowing for the detection of both short-term and long-term patterns. Sliding windows facilitate efficient processing of sequential data by breaking it into smaller, manageable segments. This enables parallelization of feature extraction computations and reduces the computational complexity associated with analyzing long sequences of recordings 21. Sliding windows can be applied in real-time settings for online feature extraction and analysis of streaming data. By continuously updating the window with new recordings points as they become available, real-time insights can be generated and used for decision-making or feedback purposes.

[0085] After identification of the features, the recordings 21 may be labelled with the activity status corresponding to the identified features. In one embodiment, where the recordings 21 are segmented into two or more segments, each obtained segment is labelled. The labelling may be performed by a user or by an automated classifier. This operation allow to obtain multiple data samples, each data sample comprising at least onerecording with its corresponding label or at least one segment with its corresponding label.

[0086] It may be observed that the operations by the modules 11, 12 and 13 are not necessarily successive in time, and may overlap, proceed in parallel or alternate, in any appropriate manner. For example, a new recording may be progressively received over time and preprocessed, while the module 13 is dealing with the previously obtained recording. In alternative examples, a batch of recordings 21 corresponding to a complete time sequence range may be fully received and preprocessed before it is submitted to the module 13.

[0087] The device 1 further comprises a module 14 for obtaining the trained prediction model 31, by feeding the training dataset generated by the module 13 to a trainable neural network 22, wherein the trained prediction model 31 is configured to:- receive as an input at least one recording 21 of said user performing said at least one activity obtained using said at least one sensor; and,- provide as output said activity status 51 of the user.

[0088] In other words, the trained prediction model 31 is obtained by iteratively providing to a trainable neural network 22 each one of the labelled training data samples until convergence. A categorial cross-entropy of a binary cross-entropy may be used as loss functions during the training. A stochastic gradient descent may be used as optimization algorithm during training.

[0089] According to one embodiment, the trainable neural network 22 is a convolutional neural network or a recurrent neural network.

[0090] In one embodiment, the device 1 further comprises a module 15 configured to retrain the trained prediction model 31 on a new training dataset comprising activity data obtained (e.g., recorded) from a user.

[0091] In this embodiment, reception module 11 is further configured to receive at least one recording 23 of said user performing at least one activity obtained using said at least one sensor. The at least one sensor may be as discussed before and IMU and / or a EMG.

[0092] Module 12 may be used to pre-process the user’s recording(s) 23 and module 13 may be further configured to generate a user dataset by, for each recording, extractingsaid at least one feature representative of said activity status 51 and labelling said recording using said at least one feature, as described above.

[0093] In this embodiment, module 15 is further configured to retrain said trainable neural network 22 or finetune the trained prediction model 31 using said user dataset.

[0094] In one embodiment, module 15 is further configured to finetune the trained prediction model 31 using said user dataset so as to obtain a finetuned user prediction model. Advantageously, finetune an already trained prediction model allows to use a smaller training dataset as the user dataset, but still to adapt the prediction model to a preferred task, predicting status activity for the specific user. This allows to avoid retraining the model from scratch. This step of finetuning is also referred to as transfer learning.

[0095] In its automatic actions, the device 1 may for example execute the following computer implemented method (Figure 2):- receiving 41 for each subject of a plurality of subjects, at least one recording 21 previously obtained using at least one sensor, wherein during each recording, said subject is performing at least one activity,- generating 43 a training dataset by, for each recording of each subject of said plurality of subjects: o extracting at least one feature representative of said activity status; o labelling said recording using said at least one feature;- obtaining 44 said trained prediction model 31 by feeding said training dataset to a trainable neural network 22, wherein the trained prediction model 31 is configured to: o receive as an input at least one recording of said user 23 performing said at least one activity obtained using said at least one sensor; and, o provide as output said activity status 51 of the user;- outputting said trained prediction model 31.

[0096] The present invention also relates to a device 2 for prediction of at least one activity status 51 of a user using the trained prediction model 31 obtained from thedevice 1. The device 2 will be described in reference to a particular function embodiment as illustrated in Figure 3.

[0097] The device 6 is configured to receive at least one recording 23 of the user performing said at least one activity obtained using at least one sensor, and to provide as output at least one activity status 51 of the user. The device 6 will be described in respect to Figure 3.

[0098] The device 6 comprises a module 61 for receiving at least one recording 23 of said user performing said at least one activity obtained using at least one sensor, the trained prediction model 31 and / or the finetuned user prediction model. The at least one recording 23 of the user, the finetuned user prediction model and the trained prediction model 31 may be stored in one or more local or remote database(s) 60. The latter can take the form of storage resources available from any kind of appropriate storage means, which can be notably a RAM or an EEPROM (Electrically-Erasable Programmable Read-Only Memory) such as a Flash memory, possibly within an SSD (Solid-State Disk).

[0099] The device 6 further comprises optionally a module 62 for preprocessing the received at least one recording 23 of the user. The module 62 is able to perform at least of the preprocessing steps that can be executed by module 12.

[0100] The standardization is advantageously applied to the at least one recording 21 of the patients and to recording(s) of the user in a similar way. Notably, the device 6 may deal with recordings coming from a single type of sensor, while the prediction model obtained from device 1 is trained with different types of sensors. Thanks to standardization, differences between types of sensors may then be neutralized or at least minimized. This may make the devices 1 and 6 more efficient and reliable.

[0101] The device 6 may further comprise a module 63 for obtaining the at least one activity status 51 of the user by providing as input, to the trained prediction model 31 or finetuned user prediction model obtained from the device 1, the recording 23.

[0102] It may be observed that the operations by the modules 61, 62 and 63 are not necessarily successive in time, and may overlap, proceed in parallel or alternate, in any appropriate manner. For example, a new recording 23 may be progressively received over time and preprocessed, while the module 63 is dealing with the previously receivedrecording. In alternative examples, a batch of recordings 23 corresponding to a complete time sequence range may be fully received and preprocessed before it is submitted to the module 63. The device 6 further comprises a module 64 for providing the at least one activity status 51 of the user.

[0103] In its automatic actions, the device 6 may for example execute the following computer implemented method for predicting at least one activity status 51 of a user, said method comprising: receiving at least one recording 23 of said user performing said at least one activity obtained using at least one sensor; providing as input to the trained prediction model 31 or finetuned user prediction model (obtained by device 1) said at least one recording so as to obtain said at least one activity status 51 of said user; outputting said at least one activity status 51 of said user.

[0104] The device 6 is interacting with a user interface 71, via which information can be entered and retrieved by a user. The user interface 71 includes any means appropriate for entering or retrieving data, information or instructions, notably visual, tactile and / or audio capacities that can encompass any or several of the following means as well known by a person skilled in the art: a screen, a keyboard, a trackball, a touchpad, a touchscreen, a loudspeaker, a voice recognition system.

[0105] Further to the generation of the prediction of the activity status 51 of the user, the device 2 may be configured to provide said prediction of the activity status 51 of the user to a device for electrical neuromuscular stimulation of said patient so as to obtain a stimulation profile for said patient that is specific to the gait pathology group in which said patient has been classified.

[0106] Such device for electrical neuromuscular stimulation may include several elements.

[0107] A first element is a multi-channel electrostimulator configured to send electric pulses to the muscles of the patient to stimulate them.

[0108] The electro stimulator includes a portable electrical muscle stimulation (EMS) generator that may be housed within a housing such as a bag or a vest. The bag may becarried on the back or on the front or on one shoulder, or around the waist. Alternatively, the generator may be housed in a pocket of an article of clothing, such as trousers.

[0109] The generator may deliver a voltage comprised between 0 and 350 V and a current intensity comprised between 0 and 170 mA. The generator is associated with an intensity variator to deliver a pulse sequence with a pre-determined intensity, width and frequency.

[0110] The generator may have several channels that can be each configured to deliver a different pulse sequence.

[0111] The generator is in electrical connection with a set of electrodes configured to be positioned on the skin of the patient, either on the motor points of muscles, where nerve endings enter the muscle or on sensory receptors in the skin that are innervated or at sensory nerves endings.

[0112] The electrodes may be connected to the generator via conductive wires, electrical traces, conductive fibers, or a combination thereof. The conductive wires, electrical traces, conductive fibers, or a combination thereof can be embedded within the article of clothing, for instance in a layer of the article of clothing or interwoven with fibers used to make the article of clothing.

[0113] The electrodes may comprise several layers including a contact layer configured to attach to the patient’s skin, a connector layer configured to be connected to the generator and conductive layers positioned between the contact layer and connector layer. The contact layer can be made of a biocompatible polymeric layer. It may include an adhesive and / or a hydrogel. Alternatively, the contact layer may be dry and may require that a conductive gel or a hydrating lotion hydrates the skin surface of the patient.

[0114] The electrodes may be coupled to an inner surface of the article of clothing by adhesives, clips, straps, hook-and-loop fasteners, stitches or a combination thereof. The electrodes may be positioned such that when the patient puts the article of clothing on, the contact layer of the electrodes is automatically positioned in contact with the skin over the motor points, the sensory receptors or the sensory nerves endings that needs to be stimulated. Advantageously, the electrodes can be detached from the article of clothing to allow replacing a defective electrode.

[0115] A second element of the device for electrical neuromuscular stimulation is at least one IMU sensor and at least one pressure detection unit. The IMU sensor may be the as the one used to capture the recordings 23.

[0116] The IMU sensor includes at least one motion sensor such as an accelerometer, a gyroscope, a magnetometer and / or a combination thereof. Advantageously, the IMU sensor includes three motion sensors for each leg and one motion sensor for the hip. A first motion sensor may be positioned on the thigh, a second motion sensor may be positioned on the calf and a third motion sensor may be positioned on the foot. The motion sensors are configured to sense the three-dimensional movements of the leg throughout a gait cycle. The motion sensors may be embedded within the article of clothing, for instance in a layer of the article of clothing or interwoven with fibers used to make the article of clothing.

[0117] The pressure detection unit includes pressure sensors that may be positioned under the feet to sense the strength of a contact of the different parts of the patient’s feet with the ground. The expression “strength of a contact” refers to the pressure exerted by a at least one region of one foot on the ground, the pressure being defined as a force per unit of area. Advantageously, the pressure sensors may be included in an insole or a sole of a shoe. For instance, the pressure detection unit may include between 1 and 1000 sensors distributed on the sole to collect data on the pressure applied, the position and the pressure changes when the patient is walking, running, jumping, climbing, descending, sitting or standing. In a preferred embodiment, there are five pressure sensors per foot. Two pressure sensors may be positioned under the heel, on the medial and lateral side, and a third pressure sensor may be positioned under the big toe. The fourth and fifth sensors may be positioned under the metatarsal bones.

[0118] The pressure sensors may be capacitive sensors, resistive sensors, piezoelectric sensors, piezoresistive sensors or a combination thereof. The pressure sensors provide an electrical signal output, which is either a voltage or a current, that is proportional to the pressure exerted on said pressure sensors.

[0119] Advantageously, the device for electrical neuromuscular stimulation may include other sensors such as electromyography sensors to access muscle fatigue and movementintention or encoders positioned at the legs joints to access the angular position of the different parts on the legs.

[0120] For instance, the patient motion data coming from the motion sensors and the patient foot plantar pressure data coming from the pressure sensors may be used to determine the foot strike pattern, the foot inclination angle, the tibia angle, the hip flexion and extension, the trunk lean, the ankle inversion and eversion, the foot progression angle, the pelvic drop, the knee flexion and extension, the stride length, or the displacement of the center of mass, the speed of gait, the cadence, etc.

[0121] A third element of the device for electrical neuromuscular stimulation is a device for computing a start timepoint and an end timepoint of at least one sequence of electrical pulses to apply to at least one of a nerve and a muscle of a patient using the electrodes. Such device may receive the prediction of the activity status 51 of the user and the additional sensor data (such as pressure data) to compute the start timepoint and an end timepoint of the at least one sequence of electrical pulses. Such device may be housed within the housing with the generator. Such device may communicate with a control device such as a tablet, a PC or a smartphone, hosting the user interface. The communication may be wireless, using Bluetooth, WiFi or other protocols. The control device may be used to parameter the device for electrical neuromuscular stimulation and visualize information on the state of the device for electrical neuromuscular stimulation and on how the patient is faring.

[0122] A particular apparatus 9, visible on Figure 4, is embodying the device 1 as well as the device 6 described above. It corresponds for example to a workstation, a laptop, a tablet, a smartphone, or a head- mounted display (HMD).

[0123] That apparatus 9 is suited to IVF outcome predictions and to related ML training. It comprises the following elements, connected to each other by a bus 95 of addresses and data that also transports a clock signal:- a microprocessor 91 (or CPU);- a graphics card 92 comprising several Graphical Processing Units (or GPUs) 920 and a Graphical Random Access Memory (GRAM) 921;- a non-volatile memory of ROM type 96;- a RAM 97;- one or several I / O (Input / Output) devices 94 such as for example a keyboard, a mouse, a trackball, a webcam; other modes for introduction of commands such as for example vocal recognition are also possible;- a power source 98; and- a radiofrequency unit 99.

[0124] According to a variant, the power supply 98 is external to the apparatus 9.

[0125] The apparatus 9 also comprises a display device 93 of display screen type directly connected to the graphics card 92 to display synthesized images calculated and composed in the graphics card. The use of a dedicated bus to connect the display device 93 to the graphics card 92 offers the advantage of having much greater data transmission bitrates and thus reducing the latency time for the displaying of images composed by the graphics card. According to a variant, a display device is external to apparatus 9 and is connected thereto by a cable or wirelessly for transmitting the display signals. The apparatus 9, for example through the graphics card 92, comprises an interface for transmission or connection adapted to transmit a display signal to an external display means such as for example an LCD or plasma screen or a video-projector. In this respect, the RF unit 99 can be used for wireless transmissions.

[0126] It is noted that the word “register” used hereinafter in the description of memories 97 and 921 can designate in each of the memories mentioned, a memory zone of low capacity (some binary data) as well as a memory zone of large capacity (enabling a whole program to be stored or all or part of the data representative of data calculated or to be displayed). Also, the registers represented for the RAM 97 and the GRAM 921 can be arranged and constituted in any manner, and each of them does not necessarily correspond to adjacent memory locations and can be distributed otherwise (which covers notably the situation in which one register includes several smaller registers).

[0127] When switched-on, the microprocessor 91 loads and executes the instructions of the program contained in the RAM 97.

[0128] As will be understood by a skilled person, the presence of the graphics card 92 is not mandatory, and can be replaced with entire CPU processing and / or simpler visualization implementations.

[0129] In variant modes, the apparatus 9 may include only the functionalities of the device 1, and not those of the device 6. In addition, the device 1 and / or the device 6 may be implemented differently than a standalone software, and an apparatus or set of apparatus comprising only parts of the apparatus 9 may be exploited through an API call or via a cloud interface.

Claims

CLAIMS1. A device (1) for obtaining a trained prediction model (31) for prediction of at least one activity status (51) of a user, wherein the device (1) comprises: at least one input configured to receive (11), for each subject of a plurality of subjects, at least one recording (21), previously obtained using at least one sensor, wherein during each recording, said subject is performing at least one activity; at least one processor configured to: o generate (13) a training dataset by, for each recording of each subject of said plurality of subjects:■ extracting at least one feature representative of said activity status (51);■ labelling said recording using said at least one feature; o obtain (14) said trained prediction model (31) by feeding said training dataset to a trainable neural network (22), wherein the trained prediction model (31) is configured to:■ receive as an input at least one recording (23) of said user performing said at least one activity obtained using said at least one sensor; and,■ provide as output said prediction of said at least one user activity status (51); at least one output configured to provide said trained prediction model (31).

2. The device according to claim 1, wherein said at least one sensor is a wearable sensor.

3. The device according to either one of claims 1 and 2, wherein said at least one activity feature is extracted using at least one of a statistical analysis, a frequency domain analysis, and a time domain analysis.

4. The device according to any of claims 1 to 3, wherein said at least one recording comprises a first recording obtained using a first sensor and a second recordingobtained using a second sensor, and wherein generating said training dataset further comprises aligning and synchronizing the first recording and the second recording.

5. The device according to any of claims 1 to 4, wherein said at least one recording is segmented into time windows of fixed size using a sliding window, the extraction of said at least one feature being performed on each time window.

6. The device according to any of claims 1 to 5, wherein generating said training dataset further comprises extracting at least two features and selecting one feature from the at least two features based on at least one of a correlation analysis, a feature importance ranking, and a domain expertise.

7. The device according to any of claims 1 to 6, wherein said trainable neural network (22) is a convolutional neural network or a recurrent neural network.

8. The device according to any of claims 1 to 7, wherein said at least one processor is further configured to receive at least one recording (23) of said user performing said at least one activity obtained using said at least one sensor, said at least one processor being further configured to generate a patient dataset by, for each recording, extracting said at least one feature representative of said activity status (51) and labelling said recording using said at least one feature; said at least one processor being further configured to finetune the trained prediction model (31) using said user dataset so as to obtain a finetuned user prediction model.

9. The device according to any of claims 1 to 7, wherein said at least one processor is configured to optimize at least one parameter of said trainable neural network (22) using stochastic gradient descent.

10. A device (6) for prediction of at least one activity status (51) of a user using a trained prediction model (31) obtained from the device according to any of claims 1 to 9, wherein said device (6) comprises: at least one input configured to receive (61) at least one recording (23) of said user performing said at least one activity obtained using at least one sensor; at least one processor configured to:o provide as input to said trained prediction model (31) said at least one recording, so as to obtain (63) said prediction of said at least one activity status (51) of said user; at least one output configured to provide (64) said prediction of said at least one activity status (51).

11. A device according to claim 10, wherein said prediction of said at least one user activity status (51) is provided as an input to a device for electrical neuromuscular stimulation so as to obtain a stimulation profile for said user, said stimulation profile corresponding to said user activity status (51).

12. A computer implemented method for obtaining a trained prediction model (31) for prediction of at least one activity status (51) of a user, said method comprising: receiving (41) for each subject of a plurality of subjects, at least one recording previously obtained using at least one sensor, wherein during each recording, said subject is performing at least one activity; generating (43) a training dataset by, for each recording of each subject of said plurality of subjects: o extracting at least one feature representative of said activity status; o labelling said recording using said at least one feature; obtaining (44) said trained prediction model (31) by feeding said training dataset to a trainable neural network (22), wherein the trained prediction model (31) is configured to: o receive as an input at least one recording (23) of said user performing said at least one activity obtained using said at least one sensor; and, o provide as output said activity status (51) of the user; outputting said trained prediction model (31).

13. A computer implemented method for prediction of at least one activity status (51) of a user using a trained prediction model (31) obtained from the method according to claim 12, said method comprising: receiving at least one recording (23) of said user performing said at least one activity obtained using at least one sensor;providing as input to said trained prediction model (31) said at least one recording so as to obtain said prediction of said at least one activity status (51) of said user; outputting said at least one prediction of the activity status (51) of said user.

14. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the methods of any of claims 12 or 13.

15. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the methods of any of claims 12 or 13.

16. A non-transitory program storage device, readable by a computer, tangibly embodying a program of instructions executable by the computer to perform a computer-implemented method of any of claims 12 or 13.

Images