A system and method for continuous fatigue modelling for manual handling in occupational settings

Abstract

A system and method to monitor fatigue by estimating fatigue during the performance of physical occupational tasks and providing real-time feedback to workers and employers is disclosed. This system and method utilizes at least one inertial measurement unit (IMU) used to measure human movement kinematic data, a computing device to collect and save the data, and algorithms and machine learning models on the computing device, presented in an application, to process and estimate fatigue levels based on the kinematic data.

Description

[0001] A SYSTEM AND METHOD FOR CONTINUOUS FATIGUE MODELLING FOR MANUAL HANDLING IN OCCUPATIONAL SETTINGS

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 668,935 filed July 9, 2024, the disclosure of which is fully incorporated by reference herein in its entirety.

[0004] BACKGROUND OF THE DISCLOSURE

[0005] An estimated 2.3 million occupational injuries and accidents occurred in the United States in 2022, leading to a median of 10 days away per incident (U.S. Bureau of Labor Statistics, 2023). Many of these injuries and accidents are associated with workplace fatigue, i.e., fatigue resulting from repetitive manual handling and work altering movement behaviors in a manner that can increase the risk for musculoskeletal disorders (MSDs) and accidents. Thus, the ability to monitor fatigue progression in the workplace can inform the optimization of workrest ratios, break and rotation scheduling, and the provision of real-time movement quality feedback.

[0006] Monitoring fatigue at work is difficult because existing methods (e.g., measuring force generation capacity) are obtrusive. Inertial measurement units (IMUs) are suitable for this purpose because kinematics can provide insight into fatigue status and can be worn on or under clothing and equipment. IMUs are an inexpensive and unobtrusive tool to monitor and recognize human activity, and typically include a tri-axial accelerometer, gyroscope, and magnetometer to measure three-dimensional linear acceleration, angular velocity, and magnetic field, respectively.

[0007] Fatigue modelling systems are novel and have been presented in occupational settings with two primary shortcomings. First, fatigue has been represented as a categorical entity, which limits predictive capabilities and is inconsistent with the current understanding of continuous fatigue progression. For example, in Aryal, A., Ghahramani, A., & Becerik-Gerber, B. (2017). Monitoring fatigue in construction workers using physiological measurements. Automation in Construction, 82, 154-165, fatigue was monitored across various construction activities, but used heart rate, temperature, and brain activity measurements to classify 1 of 4 fatigue states. Second, current systems have only analyzed the performance of one activity (e.g., walking), which is insufficient because fatigue develops and manifests during the performance of various occupational activities. For example, Karg, M., Venture, G., Hoey, J., & Kulic, D. (2014). Human movement analysis as a measure for fatigue: A hidden Markov-based approach. IEEE transactions on neural systems and rehabilitation engineering, 22(3), 470-481 is one of a few studies to attempt to model the continuous increases in fatigue. However, this was done for a squat exercise only, and utilized full-body marker-based optical motion capture. 3D motion capture and pressure insoles were used to obtain kinematic and kinetic data, respectively, to compute joint forces to estimate joint-level work load as a proxy for joint fatigue by Yu, Y, Li, H., Yang, X, Kong, L., Luo, X, & Wong, A. Y. (2019). An automatic and non-invasive physical fatigue assessment method for construction workers. Automation in construction, 103, 1-12. Similarly, in Baghdadi, A., Cavuoto, L. A., Jones-Farmer, A., Rigdon, S. E., Esfahani, E. T., & Megahed, F. M. (2021). Monitoring worker fatigue using wearable devices: A case study to detect changes in gait parameters. Journal of quality technology, 53(1), 47-71. There, an I MU was used on the ankle to study how fatigue develops over time during walking. Specifically, the research considered how gait parameters change, how they relate to subjective ratings, whether they were consistent across individuals over time and whether they were affected by participant characteristics.

[0008] Thus, it should be appreciated that a need exists for a way to reduce the fatigue-related risk for injuries, accidents, and reduced productivity by providing an unobtrusive system to monitor fatigue. Further, such a system should estimate fatigue during the performance of physical occupational tasks and provide real-time feedback to the user. The need also exists for a way to automatically monitor fatigue status, allowing the user to subsequently use this information to justify necessary breaks, shift rotations, and / or feedback regarding safe movement techniques.

[0009] SUMMARY OF THE DISCLOSURE

[0010] The present invention addresses the foregoing needs by providing an unobtrusive system to monitor fatigue. This system estimates fatigue during the performance of physical occupational tasks and provides real-time feedback to workers and employers. This system is comprised of an inertial measurement unit (IMU) used to measure human movement kinematic data, a computing device (e.g., smartphone) to collect / save the data, and algorithms and machine learning models on the computing device (presented in a software application) to process and estimate fatigue levels based on the kinematics, respectively.

[0011] While the primary beneficiaries of this invention are the wearer who can experience a rise in productivity and a decline in bodily fatigue and injury, unions and worker protection agencies will also be incentivized to promote the widespread implementation of such a system in workplaces that requires manual labor. Furthermore, the mitigated injury and accident risk resulting from the use of this invention will greatly improve the safety and quality of life experienced by workers. Thus, this invention will lead to positive impacts on companies’ output and workers’ physical and psychological well-being, amounting to benefits to both the economy and society.

[0012] BRIEF DESCRIPTION OF DRAWINGS

[0013] FIG. 1 is a schematic showing the path of the types of movements measured in the manual handling protocol. The protocol required participants to pull, lift, carry, push, walk, and stand with a crate.

[0014] FIG. 2 depicts a series of graphs showing the loss (A), accuracy (B) and weighted average Fl -score (C) of the fully connected (FC) and deep convolutional long short-term memory (DeepConvLSTM) human activity recognition models and defining sensor combinations (x-axes).

[0015] FIG. 3 shows that the FCI captured increase in fatigue throughout the manual handling protocol across all participants used to develop the system.

[0016] FIG. 4 is an example of how a fatigue notification for a worker triggered by the system of this invention may look on an Apple Watch.

[0017] FIG. 5 is an example of how the models developed could be organized in a cascading ensemble.

[0018] FIG. 6 is an example of the human activity recognition (HAR on top) and task-specific fatigue composite index (FCI) estimation models working together to estimate the FCI across the entire manual handling protocol for one participant (P5 on bottom). DETAILED DESCRIPTION

[0019] This invention is a system and method comprised of at least one wearable IMU connected to a computing device having an artificial neural network (ANN) for characterizing different types of activities the user has engaged in during the activity session, and at least one feedback interface. In one embodiment, the system can include a software application communicatively coupled to the IMU. The IMU and the application can operate cooperatively in configured processing of collected kinematic data and generating resulting interactions.

[0020] An IMU is an electronic device that includes a combination of accelerometers and / or gyroscopes to measure characteristics of an object, such as the object's velocity, orientation, and / or gravitational forces. For example, an IMU can measure three degrees of freedom of the acceleration and three degrees of freedom of the angular rate of the IMU. The IMU sensor orientation may be obtained by integrating angular velocity using dead reckoning while the gravitational and magnetic components are used to stabilize the signal and minimize integration drift. To obtain sensor position, accelerometer signals may be double integrated, and then gravity may be compensated for by applying the estimated sensor orientation and subtracting gravitation acceleration. One or more IMU may be physically coupled to the body of the user. Depending on the activity, the IMU can be worn on one or both legs and / or one or both arms of a person to provide the information necessary to approximate fatigue. Likewise, one or more IMU may be worn elsewhere on the body depending on the task at hand. One skilled in the art will appreciate that IMU can be worn on the feet, forearms, back, pelvis and elsewhere.

[0021] In some embodiments, at least one IMU supplies information or data related to the activity to the computing device. The computing device is configured to receive and process signals from the IMU so as to output an activity of the subject. The computing device may include an internal memory area used for temporary storage of data configured to store an observation sequence formed of a plurality set of the IMU data outputs for a predetermined period of time; a storage configured to store a plurality of reference activity sequences, the reference activity sequences corresponding to motion signatures of different activities; and a matching processor configured to match the sequence recorded by the IMU against the reference sequences to find a best matched reference activity sequence. Further, the matching processor may find an optimal rotation and temporal correspondence between the observation sequence and one of the reference sequences so as to obtain the best matched reference sequence. The computing device performs a calculation that uses the signals from both linear motion sensors and rotational motion sensors of the IMU to determine the activity independent of the orientation of the sensor unit. In some embodiments, the computing device may include an activity processing tool. The activity processing tool processes the data collected by the IMU and will identify the different types of activities the user has engaged during the activity session. Types of activity can include stand, walk, pull, lift, carry, and push. Based on the activity, activity-specific neural networks will partition the data into one or more data streams or blocks relating to the one or more activities performed by the user during the activity session. Algorithms adapt to various manual handling activities being performed, then model fatigue as a continuous variable.

[0022] Specifically, all data measured and reported by the IMU are concatenated into a single matrix. Feature normalization is conducted on a feature-by-feature basis, and then all data is rounded to 4 decimal places. An artificial neural network (ANN) performs the human activity recognition. This neural network is comprised of an input layer, three fully connected dense layers with rectified linear unit activation functions, and a dense output layer with a Softmax activation function. During training, the model parameters are optimized using stochastic gradient descent, which minimizes categorical cross-entropy loss. Real-time information regarding tasks being performed (including the frequency and duration) and fatigue status is available, allowing the wearer of the IMU to subsequently use this information to justify necessary breaks, rotations, and / or feedback regarding safe movement techniques.

[0023] In some embodiments, the system measures data using wearable IMUs with the algorithms running in real-time on a wirelessly-connected smartphone. Altogether, this system can recognize which manual handling activity is being performed, and estimate the associated fatigue level based on how the activity is performed in near real-time. If the estimate fatigue exceeds a pre-defined threshold for a period of time (which depends on the magnitude for which the fatigue exceeds a pre-defined threshold), the user and employers are warned to take a break, rotate jobs, and / or focus on improving movement technique to reduce the risk for injury or accidents. A feedback interface functions to provide some form of feedback to the user. The feedback interface may be integrated with the IMU, the computing device, and / or any suitable device. The feedback interface is preferably activated in response to at least one fatigue metric. A feedback interface preferably enables activation of one or more feedback outlets such as a display, an audio system, haptic feedback, and the like. In one variation, the system can enable optional use of an application. In one example, the user can also use the wearable device without the companion app. During this use case, the wearable device will track the activities performed, and store the data for upload in the future.

[0024] The process to develop this system is to collect ground truth data from individuals performing manual handling activity until exhaustion. The ground truth data for the movements that they are performing were obtained by using their whole body motion to manually identify the activities they are performing, and then using these labels to train machine learning models to identify the activity using only the data collected using the IMUs. The ground truths used to represent fatigue in the development of this system includes subjective fatigue rating, maximal force generation capacity, heart rate variability, instantaneous heart rate, electromyographic measurements of the biceps and triceps brachii, deltoids, erector spinae, vastus medialis, and vastus lateralis.

[0025] Human activity recognition (HAR) is an intermediate step for many injury and accident mitigation systems. As noted above, inertial measurement units (IMUs) are an inexpensive and unobtrusive tool to implement HAR. In developing this invention, HAR performance resulting from the use of raw versus fused IMU data collected during a fatiguing, simulated manual handling protocol was determined. It was believed that the approaches would lead to different classification performance and training times.

[0026] The method employed in this invention involved having 13 male participants and 11 female participants donned an Xsens MVN Link inertial motion capture system (Xsens, Enschede, Netherlands; 120 Hz), an Apple smartwatch (Apple Inc., Cupertino, USA; 40 Hz), and 7 Delsys Trigno electromyography (EMG) and electrocardiogram (EKG) sensors (Delsys, Natick, USA; 2048 Hz) to perform a manual handling protocol until the onset of excessive fatigue. The path of this protocol is illustrated in Fig. 1. Participants stood behind the line at point A, descended, and pulled a crate 100 (8.1 kg) towards themselves. Then, they lifted the crate, turned, and carried it to point B by walking around pylon 200. At point B they released the crate onto a low- friction table 300 and pushed it across and down the roller conveyor 400 where it stopped between points A and C. Participants then walked to point C, where they performed the same sequence from the other side, around pylon 500. Each activity from points A to C, or points C to A, was considered one round, and a set was comprised of 20 rounds. Each set of the protocol lasted 5 minutes and minimal technique constraints were imposed.

[0027] Between sets, participants performed a fatigue assessment comprised of self-reported fatigue on a visual analog scale (VAS) and a maximal lift strength (MLS) assessment. Participants alternated between the manual handling protocol and fatigue assessments without breaks until one of the following conditions were met: VAS > 90%, MLS < 70% of their baseline value, they could not maintain pace during manual handling for 3 consecutive rounds, or they wished to stop. VAS, MLS, EMG, and EKG were processed to obtain seven variables to holistically capture fatigue progression: fatigue VAS, MLS force, instantaneous heart rate (HR), HR variability low / high frequency content, EMG mean and median frequencies, and a composite index of the above. These variables were considered the ground truth for fatigue, where kinematic Xsens and smartwatch data were used to estimate these variables.

[0028] The raw data were exported and concatenated to create one matrix each for the Xsens and smartwatch IMUs. Then, all movements were labelled as one of six manual handling activities by visually observing the Xsens data. Data were segmented using a sliding window approach, where the label was the activity performed for the majority of the window.

[0029] Fully connected (FC) and deep convolutional long short-term memory (DeepConvLSTM) neural networks were employed for HAR. Eighteen sensor combinations were studied and the neural network hyperparameters were performed for 7 combinations. Eight-fold cross- validation was employed to train and evaluate each model and sensor combination permutation, with 3 randomly selected participants in each fold (to ensure evaluation was performed on unseen data). The IMU combinations tested are set forth below in Table 1; bolded combinations were used for hyperparameter optimization; IMU = inertial measurement units, R = right, L = left and T8 = T8 vertebrae.

[0030] Table 1

[0031] All sensor combinations and neural network performed well (e.g., weighted average Fl- score > 90.11%), while more sensors generally improved performances as shown in Fig. 2. The graphs of Fig. 2 show the loss (A), accuracy (B) and weighted average Fl -score (C) of the fully connected (FC) and deep convolutional long short-term memory (Deep ConvLSTM) and defining sensor combinations (x-axes). While 6 was the best performing combination, a commercial-grade smartwatch sampled at % of the frequency led to accuracy within 3% of 6. The T8 with DeepConvLSTM led to the strongest single sensor performance, while the performance of 3 A was comparable to using 6 or 8 IMUs in related work. DeepConvLSTM was only superior to FC for some combinations. Thus, the additional complexity and computational resources required to implement deep neural networks may not be warranted for manual handling HAR.

[0032] Data were processed with two methods: raw (RAW) or sensor fusion to obtain orientation and position (FUSION). Tasks were manually labelled, then used to train fully connected neural networks (ANNs). Classification performances between RAW and FUSION were compared and then the better method was employed for a sensitivity analysis of eighteen combinations, each comprised of one to six IMUs.

[0033] To quantify fatigue, self-reported fatigue level, maximal lift strength, electromyography median frequency of seven muscles, heat rate (HR), HR variability (low to high frequency ratio), and pelvic jerk magnitude were measured throughout the protocol and combined into the fatigue composite index (FCI). The FCI ranged from 0 - 1, where larger values indicated greater fatigue. Task-specific ANNs were trained to estimate the FCI using T8 IMU and HR data. All models were trained using Ufold cross-validation in TensorFlow 2.12, where k = 8 (three participants per fold). The FCI captured an increase in fatigue throughout the protocol as shown in Fig. 3, showing the mean (black line) and standard deviation (grey shading) of the FCI across the protocol completion. Using task-specific models, the FCI was estimated using the T8 IMU and HR with an average Pearson’s correlation of r = 0.59 and mean absolute error = 0.05. The performance on each task, in terms of Pearson's correlation coefficient (r) and mean absolute error (MAE) on the task-specific estimations of the FCI is presented in Table 2 below. Table 2

[0034] The kinematic data from the Xsens and smartwatch were processed and used to train and evaluate manual handling activity recognition neural networks to automatically determine which activities were performed: stand, walk, pull, lift, carry, and push. Based on the activity, activity-specific neural networks were trained to model each fatigue variable as continuous variables. The performances were evaluated using root mean squared and mean absolute error.

[0035] Across 441 sets of manual handling, the participants performed 52,920 activities over 36.7 hours. The reasons for stopping the manual handling protocol included maximal lifting strength (MLS), fatigue visual analog scale (VAS), failure to maintain and voluntary stoppage. A tally of the occurrences of stopping are presented in Table 3 below.

[0036] Table 3

[0037] Fatigue during manual handling can affect kinematics in a manner than increases musculoskeletal disorders (MSD) and accident risk. In conjunction with pre-existing activity recognition models, the foregoing results are the final component of a system that will adapt to the activity then model fatigue as a continuous variable using IMUs.

[0038] The envisioned use of the system of this invention is to unobtrusively monitor workers' fatigue using kinematic data (with or without HR data), employing HAR models to classify tasks, and estimating continuous fatigue levels. If the rate of fatigue increase was too large and / or if the fatigue level was excessive for an extended period of time, a configurable alert would be triggered to the worker and / or employer to inform breaks, rotations, and provide technique reminders. An example of how an alert could look on an Apple Watch is depicted in Fig. 4. In order to arrive at real-world implementation, various procedures must be performed including cascading HAR and regression models together in an ensemble, model deployment, front- end software development, and the selection of notification thresholds.

[0039] Cascading ensembles refers to the “stacking” of numerous machine learning (ML) models together, to provide outputs not possible using isolated models. Though various configurations are possible, a suitable configuration for the models developed in the present invention is proposed in Fig. 5. Kinematic (with or without HR) data is the input with a constant sliding window size, the HAR model classifies the task being performed, the output of the HAR model informs which task-specific model to employ for fatigue estimation, and then the task-specific regression model estimates the FCI value, representing holistic fatigue level. The input data has the dimensions equal to n number of features x m sliding window size. The human activity recognition (HAR) model classifies the windows as being one of six manual handling tasks, then the corresponding regression model estimates the fatigue composite index (FCI) as a continuous variable ranging from 0 - 1.

[0040] An example of this integration estimating the FCI on one participant is presented in Fig. 6, showcasing the results on participant 5. The first 5 min of the manual handling protocol HAR performance is depicted. The model performing the FCI estimation for each window (bottom) was dependent on the outputs of the HAR: Task 0 = stand; 1 = pull; 2 = lift; 3 = carry; 4 = push; 5 = walk; RMSE = root mean squared error; MAE = mean absolute error. Using only the T8 IMU, the HAR model classified the task being performed with an accuracy of 93.88% and a weighted-average Fl -score of 94.16%. Using these classifications to inform which task-specific model to employ, the FCI was estimated using the T8 data supplemented with HR with MAE = 0.048 and a Pearson’s correlation of r = 0.88 across all tasks during their manual handling protocol.

[0041] In the context of deploying ML models for on-device mobile inference, model compression is required and refers to the packaging and compression of ML models to ensure they are deployable to mobile devices. In short, various procedures are implemented to reduce the size and computational demands of ML models through various methods, such as quantization (reducing numbers’ significant figures used in the models), pruning (identifying and removing less important nodes from the model), and weight sharing (identifying parameters that can be shared across the model). These procedures can be employed using an open-source runtime, such as Google LiteRT (formerly TensorFlow Lite; Alphabet Inc., Mountain View, USA).

[0042] The present invention involves the development of front-end software to design and create a graphical user interface and integration with on-device models combined into software that can be used on mobile devices (e.g., laptops, smart phones, smart watches). This allows users to interact with the application to set up this system (including pairing devices), configure alert settings, and to receive near real-time alerts from the system about fatigue status via push notifications. An example of how a push notification may appear on an Apple Watch is shown in Fig. 4. Overall, the present invention allows the work from this dissertation to be employed in the field for fatigue estimation that adapts to the performance of various manual handling tasks. To widen the applicability of this work, future work should also aim to broaden the models by collecting more data.

[0043] Another important step for implementation of the present invention is the determination of appropriate thresholds for triggering notifications. A benefit of estimating holistic fatigue as a continuous value is the ability to monitor excessive FCI values, in addition to large rates in increase. While it is possible to allow end users to determine their own thresholds, one or more default settings should be defined based on training data. Then, the absolute FCI values and the rate of change of FCI values that are associated with the change of established risk factors into the high-risk categories can be noted and used as thresholds.

[0044] Many manual handling tasks exist beyond the ones presented, including overhead lifting, pushing and pulling, asymmetrical lifting, carrying, pulling, pushing, etc. To ensure the models can be more generalizable for real-world implementation, more data should be collected on the tasks not yet used herein to further expand the existing HAR model, while using these data to develop new task-specific FCI estimation models. Additionally, collecting additional data on tasks already present in the current work will also be important, as it can still yield benefits in improving model performance and generalizability.

[0045] While the present invention has been described with reference to what are presently considered to be the preferred examples of the system and method for continuous fatigue modelling for manual handling in occupational settings, it is to be understood that the invention is not limited to the disclosed embodiments. Instead, the invention is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims.

Claims

What is claimed is:

1. A system for continuous fatigue modelling for manual handling in occupational settings comprising: at least one inertial measurement unit (IMU) used to measure human movement kinematic data; a computing device to collect and save the data; and algorithms and machine learning models, used in conjunction with the computing device and presented in the form of an application, to process and estimate fatigue levels based on the kinematic data.

2. The system of claim 1 comprising 2 or more IMUs to measure various aspect of human movement kinematic data.

3. The system of claim 1 wherein the computing device provides notifications as to the levels of fatigue and warnings to assist in monitoring fatigue levels.

4. A method for continuous fatigue modelling for manual handling in occupational settings comprising: connecting at least one inertial measurement unit (IMU) used to measure human movement kinematic data to a subject; using a computing device to collect and save the data; and using algorithms and machine learning models, in conjunction with the computing device and presented in the form of an application, to process and estimate fatigue levels based on the kinematic data.

5. The method of claim 4 comprising 2 or more IMUs to measure various aspects of human movement kinematic data.

6. The method of claim 4 wherein the computing device provides notifications as to the levels of fatigue and warnings to assist in monitoring fatigue levels.

Images