Mobile device-based hand grip strength measurement

Abstract

Systems, devices, and methods for mobile device-based hand grip strength measurement are disclosed. In some aspects, the mobile device includes a vibration motor, an inertial motion unit (IMU), a processor, and a non-transitory computer-readable memory, and the processor is configured to perform operations including vibrating the vibration motor, acquiring IMU data using the inertial measurement unit of the mobile device indicative of vibration damping caused by a user applying a force to the mobile device, and determining the applied force based on a force model and the IMU data.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This patent document claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 385,379, entitled "SMARTPHONE-BASED HAND GRIP STRENGTH MEASUREMENT," filed November 29, 2022. The entire contents of the aforementioned patent application are incorporated by reference as part of the disclosure of this patent document.

[0002] This patent document relates to mobile device-based hand grip strength measurement. [Background technology]

[0003] Hand grip strength (HGS) is a functional biomarker of a user's overall condition and may be used as an individual test or in a series of tests. As an indicator of physical force generation, HGS may provide insight into muscle mass loss or bone density issues in a user. Correlations may also exist between HGS and cognitive function. These correlations may be more than a simple correlation between age-related muscle mass loss and cognitive function. Nervous system decline can affect motor system performance and dexterity independently of musculoskeletal decline. Therefore, conditions that affect neurological decline may also affect the nervous system and HGS. Thus, HGS measurement may be an indicator of frailty. Summary of the Invention

[0004] This document discloses methods, devices, and systems for performing hand grip strength measurements using sensor data acquired by built-in sensors in a mobile device without additional accessories.

[0005] Aspects of the present document relate to a mobile device configured to measure hand grip strength using sensor data acquired by an integrated sensor of the mobile device without accessories. The mobile device may include a vibration motor, an inertial motion unit (IMU), a processor, and a non-transitory computer-readable memory storing instructions that, when executed by the processor, cause the processor to perform operations including: vibrating the vibration motor; acquiring IMU data using the inertial measurement unit of the mobile device indicative of vibration damping resulting from a user applying a force to the mobile device; and determining the applied force based on a force model and the IMU data.

[0006] Aspects of this document relate to a method for measuring hand grip strength using sensor data acquired by a built-in sensor of a mobile device without accessories. The method may include vibrating a vibration motor of the mobile device, acquiring IMU data using an inertial measurement unit (IMU) of the mobile device indicative of vibration damping caused by a user applying a force to the mobile device, and determining the applied force based on a force model and the IMU data.

[0007] A further aspect of this document relates to one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a mobile device, cause the mobile device to perform any one or more of the solutions described herein.

[0008] These and other aspects of this document, as well as their implementations and applications, are explained in more detail in the drawings, description, and claims. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates hand grip strength measurement using a hand dynamometer.

[0010] [Figure 2] 1 is a diagram of an exemplary system implementing the disclosed technology in accordance with the disclosed technology.

[0011] [Figure 3] FIG. 1 is an exemplary block diagram of various components of a mobile device in accordance with some embodiments of the present document.

[0012] [Figure 4] FIG. 1 illustrates a process flowchart for mobile device-based hand grip strength measurement according to some embodiments of the present document.

[0013] [Figure 5] FIG. 10 illustrates regression plots and Bland-Altman plots for force measurements of three different smartphones, according to some embodiments of the present document.

[0014] [Figure 6] FIG. 10 illustrates a comparison between forces measured using a force sensor and corresponding forces determined based on a force model according to some embodiments of the present document.

[0015] [Figure 7] FIG. 10 illustrates the results of HGS measurements using a mobile device classified as levels according to some embodiments of this document.

[0016] [Figure 8] FIG. 1 illustrates an exemplary graphical user interface of an application for HGS measurements according to some embodiments of the present document. DETAILED DESCRIPTION OF THE INVENTION

[0017] This document describes a system, device, and method for measuring hand grip strength using a mobile device's built-in sensors. This method is applicable to any mobile device equipped with a vibration motor and an inertial measurement unit (IMU). The mobile phone application disclosed herein may drive the mobile device to vibrate using its internal vibration motor, use the mobile device's IMU to monitor the damping of these vibrations resulting from user-applied forces, and quantify the applied forces based on a force model and the IMU data. The mobile device may determine a user's hand grip strength based on IMU data acquired using built-in sensors already within the mobile device, requiring no hardware modification or physical installation; it can be software-enabled by downloading an application onto any mobile device, as described in further detail below. This technology may therefore provide an opportunity for large-scale screening for mental or physical disabilities. As a software-enabled measurement for any mobile device equipped with an IMU and vibration motor, this technology can be rapidly disseminated, for example, to underserved communities.

[0018] Because cognitive and musculoskeletal decline can affect HGS, HGS can be used as a functional biomarker to provide health care professionals with a warning sign that a patient is experiencing significant mental or physical decline. While loss of muscle mass and decline in HGS can be expected as individuals age, a significant and unexpected decline in HGS may signal a mental or physical disability and, if left untreated, could lead to continued health decline.

[0019] HGS can be measured using a hand grip dynamometer (a device with strain gauge sensors), as shown in Figure 1. Hand grip dynamometers can typically measure isometric strain for up to 5 seconds. For example, a user can perform an HGS test using one hand, placing the metacarpal bone on the base of the device and placing the pull bar between the first and second knuckles. The user can pull as hard as they can with the measuring hand and receive a strength classification (weak, medium, or strong) based on the measurements and demographics.

[0020] The act of measuring the force applied by a user's hand using a dynamometer can similarly be performed using the built-in sensors and actuators of commercially available mobile devices. While the mobile device is vibrating, the readings of the IMU (including an accelerometer and / or gyroscope) may reflect the vibration. These signals may be attenuated by the application of force (or weight). Instead of focusing on static or binary metrics, HGS measurements may be made using the attenuation of the mobile device's IMU (e.g., including an accelerometer and gyroscope) during vibration. As described elsewhere in this document, HGS sensing on mobile devices may provide results on a continuous scale and / or categorized as levels.

[0021] The mobile device-based HGS measurement disclosed herein does not require any accessories and works on various types of mobile devices equipped with a vibration motor and an IMU. Most mobile devices already include an IMU with an accelerometer and a gyroscope for other purposes, such as measuring linear acceleration and angular velocity, which enables location estimation by widely used mobile devices for rotating the screen, playing games, counting steps, gesture recognition, and many others. Similarly, most mobile devices already include a vibration motor to provide haptic feedback for messages, calls, and on-screen interactions. Such mobile devices may implement HGS measurements based on force damping techniques utilizing only the mobile device's vibration motor and IMU. To measure force, the mobile device may be set to vibrate (e.g., at full vibration), and the IMU measures the movement of the device when it vibrates (due to vibration). When a force is applied during vibration, the IMU signal changes as the vibration changes or damps. According to an embodiment of this document, vibration damping is measured and, based on that, the applied force is modeled as a damping force on the vibration. By relying on components common to different types of mobile devices, the HGS measurements disclosed herein can function across different types of mobile devices (eg, type may correspond to model by manufacturer).

[0022] According to some embodiments of the present document, IMU data, including multi-axis accelerometer data and multi-axis gyroscope data that collectively represent vibration damping, is used to determine the corresponding forces that cause vibration damping. The method recognizes the relationship between accelerometer data and different axes of the gyroscope, which respond differently to applied forces. Due to conservation of energy, an applied force not only causes vibration damping or a decrease in amplitude along all IMU axes, but may even increase some axes because a force applied to damp vibration motion in one axis may transfer vibration energy to another axis. The disclosed method takes multiple inputs from both the accelerometer and the gyroscope as inputs and examines the frequency, amplitude, and relationship between the different axes. As used herein, "damping" or "vibration damping" refers to the recorded change in device motion caused by an applied hand grip force.

[0023] The applied force that causes vibration damping of a mobile device may be quantified based on a force model trained to correlate (1) IMU data measured by the mobile device and indicative of the mobile device's vibration damping and (2) forces applied to the mobile device that cause corresponding vibration damping. The force model may include a machine learning algorithm (e.g., a multivariate linear regression algorithm). The correlation may vary depending on the configuration of the mobile device. Vibration damping may depend on one or more factors that contribute to how the mobile device absorbs and dissipates vibration energy. Examples of such factors include the intensity of the vibration, the load distribution within the mobile device (e.g., including how the weight and components are distributed within the mobile device), the materials used within the mobile device (e.g., including density, elasticity, and internal friction), the overall design (e.g., including shape and structural integrity, features such as ribs, gussets, etc.), the interactions between components within the device, etc., or a combination thereof. Furthermore, the damping measured by the mobile device's IMU may depend on one or more factors, such as the location of the IMU relative to the vibration motor(s), sensor parameters such as sensitivity, resolution, noise level, etc., or a combination thereof. The techniques disclosed herein are generic and transferable across different types of mobile devices, allowing for similar force estimation performance across different mobile devices. A force model can be determined for a type of mobile device (e.g., a model by manufacturer) and used to calibrate mobile devices of the same type.

[0024] A calibration step can enable standardized measurements across mobile devices. Calibration can ensure that different mobile devices of the same type or different types are configured to measure force consistently. This is useful because force model development is an automatic or at least partially automatic process, can be completed within a short period of time (e.g., one day), and does not require large-scale participant recruitment or clinical measurements.

[0025] Technical improvements proposed in the embodiments disclosed herein include methods for performing HGS measurements that can be applied across different types (e.g., models) of mobile devices without hardware modifications or accessories, and instead can be achieved through a downloadable application. Generalizing sensor-based measurements across a wide variety of mobile devices is a well-established problem that significantly limits the impact of research in the mobile health space. The results of the device-to-device compatibility study disclosed herein provide evidence that the disclosed force attenuation technology enables HGS measurements to function on a variety of mobile devices. Data processing and modeling demonstrate that many different types of mobile devices can perform HGS measurements after calibration based on the type of mobile device. Because there are tens of millions of users for each common type of mobile device, a single factory calibration could potentially enable millions of mobile devices to perform hand grip strength measurements.

[0026] FIG. 2 illustrates an exemplary system implementing the disclosed mobile device-based hand grip strength measurement technology. The system 200 includes a mobile device 202, which may include at least one of a camera 204, a vibration motor 205, an IMU 206, a processor 208, a wireless transmitter 210, and a display 212. The processor 208 can control the operation of the mobile device 202 (e.g., vibrate the vibration motor 205, cause the camera 204 and / or the IMU 206 to acquire data), receive and process sensor data (e.g., image data acquired by the camera 204, IMU data acquired by the IMU 206), run algorithms (e.g., force models) on the sensor data, and generate results (e.g., applied force to provide vibration damping, force plots, diagnosis based on the determined HGS, etc.). As illustrated in FIG. 2, the mobile device 202 can communicate with a user 214 or an external device or system (e.g., a cloud server 216). For example, the mobile device 202 may transmit a report of the status of the system to the cloud server 216, e.g., using the wireless transmitter 210. As another example, the mobile device 202 may communicate with a user 214 via the display 212. The display 212 may be a touchscreen configured as a graphical user interface, and the mobile device 202 may display data or results to the user 214 via the display 212 and receive user input via the display 212. In some embodiments, the mobile device 202 may be a smartphone, a tablet, or the like.

[0027] FIG. 3 illustrates an exemplary block diagram of various components of a mobile device according to some embodiments of the present document. The mobile device 300 is an example of the mobile device 202 illustrated in FIG. 2. In some embodiments, the mobile device 300 may be a smartphone, a tablet, or the like. In some embodiments, the mobile device 300 is driven to vibrate by an internal vibration motor 303, and an IMU is used to measure vibration damping caused by a user applying force to the mobile device 300. Based on the IMU data, a force model can be used to estimate the HGS. The mobile device 300 includes one or more sensors 302 capable of collecting data, a processing unit 304 coupled to the one or more sensors 302 and capable of running a force model on the collected data, a wireless transceiver 306 coupled to the processing unit 304, and a display 308 coupled to the processing unit 304. The one or more sensors 302 may include one or more of a camera or an IMU. The IMU of the mobile device 300 may include an accelerometer and a gyroscope.

[0028] 4 shows a flowchart of a process for mobile device-based hand grip strength measurement according to some embodiments of this document. The process 400 may be implemented on a mobile device (e.g., mobile device 202, mobile device 300).

[0029] At block 410, the process 400 includes vibrating a vibration motor of the mobile device. The vibration motor may be one already built into the mobile device. For example, the vibration may be driven by a vibration motor used to provide haptic feedback for messages, calls, screen interactions, etc. The vibration motor may vibrate at a particular level (e.g., a maximum level) of the vibration motor.

[0030] At block 420, the process 400 includes using an inertial measurement unit (IMU) of the mobile device to acquire IMU data indicative of vibration damping caused by a user applying a force to the mobile device. To measure HGS, the user may apply the force by gripping the mobile device in their hand.

[0031] In some embodiments, process 400 may include providing a visual guide to be displayed on the display of the mobile device. The visual guide (also referred to as visual instructions) may include indicia that guide the user in applying force by gripping the mobile device. As part of the preparation, process 400 may include providing the user with information about how to use the mobile device to perform the HGS measurement (e.g., how the user should sit, how the user should apply force, etc., or a combination thereof). By way of example only, the instructions may include that the user should hold the measurement device (mobile device) directly in front of the user and sit upright with the elbows bent at 90 degrees and a neutral wrist position; that the user's feet should be flat on the ground; that the user should breathe calmly and normally during the measurement; that the user should grip the mobile device with the edge of the mobile device between the user's fingers and palm; etc. The mobile device may provide such instructions to the user via a user interface implemented on the display of the mobile device, for example. The instructions may be presented in the form of text, images, illustrations, videos, audio messages, etc., or a combination thereof. For example, the instructions may be presented as a visual guide including an image or illustration showing the recommended position of the user's fingers and / or the fingers of the user's hand when the user grips the mobile device. As another example, the instructions may be presented as a video showing a model user performing an HGS measurement using a sample mobile device.

[0032] Following the instructions, the user may apply a damping force by squeezing the mobile device when it vibrates. The user may apply maximum force for a period of time (e.g., about 5 seconds) until the measurement is complete. The mobile device may notify the user when the measurement is complete.

[0033] In some embodiments, process 400 may include receiving information regarding a user's performance of an HGS measurement and evaluating whether the user is ready to perform the measurement and / or whether the measurement is valid. For example, the mobile device may detect the position of the user's fingers gripping the mobile device and evaluate whether the grip is appropriate for the user to proceed with the measurement. In some embodiments, position detection may be performed based on one or more images captured by a camera on the mobile device or another device (e.g., another mobile device, a wall-mounted camera, or a camera positioned on a surface, etc.). In some embodiments, the mobile device includes a touchscreen (e.g., display 212) configured to detect contact by the user, and process 400 may detect contact of one or more of the user's fingers with the touchscreen and determine the position(s) of the one or more fingers based on the detected contact. As a further example, the mobile device may detect the user's posture or a portion thereof (e.g., the posture of the user's arm or wrist) by capturing one or more images or portions thereof of the user and evaluate whether the user is appropriately positioned to perform the measurement. In some embodiments, process 400 may include providing a notification to the user based on the evaluation. For example, in response to an evaluation concluding that the user needs to make an adjustment (e.g., an adjustment related to grip or arm posture), the mobile device may provide a notification to the user suggesting the adjustment and / or perform further detection or evaluation regarding whether the user made the suggested adjustment.

[0034] The IMU of a mobile device may include a multi-axis accelerometer and a multi-axis gyroscope. The vibration motor may already be built into the mobile device for other purposes, such as measuring linear acceleration and angular velocity to enable position estimation by mobile devices commonly used for rotating screens, playing games, counting steps, gesture recognition, etc.

[0035] The IMU data may include multi-axis accelerometer data and multi-axis gyroscope data. For example, the IMU data may include tri-axis accelerometer data and tri-axis gyroscope data. The tri-axis accelerometer data may include the linear acceleration of the IMU in various axes (indicative of smartphone movement). Depending on how energy is dissipated during vibration damping, the linear acceleration of the IMU may change in various patterns. For example, during a portion(s) of time that a damping force is applied, linear acceleration along different directions may decrease at different rates or may even follow opposite trends (e.g., linear acceleration in some directions increases while linear acceleration in other directions decreases). Similarly, the angular velocity of the IMU along various directions (indicative of smartphone movement) may change in various patterns in response to a user-applied damping force. Therefore, IMU data, including multi-axis accelerometer data and multi-axis gyroscope data, may be used in combination to provide a comprehensive representation of vibration damping and, therefore, improve the accuracy of the determined applied force.

[0036] Each axis of the multi-axis accelerometer data and the multi-axis gyroscope data may include a time-series signal. The signal for an axis of the IMU data may include high-frequency components from the vibration motor and low-frequency components corresponding to noise. In some embodiments, the raw IMU data may be sampled from the mobile device at maximum speed without on-device filtering or post-processing. In some embodiments, the raw IMU data may undergo a series of processing steps. For example, the raw IMU data may be processed using one or more of the following techniques: low-pass filtering, high-pass filtering, band-pass filtering, Savitzky-Golay (Savgol) filtering, standard deviation, or empirical mode decomposition. Filtering may also include multiple steps, such as in the case of a filter bank or multiple stages of different filtering types. Additional features (e.g., signal power for axis combinations) may be obtained from the filtered or raw IMU data. One or more features may be evaluated using various metrics (e.g., including principal component analysis, recursive feature elimination, lasso regression, and correlation).

[0037] At block 430, the process 400 includes determining an applied force based on the force model and the IMU data.

[0038] In some embodiments, process 400 may include decomposing the IMU data based on an empirical mode decomposition (EMD) technique. According to the EMD technique, process 400 may include, for each axis of the multi-axis accelerometer data and the multi-axis gyroscope data, decomposing the axis's signal into multiple intrinsic mode functions (IMFs) representing different frequency components of the signal, so that the applied force may be determined based at least in part on the IMFs corresponding to the various axes of the multi-axis accelerometer data and the multi-axis gyroscope data.

[0039] Given that the high-frequency components of the signals for various axes from the vibration motor are the strongest components of the corresponding signals, the first IMFs for various axes of the IMU data primarily correspond to the vibration motor signals. The upper envelope of the first IMFs for each axis of the multi-axis accelerometer data and the multi-axis gyroscope data may be used as a feature to identify the applied force. In some embodiments, the upper envelope of the first IMFs on each IMU axis serves as a typical feature, and the first IMFs collectively may be most significantly correlated with the force data and thus may be used as an input to a force model to determine the applied force, while other features may be excluded. Thus, by way of example only, process 400 may include, for each axis of the multi-axis accelerometer data and the multi-axis gyroscope data, identifying a first IMF corresponding to the highest frequency from the multiple IMFs of the axis's signals, identifying an upper envelope of the first IMFs for the axis, and inputting the upper envelope of the first IMFs corresponding to all axes of the multi-axis accelerometer data and the multi-axis gyroscope data, respectively, into the force model.

[0040] EMD techniques for processing IMU data are described herein for purposes of illustration and not limitation. As described elsewhere herein, before being input to the force model, the IMU data may be processed by at least one of filtering, filter banking, averaging, standard deviation, wavelet decomposition, spectral analysis, EMD, or the like. In some embodiments, signals from at least two different axes of IMU data (including multi-axis accelerometer data and multi-axis gyroscope data) may be processed using various techniques. In some embodiments, raw IMU data may be input directly to the force model.

[0041] The force model may include a machine learning model trained to correlate (1) IMU data measured by a mobile device and indicative of vibration damping of the mobile device and (2) forces applied to the mobile device that cause corresponding vibration damping. The force model may be a data-driven model including parametric modeling, linear regression models, ensemble learning models (e.g., random forests, Adaboost, etc.), support vector machine models, neural networks (e.g., transformer models, convolutional neural networks (CNNs), etc.), or variations or combinations thereof. By way of example only, the force model may include a multivariate linear regression model that provides values ​​of applied force on a continuous scale.

[0042] The force model can be trained using training data including applied forces measured using force sensors (e.g., force-sensing resistors (FSRs)) and measured damping caused by the applied forces. An exemplary test case was performed involving three different types of smartphones, including a Google Pixel 4, a Samsung Galaxy A53, and a Motorola Moto G Power, from various smartphone manufacturers, physical shapes, costs, and built-in components. In the exemplary test case, each participant applied pressure by gripping the smartphone with a 0.3 mm-thick force sensor positioned under each finger on the side of the smartphone. During the measurement, real-time measurements from the force sensor were displayed on a screen in front of the user along with a force guide. The user was instructed to follow the force guide when applying various pressures using real-time feedback. The acquired IMU data was processed in substantially the same manner as described above to obtain vibration damping data. The forces measured using the force sensors and the corresponding vibration damping data were used to train and validate the force model. The mean absolute error, correlation coefficient, and bias for each smartphone are shown in Figure 5. In FIG. 5, black dots indicate data corresponding to the Motorola Moto G Power, crosses indicate data corresponding to the Samsung Galaxy A53, and stars indicate data corresponding to the Google Pixel 4.

[0043] As described elsewhere in this document, the vibration damping behavior of a mobile device may depend on its configuration. Results from exemplary test cases suggest that relevant differences in force measurements from different types of mobile devices (e.g., types may correspond to models by manufacturer) may be calibrated using different force models. Force models may be trained when new types of mobile devices (e.g., new models by manufacture) are released, and the resulting force models may be used to calibrate mobile devices of the same type. The type of mobile device may correspond to the model and / or manufacturer of the mobile device and may be related to factors (e.g., including the configuration of the mobile device's vibration motor, the configuration of the IMU, etc., or a combination thereof).

[0044] A model library may be built by collecting force models corresponding to different types of mobile devices, and in some embodiments, process 400 may include obtaining a force model based on the type of mobile device from such a model library in preparation for a hand grip strength measurement procedure.

[0045] Process 400 may determine the applied force substantially in real time. In some embodiments, during measurement, process 400 may include providing a visual guide to be displayed on a display of the mobile device, the visual guide including a force plot showing the applied force substantially in real time. In some embodiments, the visual guide may further include force guidelines superimposed on the applied force. The user may be instructed to follow the force guide in applying various forces using real-time feedback.

[0046] In some embodiments, process 400 may include measuring the area of ​​a surface of the body part, where the force is applied by pressing the surface of the body part against a camera. By way of example only, process 400 may cause a camera to take a photograph of the surface of the body part as the body part is pressed against the camera and a damping force is applied, and determine the area based on the images. As another example, the mobile device may include a touchscreen configured to detect contact by a user, and process 400 may determine the area by identifying an area of ​​contact of the body part with the display. Process 400 may include determining the applied pressure based on the applied force and the area.

[0047] FIG. 6 shows a comparison between the force measured using a force sensor (curve 602) and the corresponding force determined based on a force model (curve 604), according to some embodiments of the present document. A calibrated linear force sensing resistor (FSR) was used to measure the applied force during the grasp. As an initial setup, the FSR was placed between the user's finger and the smartphone. The accelerometer and gyroscope were sampled during the grasp while the FSR recorded the applied force. A linear regression fit demonstrates that smartphone-based HGS measurements can be performed to continuously track grip force.

[0048] Figure 7 shows the results of HGS measurements using a mobile device classified into levels according to some embodiments of the present document. Panel (I) shows the linear X-axis acceleration of the smartphone IMU over time when the smartphone was vibrated and a force was applied to cause vibration damping. Panel (II) shows the HGS determined according to embodiments of the present disclosure and classified as a level based on the user's HGS and / or demographic information. Exemplary relevant demographic information includes age, gender, health status, medical history, etc., or a combination thereof. As illustrated, higher HGS levels generally correspond to lower linear acceleration along the X-axis.

[0049] As described elsewhere in this document, the IMU of a mobile device may include an accelerometer and a gyroscope. While only linear acceleration along the X-axis is shown, according to some embodiments of this document, the IMU of a mobile device measures acceleration in multiple directions (e.g., three perpendicular directions, including a direction along the Z-axis and two directions in a plane perpendicular to the Z-axis) and also measures gyroscope data in multiple directions (e.g., the same three directions as the accelerometer). Vibration and its damping of a mobile device may be monitored based on IMU data including a combination of multi-axis accelerometer data and multi-axis gyroscope data. Depending on how vibration damping occurs, the linear acceleration of the IMU along various axes (indicating smartphone movement) may change in various patterns. For example, as the applied force increases, the linear acceleration along the X-axis decreases. However, during a portion(s) of the time that the damping force is applied, the linear acceleration along different axes may decrease at different rates or even increase. Similarly, the angular velocity of the IMU along various directions (indicating smartphone movement) may change in various patterns in response to a damping force applied by the user. Therefore, IMU data including multi-axis accelerometer data may be combined with multi-axis gyroscope data for use in determining applied forces.

[0050] FIG. 8 illustrates an exemplary graphic user interface (GUI) of an application for HGS measurement according to some embodiments of the present document. During measurement, a user may grip a mobile device (e.g., a smartphone) with force (e.g., the user's maximum force) for a period of time (e.g., 5 seconds), during which the mobile device vibrates. The resulting signal can be processed to determine the grip force substantially in real time. The results may be displayed in real time on the GUI as shown. The GUI may display additional information, such as instructions to guide the user through performing the HGS measurement using the smartphone, characteristics determined based on the determined HGS value, further analysis or actions desired by the user (e.g., different ways of displaying the results (e.g., categorized results, textual results, a summary of the results and related information), or portions thereof, sending the results to another person or device, etc.).

[0051] The disclosed mobile device-based HGS measurements can be used to assess a user's condition. For example, for surgical patients, hand grip strength typically declines after surgery and recovers as the patient recovers. Failure of hand grip strength to recover to pre-operative levels may be a sign of surgical complications. Similarly, hand grip strength can be monitored at home to screen for unexpected muscle weakness, which may be a sign of illness or other frailty. As another example, the disclosed technology can be used to assess finger strength to assess readiness to engage in physical activity. Maximum force grips can assess the body's readiness to engage in high levels of physical activity. These grips may not necessarily be based on hand strength. For example, gripping a smartphone tightly between the thumb and index finger may be more effective for individuals with particularly strong hands. A notification regarding the results of the user's HGS measurement can be generated and provided to the user or another person (e.g., a healthcare provider), another device, and / or another entity (e.g., a clinic, research institution, device manufacturer). The notification may include a report including the results and / or the user's condition determined based on the results. [Example]

[0052] The following examples illustrate some embodiments in accordance with the present technology. Other exemplary embodiments of the present technology may be presented before or after the examples listed below.

[0053] 1. A mobile device comprising: a vibration motor; an inertial motion unit (IMU); a camera; a processor; and non-transitory computer-readable memory storing instructions that, when executed by the processor, cause the processor to perform operations including: vibrating the vibration motor; using the inertial measurement unit of the mobile device to obtain IMU data indicative of vibration damping caused by a user applying a force to the mobile device; and determining the applied force based on a force model and the IMU data.

[0054] 2. A mobile device of any one or more of the solutions described herein, wherein the force model includes a machine learning algorithm trained to correlate (1) the IMU data measured by the mobile device and indicative of the vibration damping of the mobile device, and (2) forces applied to the mobile device and causing the vibration damping.

[0055] 3. A mobile device of any one or more of the solutions described herein, wherein the IMU includes an accelerometer and a gyroscope, and the IMU data includes multi-axis accelerometer data and multi-axis gyroscope data.

[0056] 4. A method for measuring hand grip strength using a mobile device, the method comprising: vibrating a vibration motor of the mobile device; using an inertial measurement unit (IMU) of the mobile device to acquire IMU data indicative of vibration damping caused by a user applying a force to the mobile device; and determining the applied force based on a force model and the IMU data.

[0057] 5. The method of any one or more of the solutions described herein, wherein the IMU data includes multi-axis accelerometer data and multi-axis gyroscope data.

[0058] 6. The method of any one or more of the solutions described herein, wherein determining the applied force from the IMU data includes processing the IMU data based on at least one of filtering, filter bank, averaging, standard deviation, wavelet decomposition, spectral analysis, or EMD.

[0059] 7. The method of any one or more of the solutions described herein, wherein each axis of the multi-axis accelerometer data and the multi-axis gyroscope data includes a signal, and determining the applied force from the IMU data includes, for each axis of the multi-axis accelerometer data and the multi-axis gyroscope data, decomposing the signal of the axis into a plurality of intrinsic mode functions (IMFs) representing different frequency components of the signal, and the applied force is determined based at least in part on the IMFs corresponding to the various axes of the multi-axis accelerometer data and the multi-axis gyroscope data.

[0060] 8. The method of any one or more of the solutions described herein, wherein determining the applied force from the IMU data further includes, for each axis of the multi-axis accelerometer data and the multi-axis gyroscope data, identifying a first IMF corresponding to a highest frequency from the multiple IMFs of the signals for that axis, identifying an upper envelope of the first IMF for that axis, and inputting the upper envelopes of the first IMFs corresponding to all axes of the multi-axis accelerometer data and the multi-axis gyroscope data, respectively, into the force model.

[0061] 9. The method of any one or more of the solutions described herein, further comprising providing a visual guide to be displayed on a display of the mobile device, the visual guide comprising a force plot showing the applied force substantially in real time.

[0062] 10. The method of any one or more of the solutions described herein, wherein the visual guide further comprises force guidelines superimposed on the applied force.

[0063] 11. The method of any one or more of the solutions described herein, further comprising providing a visual guide to be displayed on a display of the mobile device, the visual guide including markings to guide the user in applying the force over time.

[0064] 12. The method of any one or more of the solutions described herein, further comprising providing a visual guide to be displayed on a display of the mobile device, the visual guide including markings to guide the user in applying the force by gripping the mobile device using the user's hand.

[0065] 13. The method of any one or more of the solutions described herein, further comprising: assessing a state of the user based on the applied force; and generating a notification based on a result of the assessment.

[0066] 14. The method of any one or more of the solutions described herein, further comprising retrieving the force model from a model library based on a type of the mobile device, the type of the mobile device being associated with at least one of a configuration of the IMU or a configuration of the vibration motor.

[0067] 15. The method of any one or more of the solutions described herein, wherein the force model includes a machine learning algorithm trained to correlate (1) the IMU data measured by the mobile device and indicative of the vibration damping of the mobile device, and (2) forces applied to the mobile device and causing the vibration damping.

[0068] 16. The method of any one or more of the solutions described herein, wherein the force model includes a multivariate linear regression model trained based on training data obtained using a force sensor and a training mobile device of the same type as the mobile device.

[0069] 17. The method of any one or more of the solutions described herein, wherein the force model is configured to generate results on a continuous scale, and the method further comprises assigning the applied force to one of a plurality of categories.

[0070] 18. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a mobile device, cause the mobile device to perform any one or more of the solutions described herein.

[0071] Implementations of the subject matter and functional operations described in this patent document can be implemented in various systems, digital electronic circuits, or computer software, firmware, or hardware (including the structures disclosed herein and their structural equivalents), or one or more combinations thereof. Implementations of the subject matter described herein can be implemented as one or more computer program products, i.e., as one or more modules of computer program instructions encoded on a tangible and non-transitory computer-readable medium for execution by or controlling the operation of a data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter providing a machine-readable propagated signal, or one or more combinations thereof. The term "data processing unit" or "data processing device" includes all devices, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, an apparatus can include code that creates an execution environment for a given computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0072] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, e.g., a compiled or interpreted language, and can be deployed in any form, e.g., as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored as part of files that maintain other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple associated files (e.g., files storing one or more modules, subprograms, or portions of code). A computer program can be deployed to run on one computer or on multiple computers, either located at one site or distributed across multiple sites and interconnected by a communications network.

[0073] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs that perform functions by operating on input data to generate output. The processes and logic flows may also be performed by, or apparatus implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0074] Processors suitable for executing a computer program include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, a processor receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes, or is operatively coupled to, one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, and receives and / or transfers data therefrom. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices. The processor and memory can be supplemented by, or incorporated in, special-purpose logic circuitry.

[0075] This specification, together with the drawings, are intended to be considered illustrative only, and by illustrative is meant to be exemplary. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, the use of "or" is intended to include "and / or" unless the context clearly indicates otherwise.

[0076] While this patent document contains many details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in this patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as working in a particular combination, and even initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from that combination, and the claimed combination may be directed to subcombinations or variations of subcombinations.

[0077] Similarly, although operations are shown in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all of the illustrated operations be performed, to achieve desirable results. Further, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0078] Various embodiments described herein are described in the general context of a method or process, which in one embodiment may be implemented by a computer program product embodied in a computer-readable medium, e.g., computer-executable instructions, e.g., program code, executed by a computer in a network environment. Computer-readable media may include removable and non-removable storage devices, such as, but not limited to, read-only memory (ROM), random access memory (RAM), compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs, etc. Accordingly, computer-readable media as described in this application includes non-transitory storage media. Generally, program modules may include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps or processes.

[0079] For example, one aspect of the disclosed embodiments relates to a computer program product embodied on a non-transitory computer-readable medium, the computer program product including program code for performing any one and / or all of the operations of the disclosed embodiments.

[0080] Only some embodiments and examples have been described; other embodiments, extensions, and variations can be made based on what is described and illustrated in this patent document.

Claims

1. 1. A method for performing force measurements using a mobile device, comprising: vibrating a vibration motor of the mobile device; using an inertial measurement unit (IMU) of the mobile device to acquire IMU data indicative of vibration damping caused by a user applying a force to the mobile device; determining the applied force based on a force model and the IMU data; A method comprising:

2. The method of claim 1 , wherein the IMU data includes multi-axis accelerometer data and multi-axis gyroscope data.

3. 3. The method of claim 2, wherein determining the applied force includes processing the IMU data based on at least one of filtering, a filter bank, averaging, standard deviation, wavelet decomposition, spectral analysis, or empirical mode decomposition (EMD).

4. each axis of the multi-axis accelerometer data and the multi-axis gyroscope data includes a signal; determining the applied force includes, for each axis of the multi-axis accelerometer data and the multi-axis gyroscope data, decomposing the signal for that axis into a plurality of intrinsic mode functions (IMFs) representing different frequency components of the signal; the applied force is determined based at least in part on the IMFs corresponding to different axes of the multi-axis accelerometer data and the multi-axis gyroscope data. The method of claim 3.

5. Determining the applied force further comprises: For each axis of the multi-axis accelerometer data and the multi-axis gyroscope data, identifying a first IMF from the plurality of IMFs of the signal for the axis corresponding to a highest frequency; identifying an upper envelope of the first IMF for the axis; inputting into the force model the upper envelope of the initial IMF corresponding to all axes of the multi-axis accelerometer data and the multi-axis gyroscope data, respectively; 5. The method of claim 4, comprising:

6. providing a visual guide to be displayed on a display of the mobile device, the visual guide including a force plot showing the applied force substantially in real time. The method of claim 1.

7. The method of claim 6 , wherein the visual guide further comprises force guidelines superimposed on the applied force.

8. The method of claim 7 , wherein the visual guide includes markings to guide the user in applying the force over time.

9. providing a visual guide to be displayed on a display of the mobile device, the visual guide including markings to guide the user in applying the force by gripping the mobile device with the user's hand. The method of claim 1.

10. assessing a condition of the user based on the applied force; and generating a notification based on a result of said evaluation; The method of claim 1 further comprising:

11. and further comprising: obtaining the force model from a model library based on a type of the mobile device, the type of the mobile device being associated with at least one of a configuration of the IMU or a configuration of the vibration motor. The method of claim 1.

12. 2. The method of claim 1 , wherein the force model comprises a machine learning algorithm trained to correlate (1) the IMU data measured by the mobile device and indicative of the vibration damping of the mobile device, and (2) the forces applied to the mobile device and causing the vibration damping.

13. 10. The method of claim 1, wherein the force model comprises a multivariate linear regression model trained based on training data acquired using a force sensor and a training mobile device of the same type as the mobile device.

14. the force model is configured to generate results on a continuous scale; The method further includes assigning the applied force to one of a plurality of categories. The method of claim 1.

15. 1. A mobile device comprising: a vibration motor; an inertial motion unit (IMU); a processor; and a non-transitory computer-readable memory storing instructions that, when executed by the processor, cause the processor to: vibrating the vibration motor; using the IMU of the mobile device to acquire IMU data indicative of vibration damping caused by a user applying a force to the mobile device; determining the applied force based on a force model and the IMU data. Mobile devices.

16. 16. The mobile device of claim 15, wherein the force model comprises a machine learning algorithm trained to correlate (1) the IMU data measured by the mobile device and indicative of the vibration damping of the mobile device, and (2) the forces applied to the mobile device and causing the vibration damping.

17. 16. The mobile device of claim 15, wherein the IMU includes an accelerometer and a gyroscope, and the IMU data includes multi-axis accelerometer data and multi-axis gyroscope data.

18. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a mobile device, cause the mobile device to: vibrating a vibration motor of the mobile device; using an inertial measurement unit (IMU) of the mobile device to acquire IMU data indicative of vibration damping caused by a user applying a force to the mobile device; determining the applied force based on a force model and the IMU data; One or more non-transitory computer-readable media for causing operations to be performed, including:

19. The operation is and further comprising: obtaining the force model from a model library based on a type of the mobile device, the type of the mobile device being associated with at least one of a configuration of the IMU or a configuration of the vibration motor.

20. One or more non-transitory computer-readable media as recited in claim 18.

20. 20. The one or more non-transitory computer-readable media of claim 18, wherein the force model comprises a machine learning algorithm trained to correlate (1) the IMU data obtained from simultaneous measurements by an accelerometer and a gyroscope of the mobile device and indicative of the vibration damping of the mobile device, and (2) the forces applied to the mobile device that cause the vibration damping.

Images