Mobile device-based hand grip measurement

Abstract

Systems, devices, and methods for mobile device-based hand grip measurement are disclosed. In some aspects, a mobile device includes a vibration motor, an inertial motion unit (IMU), a processor, and a non-transitory computer readable memory, where the processor is configured to perform operations including: vibrating the vibration motor; obtaining IMU data using an inertial measurement unit of the mobile device, the IMU data indicating vibration damping caused by a user applying a force to the mobile device; and determining an applied force based on a force model and the IMU data.

Description

[0001] Cross-references to related applications

[0002] This patent document claims priority and benefit to U.S. Provisional Patent Application No. 63 / 385,379, filed November 29, 2022, entitled “SMARTPHONE-BASED HAND GRIP STRENGTH MEASUREMENT”. The entire contents of the aforementioned patent application are incorporated herein by reference as part of the disclosure of this patent document. Technical Field

[0003] This patent document relates to hand grip strength measurement based on a mobile device. Background Technology

[0004] Hand grip strength (HGS) is a functional biomarker of a user's overall condition, which can be used for individual or series of tests. As an indicator of physical strength, HGS can provide insight into a user's muscle mass decline or bone density problems. Furthermore, there may be correlations between HGS and cognitive function. These correlations may not be simply the correlation between age-related muscle mass decline and the cognitive system. Neurological decline can affect the ability and flexibility of the motor system, independent of musculoskeletal decline. Therefore, diseases affecting neurological decline may also affect the nervous system and HGS. In this way, HGS measurements can provide an indicator of vulnerability. Summary of the Invention

[0005] This document discloses methods, apparatus, and systems for measuring hand grip strength using sensor data acquired by built-in sensors of a mobile device without additional accessories.

[0006] One aspect of this document relates to a mobile device configured to measure hand grip force using sensor data acquired by built-in sensors of the mobile device without accessories. The mobile device may include: a vibration motor, an inertial measurement unit (IMU), a processor, and a non-transitory computer-readable storage device storing instructions that, when executed by the processor, cause the processor to perform operations including: vibrating the vibration motor; acquiring IMU data using the mobile device's inertial measurement unit, the IMU data indicating vibration damping caused by a force applied to the mobile device by a user; and determining the applied force based on a force model and the IMU data.

[0007] One aspect of this document relates to a method for measuring hand grip force using sensor data acquired by a built-in sensor of a mobile device without accessories. The method may include: vibrating a vibration motor of the mobile device; acquiring IMU data using an inertial measurement unit (IMU) of the mobile device, the IMU data indicating vibration damping caused by a force applied by a user to the mobile device; and determining the applied force based on a force model and the IMU data.

[0008] Another aspect of this document relates to one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a mobile device, cause the mobile device to perform any or more of the solutions described herein.

[0009] The above and other aspects of this document, as well as its embodiments and applications, are described in more detail in the accompanying drawings, description, and claims. Attached Figure Description

[0010] Figure 1 This shows a hand grip strength measurement performed using a hand dynamometer.

[0011] Figure 2 A diagram showing an example system implementing the disclosed technology.

[0012] Figure 3 Exemplary block diagrams of various components of a mobile device according to some embodiments of this document are shown.

[0013] Figure 4 A flowchart illustrating a process for measuring hand grip strength on a mobile device according to some embodiments of this document is provided.

[0014] Figure 5 Regression and BlandAltman plots of force measurements for three different smartphones according to some embodiments of this document are shown.

[0015] Figure 6 The document illustrates a comparison between forces measured using force sensors and stresses determined based on force models, according to some embodiments of this document.

[0016] Figure 7 The results of HGS measurements using mobile devices classified by level are shown according to some embodiments of this document.

[0017] Figure 8 This document illustrates an example graphical user interface for an application for HGS measurement according to some embodiments of this document. Detailed Implementation

[0018] This document describes systems, apparatus, and methods for measuring hand grip strength using built-in sensors in mobile devices. This method is applicable to any mobile device with a vibration motor and an inertial measurement unit (IMU). For the mobile phone applications disclosed herein, the mobile device can be made to vibrate due to the drive of its internal vibration motor, and the damping of these vibrations generated by the force applied by the user using the mobile device's IMU is monitored, and the applied force is quantified based on a force model and IMU data. The mobile device can determine the user's hand grip strength based on IMU data acquired using existing built-in sensors in the mobile device, without hardware modification, without physical accessories, and can be enabled solely using software by downloading an application on any mobile device, as discussed in further detail below. Therefore, the technology can provide large-scale screening opportunities for mental or physical disabilities. As a software-enabled measurement technology for any mobile device with an IMU and vibration motor, this technology can be rapidly deployed, including to underserved communities.

[0019] Because cognitive and musculoskeletal decline can affect HGS, HGS can be used as a functional biomarker to provide healthcare professionals with a warning sign of patients with severe mental or physical decline. As individuals age, muscle mass loss and HGS decline are expected; however, significant and unexpected declines in HGS may represent mental or physical impairments that, if left untreated, may continue to deteriorate.

[0020] Available Figure 1 The hand grip dynamometer shown is used to measure HGS (Hand Grip Strength). Hand grip dynamometers typically measure isometric strain over a maximum of 5 seconds. For example, a user can perform an HGS test using one hand, placing the metacarpal bone at the bottom of the device and the lever between the first and second knuckles. The user can pull with maximum effort using the measuring hand, and the strength classification (weak, medium, or strong) is obtained based on the measurement results and demographic data.

[0021] Measuring the force applied by a user's hand using a force gauge can be similarly performed using the built-in sensors and actuators of commercial mobile devices. When the mobile device vibrates, IMU (containing accelerometers and / or gyroscopes) readings may reflect the vibration. These signals may attenuate due to increased force (or weight). In addition to focusing on static or binary metrics, the attenuation of the mobile device's IMU (containing, for example, accelerometers and gyroscopes) during vibration can be used to measure HGS. As shown elsewhere in this document, HGS sensing on a mobile device can provide results that are continuous in scale and / or categorized by level.

[0022] The HGS measurement based on mobile devices disclosed herein requires no accessories and operates on various types of mobile devices with vibration motors and IMUs. Most mobile devices already contain IMUs with accelerometers and gyroscopes for purposes including, for example, measuring linear acceleration and angular velocity, enabling these mobile devices to perform position inference; these mobile devices are widely used for screen rotation, gaming, step counting, gesture recognition, etc. Similarly, most mobile devices already contain vibration motors to provide haptic feedback for messaging, phone calls, and screen interactions. HGS measurements can be implemented on such mobile devices based solely on force-damping techniques utilizing only the mobile device's vibration motor and IMU. To measure force, the mobile device can be set to vibrate (e.g., vibrate to maximum extent), while the IMU measures the motion as the device oscillates (caused by the vibration). When a force is applied during vibration, the IMU signal changes as the oscillation is altered or attenuated. According to embodiments of this document, vibration attenuation is measured and relied upon to model the applied force as a damping force against the vibration. By relying on components common to different types of mobile devices, the HGS measurements disclosed herein function across different types of mobile devices (e.g., the type may correspond to a manufacturer's model).

[0023] According to some embodiments of this document, IMU data, including multi-axis accelerometer data and multi-axis gyroscope data (collectively representing vibration damping), is used to determine the stresses causing vibration damping. This method identifies couplings between different axes of the gyroscope and accelerometer data, which respond differently to applied forces. Due to energy conservation, the applied force does not simply result in a decrease in vibration damping or amplitude along all IMU axes. On the contrary, some axes may even see an increase, as the applied force used to attenuate vibrational motion on one axis may cause vibrational energy to be transferred to another axis. The disclosed method takes multiple inputs from both the accelerometer and gyroscope as inputs to study the relationship between frequency, amplitude, and different axes. As used herein, "damping" or "vibration damping" refers to the recorded changes in device motion caused by an applied hand grip force.

[0024] The applied force causing vibration damping of a mobile device can be quantified based on a force model trained to correlate (1) IMU data indicating vibration damping of the mobile device as measured by the mobile device with (2) the force applied to the mobile device and causing the corresponding vibration damping. The force model may incorporate machine learning algorithms, such as multiple linear regression algorithms. The correlation may depend on the configuration of the mobile device. Vibration damping may depend on one or more factors affecting how the mobile device absorbs and dissipates vibration energy. Examples of these factors include vibration intensity, load distribution within the mobile device (including, for example, the distribution of weight and components within the mobile device), materials used in the mobile device (including, for example, density, elasticity, and internal friction), overall design (including, for example, shape and structural integrity, features such as ribs, gussets), interactions between components within the device, etc., or combinations thereof. Furthermore, damping measured by the mobile device's IMU may depend on one or more factors, including, for example, the position of the IMU relative to one or more vibration motors, sensor parameters (including, for example, sensitivity, resolution, noise level, etc.), or combinations thereof. The techniques disclosed herein are widely applicable and adaptable to various types of mobile devices, enabling similar force estimation performance to be obtained across a variety of mobile devices. A force model can be determined for a certain type of mobile device (e.g., a manufacturer's model), and the force model can be used to calibrate mobile devices of the same type.

[0025] The calibration procedure allows for standardized measurements across mobile devices. Calibration ensures that different mobile devices of the same or different types are configured to measure force consistently. This is beneficial because developing force models is an automated or at least partially automated process that can be completed in a short time (e.g., a day) and does not require large-scale participant recruitment or clinical measurements.

[0026] The technical improvements provided in the embodiments disclosed herein include methods for performing HGS measurements that are applicable to different types (e.g., models) of mobile devices without hardware modifications or accessories, and can be implemented virtually through a downloadable application. Promoting sensor-based measurements across a wide variety of mobile devices is a proven problem that severely limits the impact of work in the mobile health field. The cross-device compatibility results disclosed herein demonstrate that the disclosed force-damping technique allows HGS measurements to be performed on a range of mobile devices. Data processing and modeling show that many different types of mobile devices can perform HGS measurements after calibration based on mobile device type. Each popular type of mobile device has tens of millions of users, and a single factory calibration could enable millions of mobile devices to perform hand grip strength measurements.

[0027] Figure 2An exemplary system demonstrating the disclosed technology for implementing hand grip strength measurement based on a mobile device is provided. System 200 includes a mobile device 202, which may include at least one of a camera 204, a vibration motor 205, an IMU 206, a processor 208, a wireless transmitter 210, and a display 212. The processor 208 can control the operation of the mobile device 202 (e.g., cause the vibration motor 205 to vibrate, cause the camera 204 and / or IMU 206 to acquire data), receive and process sensor data (e.g., image data acquired by the camera 204, IMU data acquired by the IMU 206), and run algorithms (e.g., force models) on the sensor data and generate results (e.g., the applied force causing vibration damping, force map, diagnostics based on the determined HGS, etc.). Figure 2 As shown, mobile device 202 can communicate with user 214 or external devices or systems (e.g., cloud server 216). For example, mobile device 202 can use, for example, a wireless transmitter 210 to send system status reports to cloud server 216. As another example, mobile device 202 can communicate with user 214 via display 212. Display 212 can be a touchscreen configured as a graphical user interface, allowing mobile device 202 to present data or results to user 214 and receive user input via display 212. In some embodiments, mobile device 202 can be a smartphone, tablet computer, etc.

[0028] Figure 3 Exemplary block diagrams of various components of a mobile device according to some embodiments of this document are shown. The mobile device 300 is... Figure 2 An example of a mobile device 202 is shown. In some embodiments, the mobile device 300 may be a smartphone, tablet computer, etc. In some embodiments, the mobile device 300 may vibrate by a built-in vibration motor 303, using an IMU to measure the vibration damping caused by a force applied to the mobile device 300 by a user, and using a force model to estimate the HGS based on the IMU data. The mobile device 300 includes: one or more sensors 302 for collecting data; a processing unit 304 connected to the one or more sensors 302 and capable of performing a force model on the collected data; a wireless transceiver 306 connected to the processing unit 304; and a display 308 connected to the processing unit 304. The one or more sensors 302 may include one or more of a camera or an IMU. The IMU of the mobile device 300 may include an accelerometer and a gyroscope.

[0029] Figure 4 A flowchart illustrating a process for measuring hand grip strength on a mobile device according to some embodiments of this document is provided. Process 400 may be implemented on a mobile device (e.g., mobile device 202, mobile device 300).

[0030] At box 410, process 400 includes vibrating a vibration motor of the mobile device. The vibration motor may be a vibration motor already built into the mobile device. For example, the vibration may be driven by a vibration motor used to provide haptic feedback for messages, telephone calls, and screen interactions. The vibration motor may be made to vibrate at a specific level (e.g., the maximum level) of the vibration motor.

[0031] At box 420, process 400 includes acquiring IMU data using the inertial measurement unit (IMU) of the mobile device, the IMU data indicating vibration damping caused by a force applied to the mobile device by a user. To measure HGS, the user applies a force by gripping the mobile device with their hand.

[0032] In some embodiments, process 400 may include providing a visual guide to be presented on the display of the mobile device. The visual guide (also known as a visual signifier) ​​includes markings to guide the user in applying force by gripping the mobile device. Process 400 may include providing information on how the user should perform an HGS measurement using the mobile device, including, for example, how the user should sit, apply force, etc., or combinations thereof, as part of preparation. For example, instructions may include: the user should sit upright, place the measuring device (mobile device) directly in front of the user, relax the wrist naturally, bend the elbow at 90 degrees, place the user's feet flat on the ground, remain calm and breathe normally during the measurement, and grip the mobile device with the edge of the device between the user's fingers and palm. The mobile device may provide these instructions to the user via a user interface implemented on, for example, the display of the mobile device. Instructions may be presented in the form of text, images, cartoons, video, audio messages, etc., or combinations thereof. For example, instructions may be presented as visual guidance, including images or cartoons, showing suggested user fingers and / or suggested finger positions of the user's hand when the user grips the mobile device. As another example, instructions may be presented as a video showing a model user performing HGS measurements using the sample mobile device.

[0033] As instructed, the user can apply damping force by gripping the mobile device when it vibrates. The user can apply maximum force for a period of time (e.g., approximately 5 seconds) until the measurement is complete. The mobile device can notify the user when the measurement is complete.

[0034] In some embodiments, process 400 may include receiving information about a user performing an HGS measurement and assessing whether the user is ready to perform the measurement and / or whether the measurement is valid. For example, the mobile device may detect the position of a user's fingers gripping the mobile device and assess whether the grip is suitable for the user to continue the measurement. In some embodiments, position detection may be performed based on one or more images captured by a camera of the mobile device or another device (e.g., another mobile device, a camera mounted on a wall or placed on a surface, etc.). In some embodiments, the mobile device includes a touchscreen (e.g., display 212) configured to sense user touch, and process 400 may detect contact between one or more of the user's fingers and the touchscreen and determine one or more positions of one or more fingers based on the detected contact. As another example, the mobile device may detect the posture of the user or a part thereof (e.g., the posture of the user's arm or wrist) by capturing one or more images of the user or a portion thereof to assess whether the user is properly positioned to perform the measurement. In some embodiments, process 400 may include providing a notification to the user based on the assessment. For example, in response to an assessment that infers the user needs to make adjustments (e.g., adjustments in grip or arm posture), the mobile device may provide the user with a notification to suggest adjustments and / or further detect or assess whether the user has made the suggested adjustments.

[0035] The IMU of a mobile device may include a multi-axis accelerometer and a multi-axis gyroscope. For other purposes, the vibration motor may be a vibration motor already built into the mobile device, such as measuring linear acceleration and angular velocity, enabling these mobile devices to perform position inference, and these mobile devices are widely used for screen rotation, gaming, step counting, gesture recognition, etc.

[0036] IMU data can include multi-axis accelerometer data and multi-axis gyroscope data. For example, IMU data can include three-axis accelerometer data and three-axis gyroscope data. The three-axis accelerometer data can include the linear acceleration of the IMU on each axis (indicating smartphone motion). Depending on the degree of energy dissipation during vibration damping, the linear acceleration of the IMU can change in different modes. For example, during certain periods of time when damping force is applied, the linear acceleration along different directions may decrease at different rates, or even follow opposite trends (e.g., linear acceleration increases in one direction while decreasing in another). Similarly, the angular velocity of the IMU along different directions (indicating smartphone motion) can change in different modes in response to the damping force applied by the user. Therefore, IMU data including multi-axis accelerometer data and multi-axis gyroscope data can be used in combination to provide a comprehensive representation of vibration damping, and thus improve the accuracy of determining the applied force.

[0037] Each axis of the multi-axis accelerometer and multi-axis gyroscope data may contain a time-series signal. The signal of the axis in the IMU data may contain a high-frequency component from the vibration motor and a low-frequency component corresponding to noise. In some embodiments, raw IMU data can be sampled from the mobile device at maximum rate without device filtering or post-processing. In some embodiments, the raw IMU data may undergo a series of processing steps. For example, one or more of the following techniques can be used to process the raw IMU data: low-pass filtering, high-pass filtering, band-pass filtering, savgol filtering, standard deviation, or empirical mode decomposition. Filtering may also involve multiple steps, such as filter banks or multiple stages with different filter types. Additional characteristics, such as the signal power of the axis combination, can be obtained from the filtered or raw IMU data. Various metrics can be used to evaluate one or more characteristics, including, for example, principal component analysis, recursive feature elimination, LASSO regression, and correlation.

[0038] At box 430, process 400 includes determining the applied force based on the force model and IMU data.

[0039] In some embodiments, process 400 may include decomposing IMU data based on empirical mode decomposition (EMD) techniques. According to EMD techniques, process 400 may include, for each axis of the multi-axis accelerometer data and multi-axis gyroscope data, decomposing the signal of the axis into a plurality of intrinsic mode functions (IMFs) representing different frequency components of the signal, such that the applied force can be determined based on at least a portion of the IMF corresponding to each axis of the multi-axis accelerometer data and multi-axis gyroscope data.

[0040] Since the high-frequency components of the signals for each axis of the vibration motor are the strongest components of the corresponding signals, the first few IMFs of each axis of the IMU data primarily correspond to the vibration motor signals. The upper envelope of the first IMF for each axis of the multi-axis accelerometer data and multi-axis gyroscope data can be used as a feature for identifying the applied force. In some embodiments, the upper envelope of the first IMF on each IMU axis serves as a representative feature, and the first IMF can collectively and most significantly correlate with the force data and thus be used as input to the force model to determine the applied force, while excluding other features. Thus, by way of example only, process 400 may include, for each axis of the multi-axis accelerometer data and multi-axis gyroscope data, identifying the first IMF corresponding to the highest frequency from the multiple IMFs of the axis's signals; identifying the upper envelope of the first IMF of the axis; and inputting the upper envelopes of the first IMFs corresponding to all axes of the multi-axis accelerometer data and multi-axis gyroscope data, respectively, into the force model.

[0041] The EMD techniques for processing IMU data described herein are for illustrative purposes only and are not intended to be limiting. As described elsewhere in this document, IMU data can be processed by at least one of the following methods before being input into the force model: filtering, filter bank, averaging, standard deviation, wavelet decomposition, spectral analysis, or EMD. In some embodiments, different techniques may be used to process signals from at least two different axes of the IMU data (including multi-axis accelerometer data and multi-axis gyroscope data). In some embodiments, the raw IMU data may be directly input into the force model.

[0042] The force model may include a machine learning model trained to correlate (1) IMU data indicating the vibration damping of the mobile device as measured by the mobile device with (2) the force applied to the mobile device that causes the corresponding vibration damping. The force model may be a data-driven model, including parametric modeling, linear regression models, ensemble learning models (random forest, AdaBoost, etc.), support vector machine models, neural networks (e.g., transformer models, convolutional neural networks (CNNs), etc.), variations thereof, or combinations thereof. For example only, the force model may include a multiple linear regression model that provides values ​​of the applied force on a continuous scale.

[0043] The force model can be trained using training data comprising the applied force and the measured damping caused by the applied force, measured using a force sensor (e.g., a force-sensitive resistor (FSR)). Example test cases were executed across various smartphone manufacturers, physical shapes, costs, and embedded components, involving three different types of smartphones: the Google Pixel 4, Samsung Galaxy A53, and Motorola Moto G Power. In the example test cases, each participant applied pressure by gripping the smartphone, with a 0.3mm thick force sensor placed under each finger on the side of the smartphone. During the measurement, real-time readings of the force sensors were displayed on a screen in front of the user using a force guide. The user was instructed to apply a series of pressures following the force guide using real-time feedback. The acquired IMU data was processed in substantially the same manner as described above to provide vibration damping data. The force and corresponding vibration damping data measured by the force sensors were used to train and validate the force model. Figure 5 The table displays the mean absolute error, correlation coefficient, and bias for each smartphone. Figure 5 In the diagram, solid dots represent data corresponding to the Motorola Moto G Power, crosses represent data corresponding to the Samsung Galaxy A53, and stars represent data corresponding to the Google Pixel 4.

[0044] As discussed elsewhere in this document, the vibration damping behavior of a mobile device can depend on its configuration. Example test cases demonstrate that the relevant differences in force measurement between different types of mobile devices (e.g., type may correspond to manufacturer model) can be calibrated using different force models. When a new type of mobile device (e.g., a new model from a manufacturer) is released, the force model can be trained, and the obtained force model can be used to calibrate mobile devices of the same type. The type of mobile device can correspond to the model and / or manufacturer of the mobile device and is associated with multiple factors, including, for example, the configuration of the mobile device's vibration motor, the configuration of the IMU, etc., or combinations thereof.

[0045] A model library can be built by compiling force models corresponding to different types of mobile devices. In some embodiments, process 400 may include obtaining force models from such a model library based on the type of mobile device as preparation for a hand grip strength measurement procedure.

[0046] Process 400 can determine the applied force substantially in real time. In some embodiments, during measurement, process 400 may include providing visual guidance to be presented on a display of a mobile device, wherein the visual guidance includes a force map showing the applied force substantially in real time. In some embodiments, the visual guidance may further include a force guide line superimposed on the applied force. The user may be instructed to apply a series of forces following the force guide using real-time feedback.

[0047] In some embodiments, process 400 may include measuring the surface area of ​​a body part, wherein a force is applied by pressing the surface of the body part against a camera. For example only, when the body part is pressed against the camera to apply a damping force, process 400 may cause the camera to capture an image of the surface of the body part and determine the area based on the image. As another example, the mobile device includes a touchscreen configured to sense user contact, and process 400 may determine the area by identifying the contact area between the body part and the display. Process 400 may include determining the applied pressure based on the applied force and area.

[0048] Figure 6 This document illustrates a comparison between the force measured using a force sensor (curve 602) and the stress determined based on a force model (curve 604) according to some embodiments of this document. A calibrated linear force-sensitive resistor (FSR) is used to measure the applied force during gripping. As an initial setup, the FSR is placed between the user's finger and the smartphone. During gripping, accelerometer and gyroscope data are sampled while the FSR records the applied force. Linear regression fitting indicates that smartphone-based HGS measurements can be performed to continuously track grip force.

[0049] Figure 7The results of HGS measurements using a mobile device categorized by level according to some embodiments of this document are shown. Figure (I) shows the linear X-axis acceleration of a smartphone IMU over a time period during which the smartphone vibrates and forces are applied to cause vibration damping. Figure (II) shows the HGS determined according to embodiments of this disclosure and categorized by level based on the user's HGS and / or demographic information. Relevant demographic information includes age, sex, health status, medical history, etc., or combinations thereof. As shown, higher HGS levels generally correspond to lower linear acceleration along the X-axis.

[0050] As described elsewhere in this document, the IMU of a mobile device may include accelerometers and gyroscopes. Although only linear acceleration along the X-axis is shown, according to some embodiments of this document, the IMU of the mobile device measures acceleration in multiple directions (e.g., three vertical directions, including a direction along the Z-axis and two directions in a plane perpendicular to the Z-axis), and gyroscope data in multiple directions (e.g., the same three directions as the accelerometers). Vibration and damping of the mobile device can be monitored based on IMU data that includes multi-axis accelerometer data combined with multi-axis gyroscope data. Due to the way vibration damping occurs, the linear acceleration of the IMU on each axis (indicating smartphone motion) can change in different patterns. For example, as the applied force increases, the linear acceleration along the X-axis decreases; however, the linear acceleration along different axes can decrease at different rates or even increase for certain periods of time when damping force is applied. Similarly, the angular velocity of the IMU along different directions (indicating smartphone motion) can change in different patterns in response to damping force applied by the user. Therefore, IMU data from a multi-axis accelerometer, which includes data from a multi-axis gyroscope, can be used to determine the applied force.

[0051] Figure 8 Examples of graphical user interfaces (GUIs) for HGS measurements according to some embodiments of this document are shown. During the measurement, a user may grip a mobile device (e.g., a smartphone) with a force (e.g., the user's maximum force) for a period of time (e.g., 5 seconds) while the device vibrates. The resulting signal can be processed to determine the grip force substantially in real time. As shown, the results can be presented on the GUI in real time. The GUI may present additional information, including, for example, instructions to guide the user in performing HGS measurements using a smartphone, features determined based on the determined HGS value, further analysis or actions desired by the user (e.g., presenting the results in different ways (e.g., categorized results, text results, an overview of the results and related information) or a portion thereof, sending the results to another person or device, etc.).

[0052] The disclosed mobile device-based HGS measurements can be used to assess a user's condition. For example, for surgical patients, hand grip strength typically decreases after surgery and recovers as the patient recovers. If hand grip strength does not return to pre-operative levels, it may indicate surgical complications. Similarly, hand grip strength can be monitored at home to screen for unexpected strength declines, which may indicate illness or other vulnerabilities. As another example, the disclosed technology can be used to assess finger strength to determine the body's readiness for exercise. A maximum squeeze might assess whether the body is ready for high-intensity physical activity. These squeezes may not necessarily be hand grip squeezes. Especially for people with greater strength, squeezing a smartphone with the thumb and forefinger may be more effective. Notifications related to a user's HGS measurement results can be generated and provided to the user or others (e.g., healthcare providers), another device, and / or another entity (e.g., clinics, research institutions, device manufacturers). The notification may include a report containing the results and / or a user's condition determined based on the results.

[0053] Example

[0054] The following examples illustrate several embodiments of the present technology. Other exemplary embodiments of the present technology may be presented before or after the examples listed below.

[0055] 1. A mobile device comprising: a vibration motor, an inertial measurement unit (IMU), a camera, a processor, and a non-transitory computer-readable storage device storing instructions, the instructions, when executed by the processor, causing the processor to perform operations including: vibrating the vibration motor; acquiring IMU data using the inertial measurement unit of the mobile device, the IMU data indicating vibration damping caused by a force applied to the mobile device by a user; and determining the applied force based on a force model and the IMU data.

[0056] 2. Any one or more of the mobile devices described in the solutions described herein, wherein the force model includes a machine learning algorithm trained to correlate (1) IMU data of vibration damping of the mobile device measured by the mobile device with (2) the force applied to the mobile device and causing vibration damping.

[0057] 3. Any one or more of the mobile devices described in the solutions described herein, wherein the IMU includes an accelerometer and a gyroscope, and its IMU data includes multi-axis accelerometer data and multi-axis gyroscope data.

[0058] 4. A method for measuring hand grip strength using a mobile device, the method comprising: vibrating a vibration motor of the mobile device; acquiring IMU data using an inertial measurement unit (IMU) of the mobile device, the IMU data indicating vibration damping caused by a force applied to the mobile device by a user; and determining the applied force based on a force model and the IMU data.

[0059] 5. The method described in any one or more of the solutions described herein, wherein the IMU data includes multi-axis accelerometer data and multi-axis gyroscope data.

[0060] 6. The method according to any one or more of the solutions described herein, wherein determining the applied force based on IMU data comprises processing the IMU data based on at least one of filtering, filter bank, averaging, standard deviation, wavelet decomposition, spectral analysis, or EMD.

[0061] 7. The method according to any one or more of the solutions described herein, wherein: each axis of the multi-axis accelerometer data and multi-axis gyroscope data contains a signal; the applied force is determined based on IMU data for each axis of the multi-axis accelerometer data and multi-axis gyroscope data; the signal of the axis is decomposed into a plurality of intrinsic mode functions (IMFs) representing different frequency components of the signal; and the applied force is determined based on at least a portion of the IMFs corresponding to each axis of the multi-axis accelerometer data and multi-axis gyroscope data.

[0062] 8. The method according to any one or more of the solutions described herein, wherein determining the applied force based on IMU data further comprises: for each axis of the multi-axis accelerometer data and multi-axis gyroscope data, identifying a first IMF corresponding to the highest frequency from a plurality of IMFs of the axis's signal; and identifying the upper envelope of the first IMF of the axis; and inputting the upper envelopes of the first IMFs corresponding to all axes of the multi-axis accelerometer data and multi-axis gyroscope data, respectively, into the force model.

[0063] 9. The method according to any one or more of the solutions described herein, further comprising: providing a visual guide to be presented on a display of a mobile device, wherein the visual guide includes a force map showing the applied force substantially in real time.

[0064] 10. The method of any one or more of the solutions described herein, wherein the visual guidance further comprises a force guide line superimposed on the applied force.

[0065] 11. The method according to any one or more of the solutions described herein, further comprising: providing visual guidance to be presented on a display of a mobile device, wherein the visual guidance includes markers for guiding a user to apply force over a period of time.

[0066] 12. The method according to any one or more of the solutions described herein, further comprising: providing visual guidance to be presented on a display of a mobile device, wherein the visual guidance includes markings for guiding a user to apply force by gripping the mobile device with the user's hand.

[0067] 13. The method described in any one or more of the solutions described herein further comprises: assessing the user's condition based on the applied force; and generating a notification based on the assessment result.

[0068] 14. The method described in any one or more of the solutions described herein further comprises obtaining a force model from a model library based on the type of mobile device, wherein the type of mobile device is associated with at least one of the configuration of the IMU or the configuration of the vibration motor.

[0069] 15. The method according to any one or more of the solutions described herein, wherein: the force model includes a machine learning algorithm trained to correlate (1) IMU data indicating the vibration damping of the mobile device measured by the mobile device with (2) the force applied to the mobile device and causing vibration damping.

[0070] 16. The method described in any one or more of the solutions described herein, wherein the force model comprises a multiple linear regression model trained based on training data acquired using a force sensor and a training mobile device of the same type as the mobile device.

[0071] 17. The method according to any one or more of the solutions described herein, wherein the force model is configured to generate results on a continuous scale, and the method further comprises: assigning the applied force to one of a plurality of categories.

[0072] 18. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a mobile device, cause the mobile device to perform any or more of the solutions described herein.

[0073] The embodiments of the subject matter and operations described in this patent document can be implemented in various systems, digital electronic circuit systems, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in one or more combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more computer program instruction modules encoded on a tangible and non-transitory computer-readable medium for execution by or control of the operation of a data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage device substrate, a memory device, a material that performs machine-readable propagation signals, or a combination thereof. The terms "data processing unit" or "data processing device" encompass all devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. In addition to hardware, the device may also contain code that generates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination thereof.

[0074] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language, including compiled or interpreted languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. Computer programs do not necessarily correspond to files in a file system. Programs can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language file), in a single file dedicated to the program under discussion, or in multiple coordinating files (e.g., files storing portions of one or more modules, subroutines, or code). Computer programs can be deployed to execute on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected via a communication network.

[0075] The processes and logic flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be executed by dedicated logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as such dedicated logic circuit systems.

[0076] Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer does not necessarily need to have these devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit system.

[0077] The specification and accompanying drawings are intended to be illustrative only, where illustrative means example. As used herein, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” are intended to also include the plural forms. Additionally, unless the context clearly indicates otherwise, the use of “or” is intended to include “and / or”.

[0078] Although this patent document contains numerous specific details, these details should not be construed as limiting the scope or claimable content of any invention, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the context of individual embodiments in this patent document may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually in multiple embodiments or in any suitable sub-combination. Furthermore, although the features described above may be described as operating in certain combinations, and even initially claimed, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations thereof.

[0079] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or in chronological order, or requiring all shown operations to achieve the desired result. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring this separation in all embodiments.

[0080] Various embodiments described herein are described in the general context of methods or processes, which may be implemented in one embodiment by a computer program product contained in a computer-readable medium, comprising computer-executable instructions such as program code that are executed by a computer in a networked environment. The computer-readable medium may comprise removable and non-removable storage devices, including, but not limited to, read-only memory (ROM), random access memory (RAM), optical disc (CD), digital versatile optical disc (DVD), Blu-ray disc, etc. Therefore, the computer-readable medium described herein includes non-transitory storage media. Generally, a program module may comprise routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. The computer-executable instructions, associated data structures, and program modules represent instances of program code for performing steps of the methods disclosed herein. Specific sequences of these executable instructions or associated data structures represent instances of corresponding actions for implementing the functions described in these steps or processes.

[0081] For example, one aspect of the disclosed embodiments relates to a computer program product embodied on a non-transitory computer-readable medium. The computer program product includes program code for performing any and / or all of the operations of the disclosed embodiments.

[0082] Only a few implementation schemes and examples are described, and other implementation schemes, enhancements and variations may be made based on the content described and illustrated in this patent document.

Claims

1. A method for force measurement using a mobile device, the method comprising: The vibration motor of the mobile device is made to vibrate; The mobile device uses its inertial measurement unit (IMU) to acquire IMU data indicating vibration damping caused by forces applied by the user to the mobile device; and The applied force is determined based on the force model and the IMU data.

2. The method according to claim 1, wherein the IMU data includes multi-axis accelerometer data and multi-axis gyroscope data.

3. The method of claim 2, wherein determining the applied force comprises processing the IMU data based on at least one of filtering, filter bank, averaging, standard deviation, wavelet decomposition, spectral analysis, or empirical mode decomposition (EMD).

4. The method according to claim 3, wherein: Each axis of the multi-axis accelerometer data and the multi-axis gyroscope data includes a signal. The applied force is determined to include, for each axis of the multi-axis accelerometer data and the multi-axis gyroscope data, the signal of that axis is decomposed into multiple intrinsic mode functions (IMFs) representing different frequency components of the signal, and The applied force is determined based on at least a portion of the IMF corresponding to each axis of the multi-axis accelerometer data and the multi-axis gyroscope data.

5. The method of claim 4, wherein determining the applied force further comprises: For each axis of the multi-axis accelerometer data and the multi-axis gyroscope data Identify the first IMF corresponding to the highest frequency from the plurality of IMFs of the signal of the axis; Identify the upper envelope of the first IMF of the axis; as well as The upper envelope of the first IMF, which corresponds to all axes of the multi-axis accelerometer data and the multi-axis gyroscope data, is input into the force model.

6. The method of claim 1, further comprising: Provide visual guidance to be presented on the display of the mobile device, wherein the visual guidance includes a force diagram showing the applied force substantially in real time.

7. The method of claim 6, wherein the visual guidance further comprises a force guide line superimposed on the applied force.

8. The method of claim 7, wherein the visual guidance includes a marker for guiding the user to apply the force over a period of time.

9. The method of claim 1, further comprising: Provide visual guidance to be presented on the display of the mobile device, wherein the visual guidance includes markings for guiding the user to apply the force by gripping the mobile device with the user's hand.

10. The method of claim 1, further comprising: The user's condition is assessed based on the applied force; as well as A notification is generated based on the results of the assessment.

11. The method of claim 1, further comprising: The force model is obtained from a model library based on the type of the mobile device, wherein the type of the mobile device is associated with at least one of the configuration of the IMU or the configuration of the vibration motor.

12. The method of claim 1, wherein the force model comprises a machine learning algorithm trained to correlate (1) the IMU data, measured by the mobile device, indicating the vibration damping of the mobile device, with (2) the force applied to the mobile device and causing the vibration damping.

13. The method of claim 1, wherein the force model comprises a multiple linear regression model trained based on training data acquired using a force sensor and a training mobile device of the same type as the mobile device.

14. The method according to claim 1, wherein: The force model is configured to generate results at a continuous scale, and The method further includes: assigning the applied force to one of a plurality of categories.

15. A mobile device comprising: A vibration motor, an inertial motion unit (IMU), a processor, and a non-transitory computer-readable storage memory for storing instructions, which, when executed by the processor, cause the processor to perform operations including: The vibration motor is made to vibrate; The mobile device uses its IMU to acquire IMU data indicating vibration damping caused by a force applied by the user to the mobile device; and The applied force is determined based on the force model and the IMU data.

16. The mobile device of claim 15, wherein the force model comprises a machine learning algorithm trained to correlate (1) the IMU data, measured by the mobile device, indicating the vibration damping of the mobile device, with (2) the force applied to the mobile device and causing the vibration damping.

17. The mobile device of claim 15, wherein the IMU includes an accelerometer and a gyroscope, and the IMU data includes multi-axis accelerometer data and multi-axis gyroscope data.

18. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a mobile device, cause the mobile device to perform operations comprising: The vibration motor of the mobile device is made to vibrate; The mobile device uses its inertial measurement unit (IMU) to acquire IMU data indicating vibration damping caused by forces applied by the user to the mobile device; and The applied force is determined based on the force model and the IMU data.

19. One or more non-transitory computer-readable media according to claim 18, wherein the operation further comprises: The force model is obtained from a model library based on the type of the mobile device, wherein the type of the mobile device is associated with at least one of the configuration of the IMU or the configuration of the vibration motor.

20. One or more non-transitory computer-readable media according to claim 18, wherein the force model includes a machine learning algorithm trained to correlate (1) the IMU data obtained from concurrent measurements by the accelerometer and gyroscope of the mobile device and indicating the vibration damping of the mobile device with (2) the force applied to the mobile device and causing the vibration damping.

Images