Systems and methods for exercise type identification using wearable devices

Abstract

The present disclosure provides for using multiple inertial measurement units (IMUs) to identify specific user activities, such as specific types of exercise and repetitions of such exercise. The IMUs can be located in consumer products such as smartwatches and earbuds. Each IMU can include an accelerometer and a gyroscope, with each accelerometer and gyroscope having three measurement axes, for a total of 12 raw measurement streams. A training image includes a plurality of sub-curves or tiles, each sub-curve or tile depicting an individual data stream. The training image is then used to train a machine learning model to identify the IMU data as corresponding to a specific exercise type.

Description

Background Art

[0001] Historically, activity recognition using inertial sensors has been achieved by designing complex "features" from raw sensor data, such as biases, peak-to-peak values, zero crossings, kurtosis, and a large number of other measurable quantities. These features are then used as inputs to machine learning models, such as support vector machines, random forests, k-nearest neighbors, etc. Figuring out which features are important or useful for machine learning, or which model type to use, is a very difficult task that requires a great deal of expertise and know-how.

[0002] There has been some preliminary research using image-based neural networks to perform inertial measurement unit (IMU) activity recognition. Image-based models tend to be more mature and general. In this sense, "feature"-based models seem similar to "feature"-based extraction in the early days of image recognition, which had limited success. Existing image-based systems use peak and threshold detectors to determine where to window individual repetitions of exercise types. However, the thresholds will vary from person to person and can be easily disrupted in the presence of noise. Additionally, there is no general guidance for this method across all exercise types, which requires a customized analysis window detection method for each possible exercise type while also trying to make the method robust across multiple users. Summary of the Invention

[0003] The present disclosure provides for using multiple IMUs to recognize specific user activities, such as specific types of exercise and repetitions of such exercise. The IMUs can be located in consumer products such as smartwatches and earbuds.

[0004] One aspect of the present disclosure provides a method for detecting exercise. The method includes receiving first sensor data from one or more first sensors of a first wearable device by one or more processors; receiving second sensor data from one or more second sensors of a second wearable device by the one or more processors; generating, by the one or more processors, an image based on the first sensor data and the second sensor data, the image including a plurality of sub-curves, where each sub-curve depicts a data stream; and determining, using the one or more processors, an exercise type performed by a user during the receiving of the first sensor data and the second sensor data based on the image. Determining the exercise type performed may include executing a machine learning model, such as an image-based machine learning model. Additionally, the model may be trained, for example, by generating one or more first training images based on data collected from the first wearable device and the second wearable device when the user performs a first type of exercise and inputting the one or more first training images as training data into the machine learning model. Training may also include generating one or more second training images based on data collected from the first wearable device and the second wearable device when the user performs a second type of exercise and inputting the one or more second training images as training data into the machine learning model, and generating one or more third training images based on data collected from the first wearable device and the second wearable device when the user performs one or more activities not classified as exercise and inputting the one or more third training images as examples of non-exercise into the machine learning model. Generating one or more first training images may include peak detection window segmentation or fixed window segmentation. Fixed window segmentation may include changing a start time of a window having a predetermined time length, where each change in the start time in the window generates a separate first training image among the one or more first training images. The method may also include automatically scaling a time length of the window based on a recent repetition frequency estimate.

[0005] According to some examples, the method may further include determining a number of repetitions of the determined exercise type. For example, this may include determining a repetition frequency using a sliding autocorrelation window and calculating the number of repetitions based on the repetition frequency and a duration of the exercise type. In another example, determining the number of repetitions may include counting the number of repetitions in a training image, marking the repetitions in the training image, and inputting the training image as training data into a machine learning model.

[0006] Another aspect of the present disclosure provides a system for detecting exercise, including one or more first sensors in a first wearable device; one or more second sensors in a second wearable device; and one or more processors communicatively coupled to the one or more first sensors and the one or more second sensors, the one or more processors being configured to: receive first sensor data from the one or more first sensors of the first wearable device; receive second sensor data from the one or more second sensors of the second wearable device; generate an image based on the first sensor data and the second sensor data, the image including a plurality of sub-curves, where each sub-curve depicts a data stream; and determine an exercise type performed by a user during receipt of the first sensor data and the second sensor data based on the image. According to some examples, the first wearable device may be earbuds, and the second wearable device may be a smartwatch. The one or more processors may reside in at least one of the first wearable device or the second wearable device and / or in a host device coupled to the first wearable device and the second wearable device.

[0007] Yet another aspect of the present disclosure provides a non-transitory computer-readable medium storing instructions executable by one or more processors for performing a method of detecting an exercise type. The method may include receiving first sensor data from the one or more first sensors of the first wearable device; receiving second sensor data from the one or more second sensors of the second wearable device; generating an image based on the first sensor data and the second sensor data, the image including a plurality of sub-curves, where each sub-curve depicts a data stream; and determining an exercise type performed by a user during receipt of the first sensor data and the second sensor data based on the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a schematic diagram of an example system used by a user in accordance with aspects of the present disclosure.

[0009] Figure 2 is a block diagram of an example system in accordance with aspects of the present disclosure.

[0010] Figure 3 is a schematic diagram of another example system in accordance with aspects of the present disclosure.

[0011] Figures 4A - 4B shows example raw data in accordance with aspects of the present disclosure.

[0012] Figures 5A - 5E shows example images generated from data using a first technique and corresponding to various exercise types in accordance with aspects of the present disclosure.

[0013] Figure 6 Shows an example image generated from data corresponding to non-exercise according to aspects of the present disclosure.

[0014] Figures 7A - 7B Shows an image generated from data using a second technique and corresponding to an example exercise according to aspects of the present disclosure.

[0015] Figure 8 Is an example graph showing autocorrelation according to aspects of the present disclosure.

[0016] Figure 9 Is a flowchart showing an example method of detecting the type of exercise a user is performing according to aspects of the present disclosure.

[0017] Figure 10 Is a flowchart showing an example method of training a machine learning model to detect exercise types according to aspects of the present disclosure. Detailed Description

[0018] Overview

[0019] The present disclosure provides for identifying specific user activities, such as specific types of exercise and repetitions of such exercises, using multiple IMUs. The IMUs can be located in consumer products such as smartwatches and earbuds. Each IMU can include an accelerometer and a gyroscope, with each accelerometer and gyroscope having three measurement axes, for a total of 12 raw measurement streams. In some examples, additional IMUs can also provide data, resulting in additional measurement streams. Additionally, each set of three axes can be combined into a spatial norm. Thus, in an example of two IMUs that produce 12 data streams, adding the spatial norms will provide a final total of 16 data capture tiles for each training image.

[0020] The systems and methods described herein provide the ability to count repetitions of a single exercise without having to resort to using complex window detection methods. Simple fixed-overlap windows can be used, and autocorrelation techniques, instantiated type convolutional neural networks (CNNs), or combinations of these or other techniques can be used to determine the number of repetitions. The method can be deployed using an efficient model that can be retrained by the user themselves in the final layer. For example, a user can retrain the model on their own device to identify customizable exercise types, as well as their own unique motion profile during exercise.

[0021] Only a few examples of exercise types that can be detected include bicep curls, barbell presses, push-ups, sit-ups, squats, pull-ups, burpees, jumping jacks, etc. It should be understood that any of a variety of additional exercise types can also be detected. The system can be trained by the user to detect specific exercises selected or created by the user. When training a machine learning model, non-exercises can also be included. For example, this can help identify and distinguish other types of movements of the user and thereby reduce false positive detections of exercise. Only by way of example, such non-exercises can include walking, climbing stairs, opening a door, lifting various objects, sitting on a chair, etc.

[0022] Transfer learning can be implemented to retrain the top few layers of an efficient image recognition model such as the MobileNet image recognition model. Images of the IMU raw data subgraph can be used as training examples, allowing for high accuracy with a small number of training examples.

[0023] Example system

[0024] Figure 1 is a schematic diagram of an example system in use. User 102 wears wireless computing devices 180, 190. In this example, the wireless computing devices include earbuds 180 worn on the user's head and a smartwatch 190 worn on the user's wrist. The earbuds 180 and the smartwatch 190 can communicate wirelessly with a host computing device 170 such as a mobile phone. The host computing device 170 can be carried by the user, such as in the user's hand or pocket, or can be placed anywhere near the user. In some examples, the host computing device 170 may not be needed at all.

[0025] The wireless computing devices 180, 190 worn by the user can detect specific types of exercises performed by user 102 and the repetitions of such exercises. For example, as shown, user 102 is performing jumping jacks. The smartwatch 190, which is typically fixed to the user's arm, will detect relatively large, rapid waving motions. The earbuds 190, which are typically fixed in the user's ears, will detect up-and-down bouncing. The wireless computing devices 180, 190 can communicate such detections with each other or to the host device 170. Based on the combination of detected movements, one or more of devices 170 - 190 can detect that user 102 is performing jumping jacks.

[0026] Although in the example shown, the wireless computing devices 180, 190 include earbuds and a smartwatch, it should be understood that in other examples, any of a variety of different types of wireless devices can be used. For example, the wireless devices can include headphones, head-mounted displays, smart glasses, pendants, ankle strap devices, belts, etc. Additionally, although in Figure 1Two wireless devices are shown for detecting exercise, but additional wireless devices may also be used. Additionally, while two earbuds 180 are shown, the readings detected by each earbud may be redundant, and thus only one earbud in combination with the smartwatch 190 or another device worn by the user 102 may be used to perform detection of the user's movement.

[0027] Figure 2 Also shown are wireless computing devices 180, 190 and their features and components that communicate with the host computing device 170.

[0028] As shown, each of the wearable wireless devices 180, 190 includes various components such as processors 281, 291, memories 282, 292, transceivers 285, 295, and other components typically present in wearable wireless computing devices. The wearable devices 180, 190 may have all components typically associated with wearable computing devices, such as a processor, a memory for storing data and instructions (e.g., RAM and internal hard drive), user input and output.

[0029] Each of the wireless devices 180, 190 may also be equipped with short - range wireless pairing technology such as a Bluetooth transceiver, allowing wireless coupling with each other and with other devices. For example, transceivers 285, 295 may each include an antenna, a transmitter, and a receiver that allow wireless coupling with another device. Any of various technologies such as Bluetooth, Bluetooth Low Energy (BLE), etc. may be used to establish the wireless coupling.

[0030] Each of the wireless devices 180, 190 may also be equipped with one or more sensors 286, 296 capable of detecting the movement of the user. The sensors may include, for example, IMU sensors 287, 297 such as accelerometers, gyroscopes, etc. For example, the gyroscope may detect the inertial position of the wearable devices 180, 190, while the accelerometer detects the linear movement of the wearable devices 180, 190. Such sensors may detect the direction, speed, and / or other parameters of the movement. The sensors may additionally or alternatively include any other type of sensor capable of detecting changes in the received data, where such changes may be related to user movement. For example, the sensors may include a barometer, a motion sensor, a temperature sensor, a magnetometer, a pedometer, a Global Positioning System (GPS), a camera, a microphone, etc. The one or more sensors of each device may operate independently or cooperatively.

[0031] The host computing device 170 can be, for example, a mobile phone, a tablet computer, a laptop computer, a gaming system, or any other type of mobile computing device. In some examples, the mobile computing device 170 can be coupled to a network, such as a cellular network, a wireless Internet network, and the like.

[0032] The host device 170 can also include one or more processors 271 that communicate with a memory 272 that includes instructions 273 and data 274. The host device 170 can also include elements commonly found in computing devices, such as an output 275, an input 276, a communication interface, and the like.

[0033] The input 276 and the output 275 can be used to receive information from the user and provide information to the user. The input can include, for example, one or more touch-sensitive inputs, a microphone, a camera, a sensor, and the like. Additionally, the input 276 can include an interface for receiving data from the wearable wireless devices 180, 190. The output 275 can include, for example, a speaker, a display, haptic feedback, an interface with the wearable wireless devices for providing data to such devices, and the like.

[0034] The one or more processors 271 can be any conventional processor, such as a commercially available microprocessor. Alternatively, the one or more processors can be a dedicated device, such as an application specific integrated circuit (ASIC) or other hardware-based processor. Although Figure 2 functionally the processor, memory, and other elements of the host 170 are shown as being within the same block, those skilled in the art will appreciate that the processor, computing device, or memory can actually include multiple processors, computing devices, or memories that may or may not be stored within the same physical housing. Similarly, the memory can be a hard disk drive or other storage medium located in a housing different from the housing of the host 170. Thus, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0035] The memory 272 can store information accessible by the processor 271, including instructions 273 executable by the processor 271 and data 274. The memory 272 can be a type of memory operable to store information accessible by the processor 271, including non-transitory computer-readable media, or other media that stores data readable by an electronic device, such as a hard disk drive, a memory card, a read-only memory (“ROM”), a random access memory (“RAM”), an optical disk, and other writable and read-only memories. The subject matter disclosed herein can include different combinations of the foregoing, whereby different portions of the instructions 273 and data 274 are stored on different types of media.

[0036] Data 274 can be retrieved, stored, or modified by processor 271 according to instructions 273. For example, although the present disclosure is not limited to a particular data structure, data 274 can be stored in a computer register, in a relational database as a table with multiple different fields and records, an XML document, or a flat file. Data 274 can also be formatted in a computer-readable format such as, but not limited to, binary values, ASCII, or Unicode. As a further example only, data 274 can be stored as a bitmap, which consists of pixels stored in a compressed or uncompressed or various image formats (e.g., JPEG), a vector-based format (e.g., SVG), or computer instructions for drawing graphics. Additionally, data 274 can include information sufficient to identify related information such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories (including other network locations), or information used by a function to calculate related data.

[0037] Instructions 273 can be executed to detect the type of exercise performed by a user based on raw data received from sensors 286, 296 of wireless wearable devices 180, 190. For example, processor 271 can execute a machine learning algorithm, whereby it compares an image of the received raw data with a stored image corresponding to a particular exercise and detects the exercise performed based on that comparison. Additionally, instructions 273 can be executed to detect the number of repetitions of the exercise, such as by using a window.

[0038] In other examples, the analysis of sensor data can be performed by either or both of the wearable wireless devices 180, 190. For example, each of devices 180, 190 includes processors 281, 291 and memories 282, 292, similar to those described above in connection with host device 170. These processors 281, 291 and memories 282, 292 can receive data and execute machine learning algorithms to detect the type of exercise performed.

[0039] Figure 3 Wireless wearable devices 180, 190 are shown communicating with each other and with host device 170. The wireless connection between the devices can be, for example, a short-range paired connection such as Bluetooth. Other types of wireless connections are also possible. In this example, devices 170 - 190 also communicate with server 310 and database 315 via network 150. For example, wireless wearable devices 180, 190 can be indirectly connected to network 150 via host device 170. In other examples, one or both of the wireless wearable devices 180, 190 can be directly connected to network 150 regardless of the presence of host device 170.

[0040] These networks 150 can be, for example, LAN, WAN, the Internet, etc. The connection between the device and the network can be wired or wireless.

[0041] The server computing device 310 can actually include multiple processing devices that communicate with each other. According to some examples, the server 310 can execute a machine learning model to determine a specific type of exercise being performed based on inputs from IMUs of multiple wearable devices. For example, the wearable devices 180, 190 can transmit the raw data detected from their IMUs to the server 310. The server 310 can use the received raw data as input to perform calculations, determine the type of exercise performed, and send the result back to the host 170 or one or both of the wearable devices 180, 190.

[0042] The database 315 can be accessible by the server 310 and the computing devices 170 - 190. The database 315 can include, for example, data sets from various sources corresponding to a specific type of exercise. For example, the data can include images of the raw data streams from the IMUs or other sensors in the wearable devices, where the raw data streams correspond to a specific type of exercise. Such data can be used in the machine learning models executed by the server 310 or by any of the host device 170 or the wearable devices 180, 190.

[0043] Regardless of whether the detection of the exercise type is performed at the server 310, at the host 170, at one or both of the wearable devices 180, 190, or some combination thereof, any of several different types of calculations can be performed. These different types at least include (1) peak detector window segmentation using inputs from multiple IMUs located at different positions on the user's body or (2) fixed window segmentation from multiple IMUs. In addition, any of the devices 170 - 190 or 310 can be capable of counting the number of repetitions of each exercise.

[0044] Peak detector window segmentation using inputs from multiple IMUs located at different positions on the user's body to detect "peaks" in the signal data, such as raw accelerometer and gyroscope data. Figure 4A Show an example of the raw accelerometer data from the earbuds and Figure 4B Show an example of the raw accelerometer data from the smartwatch. In this specific example, the data was obtained while doing squats, but it should be understood that the analysis can be applied to data for any of various exercises. Each figure includes three waveforms: one waveform for each of the x, y, and z directions. For example, in Figure 4AIn [the figure], wave 402 corresponds to the y direction, such as the vertical direction; wave 406 corresponds to the x direction, such as a lateral or side-to-side movement relative to the user; and wave 404 corresponds to the z direction, such as a forward / backward movement relative to the user. Similarly, in Figure 4B wave 412 corresponds to the y direction, wave 416 corresponds to the x direction, and wave 414 corresponds to the z direction.

[0045] Data from the first IMU can be timescaled to match the time from the second IMU. For example, measurements from an IMU in earbuds can be timescaled to match the measurement time from an accelerometer in a smartwatch. This can include resampling all data to the length of the smartwatch accelerometer data. A low-pass filter can be applied to the raw data. By way of example only, the low-pass filter can be a Butterworth filter or any one of various other types of filters.

[0046] The "window" uses custom peak detection and threshold processing to capture each repetition. The window in the peak detection segmentation can refer to capturing a complete repetition of an exercise. The window can start anywhere during the exercise repetition. For example, it can start in the middle of one push-up and end in the middle of the second push-up. The window can start / end at a zero crossing, or some peak or valley, or any other feature that can be extracted from the data. For a fixed window, the window can be defined by a time length. For example, the time length can be 4 seconds, where the window overlaps every 2 seconds (50% overlap). The window can capture partial repetitions or multiple repetitions depending on how fast the individual exercise is. Another option is to have an auto-scaling window that automatically scales the time length of the window based on an estimate of the nearest repetition frequency from an autocorrelation calculation. In this case, the window is still determined by time and may not accurately capture a complete repetition. For example, the repetition length may be overestimated or underestimated, but the time window will generally be close to the repetition length of the exercise.

[0047] Thus, for example, for the waveform of the raw data received from the IMU during an exercise, detect the peaks of the waveform. Analysis of the waveform can further identify other characteristics to indicate the start and end of a repetition. If two IMUs are used, each with x, y, and z axes, then 12 raw data streams are included in each window. Adding 4 norms for each set of x, y, and z brings this to a total of 16 data streams. The norm can be calculated as, for example, square_root(x^2 + y^2 + z^2). Other mathematical operations may also be possible, which can provide additional useful information to the model. Thus, 16 sub-curves are provided for each image, for example, in a 4×4 grid.

[0048] Figures 5A - 5EShows example images created using the peak detection window segmentation technique described above with two IMUs for various types of exercises. For example, Figure 5A shows an example image for bicep curls, Figure 5B shows an example image for weight presses, Figure 5C shows an example image for push - ups, Figure 5D shows an example image for sit - ups, and Figure 5E shows an example image for squats. These images can be used, for example, to train machine learning models to recognize exercises from movements detected by IMUs in wearable devices.

[0049] Figure 6 Shows example images of non - exercises. Examples of non - exercises can include, but are not limited to, walking, climbing stairs, working at a table, opening a door, picking up an object, etc. Non - exercise images can be included in the training data to help distinguish between specific exercise types and non - exercises. Identifying such differences can help prevent false - positive detections of exercises. Although one example image is shown in Figure 6 , it should be understood that any number of example images corresponding to various non - exercises can be included in the training data.

[0050] According to another type of calculation, fixed window segmentation from multiple IMUs modifies the above - described data - processing flow by using a fixed window of a predetermined width. As an example only, the width of the fixed window can be 4 seconds with 50% overlap. Since it is fixed, it is not synchronized with exercise repetitions and can even have multiple or partial repetitions within a single windowed training or test example image. Since the window start is random relative to the exercise frequency, more training examples can be generated from the same data set by changing the initial window start time. By stepping the window start time by 0.5 s, the total number of training examples for each exercise is multiplied by 8.

[0051] According to another example, the fixed window can be automatically scaled based on the estimated repetition frequency. This option does not require peak detection or threshold processing.

[0052] Figures 7A - 7B Shows example images of exercises using fixed window segmentation. In these examples, Figure 7A represents a first example of bicep curls, while Figure 7B represents a second example of bicep curls. For example, Figure 7A 's first example is generated using the first start time of the window, while Figure 7BA second example of [[ID=]] is generated using a second start time that is different from the first start time. For example, if a 4s window is used, the second start time can be any time between 0.5s and 3.5s later than the first start time. However, it should be understood that the width of the window can vary, for example by using 2s, 6s, 10s, etc. Additionally, the start time increment of the window can also vary from 0.5s to any other value. The machine learning model thus learns the patterns within the image without relying on specific start or stop points.

[0053] In addition to being able to classify the type of exercise, the number of repetitions of a given exercise type can be counted in real time. Using peak detector windowing, each test image becomes a single repetition when classified. Using fixed windowing, different methods such as autocorrelation or model instantiation can be used to perform the repetition count.

[0054] When using autocorrelation, a sliding autocorrelation window of fixed duration is used during the exercise. The repetition frequency can thus be extracted. Figure 8 An example showing autocorrelation is presented. Using a 4 - second sliding window during a 10 - second push - up, an indicator of the repetition frequency can be extracted. For example, the indicator can be the position of the highest peak, excluding the peak at time t = 0. Once the repetition frequency is known, the number of repetitions in a given exercise sequence can be calculated as the exercise duration in seconds multiplied by the repetition frequency. As long as the device detects a specific exercise type, a timer can run to track the total duration. Using this method, the user can receive an update on the repetition counter every few seconds.

[0055] When using instantiation, the model can learn to count repetitions on its own by using instantiation. In this method, the model is trained not only with labeled examples, but also with "instances" of segments or repetitions. For example, in a training example image containing 10 repetitions, each repetition can be labeled within the training example. In this way, the model learns to estimate how many repetitions there are in a given image.

[0056] Example method

[0057] In addition to the operations described above and shown in the figures, various operations will now be described. It should be understood that the following operations do not have to be performed in the exact order described below. Instead, the individual steps can be disposed of in a different order or simultaneously, and steps can also be added or omitted.

[0058] Figure 9 An example method 900 for detecting the type of exercise a user is performing is shown. The method can be executed by one or more processors in a first or second wearable device or in a host device communicating with the wearable device.

[0059] In block 910, first sensor data is received from one or more first sensors of a first wearable device. The first wearable device can be, for example, earbuds. The one or more first sensors can include an IMU, such as including a gyroscope and an accelerometer. Each of the gyroscope and the accelerometer can generate a data stream of measurements in the x, y, and z directions.

[0060] In block 920, second sensor data is received from one or more second sensors of a second wearable device. The second wearable device can be, for example, a smartwatch. Similar to the first sensors, the one or more second sensors can also include an IMU, which generates a data stream of measurements in multiple spatial directions.

[0061] In block 930, an image is generated based on the first sensor data and the second sensor data. The image can include a plurality of tiles or sub-curves, where each tile or sub-curve depicts a separate data stream from the first and second sensors. For example, the first tile depicts the data stream of the accelerometer from the first sensor in the x direction, the second tile depicts the data stream of the accelerometer from the first sensor in the y direction, and so on. Using two IMUs, each with two sensors measuring in three directions, twelve tiles can be generated. Additionally, each set of three (x, y, z) axes can be combined into a spatial norm to obtain a final total of 16 data capture tiles for each image.

[0062] In block 940, the type of exercise performed by the user during the reception of the first and second sensor data is determined based on the type of exercise. For example, a machine learning model is applied to the generated image. The machine learning model can be trained using the image data, as described in further detail below in conjunction with Figure 10 The machine learning model can be, for example, an image-based deep neural network, such as a convolutional neural network (CNN).

[0063] In block 950, the number of repetitions of the exercise can be determined. For example, the machine learning model can be trained to recognize repetitions if it is trained using images in which the repetitions are labeled. According to other examples, the repetition frequency can be determined by using autocorrelation and then multiplying the repetition frequency by the duration of the exercise type to detect repetitions.

[0064] Figure 10 An example method 1000 of training a machine learning model to detect exercise types is shown. The method can be performed, for example, by one or more processors in a wearable device, a coupled host device, and / or on a remote server connected to the wearable device or the host via a network.

[0065] In blocks 1010 - 1020, the data can be as described above in conjunction with Figure 9Received as described for the frames 910 - 920. However, when the user performs several types of exercises and non - exercises, data can be received over an extended period of time.

[0066] In block 1030, the received data can be used to generate one or more first training images. For example, similar to the generation of the images described above in connection with Figure 9 the block 930, the first training image can include a plurality of image tiles, each image tile depicting a different data stream of the data collected while the user is performing a first type of exercise. Similarly, in block 1040, the received data can be used to generate a second training image depicting the data stream collected while the user is performing a second type of exercise different from the first type. This can be repeated for any number of different types of exercises.

[0067] In block 1050, techniques similar to those described in block 1030 are used to generate a third training image. However, the third training image can correspond to non - exercise. For example, the data represented in the tiles of the third image can be collected while the user is performing an activity other than a specific exercise. Such activities can include any of a variety of non - exercises, such as brushing teeth, cooking, sitting, etc.

[0068] In block 1060, the first, second, and third images are input as training data into a machine - learning model. In this regard, the machine - learning model learns to identify various types of exercises and to distinguish such exercises from non - exercises. This distinction can help reduce false - positive detections.

[0069] The advantage of the foregoing systems and methods is that they provide highly accurate detection of exercise by using an IMU in a product that is accessible to and commonly used by the user.

[0070] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the above - described features can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be understood by way of illustration rather than by way of limitation of the subject matter defined by the claims. Additionally, the provision of the examples described herein and clauses phrased as "such as", "including", etc. should not be construed as limiting the subject matter of the claims to specific examples; rather, these examples are intended to illustrate only one of many possible embodiments. Furthermore, the same reference numerals in different figures can identify the same or similar elements.

Claims

1. A method for detecting exercise, comprising: Receiving, by one or more processors, first sensor data from one or more first sensors of a first wearable device; Receiving, by the one or more processors, second sensor data from one or more second sensors of a second wearable device; Generating, by the one or more processors, an image based on the first sensor data and the second sensor data, the image including a plurality of sub-curves, wherein each sub-curve depicts a corresponding data stream received from the one or more first sensors and the one or more second sensors; Using the one or more processors to determine, based on the image, the type of exercise performed by a user during the reception of the first sensor data and the second sensor data, wherein determining the type of exercise performed includes: executing a machine learning model; and Training the machine learning model, the training including: Generating, based on sensor data collected from the one or more first sensors and the one or more second sensors of the first wearable device and the second wearable device when the user performs a first type of exercise, one or more first training images, wherein the one or more first training images include a plurality of image tiles, each image tile depicting a different data stream of the sensor data; and Inputting the one or more first training images as training data into the machine learning model, wherein Generating the one or more first training images includes fixed window segmentation and automatically scaling the time length of the window based on an estimated nearest repetition frequency of the first type of exercise.

2. The method according to claim 1, wherein The machine learning model is an image-based machine learning model.

3. The method according to claim 1, wherein the training further comprises: Generating, based on data collected from the first wearable device and the second wearable device when the user performs a second type of exercise, one or more second training images; And Inputting the one or more second training images as training data into the machine learning model.

4. The method according to claim 1, wherein the training further comprises: Generating, based on data collected from the first wearable device and the second wearable device when the user performs one or more activities not classified as exercise, one or more third training images; Determining the number of repetitions includes: Using a sliding autocorrelation window to determine the repetition frequency; and 5. The method according to claim 1, wherein Calculating the number of repetitions based on the repetition frequency and the duration of the exercise type.

6. The method according to claim 1, wherein Determining the number of repetitions includes:

7. The method according to claim 1, further comprising: Counting the number of repetitions in the training images; 8. The method according to claim 7, wherein Marking the repetitions in the training images; Inputting the training images as training data into a machine learning model.

10. A system for detecting exercise, comprising:

9. The method according to claim 7, wherein ​ ​ ​ ​ ​ One or more first sensors in a first wearable device; One or more second sensors in a second wearable device; And One or more processors in communication with the one or more first sensors and the one or more second sensors, the one or more processors being configured to: Receive first sensor data from the one or more first sensors of the first wearable device; Receive second sensor data from the one or more second sensors of the second wearable device; Generate an image based on the first sensor data and the second sensor data, the image including a plurality of sub-curves, wherein each sub-curve depicts a corresponding data stream received from the one or more first sensors and the one or more second sensors; and Determine an exercise type performed by a user during receipt of the first sensor data and the second sensor data based on the image, wherein determining the exercise type performed includes: executing a machine learning model, wherein The one or more processors are further configured to train the machine learning model, the training including: Generating one or more first training images based on sensor data collected from the one or more first sensors and the one or more second sensors of the first wearable device and the second wearable device when the user performs a first type of exercise, wherein the one or more first training images include a plurality of image tiles, each image tile depicting a different data stream of the sensor data; and Inputting the one or more first training images as training data into the machine learning model, wherein Generating the one or more first training images includes fixed window segmentation and automatically scaling a time length of the window based on a nearest repetition frequency estimate of the first type of exercise.

11. The system according to claim 10, wherein, The first wearable device is an earbud, and the second wearable device is a smartwatch.

12. The system according to claim 10, wherein, The one or more processors reside in at least one of the first wearable device or the second wearable device.

13. The system according to claim 10, wherein, At least one of the one or more processors resides in a host device coupled to the first wearable device and the second wearable device.

14. The system according to claim 10, wherein Determining the exercise type performed includes: executing an image-based machine learning model.

15. The system according to claim 14, wherein The one or more processors are further configured to train the machine learning model, the training including: Generating one or more second training images based on data collected from the first wearable device and the second wearable device when the user performs a second type of exercise; Generating one or more third training images based on data collected from the first wearable device and the second wearable device when the user performs one or more activities not classified as exercise; and Inputting the one or more second training images and the one or more third training images as training data into the machine learning model.

16. The system according to claim 15, wherein, Generating the one or more first training images includes peak detection window segmentation.

17. The system according to claim 15, wherein The fixed window segmentation includes: changing the start time of a window having a predetermined time length, wherein each change in the start time in the window generates a separate one of the one or more first training images.

18. The system according to claim 10, wherein The one or more processors are further configured to determine the number of repetitions of the determined exercise type by: Determining a repetition frequency using a sliding autocorrelation window; and Calculating the number of repetitions based on the repetition frequency and the duration of the exercise type.

19. The system according to claim 10, wherein The one or more processors are further configured to determine the number of repetitions of the determined exercise type by: Counting the number of repetitions in the training images; Marking the repetitions in the training images; Inputting the training images as training data into a machine learning model.

20. A non-transitory computer-readable medium storing instructions that can be executed by one or more processors to perform a method for detecting an exercise type, the method including: Receiving first sensor data from one or more first sensors of a first wearable device; Receiving second sensor data from one or more second sensors of a second wearable device; Generating an image based on the first sensor data and the second sensor data, the image including a plurality of sub-curves, wherein each sub-curve depicts a data stream received from the one or more first sensors and the one or more second sensors respectively; and Determining an exercise type performed by a user during the reception of the first sensor data and the second sensor data based on the image, wherein determining the performed exercise type includes: executing a machine learning model; and Training the machine learning model, the training including: Generating one or more first training images based on sensor data collected from the one or more first sensors and the one or more second sensors of the first wearable device and the second wearable device when the user performs a first type of exercise, wherein the one or more first training images include a plurality of image tiles, each image tile depicting a different data stream of the sensor data; and Inputting the one or more first training images as training data into the machine learning model, wherein Generating the one or more first training images includes fixed window segmentation and automatically scaling the time length of the window based on an estimation of the most recent repetition frequency of the first type of exercise.

Images