State-aware cascade machine learning system and method

Abstract

A cascaded machine learning inference system and method is disclosed. The cascaded system and method can be designed to be employed in resource-constrained environments. The cascaded system and method can be suitable for applications that operate with limited power (e.g., wearable smart watches). The cascaded system and method can employ two or more subsystems that are operable to classify input signals provided by any number or type of sensors suitable for a given application. For example, the sensors used can include a gyroscope, an accelerometer, a magnetometer, or a barometric altimeter. The system and method can also further separate functionality across additional or new subsystems. By separating operations and functionality across additional subsystems, overall power consumption can be further reduced.

Description

Technical Field

[0001] The present disclosure relates to a machine learning system comprising two or more subsystems operable to classify input signals that may be provided by an inertial sensor. Background Art

[0002] The task of classifying input data using machine learning algorithms is understood to be applicable across diverse applications. Depending on the given application, a single machine learning algorithm can be very large, containing tens of millions of variables. Such large networks typically require significant computational processing power and vast data storage resources. Consequently, such machine learning models can have a large energy footprint. Typically, such large machine learning models cannot be deployed in resource-constrained settings such as wearables, smartphones, or IoT devices. Summary of the Invention

[0003] A cascaded machine learning system and method are disclosed. The system and method may include a first subsystem and a second subsystem, the first subsystem including multiple operating states and at least a first machine learning algorithm, and the second subsystem including a second machine learning algorithm. Upon receiving a first set of sensory data signals, the second subsystem may be activated. A second algorithm (i.e., the second machine learning algorithm) may classify the first set of sensory data signals. A first operating state may be selected from the multiple operating states for the first subsystem to adopt. A set of activity parameters to be used by the first subsystem may also be selected based on the first operating state. The second subsystem may be deactivated (e.g., to save power), and the first algorithm (i.e., the first machine learning algorithm) may classify a second set of sensory data signals. The classification of the second set of sensory data signals may then be compared to the set of activity parameters. Furthermore, if the output classification performed using the first machine learning algorithm is invalid, the second subsystem may be reactivated.

[0004] The cascaded machine learning system and method may also compare a confidence level of the classification of the second set of sensory data signals using the first machine learning algorithm to a threshold. If the confidence level is less than the threshold, the second subsystem may be reactivated. It is also contemplated that the second set of sensory data signals may be classified using a second machine learning algorithm. If the classification of the second set of sensory data signals using at least the first machine learning algorithm is equal to the classification of the second set of sensory data signals using the second machine learning algorithm, the threshold may be increased. Alternatively, if the classification of the second set of sensory data signals using the second machine learning algorithm is equal to one of the set of activity parameters, the threshold may be decreased. If the threshold is adjusted, the second set of sensory data signals may be classified using the second machine learning algorithm. It is also contemplated that the threshold may be temporarily or permanently stored in memory.

[0005] A second machine learning algorithm may also be used to classify the second set of sensory data signals after the predetermined period has expired. It is contemplated that the first subsystem may be employed on a first processor (e.g., a low-power microprocessor), while the second subsystem may be employed on a second processor (e.g., an application processor). The second machine learning algorithm may also be trained using the full training data set and multiple labels. Furthermore, the first subsystem may be trained using the full training data set, but with one or more designated labels grouped into a single category. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 An exemplary embodiment of a cascade system having a first subsystem and a second subsystem is shown;

[0007] Figure 2 illustrates an exemplary CNN that may be employed by the first subsystem or the second subsystem;

[0008] Figure 3 shows an exemplary state diagram implemented by a cascade system;

[0009] Figure 4 Another exemplary state diagram implemented by a cascade system is shown. DETAILED DESCRIPTION

[0010] Embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The drawings are not necessarily to scale; some features may be exaggerated or minimized to show details of particular components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to employ the embodiments in various ways. As will be understood by those of ordinary skill in the art, the various features shown and described with reference to any one of the figures may be combined with features shown in one or more of the other figures to produce embodiments that are not explicitly shown or described. The combinations of features shown provide representative embodiments for typical applications. However, various combinations and modifications of features consistent with the teachings of the present disclosure may be desirable for particular applications or implementations.

[0011] For machine learning algorithms, the use of neural networks (e.g., CNNs or DNNs) has increased and is now employed across a wide variety of applications, including classifying input data to provide event detection, activity classification, image classification, and object classification (e.g., object recognition or facial recognition). However, conventional neural networks used to classify input data from inertial sensors can be extremely large, requiring significant computational and data storage resources. Consequently, such large networks consume significant energy and typically have a large power footprint. Furthermore, conventional neural networks are typically not deployable in resource-constrained settings (e.g., within wearable devices such as smartwatches or IoT devices).

[0012] It is therefore conceivable that cascaded machine learning inference can be designed to be employed in resource-constrained settings or environments. The cascaded system and method may be applicable to applications that operate with limited power (e.g., wearable smart watches, smart phones, or IoT devices). The cascaded system and method may employ two or more subsystems that are operable to classify input signals provided by one or more sensors. Sensors commonly used in consumer devices are MEMS-based gyroscopes, accelerometers, magnetometers, pressure sensors, or barometric altimeters. The system and method may also further separate functionality across additional or new subsystems. By separating operations and functionality across additional subsystems, overall power consumption can be further reduced.

[0013] It is contemplated that a cascaded system could employ one subsystem (i.e., a first subsystem executing a first machine learning algorithm) that receives data and classifies it according to a limited set of activities, images, categories, objects, or groups of objects. Then, in some cases and depending on the application, the classified data could be sent to a more complex subsystem (i.e., a second subsystem executing a second machine learning algorithm) that performs the full classification. By employing two subsystems with different levels of complexity in terms of the classifications performed and area consumed, a multi-stage network can require less power than a conventional machine learning network.

[0014] Figure 1An exemplary embodiment of a cascade system 100 is shown, including a first subsystem 102 and a second subsystem 104. As shown, the first and second subsystems 102, 104 can be designed to include at least one processor 106, 108, operably connected to respective memory units 114, 116. One or more sensors 140 can also be electrically connected to the first and second subsystems 102, 104. The one or more sensors 140 can include time series sensors such as gyroscopes, accelerometers, magnetometers, pressure sensors, barometric altimeters, photoplethysmograms (PPGs), heart rate sensors, or piezoelectric sensors. However, these are merely exemplary sensors, and it should be understood that the cascade system 100 can be designed to operate with any other type of sensor, depending on the application. The one or more sensors 140 can provide signal data 142 to the first and second subsystems 102, 104.

[0015] Each processor 106, 108 may be one or more integrated circuits that implement the functionality of a corresponding processing unit (PU) 110, 112. PUs 110, 112 may be commercially available processing units that implement an instruction set such as one of the x86, ARM, Power, or MIPS instruction set families. However, it is contemplated that commercially available processing units may be exemplary methods for implementing the first subsystem 102 and the second subsystem 104. For example, it is also contemplated that PUs 110, 112 may be specialized accelerator units that include specialized functionality or instructions optimized for implementing machine learning algorithms or neural networks (e.g., CNNs, DNNs, or decision trees), such as analog computational units or units that perform computational operations in memory. It is further contemplated that PUs 110, 112 may be designed using specialized integrated circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0016] During operation, each PU 110, 112 can execute stored program instructions retrieved from an associated memory unit 114, 116. The stored program instructions may include software that controls the operation of each PU 110, 112 to perform the operations described herein. In some examples, each processor 106, 108 can be a system on a chip (SoC) that integrates the functionality of each PU 110, 112; each memory unit 114, 116; associated network interfaces 118, 120; and associated input / output interfaces 122, 124. It is contemplated that the first subsystem 102 and the second subsystem 104 can implement an operating system for managing various aspects of operation.

[0017] Each memory unit 114, 116 may include volatile memory and non-volatile memory for storing instructions, data, and parameters. Non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the first subsystem 102 and the second subsystem 104 are deactivated or lose power. Volatile memory may include static and dynamic random access memory (RAM) that stores program instructions and data. For example, each memory unit 114, 116 may store a machine learning model 126, 128 or algorithm. Memory units 114, 116 may also store parameters or data (e.g., coefficients, weights), which can then be loaded by the PU 110, 112 for use by the machine learning model 126, 128. Specifically, the first subsystem 102 may store parameters 130 that can be used by the machine learning model 126 to classify a given activity, image, category, object, or group of objects.

[0018] It is contemplated that the first subsystem 102 and the second subsystem 104 may also pre-process the data 142 received from the sensor 140 using pre-processing circuitry or algorithms 143. Pre-processing of the data 142 may include implementing known smoothing, filtering, and averaging techniques to remove potential noise. After pre-processing, the data 142 may then be stored as source data 134, 136 within the memory units 114, 116. Alternatively, it is also contemplated that the pre-processing of the data 142 may occur externally to the first subsystem 102 and the second subsystem 104. In this way, the raw data 141 may be provided directly to the first subsystem 102 and the second subsystem 104 for use. Even if the pre-processing occurs externally, the data 142 may still be stored as source data 134, 136 within the memory units 114, 116. Alternatively, the pre-processed data 142 may be provided directly for use by the PUs 110, 112.

[0019] Likewise, the first subsystem 102 and the second subsystem 104 may also include associated network interface devices 118, 120. It is contemplated that the network interface devices 118, 120 may be configured to provide a communication link 138 between the first subsystem 102 and the second subsystem 104. Alternatively, the network interface devices 118, 120 may be configured to communicate with additional external systems and devices (e.g., sensors). The network interface devices 118, 120 may utilize I 2The network interface devices 118 and 120 may be configured to communicate with each other using SPI, C, or similar methods that allow communication between microcontrollers, EEPROMs, A / D and D / A converters, I / O interfaces, and other similar peripherals in embedded systems. Additionally, the network interface devices 118 and 120 may be configured to include wired and / or wireless Ethernet interfaces as defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards. The network interface devices 118 and 120 may include a cellular communication interface for communicating with a cellular network (e.g., 3G, 4G, 5G).

[0020] The first subsystem 102 and the second subsystem 104 may also be operable to communicate with an external network (e.g., the World Wide Web or the Internet). The external network may be designed to establish a standard communication protocol between computing devices. The external network may allow for easy exchange of information and data between the first subsystem 102, the second subsystem 104, and the external network. For example, one or more servers may connect to and communicate with the first subsystem 102 and the second subsystem 104 using the external network.

[0021] The first and second subsystems 102 and 104 may implement associated machine learning algorithms 126 and 128 configured to analyze data 134 and 136, which may include unprocessed data 141 or pre-processed data 142 received from a sensor 140 (e.g., an inertial sensor). Alternatively, the first and second subsystems 102 and 104 may implement associated machine learning algorithms 126 and 128 to directly analyze the unprocessed data 141 or pre-processed data 142 received from the sensor 140. The data 134 and 136 (or the unprocessed data 141 and pre-processed data 142) may be videos, video clips, images, and raw or partially processed sensor data (e.g., data from a digital camera or LiDAR sensor). In some examples, the machine learning algorithms 126 and 128 may be machine learning algorithms (e.g., CNNs or DNNs) designed to perform a predetermined function.

[0022] However, it is further contemplated that the first and second subsystems 102, 104 can be designed to implement non-machine learning algorithms, depending on the application. For example, the first and second subsystems 102, 104 can implement an algorithm that evaluates the gravity vector, rather than a machine learning algorithm. The algorithm can then be operable to infer a decision based on the noise of the vector (e.g., the mobile device is in motion) and the orientation of the vector for the mobile device. The algorithm can use the orientation of the vector to determine whether the mobile device is positioned face-up or face-down on a surface.

[0023] Although the cascaded system 100 is illustrated as executing the first subsystem 102 and the second subsystem 104 on separate devices, it is contemplated that both subsystems can be implemented on a single device. For example, the PU 110 can be operable to load and execute both the machine learning model 126 (i.e., the machine learning algorithm employed by the first subsystem 102) and the machine learning model 128 (i.e., the machine learning algorithm employed by the second subsystem 104). Furthermore, the PU 110 can be operable to load and execute additional machine learning algorithms.

[0024] It is contemplated that the first subsystem 102 and the second subsystem 104 may also be executed concurrently on the PU 110, regardless of whether the PU 110 is designed using a commercial processor (e.g., x86) or a specialized processor (e.g., an ASIC). For example, if the PU 110 is designed using a specialized integrated circuit (e.g., an ASIC), a portion of the logic may be designed to implement the machine learning model 126 employed by the first subsystem 102, and a separate portion of the logic may be designed to implement the machine learning model 128 employed by the second subsystem 104.

[0025] Alternatively, if the PU 110 is designed using a commercially available processor, both the machine learning models 126 and 128 can be loaded. In either scenario, when the first subsystem 102 is active, the portion of the PU 110 associated with the second subsystem 104 can be deactivated or placed in a low-power sleep mode to save power. When the second subsystem 104 is needed, the PU 110 associated with the machine learning model 128 can be activated, and the machine learning model 126 employed by the first subsystem 102 can be deactivated or placed in a low-power sleep mode. By switching between operating the first subsystem 102 and the second subsystem 104 or deactivating portions of the PU 110, the overall power consumption of the cascaded system 100 can be reduced.

[0026] Again, the first subsystem 102 and the second subsystem 104 can be designed to operate using machine learning models 126, 128. It is contemplated that the machine learning models 126, 128 can be implemented using convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), long short-term memory (LSTMs), or decision trees. However, it is contemplated that the machine learning models can include other known algorithms for classifying data received from sensors.

[0027] For example, Figure 2An exemplary CNN 200 is shown that can be used to implement the first subsystem 102 or the second subsystem 104 to provide partial, specific, or complete classification of the provided data or input signal. As shown, the CNN 200 can include one or more convolutional layers 220-240; one or more pooling layers 250-270; one or more fully connected layers 260; and a softmax layer 270.

[0028] CNN 200 may receive data 210 (e.g., Figure 1 ), which may include image data, speech data, text data, or measurements received from a sensor or transducer. Again, the data 210 may also be lightly processed before being provided to the CNN 200. The convolutional layers 220-240 may be designed to extract features from the data 210. For example, the convolutional layers 220-240 may employ filtering operations (e.g., kernels) before passing the results to the next layer of the CNN 200. The filtering operations may include image identification, image edge detection, and image sharpening, which are applied when the received data 210 is an image. Alternatively, the filtering operations may include time series feature extraction, such as mean, standard deviation, peak detection, or slope detection, which are applied when the received data 210 is an image.

[0029] CNN 200 may also include one or more pooling layers 250-270 that receive convolution data from corresponding convolutional layers 220-240. Pooling layers 250-270 may include one or more pooling layer units that apply a pooling function to one or more convolutional layer outputs computed at different bands using the pooling function. For example, pooling layer 250 may apply a pooling function to the kernel output received from convolutional layer 220. The pooling function implemented by pooling layers 250-270 may be an average or maximum function or any other function that aggregates multiple values ​​into a single value.

[0030] Next, one or more fully connected layers 280 may attempt to learn nonlinear combinations of high-level features in the output data received from the convolutional layers 220-240 and pooling layers 250-270. Finally, the CNN 200 may include a softmax (soft maximum transfer function) layer 290 that combines the outputs of the fully connected layers 280 using a softmax function. It is also contemplated that the CNN 200 may further employ batch normalization layers, maximum pooling layers, and dropout layers.

[0031] Figure 3An exemplary state diagram 300 that can be employed by the cascade system 100 is shown. In state 302, the second subsystem 304 can provide classification of the source data 134, 136. Based on the classification performed by the second subsystem 304, the state diagram 300 can proceed to one or more states 304, 308, 312, or 316, wherein the first subsystem 302 can execute one or more various algorithms, such as machine learning models. It is contemplated that each algorithm executed by the first subsystem 102 can be designed to provide a classification of the result or an unclassifiable result.

[0032] For example, the cascade system 100 can be implemented for use within or in association with a wearable device, IoT device, edge device, or mobile device (e.g., a smartphone) to provide a given classification (i.e., categorizing a user's activity). Based on source data 136 (i.e., unprocessed data 141 or preprocessed data 142 provided by sensor 140), the second subsystem 104 can determine that the user is in motion (i.e., user activity). More specifically, the second subsystem 104 can determine that the source data 136 indicates that the user's activity is "running." The second subsystem 104 can then trigger the activation of the first subsystem 102, which may have been deactivated or in sleep mode to conserve power. Once activated, the state diagram 300 can proceed to state 304, where the first subsystem 302 loads parameters 130 associated with the user's activity.

[0033] The loaded parameters 130 may include a user activity associated with "running" (e.g., "sprinting" or "jogging"). Alternatively, the parameters 130 may include a user activity transitioning from "running" to "walking," for example. The loaded parameters 130 may also include a default "other" classification, such as where the first subsystem 102 is unable to classify the user activity (e.g., the user has stopped running and is now standing still).

[0034] The state machine 300 then proceeds to state 306, where the parameters 130 are used by the machine learning model 126 of the first subsystem 102 to determine whether the future data 134 (i.e., the input 142 received from the sensor 140) still correlates with the user activity previously determined by the second subsystem 104. If so, the state machine remains in state 306 to evaluate and classify the new source data 134 (i.e., the unprocessed data 141 or the processed data 142) acquired from the sensor 140. In the current example, the first subsystem 102 may still determine that the next data 134 (i.e., the unprocessed data 141 or the processed data 142) still indicates that the user activity is "running." Because the user activity is still classified as "running," the first subsystem 102 will maintain the current set of parameters, and the state diagram 300 will remain in state 306. Again, classification of new source data 134 (ie, unprocessed data 141 or processed data 142 ) by first subsystem 102 may consume less power than classification performed by the more complex machine learning model 128 of second subsystem 104 .

[0035] If state 306 determines that the user activity has changed (e.g., the user activity has changed from "running" to "walking") or that the first subsystem 102 cannot classify the user activity (i.e., the user activity is "other"), the second subsystem 104 can be reactivated (e.g., using a trigger signal) and the state diagram 300 returns to state 302. Once the state machine 300 returns to state 302, the second subsystem 104 will again process and provide a complete classification of the data 136 (i.e., the input 142 received from the sensor 140). Based on the complete classification, the state diagram 300 can then proceed to one of states 304, 308, 312, or 316.

[0036] For example, like the first subsystem 102, the second subsystem 104 operating in state 302 may also classify the user activity as "walking." The state diagram 300 may then proceed to state 308 (i.e., Activity 2), where the first subsystem 102 loads parameters 130 that will enable the classification associated with "walking." The parameters 130 may include "walking," "standing," "running," or "other." Again, the parameters loaded for "Activity 2" may include a transitional user activity (i.e., "walking" or "running") or a classification indicating that the first subsystem 102 is unable to detect the user activity (i.e., "other").

[0037] It is contemplated that if the first subsystem 102 does classify the user activity as "other," the second subsystem 104 can perform a full classification to determine the current state. The second subsystem 104 can then proceed to the correct state (e.g., states 304, 308, 312, 316) where appropriate parameters are loaded for use by the first subsystem 102.

[0038] It is also contemplated that the confidence level of the classification performed by the first subsystem 102 may need to be assessed to determine whether the loaded parameter set (e.g., "walking," "standing," "running," or "other") is still applicable. The cascade system 100 can check the confidence level by defining a threshold. For example, the PU 110 can be programmed (or instructions can be loaded) to define the threshold. If the confidence level falls below the threshold, the cascade system 100 can activate the second subsystem 104 so that a full classification can be performed. The threshold can be a fixed value or a value optimized over time by the cascade system 100.

[0039] For example, the threshold may be initially set to a value of 50%. If the PU 110 determines that the confidence level has fallen below the 50% threshold, the second subsystem 104 may be reactivated. However, if the second subsystem 104 reclassifies the user activity as previously classified, the threshold may be increased, for example, to 55% or as low as 45%. In other words, the second subsystem 104 may have classified the user activity as "running." The first subsystem 102 may have loaded parameters associated with "running," but classified the data 134 received from the sensor 140 as associated with the user activity of "walking." Upon performing the full classification, the second subsystem 104 may determine that the first subsystem 102 was incorrect and that the user activity remains "running." By adjusting the threshold, the first subsystem 102 may not incorrectly classify the user activity as "walking" again. By implementing a confidence level threshold, the power consumption of the cascade system 100 can be reduced because the system becomes personalized to the user.

[0040] It is also contemplated that the threshold value may be saved (e.g., in memory 114 or 116) and reused each time the cascade system 100, subsystem 102, or subsystem 104 is activated. Alternatively, the threshold value may be temporarily stored and valid only during the current activation period. For example, a temporarily stored threshold value may be implemented when a user attempts to incorrectly apply the wearable device and excessive movement is detected. In this given situation, the cascade system 100 may be operable to modify the threshold value once and return to the previous (or original) threshold value during the next iteration. The cascade system 100 may also detect and determine that a given threshold value has been modified because the user is moving in a given manner (e.g., walking). The cascade system 100 may thus be operable to "personalize" the threshold value unique to a given user and permanently store the personalized threshold value in memory 114 or 116. Finally, the cascade system 100 may be operable to monitor and change the threshold value within a given number of uses (e.g., after wearing the smartwatch 10 times), and store the threshold value after the given number of uses has been exceeded. It is also contemplated that the algorithm may not involve a learning cycle. For example, an algorithm may be used to detect the occurrence of a "lights on" or "lights off" event within a light sensor. It is also contemplated that the threshold value may be set manually.

[0041] It is contemplated that the threshold value could also be reset back to its original predefined value (e.g., 50%). If a user running on a flat road begins walking up a hill, then descends, and then continues down a flat road again, the threshold value may have been modified and reset to its original value. In this scenario, the user's activity may change between classifications of running (e.g., while on a flat surface) and walking (e.g., while walking up a hill). When the user is running on a flat surface, the first subsystem 102 may provide a very clear (high confidence) classification. However, when the user begins running uphill, the user's characteristics may change, and the first subsystem 102 may have difficulty classifying the user as still running.

[0042] Changes in user characteristics may cause the first subsystem 102 to lose confidence and fall back (i.e., activate) the second subsystem 104 for a more accurate full classification. This may occur when a user is walking up a hill rather than "running" as quickly as they would on a flat surface. Consequently, the first subsystem 102 may mistakenly believe the user is no longer running but walking. However, by performing a full classification, the second subsystem 104 can still classify the user's activity as running. After several instances of this "backwards and forwards" between the two subsystems, the threshold of the first subsystem 102 can be reduced by 5% (e.g., to a 45% confidence threshold for running to ensure that the user's activity is classified as running). If or when the runner begins running on a flat surface, the confidence of the first subsystem 102 can increase again, and the threshold can be increased and / or reset. Furthermore, if the threshold falls below a certain percentage (e.g., 30%), the second subsystem 104 can be activated to perform a full classification to verify the classification of the first subsystem 102.

[0043] It is also contemplated that the cascaded system 100 can also verify the threshold by running the second subsystem 104 to perform a full classification after a predetermined duration. For example, after the first subsystem 102 has performed 10 or 100 classifications of user activity, the second subsystem 104 can be activated to perform a full classification. Alternatively, the duration can be time-based (e.g., 100 μs), and the second subsystem 104 can be activated after the time-based duration has expired. The second subsystem 104 can be reactivated after the predetermined duration to ensure that the first subsystem 102 does not loop and provide the same incorrect classification.

[0044] It is also contemplated that the first subsystem 102 may be activated by a hardware switch or a software switch that detects a related substate, which may be one of the states included in the state diagram 300 (e.g., states 304, 308, 312, or 316). The hardware switch or the software switch may be used to detect significant changes within the device. The cascade system 100 may also include additional substates, including "NO_MOTION," "NO_SIGNIFICANT_MOTION," and "NO_LOCATION_CHANGE." The cascade system 100 may use these additional substates to generate an intrinsic relationship, such as: NO_MOTION => NO_SIGNIFICANT_MOTION => NO_LOCATION_CHANGE. This intrinsic relationship may include multiple cascade systems—for example, cascade 1 may be a hardware interrupt, while cascade 2 may be used to check for "noise" on a sensor signal. It is also contemplated that hardware and software switches may be employed to activate state-specific sensors, such as accelerometers to detect motion, pressure sensors to detect changes in altitude / altitude, or gyroscopes to detect changes in rotation.

[0045] Figure 4 Another state diagram 400 is shown according to an alternative embodiment of the cascade system 100. It is contemplated that the alternative embodiment may share similar functionality and operating states with those described with reference to the state diagram 300. For example, state 402 may operate in a similar manner to state 302. Likewise, states 404, 408, 412, and 416 may operate as states 304, 308, 312, and 316 described above. However, the state diagram 400 may change based on decisions made by the first subsystem 102 regarding states 406, 410, and 414. More specifically, the first subsystem 102 may switch to another classification activity without reactivating or performing classification using the second subsystem 104.

[0046] For example, second subsystem 104 may have classified the user activity as "running" at state 402. State diagram 400 may then proceed to state 404, where parameters 130 associated with "running" are loaded by first subsystem 102. At state 406, first subsystem 102 may continue to evaluate data 134 from sensor 140. Unlike state diagram 300, if first subsystem 102 classifies the user activity as associated with a second activity (e.g., "walking"), first subsystem 102 will switch to state 408, where parameters 130 associated with "walking" are loaded. State diagram 400 will then proceed to state 410, where first subsystem 102 will continue to classify data 134 received from sensor 140. By switching between states associated with first subsystem 102, second subsystem 104 will not be continuously reactivated, further reducing power consumption.

[0047] It is contemplated that the first subsystem 102 may also modify parameters of a signal source (e.g., a MEMS element operable to improve a key performance indicator (KPI) of the cascaded system 100). By modifying the parameters, power consumption can be further reduced due to the reduced sampling rate. It is also contemplated that the second subsystem 104 may employ a training process that includes the complete data set and all labels. The first subsystem 102 may alternatively be trained with the complete data set, but all labels belonging to activities classified as "other" may be combined into a single category. The resulting parameter sets corresponding to each category of interest may then be stored in associated memories 114, 116 and loaded by the first and second subsystems 102, 104 as needed during inference.

[0048] It is contemplated that a balanced data set should be maintained for the first subsystem 102. During the training process, a weighted cross entropy of the data can be used to maintain a balanced data set. It is further contemplated that the partitioning of the cascade system 100 described above is merely exemplary, and each subsystem (i.e., the first subsystem 102 and the second subsystem 104) can also be designed to operate on highly optimized hardware (e.g., analog in-memory computing) or a cloud-based platform.

[0049] The processes, methods, or algorithms disclosed herein may be delivered to / implemented by a processing device, controller, or computer, which may include any existing programmable electronic control unit or dedicated electronic control unit. Similarly, the processes, methods, or algorithms may be stored in many forms as data and instructions executable by a controller or computer, including but not limited to information permanently stored on non-writable storage media such as ROM devices and information revisably stored on writable storage media such as floppy disks, magnetic tapes, CDs, RAM devices, and other magnetic and optical media. The processes, methods, or algorithms may also be implemented in a software executable object. Alternatively, the processes, methods, or algorithms may be implemented in whole or in part using suitable hardware components (e.g., application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices), or a combination of hardware, software, and firmware components.

[0050] Although exemplary embodiments have been described above, it is not intended that these embodiments describe all possible forms encompassed by the claims. The words used in the specification are descriptive rather than restrictive, and it should be understood that various changes may be made without departing from the spirit and scope of the present disclosure. As previously mentioned, features of the various embodiments may be combined to form other embodiments of the present invention that may not be explicitly described or shown. Although various embodiments may have been described as providing advantages or being preferred relative to other embodiments or prior art implementations with respect to one or more desired characteristics, it should be recognized by those skilled in the art that one or more features or characteristics may be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, maintainability, weight, manufacturability, ease of assembly, and the like. Therefore, to the extent that any embodiment is described as being less desirable than other embodiments or prior art implementations with respect to one or more characteristics, such embodiments are not outside the scope of the present disclosure and may be desirable for a particular application.

Claims

1. A cascade machine learning system comprising: a first subsystem comprising a plurality of operating states and at least a first machine learning algorithm; a second subsystem comprising a second machine learning algorithm; At least a first processor operable to: activating the second subsystem when receiving a first set of sensing data signals; classifying the first set of sensory data signals using the second machine learning algorithm to produce a first classification of user activity of the cascaded machine learning system; selecting a first operating state to be adopted by the first subsystem from the plurality of operating states, wherein the selection is based on the first classification; selecting a set of activity parameters to be used by the first subsystem, wherein the set of activity parameters is selected based on the first operating state; deactivating the second subsystem; classifying a second set of sensory data signals using the first machine learning algorithm to produce a second classification of user activity; comparing the second classification to the set of activity parameters; and If the output classification performed using the first machine learning algorithm is invalid, the second subsystem is reactivated.

2. The cascaded machine learning system according to claim 1, wherein: The first processor is further operable to: comparing a confidence level of a classification of the second set of sensory data signals using the first machine learning algorithm to a threshold; and If the confidence level is less than the threshold, the second subsystem is reactivated.

3. The cascaded machine learning system according to claim 2, wherein: The first processor is further operable to: classifying the second set of sensory data signals using the second machine learning algorithm; and The threshold is increased if the classification of the second set of sensory data signals using at least the first machine learning algorithm is equal to the classification of the second set of sensory data signals using the second machine learning algorithm.

4. The cascaded machine learning system according to claim 2, wherein: The first processor is further operable to: classifying the second set of sensory data signals using the second machine learning algorithm; and If the classification of the second set of sensory data signals using the second machine learning algorithm is equal to one of the set of activity parameters, then reducing the threshold.

5. The cascaded machine learning system according to claim 2, wherein: The first processor is further operable to: If the threshold is adjusted, the second set of sensed data signals is classified using the second machine learning algorithm.

6. The cascaded machine learning system according to claim 2, wherein: The threshold value is stored for a predetermined number of iterations.

7. The cascaded machine learning system according to claim 1, wherein: The first processor is further operable to: After the predetermined period has expired, the second set of sensory data signals is classified using the second machine learning algorithm.

8. The cascaded machine learning system according to claim 1, wherein: The first subsystem is employed on the first processor, and the second subsystem is employed on a second processor.

9. The cascaded machine learning system according to claim 8, wherein: The second processor comprises an application processor, and the first processor comprises a low-power microprocessor.

10. The cascaded machine learning system according to claim 1, wherein: The second machine learning algorithm is trained using the complete training data set and the plurality of labels.

11. The cascaded machine learning system according to claim 1 , wherein: The first subsystem is trained using the complete training data set, but one or more specified labels are combined into one category.

12. A cascade algorithm method comprising: Initializing a first subsystem and a second subsystem, the first subsystem including a first algorithm and a plurality of operating states, and the second subsystem including a second algorithm; activating the first subsystem when a first set of sensing data signals is received; classifying the first set of sensory data signals using the second algorithm to produce a first classification of user activity of the first and second subsystems; selecting at least one operating state to be adopted by the first subsystem, wherein the selection is based on the first classification; selecting a set of activity parameters to be used by the first subsystem, wherein the set of activity parameters is selected based on the selected operating state; deactivating the second subsystem; classifying a second set of sensory data signals using at least the first algorithm to produce a second classification of user activity; comparing the second classification to the set of activity parameters; and If at least one activity parameter of the set of activity parameters indicates that the second classification is invalid, the second subsystem is reactivated.

13. The cascade algorithm method according to claim 12, further comprising: comparing a confidence level of a classification of the second set of sensed data signals using at least the first algorithm to a threshold; as well as If the confidence level is less than the threshold, the second subsystem is reactivated.

14. The cascade algorithm method according to claim 13, further comprising: classifying the second set of sensed data signals using the second algorithm; as well as The threshold is increased if the classification of the second set of sensed data signals using the second algorithm is equal to the classification of the second set of sensed data signals using at least the first algorithm.

15. The cascade algorithm method according to claim 13, further comprising: classifying the second set of sensed data signals using the second algorithm; as well as If the classification of the second set of sensed data signals using the second algorithm is equal to one of the set of activity parameters, the threshold is decreased.

16. The cascade algorithm method according to claim 13, further comprising: If the threshold is adjusted below a predetermined level, the second set of sensed data signals is classified using at least the first algorithm.

17. The cascade algorithm method according to claim 12, further comprising: After the predetermined period has expired, the second set of sensed data signals is classified using the second algorithm.

18. A cascade machine learning method comprising: Initializing a first subsystem and a second subsystem, the first subsystem including at least a first machine learning algorithm and a plurality of operating states, and the second subsystem including a second machine learning algorithm; activating the first subsystem when a first set of sensing data signals is received; classifying the first set of sensory data signals using the second machine learning algorithm to produce a first classification of user activity of the first and second subsystems; selecting a first operating state to be adopted by the first subsystem, wherein the selection is based on the first classification; selecting a first set of activity parameters to be used by the first subsystem, wherein the set of activity parameters is selected based on the first operating state; deactivating the second subsystem; classifying a second set of sensory data signals using at least the first machine learning algorithm to produce a second classification of user activity; comparing the second classification to the set of activity parameters; selecting a second operating state to be adopted by the first subsystem based on the comparison; and A second set of activity parameters is selected to be used by the first subsystem, wherein the second set of activity parameters is selected based on the second operating state.

19. The cascade machine learning method according to claim 18, further comprising: comparing a confidence level of a classification of the second set of sensory data signals using at least the first machine learning algorithm to a threshold; as well as If the confidence level is less than the threshold, the second subsystem is reactivated.

20. The cascade machine learning method according to claim 19, further comprising: classifying the second set of sensed data signals using the second machine learning algorithm; as well as The threshold is increased if the classification of the second set of sensory data signals using at least the first machine learning algorithm is equal to the classification of the second set of sensory data signals using the second machine learning algorithm.

Images