Systems and methods for distributed neural networks on edge devices

Abstract

Systems and methods are provided for distributed neural networks on edge devices. A system and method are disclosed for classifying time series data provided from continuous sensor signals to a machine learning model. The data can be "windowed" or "divided" into smaller pieces of data using a first stage classifier in which "events of interest" can be identified. The first stage classifier can employ an algorithm that prohibits false negatives. The pieces of data detected as including events of interest can then be transmitted to a second stage classifier that is operable to perform a full classification on the pieces of data. The multi-stage network can require less power and less complex structure.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a decentralized machine learning network for detecting and classifying events in sensor signals or edge devices and for providing a full classification upon detection of an event of interest. BACKGROUND

[0002] The task of classifying input data using machine learning based classifiers is understood to be used within a variety of applications. Depending on the given application, the machine learning model used for classification can be massive, with several hundred million variables. Such large networks typically require a large amount of computational processing power and large data storage resources. As a result, such machine learning models can have a large energy footprint. Typically, such large machine learning models are not deployable in resource constrained environments. SUMMARY

[0003] A system and method for a multi-stage machine learning classifier for receiving a sensed data signal is disclosed. The system applies a first stage classifier to parse the sensed data signal into one or more data segments. An event of interest (e.g., a binary classification) can be identified within the one or more data segments using a first machine learning algorithm employed by the first stage classifier. The one or more data segments identified as including the event of interest can be transmitted from the first stage classifier to a second stage classifier. The one or more data segments can then be classified using a second machine learning algorithm employed by the second stage classifier.

[0004] The system and method can also be operable to generate a first classification and a first classification label for the one or more data segments using the first machine learning algorithm employed by the first stage classifier. If a confidence metric of the first classification exceeds a confidence threshold, the first classification can be output as a full or complete classification. If the confidence metric of the first classification is less than the confidence threshold, the first classification and the first classification label can be provided to the second machine learning algorithm employed by the second stage classifier. A second classification can then be generated for the one or more data segments using the second machine learning algorithm employed by the second stage classifier. The second classification can then be output as a full or complete classification.

[0005] The system and method can also combine one or more data segments into a common data set, where the event of interest of the first data segment can be the same as the event of interest of the second data segment. The common data set can then be transmitted from the first stage classifier to the second stage classifier for classification. The sensed data signal can also be parsed into one or more data segments using a second machine learning algorithm. The event of interest can be identified within the one or more data segments using the second machine learning algorithm. The one or more data segments can then be classified using the second machine learning algorithm.

[0006] The system and method can also apply a first classification that identifies the event of interest within the one or more data segments. A second classification can identify that the event of interest is not present within the one or more data segments. The one or more data segments labeled using the first classification can then be transmitted to the second stage classifier. The one or more data segments labeled using the first classification can be identified as not including the event of interest. The one or more data segments can be relabeled using the second classification.

[0007] The system and method can also determine that the one or more data segments identified as having the event of interest exceed a predefined confidence threshold. The one or more data segments can then be classified using the first machine learning algorithm if they exceed the predefined confidence threshold. The one or more data segments identified as having the event of interest can also be classified into a class data set using the first machine learning algorithm. The class data set can then be transmitted from the first stage classifier to the second stage classifier. The second machine learning algorithm can then be used to classify the class data set.

[0008] A training process can be employed to apply a penalty algorithm to ensure that the one or more data segments are biased towards being labeled using the first classification. The penalty algorithm can include a first weight applied to the first classification that is greater than a second weight applied to the second classification. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 An exemplary embodiment of a multi-stage network having a first stage classifier and a second stage classifier is illustrated.

[0010] Figure 2 An exemplary CNN is illustrated.

[0011] Figure 3 Another exemplary embodiment of a multi-stage network 100 is illustrated.

[0012] Figure 4 An alternative embodiment of a first stage classifier is illustrated. DETAILED DESCRIPTION

[0013] Embodiments of the present disclosure are described herein. It is to be understood, however, that the embodiments disclosed are merely examples, and other embodiments can take various forms and alternative forms. The drawings are not necessarily to scale; some features can be exaggerated or minimized for the purpose of clarity. The specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to employ the embodiments in various ways. As will be readily understood by those skilled in the art, the various features illustrated and described with reference to any one of the figures can be combined with features illustrated in one or more other figures to produce embodiments that are not explicitly illustrated or described. The combination of features from the illustrated embodiments provides representative examples for typical applications. However, various combinations and modifications to the features consistent with the teachings of the present disclosure can be desired for particular applications or implementations.

[0014] For machine learning algorithms, the use of neural networks (e.g., CNNs or DNNs) has increased for a wide variety of applications including classifying input data to provide event detection (e.g., events of interest) or object classification (e.g., object recognition or facial recognition). However, neural networks for classifying input data can be very large and provide results with tens of millions of variables. Such large neural networks typically require large amounts of computational and data storage resources and thus can consume large amounts of energy to operate or have large power footprints. Moreover, current neural networks typically cannot be deployed in resource-constrained environments (e.g., within wearable devices such as smart watches).

[0015] It is contemplated that, for certain classification applications, time series data provided to a neural network from a continuous sensor signal can be“windowed” or“partitioned” into smaller segments. Each data segment can then be independently classified. The time series data can also be highly sparse (i.e., portions of the time series data can not have events of interest that need to be classified). Distinguishing whether a time series data segment has an“event of interest” does not require the computationally and data storage resource intensive resources typically needed for a more comprehensive classification phase. Classifying time series data segments that do not include events of interest can even be unnecessary resource usage.

[0016] Accordingly, it is contemplated that a multi-stage network can be employed with a first stage classifier (i.e., a first neural network) that identifies whether a windowed or partitioned data segment includes an event of interest. Data segments detected to include an event of interest can then be transmitted to a second stage classifier (i.e., a second neural network) that performs a full classification on the data segment. The multi-stage network can require less power and less complex structure.

[0017] Figure 1A multi-stage network 100 is illustrated having a stage 1 classifier 102 and a stage 2 classifier 104. As shown, the stage 1 classifier 102 and the stage 2 classifier 104 can be designed to include at least one processor 106, 108 that is operatively connected to a respective memory unit 114, 116. Each processor 106, 108 can be one or more integrated circuits that implement the functionality of a respective central processing unit (CPU) 110, 112, which can be a commercially available processing unit that implements an instruction set, such as one of the x86, ARM, Power, or MIPS instruction set families. However, it is contemplated that commercially available processing units are merely an example for implementing the stage 1 classifier 102 and the stage 2 classifier 104. For example, it is contemplated that the CPUs 110, 112 can be special-purpose accelerators that include special functions or instructions that are optimized for implementing machine learning algorithms or neural networks (e.g., CNNs or DNNs). It is also contemplated that the CPUs 110, 112 can be analog computing units or units that perform computing operations in memory.

[0018] During operation, each CPU 110, 112 can execute stored program instructions retrieved from the associated memory unit 114, 116. The stored program instructions can include software that controls operation of each CPU 110, 112 to perform the operations described herein. In some examples, each processor 106, 108 can be a system on a chip (SoC) that integrates the functionality of each CPU 110, 112; each memory unit 114, 116; the associated network interface 118, 120; and the associated input / output interface 122, 124. It is contemplated that the stage 1 classifier 102 and the stage 2 classifier 104 can implement an operating system for managing various aspects of operation.

[0019] Each memory unit 114, 116 can include volatile memory and non-volatile memory for storing instructions and data. The non-volatile memory can include solid state memory such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the stage 1 classifier 102 and stage 2 classifier 104 are deactivated or lose power. The volatile memory can include static and dynamic random access memory (RAM) that stores program instructions and data. For example, each memory unit 114, 116 can store a machine learning model 126, 128 or algorithm; and / or raw source data 134, 136. The memory units 114, 116 can also store values or data (e.g., coefficients, sks, weights) that can then be loaded by the CPU 110, 112 (or accelerator) for use by the machine learning model 126, 128. The memory units 114, 116 can also be used to store associated training data sets 130, 132 that can be used during a training process.

[0020] Again, the stage 1 classifier 102 and stage 2 classifier 104 can also include associated network interface devices 118, 120. It is contemplated that the network interface devices 118, 120 can be configured to provide a communication link 138 between the stage 1 classifier 102 and stage 2 classifier 104. Alternatively, the network interface devices 118, 120 can be configured to communicate with additional external systems and devices (e.g., sensors). The network interface devices 118, 120 can use I 2 C, SPI, similar methods that allow communication between microcontrollers, EEPROMs, A / D and D / A converters, I / O interfaces, and other similar peripherals in embedded systems. Additionally, the network interface devices 118, 120 can be configured to include wired and / or wireless Ethernet interfaces as defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards. The network interface devices 118, 120 can include a cellular communication interface for communicating with a cellular network (e.g., 3G, 4G, 5G).

[0021] As further shown, the stage 2 classifier 104 can be in communication with an external network 138 such as the World Wide Web or the Internet. The external network 138 can establish standard communication protocols between computing devices. The external network 138 can allow information and data to be easily exchanged between the stage 2 classifier 104 and the network. One or more servers 140 can be in communication with the external network 138.

[0022] It is also contemplated that the stage 2 classifier 104 can be implemented on a server- or "cloud"-based storage system (e.g., Amazon Web Services). The stage 1 classifier 102 can still communicate with the stage 2 classifier 104 using the communication link 138. For example, the stage 1 classifier 102 can be implemented on a security camera that is operable to use the network interface device 118 to communicate with the cloud-based storage system using known Internet protocols. The stage 1 classifier 102 can detect events of interest and transmit them (e.g., using the communication link 138) to the stage 2 classifier 104 (operating on the cloud-based storage system) for further classification.

[0023] The stage 1 classifier 102 and the stage 2 classifier 104 can implement associated machine learning algorithms 126, 128 that are configured to analyze provided or stored raw source data 134, 136, which can include raw or unprocessed sensor data. The raw source data 134, 136 can include video, video clips, images, and raw or partially processed sensor data (e.g., data from a digital camera or a LiDAR sensor). In some examples, the machine learning algorithms 126, 128 can be machine learning network algorithms (e.g., CNNs or DNNs) that can be designed to perform a predetermined function.

[0024] Figure 2 An exemplary CNN 200 is illustrated that can be used by the stage 1 classifier 102 and the stage 2 classifier 104 to classify whether a data clip includes an event of interest or to provide a full classification of a data clip. However, the CNN 200 can be just one example of implementing the stage 1 classifier 102 and the stage 2 classifier 104. For example, it is also contemplated that the stage 1 classifier 102 and the stage 2 classifier 104 can be implemented using decision trees, where a feature extractor can provide the data 134, 136. As illustrated, the CNN 200 can include one or more convolutional layers 220-240; one or more pooling layers 250-270; one or more fully connected layers 260; and a softmax layer 270. Depending on the given application, the stage 1 classifier 102 and the stage 2 classifier 104 can further include additional CNNs, DNNs, or decision trees.

[0025] The CNN 200 can receive data 210 (e.g., Figure 1The data 210 can include image data, speech data, text data, or measurements received from sensors or transducers. The data 210 can also be lightly processed before being provided to the CNN 200. The convolutional layers 220-240 can be designed to extract features from the data 210. For example, the convolutional layers 220-240 can employ a filtering operation (e.g., a kernel) before passing the results to the next layer of the CNN 200. The filtering operation can include image identification, edge detection, and image sharpening applied when the received data 210 is an image.

[0026] The CNN 200 can also include one or more pooling layers 250-270 that receive convolutional data from the respective convolutional layers 220-240. The pooling layers 250-270 can include one or more pooling layer units that apply a pooling function to one or more convolutional layer outputs computed at different frequency bands. For example, the pooling layer 250 can apply a pooling function to the kernel output received from the convolutional layer 220. The pooling function implemented by the pooling layers 250-270 can be a mean or max function or any other function that aggregates multiple values into a single value.

[0027] Next, one or more fully connected layers 280 can attempt to learn a non-linear combination of high-level features in the output data received from the convolutional layers 220-240 and the pooling layers 250-270. Finally, the CNN 200 can include a softmax layer 290 that uses a softmax function to combine the outputs of the fully connected layers 280. It is also contemplated that the CNN 200 can further employ batch normalization layers, max pooling layers, and dropout layers.

[0028] Figure 3 Another example embodiment of a multi-stage network 100 that can be employed to detect and classify events of interest provided from sensor signals or input data (e.g., data 134, 136) is illustrated. As shown, the stage 1 classifier 102 employs a machine learning model 126 to detect events of interest or activity spotting from one or more "windowed" data segments. If relevant activity (i.e., events of interest) is identified, the stage 1 classifier 102 can be operable to generate a binary classification. The stage 1 classifier 102 can employ an architecture like the CNN 200. Alternatively, the stage 1 classifier 102 can employ additional or fewer layers (e.g., convolutional layers, fully connected layers) than shown for the CNN 200. However, as discussed below, it is also contemplated that the stage 1 classifier 102 can be operable to perform a full or complete classification of data segments determined to be above a predefined confidence threshold.

[0029] The stage 1 classifier 102 can also be incorporated within one or more sensors, or embedded within a wearable or edge device. For example, within a vehicle 311, the stage 1 classifier 102 can be embedded within a radar sensor 302, a video camera 304, a global positioning system 306, a laser sensor 308 (e.g., LiDAR), or an ultrasonic sensor 310. Alternatively, the stage 1 classifier 102 can be positioned alongside each of the sensors 302-310. In other words, the stage 1 classifier 102 can be included within each of the sensors 302-310, and unique to each of the sensors 302-310. Alternatively, it is contemplated that the stage 1 classifier 102 can receive data, and be operable for more than one of the sensors 302-310. For example, the stage 1 classifier 102 can be in communication with, and receive sensed data from, more than one of the sensors 302-310. Again, the stage 1 classifier 102 can use I 2 SPI, or similar communication protocol, to communicate with the sensors 302-310.

[0030] Likewise, the stage 1 classifier 102 can be embedded within one or more wearable devices 312-318 (e.g., a smart watch, earpiece, or heart rate monitor). The wearable devices 312-318 can include a number of sensors (e.g., a heart rate monitor, pedometer, accelerometer, or gyroscope). Again, the stage 1 classifier 102 can be within each sensor within the wearable devices 312-318, or alongside each sensor. Alternatively, the stage 1 classifier 102 can receive data, and be operable for more than one of the sensors within the wearable devices 312-318.

[0031] In general operation, a sensor can output a continuous signal of sensed data. The stage 1 classifier 102 can be operable to segment this continuous sensed data signal into “windowed” data segments, which can either be completely independent of other windowed data segments, or can overlap or combine with other windowed data segments.

[0032] It is also contemplated that the stage 1 classifier 102 can be optimized to use a machine learning network that is less complex and less power intensive compared to a more robust classification network. For example, the stage 1 classifier 102 can include fewer layers (i.e., fewer convolutional layers 322-324 or fully connected layers 326) than the machine learning network employed by the stage 2 classifier 104. Again, the stage 1 classifier can also be optimized using decision trees or other known classifiers designed for a given application. The stage 1 classifier 102 can then operate to detect events of interest within each windowed data segment, or perform a low effort classification on the sensed data or windowed data segments. If the stage 1 classifier 102 detects an event of interest (or if the confidence of the classification performed by the stage 1 classifier 102 is below a given threshold), the identified windowed data segment can be provided to the stage 2 classifier 104, which includes a machine learning algorithm 128 designed to perform a full or more complex classification.

[0033] It is contemplated that, in order to handle full classifications that require higher accuracy, the stage 2 classifier 104 can be a more complex machine learning network than the machine learning network employed by the stage 1 classifier 102. For example, the stage 2 classifier 104 can employ additional layers (e.g., convolutional layers, fully connected layers) for providing a full classification of the data segments transmitted by the stage 1 classifier 102. The stage 2 classifier 104 can further include more complex design structures, such as a skip architecture 328 (i.e., a ResNet architecture), in which an output generated by one or more layers (e.g., one of the convolutional layers 330-334) can be provided as input to another layer (e.g., one of the fully connected layers 338-342). Moreover, by employing a less complex network (i.e., the stage 1 classifier 102) to handle event detection, the amount of data transmitted to the stage 2 classifier 104 can be reduced.

[0034] While the stage 2 classifier 104 can generally operate only on data segments received from the stage 1 classifier 102, it is contemplated that the stage 2 classifier 104 can receive continuous sensor signals (i.e., data) directly. In other words, the stage 2 classifier 104 can operate independently, or without the stage 1 classifier 102. The stage 2 classifier 104 can operate to window the data into segments; identify events of interest for each data segment; and provide a full classification of the data segments identified as including events of interest. It is contemplated that the stage 2 classifier 104 can be designed to handle identification of events of interest and full classification without the sensor or wearable device including the stage 1 classifier 102. Or, such operation can be necessary if the stage 1 classifier 102 becomes inoperable.

[0035] It is also contemplated that when operating to perform event detection, the stage 1 classifier 102 should be designed to capture and label all events of interest within a windowed data segment. For example, the stage 1 classifier 102 can label a windowed data segment in which an event of interest occurs as class "1" and label a windowed data segment in which no event of interest occurs as class "0." During operation, if the stage 1 classifier 102 labels a data segment from class "0" as class "1," this can be acceptable. However, it is contemplated that the stage 1 classifier 102 should not label a data segment that should (or has been labeled) as class "1" with a label of class "0."

[0036] It is to be understood that a data segment that is incorrectly identified as class "1" by the stage 1 classifier 102 will still be transmitted to the stage 2 classifier 104. And, the stage 2 classifier 104 can then be operable to identify and correctly label the data segment as class "0" (i.e., no event of interest requiring full classification). However, if a data segment is incorrectly identified as class "0" by the stage 1 classifier 102, the data segment will not be correctly relabeled as class "1" because no data will be transmitted to the stage 2 classifier 104. Thus, it is contemplated that the stage 1 classifier 102 should be biased to penalize false negatives (i.e., false classification of a data segment that should be class "1" to class "0").

[0037] Thus, it is contemplated that the stage 1 classifier 102 can be penalized (during the training process) from applying false classifications by assigning class "1" labels as higher than class "0" labels using Equation 1 below:

[0038]

[0039] where y i = 1, false negative

[0040] where y i = 0, false positive

[0041] It is contemplated that y i may be the actual label associated with the ith segment, and ​​may be an output label predicted by the neural network. The terms w1, w0may be weights associated with false negatives and false positives, respectively, used during the training process. It is contemplated that the terms w1, w0may be adjusted during the training process to penalize false negatives. For example, the stage 1 classifier 102 can be trained to ensure that w1> w0. It is also contemplated that the values used in equation 1 can be selected to achieve an acceptable low rate of false negatives (e.g., < 3%), while attempting to achieve a false positive rate below a desired threshold. When there is an equal number of training samples (e.g., training data 130) available for both class "0" and class "1", the stage 1 classifier 102 can have a low rate of false negatives when w1= 1 and w0= 0.1. However, the values of w1and w2may be assigned different values depending on the application or process.

[0042] It is also contemplated that the stage 1 classifier 102 can also employ a full classification process. The stage 1 classifier 102 can perform a full classification process by processing less complex classification samples that can not require a large amount of computational processing power. For example, the machine learning model 126 deployed by the stage 1 classifier 102 can be repurposed to perform a full classification by using a portion of the existing layers (e.g., convolutional layers) and introducing one or more additional output layers. Alternatively, the existing layers can be used with additional fully connected layers to perform a full classification.

[0043] It is contemplated that for the stage 1 classifier 102, the output for the layers for which the network is public can be used as input to additional layers, which can then be used to perform a full classification. The reuse of pre-computed outputs can not significantly increase the complexity of the stage 1 classifier 102 to perform a full classification. For multi-class classification, the output of the last layer of the stage 1 classifier 102 can provide the probability / confidence of the network for each class. It is contemplated that the output for the i-th input patch can be given as

[0044] It is also contemplated that the stage 1 classifier 102 can be designed to handle full or complex classification by determining whether the windowed data patch is above or within a predefined confidence threshold (e.g., a confidence level approximately equal to or greater than 90% or between 80-90%). For windowed data patches for which the stage 1 classifier 102 is able to perform classification with high confidence (e.g., a confidence level greater than 90%), further classification by the stage 2 classifier 104 can not be needed. However, if the windowed data patch is below the predefined confidence threshold (e.g., a data sample with a confidence level less than 90%), the stage 1 classifier 102 can transmit the data patch to the stage 2 classifier 104 for full or more complex classification.

[0045] Figure 4An example of a stage 1 classifier 102 is illustrated that can operate to process both event binary based classification (which will be passed to a stage 2 classifier 104) and lower and complex classification of data snippets based on a confidence threshold. As illustrated, the stage 1 classifier 102 can receive input data (e.g., data 134). The stage 1 classifier 102 can then include a first convolutional filter 402 and a second convolutional filter that extract features that can be classified with one or more fully connected layers. For example, fully connected layers 410, 412 can be used to generate an output 408 that can include a less complex or broad category classification of the data snippet. Alternatively, fully connected layers 410, 414 can be operable to generate an output 406 that can include a more complex, full or fine category classification of the data snippet.

[0046] For example, the stage 1 classifier 102 can perform a category classification of a data snippet to a designated group (e.g., animals, vehicles or food) that can then be passed to the stage 2 classifier 104 for a full classification of the assigned group. The stage 1 classifier 102 can determine that a windowed data snippet belongs to an "animals" category group and the stage 2 classifier 104 can then provide a full classification that the windowed data snippet is an image of a "dog." By operating to provide a broad category group classification, the stage 1 classifier 102 can increase the confidence of the full classification.

[0047] It is also contemplated that the stage 1 classifier 102 can be generally designed to have a very low density. As such, the stage 1 classifier 102 can not produce high accuracy when performing full classifications across all data snippets. Instead, the stage 1 classifier 102 can be highly accurate in performing full classifications only on a small number of data snippets when the associated probability / confidence can be high. For example, the stage 1 classifier 102 can generate an output classification 406 (i.e., a full classification) when the confidence is high. If the confidence level is low, the stage 1 classifier 102 can output a low effort classification label to the stage 2 classifier 104 for a full classification.

[0048] The stage 1 classifier 102 can use the following logic equations 2 and 3 to determine whether to perform a lower or more complex classification:

[0049] If > threshold, output class: Else, i * = output stage 2 (Equation 3)

[0050] As illustrated, if the stage 1 classifier 102 provides a classification with a confidence threshold that is high (e.g., greater than 90%), then the classification from the stage 1 classifier 102 is used. However, if the stage 1 classifier 102 provides a classification with a confidence threshold that is not high (e.g., less than 90%), then a low effort classification label is assigned and the data segment is transmitted to the stage 2 classifier for full or more complex classification.

[0051] Alternatively, it is also contemplated that the stage 1 classifier 102 can determine whether a data segment with an event of interest has a probability confidence above a predefined threshold (e.g., 90%). If yes, then the stage 1 classifier 102 can generate an output classification 406. If no, then the output data segment 408 can be transmitted to the stage 2 classifier 104. Again, to add an additional layer for full classification, additional layers can be added to the stage 1 classifier 102 to perform a broad / fine class classification.

[0052] It is contemplated that since a large portion of the stage 1 classifier 102 can be common to event detection or full classification (or some broad class classification), it can be beneficial to train the common layers of the machine learning model 126 for a full classification task (rather than a binary classification task). It is contemplated that full classification can be more complex and training for more complex problems improves the discriminative power of the network. Thus, the initial layers of the stage 1 classifier 102 network, which are generally thought to perform feature extraction, extract features that are more relevant to the full classification task. Since event detection or binary classification is a subset of the full classification problem, the extracted features can also be operable for detection.

[0053] The processes, methods, or algorithms disclosed herein can be deliverable to / implemented by a processing device, controller, or computer that can include any existing programmable electronic control unit or a specially designed electronic control unit. Similarly, the processes, methods, or algorithms can be stored as data and instructions on a non-transitory storage medium that can be readable by a controller or computer, including non-writable storage media and writable storage media, such as floppy disks, magnetic tapes, CD-ROMS, DVDs, memory devices, and other magnetic and optical media. The processes, methods, or algorithms can also be implemented in a software executable object. Alternatively, the processes, methods, or algorithms can be embodied in whole or in part by suitable hardware components, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware, software and firmware components, and can implement all or part of the processes, methods, or algorithms in conjunction with a suitable operating system.

[0054] While example embodiments have been described above, it is not intended that these embodiments describe all possible forms of the claims. The words used in the specification are words of description rather than limitation, and it is understood that various changes can be made without departing from the spirit and scope of the disclosure. As previously described, features of various embodiments can be combined to form further embodiments of the present application that can not be expressly described or illustrated. While various embodiments can have been described as providing advantages or being preferred over other embodiments or prior art implementations, this description is not intended to be construed as a limitation of the scope of the application. It is contemplated that various modifications, changes, revisions, and substitutions can be made to the embodiments described herein without departing from the spirit and scope of the application. It is intended that the application be limited only by the scope of the claims and the reasonable interpretation thereof. Any heading used in the Appendix A or Appendix B is for convenience only and does not interpret the scope or meaning of the embodiments.

Claims

1. A method for a multi-stage machine learning classifier, comprising: receiving a sensed data signal, the sensed data signal being image data; applying a first stage classifier to parse the sensed data signal into one or more data segments; identifying events of interest within the one or more data segments using a first machine learning algorithm employed by the first stage classifier; transmitting the one or more data segments identified as including events of interest from the first stage classifier to a second stage classifier; and classifying the one or more data segments identified as including events of interest using a second machine learning algorithm employed by the second stage classifier, the second stage classifier trained to directly receive the sensed data signal and perform both the identification of events of interest and full classification when the first stage classifier is inoperable.

2. The method of claim 1, wherein the first stage classifier generates a binary classification to identify events of interest within the one or more data segments.

3. The method of claim 1, further comprising: combining the one or more data segments into a common data set, wherein events of interest of a first data segment match events of interest of a second data segment; transmitting the common data set from the first stage classifier to the second stage classifier; and classifying the common data set using the second machine learning algorithm.

4. The method of claim 1, further comprising: parsing the sensed data signal into one or more data segments using the second machine learning algorithm; identifying events of interest within the one or more data segments using the second machine learning algorithm; and classifying the one or more data segments using the second machine learning algorithm.

5. The method of claim 1, further comprising: applying a first classification, the first classification identifying events of interest within the one or more data segments; and applying a second classification, the second classification identifying events of interest that are not present within the one or more data segments.

6. The method of claim 5, further comprising: transmitting the one or more data segments labeled using the first classification to the second stage classifier.

7. The method of claim 6, further comprising: identifying that the one or more data segments labeled using the first classification do not include events of interest; and relabeling the one or more data segments using the second classification.

8. The method of claim 5, further comprising: applying a penalty algorithm during a training process to ensure that the one or more data segments are biased toward being labeled using the first classification.

9. The method of claim 8, wherein the penalty algorithm includes a first weight applied to the first classification, the first weight being greater than a second weight applied to the second classification.

10. The method of claim 1, further comprising: determining that the one or more data segments identified as having events of interest exceed a predefined confidence threshold; and ​ ​ ​ ​ ​ ​ classifying the one or more data segments that exceed the predefined confidence threshold using a first machine learning algorithm.

11. The method of claim 1, further comprising: classifying the one or more data segments identified as having an event of interest into a class data set using a first machine learning algorithm; transmitting the class data set from the first stage classifier to a second stage classifier; and classifying the class data set using a second machine learning algorithm.

12. A system for a multi-stage machine learning classifier, comprising: a sensor operable to generate a sensed data signal, the sensed data signal being image data; a first stage classifier comprising a first controller operable to: receive the sensed data signal from the sensor; parse the sensed data signal into one or more data segments; identify events of interest within the one or more data segments using a first machine learning algorithm; transmit the one or more data segments identified as including events of interest from the first stage classifier to a second stage classifier; a second stage classifier comprising a second controller operable to: receive the one or more data segments identified as including events of interest; and classify the one or more data segments identified as including events of interest using a second machine learning algorithm, the second stage classifier trained to directly receive the sensed data signal and perform identification of events of interest and full classification when the first stage classifier is inoperable.

13. The system of claim 12, wherein the first machine learning algorithm comprises one or more convolutional layers and one or more fully connected layers.

14. The system of claim 12, wherein the second machine learning algorithm comprises one or more convolutional layers and one or more fully connected layers.

15. The system of claim 12, wherein the first stage classifier generates a binary classification to identify events of interest within the one or more data segments.

16. The system of claim 12, further comprising: the first controller operable to: combine the one or more data segments into a common data set, wherein an event of interest of a first data segment matches an event of interest of a second data segment; transmit the common data set from the first stage classifier to the second stage classifier; and the second controller operable to: classify the common data set using the second machine learning algorithm.

17. The system of claim 12, further comprising: the second controller operable to: parse the sensed data signal into one or more data segments; identify events of interest within the one or more data segments using the second machine learning algorithm; and classify the one or more data segments using the second machine learning algorithm.

18. The system of claim 12, further comprising: applying a first classification, the first classification identifying events of interest within the one or more data segments; and ​ applying a second classification, the second classification identifying an absence of an event of interest within the one or more data segments.

19. A method for a multi-stage machine learning classifier, comprising: receiving a sensed data signal, the sensed data signal being image data; applying a first stage classifier to parse the sensed data signal into one or more data segments; generating a first classification and a first classification label for the one or more data segments using a first machine learning algorithm employed by the first stage classifier; outputting the first classification as a final classification if a confidence metric of the first classification exceeds a confidence threshold; providing the first classification and the first classification label to a second machine learning algorithm employed by a second stage classifier if the confidence metric of the first classification is less than the confidence threshold; generating a second classification for the one or more data segments using the second machine learning algorithm employed by the second stage classifier, the second stage classifier trained to directly receive the sensed data signal and perform identification of an event of interest and perform a full classification to generate the second classification when the first stage classifier is inoperable; and outputting the second classification as the final classification.

Images