Systems and methods for interpretable sequential and time series data modeling

Abstract

Systems and methods for interpretable sequence and time-series data modeling. A novel interpretable and manipulable deep sequence modeling technique is disclosed. This technique combines prototype learning and RNNs to achieve both interpretability and high accuracy. Experiments and case studies on various real-world sequence prediction / classification tasks demonstrate that the model is not only as accurate as other state-of-the-art machine learning techniques, but also significantly more interpretable. Furthermore, large-scale user studies on Amazon Mechanical Turk demonstrate that for familiar domains, such as sentiment analysis of text, the model is able to select high-quality prototypes that well match human knowledge for prediction and interpretation. Moreover, the model achieves better interpretability without performance loss by updating the prototypes by incorporating feedback from user studies, demonstrating the benefits of involving human-machine loops for interpretable machine learning.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 849,669, filed May 17, 2019, the contents of which are hereby incorporated herein by reference in their entirety. Technical Field

[0003] This application generally relates to a system for generating explanatory information for decisions made by machine learning algorithms. Background Technology

[0004] Machine learning is increasingly used in devices and computing systems for decision-making and prediction. While they have practical applications, the lack of transparency in machine learning algorithms often limits their application in many critical decision-making scenarios. The need for more transparent and understandable machine learning systems has become even more pressing, due to recent EU regulations requiring a “right of interpretation” for algorithms used to make individual-level predictions. Summary of the Invention

[0005] Systems and methods for implementing prototype sequence machine learning networks are disclosed. The processor is operable to implement the prototype sequence machine learning network. The system and methods can use a sequence encoder to map one or more labeled sequence datasets to generate embedding vectors of fixed length. It is anticipated that the sequence encoder can be designed using Long Short-Term Memory (LSTM) networks, bidirectional LSTM networks, or Gated Recurrent Unit (GRU) networks.

[0006] A score can be determined between an embedding vector and one or more prototype vectors to generate one or more similarity vectors. A score of "zero" can be assigned when the sequential embedding of an embedding vector differs from that of a prototype vector. Conversely, a score of "one" can be assigned when the sequential embedding of a single embedding vector is identical to that of a prototype vector. The system and method can also use a fully connected layer to classify one or more predicted values, applying a weight matrix to each of the one or more similarity vectors.

[0007] Fully connected layers may also include a sparsity penalty and a nonnegation constraint that confines the weight matrix to one or more positive values. The sparsity penalty and nonnegation constraint can be operable to improve the learning process for prototype sequence machine learning algorithms, giving the one or more prototype vectors more unified and additive semantics. It is anticipated that at least one of the one or more prototypes may also be modified, corrected, and / or deleted during the training process.

[0008] The system and method are also expected to utilize a softmax layer to compute predicted probabilities for the labeled sequence dataset, whereby the softmax layer divides the exponent of each of the one or more predicted values ​​by the sum of the one or more predicted values. A diversity regularization value can also be applied to the one or more prototypes to penalize at least one of the one or more prototype vectors that is similar to a second of the one or more prototype vectors. A clustering regularization function can also be applied to the one or more labeled datasets and the one or more prototype vectors to ensure clustering structure in the latent space. An evidence regularization function can also be applied to ensure that the one or more prototype vectors are approximately equal to the one or more labeled sequence datasets.

[0009] The system and method are also operable to assign sequence embedding vectors provided from a training dataset to the one or more prototype vectors. It is contemplated that the sequence embedding vectors may be approximately equal to the one or more prototype vectors. It is also contemplated that the assignment of the one or more prototype vectors may occur at a predetermined time. The system and method can also project the one or more prototype vectors onto a subsequence of events within the training dataset. The beam search algorithm can also be implemented to select candidates determined to be substantially equal to the one or more prototype vectors. Attached Figure Description

[0010] Figure 1 This is an example configuration for an interpretable sequence modeling system 100.

[0011] Figure 2 The machine learning model configurations for improved interpretability are described.

[0012] Figure 3 An example of user interaction for updating a machine learning model is depicted.

[0013] Figure 4 An example explanation of using a machine learning model for vehicle diagnostic systems is provided.

[0014] Figure 5A An example explanation of a machine learning model, such as that used in a restaurant review app, is described.

[0015] Figure 5B An example explanation of a machine learning model, such as that used in a restaurant review app, is described.

[0016] Figure 5C An example explanation of a machine learning model, such as that used in a restaurant review app, is described.

[0017] Figure 6Examples of prototype protein sequences and adjacent sequences in the test data are depicted.

[0018] Figure 7 An example prototype depicting a heartbeat signal.

[0019] Figure 8 A heatmap was plotted to illustrate the effect of the diversity regularization term.

[0020] Figure 9 Possible beam search algorithms are described. Detailed Implementation

[0021] Embodiments of this disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various forms and alternative forms. The figures are not necessarily to scale; some features may be enlarged or minimized to show details of particular components. Therefore, the specific structural and functional details disclosed herein are not to be construed as limiting, but merely as a representative basis for teaching those skilled in the art to employ the invention in various ways. As will be understood by those skilled in the art, various features illustrated and described with reference to any of the figures may be combined with features illustrated in one or more other figures to produce embodiments not explicitly illustrated or described. Combinations of illustrated features provide representative embodiments for typical applications. However, for a particular application or implementation, various combinations and modifications of features consistent with the teachings of this disclosure may be expected.

[0022] One of the major challenges in machine learning today is providing predictions that are not only highly accurate but also user-friendly to interpret. While we have witnessed the increasing popularity of deep neural networks for sequence modeling in recent years, explaining the underlying principles behind the model's output remains difficult. This is essential for building trust and supporting domain experts to validate, critique, and improve the model.

[0023] This paper describes an interpretable and manipulable deep sequence model with natural interpretations derived from case-based reasoning. Predictions can be obtained by comparing the input with several prototypes, which are canonical cases in the problem domain. For better interpretability, several criteria, including simplicity, diversity, and sparsity, are defined for constructing the prototypes. Learning objectives and optimization processes are also defined. The described system provides a user-friendly approach to model manipulation: domain experts without any knowledge of the underlying model or parameters can easily incorporate their intuition and experience by manually improving the prototypes.

[0024] Applications include a wide range of real-world applications, including predictive diagnostics for automobiles, electrocardiograms (ECG), protein sequence classification, and sentiment analysis of text. The described system achieves accuracy comparable to state-of-the-art deep sequence models. The model utilizes high-quality prototypes that closely match human knowledge and can be interactively improved for better interpretability without sacrificing performance.

[0025] Event sequence data is becoming increasingly prevalent across various fields, such as electronic health records (HERs) in healthcare, clickstreams in software applications, and vehicle diagnostic logs in automobiles. Generally, an event sequence is a series of events arranged in chronological order. With advancements in machine learning, particularly deep learning, there is a growing research trend towards applying sequence learning to aid decision-making in these fields. For example, by modeling diagnostic sequences collected from a fleet of vehicles, a system can predict potential future diagnostic conditions. This can enable predictive maintenance for automakers and repair facilities, improving customer experience and reducing warranty costs.

[0026] The most widely adopted approach for modeling sequential data is the Recurrent Neural Network (RNN) and related variants, such as Long Short-Term Memory (LSTM) networks. RNNs have achieved superior performance in a variety of sequence modeling applications, such as document / text classification, machine translation, and speech recognition. Despite their superior performance, RNNs are often considered "black boxes" lacking transparency, thus limiting their application in many critical decision-making scenarios. The need for more transparent and understandable machine learning systems has become even more urgent due to recent EU regulations requiring "interpretation rights" for algorithms used to make individual-level predictions.

[0027] To address this challenge, various techniques have been developed to reveal the inner workings of deep sequence models by visualizing changes in hidden states, extracting feature importance, and constructing rules that mimic the behavior of RNNs. However, post-hoc interpretations can be incomplete or inaccurate in capturing the reasoning process of the original model. Therefore, it is often desirable to have models with inherent interpretability in many application scenarios.

[0028] The described system leverages the concept of prototype learning to construct deep sequence learning models with built-in interpretability. Prototype learning is a form of case-based reasoning that draws conclusions about a new input by comparing it with exemplary cases (e.g., prototypes) in the problem domain. This is a natural practice in everyday problem-solving processes. For example, physicians make diagnoses and generate prescriptions based on their experience with past patients, and mechanics predict potential malfunctions by remembering vehicles exhibiting similar symptoms. Prototype learning mimics such human problem-solving processes for better interpretability. This concept can be incorporated into convolutional neural networks to build interpretable image classifiers. However, prototype learning for modeling sequential data has not yet been explored.

[0029] The described system incorporates a prototype sequence network design, a novel sequence learning model that combines prototype learning with RNNs to achieve both interpretability and high accuracy in sequence modeling. An RNN is used as the backbone to capture the latent structure of temporal evolution. Predictions for new input sequences are performed based on similarity to prototypes in the latent space. A novel model improvement method incorporates user knowledge as constraints to enhance user trust.

[0030] The described system considers the criteria of simplicity, diversity, and sparsity for constructing prototypes for explanations. Regarding simplicity, it is possible to directly use the original sequences in the data as prototypes, but these sequences may contain irrelevant noise. In the described system, the prototype can be a subsequence of the original training data and contains only the key events that determine the output. Shorter prototypes are preferred for presenting explanations in a more concise form.

[0031] Regarding diversity, redundant prototypes should be avoided, as they increase the complexity of interpretation without improving performance. Therefore, the described system attempts to use a set of prototypes that are sufficiently different from each other. The prototypes also provide a high-level overview of the original data, which may be several orders of magnitude large.

[0032] Regarding sparsity, the desired outcome for each input is that only a few prototypes are "activated," so that people are not overwhelmed by lengthy and redundant explanations.

[0033] We introduce a novel learning objective that takes into account the aforementioned criteria and propose a training process that iteratively performs gradient descent and prototype projection. For manipulable learning, we consider a constrained training process with multiple user-specified prototypes that reflect the intuition and experience of experts in the domain.

[0034] The system has been applied to and evaluated on several real-world datasets, achieving comparable performance under state-of-the-art deep learning techniques. Experiments cover a wide range of applications, including predictive car maintenance, protein sequence classification, ECG signal annotation, and sentiment analysis of customer reviews, demonstrating the general applicability of the method. In each experiment, classification accuracy on both training and test data is reported, and intuitive interpretations of the results are presented through specific case studies and visualizations. A number of prototypes have also been disclosed. k The impact and the provision of options k The guiding principles. Further research can be conducted to explore the impact of incorporating diversity and simplicity criteria into the model.

[0035] To further evaluate the interpretability of the prototype, a user study was conducted on Amazon Mechanical Turk (MTurk) for a sentiment analysis task on customer reviews. The results show that the described system is able to select high-quality prototypes that match well with human knowledge about natural language for sentiment classification. The model can be manipulated to achieve comparable performance with even better interpretability by learning under constraints with user-specified prototypes.

[0036] The described system exposes a sequence model that learns interpretable representations via sequence prototypes for predictive tasks. An interaction scheme is also described that allows human experts to incorporate their domain knowledge by validating and updating the learned sequence prototypes.

[0037] The described system mimics our everyday problem-solving process by matching inputs to historical data and generating solutions accordingly. Unlike the nearest neighbor classifiers used in typical case-based reasoning systems, this system uses only a few selected prototypes that are memorized, simplified, and used for reasoning. This sparsity offers several advantages: 1) comparing predictions and their interpretations is easier for different inputs; 2) the learned prototypes provide a concise overview of the original data, which may be several orders of magnitude larger; and 3) a human-in-the-loop approach is possible to interactively update the prototypes, allowing human operators / users to incorporate their domain knowledge to further improve the model's interpretability. The system incorporates a combination of prototype-based reasoning and deep neural networks (DNNs) for predictive analytics on sequential data.

[0038] There is no universally applicable method for evaluating the interpretability of machine learning models, and it is often dependent on the use case and the model. Quantitative approaches measure the sparsity of features or the complexity of the model (e.g., the number of rules in a decision tree). However, how these metrics relate to human interpretability remains unknown. One technique might be to evaluate how well a prototype interprets predictions based on user research conducted on MTurk.

[0039] The systems and methods described in this paper can be implemented on computing systems. Figure 1 An example configuration for an interpretable sequence modeling system 100 is depicted. System 100 may include a computing system 102. Computing system 102 may include a processing unit 104 configured to execute instructions and programs. Computing system 102 may include volatile and non-volatile memory 106 that can be used to store programs and data. Computing system 102 may include an input / output (I / O) interface 108 for interacting with an external device 112. Computing system 102 may include a communication interface 110 configured to provide communication with other devices and systems. For example, communication interface 110 may include an Ethernet interface for connecting to an external network 116 or a cloud. User equipment 118 may be connected to the external network 116. Server 120 may be coupled to the external network 116.

[0040] The computing system 102 can also interface with a local database 114. The local database 114 can provide additional storage capacity. The server 120 can connect to an external database 122 that can store relevant data. For example, the local database 114 and / or the external database 122 can store training and testing data used to train the models disclosed herein. For example, the computing system 102 can be programmed to implement a machine learning model to generate outputs based on inputs received via an external network 116 from an external device 112 or a user device 118. The computing system 102 can be programmed to update the machine learning model based on the provided inputs received and / or training data.

[0041] This paper describes a machine learning model architecture. This machine learning model architecture can be implemented in computing system 102. The labeled sequence dataset can be represented as:

[0042] (1)

[0043] Where T is the sequence length. Steps t The input vector at that point, and The labels are for the sequences. The method described in this paper seeks to learn representative prototype sequences (not necessarily existing in the training database) that can be used as classification references and analogical explanations. For a new input sequence, the similarity between the new input sequence and each of the representative sequences is measured in the learned latent space. Predictions for new instances can then be derived and interpreted from their similar prototype sequences.

[0044] Figure 2 Possible model structures that can be implemented in computing system 102 are described. For example... Figure 2 As shown, the model can consist of three components: a sequence encoder. r 206. Prototype Layer p 210 and fully connected layer f 214. The input to the system can be input sequence 202. Input sequence 202 can be composed of input elements 204. Input sequence 202 can be represented as... Sequence encoder r 206 can map the entire input sequence 202 to a single embedding vector, which is expressed as:

[0045] (2).

[0046] Embedded vector e It can have a fixed length. Sequence encoder. r 206 can implement a backbone sequence learning model. For example, Long Short-Term Memory (LSTM), Bidirectional LSTM, and Gated Recurrent Unit (GRU) can be used. Some configurations can use a Bidirectional LSTM as a recurrent sequence encoder. r 206, and the cell state at the last step. c （t） It can be used as an embedding vector. The hidden state at the final step 208. h （t） It can be used as an embedding vector.

[0047] Prototype layer p 210 can contain k prototype vectors It has the same as e Same length. Prototype layer 210 can be configured to be paired with each prototype. p i The similarity between them is scored. Prototype layer 210 can output the squared value. L 2 Distance term, the square L 2 The distance term can be calculated as:

[0048] (3)

[0049] To improve interpretability, a similarity scoring layer 212 can be implemented. The similarity score can be calculated for each element in the output of prototype layer 210 as follows:

[0050] (4)

[0051] Similarity scoring can convert distance into a score between zero and one. "Zero" can be interpreted as sequence embedding. e With prototype vector p i Completely different, and "one" can be interpreted as sequence embedding. e With prototype vector p i same.

[0052] Once the similarity vector is calculated a = p(e) Then the fully connected layer 214 can be executed to compute. z=Wa ,in W yes C x k Weight matrix, and C This is the output size (the number of classes in the classification task). Elements in the fully connected layer 214 can be connected to the entire output of the similarity scoring layer 212. To enhance interpretability, [the following can be added]: W The constraint is non-negative. For multi-class classification tasks, a softmax layer of 2^16 can be used to calculate the predicted probabilities as follows:

[0053] (5).

[0054] The goal can be to train a model that is both accurate and interpretable. The network can be trained by setting weighting factors within the model to achieve acceptable performance. The model can be trained by updating the weighting factors and evaluating the results (e.g., based on a loss function). This process can be iterated until the desired result is obtained. During training, the weighting factors can be updated according to a predetermined algorithm (e.g., gradient descent). For accuracy, the cross-entropy loss can be minimized on the training set. The cross-entropy loss can be expressed as:

[0055] (6)

[0056] in, It is the set of all trainable parameters of the model.

[0057] Experimental results indicate that when the number of prototypes kWhen the number of classes is large (e.g., 2-3 times greater than the number of classes), this training may result in many similar or even duplicate prototypes (some prototypes are very close to each other in the latent space). Having multiple similar prototypes in the interpretation can lead to clutter and inefficiency in utilizing model parameters. To prevent this, a diversity regularization term can be incorporated, which penalizes prototypes that are close to each other. The diversity regularization term can be expressed as:

[0058] (7)

[0059] in d min It is a threshold used to classify whether two prototypes are close. In some examples, d min The value can be set to 1.0 or 2.0. R d This is soft regularization, which imposes a larger penalty on small pairwise distances. By preserving the prototypes distributed in the latent space, sparser similarity vectors can be produced. a .

[0060] To further enhance explainability, penalties can be... L 1 Add to fully connected layer f 214, to make the weight matrix W The constraint is non-negative. Fully connected layer. f On 210 L 1 Sparsity penalties and nonnegativity constraints help learn sequence prototypes with more unified and additive semantics for classification.

[0061] To improve interpretability, clustering regularization terms can be used. R c and evidence regularization terms R e Clustering regularization term R c Clustering structure in the latent space can be encouraged by minimizing the squared distance between encoded instances and their nearest prototypes. (Clustering regularization term) R c This can be expressed as:

[0062] (8)

[0063] in, It is the training set The set of all sequences in the set. Evidence regularization term. R e Each prototype vector is encouraged to be as close as possible to the encoded instance, and this can be expressed as:

[0064] (9).

[0065] The complete loss function to be minimized can be expressed as:

[0066]

[0067] Where λ c , λ e , λ d , λ l1 These are hyperparameters that control the strength of regularization. The configuration of these hyperparameters depends on the nature of the data and can be selected through cross-validation.

[0068] Stochastic gradient descent (SGD) with mini-batch can be used to minimize the loss function on the training data. Due to the gradient... R e Since computation is required across the entire training set, this minimization can be relaxed to be computed only in each individual batch. Prototype projection techniques can be used to learn simple and interpretable prototypes. The optimization process can iteratively alternate between SGD and prototype projection steps.

[0069] Due to prototype vector p i These are representations in the latent space, and therefore cannot be readily interpreted. Some approaches can jointly train a decoder that transforms the latent space into the original input sequence space, thus making the prototypes interpretable. However, such a decoder may not necessarily decode the prototypes into meaningful sequences. The disclosed system incorporates a projection during training, which assigns the prototype vectors its nearest sequence embedding in the training set. This projection step can be expressed as:

[0070] (11)

[0071] Then, each prototype vector p i It can be associated with prototype sequences in the input space. The projection step can be performed only every few training epochs (e.g., four) to reduce computational cost. The projection step saves the computationally expensive effort of jointly training the sequence autoencoder. This ensures that each prototype is an observed sequence, which guarantees that the prototype is meaningful in the real world.

[0072] Predictions from the model can be easily explained by consulting the most similar prototypes. When making predictions based on new input sequences, explanations can be generated along with the inference process. Predictions can be explained by a weighted sum of the contributions of the most similar prototypes. For example, the system can generate predictions and explanations for inputs as follows:

[0073] Input: The pizza was good, but the service was extremely slow.

[0074] Prediction: Negative

[0075] Explanation: 0.69 * Good food but worst service (-2.1)

[0076] The service is incredibly slow (0.30* -1.1).

[0077] The numerical factors preceding the explanatory prototype sequence represent the similarity between the input sequence and the prototypes. At the end of each prototype are associated weights. These weights can be interpreted as the model's confidence in the possible labels of the prototypes.

[0078] Although the prototype is already readable after being projected onto the observed sequence in the training data, it may still be difficult to understand if the prototype sequence contains irrelevant or noisy events.

[0079] A process can be introduced to simplify the projected prototype sequence. Instead of projecting the prototype onto the complete observed sequence, it can be projected onto a subsequence containing key events. The modified projection steps can be expressed as:

[0080] (12)

[0081] (13)

[0082] in, It is data The set of all possible subsequences, Calculate the effective length of the subsequence. The complexity of the above operation is O(n log n). O（2 T N） ,in N It is the size of the training set, and T yes The maximum length of the sequence in the given information. The cost of brute-force computation varies with... T Exponential growth, and even for relatively short sequences, may be unacceptable.

[0083] To find an approximate solution, a beam search algorithm can be used, which only retains... w A greedy breadth-first search algorithm for the best candidate. w The value of is called the beamwidth. The algorithm can first select the prototype... p i of wThe algorithm then selects the closest candidate sequences. It can then generate all possible subsequences, which can be obtained by... w To obtain all possible subsequences, remove one event from any of the candidates. The modified projection step can be evaluated for each subsequence. Then, the subsequence with the minimum score can be selected. w Each subsequence is retained as a candidate to continue the search in the next iteration. The subsequence with the minimum score can be the output. The complexity of the algorithm is now O(n log n). O (wT) 2 N） In some examples, w It can be set to three.

[0084] Beam search algorithms that can be used for prototype simplification, such as Figure 9 As shown in the figure. BestCandidates(S,w) takes the set of sequences S, calculates a score for each sequence using equations (12) and (13), and returns the sequence with the lowest score. w The algorithm terminates when a subsequence cannot be reduced or when there is no better subsequence than an existing sequence in the candidate set S to remove one of them.

[0085] Figure 3 This describes how user 304 can improve the trained model 302 for better interpretability and performance by validating and updating prototype 308, especially when user 304 already has some expert opinion or knowledge in the problem domain. Allowing user 304 to validate and interact with prototype 308 also increases their understanding of the model and data, which is the basis of user trust. In this example, input sequence 306 is fed into trained model 302 to generate prediction 307. Trained model 302 includes prototype 308. Associated with each of prototypes 308 can be a predicted value 310. In this example, user 304 might want to eliminate prototype 308 that matches input command 312A. In this case, prototype 308B matches input command 312A and can be removed from model 302. User 304 might want to add prototypes based on input command 312B. The system can provide a user interface for interacting with user 304 to allow command input.

[0086] Suppose that a user's knowledge can be explicitly expressed in the form of input-output patterns that the user identifies as important or typical in the domain (e.g., "The food is good" is often a comment with a "positive" sentiment). These patterns can be viewed as "prototypes" that the user has learned from his / her past experiences. Therefore, this improvement can be achieved by incorporating user-specified prototypes as constraints into the model.

[0087] Based on the user's past knowledge and observations of the model's output, there are three possible operations the user can apply to the model: create a new prototype, modify an existing prototype, or delete an existing prototype. After submitting changes, the model can be fine-tuned on the training data to reflect those changes.

[0088] When fine-tuning the model, the prototype can be fixed to reflect the user's constraints. The optimization process can be modified as follows: Instead of updating the hidden prototype vector during the gradient descent step... p i Updated sequence encoder r It can be used to directly set in each iteration. p i = r(seq i ) The prototype projection step can be skipped. After fine-tuning, the sequence encoder... r This allows for better learning of data representation. Users can verify the updated results and repeat the process until they are satisfied with the outcome.

[0089] The described systems and methods can be used in a variety of applications. A first example could be a system for predictive diagnostics based on vehicle fault log data. Modern vehicles have complex interconnected modules, and the faults typically have a significant evolutionary history throughout the vehicle's lifespan. Therefore, fault logs collected from vehicles can be used to understand the typical evolutionary path of problems and support predictive diagnostics. The fault logs for each vehicle can be modeled as a sequence of events. Each event can correspond to one fault or multiple faults occurring simultaneously. Each fault can be described by a five-digit Diagnostic Incident Code (DTC), which is standard across different brands and models. The disclosed models and systems can be applied to predict the risk of future vehicle faults (e.g., DTCs) using historical DTC logs. Events can be encoded as multi-hot vectors because multiple faults may occur simultaneously. Therefore, the input at each step is a binary vector. x(t) ∈ {0,1} n Furthermore, each element in the vector indicates whether a specific fault has occurred. The prediction problem can be formulated as a multi-label classification task to predict the risk of different DTCs. In this example, the softmax layer can be replaced by a sigmoid layer to compute the output probability.

[0090] In one example, the system was applied to vehicle diagnostic data containing a total of 12K vehicle fault sequences, which comprised 393 different types of DTCs. A classifier was trained to predict the top 92 DTCs that had occurred more than 100 times in the dataset. The sequences had a mean length of 2.31. The dataset was split into 7.2K training, 2.4K validation, and 2.4K test sets. The system was trained using an LSTM encoder (1 layer, 50 hidden units) and 100 prototypes. The factors set during training were as follows: λ l1 =1.0, λ e =0.1, λ c =0.01, λ d =0.01, and d min =1.0. For prototype simplicity, the beamwidth is set to: w =3.

[0091] exist Figure 4 Example prediction 400 of the model is shown on the input fault log sequence. Input sequence 402 shows a cyclic sequence consisting of “P030X” and “P0420”, while the model predicts a relatively high risk (0.17) for “P2187”, which has not occurred before. “P2187” indicates a problem with the fuel system in the engine at row 1. Given the interpretation, we can see that there are three prototypes 404 that match different aspects of input sequence 402. All three prototypes 404 indicate a high risk for “P2187”, which explains the reasoning for this prediction. Mechanics can use this model to predict potential future problems and make predictions based on canonical cases. The entire set of prototypes in the model provides an overview of the entire fault development path, which can help manufacturers identify systemic problems and develop preventative maintenance strategies.

[0092] This system can also be applied to sentiment classification tasks on text data. For example, restaurant reviews from the Yelp Open Dataset can be used. Each review is tokenized into a sequence of words using the Natural Language Toolkit (NLTK). In this example, only reviews with fewer than 25 words in the experiment (a total of 106K reviews) are used for user research, indicating that shorter sentences are easier for humans to read and compare. Each review includes a star rating. The stars (one to five) given under the review can be used as labels, and experiments can be conducted on both fine-grained (5 classes) and binary (positive = rating ≥ 3) classifications. The dataset is split into 60% training, 20% validation, and 20% test sets. The resulting accuracy on both the validation and test sets is comparable to algorithms such as Bi-LSTM encoders, LSTM, Bi-LSTM, and ResNet. The LSTM has 2 layers with 100 hidden units per layer. The ResNet contains 7 residual blocks. A dropout rate of 0.8 is applied during training. The initial number of prototypes was set to 100 and 200 for binary and fine-grained classification tasks, respectively. Results indicate that the published model can learn interpretable representations while achieving similar performance to state-of-the-art bidirectional LSTMs.

[0093] For the sentiment classification task on Yelp Reviews, reviews can first be filtered to contain only "restaurant" reviews based on the "category" field of the reviewed business. Then, NLTK can be used to tokenize the review text into word sequences. For human evaluation purposes, reviews longer than 25 words can be filtered. Downsampling can be used to balance the classes for both binary and fine-grained classification. The size of the largest class should not exceed twice the size of the smallest class. The vocabulary size is 6287 and 6792. A word embedding of size 100 can be used for the entire model. This embedding can be trained jointly with the model.

[0094] Figure 5 illustrates the interpretations and neighboring prototypes generated from the example sentiment analysis results. The first example comment 502 led to a prediction and interpretation that resulted in a referenced prototype illustrating different aspects of a good restaurant—good food and service. The second example 503 and the third example 504 depict sequences of neighboring prototypes associated with a given prototype. Some prototypes represent frequent, short phrases that are emotionally significant. Some prototypes capture a long range of semantics, such as mood shifts via contrastive conjunctions (e.g., "but" in the second example 503). The analysis also uncovered some interesting sequential "patterns" of how people express their emotions. For example, a typical way to express positive emotions is through multiple short compliments ending with an exclamation mark, as in the third example 504. In this example, the input sequence and prototype sequence can be matched using a latent distance measure via LSTM instead of strict pattern matching. Similar patterns between prototypes and neighboring sequences are highlighted. Bold and capitalized text shows simplified prototype sequences.

[0095] The UniProtKB database can be used to apply this system in the field of biology. This database contains 558,898 manually annotated and commented protein sequences. Protein sequences consist of 20 standard amino acids and can be grouped into multiple families. Proteins within a family originate from a common ancestor and typically have similar functions and 3D structures. This system can be used to study sequence similarity within a family.

[0096] A maximum length of 512 sequences can be used to prune the sequence, and the model can be trained to classify the top 100 families ranked by family size. The sequences are split into a 58K training set and a 24K test set. The parameters are set to: λ l1 =1.0, λ e =0.1, λ c =0, λ d =0.01, and d min =1.0. The system comprises a Bi-LSTM (2 layers × 50 hidden units) and 200 prototypes. The batch size was set to 64, and the model was trained for 40 epochs. The model achieved accuracy comparable to other strategies. However, the system learned interpretable representations that revealed important subsequences of protein families. Figure 6 An example of the prototype is shown. Matching subsequences in the neighborhood are highlighted.

[0097] This system and method can be extended to find meaningful prototypes in real-valued time-series data using the MIT-BIH arrhythmia ECG dataset. ECG is widely used in medical practice to monitor heart health. Proper classification of waveforms is crucial for appropriate diagnosis and treatment. In the dataset, each signal consists of heartbeats annotated by at least two cardiologists. ECG signals can be downsampled to 125 Hz, and the signals are broken down into annotated heartbeats according to a predetermined protocol. The annotations can be mapped into five groups: normal (N), supraventricular ectopic beats (SVEB), ventricular ectopic beats (VEB), fusion beats (F), and unknown beats (Q). The training and test sets contain 87K and 21K sequences, respectively.

[0098] Instead of discretizing time-series data into event sequences, LSTM can be used directly to encode real-valued sequences. Parameters can be set to: λ l1 =0.1 , λ e =1.0 , λ d =0.01 , d min =2.0 The dropout rate is as low as 0.1. The network can be trained using a Bi-LSTM encoder (32 hidden units × 3 layers) and 30 prototypes. Training can run for up to 36 epochs with a batch size of 128, and no prototype simplification is applied. Prototypes with small weights are removed... max(w i ) <0.1 max(W) After that, a model with 23 prototypes was obtained.

[0099] exist Figure 7 Several selected prototypes are shown. The disclosed system successfully learned several prototypes for each class. Prototype 12 (Pt 12) shows a characteristic junctional escape beat (belonging to the SVEB group), which is characterized by a long, flat line corresponding to a dropped beat. Prototype 17 (Pt 17) shows a premature ventricular contraction (PVC) with a strong contraction and a long pause between ventricular contractions. This demonstrates the disclosed system's ability to learn meaningful representations of ECG timing data, a capability that has been validated by two independent cardiologists.

[0100] The disclosed system and model achieve comparable results when compared to state-of-the-art models used for ECG heart rate classification. The disclosed model can present a verifiable and understandable prototype that is extremely useful in the healthcare field. In fact, the most similar prototype can be presented side-by-side with automatically annotated ECG signals for interpretation.

[0101] When implementing the disclosed model, the number of prototypes to choose from. k This is necessary. The impact of the number of prototypes on system performance can be analyzed using the UniProtKB and Yelp Review data discussed earlier in this paper, thus investigating the appropriate choice. Using the same hyperparameter configuration described previously, a range of models can be trained using different k values. As a result, the initial accuracy increases with... k Increased and greatly improved. Regarding UniProtKB... k More than 100 and for Yelp reviews k After 40, the increasing slope flattens out rapidly.

[0102] With the number of prototypes k Increased accuracy leads to improved performance. However, it can become more difficult to understand and distinguish such a large number of prototypes. Therefore, there is a trade-off between accuracy and interpretability. In practice, due to increases after a certain threshold... k It only brings negligible performance improvements, so one possible strategy is to start small. k Begin (for example, k=C (Assuming one prototype for each class), and gradually increasing... k Until performance improvements fall below a certain threshold.

[0103] To study the impact of the diversity regularization term, we can set... λ d = 0 to remove the item, and another set of experiments can be performed using the varying number of prototypes. It was observed that for k Different settings, without R d In cases where UniProtKB performs poorly, especially for larger... k Value, for R d The inclusion of [the data] also positively impacts the performance of Yelp Review.

[0104] A heatmap of two networks with 100 prototypes can be generated by plotting the similarity score between prototypes and test sequences on Yelp Review data. Figure 8 ), to conduct further examination R dThe impact of diversity regularization. λ d In the case of =0), most rows are in the heatmap ( Figure 8 The patterns in the diagrams are similar horizontally, indicating approximately repeating prototypes. λ d With a similarity of 0.01, the similarity heatmap is much sparser and more diagonal, which shows that the prototypes are more diverse and evenly distributed in the latent space.

[0105] You can examine the impact of prototype simplification on performance and subsequence length. This can be done using UniProt and YelpReview. λ d = 0.01 as previously set. The results indicate no significant difference in accuracy between the two datasets. However, with simplification applied, the average prototype (sub)sequence length decreased from 20.1 to 15.1 on the Yelp Review dataset and from 274.5 to 130.7 on the UniProtKB dataset.

[0106] The interpretability of machine learning models is a subjective concept and is often considered difficult to assess computationally. This paper aims to quantitatively evaluate the interpretability of networks through experiments on human subjects. In the case of a prototype learning architecture, the desired answers are: 1) How understandable and accurate is the prototype in interpreting predictions of the input sequence? 2) How will incorporating human knowledge affect the model's performance and interpretability? The publicly available model for binary sentiment classification, trained on Yelp Review, can be used for evaluation. This model has 80 valid prototypes (i.e., associated weights). max(w i ) >0.1 max(W) ).

[0107] To assess the interpretability of the explanations, human participants who are non-experts in machine learning can be recruited on Amazon Mechanical Turk. Directly asking whether an explanation is interpretable or accurate is highly subjective and varies from person to person. Therefore, a relative measure is employed by asking participants to choose one of three prototype statements that express the most similar emotion to a given input statement. The prototype with the highest similarity score to the input statement in the model is considered the suggested answer and presented as one of the options. Other options are randomly selected from the remaining prototypes. The selection also includes "None of the above" as a fourth option. The input statements are selected from the validation set using stratified sampling. That is, sequences are grouped according to their most similar prototypes, and these groups are used as strata for sampling.

[0108] A total of 70 questions were sampled, and 20 responses were collected from human subjects for each question. After filtering out answers that failed quality checks (e.g., consistency checks of answers to duplicate questions), each question had an average of 12.5 valid responses. The option with the most votes from human subjects was used as the correct answer for each question and was used to calculate the accuracy of both the human and model responses. After updating the model based on the interactions, the model typically showed improved accuracy.

[0109] The 70 questions can be evenly distributed across four questionnaires (e.g., each with 17 and 18 questions) to prevent participants from becoming overwhelmed. Three additional quality check questions can be added (e.g., repeated questions with options in different orders, or questions with clearly correct answers). Responses that fail more than one quality check question can be filtered out. The correct answer for each question can be calculated using the option with the most votes. Responses with less than 50% accuracy can be filtered out. Human accuracy and model accuracy can be calculated using the option with the most votes as the correct answer.

[0110] To investigate how input of human knowledge affects the interpretability of the model, feedback from user research can be used as a source of human knowledge to update the model, and a second round of experiments can be conducted on MTurk. Based on the results of the first round of user experiments, the model can be updated to improve the quality of the prototypes. The update protocol could be as follows: For each question with an incorrect answer, examine the prototype sequence suggested by the model as an answer and its neighboring sequences in the validation set. If the neighboring sequences do not have consistent sentiment (in the case of subjective judgment), the prototype can be deleted. If the neighboring sequences have consistent sentiment, but the provided prototype is not representative enough (e.g., parts of the statement have misleading implications), a new statement is selected from the neighboring statements to replace the old prototype.

[0111] Following the aforementioned protocol, 13 prototypes were updated and 5 were removed. After incremental training, the model's performance on the test set remained essentially unchanged (a slight increase of 0.1%). A second user experiment was then conducted using the same procedure. An average of 12.3 valid responses were collected for each question. The accuracy of the model's suggested answers increased significantly. The results demonstrate that while incorporating human knowledge via the proposed interaction scheme may not necessarily increase model performance, it can be very helpful in improving the model's interpretability.

[0112] A novel, interpretable, and manipulable deep sequence modeling technique is disclosed. This technique combines prototype learning and RNNs to achieve both interpretability and high accuracy. Experiments and case studies on various real-world sequence prediction / classification tasks demonstrate that the model is not only as accurate as other state-of-the-art machine learning techniques but also significantly more interpretable. Furthermore, large-scale user studies on Amazon Mechanical Turk demonstrate that for familiar domains, such as sentiment analysis of text, the model is able to select high-quality prototypes that well match human knowledge for both prediction and interpretation. Moreover, the model achieves better interpretability without performance loss by updating the prototypes by incorporating feedback from user studies, thus demonstrating the benefits of involving human-machine loops for interpretable machine learning.

[0113] The processes, methods, or algorithms disclosed herein may be deliverable to or implemented by a processing device, controller, or computer, which may include any existing programmable electronic control unit or dedicated electronic control unit. Similarly, the process, method, or algorithm may be stored in many forms as data and instructions executable by a controller or computer, including but not limited to: information permanently stored on a non-writable storage medium (such as a ROM device) and information reproducibly stored on a writable storage medium (such as a floppy disk, magnetic tape, CD, RAM device, and other magnetic and optical media). The process, method, or algorithm may also be implemented in a software executable object. Alternatively, the process, method, or algorithm may be embodied, in whole or in part, using suitable hardware components such as: application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0114] Although exemplary embodiments have been described above, they are not intended to describe all possible forms covered by the claims. The language used in this specification is descriptive and not restrictive, and it should be understood that various changes may be made without departing from the spirit and scope of this disclosure. As previously described, features of various embodiments may be combined to form other embodiments of the invention, which may not be explicitly described or illustrated. While various embodiments may have been described as offering advantages or being preferred over other embodiments or prior art implementations with respect to one or more desired characteristics, those skilled in the art will recognize that trade-offs can be made to one or more features or characteristics to achieve desired overall system properties, depending on the particular application and implementation. These properties may include, but are not limited to: cost, strength, durability, lifecycle cost, marketability, appearance, packaging, size, maintainability, weight, manufacturability, ease of assembly, etc. Accordingly, embodiments described as less desirable than other embodiments or prior art implementations with respect to one or more characteristics are not outside the scope of this disclosure and may be desirable for a particular application.

Claims

1. A method for implementing a prototype sequence machine learning network, comprising: A sequence encoder is used to map one or more labeled sequence datasets to generate a fixed-length embedding vector. The labeled sequence datasets include one of the following: vehicle fault log data, customer reviews, protein sequences, and electrocardiogram datasets. Determine the score between the embedded vector and one or more prototype vectors in the corresponding problem domain to generate one or more similarity vectors; as well as A fully connected layer is used to classify one or more predicted values, wherein the fully connected layer applies a weight matrix to the one or more similarity vectors; The diversity regularization value is applied to the one or more prototype vectors to penalize at least the first of the one or more prototype vectors, which is similar to the second of the one or more prototype vectors.

2. The method according to claim 1, further comprising: A softmax layer is used to calculate the predicted probability for the one or more labeled sequence datasets, wherein the softmax layer divides the exponent of the one or more predicted values ​​by the sum of the one or more predicted values.

3. The method of claim 1, wherein when the sequence embedding of the embedding vector is not substantially equal to the one or more prototype vectors, a value of zero is assigned to the score.

4. The method of claim 1, wherein the score is assigned a value of one when the sequence embedding of the embedding vector is substantially equal to the one or more prototype vectors.

5. The method of claim 1, wherein the fully connected layer includes a sparsity penalty and a non-negativity constraint, the non-negativity constraint constraining the weight matrix to one or more positive values, wherein the sparsity penalty and the non-negativity constraint improve the learning process of the prototype sequence machine learning network, such that the one or more prototype vectors have more unified semantics and additive semantics.

6. The method of claim 1, wherein a clustering regularization function is applied to the one or more labeled sequence datasets and the one or more prototype vectors, wherein the clustering regularization function ensures clustering structure in the latent space.

7. The method of claim 1, wherein an evidence regularization function is applied to ensure that the one or more prototype vectors are approximately equal to the one or more labeled sequence datasets.

8. The method of claim 1, further comprising: Assign sequence embedding vectors provided from the training dataset to the one or more prototype vectors, wherein the sequence embedding vectors are approximately equal to the one or more prototype vectors.

9. The method of claim 8, wherein the step of allocating the one or more prototype vectors occurs at a predetermined period.

10. The method of claim 1, further comprising: Project the one or more prototype vectors onto a subsequence of events within the training dataset.

11. The method of claim 10, wherein a beam search algorithm is implemented to select candidates determined to be substantially equal to the one or more prototype vectors.

12. The method of claim 1, further comprising: At least one of the one or more prototype vectors is deleted during the training process.

13. The method of claim 1, further comprising: At least one of the one or more prototype vectors is modified during the training process.

14. The method of claim 1, wherein the sequence encoder is a long short-term memory network.

15. The method of claim 1, wherein the sequence encoder is a bidirectional long short-term memory network.

16. The method of claim 1, wherein the sequence encoder is a gated loop unit network.

17. A system for implementing prototype sequence machine learning networks, comprising: A processor that is operable in: A sequence encoder is used to map one or more labeled sequence datasets to generate a fixed-length embedding vector. The labeled sequence datasets include one of the following: vehicle fault log data, customer reviews, protein sequences, and electrocardiogram datasets. Determine the score between the embedded vector and one or more prototype vectors in the corresponding problem domain to generate one or more similarity vectors; as well as A fully connected layer is used to classify one or more predicted values, wherein the fully connected layer applies a weight matrix to the one or more similarity vectors; The diversity regularization value is applied to the one or more prototype vectors to penalize at least the first of the one or more prototype vectors, which is similar to the second of the one or more prototype vectors.

18. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 16.

Images