Machine learning model and trained autoencoder for machine learning model analysis

Abstract

The disclosure provides a machine learning model and a trained autoencoder for machine learning model analysis. A method of training an autoencoder (202) for analyzing a machine learning model (102), an autoencoder (202) for analyzing a machine learning model (102), a system and a computer program product, the method comprising: extracting output data from an embedded layer of a machine learning model (102) operating on data within a sample, a dimension reduction is performed on at least a portion of the extracted output data from the embedded layer to obtain first dimension-reduced data (302), and an autoencoder (202) is trained to generate corresponding intermediate dimension-reduced data (304) at a bottleneck (206) of the autoencoder from the at least a portion of the extracted output data.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to methods of analyzing machine learning models, and in particular to training autoencoders to analyze embedding layers of machine learning models. Machine learning models are commonly used in advanced driver assistance systems (ADAS). BACKGROUND

[0002] Advanced driver assistance systems use machine learning (ML) models to significantly improve vehicle safety and driving expectations. Vehicles incorporating ADAS can utilize sensor data, such as camera or radar data, to analyze and determine vehicle driving situations and take appropriate actions or warn the operator of the vehicle of appropriate actions to take. The analysis and determination of sensor data is performed using a machine learning model that has been trained on a dataset of labeled sensor data corresponding to driving situations. The machine learning model uses an artificial neural network to learn the connection between data in the training dataset and driving situations.

[0003] Machine learning models address complex challenges in the areas of lane detection, collision avoidance, adaptive cruise control, pedestrian recognition, and the like. In each driving situation, the ML model must take into account multiple factors, such as the speed of the host vehicle, the speed of other vehicles, the distance to other vehicles, road markings, and the like. Each of these factors can have multiple data attributes both for identifying the factor and for describing the factor.

[0004] In training, the neural network establishes connections between the various factors and data attributes related to individual driving situations to generate a complex ML model that can accurately determine driving situations from input data.

[0005] However, when ML models are treated as black boxes, where the learned principles are unknown or poorly understood, this presents a significant challenge to ML developers. This lack of transparency becomes particularly problematic when the goal of the developer is to improve model performance, extend model functionality, and optimize datasets.

[0006] Accordingly, it is beneficial to analyze machine learning models at the hidden layers of the ML model to understand what is happening deep within the model. Due to the complexity of ML models, the data extracted from the hidden layers can have hundreds or even thousands of dimensions, making it impossible for the human mind to conceptualize the data. Accordingly, visualization of the data extracted from the hidden layers can be performed using conventional dimensionality reduction techniques. However, the useful output of some more powerful conventional dimensionality reduction techniques, such as T-SNE UMAP or LargeVis, is limited to a fixed dataset that is input to the ML model, which can be referred to as in-sample data, and cannot be generalized to out-of-sample data. Thus, in order to evaluate new data, it is necessary to run a new analysis, which can be time-consuming and computationally expensive.

[0007] Accordingly, it is beneficial to derive different methods of visualizing the inner workings of a machine learning model, which can guide developers in utilizing the full power of ML models in the real world. SUMMARY

[0008] According to a first aspect of the application, there is provided a method of training an autoencoder for analyzing a machine learning model, the method comprising the steps of: extracting output data from an embedding layer of the machine learning model operating on in-sample data; performing dimensionality reduction on at least a portion of the extracted output data from the embedding layer to obtain first reduced dimensionality data; and training the autoencoder to generate corresponding intermediate reduced dimensionality data at a bottleneck of the autoencoder from the at least a portion of the extracted output data.

[0009] In this way, out-of-sample data, i.e. test data encountered by the ML model in operation, can be visualized in a format that is typically only available for in-sample data training data. The original dimensionality reduction can be performed using conventional methods. When the ML model operates on labeled out-of-sample data, i.e. test data encountered by the ML model in operation but not used in the conventional dimensionality reduction methods, the trained autoencoder can facilitate visualization of the data extracted from the embedding layer. By training the autoencoder to reduce the output data from the embedding layer to a format modeled based on classical dimensionality reduction processes, such as t-SNE, when the ML model operates on out-of-sample data, the resulting data can be analyzed to obtain information about the machine learning model. This information can be used to determine how to adjust the dataset to fine-tune the model.

[0010] In an embodiment, the method further comprises: generating a custom loss function for the autoencoder, the custom loss function comprising a concatenation of: an error between the bottleneck of the autoencoder and the first reduced dimensionality data, and a reconstruction loss of the autoencoder.

[0011] In this way, the autoencoder is taught to reconstruct the original data using a specific form of bottleneck, where the bottleneck output itself corresponds to the first reduced dimensionality data. This means that the encoded representation at the bottleneck can be analyzed similarly to the reduced dimensionality data of prior art projections, and the efficiency of this bottleneck output data in preserving relationships between data points is evidenced by the low reconstruction loss.

[0012] Preferably, the portion of the extracted output data can be normalized and centered before the autoencoder generates intermediate reduced dimensionality data from the at least a portion of the extracted output data.

[0013] Preferably, at least one layer of the encoder portion of the autoencoder, more preferably the first layer, may employ the tanh activation function. This can be used to ensure that each data point has a value centered at zero, between -1 and 1.

[0014] In one implementation, the method further includes: analyzing the machine learning model using a trained autoencoder by receiving at least a portion of the output data from the embedding layer of a machine learning model operating on unseen data at a trained autoencoder, and by generating second dimensionality-reduced data by the autoencoder at a bottleneck of the autoencoder.

[0015] This allows analysis of data processed by the ML model at the embedding layer, even if that data is out-of-sample. Additionally, before the autoencoder generates the second dimensionality-reduced data, the portion of the embedding layer from the ML model's output data, which operates on unseen data, can be normalized and centered.

[0016] In one implementation, the method further includes: using intermediate dimensionality reduction data generated by the autoencoder to train a regression model, and determining, based on a comparison between the regression model and the second dimensionality reduction data, whether at least a portion of the unseen data corresponds to a predetermined model loss of the machine learning model or greater.

[0017] In this way, real-world data can be evaluated by determining the model loss for dimensionality-reduced data created by a guided autoencoder. Regions of input data with high expected model loss can be identified and thus considered for future dataset creation, aiming to address existing model weaknesses. Therefore, model performance on out-of-sample data can be predicted.

[0018] In one implementation, the method further includes recording the unseen data corresponding to a predetermined model loss or greater than that of the machine learning model.

[0019] In this way, the recordings can be used to create datasets, and therefore datasets can be created as the vehicle interacts with its test environment without requiring significant effort. Furthermore, selectively recording sensor data during test driving reduces downstream data processing and data storage workload, while simplifying the dataset creation process.

[0020] In one implementation, the method further includes: compiling at least a portion of a training dataset for the machine learning model based on the recorded unseen data.

[0021] In this way, test data collected during test driving of the vehicle can be evaluated by determining the model loss against the dimensionality-reduced data created by the guided autoencoder. Areas where the model performs poorly can be identified, allowing for targeted model adjustments.

[0022] In an embodiment, the method further comprises performing model analysis on the second reduced dimension data to obtain characteristics of the machine learning model.

[0023] In this way, the internal working mechanism of the ML model can be explored. Biases in the available data can be identified, potential problems in the model architecture can be discovered, and the learning principles of the model can be visualized in relation to specific embedding layers of the model.

[0024] In an embodiment, obtaining characteristics of the machine learning model comprises labelling data points within the second reduced dimension data with associated metadata to derive principles learned by the machine learning model.

[0025] In this way, the visualization of the characteristics of the model can be made simpler. By labelling data points with e.g. colors based on available metadata, i.e. the speed of the vehicle, the number of recognized objects, and the environmental context, the learning structure and principles of the ML model can be easily identified and conceptualized in a visualized manner.

[0026] In an embodiment, the step of obtaining characteristics of the machine learning model comprises identifying substantially similar activations in embedding layers of the machine learning model.

[0027] In this way, scenarios encountered by the ML model can be clustered. For a specific scenario, similar scenarios leading to similar activations in the embedding layers of the ML model can be found and returned, such that the clustering of scenarios delineated by the ML model can be visualized for a specific embedding layer.

[0028] According to a second aspect of the present invention, there is provided an autoencoder for analyzing a machine learning model, wherein the autoencoder is trained by: extracting output data from an embedding layer of the machine learning model operating on in-dataset data; performing dimension reduction on at least a portion of the extracted output data to obtain first reduced dimension data; and training the autoencoder to generate corresponding intermediate reduced dimension data at a bottleneck of the autoencoder from the at least a portion of the extracted output data.

[0029] In an embodiment, the autoencoder is configured to receive at least a portion of sensor data from a sensor as input and generate second reduced dimension data at a bottleneck of the autoencoder.

[0030] In this way, data encountered by the vehicle in the test and run through the ML model can be visualized in a format that is typically only available for in-dataset data. Thus, the working of the ML model in operation can be visualized.

[0031] According to a third aspect of the application, there is provided a machine learning model system for operating on sensor data relating to a vehicle during driving, the system comprising:

[0032] a machine learning model configured to generate at least one prediction of a vehicle state based on the sensor data, preferably in the context of a driving related scenario; and

[0033] one or more autoencoders as described above, and the autoencoders are configured to generate reduced dimensionality data from extracted output data from an embedding layer of the machine learning model corresponding to at least a portion of the sensor data.

[0034] In this way, the ML model can be combined with the autoencoders into a single architecture that outputs both predictions for the vehicle and reduced dimensionality data from one or more embedding layers of the model. This provides the data required for the ML model to analyze online in an efficient manner.

[0035] In embodiments, the system further comprises a regression model trained to predict the model loss of the machine learning model in relation to the at least a portion of the sensor data.

[0036] In this way, model performance on out-of-sample data can be inferred. By concatenating the input of each of the one or more predictions into a regression model trained on the ML model development dataset (in-sample data), this prediction can be generated for out-of-sample data for all extracted out-of-sample embedding layers. Furthermore, the regression model can be part of the same ML model system as the ML model and autoencoders. In this way, a model can be generated that outputs both predictions for the vehicle and loss predictions for the ML model based on reduced dimensionality data of the embedding layers of the ML model run on out-of-sample data. Thus, the architecture of the system is simplified.

[0037] In embodiments, the sensor data comprises at least one of camera, lidar, radar, speed, acceleration, and yaw sensor data.

[0038] In this way, the environment of the vehicle and the features of the vehicle interacting with the environment can be visualized mapped for interpretation by the machine learning model.

[0039] According to a fourth aspect of the application, there is provided a non-transitory computer readable medium comprising computer program instructions which, when executed by a processor, cause the processor to perform any of the preceding methods. BRIEF DESCRIPTION OF DRAWINGS

[0040] The illustrative embodiments will now be described with reference to the accompanying drawings, of which:

[0041] Figure 1 An example data processing workflow and ML model analysis application according to an aspect of the application is shown;

[0042] Figure 2 A guided autoencoder architecture according to an aspect of the application is shown;

[0043] Figure 3a A first projection generated by a dimensionality reduction method according to the prior art is shown;

[0044] Figure 3b A second projection generated by a dimensionality reduction method according to an aspect of the application is shown;

[0045] Figure 4 Multiple projections according to an aspect of the application are shown;

[0046] Figure 5 Multiple driving scenarios encountered by a model according to an aspect of the application are shown;

[0047] Figure 6a A third projection generated by a dimensionality reduction method according to the prior art is shown;

[0048] Figure 6b A fourth projection generated by a dimensionality reduction method according to an aspect of the application is shown;

[0049] Figure 7 Regression model performance evaluation according to an aspect of the application is shown;

[0050] Figure 8 Representation of data naming according to an aspect of the application is shown;

[0051] Figure 9a Gray scale representation of Figure 3a is shown;

[0052] Figure 9b Gray scale representation of Figure 3b is shown;

[0053] Figure 10 Gray scale representation of Figure 4 is shown;

[0054] Figure 11 Gray scale representation of Figure 5 is shown;

[0055] Figure 12a Gray scale representation of Figure 6a is shown;

[0056] Figure 12b Gray scale representation of Figure 6b is shown. DETAILED DESCRIPTION

[0057] In Figure 1 In the example, an example machine learning model data processing workflow and a collection of model analysis applications are shown. The machine learning model is shown as ML model 102. The ML model 102 is trained to generate predictions 104 for movement or operation of a vehicle in a driving related scenario based on input data. To train the ML model 102, a training and development dataset 106 is used. Based on the dataset, the model 102 can be trained to perform the predictions 104.

[0058] As described above, the ML model 102 takes a plurality of input data points from internal and external vehicle sensor data and generates predictions 104. How the ML model 102 generates the predictions is not actively encoded into the ML model 102, but rather the model 102 learns how to generate appropriate predictions based on the training dataset 106 by passing the dataset through a neural network. The large number of weights applied to the links of the neural network that form part of the complex internal workings of the ML model are not easily interpretable or understandable by humans, and thus the way the ML model 102 processes data in the neural network is not easily analyzed and the ML model 102 can be considered a “black box”. The ML model 102 is composed of a plurality of hidden layers, where the ML model 102 generates associations between data points by adjusting its plurality of parameters during training to minimize a loss and by doing so, capture patterns that enable prediction of outcomes.

[0059] Certain hidden layers of the ML model 102 are considered embedding layers, which can be selected by a developer for analysis. Embedding layers in a machine learning model are used to map high-dimensional inputs to a lower-dimensional continuous vector space, where the outputs (embeddings) represent the inputs in a way that preserves contextual relationships. The input data to the model can include very high-dimensional data. The dimensionality of the data can increase or decrease across all layers of the ML model. In seeking to understand how the ML model operates, it is useful to analyze the output of one or more embedding layers. The output of the hidden and embedding layers can be very high-dimensional data that cannot be visualized or interpreted by a human.

[0060] Thus, in step 108, the output of at least one embedding layer of interest is extracted from machine learning model 102. However, the output of any given embedding layer can still include high-dimensional data that is difficult or impossible for human thought to conceptualize, understand, and evaluate. Dimensionality reduction can make data easier to visualize or otherwise conceptualize by representing it in two or three dimensions. In step 110, dimensionality reduction processing is performed on at least a portion of the extracted output data from the embedding layer. This processing can be t-SNE, UMAP, LargeVis, or any other suitable dimensionality reduction technique. Dimensionality reduction aims to make high-dimensional data easier to understand by mapping it to a low-dimensional space. The result of the dimensionality reduction processing can be projected as data points onto, for example, a two-dimensional image, which may or may not be color-coded. In this way, the dimensionality reduction processing can generate an image that shows the projection of the data at the embedding layer as corresponding points in the image, and thus allows visualization of the data obtained from the embedding layer. The main limitation of these dimensionality reduction techniques is that they are trained on a fixed dataset and generally cannot be generalized to out-of-sample data points without additional effort. Furthermore, they can be computationally expensive, slowing down the analysis and making it difficult or impossible to perform in real time in a commercial setting.

[0061] Therefore, it will be noted that the ML model 102 has been trained using the development / training dataset 106 at this point. Conventional dimensionality reduction is particularly effective when applied to in-sample or labeled data. This is because conventional dimensionality reduction (such as t-SNE) is analogous to a physical simulation operation and does not learn a generalizable mapping function from the input space to the embedding space, and therefore only applies to the data input to the process. Therefore, the relevance learned through t-SNE cannot be extrapolated to other data. Thus, it is possible to expand the in-sample data and rerun t-SNE dimensionality reduction on the new in-sample data; however, this is time-consuming and resource-intensive. The trained autoencoder discussed below addresses this problem.

[0062] Therefore, while the internal workings of ML models can be visualized using the dimensionality reduction techniques described above, not all data encountered by ML models is in-sample data, and ML models will interpret this data differently than in-sample data. Therefore, it is important to understand how ML models interpret and process out-of-sample data.

[0063] Therefore, instead of using the dimensionality reduction projection of the embedding layer to analyze the ML model, the result of the dimensionality reduction step is used to train an autoencoder to perform dimensionality reduction on the data from the embedding layer of the ML model in step 112, producing an output corresponding to the dimensionality reduction projection generated in step 110. The output data of the autoencoder can be projected as points in the image in the same manner as the output of the dimensionality reduction step. In the following description and claims, the output of the dimensionality reduction step and / or the autoencoder may be referred to as a projection. It should be understood that the term "projection" in the following description and claims includes data from the dimensionality reduction process at step 110 or the output of the trained autoencoder at step 112. Unless otherwise specifically stated, this "projection" is not required to generate the projected image.

[0064] Step 114 illustrates the predictive inference step, which can be performed on a dimensionality-reduced projection of the data from the embedding layer of the ML model, generated by an autoencoder, while the ML model is operating on out-of-sample data. Step 116 illustrates the scene recognition step that can be performed using this projection, and Step 118 illustrates the model analysis and automatic dataset optimization steps that can be performed using this projection. These steps will be discussed in detail later.

[0065] For the preceding description, the dimensionality reduction technique and process will be described as t-SNE, but it should be understood that any suitable technique as described above can be used.

[0066] Figure 2 An exemplary representation of the t-SNE-guided autoencoder 202 described above is shown. The autoencoder 202 takes flattened, centered, and normalized data 210 output from the embedding layer as input. Flattening, normalization, and centering allow the data from the embedding layer to be properly represented in the autoencoder. However, it should be understood that this process is not necessary for some techniques.

[0067] The flattening process transforms the data into a single vector, which can then be compressed by the layers of encoder 204 of autoencoder 202. Preferably, the data input to the autoencoder is normalized so that it falls between -1 and 1 and is centered so that the average value is 0. The dimensionality of the data is reduced to the bottleneck 206 of autoencoder 204 during this compression process.

[0068] The purpose of the automatic encoder 202 is to generate at the target or bottleneck 206 the same as that at... Figure 1The output data / projection corresponding to the t-SNE output data / projection generated at step 110. Bottleneck 206 compresses the input data into, for example, a 2D representation; however, it should be understood that this representation can be 3D or higher dimensional for analysis by a machine rather than a human. Furthermore, when the data is projected to form a 2D or 3D image, one or more additional dimensions can be represented by color. The decoder 208 of the autoencoder is designed to reconstruct the data, meaning that the data representation at bottleneck 206 is sufficient to reconstruct the higher-dimensional data input to the autoencoder.

[0069] To achieve this, in a conventional autoencoder, the input data is compared with the output data to generate a loss function for the autoencoder 202, which decreases as the output data corresponds more closely to the input data. However, for the t-SNE-guided autoencoder 202, a custom loss function is generated. This custom loss function uses a cascade of the output 212 of the autoencoder 202 and the bottleneck 206 of the autoencoder 202, and compares it with... Figure 1 The t-SNE projection generated at step 110 is compared with the input data of the t-SNE-guided autoencoder 202.

[0070] This method generates a custom loss function 214, which is the sum of the reconstruction loss of the autoencoder (the difference between the input and output) and the mean squared error of the data points in the t-SNE projection and the data points in the 2D representation at the bottleneck 206 of the autoencoder. Therefore, the autoencoder 202 is trained using this custom loss function to generate a t-SNE-style projection at the bottleneck, representing the input data with fewer dimensions.

[0071] In other words, the autoencoder is trained to both reproduce the dataset input as in a classic autoencoder and to do so via a bottleneck (which produces a t-SNE-like projection of the input data). Once the autoencoder is shown to reliably reproduce the input via the bottleneck at the decoder output, only the encoder portion is needed to provide the projection generated by the autoencoder.

[0072] At least one dense layer of the autoencoder, preferably the first layer, may employ the tanh activation function, which ensures that each data point has a value centered at zero in the range of -1 to 1.

[0073] Once the autoencoder is trained using a custom loss function, it can be applied to tasks that analyze the embedding layers of an ML model to generate t-SNE-like projections of out-of-sample data. Figure 3a The t-SNE projection 302 of the ML model, which operates on the in-sample data used for the exemplary embedding layer, is shown, corresponding to the output of the dimensionality reduction step 110. Figure 3bThe projection 304, similar to t-SNE, generated by the autoencoder on out-of-sample data from the same embedding layer of the ML model is shown.

[0074] exist Figure 3a and Figure 3b In the figure, each point represents a single sample input to the ML model. Each point is generated by reducing the dimensionality of the data from the selected embedding layer to standardized 2D data centered between -1 and 1. The background color of the figure represents the density estimate. It should be noted that the overall structure of the two figures is similar. A perfect in-sample to out-sample reconstruction is unnecessary for useful analysis of the embedding layer. Furthermore, if needed, the autoencoder trained using the above method can be tuned to close the loop. Figure 3a t-SNE represents the relationship with Figure 3b The gap between the projections generated by the autoencoder.

[0075] Figure 3b Curve 304 represents the output of the selected embedding layer of the ML model working on out-of-sample data that the ML model encountered during testing but not in the training and development datasets (i.e., data not used to generate the t-SNE projection). As mentioned above, this data is not easily visualized using conventional t-SNE, UMAP, or LargeVis dimensionality reduction, or other conventional dimensionality reduction techniques. Because this data has been visualized using an autoencoder guided by t-SNE, information about how the ML model interprets and interacts with the test data can be obtained.

[0076] Figure 3b The diagram illustrates the clustering of data points and the sparse regions between them. Clustering indicates that some relationships between data points are stronger than others. Therefore, clustering can represent how an ML model processes data at the embedding layer and can be used to analyze the impact of certain attributes of data points on the overall processing of the ML model.

[0077] Return to reference Figure 1 Step 114, based on the autoencoder projection, allows the predictive inference process to be applied to the data. This will refer to... Figure 4 Describe it.

[0078] Figure 4The diagram shows projections of the same embedding layer using the same data generated by an autoencoder guided by t-SNE. In each projection, the data has been color-coded based on different metadata values. For example, projection 402 shows data color-coded by velocity. It can be seen that some clusters have different colors than others. This indicates that different velocities contribute to different regions in the projection derived by the autoencoder from the embedding layer's output. Therefore, the ML model assigns some weight to the vehicle's velocity, at least with respect to the chosen embedding layer, because the vehicle's velocity is highly correlated with the associations made between data points by the ML model.

[0079] Conversely, projection 408 is color-coded based on the vehicle's longitudinal acceleration. It can be seen that the entire projection is approximately the same color, and no particular cluster has a color different from any other cluster. This indicates that, at the stage of the embedding layer in question, the ML model does not assign any specific value to the vehicle's longitudinal acceleration because it is unrelated to any particular set of associations made by the ML model.

[0080] To further illustrate the versatility of this method, projection 404 shows data color-coded based on vehicle yaw rate, and correlations are seen in at least some clusters (some clusters have different colors from the others), indicating that yaw rate is a property that the ML model assigns some importance to. Projections 410 and 414 show variations related to clustering, indicating that the ML model has some significance for both the object at 30 meters and the car at 30 meters. Furthermore, it can be seen that the colors of the data points in these two projections are substantially similar, suggesting that the car is somewhat mixed with all objects in the ML model. Projection 412 shows a similar but less intense coloring, again demonstrating the correlation between pedestrians and objects in the ML model's processing.

[0081] Projections 406 and 416 show very little color correlation with clustering, indicating that the ML model does not significantly increase the weights for vehicle lateral acceleration or trucks within 30m at the embedding layer in question. Interestingly, while the color variation of trucks in projection 416 is small (the number of trucks within 30m is not considered correlated), there is significant color variation in cars in projection 414 (indicating that different numbers of nearby cars lead to significant variations in the output of the embedding layer). It should be understood that the specific associations described in this paper are specific to the ML model and the training data and will vary from model to model.

[0082] based on Figure 4 The projection in the model visualizes the principles and structure of model learning. This allows for the identification of biases in the available data, the discovery of potential problems in the model architecture, such as information bottlenecks, and the association and localization of the learning principles to specific embedding layers.

[0083] Return to reference Figure 1 Step 116, based on the autoencoder projection, can perform scene recognition on the projection. This will refer to... Figure 5 Describe it.

[0084] Figure 5 Three scenarios encountered by the ML model in the form of radar data are illustrated. The original query is shown as radar diagram 502, and the two nearest neighbors in the projection generated by the autoencoder are shown as radar diagrams 504 and 506. That is, corresponding to respectively Figure 5 The three individual points of the three scenes shown are all closely grouped together in the projection generated by the autoencoder.

[0085] These images appear distinctly different to the human eye, yet each scene still results in similar activations in the embedding layer of the ML model. These scenes all share similar vehicle speeds and the number of objects 30 meters away. Conversely, the vehicle's yaw rate varies across different scenes. Therefore, this provides insight into how the ML model groups data points within the embedding.

[0086] More generally, scenes encountered by the ML model can be clustered by utilizing distance measurements in the embedding space. For a specific scene, other similar scenes that lead to similar activations in the ML model can be found. This allows scene clustering to be visualized as the model sees or groups scenes within a specific embedding layer. Similar scenes can be quickly retrieved using approximate nearest neighbor lookups.

[0087] Therefore, identifying scenarios in this way allows ML developers to better understand the inner workings of their models.

[0088] Return to reference Figure 1 Step 116, based on the autoencoder projection, involves performing model analysis and automatic dataset optimization on the projection generated by the autoencoder. This will refer to... Figure 6a and Figure 6b Describe it.

[0089] Similar to the predictive inference methods discussed above, the projections generated by autoencoders can be color-coded based on model loss. Figure 6b The projection color encoded by the model loss is shown. Projection 604 shows that some clusters shaded in yellow have high model loss, meaning that the ML model has difficulty reliably interpreting data points in these regions of the embedding layer.

[0090] This color code can be compared with... Figure 4 The projections are compared. For example, Figure 6b The right middle cluster 605 in the diagram shows a high model loss (yellow), and... Figure 4In projection 402, the same cluster corresponds to the low-speed value (dark purple). This analysis shows that the ML model has a high loss when dealing with high speeds.

[0091] Figure 6a Projection 602 shows the t-SNE projection of the embedding layer, color-coded by the model loss. It should be understood that the general region of the model loss grouped by clustering essentially corresponds to projection 604 generated by the autoencoder. This demonstrates that this visualization of the model loss is transformed from the t-SNE projection to the autoencoder-generated projection, thus allowing regression models to be trained to predict the model loss on new data. This can help generate datasets for training or fine-tuning machine learning models.

[0092] The original development / training dataset of the ML model can be used to train the regression model itself. The regression model can then take the projections generated by the autoencoder for each extracted embedding layer as input and concatenate them into a multidimensional vector (e.g., 3 embedding layers in 2D will result in a 6D input vector). Given the resulting regression model, the model performance predictions on new data encountered by the ML model can be used to customize new datasets to fine-tune the original model.

[0093] Figure 7 The regression model curve 702 is shown, where data points expected to be well handled by the ML model follow a linear relationship between the true values ​​and predictions, as shown by line 703. This illustration demonstrates the strong predictive power of the regression model for unseen data. Any data points provided to the regression model that do not fall on line 703 can be considered as data points or scenarios where the ML model would struggle to correctly predict vehicle movement or operation based on available data. These data points can drive the generation of new datasets to overcome this particular area of ​​loss.

[0094] By cascading the predictions from one or more forecasts into a regression model trained on an ML model development dataset (in-sample test data), the prediction can be generated against out-of-sample data from all extracted out-of-sample embedding layers. This information can be used to determine how to adjust the dataset to fine-tune the model based on data encountered by the model in the real world. In particular, it can be targeted at... Figure 7 Each individual data point determines whether the predicted model loss is equal to or greater than a predetermined threshold. If so, the corresponding input data can be identified as particularly useful and included as part of a broader future training dataset to overcome the high model loss in that specific region of the model. The input data can be labeled to be included as input data in future training datasets.

[0095] The development of new datasets can be carried out through a pool-based active learning approach, where new data is labeled only when the model needs to be fine-tuned based on the output of the performance predictive regression model.

[0096] Figure 8 Exemplary data naming for the above discussion is illustrated. Specifically, 802 shows a set of annotated ML model data, including “seen data” 804 and “unseen data” 806. “Seen data” 804 is data used to train the ML model, and “unseen data” 806 is data not used to train the ML model but forms part of a broader development and testing data pool. Within the unseen data is data used to perform t-SNE or other conventional dimensionality reduction, referred to in this application as “in-sample” data. This includes data 808 used to train the guided autoencoder and data 810 not used for initial training of the guided autoencoder (but can be used as test data). For conventional dimensionality reduction processing, everything outside these two data types is considered “out-of-sample.” Therefore, this data cannot be interpreted using the t-SNE processing applied to in-sample data. Within the data not seen by the guided autoencoder during training, there exists data seen during training the regression model 812, and data 814 not seen by the regression model during training (but can be used as test data). Data 816, excluding the annotated ML model data 802, represents data encountered by the ML model during operation, such as data from sensors attached to the vehicle. Therefore, while the data outside the samples used for regular dimensionality reduction (i.e., all data except 808 and 810) cannot be visualized using conventional in-sample dimensionality reduction techniques, all data outside of data 808 (not seen by the guided autoencoder during training) can be visualized using a guided autoencoder simply by feeding the output of the embedding layer of the ML model running on that data into the autoencoder.

[0097] Figures 9a to 12b A grayscale version of the projection discussed above is shown. In these figures, the same representations are indicated by the same reference numerals. Figures 3a to 6b In cases where data is displayed using color encoding, the colors have been directly converted to... Figures 9a to 12b The corresponding grayscale component in the image. That is to say... Figures 9a to 12b Each grayscale value in the image corresponds to... Figures 3a to 6b Different colors in the text. For example, Figure 10 The darker clusters in projection 402 that correlate the vehicle's speed with the ML model show a high correlation between the model's perception of the vehicle's speed, and specific dark clusters are associated with the interpretation of low speed values. In the corresponding... Figure 12bIn the diagram, box 605 illustrates the high model loss in this cluster of the ML model, demonstrating that the ML model exhibits high model loss when handling low vehicle speeds. Typically, plots showing color values ​​at the violet end of the spectrum are represented in dark gray, while plots showing color values ​​at the yellow end of the spectrum are represented in light gray.

[0098] The preceding text referenced multiple extracted embedding layers and the data contained within (or output by) these layers. It should be understood that a t-SNE-guided autoencoder can be trained based on the t-SNE projection of a single embedding layer. Furthermore, it should be understood that in order to perform… Figure 1 Steps 114 to 118 allow the trained autoencoder to be applied to the data from a single embedding layer. It will also be understood that the t-SNE projection and the projection generated by the autoencoder can be applied to subsets of the data from each of the extracted embedding layers.

[0099] The above discussion pertains to ML models used for predicting vehicle movement and maneuvering, and to autoencoders used for visualizing the embedding layers of ML models. It should be understood that ML models can include autoencoders as part of the model; an autoencoder is a form of ML model. Furthermore, the above discussion involves regression models used to predict model loss, which are also different forms of ML models. Therefore, it should be understood that ML models can combine one or more of the autoencoder and regression models. That is, an ML model can provide one or more outputs as its outputs: vehicle prediction based on t-SNE-guided autoencoder projection, and model loss indicating the data in a specific dataset.

[0100] The above discussion concerns the interpretation of data by the ML model. In some implementations, the data relates to vehicle driving conditions. In one aspect, the data is part of an externally generated training / development dataset used to train the model. In other aspects, the data is generated from sensors on a vehicle that is running or otherwise collecting data from the ML model. In this respect, the dataset can be considered as new data analyzed using an autoencoder guided by t-SNE.

[0101] The sensor can be a camera, radar sensor, lidar sensor, vehicle sensor (such as a speedometer, accelerometer, yaw sensor, etc.), or any combination thereof. In response to the determination by the regression model that a particular situation or aspect of a situation corresponds to a region of high model loss in the ML model (e.g., where the predicted model loss is above a predetermined threshold), data from the sensor can be recorded to provide a further training dataset for the ML model and labeled as a scene of interest for the creation of a downstream dataset used for fine-tuning the ML model.

[0102] The recorded data can be labeled with metadata by a human operator or by another ML model to provide labeled training data to the ML model, which is designed to fine-tune the high-loss regions of the ML model.

[0103] The ML model can operate within the vehicle while it is in motion to assist the driver, such as warning the driver about obstacles, applying emergency braking, or autonomously operating the vehicle. The vehicle may also include a regression model as described above, where the regression model can provide outputs based on determining that at least a portion of the sensor data is currently related to a high-loss condition. In response, the regression model can cause sensors or recording devices to record sensor data in real time. The sensor data can then be used to form a training dataset for the ML model.

[0104] ML models, autoencoders, and / or regression models can be implemented using physical neural networks or via simulated neural networks. In this respect, ML models, autoencoders, and / or regression models can be implemented as software for implementation on a processor.

[0105] The projections generated using t-SNE and the autoencoder guided by t-SNE are referred to as 2D projections throughout the preceding text. However, it should be understood that a projection can also be a 3D projection for human interpretation, or in fact, a higher-dimensional projection for machine interpretation.

[0106] The reference to the t-SNE-guided autoencoder refers to the autoencoder that has been trained using the t-SNE-reduced output in step 110. However, in the above context, an autoencoder guided using any other suitable dimensionality reduction technique can be used instead of the t-SNE-guided autoencoder. For example, a UMAP-guided autoencoder or a LargeVis-guided autoencoder can be used instead of the t-SNE-guided autoencoder. The technique used to train the autoencoder will be the same as the technique used to reduce the dimensionality of the embedding layer in the in-sample data in step 110. That is, if the autoencoder is guided using t-SNE projection, the bottleneck output will resemble the t-SNE projection; similarly, if UMAP is used to perform the first dimensionality reduction and is therefore used to train the autoencoder, the output at the bottleneck will resemble the UMAP-reduced data.

[0107] The chosen dimensionality reduction methods each possess different advantages, which are applicable depending on the ML model being analyzed. For t-SNE, local structure of the embedding layers can be preserved while transforming high-dimensional data into low-dimensional data. This technique is effective in revealing clusters in the data and preserving local neighborhood structure. For LargeVis, large-scale high-dimensional data can be visualized scalably. This technique is designed for large-scale datasets and aims to capture local and global relationships between data points. For UMAP, the geometry of data in high-dimensional space is approximated as a low-dimensional manifold in a fast and scalable algorithm. For large datasets, this technique is faster than t-SNE and preserves local and some global structure; it is also very versatile and can be used for general nonlinear dimensionality reduction tasks beyond visualization.

[0108] The methods discussed above are general because they are not limited to specific types of machine learning models. These methods combine the advantages of powerful dimensionality reduction techniques optimized for high-dimensional data visualization with the ability to transfer these attributes to out-of-sample data points using trained, guided autoencoders. For example... Figure 1 As shown, the resulting coding can be used for a wide range of model analysis functions and dataset optimization opportunities. Through the "Predictive Inference" branch of the workflow, the principles of model learning become visible, helping ML developers transition from a "black box" approach to a more transparent, interpretable, and efficient model development process. These possibilities for gaining further insights are expanded as the scenario is identified. The final branch of the workflow, "Model Analysis and Automated Dataset Optimization," directly addresses model performance metrics, opening new avenues for customizing new datasets using active learning techniques. Another advantage is the workflow's versatility, which enables automation. The model-agnostic interface, providing embedded data, metadata, and model performance data, allows for the automatic generation of corresponding visual data artifacts for model analysis and insights usable by ML developers.

[0109] Therefore, although the invention has been explained with reference to applications of ML models commonly used in the vehicle and automotive industries, the invention is not limited thereto and is applicable to ML models used in any field and in any application. It should also be understood that, in one aspect, the invention provides a trained autoencoder, while in other aspects, the invention provides different uses for training and trained autoencoders. The different branches 114, 116, and 118 of the workflow are all optional, and if any branch is used, any one or more branches can be used individually or in combination.

[0110] It should be understood that one or more of the methods described above are suitable for implementation using computer code. Therefore, the methods described above can be provided as a non-transitory computer-readable medium having computer program instructions that, when executed, cause a processor to perform the methods described above.

[0111] It should be understood that the embodiments shown above are for illustrative purposes only. In practice, the embodiments can be applied to many different configurations, and those skilled in the art will be able to implement their detailed embodiments.

[0112] It should be understood that the aforementioned steps can be performed in different orders depending on the requirements of each specific application. That is, some steps can be taken out of order and executed as appropriate for a particular application.

Claims

1. A method for training an autoencoder for analyzing machine learning models, the method comprising the following steps: Output data is extracted from the embedding layer of the machine learning model that operates on in-sample data; Dimensionality reduction is performed on at least a portion of the output data extracted from the embedding layer to obtain first dimensionality-reduced data; and The autoencoder is trained to generate corresponding intermediate dimensionality reduction data at the bottleneck of the autoencoder based on at least a portion of the extracted output data.

2. The method according to claim 1, further comprising: Generate a custom loss function for the autoencoder, the custom loss function comprising a cascade of the following: the error between the bottleneck of the autoencoder and the first dimensionality reduction data, and the reconstruction loss of the autoencoder.

3. An autoencoder for analyzing machine learning models, wherein, The autoencoder has been trained through the following operations: Output data is extracted from the embedding layer of the machine learning model that operates on in-sample data; Perform dimensionality reduction on at least a portion of the extracted output data to obtain first dimensionality-reduced data; and The autoencoder is trained to generate corresponding intermediate dimensionality reduction data at the bottleneck of the autoencoder based on at least a portion of the extracted output data.

4. The automatic encoder according to claim 3, wherein, The autoencoder is configured to receive as input the extracted output data from the embedding layer of the machine learning model that operates on unseen data, and to generate second dimensionality-reduced data at the bottleneck of the autoencoder.

5. A method for analyzing a machine learning model using an autoencoder trained according to claim 1 or 2, or an autoencoder according to claim 3 or 4, the method comprising: At the trained autoencoder, at least a portion of the output data is received from the embedding layer of the machine learning model that operates on unseen data, and the autoencoder generates second dimensionality-reduced data at the bottleneck of the autoencoder.

6. The method according to claim 5, further comprising: The regression model is trained using the second dimensionality-reduced data generated by the autoencoder, and a comparison between the regression model and the second dimensionality-reduced data is used to determine whether at least a portion of the unseen data corresponds to a predetermined model loss of the machine learning model or greater.

7. The method according to claim 6, further comprising: Record the unseen data corresponding to the predetermined model loss or greater of the machine learning model.

8. The method according to claim 6 or 7, further comprising: At least a portion of the training dataset for the machine learning model is compiled based on the recorded unseen data.

9. The method according to any one of claims 5 to 7, wherein the method further comprises: Perform model analysis on the second dimensionality-reduced data to obtain the characteristics of the machine learning model.

10. The method according to claim 9, wherein, Obtaining the characteristics of the machine learning model includes tagging data points within the second dimensionality-reduced data with associated metadata to derive the principles learned by the machine learning model.

11. The method according to claim 9 or 10, wherein, The steps for obtaining the characteristics of the machine learning model include: identifying substantially similar activations in the embedding layers of the machine learning model.

12. A machine learning model system for manipulating vehicle-related sensor data during driving, the system comprising: A machine learning model configured to generate at least one prediction of the vehicle's state based on the sensor data; as well as One or more autoencoders according to any one of claims 3 and 4, wherein the autoencoder is configured to generate dimensionality-reduced data based on output data extracted from the embedding layer of the machine learning model corresponding to at least a portion of the sensor data.

13. The machine learning model system according to claim 12, wherein, The system also includes a regression model trained to predict the model loss of the machine learning model related to at least a portion of the sensor data.

14. The system according to claim 12 or 13, wherein, The sensor data includes at least one of camera, lidar, radar, speed, acceleration, and yaw sensor data.

15. A non-transitory computer-readable medium comprising computer program instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 2 and 5 to 11.

Images