Methods for controlling the operation of a system
A multi-granular dataset comparison method addresses inefficiencies in machine learning by aligning operational data with training data through feature space projection and ensemble dissimilarity measures, enhancing algorithm performance and system adaptability.
Patent Information
- Authority / Receiving Office
- GB · GB
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2026-04-01
AI Technical Summary
Existing data dissimilarity measures are challenging in high-dimensional spaces, computationally costly, and difficult to generalize across different types of data, leading to inefficiencies in machine learning algorithm performance when operational data diverges from training data.
A multi-granular approach to dataset comparison is employed, projecting data onto feature spaces and calculating dissimilarity at multiple levels of granularity, combining results to form an ensemble measure, and adjusting system parameters to minimize dissimilarity.
Enhances dataset discrepancy quantification, improving machine learning algorithm performance by aligning operational data with training data, enabling efficient and adaptable system operation.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
FIELD Embodiments described herein relate to methods for controlling the operation of a system. BACKGROUND Machine learning (ML) elements such as classifiers, regressors, predictors, etc. are often incorporated in autonomous systems. Machine learning is fuelled by empirical data, and therefore it is important to analyse and evaluate the data which influences the operation of ML elements and their parent systems. One form of data analysis which is required is the ability to compare datasets. For example, if the operational data received by a machine learning algorithm at run-time is significantly different from the dataset used to train the algorithm, the algorithm may no longer be expected to make accurate decisions during operation. Data dissimilarity measures quantify the statistical discrepancy between datasets. These measures have wide ranging applications. For example, the performance of a machine algorithm applied to an arbitrary dataset is related to the extent to which the dataset differs from the dataset used to train the algorithm. A dataset dissimilarity measure can be used to quantify the statistical discrepancy between the training dataset and another, distinct dataset. This quantification of dataset difference allows a numerical relationship to be established between algorithm performance and the dissimilarity of a dataset to the training dataset. If the performance of a trained machine learning algorithm is recorded for a set of unseen test datasets at design-time, prior to operation, then a relationship can be found between the performance returned for a test dataset, and the dissimilarity of that test dataset to the training dataset. Finally, this performance-versus-dissimilarity relationship can be used to predict the performance of the algorithm at run-time when applied to a dataset received during operation, having first measured the dissimilarity of the operational dataset to the training dataset. This particular technique, which makes use of data dissimilarity measurements, thus provides information about the reliability of the estimates or outputs of ML models with respect to the nature of the data being processed. The task of quantifying the similarity (or, equivalently, the dissimilarity) between datasets or, alternatively, between the statistical distributions from which these samples are drawn, has been extensively investigated in the literature. Different data dissimilarity measures have been proposed in the literature, each of which has its own positive and negative points. An example of such a measure is described in UK Patent Application No. 2011580.4. Standard statistical distances often require that an approximation of a relevant distribution is inferred from point measurements, which is a problematic operation in high-dimensional spaces (i.e. where the number of variables or features is large). As a result, the application of such approaches is generally challenging in the field of data science. Another common drawback is high computational cost, as exemplified by the widely applied Maximum Mean Discrepancy (MMD) measure. Although the use of the MMD can be perfectly justified for offline analysis, the use of this measure might not be feasible for online operations when the response time is crucial. Some conventional data discrepancy measures are highly specific to a given type of data and difficult to generalise to other contexts. For example, measures based on specific feature representations generated using Siamese networks or extracted in line with Computer Vision-based definitions have been investigated and proven effective with imagery. However, extending their applicability beyond images is not straightforward. Finally, there are many ways in which the dissimilarity between datasets can be defined, but not all of these forms of dissimilarity will be effective for specific applications, such as the use of dissimilarity measurements for predicting algorithm performance. SUMMARY According to a first aspect of the present invention, there is provided a method for controlling the operation of a system, the system having one or more sensors, wherein one of the sensors is a first sensor configured to capture sensor data to be used as input into a software algorithm, the method comprising: determining a measure of dissimilarity between a first dataset D? used for training or configuring the software algorithm and a second dataset D2 captured by the first sensor and input into the software algorithm; and based on the measure of dissimilarity, adjusting one or more operating parameters of the system. Determining the measure of dissimilarity may comprise comparing each of the first dataset D? and the second dataset D2 with a reference dataset Ds. The first dataset Di and the second dataset D2 may be compared with the reference dataset Ds at one or more levels of granularity. The first dataset Di may comprise a first set of data elements. The second dataset D2 may comprise a second set of data elements. The reference dataset Ds may comprise a third set of data elements. For each level of granularity, comparing the first dataset Di with the reference dataset Ds at the level of granularity may comprise computing, for one or more elements of the first dataset Di, a distance between the respective element and one or more subsets of the reference dataset Ds. For each level of granularity, comparing the second dataset D2 with the reference dataset Ds at the level of granularity may comprise computing, for one or more elements of the second dataset D2, a distance between the respective element and one or more subsets of the reference dataset Ds. The size of the subsets of the reference dataset Ds may be different for each level of granularity. The method may comprise, for each level of granularity, deriving a respective measure of dissimilarity based on the distances between the data elements and the subsets of the reference dataset at that level. The measure of dissimilarity may be computed using the respective measures of dissimilarity from each level of granularity. Computing the distance measures at a first level of granularity may comprise: projecting the elements of the first dataset Di, the second dataset D2 and the reference dataset Ds onto a feature space; determining a centroid of the projections of the elements of the reference dataset Ds on the feature space; for each one of the elements of the first dataset Di, determining a distance in the feature space between the projection of the respective element and the centroid Asc\ and for each one of the elements of the second dataset D2, determining a distance in the feature space between the projection of the respective element and the centroid Ac- Computing the distance measures at a second level of granularity may comprise: determining, for each subset of the reference dataset Ds, a centroid of the projections of the elements in the respective subset on the feature space; and for each one of the elements of the first dataset Di, determining a distance in the feature space between the projection of the respective data element and the closest centroid from among the centroids of the different subsets; and for each one of the elements of the second dataset D2, determining a distance in the feature space between the projection of the respective data element and the closest centroid from among the centroids of the different subsets. Computing the distance measures at a third level of granularity may comprise: for each one of the elements in the first dataset Di. identifying a number of k elements of the reference dataset Ds, whose projections in the feature space are closest to the projection of the respective element of the first dataset Di, each set of k data elements defining a respective one of the subsets of the reference dataset Ds, and determining an average distance between the projection of the element of the first dataset Di and the projections of the k data elements in the respective subset of the reference dataset Ds\ and for each one of the elements in the second dataset D2. identifying a number of k elements of the reference dataset Ds, whose projections in the feature space are closest to the projection of the respective element of the second dataset D2, each set of k data elements defining a respective one of the subsets of the reference dataset Ds, and determining an average distance between the projection of the element of the second dataset D2 and the projections of the k data elements in the respective subset of the reference dataset Ds. The software algorithm may be a machine learning algorithm. The machine learning algorithm may be an artificial neural network, ANN. Projections in the feature space may be defined by activation values of neurons in the ANN when the respective datasets are input to the ANN. The feature space may be defined by activation values of a subset of the neurons contained within inner layers of the ANN. The first dataset may be used to train the machine learning algorithm. The reference dataset Ds may be a validation dataset used in validating the training of the algorithm. The method may further comprise: updating the second dataset D2 to include subsequent data captured by the first sensor; obtaining an updated measure of dissimilarity between the first dataset Di and the updated second dataset D2, and based on the updated measure of dissimilarity, further adjusting the one or more operating parameters of the system. The operating parameters may be adjusted such as to reduce the dissimilarity between the first dataset Di and the second dataset D2. The operating parameters may be adjusted using reinforcement learning to reduce the dissimilarity between the first dataset Di and the second dataset D2. Updating the second dataset D2 may comprise discarding one or more elements of the second dataset D2 received at an earlier point in time. The one or more operating parameters may be adjusted in the event that the measure of dissimilarity is determined to be above a pre-defined threshold. In the event the measure of dissimilarity is determined to be below a pre-defined threshold, the method may comprise: predicting, based on the measure of dissimilarity, an expected performance of the machine learning algorithm when the second dataset D2 is input to the machine learning algorithm. The system may be a mobile system. Adjusting the one or more operating parameters may comprise adjusting the motion of one or more components of the system. Adjusting the motion of one or more components of the system may comprise: changing a speed or direction of motion of the one or more components and / or adjusting the motion of a platform on which the sensors are mounted. Adjusting the one or more operating parameters may comprise adjusting one or more acquisition settings of the one or more sensors. Adjusting the acquisition settings may comprise adjusting one or more acquisition settings of the first sensor. The system may comprise at least two sensors. Adjusting the operating parameters may comprise adjusting one or more acquisition settings of a second sensor different from the first sensor. The at least one sensor and the second sensor may comprise different types of sensor. The at least one sensor and the second sensor may comprise a camera and a radar sensor. Adjusting the operating parameters may comprise switching from capturing sensor data from the first sensor to capturing sensor data from a second sensor different from the first sensor. According to a second aspect of the present invention, there is provided a method for controlling the operation of a system, the system comprising a processing pipeline including a first sub-component and one or more other sub-components, the first subcomponent being configured to process data received from the one or more other subcomponents and to output the processed data as input to a software algorithm, the method comprising: determining a measure of dissimilarity between a first dataset Di used for training or configuring the software algorithm and a second dataset D2, the second dataset comprising the processed data output by the first sub-component; and based on the measure of dissimilarity, adjusting one or more elements of the processing pipeline. Adjusting the one or more elements of the processing pipeline may comprise: performing a search of a parameter space of each element to identify one or more elements of the processing pipeline that when adjusted will minimise the measure of dissimilarity. Adjusting the one or more elements of the processing pipeline may comprise adjusting the configuration of a de-noising algorithm to be applied to the output from the first subcomponent. Adjusting the one or more elements of the processing pipeline may comprise: determining whether or not to reduce the contribution of high spatial frequency 2D sinusoids in the second dataset and / or determining an extent by which to reduce the contribution of the high spatial frequency 2D sinusoids. Adjusting the one or more elements of the processing pipeline may comprise adjusting a means by which the first sub-component processes the data received from the one or more other sub-components. Determining the measure of dissimilarity may comprise comparing each of the first dataset Dt and the second dataset D2 with a reference dataset Ds. The first dataset Di and the second dataset D2 may be compared with the reference dataset Ds at one or more levels of granularity. The first dataset Di may comprise a first set of data elements. The second dataset D2 may comprise a second set of data elements. The reference dataset Ds may comprise a third set of data elements. For each level of granularity, comparing the first dataset D! with the reference dataset Ds at the level of granularity may comprise computing, for one or more elements of the first dataset Dj, a distance between the respective element and one or more subsets of the reference dataset Ds. For each level of granularity, comparing the second dataset D2 with the reference dataset Ds at the level of granularity may comprise computing, for one or more elements of the second dataset D2, a distance between the respective element and one or more subsets of the reference dataset Ds. The size of the subsets of the reference dataset Ds may be different for each level of granularity. The method may comprise, for each level of granularity, deriving a respective measure of dissimilarity based on the distances between the data elements and the subsets of the reference dataset at that level. The measure of dissimilarity may be computed using the respective measures of dissimilarity from each level of granularity. Computing the distance measures at a first level of granularity may comprise: projecting the elements of the first dataset Di, the second dataset D2 and the reference dataset Ds onto a feature space; determining a centroid Asc of the projections of the elements of the reference dataset Ds on the feature space; for each one of the elements of the first dataset Di, determining a distance in the feature space between the projection of the respective element and the centroid and for each one of the elements of the second dataset D2, determining a distance in the feature space between the projection of the respective element and the centroid ASC- Computing the distance measures at a second level of granularity may comprise: determining, for each subset of the reference dataset Ds, a centroid of the projections of the elements in the respective subset on the feature space; and for each one of the elements of the first dataset D1; determining a distance in the feature space between the projection of the respective data element and the closest centroid from among the centroids of the different subsets; and for each one of the elements of the second dataset D2, determining a distance in the feature space between the projection of the respective data element and the closest centroid from among the centroids of the different subsets. Computing the distance measures at a third level of granularity may comprise: for each one of the elements in the first dataset Df. identifying a number of k elements of the reference dataset Ds, whose projections in the feature space are closest to the projection of the respective element of the first dataset Di, each set of k data elements defining a respective one of the subsets of the reference dataset Ds, and determining an average distance between the projection of the element of the first dataset Di and the projections of the k data elements in the respective subset of the reference dataset Ds; and for each one of the elements in the second dataset D2: identifying a number of k elements of the reference dataset Ds, whose projections in the feature space are closest to the projection of the respective element of the second dataset D2, each set of k data elements defining a respective one of the subsets of the reference dataset Ds, and determining an average distance between the projection of the element of the second dataset D2 and the projections of the k data elements in the respective subset of the reference dataset Ds. The software algorithm may be a machine learning algorithm. The machine learning algorithm may be an artificial neural network, ANN. Projections in the feature space may be defined by activation values of neurons in the ANN when the respective datasets are input to the ANN. The feature space may be defined by activation values of a subset of the neurons contained within inner layers of the ANN. The first dataset may be used to train the machine learning algorithm. The reference dataset Ds may be a validation dataset used in validating the training of the algorithm. The method may further comprise: updating the second dataset D2 to include subsequent data output by the first sub-component; obtaining an updated measure of dissimilarity between the first dataset Di and the updated second dataset D2, and based on the updated measure of dissimilarity, further adjusting the one or more elements of the processing pipeline. The one or more elements of the processing pipeline may be adjusted such as to reduce the dissimilarity between the first dataset D? and the second dataset D2. The one or more elements of the processing pipeline may be adjusted using reinforcement learning to reduce the dissimilarity between the first dataset Di and the second dataset D2. Updating the second dataset D2 may comprise discarding one or more elements of the second dataset D2 received at an earlier point in time. The one or more elements of the processing pipeline may be adjusted in the event that the measure of dissimilarity is determined to be above a pre-defined threshold. In the event the measure of dissimilarity is determined to be below a pre-defined threshold, the method may comprise: predicting, based on the measure of dissimilarity, an expected performance of the machine learning algorithm when the second dataset Dz is input to the machine learning algorithm. According to a third aspect of the present invention, there is provided a computer-readable storage medium comprising computer executable instructions that when executed by one or more computers will cause the one or more computers to carry out a method according to the first or second aspect of the present invention. According to a fourth aspect of the present invention, there is provided a system comprising: one or more sensors; and one or more computers configured to carry out the method according to the first aspect of the present invention. According to a fifth aspect of the present invention, there is provided a system comprising: a processing pipeline including a first sub-component and one or more other sub-components, the first sub-component being configured to process data received from the one or more other sub-components and to output the processed data as input to a software algorithm; and one or more computers configured to carry out the method according to the second aspect of the present invention. According to a sixth aspect of the present invention, there is provided an autonomous vehicle comprising a system according to the fourth or fifth aspect of the present invention. Embodiments described herein provide a novel means of quantifying the discrepancy between two sample datasets. The data dissimilarities so obtained may be used for a variety of applications, including use as feedback signals / inputs to decision-making processes for controlling or guiding the behaviour of an autonomous system and the modes by which that system generates data. In particular, embodiments described herein allow for data comparisons to be performed at multiple levels of detail, examining coarse as well as more fine-grained properties of the various datasets. This leads to the introduction of an ensemble-based method of comparison where multiple data dissimilarities are first independently calculated at different levels of granularity, and then combined together to produce an enhanced data discrepancy value. Distances (dissimilarities) can be computed by projecting the original data items in the respective datasets into suitable feature spaces, selected in line with the complexity of the considered problem and the available computational resources. Dataset comparisons at different levels of detail may be carried out independently of each other, making it possible to numerically perform these operations simultaneously through the adoption of parallelised implementations. The measure of dissimilarity may be tailored to particular applications by selecting a specific set of levels of granularity for analysis. In some cases, all levels from coarse to fine might be selected - in other cases coarse to medium, or medium to fine. To detect that one dataset is drawn from a different data-generating distribution than a second dataset, for example, measurements made at a wide range of scales might be combined, employing coarse- to fine-grained analysis, since extracting more information will generally be beneficial. In contrast, the use of only fine-grained analysis, which compares the local properties of datasets, might be more effective for predicting classifier performance in some circumstances. Computational economy is another factor which will influence the range of levels incorporated into the measure. The number of levels of granularity can be varied, as can the size of the grain. Dataset comparison at finer levels of detail may effectively employ suitable sub-datasets rather than the whole dataset. The multi-granular approach not only attempts to characterise and compare data properties at different levels of abstraction, but also provides a systematic strategy to combine the results of these comparisons. Further flexibility also ensues from the fact that, for each degree of data coarseness considered, several measure variations could be defined. For example, when specifying different measure variants which employ clustering, any of a number of clustering techniques (both supervised and unsupervised) could be selected and applied to appropriately divide a dataset. BRIEF DESCRIPTION OF DRAWINGS Embodiments of the invention will now be described by way of example with reference to the accompanying drawings in which: Figure 1 shows a schematic of a system used to determine a measure of dissimilarity between two datasets, in an embodiment; Figure 2 shows a flow-chart of steps used for determining the measure of dissimilarity in an embodiment; Figure 3 shows a further flow-chart of steps used for determining the measure of dissimilarity in an embodiment; Figure 4A shows a schematic of a system that uses a dissimilarity measure to control the adjustment of one or more operating parameters, in an embodiment; Figure 4B shows another schematic of a system that uses a dissimilarity measure to control the adjustment of one or more operating parameters, in an embodiment; Figure 4C shows another schematic of a system that uses a dissimilarity measure to control the adjustment of one or more operating parameters, in an embodiment; Figure 5 shows an example of a classifier resilience function used to measure the generalisation capability of a machine learning algorithm, in an embodiment; Figure 6 shows a flow-chart of how the classifier resilience function of Figure 5 may be used to predict the performance of a classifier, in an embodiment; Figure 7 shows a schematic of a system that uses a dissimilarity measure to control one or more elements of a processing pipeline, in an embodiment; and Figure 8 shows another schematic of a system that uses a dissimilarity measure to control one or more elements of a processing pipeline, in an embodiment. DETAILED DESCRIPTION In embodiments described herein, a measure of dissimilarity between two datasets of interest, Di and D2 is determined. In some embodiments, the dataset Di comprises a training set used to train a machine-learning algorithm, and D2 comprises an “operational” dataset acquired after the algorithm has been trained. The datasets Di and D2 will be referred to as ‘acquired’ datasets. The two respective datasets are compared with a source or ‘reference’ dataset Ds. The dataset Ds may comprise a validation dataset also used during training of the machine-learning algorithm for validation purposes e.g. hyperparameter setting I stopping criteria, and should come from the same distribution from which the training dataset Di is drawn. On the basis of these individual comparisons with the dataset Ds, it is possible to obtain an estimate of the dissimilarity between the datasets Di and D2. Thus, in some embodiments: . Di is a training set used to train a machine-learning algorithm . Dz is an operational dataset acquired after the algorithm has been trained . Ds is a validation dataset, drawn from the same distribution as the training dataset Di, which is used during training of the machine-learning algorithm More specifically, a data dissimilarity value is produced for the two datasets Di and D2 at each one of a number of levels of granularity. In theory, the result of computing the dissimilarity at a particular level of granularity could itself be seen as a fully defined measure of the distance between Di and D2. However, in embodiments described, the results of computing the dissimilarity at these different levels of granularity are combined with one another to form an ensemble measure of the distance between Di and D2. The adoption of such an ensemble-based strategy can provide an enhanced measure that performs better than each of the granular components when used individually. In particular, the embodiments disclosed herein make use of a breadth and diversity of data properties, thus generalising beyond the use of a limited number, which are used to return a summary, single value of dissimilarity. The benefit of the described multi-granular analysis is that it defines measurements based on a relatively wide range of data properties, which are able to reflect more of the information latent in the data. The measure of dissimilarity can be computed after data items from the datasets Di, D2 and Ds have been projected onto a suitable feature space, the definition of which is problem specific. No specific assumptions about this data representation need be introduced. In some embodiments, the space in which the raw data is embedded may be entirely suitable for computing the measure of dissimilarity. An example of a dissimilarity measure based on the multi-granular approach will now be described with reference to Figures 1 to 3. In this embodiment, dataset comparisons are practically realised by adopting the following levels of coarseness of the reference dataset: Ds. Level 1: involves the whole reference set Ds; Level 2: involves medium-sized subsets of Ds; and Level 3: involves small-sized subsets of Ds. Figure 1 shows the setup used to determine the measure of dissimilarity in the present embodiment. In the present embodiment, data comparisons are made in a feature space defined by the neuron outputs of a suitably chosen artificial neural network (ANN) classifier 101 that has been trained on the (entirety) of the training dataset Di. Once the algorithm has been trained, the data items in each of the datasets Di, D2 and Ds will be fed into the ANN classifier as inputs and the neuronal activation recorded in each case. No a priori assumptions are made about the architecture of the chosen ANN classifier, although it is recognised that constraints might be imposed by the properties of the examined data (e.g. the complexity of the model should match that of the datasets of interest). Figure 2 shows an overview of the steps used for determining the measure of dissimilarity. Beginning in step S201, the datasets Di and Ds are used to train the ANN. By way of example, the ANN may comprise a classifier algorithm and the dataset Di may comprise training data including examples of positive occurrences (for example, images containing an object that the classifier is being trained to identify) as well as examples of negative occurrences (for example, images in which the object is absent). Each item in the dataset Di includes a label identifying the item as a positive or negative example. The classifier itself comprises a model that extracts features from the training data to determine whether or not a particular item is a positive or negative example. Each time the classifier is presented with an example from the training set Di, it will judge whether or not the example is a positive or negative occurrence, and compare its output with the label for that example. A cost function can be defined that reflects the difference between the classifier output and the label, as a function of the parameters of the model. By repeatedly exposing the model to different items of training data and modifying the parameters of the model to minimise the cost function, it is possible to improve the classifier performance, such that it will identify positive and negative examples with increasing accuracy. Once the cost function has been reduced to a satisfactory degree, the classifier can be considered as “trained”. Stopping criteria for the training process may be defined based on the classifier performance on the validation dataset Ds. In order to compare the datasets Di and D2, the activation values of neurons in the ANN can be recorded when those different datasets are input into the ANN. The activation values of a proper subset of the ANN neurons might be recorded rather than the whole set of neurons. To begin with, in step S203, neuronal activations are recorded for data belonging to the dataset Di. Following this, neuronal activations are measured when elements in the dataset D2 are input into the ANN (step S205) and when elements in the dataset Ds are input into the ANN (step S207). The measure of dissimilarity can be obtained by then comparing the neuronal activations generated by each one of the three datasets (step S209). The precise manner in which this comparison is carried out will now be described in more detail with reference to Figure 3. We will use NA to denote the number of neurons selected to define the feature space for sample comparisons. We can denote as 4(-) e IR"4 the corresponding activation trace, namely a vector whose components are the activation values of these selected neurons in response to a particular input. In other words, we adopt 4(%) as the alternative representation of a generic input data item x. Moreover, we specify that this operation is carried out assuming that the network has previously been trained on the training dataset Di where: x^.n = 1,..., Nr are input data items; and 6 {1, ...,C],n = 1, are the corresponding categorical labels, assuming the existence of C distinct classes. A data item within either D} or D2, when represented as a point in feature space (corresponding to 4(-)), will be referred to as an acquired point. Then, we calculate distances in the defined alternative feature space to obtain the dissimilarity between Di and D2. The term ‘distance’ will be used in this mathematical description of the computation of the dissimilarity measure, and only in this strict sense within this section, to refer to a value that is computed as a function of properties of Di and Ds or as a function of properties of D2 and Ds. Dissimilarities are, in turn, defined as functions of these distances. In the following text, different distance definitions are introduced that enable the acquired datasets to be compared with the reference dataset at different levels of granularity. Specifically, three granularity levels are defined below. First measure of granularity (level 1): A dissimilarity measure is computed which is a function of the distances between acquired points and the entire reference dataset Ds. Here, we calculate the reference centroid Asc e as the centre of the projections of the reference data items on the considered feature space as follows: n=l where x^s\ n = 1,..., Ns, denote the data items in Ds. Then, for a given acquired point x e Dlt we determine its Euclidean distance from the reference centroid, i.e. ||A(x) - Asc || We repeat the latter operation with respect to all data items included in Dr, and subsequently compute the associated empirical mean value. The same procedure is carried out with respect to the acquired points contained in D2. Finally, the absolute value of the difference between the empirical means of the compared sets is taken as the ‘level T component of the dissimilarity measure. Mathematically, this corresponds to the following formula: dlevell^D^ D2) where x^2) e D2,n = 1,...,N2 and x^ ED1,n = 1, ...,N1. Second measure of granularity (level 2): The dissimilarity measure computed at the second level of granularity is a function of distances between acquired points and medium-sized subsets of Ds. We begin by splitting the dataset Ds into nc clusters and computing the corresponding centroids A^,m = 1,..., nc in the considered feature space. Both supervised and unsupervised clustering approaches may be applicable. For some datasets, data items can be assigned to one of a number of classes, using some set of criteria, prior to the application of any clustering technique. This class information can be used by a clustering technique. The discrepancy between a given acquired point x e D} and the reference set is quantified by computing its Euclidean distance from the nearest cluster centroid, i.e. min ||71(x) — AU|2. m=l,...,nc The above operation is repeated for all data instances included in Di. The associated mean is then calculated analogously to what has been described above for the first measure of granularity. The same procedure is now carried out with respect to the acquired points included in Dz. Finally, the absolute value of the difference between the mean calculated for Di and the mean calculated for Dz is taken as the level 2 component of the dissimilarity measure. Mathematically, this corresponds to the following formula: dlevel2 (A- ) «1 — y min |U (x^1^ — ^mll >Z-i m=l nc II V n 7 m „2 n=l where x^ e D2,n = 1,...,N2 and x^D1) e D^n = 1, ...,N1. Third measure of granularity (level 3): The dissimilarity measure computed at the third level of granularity is a function of the distances between acquired points and small subsets of Ds. A k-Nearest Neighbours (kNN) algorithm is employed to identify the k reference points that are closest in Euclidean distance to a given acquired point, all the relevant data items having been projected into the considered feature space. The average distance between these k reference points and that of the acquired point is then determined. This average distance is itself averaged over first Di and then Dz, before some final steps give the level 3 data discrepancy. Mathematically, for a given acquired input x e Dlt we first identify the k reference points ..., x£Ds)j whose activation traces {A(x^Ds)),..., A(x£Ds))} are closest to that of x, and consequently calculate the following average distance: k —||A(x) — A (x™) || . 7=1 2 The operation is repeated for all acquired points included in Di, and the empirical mean determined analogously to what has been discussed for the first and second granularity levels. The same procedure is then carried out with respect to the acquired points included in D2. Finally, the absolute value of the difference between the empirical means computed for the datasets being compared is taken as the ‘level 3’ dissimilarity measure component. Mathematically, this corresponds to the following formula: where x^ e D2,n = 1,...,N2 and x^D1) e D1,n = 1, ...,N1. It will be understood that the above equation requires computing the k nearest neighbours of each acquired data point considered. In other words, each acquired point x is associated with a set of neighbouring reference points ^Ds)(x),..., x^Ds)(x)j. For simplicity, however, we adopt a notation in which the dependence on x is not explicitly indicated i.e. ...,x£Ds)j. As an alternative to the third measure of granularity (level 3), the following formula may also be considered, interpreting the terms in the same way as the first level 3 formula given: where x^ e D2,n = 1,...,N2 and x,^ e D^n = 1, ft ' ' £. fl X' ' ' J. Here, dav denotes an average k-NN distance for the reference set and can be seen as a normalisation factor. dav can be found as: with e Ds,n = 1, ..., Ns and as above, the understanding that the k nearest neighbours of a point in Ds are dependent on that point The alternative formulation d^g^D^D^ mainly differs from dievel2(Pi,D2) because the influence of any distant neighbours amongst the / c-nearest neighbours will gradually diminish with distance. Thus far, we have introduced formulas to compute the dissimilarity between Di and D2 by performing sample comparisons at different levels of granularity. The measures so obtained at each level of dissimilarity may now be combined to provide an overall measure of dissimilarity MGrMtPi^) by combining them in line with simple model bagging techniques. This approach is motivated by the fact that different levels of dataset comparison, each relying on a distinct mode of data analysis, can capture complementary information about the datasets being compared. Two alternative formulations for combining the measures obtained at each level of granularity to obtain the overall measure of dissimilarity MGrM(D1,D2) are as follows: (i) MGrMptDi, D2) = dieveii(Di> D2) • dievei2(Di, D2) ■ d^g^^Di, O2); and (ii) MGrMs^D^D^ = dieveii(Di, D2) + dievei2(^1^2) T dievei2(Dt,D2). The steps described above are summarised in the flow-chart of Figure 3. It will be appreciated that the above formulations are provided by way of example and other examples of multi-granular discrepancy estimators could be attained by modifying these formulations. For instance, the precise steps by which the overall measure of dissimilarity is calculated could be modified by taking into account the considerations listed below. 1. In the embodiment described above, sample comparisons are carried out by first projecting the original datasets into a feature space defined by the neuron outputs of a suitably chosen ANN classifier. To improve computational efficiency, in some embodiments, only the activation outputs produced by the neurons included in the deepest layers (i.e. the most specialised layers) of the AN Ns might be employed. It is also the case that it might be beneficial to use the outputs within deeper layers, since these tend to extract semantic information whilst discarding incidental information. However, different layers of an ANN may appear to learn features at diverse levels of abstraction; thus, in some embodiments, the activation levels of combinations of neurons belonging to a number of different layers embedded at different depths in the ANN may be recorded, rather than simply recording those neurons in the deepest layers of the ANN. More broadly, the dissimilarity measure may be obtained by projecting the datasets onto an altogether different feature space. Different data projection mechanisms lead to diverse dissimilarity estimates, thus increasing the flexibility of the method. The sample representations discussed above are derived by feeding the data of interest to ANNs. Such representations include those based on neuron outputs (i.e. activation values) or convolutional neural network (CNN) kernel outputs. In contrast, an example of a data projection mechanism that does not rely on Al algorithms is simply given by the identity function, which leads to sample comparisons directly performed in the original data space. It will be appreciated here that different data representations might vary significantly in terms of the computational costs associated with the operation of projecting the original data onto new feature spaces. For example, memory constraints could arise when high-dimensional feature representations are adopted for large datasets. Therefore, a balance between available computational resources and other constraints, as well as the effectiveness of the representation should be sought. Depending on the nature of the data collected, privacy and security concerns might also arise, which could require the use of specific representations (e.g. anonymised formats) to address these issues. In some applications, feature representations that promote interpretability may be preferred. 2. Modifications of the measure components could be introduced for any of the granularity levels considered. For example, instead of using Euclidean distances, alternative metrics might be employed. Furthermore, the intermediate granularity levels could employ any one of a number of possible clustering algorithms (both supervised and unsupervised). 3. The number of measure components, each corresponding to a different level of granularity analysis, could be varied. This flexibility takes into account the fact that some measure components could be better suited to specific inputs, or specific applications of the measure, than others. 4. Different ensemble-based approaches could be used to combine measure components associated with multiple degrees of granularity. The choice of a suitable ensemble strategy will depend on several factors, including the number of granularity levels considered, the complexity of the feature representation adopted, the properties of the examined datasets, and the amount of data available in each of the sets of interest. 5. Depending on the considered granularity levels and their modes of combination, as well as the use to which the dissimilarity measure is put, it could be the case that data analyses are realised by directly or effectively comparing only specific subsets of the original datasets. In other words, the multi-granular approach not only introduces high flexibility in the definition of discrepancy measures, but also provides ways to systematically choose subsets of the datasets of interest. The selection of subsets may be beneficial, depending on the task for which measurements are made and the nature of the data. 6. The sample representation described above refers directly to point clouds (i.e. points are mapped from the original data space to points in a new feature space, namely the activations of neurons in an ANN). However, it will be understood that statistical representations are also possible. For example, it is possible to first compute empirical estimates of the probability distributions corresponding to the examined datasets, and then calculate probability values to attain data dissimilarity measurements. It is recognised that estimating distributions from a set of points is generally a problematic operation, especially in high-dimensional data spaces. Taking such a statistical approach, different levels of granularity could be defined by approximating the distributions associated with diverse subsets of the original datasets of interest. In contrast to the case where point-based representations are used, and where comparing datasets amounts to comparing the associated point clouds, comparing datasets when probabilistic representations are employed requires calculating the divergence between the associated distributions. 7. The multi-granular approach can be applied to a single data instance, effectively a dataset with one member. It could be the case that Di or Dz or both of these datasets could contain a single member. A dissimilarity measure as obtained using the methods described above can be employed by any type of autonomous and automatic system requiring the computation of data similarities and / or data comparisons for a decision-making process. The decision-making process would then use the dissimilarity measure to guide system actions. Data dissimilarity measurement can inform tasks such as domain or distributional shift analysis, out-of-distribution detection and anomaly detection, and the verification of the generalisation capability of Al-based components. Moreover, data dissimilarity measurement can also be used to provide feedback signals I inputs for guiding or controlling system behaviour. In this regard, example applications include: controlling the behaviour of an autonomous system; guiding the manner in which a system generates or processes data; and enabling the use of reinforcement learning to train an agent to select system actions (for example actions which optimise the configuration of a sensor or process), where the reward functions are based on observed data dissimilarities. Figures 4A to 4C show an example of a system that utilises a dissimilarity measure such as that described above to control the adjustment of one or more operating parameters. In the present example, the operating parameters pertain to parameters of a camera or other sensor 401 that is used to capture sensor data from the surrounding environment. The captured data is input to a software algorithm 403 that processes the data to provide an output 405. In the present embodiment, the algorithm 403 is a machine learning algorithm, such as an ANN classifier that processes the captured sensor data to identify the presence of certain objects or features in the environment. It will, however, be appreciated that the software algorithm need not involve a machine learning element-the algorithm may, for example, comprise a classifier constructed using manually designed rules. The output 405 takes the form of one or more control signals used to control one or more parts of the system. For example, the control signal may initiate an alarm to warn of the presence of an object identified as a threat, or alter the motion of a vehicle in anticipation of the behaviour of a particular object (such as slowing an autonomous vehicle if an animal is identified). The software algorithm is configured using a first dataset. Figure 4A shows an example of algorithm configuration, in which the algorithm 403 comprises a machine learning algorithm such as a classifier, and the first dataset comprises a set of training data received from the camera / sensor and used to train the algorithm. Here, the training dataset can be understood to correspond to the dataset Di described above. It will be appreciated that the training dataset Di need not necessarily be provided by the camera / sensor and could instead comprise a set of images obtained from another source or repository of training data. In this latter case, the camera would only be used during operation, after the training / configuration phase is complete. Also shown in the Figure 4A is the validation dataset Ds used to validate the software algorithm’s performance during training. The validation dataset Ds does not form part of the training dataset itself, but is drawn from the same distribution from which the training dataset Di is drawn. Like the dataset Di, the validation dataset may be provided by the camera / sensor or could instead comprise a set of images obtained from another source or repository of training data. In other embodiments, in which the algorithm 403 is not a machine-learning algorithm, but a conventional software algorithm such as a rule-based classifier wholly designed by a human engineer, the first dataset as used to configure the algorithm may comprise a dataset from which the engineer has derived rules on which the classifier is intended to operate. In another example, the first dataset may be a dataset used to select / tune the free parameters of a non-ML based algorithm. For instance, the software algorithm 403 might perform background subtraction of image data; in this case, the first dataset may comprise a set of images used to compute a background profile. Figure 4B shows the system at run-time, in which the now trained I configured algorithm 403 is receiving a second dataset, comprising operational data, from the camera I sensor 401. Here, the second dataset can be understood to correspond to the dataset D2 described above. Depending on the degree of (dis)similarity between the datasets Di and D2, the accuracy I trustworthiness of the output 405 from the algorithm may be compromised. As an example, the training dataset Di might comprise a sequence of images captured when the camera / sensor was aligned in a certain orientation or with a particular aperture setting or exposure time. If the operational data D2 is captured with different exposure settings on the camera, the algorithm 403 may no longer be capable of making accurate classifications of objects seen in the images. To address this problem, the training dataset Di and the operational dataset D2 are compared with one another using a comparison algorithm 407 to obtain a measure of the dissimilarity 409 between them. In some embodiments, the comparison algorithm 407 may determine the measure of dissimilarity using the multi-granular approach described above in relation to Figures 1 to 3. In the embodiment shown in Figure 4B, the comparison algorithm 407 comprises its own ANN, separate from the software algorithm 403, and trained using the dataset Di and validation dataset Ds. By recording the activation of neurons within the (trained) ANN of the comparison algorithm 407 when the respective datasets Di, D2 and Ds are input to the ANN, the degree of dissimilarity can be established by a process akin to that shown in Figure 3. Figure 4C shows an alternative embodiment in which the datasets Di, D2 and Ds are fed into the same algorithm from which the output 405 is obtained, and the comparison algorithm 407 records the activation of neurons within the software algorithm 403 itself to determine the degree of dissimilarity. More generally, in the event that the algorithm 407 were not an ANN or ML algorithm, the comparison algorithm 407 would record the feature representations within the software algorithm 403. If the degree of dissimilarity is above a certain predefined threshold, this may serve as an indication that the algorithm 403 is not equipped to process the captured operational data from the camera I sensor with the level of accuracy required. Thus, an adjustment 411 may be made to the camera I sensor in order that the data subsequently captured by that sensor is brought into closer conformity with the first dataset Di used to configure the algorithm 403 at the outset. The camera may, for example, be rotated about one or more axes to regain the orientation that was adopted when capturing Di. Again, one or more of its exposure settings may be varied such that the images now output by the camera have similar exposure characteristics to those used to train I configure the software algorithm 403. It will be appreciated here that the process shown in Figure 4 and subsequent Figures may continue iteratively over the course of multiple cycles. In each cycle, the second dataset D2 may be updated with the newly received data item(s) captured by the camera I sensor, and an updated measure of dissimilarity obtained by comparing the now updated second dataset D2 with the first dataset Di. The camera I sensor may continue to be adjusted until such time as the dissimilarity between the first dataset Di and the updated dataset D2 falls below the threshold (the dissimilarity may continue to be measured after that point, in case it rises above the threshold again). It will further be appreciated that when updating the second dataset D2, one or more elements of the second dataset that were received at an earlier point in time may be discarded. In one embodiment, a first-in-first-out (FIFO) buffer may be implemented, whereby, in each cycle, the latest received data element from the camera / sensor is added to the buffer and the oldest remaining data element in the buffer is removed, with the updated second dataset then comprising the contents of the buffer. In this way, the second dataset can be understood to comprise a dynamic dataset that is modified in a rolling manner over time. It will also be appreciated that when determining that a particular sensor (e.g. camera) is misaligned, the same adjustments made to re-align the camera may be applied to other sensor(s) or parts of the system. For example, the camera of Figures 4A- C may be attached to a vehicle or platform that also includes one or more other sensors, such as other cameras, or radar sensors. In this case, the alignment of one or more of the other sensors (and / or other settings) may be adjusted in line with the camera. Thus, in detecting that one sensor (the camera) is misaligned, this determination may also be used to correct the alignment of other collocated sensors, as well as the camera itself. As well as using information gained from one sensor to infer and change the state of a second sensor, such information can be used to infer the behaviour of components downstream from the second sensor. For instance, an autonomous system may include two classifiers Ci and C2, that operate on data collected and output by a camera and radar, respectively. In this case, the data dissimilarity may be determined between the operational and training samples input into the classifier G; that is, the algorithm 403 may comprise the classifier Ci and the dissimilarity between the operational data and training data may be computed in a feature space defined by the activation values of Ci. By studying the relationship between performance of the classifier C2 and input dissimilarity computed via Ci, information on the accuracy of the classifier C2 might be obtainable on the basis of data received by the classifier Ci. Following this, one response to obtaining information about the accuracy of C2 might be to initiate the sensor adjustment steps described above. In another embodiment, the system may comprise an autonomous robot with the camera / sensor 401 being mounted on that robot. The robot’s motion may be altered in the event that the dissimilarity between operational data captured by the camera and the data used to configure the algorithm 403 is found to be above a predefined threshold. For example, the autonomous robot may be caused to slow down or stop moving in the event the dissimilarity is above the threshold. The robot (e.g. a Mars rover) might be using the camera and an associated classifier to search for rocks from which to take samples. Data dissimilarities measured between the images captured on the camera during operation and those used for training the classifier might indicate that the level of performance of the classifier will be lower than required. Given this indication of an inadequate level of performance, the rover may move on to another outcrop to try and reduce the dissimilarity to a satisfactory level, or the rover may attempt to change the viewing angle of the camera. In other embodiments, a fail-safe mechanism may be activated when the dissimilarity measure reaches a threshold. The fail-safe might comprise reverting the autonomous robot to manual control by an operator, for example. It will be appreciated that the above processes could be implemented using Reinforcement Learning (RL), whereby a positive reward is obtained by reducing the magnitude of the dissimilarity measure. As an example, consider the case where the system includes a camera mounted on an autonomous vehicle and images from the camera are fed into a particular algorithm, such as a classifier. A second algorithm would be incorporated into the system which would make use of reinforcement learning (RL). The RL objective can be defined as identifying a viewing angle or rotational alignment of the camera that minimises the dissimilarity between the images collected during operation and the images used for training / configuring the classifier. By providing measures of data discrepancies, embodiments described herein also define a framework for creating verifiable specifications for Al-based systems. In this respect, embodiments serve to build on the methods of verifying the performance of autonomous systems as described in UK Patent Application No. GB 2011580.4. Here, a classifier resilience function (CRF) is used to measure the generalisation capability of a machine learning algorithm. An example CRF is shown in Figure 5. The CRF captures the relationship between performance, denoted Y, and some measure of test dataset dissimilarity, denoted Z in Figure 5. The CRF shows the extent to which performance is maintained as the algorithm is tested on a series of datasets which, by some measure, are progressively further from the dataset on which the algorithm was trained. By doing so, the CRF indicates how well a machine learning algorithm, such as an ANN, can generalize to progressively more difficult datasets i.e. datasets that differ from the training set to an increasing degree. In doing so, the CRF provides a basis for the verification of the performance of the algorithm. Figure 6 shows a flow-chart of how the CRF may be employed in embodiments described herein. In this example, we again consider the case of a machine learning classifier, but it will be understood that the same approach is applicable to different types of algorithm, and not only machine learning algorithms. The process begins in step S601 by estimating the CRF of the classifier offline. In step S603, a set of operational data is captured and in step S605, the dissimilarity between the operational dataset (acting in the role of Dz) and the training data used to train the classifier (acting in the role of Di) is determined using the multi-granular approach described above. If the measured dissimilarity value is within an acceptable range (e.g. below a predetermined threshold) (step S607), a prediction of the expected performance of the classifier, as defined by the classifier’s accuracy in classifying different input items, may be made in line with the approach described in UK Patent Application No. GB 2011580.4 (step S609). In the event the dissimilarity is found to be above the threshold, actions such as those described above may be taken to reduce the dissimilarity by changing the mode of operation used by the system to generate data (step S611). Once the adjustment is made, steps S603 to S607 may be repeated and if the newly measured dissimilarity is below the threshold, a prediction of the system performance may be made in line with the approach set out in in UK Patent Application No. GB 2011580.4 (step S609). Thus, Figure 6 shows how the self-verification process described in UK Patent Application No. GB 2011580.4 may be generalised by adding an additional step in which an attempt to reduce data dissimilarities is made, if needed, before making predictions of model performance. This extension is a consequence of the idea of interpreting data dissimilarities as the sum of two contributions: i) the extent to which the mode of data generation used by a system during operation differs from the mode used by the system during training (when collecting comparative data for a component algorithm), ii) domain shift due to other factors, for example environmental factors. It will be appreciated that the ability to verify machine learning systems as described above is important for complex systems. It might be vital, for example, to verify how well ANN-based algorithms generalise to perturbed and unseen data. ANNs, for example, are used in object classification across a range of modalities including visible and IR imagery, and sonar signals. Operationally, ANNs can be applied to many diverse domains. In the maritime domain, object classification is required as part of the detection, recognition and identification process of potentially hazardous objects and surface vessels. It is crucial to the safety of Maritime Autonomous Systems (MASs) including both Unmanned Surface Vehicles (USVs) and Unmanned Underwater Vehicles (UUVs). The number of potential applications for MASs continues to grow across a range of sectors, including oil and gas, where both unmanned surface and underwater vehicles could be used for pipeline inspection; the growing offshore wind farm industry, where MASs will be applicable for routine inspection and maintenance; the security and defence industry; and the scientific community for collecting important environmental data. Situational awareness is crucial to the safety of such systems; it follows that any object classification algorithm employed by such systems must behave as expected, even in an uncontrolled environment such as the sea or ocean. Embodiments described herein can assist in assuring the safety and reliability of USV / MAS operations and provide meaningful verification evidence that will be required for regulation, certification and assurance purposes. It will further be appreciated that object classification might be required for the operation of Unmanned Aerial Vehicles (UAVs). The objects to be classified might be located on land, or in the sea or air. The atmosphere is another uncontrolled environment where many factors will affect the data collected by sensors, such as imagery, and which may have an impact on the performance of an associated classifier. As well as their use as a tool within methods for directly verifying machine learning algorithms, dissimilarity measures can be used to verify the properties of environments used to test such algorithms. Taking the example of autonomous vehicles, it is almost impossible to collect sufficiently diverse sets of test data that reflect the wide variety of situations in the physical world that an autonomous vehicle may confront. For this reason, synthetic data and synthetic environments become quintessential elements in the development of safety systems. For instance in the field of self-driving cars, developers need to be able to mimic driver behaviour in virtual environments to test their models across a broad set of settings and situations without having to build and deploy fleets of vehicles. Synthetic data fills in the gaps not covered by real-world data by means of modelling, e.g. using computer-generated roads, traffic lights, parking spaces, etc. Therefore, huge sets of synthetic data might need to be generated. In this regard, the multi-granular data dissimilarity measure as described herein may be employed to assess the realism of synthetic data-generation mechanisms by comparing sets of artificially-created data with real-world samples, showing to what extent synthetic data is close to or far from real-world data. The embodiments described herein may thus be employed to contribute to the verification of synthetic simulation environments. In addition to assessing the realism of synthetically generated data, embodiments described herein may utilise the dissimilarity measure to actively select out-of-distribution data, which could then be employed to re-train underperforming Al-based components included within a system. For example, an autonomous vehicle may be fitted with a camera or other sensor that captures data for input to a classifier or other algorithm. The autonomous vehicle might continue moving until it reaches a point where the dissimilarity between the data being captured and that used to configure / train the algorithm reaches or exceeds a predetermined threshold. At this point, the data being captured may be stored, with the intent of subjecting it to groundtruth labelling and using it for retraining purposes at a later point in time. Here then, is a further use case, where the data dissimilarity measure as described herein may be used to design Al algorithms that are robust to perturbed data (due to changing environmental conditions, for example) by select training data for data augmentation. The multi-granular dissimilarity measure described herein may also be used for anomaly detection. It was noted above that Di or D2 could contain a single member. Let i) Di be a dataset comprising multiple data items used to train a machine algorithm, ii) Ds be a dataset comprising multiple data items used to validate a machine learning algorithm during training, and iii) D2 contain a single data item acquired during system operation. If the dissimilarity measure is used to return the dissimilarity between D2 and Di, then this value can be used to make a binary decision, perhaps by applying a threshold, as to whether the single data item contained within D2 is anomalous or not. In other words, a judgement would be made as to whether the single data item contained within D2 is anomalous with respect to the normality represented by Di. Where Di corresponds to the ML training dataset, then anomaly detection performed in this manner is a form of out-of-distribution detection, with data items being classified as out-of-distribution relative to the training dataset. Out-of-distribution (OOD) detection can be used for making decisions about how to respond to the output of an ML algorithm given the detected status of an input data item. As well as detecting whether a single data item is out-of-distribution, the dissimilarity measure may be used to provide evidence as to whether a dataset D2 which comprises multiple items is not drawn from the same distribution as Di. By way of example, a system might respond to OOD detection by: - Deactivating the classifier - Employing an alternative classifier which is more robust, but not the first-choice because it is computationally more expensive. - Substituting a different imaging modality; for example switch from a classifier applied to visible-band imagery to one applied to infrared imagery - Bringing a mobile system to a halt. Although in the embodiments described above, the dissimilarity measure is used to quantify the discrepancy between two datasets Di and D2 using a reference set Ds, it will be noted that the multi-granular approach could be employed, broadening its range of variants, to directly quantify the dissimilarity between a single dataset of interest Di and a reference set Ds. In the following, we denote this general multi-granular measure as dmg(D1,Ds). Within the mathematical formulation given above, we calculate mean distances at the third level of granularity. For Di, the mean distance is given as: We could, among other possibilities, define dmg as: d-mg >^s) In other words, what was treated earlier as a distance, in that it was obtained as an intermediate value, to be combined with other intermediate values when calculating a dissimilarity measure, can be used as a dissimilarity measure in its own right. Taking this further, a general multi-granular discrepancy measure between two datasets Di and D2 may be computed by first calculating d^^D^Ds) and dmg(D2,Ds), and consequently applying further operations to these two dissimilarities. An analogous strategy could be adopted when comparing more than two test datasets. Figure 7 shows a further embodiment in which a dissimilarity measure as described herein may be used in controlling the operation of a system, where that system comprises a processing pipeline having a first sub-component 701 and one or more other sub-components 703. The first sub-component may be configured to process data received from the one or more other sub-components 703 and to output the processed data into a software algorithm 705. The software algorithm 705 may function in a similar manner to the algorithm 403 of Figure 4, having been trained or configured with a first dataset to generate an output 707. The first dataset may be understood to correspond to the dataset Di as described earlier, with a further dataset Ds drawn from the same distribution as Di being used in validating the training of the algorithm. The processed data output by the first sub-component and fed into the software algorithm may be understood to correspond to the second dataset D2 as described earlier. As in the embodiment of Figures 4A - 4C, a comparison algorithm 709 is used to determine a measure of dissimilarity 711 between the processed data output by the first sub-component 701 and fed into the software algorithm, and the first dataset used to train or configure the software algorithm. Based on the measured dissimilarity 711, an adjustment 713 may be made to one or more elements of the processing pipeline. Each sub-component may comprise a processing unit function tasked with performing a particular calculation or executing a particular algorithm. By way of example, subcomponents may include, for example, components used for image and / or video processing, data encryption or compression I decompression. The one or more elements of the processing pipeline may include processing devices (including the various sub-components themselves), as well as other physical devices including sensors, cameras, actuators etc. The adjustment 713 might include adding or adjusting a post-processing algorithm, such as a de-noising algorithm, at the output of the first sub-component 701. The adjustment may be such as to reduce the dissimilarity measure between the (subsequent) output of the first sub-component 701 and the first dataset used to train or configure the software algorithm 705. As one example, the adjustment 713 might involve selecting and dampening a particular frequency band of signal sinusoids in the output from the first sub-component 701 using a Fourier transform to the frequency domain. More generally, where a post-processing algorithm has adjustable parameters, a search of the parameter space may be conducted in order to identify the selection of parameter values which most reduce the dissimilarity. It will be appreciated that features of the embodiment shown in Figures 4A - C and Figure 7 may be combined. For example, referring back to Figures 4A - C, the adjustment 411 made to the system may include adjusting a post-processing algorithm applied to the output of the camera or sensor. Figure 8 shows schematically how this might be implemented in practice. As before, the camera I sensor 801 is used to capture data from the surroundings. In contrast to Figure 4, before the data capture by the camera I sensor is input to the software algorithm 805, the captured data undergoes post-processing 803 to obtain processed data. The software algorithm may be trained to process input data to product an output 807, using a first dataset corresponding to Di as described above. A further dataset Ds may be used for validating the training. During operation, the processed data that is fed into the software algorithm (and which can be understood to correspond to the dataset Dz as described above) is compared with the first dataset Di to determine a measure of dissimilarity 811. The comparison is carried out using the comparison algorithm 809, and may be based on recording the activation of neurons within the software algorithm 805 when the respective datasets Di, Dz and Ds are input into that algorithm. Depending on the magnitude of the dissimilarity, an adjustment 813 may be made to the system by adjusting 815 the post-processing applied to the captured data and / or by adjusting 817 one or more components of the system such as the camera I sensor. Thus, adjustments may comprise both mechanical adjustments made to the camera / sensor (e.g. adjustment of position, tilt, rotational alignment etc.) and / or digital adjustment of the data output from that camera I sensor, prior to its being used by the software algorithm 805. In another similar example to that shown in Figure 8, the software algorithm 805 may comprise a classifier that receives a signal describing the data contents of an output channel from a router within a data network. Here, the signal may correspond to the processed data (Dz) shown in Figure 8. The signal may represent the number of bits output per second over a period of time. Classes may be assigned depending on the nature of the signal. The signal may be de-noised before classification; however, if the signal representing the output of the router channel has a high dissimilarity compared to the dataset (Di) used to train the classifier, an adjustment may be made to the system. Such adjustment may take the form of altering the inputs into the router; for example, it might be that one of the router inputs is malfunctioning and should be reconfigured - just as a camera might malfunction and need adjusting. Alternatively, the router inputs may be adjusted digitally and / or mechanically. In a further example, the sampling frequency used to extract a signal from the router output data might also be adjusted. In further embodiments, data processed by more than one ML algorithm may be analysed to guide the way in which the system generates data. For instance, there may be two different cameras on board the same or different platforms, and each of these cameras might feed its own, dedicated machine learning algorithm. Two different dissimilarity measurements could be made relative to each of the machine learning algorithms, and these dissimilarities could be integrated in some way to influence the behaviour of a system. Thus multiple dissimilarity measurements of the data produced by multiple sensors may be used to guide the way in which the system generates data. Implementations of the subject matter and the operations described in this specification can be realized in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be realized using one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). While certain embodiments have been described, these embodiments have been presented by way of example only and are not intended to limit the scope of the invention. Indeed, the novel methods, devices and systems described herein may be embodied in a variety of forms; furthermore, various omissions, substitutions and changes in the form of the methods and systems described herein may be made without departing from the spirit of the invention. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the invention.
Claims
1. A method for controlling the operation of a system, the system having one or more sensors, wherein one of the sensors is a first sensor configured to capture sensor data to be used as input into a software algorithm, the method comprising:determining a measure of dissimilarity between a first dataset Di used for training or configuring the software algorithm and a second dataset D2 captured by the first sensor and input into the software algorithm; andbased on the measure of dissimilarity, adjusting one or more operating parameters of the system.
2. A method according to claim 1, wherein determining the measure of dissimilarity comprises comparing each of the first dataset Di and the second dataset D2 with a reference dataset Ds.
3. A method according to claim 2, wherein the first dataset Di and the second dataset D2 are compared with the reference dataset Ds at one or more levels of granularity.
4. A method according to claim 3, wherein the first dataset Di comprises a first set of data elements, the second dataset D2 comprises a second set of data elements and the reference dataset Ds comprises a third set of data elements;wherein for each level of granularity, comparing the first dataset Di with the reference dataset Ds at the level of granularity comprises computing, for one or more elements of the first dataset Di, a distance between the respective element and one or more subsets of the reference dataset Ds; andwherein for each level of granularity, comparing the second dataset D2 with the reference dataset Ds at the level of granularity comprises computing, for one or more elements of the second dataset D2, a distance between the respective element and one or more subsets of the reference dataset Ds\wherein the size of the subsets of the reference dataset Ds are different for each level of granularity.5: A method according to claim 4, wherein the method comprises, for each level of granularity, deriving a respective measure of dissimilarity based on the distances between the data elements and the subsets of the reference dataset at that level.
6. A method according to claim 5, wherein the measure of dissimilarity is computed using the respective measures of dissimilarity from each level of granularity.
7. A method according to any one of claims 4 to 6, wherein computing the distance measures at a first level of granularity comprises:projecting the elements of the first dataset Di, the second dataset D2 and the reference dataset Ds onto a feature space;determining a centroid Asc of the projections of the elements of the reference dataset Ds on the feature space;for each one of the elements of the first dataset Di, determining a distance in the feature space between the projection of the respective element and the centroid Asc\ andfor each one of the elements of the second dataset D2, determining a distance in the feature space between the projection of the respective element and the centroid ASC-8. A method according to claim 7, wherein computing the distance measures at a second level of granularity comprises:determining, for each subset of the reference dataset Ds, a centroid of the projections of the elements in the respective subset on the feature space; andfor each one of the elements of the first dataset Di, determining a distance in the feature space between the projection of the respective data element and the closest centroid from among the centroids of the different subsets; andfor each one of the elements of the second dataset D2, determining a distance in the feature space between the projection of the respective data element and the closest centroid from among the centroids of the different subsets.
9. A method according to claim 8 wherein computing the distance measures at a third level of granularity comprises:for each one of the elements in the first dataset Di.identifying a number of k elements of the reference dataset Ds, whose projections in the feature space are closest to the projection of the respective element of the first dataset Di, each set of k data elements defining a respective one of the subsets of the reference dataset Ds, anddetermining an average distance between the projection of the element of the first dataset D2 and the projections of the k data elements in the respectivesubset of the reference dataset Ds', andfor each one of the elements in the second dataset D2:identifying a number of k elements of the reference dataset Ds, whose projections in the feature space are closest to the projection of the respective element of the second dataset D2, each set of k data elements defining a respective one of the subsets of the reference dataset Ds, anddetermining an average distance between the projection of the element of the second dataset D2 and the projections of the k data elements in the respective subset of the reference dataset Ds.
10. A method according to any one of the preceding claims, wherein the software algorithm is a machine learning algorithm.
11. A method according to claim 10, wherein the machine learning algorithm is an artificial neural network, ANN, and wherein projections in the feature space are defined by activation values of neurons in the ANN when the respective datasets are input to the ANN.
12. A method according to claim 11, wherein the feature space is defined by activation values of a subset of the neurons contained within inner layers of the ANN.
13. A method according to any one of the preceding claims wherein the first dataset is a dataset used to train the machine learning algorithm and the reference dataset Ds is a validation dataset used in validating the training of the algorithm.
14. A method according to any one of the preceding claims, further comprising:updating the second dataset D2 to include subsequent data captured by the first sensor;obtaining an updated measure of dissimilarity between the first dataset Di and the updated second dataset D2, andbased on the updated measure of dissimilarity, further adjusting the one or more operating parameters of the system.
15. A method according to any one of the preceding claims, wherein the operating parameters are adjusted such as to reduce the dissimilarity between the first dataset Di and the second dataset D2.
16. A method according to claim 15, wherein the operating parameters are adjusted using reinforcement learning to reduce the dissimilarity between the first dataset Di and the second dataset D2.
17. A method according to any one of claims 14 to 16, wherein updating the second dataset D2 comprises discarding one or more elements of the second dataset D2 received at an earlier point in time.
18. A method according to any one of the preceding claims, wherein the one or more operating parameters are adjusted in the event that the measure of dissimilarity is determined to be above a pre-defined threshold.
19. A method according to any one of the preceding claims, wherein in the event the measure of dissimilarity is determined to be below a pre-defined threshold, the method comprises:predicting, based on the measure of dissimilarity, an expected performance of the machine learning algorithm when the second dataset D2 is input to the machine learning algorithm.
20. A method according to any one of the preceding claims, wherein the system is mobile and adjusting the one or more operating parameters comprises adjusting the motion of one or more components of the system.
21. A method according to claim 20, wherein adjusting the motion of one or more components of the system comprises:changing a speed or direction of motion of the one or more components and / or adjusting the motion of a platform on which the sensors are mounted.
22. A method according to any one of the preceding claims, wherein adjusting the one or more operating parameters comprises adjusting one or more acquisition settings of the one or more sensors.
23. A method according to claim 22, wherein adjusting the acquisition settings comprises adjusting one or more acquisition settings of the first sensor.
24. A method according to any one of the preceding claims, wherein the systemcomprises at least two sensors, and adjusting the operating parameters comprises adjusting one or more acquisition settings of a second sensor different from the first sensor.
25. A method according to claim 24, wherein the at least one sensor and the second sensor comprise different types of sensor.
26. A method according to claim 25, wherein the at least one sensor and the second sensor comprise a camera and a radar sensor.
27. A method according to any one of the preceding claims, wherein the system comprises at least two sensors, and adjusting the operating parameters comprises switching from capturing sensor data from the first sensor to capturing sensor data from a second sensor different from the first sensor.
28. A method for controlling the operation of a system, the system comprising a processing pipeline including a first sub-component and one or more other subcomponents, the first sub-component being configured to process data received from the one or more other sub-components and to output the processed data as input to a software algorithm, the method comprising:determining a measure of dissimilarity between a first dataset D! used for training or configuring the software algorithm and a second dataset D2, the second dataset comprising the processed data output by the first sub-component; andbased on the measure of dissimilarity, adjusting one or more elements of the processing pipeline.
29. A method according to claim 28, wherein adjusting the one or more elements of the processing pipeline comprises:performing a search of a parameter space of each element to identify one or more elements of the processing pipeline that when adjusted will minimise the measure of dissimilarity.
30. A method according to claim 28 or 29, wherein adjusting the one or more elements of the processing pipeline comprises adjusting the configuration of a de-noising algorithm to be applied to the output from the first sub-component.
31. A method according to claim 30, wherein adjusting the one or more elements comprises:determining whether or not to reduce the contribution of high spatial frequency 2D sinusoids in the second dataset and / ordetermining an extent by which to reduce the contribution of the high spatial frequency 2D sinusoids.
32. A method according to any one of claims 28 to 30, wherein adjusting the one or more elements of the processing pipeline comprises adjusting a means by which the first sub-component processes the data received from the one or more other subcomponents.
33. A method according to any one of claims 28 to 32, wherein determining the measure of dissimilarity comprises comparing each of the first dataset D? and the second dataset D2 with a reference dataset Ds.
34. A method according to claim 33, wherein the first dataset Di and the second dataset D2 are compared with the reference dataset Ds at one or more levels of granularity.
35. A method according to claim 34, wherein the first dataset Di comprises a first set of data elements, the second dataset D2 comprises a second set of data elements and the reference dataset Ds comprises a third set of data elements;wherein for each level of granularity, comparing the first dataset Di with the reference dataset Ds at the level of granularity comprises computing, for one or more elements of the first dataset Di, a distance between the respective element and one or more subsets of the reference dataset Ds; andwherein for each level of granularity, comparing the second dataset D2 with the reference dataset Ds at the level of granularity comprises computing, for one or more elements of the second dataset D2, a distance between the respective element and one or more subsets of the reference dataset Ds,wherein the size of the subsets of the reference dataset Ds are different for each level of granularity.36: A method according to claim 35, wherein the method comprises, for each level of granularity, deriving a respective measure of dissimilarity based on the distancesbetween the data elements and the subsets of the reference dataset at that level.
37. A method according to claim 36, wherein the measure of dissimilarity is computed using the respective measures of dissimilarity from each level of granularity.
38. A method according to any one of claims 35 to 37, wherein computing the distance measures at a first level of granularity comprises:projecting the elements of the first dataset Di, the second dataset D2 and the reference dataset Ds onto a feature space;determining a centroid Asc of the projections of the elements of the reference dataset Ds on the feature space;for each one of the elements of the first dataset Dh determining a distance in the feature space between the projection of the respective element and the centroid Asc\ andfor each one of the elements of the second dataset D2, determining a distance in the feature space between the projection of the respective element and the centroid39. A method according to claim 38, wherein computing the distance measures at a second level of granularity comprises:determining, for each subset of the reference dataset Ds, a centroid of the projections of the elements in the respective subset on the feature space; andfor each one of the elements of the first dataset Di, determining a distance in the feature space between the projection of the respective data element and the closest centroid from among the centroids of the different subsets; andfor each one of the elements of the second dataset D2, determining a distance in the feature space between the projection of the respective data element and the closest centroid from among the centroids of the different subsets.
40. A method according to claim 39 wherein computing the distance measures at a third level of granularity comprises:for each one of the elements in the first dataset Df.identifying a number of k elements of the reference dataset Ds, whose projections in the feature space are closest to the projection of the respective element of the first dataset Di, each set of k data elements defining a respective one of the subsets of the reference dataset Ds, anddetermining an average distance between the projection of the element of the first dataset Di and the projections of the k data elements in the respective subset of the reference dataset Ds', andfor each one of the elements in the second dataset D2:identifying a number of k elements of the reference dataset Ds, whose projections in the feature space are closest to the projection of the respective element of the second dataset D2, each set of k data elements defining a respective one of the subsets of the reference dataset Ds, anddetermining an average distance between the projection of the element of the second dataset D2 and the projections of the k data elements in the respective subset of the reference dataset Ds.
41. A method according to any one of claims 28 to 40, wherein the software algorithm is a machine learning algorithm.
42. A method according to claim 41, wherein the machine learning algorithm is an artificial neural network, ANN, and wherein projections in the feature space are defined by activation values of neurons in the ANN when the respective datasets are input to the ANN.
43. A method according to claim 42, wherein the feature space is defined by activation values of a subset of the neurons contained within inner layers of the ANN.
44. A method according to any one of claims 28 to 43 wherein the first dataset is a dataset used to train the machine learning algorithm and the reference dataset Ds is a validation dataset used in validating the training of the algorithm.
45. A method according to any one of claims 28 to 44, further comprising:updating the second dataset D2 to include subsequent data output by the first sub-component;obtaining an updated measure of dissimilarity between the first dataset Di and the updated second dataset D2, andbased on the updated measure of dissimilarity, further adjusting the one or more elements of the processing pipeline.
46. A method according to any one of claims 28 to 45, wherein the one or moreelements of the processing pipeline are adjusted such as to reduce the dissimilarity between the first dataset and the second dataset D2.
47. A method according to claim 46, wherein the one or more elements of the processing pipeline are adjusted using reinforcement learning to reduce the dissimilarity between the first dataset Di and the second dataset D2.
48. A method according to any one of claims 45 to 47, wherein updating the second dataset D2 comprises discarding one or more elements of the second dataset D2 received at an earlier point in time.
49. A method according to any one of claims 28 to 48, wherein the one or more elements of the processing pipeline are adjusted in the event that the measure of dissimilarity is determined to be above a pre-defined threshold.
50. A method according to any one of the preceding claims, wherein in the event the measure of dissimilarity is determined to be below a pre-defined threshold, the method comprises:predicting, based on the measure of dissimilarity, an expected performance of the machine learning algorithm when the second dataset D2 is input to the machine learning algorithm.
51. A computer-readable storage medium comprising computer executable instructions that when executed by one or more computers will cause the one or more computers to carry out a method according to any one of the preceding claims.
52. A system comprising:one or more sensors; andone or more computers configured to carry out the method according to any one of claims 1 to 27.
53. A system comprising:a processing pipeline including a first sub-component and one or more other sub-components, the first sub-component being configured to process data received from the one or more other sub-components and to output the processed data as input to a software algorithm; andone or more computers configured to carry out the method according to anyone of claims 28 to 50.
54. An autonomous vehicle comprising a system according to claim 52 or 53.01 09 25AMENDMENTS TO THE CLAIMS HAVE BEEN FILED AS FOLLOWS:-CLAIMS1. A method for controlling the operation of a system, the system having one or more sensors, wherein one of the sensors is a first sensor configured to capture sensor data5 to be used as input into a software algorithm, the method comprising:determining a measure of dissimilarity between a first dataset Di used for training or configuring the software algorithm and a second dataset D2 captured by the first sensor and input into the software algorithm, wherein determining the measure of dissimilarity comprises comparing each of the first dataset Di and the second dataset10 D2 with a reference dataset Ds', andbased on the measure of dissimilarity, adjusting one or more operating parameters of the system.
2. A method according to claim 1, wherein the first dataset Di and the second dataset 15 D2 are compared with the reference dataset Ds at one or more levels of granularity.
3. A method according to claim 2, wherein the first dataset Di comprises a first set of data elements, the second dataset D2 comprises a second set of data elements and the reference dataset Ds comprises a third set of data elements;20 wherein for each level of granularity, comparing the first dataset Di with thereference dataset Ds at the level of granularity comprises computing, for one or more elements of the first dataset Di, a distance between the respective element and one or more subsets of the reference dataset Ds; andwherein for each level of granularity, comparing the second dataset D2 with the25 reference dataset Ds at the level of granularity comprises computing, for one or more elements of the second dataset D2, a distance between the respective element and one or more subsets of the reference dataset Ds;wherein the size of the subsets of the reference dataset Ds are different for each level of granularity.304: A method according to claim 3, wherein the method comprises, for each level of granularity, deriving a respective measure of dissimilarity based on the distances between the data elements and the subsets of the reference dataset at that level.35 5. A method according to claim 4, wherein the measure of dissimilarity is computedusing the respective measures of dissimilarity from each level of granularity.01 09 256. A method according to any one of claims 3 to 5, wherein computing the distance measures at a first level of granularity comprises:projecting the elements of the first dataset Di, the second dataset D2 and the5 reference dataset Ds onto a feature space;determining a centroid Ac of the projections of the elements of the reference dataset Ds on the feature space;for each one of the elements of the first dataset Di, determining a distance in the feature space between the projection of the respective element and the centroid 10 a£; andfor each one of the elements of the second dataset D2, determining a distance in the feature space between the projection of the respective element and the centroid15 7. A method according to claim 6, wherein computing the distance measures at asecond level of granularity comprises:determining, for each subset of the reference dataset Ds, a centroid of the projections of the elements in the respective subset on the feature space; andfor each one of the elements of the first dataset Di, determining a distance in 20 the feature space between the projection of the respective data element and the closest centroid from among the centroids of the different subsets; andfor each one of the elements of the second dataset D2, determining a distance in the feature space between the projection of the respective data element and the closest centroid from among the centroids of the different subsets.
258. A method according to claim 7 wherein computing the distance measures at a third level of granularity comprises:for each one of the elements in the first dataset Df.identifying a number of k elements of the reference dataset Ds, whose30 projections in the feature space are closest to the projection of the respective element of the first dataset Di, each set of k data elements defining a respective one of the subsets of the reference dataset Ds, anddetermining an average distance between the projection of the element of the first dataset Di and the projections of the k data elements in the respective35 subset of the reference dataset Ds', andfor each one of the elements in the second dataset D2:01 09 25identifying a number of k elements of the reference dataset Ds, whose projections in the feature space are closest to the projection of the respective element of the second dataset D2, each set of k data elements defining a respective one of the subsets of the reference dataset Ds, and5 determining an average distance between the projection of the elementof the second dataset D2 and the projections of the k data elements in the respective subset of the reference dataset Ds.
9. A method according to any one of the preceding claims, wherein the software 10 algorithm is a machine learning algorithm.
10. A method according to claim 9, wherein the machine learning algorithm is an artificial neural network, ANN, and wherein projections in the feature space are defined by activation values of neurons in the ANN when the respective datasets are input to15 the ANN.
11. A method according to claim 10, wherein the feature space is defined by activation values of a subset of the neurons contained within inner layers of the ANN.20 12. A method according to any one of the preceding claims wherein the first dataset isa dataset used to train the machine learning algorithm and the reference dataset Ds is a validation dataset used in validating the training of the algorithm.
13. A method according to any one of the preceding claims, further comprising:25 updating the second dataset D2 to include subsequent data captured by the firstsensor;obtaining an updated measure of dissimilarity between the first dataset D, and the updated second dataset D2, andbased on the updated measure of dissimilarity, further adjusting the one or30 more operating parameters of the system.
14. A method according to any one of the preceding claims, wherein the operating parameters are adjusted such as to reduce the dissimilarity between the first dataset Di and the second dataset D2.3515. A method according to claim 14, wherein the operating parameters are adjusted01 09 25using reinforcement learning to reduce the dissimilarity between the first dataset Di and the second dataset D2.
16. A method according to any one of claims 13 to 15, wherein updating the second 5 dataset D2 comprises discarding one or more elements of the second dataset D2received at an earlier point in time.
17. A method according to any one of the preceding claims, wherein the one or more operating parameters are adjusted in the event that the measure of dissimilarity is 10 determined to be above a pre-defined threshold.
18. A method according to any one of the preceding claims, wherein in the event the measure of dissimilarity is determined to be below a pre-defined threshold, the method comprises:15 predicting, based on the measure of dissimilarity, an expected performance ofthe machine learning algorithm when the second dataset D2 is input to the machine learning algorithm.
19. A method according to any one of the preceding claims, wherein the system is 20 mobile and adjusting the one or more operating parameters comprises adjusting the motion of one or more components of the system.
20. A method according to claim 19, wherein adjusting the motion of one or more components of the system comprises:25 changing a speed or direction of motion of the one or more components and / oradjusting the motion of a platform on which the sensors are mounted.
21. A method according to any one of the preceding claims, wherein adjusting the one or more operating parameters comprises adjusting one or more acquisition settings of 30 the one or more sensors.
22. A method according to claim 21, wherein adjusting the acquisition settings comprises adjusting one or more acquisition settings of the first sensor.35 23. A method according to any one of the preceding claims, wherein the systemcomprises at least two sensors, and adjusting the operating parameters comprises adjusting one or more acquisition settings of a second sensor different from the first01 09 25sensor.
24. A method according to claim 23, wherein the at least one sensor and the second sensor comprise different types of sensor.
525. A method according to claim 24, wherein the at least one sensor and the second sensor comprise a camera and a radar sensor.
26. A method according to any one of the preceding claims, wherein the system10 comprises at least two sensors, and adjusting the operating parameters comprises switching from capturing sensor data from the first sensor to capturing sensor data from a second sensor different from the first sensor.
27. A computer-readable storage medium comprising computer executable instructions15 that when executed by one or more computers will cause the one or more computers to carry out a method according to any one of the preceding claims.
28. A system comprising:one or more sensors; and20 one or more computers configured to carry out the method according to anyone of claims 1 to 26.
29. An autonomous vehicle comprising a system according to claim 28.
Citation Information
Patent Citations
Methods and systems for verifying and predicting the performance of machine learning algorithms
GB2597666A