Training system for AI agents

Abstract

Updating the sample data generator module based on the determination of success or failure or the AI, whether sample data is distinguishable, or whether the sample data meets a realism test. This may be used to improve AI agents, used in robots or autonomous vehicles, by ensuring that the training process meets these criteria. An aspect of the AI agent performance associated with success or failure may be determined to inform further sample data, and cost and reward functions. An artificial neural network may be used, and sample data approximate a scenario such as geographical layout, parameters of static or moving objects within the environment.

Description

The field of vehicle autonomy is growing rapidly, and the requirement for intelligent autonomous agents to control vehicles is well established. The systems in place to train autonomous agents through machine learning techniques require a vast amount of data. Over the past decade, many companies and research institutions have been involved in collecting road use data, and data relating to environments for driving. Such data includes image data and particularly video data, LIDAR data gathered from LIDAR sensors installed on vehicles, satellite data, radar data, and map data containing route and road network data. Training an autonomous agent to drive using Al and machine-learning techniques is inherently data intensive. The event space for events occurring during driving is huge, given the numbers of drivers on the roads, pedestrians in proximity to roads, and the large variety of street signs, road markings, and street furniture. No pair of journeys is ever the same, and agents must be trained to cope with virtually any eventuality. Given the safety-critical nature of autonomous driving, agents must be trained to a very high level of proficiency before they can be deployed in a real-world driving situation. Real-world data is important for understanding the nature of situations that are faced routinely, for training agents to fine tune their behaviour and responses, and for testing agents to gain assurance that safe and acceptable performance has been achieved. This real-world data can take the form of 'mileage accumulation', in which the training and / or testing is performed through extended operation in uncontrolled environments (particularly public roads) such that the system experiences situations that have not been actively set up, but instead are experienced by chance. It could also take the form of discrete scenarios, which could be isolated retrospectively from mileage accumulation data or could be designed or generated by some other means. However, due to the volume of training data required for advancing an autonomous agent to a level in which it can control a vehicle safely, artificial generation of training data provides an excellent tool for systems both in training and testing autonomous agents. Importantly, autonomous agents must receive rigorous training before being able to perform to a level deemed acceptable. As the agents are trained, they require extensive testing to determine how they will act in situations presented to them. Scenario-based testing (SBT) of autonomous agents consists of exposing the agent to a programme of defined test scenarios. The scenarios are specifically selected to provide coverage of a very wide range of permutations of events that may occur during a journey, and that could challenge the agent. The scenarios are performed in a controlled manner; they may be set up on a proving ground or in a simulation, for example. SBT has an advantage in being able to ensure a test programme provides coverage of a very large range of permutations of events and environments that are possible. In both training and testing processes, the agent must be exposed to a large variety of scenarios. The same scenarios may be appropriate both for training the agent and also for testing the agent, and so where we refer to training or testing an agent herein, or to the data used for either purpose, it is intended that the terms are generally interchangeable and it should be understood that the data used for one purpose is also suitable for the other. However, it should also be understood that different scenario data should be used for testing the agent than was used in its training, or else the agent may be tested on specific scenarios for which it is prepared rather than testing its general aptitude to handle new scenarios. SBT also provides for training and testing systems that are able deliberately to target areas where there is reason to conclude that more training would be valuable. For example, performance may be close to some threshold level of acceptability such as one defined within regulations or requirements, or statistical confidence may be determined to be low in the agent successfully (e.g., ‘safely’, or in a manner that provides satisfactory user comfort, or efficiently) navigating through a particular environment or set of events / obstacles. In this case, the agent may be trained repeatedly on similar scenarios / events, so that it learns to navigate those more effectively, raising its performance level over iterations. This approach requires ‘search space optimisation’ (SSO), in which a feedback loop allows new test scenarios to be selected based upon the results of previously undertaken scenarios. However, designing a system providing SBT, and particularly using SSO, poses difficulties. For example, ensuring that such a system generates a sufficient range of realistic environments and events, ensuring that the agent is trained in a way that means it is able to handle a full variety of environments and events, and that the scenarios and events are true-to-life. The present invention aims to overcome or reduce one or more of the difficulties associated with the state of the art. BRIEF DESCRIPTION OF THE INVENTION According to a first aspect of the invention we provide a computer-implemented method of training or testing an Al agent, comprising the following steps: generating a set of first sample data using a sample data generator module; training or testing the Al agent using an Al agent module and the first sample data, and determining a success or failure of the Al agent in performing a task based on the first sample data; and updating the sample data generator module based on the determination of success or failure. The method may further include updating the Al agent module based on the determination of success or failure, to improve the Al agent performance at the task based on the first sample data. Updating the sample data generator module based on the determination of success or failure may involve determining an aspect of the Al agent performance associated with the success or failure, and updating the sample data generator module based on that determination so that further generated sample data is more likely to cause the Al agent to fail when performing a task based on that further sample data. The method may further comprise the steps of: prior to training or testing the Al agent: using a sample data compliance module, determining whether items of the first sample data are distinguishable from items of a set of second sample data accessible from the sample data compliance module, where the sample data compliance module determines compliance of the first sample data where the items of first and second sample data are found not to be distinguishable; updating the sample data generator module based on the determination of compliance or non-compliance. The method may include, where the first sample data is found to be non-compliant, repeating the step of generating a set of first sample data to produce a further set of first sample data and repeating the step of determining whether items of that further set of first sample data are compliant. If the first sample data is found to be compliant, the method may include providing the first sample data to the Al agent module. The method may include either: where the first sample data are found to be compliant, updating the sample data compliance module to reduce the likelihood of those specific first sample data being determined to be compliant in future, using a cost function; or where the first sample data are found to be non-compliant, updating the sample data compliance module to improve the likelihood of those first sample data being determined to be non-compliant in future, using a reward function. The method may include either: where the first sample data are found to be compliant, updating the sample data generator module to increase the likelihood of similar first sample data being generated in future, using a reward function; or where the first sample data are found to be non-compliant, updating the sample data generator module to reduce the likelihood of similar first sample data being generated in future, using a cost function. According to a second aspect of the invention, we provide a computer-implemented method of generating data for training or testing an Al agent, comprising the following steps: generating a set of first sample data using a sample data generator module; using a sample data compliance module, determining whether items of the first sample data are distinguishable from items of a set of second sample data accessible from the sample data compliance module, where the sample data compliance module determines compliance of the first sample data where the items of first and second sample data are found not to be distinguishable; updating the sample data generator module based on the determination of compliance or non-compliance. The method may include, if the first sample data is found to be compliant, providing the first sample data to the Al agent module. The method may include, either: where the first sample data are found to be compliant, updating the sample data compliance module to reduce the likelihood of those specific first sample data being determined to be compliant in future, using a cost function; or where the first sample data are found to be non-compliant, updating the sample data compliance module to improve the likelihood of those first sample data being determined to be non-compliant in future, using a reward function. The second sample data may be generated from sensor data and relates to measurements of properties of objects or of an environment, and is preferably reflective of or approximates a real-world environment. The second sample data may record a scenario for testing or training behaviour within an environment, the record of the scenario including one or more of: a geographical layout of an environment, environmental conditions, parameters of static objects, and parameters of moving objects within the environment. According to a third aspect of the invention, we provide a computer-implemented method of training or testing an Al agent, comprising the following steps: generating a set of first sample data using a sample data generator module; using a sample data compliance module, determining whether items of the first sample data pass a realism test, updating the sample data generator module based on the determination of whether the items of the first sample data were found to pass the realism test, to improve the likelihood of future generated sample data passing the realism test, using the first sample data to train or test an Al agent, determining a success or failure of the Al agent in performing a task based on the first sample data; and updating the sample data generator module based on the determination of success or failure, to improve the likelihood of future generated sample data causing the Al agent to fail a task performed on that data. The method may further include updating the Al agent module based on the determination of success or failure, to improve the Al agent performance at the task based on the first sample data. The method may further include updating the sample data compliance module to alter the realism test, to reduce the likelihood of the generated sample data being determined to pass the realism test. The sample data generator module may use an artificial neural network to generate the first sample data. The first sample data may approximate a scenario for testing or training behaviour within an environment, the scenario including one or more of: a geographical layout of an environment, parameters of static objects, and parameters of moving objects within the environment. The Al agent may be configured to drive a vehicle, or to assist in driving a vehicle. The Al agent may be configured to operate a robotic device. The Al agent may be configured to detect and / or classify an object based on sensor data, wherein the second sample data is generated from sensor data and relates to measurements of properties of the object, and the first sample data is generated by a neural network configured to approximate properties of the object. According to a fourth aspect of the invention, we provide a data processing system comprising a processor configured to perform the steps of any one of the first three aspects. According to a fifth aspect of the invention, we provide a computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of the first three aspects. BRIEF DESCRIPTION OF THE FIGURES In order that the present disclosure may be more readily understood, preferable embodiments thereof will now be described, by way of example only, with reference to the accompanying drawings, in which: FIGURE 1 is a diagrammatic view of a system architecture, embodying the present disclosure; FIGURES 2 and 3 are illustrative diagrams of an example steps of a training or testing process for training or testing an Al agent; FIGURE 4 is a diagrammatic representation of an Al agent module, embodying aspects of the present disclosure; FIGURE 5 is a diagrammatic representation of a sample data generator module, embodying aspects of the present disclosure; FIGURE 6 is a diagrammatic representation of a sample data compliance module, embodying aspects of the present disclosure; FIGURE 7 is a flow diagram representing method steps according to embodiments of the technology; FIGURE 8 is a diagram showing the flow of data around a system according to embodiments of the technology; and FIGURE 9 is a flow diagram representing method steps according to embodiments of the technology. DETAILED DESCRIPTION OF THE DISCLOSURE With reference to the Figures of the appended drawings, we describe computer-implemented methods of training and / or testing an artificially intelligent agent (referred to as an ‘Al agent’), methods of generating data for training or testing an Al agent, a system for performing such training or testing, and a computer program product for the same. By an Al agent, we mean a module or collection of algorithms configured to receive inputs providing data about a task, to process those inputs, and to output a response. Of particular interest are Al agents for driving (or assisting the driving of) vehicles. As input, such agents receive data from sensors such as LIDAR, video cameras, GNSS (e.g., GPS, Galileo, GLONASS), radar, odometers, as is known in the art. In response to those inputs, the Al agent determines information about the environment surrounding the vehicle, including details of the road layout, street signs, traffic signals, other vehicles and pedestrians, for example, and determines how best to control the vehicle to guide it to its destination. It should be understood that other implementations of Al agents are also envisaged. For example, Al agents for controlling robots based on sensor data. Such robots may operate in industrial environments - in factories and on production lines, for sorting objects, for identifying objects, and in all manner of tasks. Robots may also be employed in medical environments, as surgical robots, and in a wide variety of settings. The technology described herein may apply widely to processes fortraining such Al agents to improve their functionality, and on testing trained Al agents to determine whether they are trained sufficiently. With reference to Figure 1, and in broad terms, the system for training or testing Al agents 10 provides three core modules: an Al agent module 12, a sample data generator module 14, and a sample data compliance module 16. The Al agent module 12, is configured to test or train an Al agent using sample data. In a training process, the Al agent module 12 updates the behaviour of the Al agent by processing sample data to learn patterns in the data and update its outputs accordingly, during a training process. In such a setting, the sample data discussed herein is ‘training data’ for the purpose of training the Al agent, and those terms (“training data” and “sample data”) may be used interchangeably. Note that in other settings, the Al agent module 12 is used for testing the performance of a trained Al agent, and in that case the Al agent module 12 may perform no training function as the Al agent may not be changed or updated during the process. In such a case, the sample data may be referred to as “test data” since it is used in testing the Al agent. In the context of this specification, the terms “sample data”, “test data” and “training data” are considered to relate to the same types of data, used in the same way in terms of simulating tasks for an Al agent to perform. Where Al agent training is involved, the Al agent module 12 may provide a machine learning framework for training the Al agent. The Al agent module may comprise one or more artificial neural networks (ANN), for example, and training may involve feeding labelled training data as input to the ANN, and updating the ANN using backpropagation for example. Techniques for training an ANN using training data are well known in the art and will not be discussed here in detail. The second core module is the sample data generator module 14. The sample data generator module 14 is configured to generate sample data to be used by the Al agent module 12 in training or testing the Al agent. The sample data generator module 14 is configured to generate a variety of data representing the data likely to be received for processing by the Al agent in use.. For example, in the context of an Al agent for use in driving an autonomous vehicle, it is necessary for the Al agent to be capable of many tasks including navigation, identifying street signs and traffic signals and responding accordingly, identifying other vehicles and their behaviour, identifying and responding to pedestrians, and many other tasks and behaviours. In other words, the search space is vast, and in order to train the Al agent to perform appropriate responses, or to optimise its responses, it is necessary to expose the Al agent to a very large number of different scenarios, and to learn and modify its behaviour in response to the potential outcomes from each situation, based on a large variety of potential actions taken. The vast range of possibilities to be faced by the Al agent, which must be covered broadly with the sample data in order for the Al agent to be competent to act appropriately, is called the search space. The task of covering the search space with an appropriate selection of sample data is known as search space optimisation, and is traditionally performed using an algorithmic approach, involving first characterising system behaviour to interpolate between the results of scenarios that have been undertaken, and determining where in the performance ‘gaps’ can be predicted. Subsequently, the optimisation process selects where within the space defined by the parameter ranges available, the new scenarios should be selected in order to test and improve the Al agent performance most efficiently. Therefore, in such a setting, the sample data generator module 14 is operable to generate artificial scenarios for use in training the Al agent, the scenarios including parameters defining at least one of: an environment layout, a layout of objects within an environment (either stationary, or moving, or a mixture of both stationary and moving objects), and environmental conditions such as lighting conditions (details of the lighting due to the sun being positioned low in the sky, an absence of lighting, or artificial lighting), low visibility conditions due to fog, smoke or precipitation, properties of the road surface (such as its frictional grip, wet conditions, icy conditions). The parameters may also include parameters relating to the vehicle itself, such is its dimensions and performance, degradation, and the like. Figures 2 and 3 of the drawings provide illustrative examples of aspects of scenarios which may be included in sample data; road layouts, objects positioned relative to the roads (e.g., trees, buildings). An Al agent configured to drive a vehicle may be required to identify objects such as these, and corners of roads forming junctions, for example, as a standard part of their task. In embodiments of the technology, the sample data generator module 14 comprises one or more ANN configured to output parameters representing items of sample data. There is little benefit in generating large sets of similar data on which the Al agent performs well. Doing so provides a false impression of the ability of the Al agent, whereas in reality it may have become specialised at a small subset of the scenarios it may encounter during use. To avoid this, the sample data generator module 14 is incentivised to generate sets of sample data which cause the Al agent to perform poorly. Note that where the term “set” is used, it is intended to mean a set of one or more items (i.e., it may contain a single item). To achieve this, a feedback loop is formed between the Al agent module 12 and the sample data generator module 14. The sample data generator module 14 communicates a set of generated sample data 13’ to the Al agent module 12 to be used in training or test the Al agent. Following the training or testing, the Al agent module 12 communicates data about the performance of the Al agent 13” back to the sample data generator module 14. This data may include an indication of the success or failure of the Al agent to perform successfully on the data. The indication may include data about the Al agent performance where it is close to an acceptance / rejection threshold, where statistical confidence in its performance is low, or where there are discontinuities in its outputs as the inputs are varied (e.g., where an Al Agent acting as an autonomous driver is on the threshold of a go / no-go decision at a junction, where small changes in variables can result in highly divergent responses). Where the sample data generator module 14 receives feedback that the Al agent is successfully performing on the data being generated, the sample data generator module 14 is penalised, and in this way it is incentivised to change the sample data being generated. In other words, it is of more benefit to the system 10 for the generated sample data to challenge the Al agent so that it learns from the new sample data. This approach involves using a cost function to penalise the sample data generator module 14 so as to shape its performance, and incentivise it to generate data on which the Al agent fails to perform well. In addition, or alternatively, where the feedback is that the Al agent is failing to perform well on the sample data being generated, the sample data generator module 14 is incentivised to generate similar data again, using a reward function. In these ways, the sample data generator module 14 may adapt to generate a range of data which is likely to cause similar underperformance in the Al agent. A benefit of the systems described herein is that the sample data generator module 14 changes the sample data generated over time, adapting to the performance of the Al agent module 12 to find weaknesses in the Al agent performance on certain types of data, and to generate data on which the Al agent performs weakest. In some embodiments of the technology, in which the Al agent module 12 adapts the behaviour of the Al agent to train it on the data, the sample data generator module 14 is used in conjunction with the Al agent module 12, such that the two modules engage in a competitive ‘game’, each attempting to beat the other. Each module learns i.e. optimising weights and / or biases within their respective ANN(s) in real time as the sample data continue to be generated and training carried out. In other words, within this described architecture, the sample data generator module 14 generates one or more sets of sample data to challenge the Al agent module 12. Each epoch in which training or testing is run, and learning undertaken, may consist of either a single scenario, or a batch of multiple scenarios. As described above, the sample data generator module 14 learns via a cost function and / or a reward function that penalises the sample data generator module 14 for providing sample data that fail to uncover interesting or concerning behaviour in the Al agent, to show that there are issues with its performance which require improvement and / or rewards the sample data generator module 14 for providing data that do uncover such behaviour. Where the Al agent is being trained, the Al agent module 12 reinforcement learning, on the other hand, is penalised for failing to provide the desired performance, or rewarded for succeeding in providing it. The net result is a ‘game’ between the two modules, where the sample data generator module 14 is attempting to beat the Al agent module 12 by finding ways to catch it out, but the Al agent module 12 is attempting to beat the sample data generator module 14 by avoiding being caught out and succeeding on the tasks being set based on the generated sample data. As the cost / reward functions trigger reinforcement learning as the game progresses, therefore each subsystem will get progressively better at its role, with such improvements happening on each epoch / feedback loop and with the learning occurring in a way that is scalable such that vast numbers of sample data can be used and significant learning can result. Over time, as the Al agent sees sample data for which it is underperforming, its performance will improve as it is optimised using machine learning techniques, until it becomes proficient at operating on those data and similar data. So, in broad terms, the sample data generator module 14 is updated based on the determination of success or failure of the Al agent module 12, to improve the likelihood of future generated sample data causing the Al agent to fail a task performed on that data. Similarly, where the Al agent module 12 performs training on the Al agent to update its behaviour, the Al agent module 12 is also updated based on the determination of success or failure, to improve the Al agent performance at the task based on that sample data. It should be understood that while reference is made to the sample data generator module 14 generating sets of sample data, it may be the case that this generation of sample data involves selecting subsets of sample data from a database of sample data. So, in this manner, there may be a large database of potential sample data, and the sample data generator module 14 may be configured to select a subset of those data to be used. As an example, where training an Al agent to park a vehicle in a designated parking space within a car park environment, the Al agent may be under-trained in manoeuvring the vehicle within a certain restricted space requiring a multi-point turn. The failure of the Al agent to park the vehicle correctly in an efficient time leads to a deemed failure of the task, and the Al agent module 12 provides feedback of this to the sample data generator module 14, resulting in that module being rewarded. The sample data generator module 14 is therefore incentivised to produce similar sample data. Over time, the Al agent module 12 will receive more similar data from the sample data generator module 14, and so the Al agent will learn from these scenarios until it becomes adept at completing the parking manoeuvre successfully. At this point, the feedback to the sample data generator module 14 is that the Al agent is completing the tasks successfully, and so the sample data generator module 14 receives a penalty. In turn, this penalty results in the sample data generator module 14 generating different types of sample data focussing on different scenarios, ensuring that the Al agent is trained or tested on a wide range of scenarios over time. ANNs are able to characterise complex multidimensional responses to inputs through curve-fitting. By obtaining feedback from the Al agent module 12 on the performance based on the sample data being generated, the search space may be optimised by learning the Al agent’s performance using an ANN. This allows identification of discontinuities within the curve ‘surface’ generated over the search space. This is extremely difficult for traditional algorithmic curve fitting approaches to model accurately, and so in this way, the use of ANN with feedback, provides a distinct advantage in modelling performance of the Al agent training. Further, through reinforcement learning, the sample data generator module 14 is able to adapt and learn dynamically, ‘rewarding’ the system for selecting sample data that uncovers significant results (such as the Al agent failing to meet, or being close to failing to meet, a quantitative requirement). In this way, the ANN ‘learns’ the Al agent behaviour and becomes extremely effective at finding valuable training results and new sample data on which to train the Al agent. This leads to great efficiencies in the training process. The same benefits are also derived when testing a trained Al agent. So, there is a distinct advantage both in employing an ANN to learn the behaviour of the Al agent from the feedback from the Al agent module 12, and in using the (or another) ANN to generate new sample data to be used in future training or testing. Whereas in the above description the Al agent module 12 and the sample data generator module 14 are described as distinct modules, in embodiments of the technology, a single ANN may receive data relating to the training of the Al agent as an input (i.e , feedback received from the Al agent module 12), and outputs a new set of sample data as an output. In embodiments, this may require additional processing of the inputs and outputs outside the ANN. As such, the ANN would selforganise, as part of the reinforcement learning process, to maximise the number of sets of sample data selected that yield results that are of benefit to the training. It should also be noted that in embodiments of the technology, the sample data generator module 14 may comprise multiple ANNs, for generating different aspects of the sample data. For example, a first ANN may be tasked with generating a road layout of an example scenario, and a second ANN may be tasked with generating movement patterns of one or more pedestrians or vehicles within the environment. Where feedback is discussed, and the behaviour of the sample data generator module 14 is said to be updated, this may apply to one or more of the ANN. The feedback may be such that it is itemised, so that different elements of the sample data are awarded different feedback - some of which may be positive, and others negative. In this way, multiple ANNs generating multiple aspects of the sample data may be updated in different ways based on positive feedback, negative feedback, or a respective mixture of positive and negative feedback. The third module is the sample data compliance module 16. The sample data compliance module 16 is configured to review the sample data being generated. In the context of the example described above, in which the sample data relates to driving scenarios, the sample data compliance module 16 effectively provides a realism check to ensure that the sample data being passed to the Al agent module 12 represents scenarios that are realistic. In other words, the sample data should be representative of situations that a vehicle may encounter in use when the Al agent is operating a vehicle. If the data is contrived in such a way that the Al agent is training on data that is unrealistic, and unrepresentative of a real-life situation, then this is undesirable. The sample data compliance module 16 preferably comprises one or more artificial neural networks (ANN). Where the sample data comprises multiple elements (as described above, for example, where the data describes an environment, and described the behaviour of actors within the environment), AN Ns may be trained on respective elements of the sample data. Since the sample data generator module 14 is encouraged to generate scenarios which the Al agent will fail to complete, if left unchecked, the sample data generator module 14 may produce unrealistic environments in which a real-life driver would be unable to achieve the task. Looking at the parking example where the sample data represents a configuration of a car park and a task of steering the vehicle safely into a free space, the sample data generator module 14 may be rewarded where the parking space is very narrow. At some point, where the width of the available space becomes as narrow as the width of the vehicle, and at that stage (and for any narrower space) the task becomes impossible to complete successfully. There is no benefit to the training process in using such scenarios as training data, as the Al agent will never be able to accomplish the task. One method of trying to ensure a good distribution of sample data across the search space is to seed the system with a set of scenarios designed to be spread across the entire search space (e.g. using sampling methods such as Monte Carlo, Latin Hypercubes, or Latin Hypercubes with Multidimensional Uniformity). However, by providing another adversarial ‘game’ using reinforcement learning techniques, the system 10 provides a self-balancing functionality, and so the purpose of the sample data compliance module 16, is to guard against the sample data becoming unrealistic in this way. In the context of the system 10, the sample data compliance module 16 receives a set of sample data 15’ from the sample data generator module 14. The sample data compliance module 16 determines whether the sample data 15’ passes a realism test in order for it to be found compliant. For example, the realism test is passed where the artificially generated sample data 15’ is found to be indistinguishable from ‘real world’ data. It should be noted that the sample data 15’ being tested for compliance, is the same data as the sample data 13’ being provided to the Al agent module 12 as described above. While in the above example of the narrow parking space, the task being set for the Al agent became impossible to complete successfully, there may be other cases in which the sample data generator module 14 is rewarded by the Al agent module 12 for generating scenarios that are not strictly impossible to complete, but which do not resemble realistic sample data. While it is important for the Al agent to be trained or tested across the breadth of the event space of situations it may encounter when deployed, focussing its training or testing on a decreasingly realistic scenario serves little benefit to its overall development or testing. Therefore, the important aspect is determining whether the sample data 15’ is sufficiently realistic. In other words, is the sample data 15’ distinguishable from "real” data - deemed to be representative of sensor data to be encountered by the Al agent during use. For example, the sample data compliance module 16 preferably has access to a data store 46’ of second sample data, such as sensor data captured of the objects or environments with which the Al agent will interact in use. In embodiments of the technology, the data relates to measurements of properties of objects or of an environment, based on captured sensor data. So, in the example of training an Al agent for driving an autonomous vehicle, the data store 46’ may contain scenarios resembling real world locations and / or road layouts, based on one or more of captured video footage, recorded map information, captured LIDAR sensor information, for example. It is of course not strictly necessary for the data store 46’ of second sample data to be of actual real captured data, but it should be data that has been assessed to be sufficiently realistic for the purpose of training or testing the Al agent. In embodiments of the technology, the sample data compliance module 16 provides a machine learning framework in which the module is trained to analyse an item of a set of sample data and to determine whether the item was generated by the sample data generator module 14 or whether the item is of the dataset of ‘real’ data - the second sample data. In other words, is the data item “real”, from captured sensor data or a source of data that is known to be sufficiently realistic - or has it been artificially generated by the sample data generator module 14? In embodiments of the technology, the sample data compliance module 16 is rewarded for correctly identifying the sample data items generated by the sample data generator module 14 and / or penalised for misidentifications, so that its performance in identifying the generated sample data improves. A false positive determination - in which a sample data item generated by the sample data generator module 14 is wrongly determined to comply with the realism test - may result in feedback to the sample data generator module 14 (i.e. it may be rewarded) and may also result in feedback to the sample data compliance module 16, so that it updates its behaviour in order to alter (correct) its determination when identifying similar items in future. False negative determinations - where an item is determined not to comply where it does in fact comply (i.e., belongs to the second sample data set and so should be found to pass the realism test) - may result in feedback to the sample data compliance module 16 to update is behaviour in order to alter (correct) its determination when identifying similar items in future. The sample data generator module 14 may be updated based on that determination of whether the items of the first sample data were found to pass the realism test, to improve the likelihood of future generated sample data passing the realism test. This can be achieved via a reward function or via a cost function, incentivising creation of data similar to that which was misclassified by the sample data compliance module 16, and / or penalising the creation of data similar to that which was correctly classified by the sample data compliance module 16. A feedback loop is formed between the sample data generator module 14 and the sample data compliance module 16. The sample data generator module 14 communicates a set of generated sample data 15’ to the sample data compliance module 16, for analysis. The sample data compliance module 16 subsequently communicates data 15” about the success or failure of the sample data compliance module 16 to recognise that the sample data 15’ had been artificially generated. This feedback may include a Boolean pass / fail, for example, or may be a value on a scale (e.g., a score representing a likelihood of that sample data being artificial as determined by the sample data compliance module 16). It should be noted that the feedback may include multiple components, each relating to a different element of the sample data. For example, sample data representing a road traffic scenario may be assessed according to multiple criteria including an assessment of whether the road layout is realistic, and whether the movement of dynamic actors (i.e., vehicles / pedestrians) within the environment is realistic. The sample data compliance module 16 may indicate that one aspect of the sample data is found to comply - to pass the realism test -while another aspect is found not to comply. Based on such feedback, the sample data generator module 14 may be rewarded for one aspect, and penalised for another aspect, of the data being generated. Where the sample data generator module 14 receives feedback that the sample data compliance module 16 failed to identify that the sample data was artificial, the sample data generator module 14 may be rewarded. In this way, the sample data generator module 14 is incentivised to generate sample data that is indistinguishable from the ‘real’ data (e.g., real world data). Where the feedback is that the sample data was recognised as being artificial, then the sample data generator module 14 is penalised. In this way, the sample data generator module 14 adapts to generate sample data that is less easily distinguished from ‘real world’ data, so that future iterations of sample data are more likely to be found to comply with the criterion. In broad terms, the sample data compliance module 16 is updated, based on whether the first sample data was found to pass the realism test, to alter the realism test, to reduce the likelihood of the generated sample data being determined to pass the realism test. In the context of the wider system 10, the system 10 operates by generating a set of first sample data using the sample data generator module 14, and then determining, using the sample data compliance module 16, whether items of that first sample data pass the realism test, to determine whether that data is “compliant” and ready for use by the Al agent module 12. A determination of non-compliance is made where items of the first sample data are found to be distinguishable from items of a set of second sample data accessible from the sample data compliance module 16, the second sample data being “real world” data - or at least, known to be sufficiently realistic for the purpose of the testing or training to take place. A determination of compliance is made where the items of first and second sample data are found not to be distinguishable, in which case the realism test is passed. So, in other words, the “realism test” is defined by the configuration of the sample data compliance module 16, and defines a classifier in the domain separating data to be labelled real (or at least ‘sufficiently real’ and suitable for use by the system 10) from data to be labelled artificial (or not sufficiently realistic). The realism test therefore varies as the machine learning model of the sample data compliance module 16 is updated over time. Note that reference to sample data being determined to be compliant, relates to it passing the realism test. Where the sample data compliance module 16 provides one or more ANNs for determining whether sample data passes the realism test, the outcome of the test is defined by the weights and biases mapped onto the neural network. Where the data compliance module 16 is updated to alter the realism test, it is typically these weights and biases that are updated. Where the first sample data is found to be non-compliant, feedback is provided to the sample data generator module 14 in order to update the configuration of the sample data generator module 14 in order to produce sample data that is more likely to be found to pass the realism test. The steps are subsequently repeated, as the sample data generator module 14 is itself trained to produce such sample data that can pass the test of the sample data compliance module 16, being found to pass the realism test. In embodiments of the technology, the requirement might be that all of the sample data generated must comply with the criterion, or else it may be required that a certain proportion of the sample data generated complies - above a predetermined threshold (for example 95%, or 90%, or 80%). As an example of training an Al agent, testing of the Al agent across an initial set of sample data covering a range of test cases may be performed to establish a model or characterisation of the Al agent behaviour at the outset of the process (i.e., where it is set up with an initial set of operating parameters). This initial set of sample data should be spread as evenly as possible across the range of possible permutations (i.e., across the search space). Once the training / testing has been performed and results obtained, the next stage is to model / characterise system behaviour, this model being key to predicting where it’s best to sample further test cases. This modelling may be achieved at the sample data generator module 14, using its first ANN to learn from the feedback received from the Al agent module 12. This ANN is then progressively updated as further sample data is processed, and further feedback received. In embodiments of the technology, in order to make the sample data generator module 14 function optimally, it is preferable for the sample data generator module 14 to be trained to a level at which reasonable data is being generated, prior to the involvement of the Al agent module 12. Therefore, typically, the sample data generator module 14 and sample data compliance module 16 may undertake many epochs of ‘training’ to improve the quality of the data being generated, before any sample data is communicated to the Al agent module 12 for testing or training an Al agent. In this way, particularly when training an Al agent, the ANNs are more likely to converge on suitable values based on greater consistency in the data being used. In other embodiments, the ANN(s) used by the sample data generator module 14 may be trained entirely through pre-existing datasets via backpropagation, with no ‘live’ reinforcement learning being required. In some embodiments of the technology, as discussed above, the sample data generator module 14 may be used as a test data generator module. In this setting, test data is generated for supporting the testing of an Al agent that has been pre-trained. In such a scenario, the Al agent does not gain further learning in response to the test data it is presented with by the test data generator module. In embodiments of the technology, the sample data compliance module 16 is augmented by, or even replaced by, algorithmic rules within the sample data generator module 14 to prohibit or discourage unrealistic sample data being generated. However, a downside of this approach is that it is difficult to write algorithms that would eliminate all unrealistic scenarios and not eliminate any realistic ones, and therefore a likely permutation is for such additional algorithms to support, rather than replace, the sample data compliance module 16. Example sample data, in addition to the parameters described above, may include for example a set of road characteristics, positions and speeds of other road users, to be used in the context of a predetermined simulation environment. In this context, the set of second sample data (i.e., the dataset of real scenarios) to be used by the sample data compliance module 16 for comparison would therefore need to capture the same parameters in the same format. An example of this would be ASAM OpenDrive + OpenScenario. An alternative is to adopt whatever format is used within the Al agent module 12 to pass data from the perception layer to the planning layer (e.g. an occupancy grid or list of tracked objects) - this would support training / testing the planning layer in isolation, bypassing the perception layer so it is not part of the process. In embodiments, the sample data may include realistic images or video footage. This may be multilayered to include simulated radar, lidar or ultrasonic ‘raw data’ as well as visual data, for example. In embodiments, the sample data may include instantaneous snapshots (analogous to photos) or scenarios that have a time-based aspect (analogous to videos) - latter is expected to be generally preferable. Generating footage for entire scenarios prior to using them to train / test the Al agent would only work for open-loop testing of the perception layer of the ANN of the agent, as the footage would not be able to adapt to vehicle behaviour (i.e., steering the vehicle would change the view and therefore the footage would fail to match the new situation). Training the perception layer in this way is still valuable. In some embodiments of the technology, the feedback loops between the sample data generator module 14 and the Al agent module 12 are set up so that at each timestep a scenario evolves. In this way, the sample data generator module 14 adds more variations to try to trip up the Al agent module 12, with the response of the Al agent module 12 at each timestep, and its feedback to the sample data generator module 14 influencing the scene it is presented with at the next timestep. This could take the form of a single extended drive containing many events generated in real-time to train and assess the system, rather than discrete scenarios. It is worth noting that the same sample data generated by the sample data generator module 14 is passed to both the sample data compliance module 16 and the Al agent module 12. However, in the latter case, it may not be passed directly to the Al agent module 12, and may instead be passed to simulation software that interprets scenario attributes based on the sample data in order to present a test scenario to the Al agent module 12. The data sent by the sample data generator module 14 to the sample data compliance module 16 and to the Al agent module 12 / Simulation software must be identical, although optionally may be captured in a different format. However, the data passed to the first sample data compliance module 16 must be both identical to and in the same format as the second sample data used for comparison within the sample data compliance module 16. For all three core modules 12, 14, 16, pretraining on generic data prior to commencing the adversarial ‘game’ between the subsystems, may help the system 10 to converge upon an acceptable result. This pretraining could potentially be undertaken using an alternative method to reinforcement learning (e.g. via supervised or unsupervised learning) and may be applied to any number of the modules 12, 14, 16. An alternative is to commence the adversarial game without any such pretraining for one or all modules, using an initial seed for the weights and biases of the ANN(s) (e.g. randomisation) that is not optimised via any form of training. It should be noted that the system 10 and methods described are equally applicable to automated driving (i.e. self-driving vehicles of SAE level 3 and above) and advanced driver assistance systems as seen on present-day production vehicles (i.e. SAE level 2 and below). Furthermore, it is also applicable to training and / or testing many other systems on road vehicles, or other types of vehicles, such as detecting terrain / road surface types to optimise the powertrain, or reacting to climatic conditions to automatically optimise the HVAC system, to name two of many possible examples. As mentioned elsewhere, the methods and systems described are also applicable to other applications involving Al agents, not necessarily relating to vehicles or autonomous driving systems. A data processing system 10, comprising at least a processor, is configured to perform the steps of the methods described herein, and a computer program product for execution to cause a computer to carry out the methods described herein is also provided. The three core modules 12, 14, 16 and their functionality as described, may be provided by one or more computing devices. We now describe suitable hardware, in which example the three modules are provided to run on separate computing devices. However, it should be readily understood that this need not be the case. With reference to Figure 4 of the drawings, an exemplary computing device is described in relation to the Al agent module 12. The module 12 provides at least one processing device 18, a memory device 20, has access to a storage device 22. Communication modules provide an input module 23 and output module 24 for sending and receiving data. The input and output modules 23, 24 may be provided by a single piece of hardware or alternatively by distinct input and output devices 23, 24. With reference to Figure 5 of the drawings, an exemplary computing device is described in relation to the test data generator module 14. The module 14 provides at least one processing device 32, a memory device 34, has access to a storage device 36. Communication modules provide an input module 38 and output module 40 for sending and receiving data, and as previously described, these input and output modules 38, 40 may be provided as a single unit. With reference to Figure 6 of the drawings, an exemplary computing device is described in relation to the sample data compliance module 16. The module 16 provides at least one processing device 42, a memory device 44, has access to a storage device 46. In embodiments of the technology, the module 16 may be in communication with a remote storage device 46’, for hosting sample data providing second sample data, for example. Again, as before, communication modules provide either distinct or a combined input module 48 and output module 50 for sending and receiving data. With reference to Figure 7 of the drawings, we describe an example method 100 of generating data for training or testing an Al agent. The method 100 broadly involves the steps of generating a set of first sample data (step 102) using the sample data generator module 14, and then subsequently using the sample data compliance module 16 to determine whether items of the first sample data pass the realism test (i.e., are found to be compliant) (step 104). A determination of non-compliance is made where items of the first sample data are found to be distinguishable from items of a set of second sample data accessible from the sample data compliance module 16. The second sample data being, as described above, is typically the data resembling real world data (i.e., sensor data), or has at least been confirmed as sufficiently realistic for the purposes of the system 10. The sample data compliance module 16 determines compliance where the items of first and second sample data are found not to be distinguishable. Where the first sample data is found to be non-compliant, the method involves updating the sample data generator module 14 (step 106); and where the first sample data is found to be compliant, updating the sample data compliance module 16 (step 108). Figure 8 of the drawings illustrates the flow of data through the system 10. The data flow through the system is best described in relation to two distinct feedback loops; one relating to improving the training of the Al agent through adversarial reinforcement learning between the Al agent module 12 and the sample data generator module 14, and the second relating to improving the realism of the sample data being generated, between the sample data generator module 14 and the sample data compliance module 16. Looking first at the right-hand feedback loop, we see that a first set of sample data is generated (step 102), and communicated 15’ to the sample data compliance module 16, and received (step 114) for processing. A determination is made (step 104) as to whether items of the first sample data pass the realism test (i.e., non-compliance is found where items of the first sample data are found to be distinguishable from items of the set of second sample data). The result of compliance determination is communicated 15” to the sample data generator module 14. The sample data generator module 14 is updated based on the outcome of the determination (step 106), either using a reward or a cost function, as described elsewhere. As an example, where the first sample data is found to be compliant, the sample data compliance module 16 is updated (step 108). This loop of generating sample data, checking for compliance, and then updating the sample data generator module 14 may continue through many iterations until data is produced that is found to be compliant, and suitable for use in training the Al agent. Where the first sample data is found to be compliant, it may be communicated 13’ to the Al agent module 12 for use in training or testing. Looking at the left-hand loop, we see that the first set of sample data, once found to be compliant, is communicated 13’ to the Al agent module 12. The sample data is subsequently used to train or test the Al agent (step 110). A determination is made (step 112) as to whether the Al agent performed successfully on the basis of the first sample data. Where the system 10 is training he Al agent, the Al agent module 12 is updated (step 120) to take account of its performance. Finally, the sample data generator module 14 receives feedback 13” from the Al agent module 12 based on the determination of success or failure, and the sample data generator module 14 is updated accordingly (step 106). With reference to Figure 9, we describe the method 122 of training or testing an Al agent, which involves generating a set of first sample data (step 102) using the sample data generator module 14. Subsequently, the sample data is provided to the Al agent module 12, and used to train or test the Al agent using those data (step 110). A determination is made (step 112) as to whether the Al agent performed successfully on the basis of the first sample data. In embodiments involving training the Al agent, the Al agent generator module 12 is then updated (step 120) to take account of its performance. Whether testing or training the agent, feedback is then provided from the Al agent module 12 to the sample data generator module 14, which is updated accordingly (step 106). In addition to embodiments described, which generally relate to use of an Al agent in controlling a vehicle, or in robotics, trained Al systems may be used in other settings such as systems for determining medical diagnosis / treatment, as an example. Such a system may involve generating artificial data relating to ailments - such as medical images for use in diagnosis, patient test results and medical history, and other forms of data commonly used for diagnostic or treatment purposes. That data may then be subjected to scrutiny to ensure it is representative of “real world” data in the manner described herein, and subsequently used in training an Al agent configured to suggest a diagnosis based on the data, or to propose a treatment based on the data. It is envisaged that the methods described can be applied to that setting, and more widely to other Al-based tools and applications not described explicitly herein. When used in this specification and claims, the terms "comprises" and "comprising" and variations thereof mean that the specified features, steps or integers are included. The terms are not to be interpreted to exclude the presence of other features, steps or components. The invention may also broadly consist in the parts, elements, steps, examples and / or features referred to or indicated in the specification individually or collectively in any and all combinations of two or more said parts, elements, steps, examples and / or features. In particular, one or more features in any of the embodiments described herein may be combined with one or more features from any other embodiment(s) described herein. Protection may be sought for any features disclosed in any one or more published documents referenced herein in combination with the present disclosure. Although certain example embodiments of the invention have been described, the scope of the appended claims is not intended to be limited solely to these embodiments. The claims are to be construed literally, purposively, and / or to encompass equivalents.

Claims

1. A computer-implemented method of training or testing an Al agent, comprising the following steps:5generating a set of first sample data using a sample data generator module;training or testing the Al agent using an Al agent module and the first sample data, and determining a success or failure of the Al agent in performing a task based on the first sample data; 10 andupdating the sample data generator module based on the determination of success or failure of the Al agent in performing the task based on the first sample data;wherein based on determining success of the Al agent in performing the task based on the 15 first sample data, the method comprises updating the sample data generator module so that further sample data is more likely to cause the Al agent to fail when performing a task based on the further sample data, and wherein the first sample data approximates a scenario fortesting or training behaviour within an environment.20 2. The method of claim 1, wherein the method further includes updating the Al agent modulebased on the determination of success or failure, to improve the Al agent performance at the task based on the first sample data.25 3. The method of any preceding claim, wherein the method further comprises the steps of:prior to training or testing the Al agent:using a sample data compliance module, determining whether items of the first30 sample data are distinguishable from items of a set of second sample data accessible fromthe sample data compliance module, where the sample data compliance module determines compliance of the first sample data where the items of first and second sample data are found not to be distinguishable;35 updating the sample data generator module based on the determination ofcompliance or non-compliance.24where the first sample data is found to be non-compliant, repeating the step of generating a set of first sample data to produce a further set of first sample data and repeating the step of determining whether items of that further set of first sample data are compliant.

55. The method of claim 3 or claim 4, wherein if the first sample data is found to be compliant, providing the first sample data to the Al agent module.

6. The method of any one of claims 3 to 5, wherein either:10where the first sample data are found to be compliant, updating the sample data compliance module to reduce the likelihood of those specific first sample data being determined to be compliant in future, using a cost function; or15 where the first sample data are found to be non-compliant, updating the sampledata compliance module to improve the likelihood of those first sample data being determined to be non-compliant in future, using a reward function.

7. The method of any one of claims 3 to 6, wherein either:20where the first sample data are found to be compliant, updating the sample data generator module to increase the likelihood of similar first sample data being generated in future, using a reward function; or25 where the first sample data are found to be non-compliant, updating the sampledata generator module to reduce the likelihood of similar first sample data being generated in future, using a cost function.

8. A computer-implemented method of generating data fortraining or testing an Al agent, 30 comprising the following steps:generating a set of first sample data using a sample data generator module;using a sample data compliance module, determining whether items of the first35 sample data are distinguishable from items of a set of second sample data accessible fromthe sample data compliance module, where the sample data compliance module determines compliance of the first sample data where the items of first and second sample data are found not to be distinguishable;24updating the sample data generator module based on the determination of compliance or non-compliance.

9. The method of claim 8, wherein if the first sample data is found to be compliant, providing 5 the first sample data to the Al agent module.

10. The method of claim 8 or claim 9, wherein either:where the first sample data are found to be compliant, updating the sample data10 compliance module to reduce the likelihood of those specific first sample data beingdetermined to be compliant in future, using a cost function; orwhere the first sample data are found to be non-compliant, updating the sample data compliance module to improve the likelihood of those first sample data being15 determined to be non-compliant in future, using a reward function.

11. The method of any one of claims 3 to 10, wherein the second sample data is generated from sensor data and relates to measurements of properties of objects or of an environment.20 12. The method of any one of claims 7 to 11, wherein the second sample data records ascenario fortesting or training behaviour within an environment, the record of the scenario including one or more of: a geographical layout of an environment, environmental conditions, parameters of static objects, and parameters of moving objects within the environment.25 13. A computer-implemented method of training or testing an Al agent, comprising thefollowing steps:generating a set of first sample data using a sample data generator module;30 using a sample data compliance module, determining whether items of the first sampledata pass a realism test,updating the sample data generator module based on the determination of whether the items of the first sample data were found to pass the realism test, to improve the likelihood of future 35 generated sample data passing the realism test,using the first sample data to train or test an Al agent,24determining a success or failure of the Al agent in performing a task based on the first sample data; andupdating the sample data generator module based on the determination of success or5 failure, to improve the likelihood of future generated sample data causing the Al agent to fail a task performed on that data.

14. The method of claim 13, further including updating the Al agent module based on the determination of success or failure, to improve the Al agent performance at the task based on the 10 first sample data.

15. The method of claim 13 or claim 14, further including updating the sample data compliance module to alter the realism test, to reduce the likelihood of the generated sample data being determined to pass the realism test.1516. The method of any preceding claim, wherein the sample data generator module uses an artificial neural network to generate the first sample data.

17. The method of any preceding claim, wherein the scenario for testing or training behaviour 20 within the environment includes one or more of: a geographical layout of the environment, parameters of static objects, and parameters of moving objects within the environment.

18. The method of any preceding claim, wherein the Al agent is configured to drive a vehicle, or to assist in driving a vehicle.2519. The method of any one of claims 1 to 17, wherein the Al agent is configured to operate a robotic device.

20. The method of any one of claims 1 to 10, wherein the Al agent is configured to detect 30 and / or classify an object based on sensor data, wherein the second sample data is generated from sensor data and relates to measurements of properties of the object, and the first sample data is generated by a neural network configured to approximate properties of the object.

21. A data processing system comprising a processor configured to perform the steps of any 35 one of claims 1 to 20.

22. A computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1 to 20.

Images